fabi/Syplheed-Reborn
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re: decode the IDXD/IXUD record table — and there is no schema hash

Browse Source 
The binary region in front of the string pool was the parser's oldest open
note ("Not yet decoded"). It is a uniform 16-byte record array sorted by
name hash, a field count, a 12-byte field array sorted by key, a pool size,
and the pool. The trailing `pool_size == file_len - pool_base` identity makes
the layout self-checking, which is what caught the first wrong version.

Verified over the WHOLE disc with zero failures: 7750/7750 IDXD objects,
190782/190782 records reproducing their stored tag_hash, 1271462/1271462
named fields reproducing their key. IXUD is the same container with
ixud_hash, UTF-16BE and every offset in chars — 1104/1104 objects,
628165/628165 fields, checked with an independent parser.

Field names are stored on disc, so no preimage search is needed: a field's
middle word points at its own name. Only 504 fields disc-wide are hash-keyed
with no name; the other 1485073 nameless fields are positional, keyed by a
literal integer (line slots, movie ids).

Two long-held beliefs are WITHDRAWN:

* The word at 0x08 is not a schema hash. It is record 0's name_hash — the
  format has no type field at all, and an object's kind is known only from
  the caller that loads it. It survived as "schema" because tables of one
  kind share their lowest-hashed record name. Caught by a test asserting
  every movie id names a real record: 1005 -> STAGE10_PHASE01 failed because
  tag_hash("STAGE10_PHASE01") IS 0x067025B9, that table's supposed schema id.
* The field's middle word is not an always-0xFFFFFFFF flags word. It is
  0xFFFFFFFF for 54% of fields, enough to look constant in a small sample;
  the tell was that it is constant per key ACROSS records, which a per-record
  flag cannot be but a per-name pointer must.

`schema_hash` keeps its name rather than churn 33 call sites, with corrected
docs. The first sweep globbed dat/** and missed hidden/DefTables.pak (1425
objects); the test now walks the whole disc root.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-25 21:40:23 +00:00
parent 34c99a1568
commit af32540190
5 changed files with 685 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/re/BACKLOG.md
View File

@@ -1033,6 +1033,26 @@ premise was wrong.**
Candidates: the **7 `.embsec_` sections** (VAs 0x84D0000–0x86AC000, ~129 KB
total, executable) or a hashed record in `hidden/MiscBin.pak`. **Finding it
gives the actual per-phase clear condition for every stage.**
* ✅ **(2026-08-25) The IDXD/IXUD container is fully decoded** — the "binary
node/index region" in front of the string pool is a **uniform 16-byte record
array** `{name_hash, name_off, field_begin, field_end}` sorted by hash, then a
field count, **12-byte fields** `{key, name_off, value_off}` sorted by key, then
a pool size and the pool. Verified over the *whole* disc with zero failures:
IDXD **7 750/7 750** objects, **190 782/190 782** records, **1 271 462/1 271 462**
named fields; IXUD **1 104/1 104** objects, **628 165/628 165** fields (offsets in
chars). **Field names are stored on disc**, so no preimage search is needed —
only **504** fields disc-wide are hash-keyed with no name.
🔴 **Two corrections:** the header word at `0x08` is **not a schema hash**, it is
record 0's `name_hash` (7 750/7 750) — the format has no type field at all, so an
object's kind is known only from its loader; and the field's middle word is not
an `aux` flags word. See [`structures/idxd-container.md`](structures/idxd-container.md).
⚠️ My first disc sweep globbed `dat/**` and **missed `hidden/DefTables.pak`**
(1 425 objects); the test now walks the whole disc root.
▶️ **Follow-up now open:** the legacy value-before-key string-pool reader is an
*approximation* of the real table, and every number in this corpus that came out
of `get_f32`/`get_raw` is re-checkable against ground truth but **not yet
re-checked**. First step: diff the two readers across the disc and count
disagreements. Also open: recover the 504 unnamed hash keys.
* ✅ **(2026-08-25) Both guest hash routines located** — `sub_82447DF0` (IDXD)
and `sub_82447E70` (IXUD), transcribed instruction-for-instruction into Python
and Rust; `cargo test -p sylpheed-formats --lib hash` 10/10. **IXUD SOLVED:**