# The IDXD container — record/field table Status: ✅ **CONFIRMED**, decoded in full and verified over **the whole disc** (not just `dat/` — `hidden/DefTables.pak` holds another 1425 objects, which the first version of this sweep missed): **7750 / 7750** objects parse, **190,782 / 190,782** records reproduce their stored name hash, **1,271,462 / 1,271,462** named fields reproduce their key from their stored name. Zero failures of any kind. This closes the **"binary node/index region — not yet decoded"** note that stood in `crates/sylpheed-formats/src/idxd.rs` for the whole life of the parser, and it **demotes two beliefs the corpus was built on** (see *Two corrections* below). Tests: `crates/sylpheed-formats/tests/idxd_records_disc.rs` — `records_roundtrip_disc`, `first_header_word_is_record0_hash`, `field_names_are_stored_disc`, `movie_table_ids_are_literal_field_keys`. ## ✅ The layout All fields big-endian. ```text Offset Size Field 0x00 4 "IDXD" 0x04 4 record_count n 0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end } .. 4 field_count m (equals max(field_end)) .. 12*m fields { u32 key, u32 name_off, u32 value_off } .. 4 pool_size (equals file_len - pool_base) .. .. string pool (every *_off above is a byte offset from here) ``` * **Records are sorted ascending by `name_hash`** and the guest binary-searches them with a 16-byte stride (`sub_82448AA0`). Verified: sorted in every object. * `name_hash` is [`tag_hash`](idxd-tag-hash.md) of the record's **own name** — not the pak TOC hash (different modulus, and not lowercased). * A record owns the half-open field range `[field_begin, field_end)`. Ranges are *not* contiguous in record order — records are in hash order while their fields sit in name order, so the field array is shared, not partitioned by position. * **Fields are sorted ascending by `key`** and lower-bounded (`sub_8244E338`). * `field_count` and `pool_size` are what make the layout self-checking: a wrong record stride lands on a `pool_size` that does not equal the remaining bytes. That identity is what caught the first wrong version of this decode. ## ✅ Field names are on the disc A field's middle word is **its own name's pool offset**, and then `key == tag_hash(name)`. Field names never have to be recovered from their hash. `name_off == 0xFFFF_FFFF` means the field has **no name**; its `key` is then a literal positional integer — a line-slot index (`0,1,2,3`), a movie id (`105`, `1303`). So a field is positional *exactly* when it stores no name; you do not have to guess from the key's magnitude. Disc-wide census of all **2,757,039** fields: | kind | count | |---|---| | named, `key == tag_hash(name)` | 1,271,462 | | unnamed, literal key (`< 0x10000`) | 1,485,073 | | unnamed, hash-shaped key | 504 | Those **504** are the entire remaining preimage problem on the disc — but the count is easy to misread, so state it precisely: 504 is a count of **field entries**, not of distinct names. They are **42 distinct keys**, each appearing in 12 places (six byte-identical language copies of one object × two records). The preimage target was never 504 names; it is 42. ❔ Their names are unrecovered, and [idxd-unnamed-keys.md](idxd-unnamed-keys.md) shows why that is not a matter of more compute: the 42 belong to `\script\ID.tbl`, and at 7 characters one of them already has 1 176 preimages under a 24-bit modulus. 30 of the 42 have their trailing digits pinned algebraically; the prefixes cannot be recovered from the hash alone. ## Two corrections ### ❌ WITHDRAWN — "the word at `0x08` is a schema hash" The corpus (and `IdxdObject::schema_hash`, and every `pak list` line printing `schema 067025b9`) read the third header word as an object-type id whose preimage was unknown. **It is not a schema id.** It is simply **record 0's `name_hash`**: the record array is uniform 16-byte entries starting at `0x08`, and the header has no type field at all. Evidence: `tag_hash(records[0].name) == word@0x08` for **all 7750** objects on the disc, zero exceptions (`first_header_word_is_record0_hash`). How it was caught, which is the useful part: a test asserted that every movie id in `BASE_INFO` names a real record, and one — `1005 -> "STAGE10_PHASE01"` — did not resolve. The reason was that `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, the movie table's supposed schema id. A "coincidence" at 1-in-2^32 is not a coincidence; the record was being eaten by the header. It still *works* as a type discriminator, because tables of one kind share their lowest-hashed record name — which is exactly why it went unquestioned for so long. `schema_hash` is therefore kept under its established name, with its docs corrected, rather than renamed across 33 call sites. **Nothing on disc names an object's type.** ### ❌ WITHDRAWN — "the field's middle word is an `aux`/flags word, always `0xFFFFFFFF`" It is `0xFFFFFFFF` for 54% of fields, which is enough to look constant in a small sample. It is a name offset (above). The tell was that the word is constant *per key across records* (`key=0x6c43a78d` always carried `1743`) — a per-record flag cannot do that, but a per-name pointer must. ## Worked example — the movie table `dat/tables.pak`, the object whose record 0 is `STAGE10_PHASE01` (105 records, 433 fields). Its `BASE_INFO` record mixes both field kinds: ```text named: PATH = "dat\movie\" VERSION = "0x060329" SUBTITLE_FONT … positional: 105 -> "STAGE01_PHASE01" 205 -> "STAGE02_PHASE01" 1005 -> "STAGE10_PHASE01" ``` Each positional value names another record in the same object, which carries the real files: ```text STAGE02_PHASE01: MOVIE = "RT02A.wmv" VOICETRACK = "VOICE_RT02A" SUBTITLE = "…+SUBTITLE_RT02A.tbl" TELOP = "…+pwrt02.prt" ``` All 104 positional ids resolve to a real record — verified, no dangling entries. This is the id space the mission script's cutscene request uses; see [movie-subtitle-link](../movie-subtitle-link.md). ## ✅ `IXUD` is the same container The wide-string sibling has an identical shape, with three substitutions: the hash is `ixud_hash`, strings are UTF-16BE, and **every offset — record name, field name and value alike — is in 16-bit chars**, so `pool_base + 2*off`. `pool_size` is likewise a char count, which is the same identity as the already-known `STR + 2*strsize == filesize` ([ixud-localised-text](ixud-localised-text.md)). Verified independently over all **1104** IXUD objects on the disc: the header word at `0x08` is record 0's `ixud_hash` (1104/1104), and `key == ixud_hash(field name)` for **628,165 / 628,165** named fields, zero mismatches. Only 48 fields are unnamed — 6 objects × 8, all in a `CATEGORY_DESC` record with keys `0..7`, a positional array of weapon-category descriptions. One extra rule shows up in IXUD's comparator (`sub_82447F38`) and is worth assuming for IDXD too: `name_off == 0xFFFF_FFFF` is the **primary** sort key, so a record's field slice is *unnamed fields first, then named fields*, each key-ascending (1476/1476 slices). That is why there are two getters — lookup-by-integer searches only the unnamed run, lookup-by-name only the named run. 🟡 The loader `sub_82448D00` also accepts legacy magics `IIDX`, `IDX2`, `IDX3`, `IDXC` and rejects `IDXD` on that path with `"Old virsion binary table. Not supported."` [sic]. **None of them occur on this disc** (magic census over every pak entry: `IDXD` 7750, `T8aD` 4525, `RATC` 2985, `IXUD` 1104, `LSTA` 64, and zero of the four legacy magics), so that path is read-only knowledge, untestable here. ## What this does *not* settle * ❔ The names of the 42 unnamed hash-keyed fields (504 entries — see [idxd-unnamed-keys.md](idxd-unnamed-keys.md); shown to be unrecoverable from the hash alone). * ❔ How much of the existing corpus the old string-pool reader got wrong. The legacy `get_f32`/`get_raw` path infers a field's value from *pool adjacency* (`\0\0`). That adjacency is a consequence of the field table, not a rule of the format, and it cannot represent a field whose value string is shared or reordered. Every number in `docs/re/` that came from it is now re-checkable against the true table, and has not yet been re-checked. * ❔ Whether `IXUD`, the wide-string sibling, uses the same shape. Its offsets are in 16-bit chars and its hash is different ([ixud-localised-text](ixud-localised-text.md)). * The type-identification question is now open rather than closed: with no schema field, an object's kind is known only from the caller that loads it.