Sylpheed/docs/re/structures/idxd-container.md

# The IDXD container — record/field table

Status: ✅ **CONFIRMED**, decoded in full and verified over **the whole disc**
(not just `dat/` — `hidden/DefTables.pak` holds another 1425 objects, which the
first version of this sweep missed): **7750 / 7750**
objects parse, **190,782 / 190,782** records reproduce their stored name hash,
**1,271,462 / 1,271,462** named fields reproduce their key from their stored name.
Zero failures of any kind.

This closes the **"binary node/index region — not yet decoded"** note that stood
in `crates/sylpheed-formats/src/idxd.rs` for the whole life of the parser, and it
**demotes two beliefs the corpus was built on** (see *Two corrections* below).

Tests: `crates/sylpheed-formats/tests/idxd_records_disc.rs` —
`records_roundtrip_disc`, `first_header_word_is_record0_hash`,
`field_names_are_stored_disc`, `movie_table_ids_are_literal_field_keys`.

## ✅ The layout

All fields big-endian.

```text
Offset   Size    Field
0x00     4       "IDXD"
0x04     4       record_count  n
0x08     16*n    records      { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
..       4       field_count  m       (equals max(field_end))
..       12*m    fields       { u32 key, u32 name_off, u32 value_off }
..       4       pool_size            (equals file_len - pool_base)
..       ..      string pool          (every *_off above is a byte offset from here)
```

* **Records are sorted ascending by `name_hash`** and the guest binary-searches
  them with a 16-byte stride (`sub_82448AA0`). Verified: sorted in every object.
* `name_hash` is [`tag_hash`](idxd-tag-hash.md) of the record's **own name** —
  not the pak TOC hash (different modulus, and not lowercased).
* A record owns the half-open field range `[field_begin, field_end)`. Ranges are
  *not* contiguous in record order — records are in hash order while their fields
  sit in name order, so the field array is shared, not partitioned by position.
* **Fields are sorted ascending by `key`** and lower-bounded (`sub_8244E338`).
* `field_count` and `pool_size` are what make the layout self-checking: a wrong
  record stride lands on a `pool_size` that does not equal the remaining bytes.
  That identity is what caught the first wrong version of this decode.

## ✅ Field names are on the disc

A field's middle word is **its own name's pool offset**, and then
`key == tag_hash(name)`. Field names never have to be recovered from their hash.

`name_off == 0xFFFF_FFFF` means the field has **no name**; its `key` is then a
literal positional integer — a line-slot index (`0,1,2,3`), a movie id (`105`,
`1303`). So a field is positional *exactly* when it stores no name; you do not
have to guess from the key's magnitude.

Disc-wide census of all **2,757,039** fields:

| kind | count |
|---|---|
| named, `key == tag_hash(name)` | 1,271,462 |
| unnamed, literal key (`< 0x10000`) | 1,485,073 |
| unnamed, hash-shaped key | 504 |

Those **504** are the entire remaining preimage problem on the disc — but the
count is easy to misread, so state it precisely: 504 is a count of **field
entries**, not of distinct names. They are **42 distinct keys**, each appearing
in 12 places (six byte-identical language copies of one object × two records).
The preimage target was never 504 names; it is 42.

❔ Their names are unrecovered, and
[idxd-unnamed-keys.md](idxd-unnamed-keys.md) shows why that is not a matter of
more compute: the 42 belong to `<lang>\script\ID.tbl`, and at 7 characters one
of them already has 1 176 preimages under a 24-bit modulus. 30 of the 42 have
their trailing digits pinned algebraically; the prefixes cannot be recovered
from the hash alone.

## Two corrections

### ❌ WITHDRAWN — "the word at `0x08` is a schema hash"

The corpus (and `IdxdObject::schema_hash`, and every `pak list` line printing
`schema 067025b9`) read the third header word as an object-type id whose preimage
was unknown. **It is not a schema id.** It is simply **record 0's `name_hash`**:
the record array is uniform 16-byte entries starting at `0x08`, and the header has
no type field at all.

Evidence: `tag_hash(records[0].name) == word@0x08` for **all 7750** objects on the
disc, zero exceptions (`first_header_word_is_record0_hash`).

How it was caught, which is the useful part: a test asserted that every movie id in
`BASE_INFO` names a real record, and one — `1005 -> "STAGE10_PHASE01"` — did not
resolve. The reason was that `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, the
movie table's supposed schema id. A "coincidence" at 1-in-2^32 is not a
coincidence; the record was being eaten by the header.

It still *works* as a type discriminator, because tables of one kind share their
lowest-hashed record name — which is exactly why it went unquestioned for so long.
`schema_hash` is therefore kept under its established name, with its docs corrected,
rather than renamed across 33 call sites. **Nothing on disc names an object's type.**

### ❌ WITHDRAWN — "the field's middle word is an `aux`/flags word, always `0xFFFFFFFF`"

It is `0xFFFFFFFF` for 54% of fields, which is enough to look constant in a small
sample. It is a name offset (above). The tell was that the word is constant *per
key across records* (`key=0x6c43a78d` always carried `1743`) — a per-record flag
cannot do that, but a per-name pointer must.

## Worked example — the movie table

`dat/tables.pak`, the object whose record 0 is `STAGE10_PHASE01` (105 records,
433 fields). Its `BASE_INFO` record mixes both field kinds:

```text
named:       PATH = "dat\movie\"   VERSION = "0x060329"   SUBTITLE_FONT …
positional:  105 -> "STAGE01_PHASE01"   205 -> "STAGE02_PHASE01"   1005 -> "STAGE10_PHASE01"
```

Each positional value names another record in the same object, which carries the
real files:

```text
STAGE02_PHASE01:  MOVIE = "RT02A.wmv"   VOICETRACK = "VOICE_RT02A"
                  SUBTITLE = "…+SUBTITLE_RT02A.tbl"   TELOP = "…+pwrt02.prt"
```

All 104 positional ids resolve to a real record — verified, no dangling entries.
This is the id space the mission script's cutscene request uses; see
[movie-subtitle-link](../movie-subtitle-link.md).

## ✅ `IXUD` is the same container

The wide-string sibling has an identical shape, with three substitutions: the hash
is `ixud_hash`, strings are UTF-16BE, and **every offset — record name, field name
and value alike — is in 16-bit chars**, so `pool_base + 2*off`. `pool_size` is
likewise a char count, which is the same identity as the already-known
`STR + 2*strsize == filesize` ([ixud-localised-text](ixud-localised-text.md)).

Verified independently over all **1104** IXUD objects on the disc: the header word
at `0x08` is record 0's `ixud_hash` (1104/1104), and `key == ixud_hash(field name)`
for **628,165 / 628,165** named fields, zero mismatches. Only 48 fields are
unnamed — 6 objects × 8, all in a `CATEGORY_DESC` record with keys `0..7`, a
positional array of weapon-category descriptions.

One extra rule shows up in IXUD's comparator (`sub_82447F38`) and is worth assuming
for IDXD too: `name_off == 0xFFFF_FFFF` is the **primary** sort key, so a record's
field slice is *unnamed fields first, then named fields*, each key-ascending
(1476/1476 slices). That is why there are two getters — lookup-by-integer searches
only the unnamed run, lookup-by-name only the named run.

🟡 The loader `sub_82448D00` also accepts legacy magics `IIDX`, `IDX2`, `IDX3`,
`IDXC` and rejects `IDXD` on that path with `"Old virsion binary table. Not
supported."` [sic]. **None of them occur on this disc** (magic census over every
pak entry: `IDXD` 7750, `T8aD` 4525, `RATC` 2985, `IXUD` 1104, `LSTA` 64, and zero
of the four legacy magics), so that path is read-only knowledge, untestable here.

## What this does *not* settle

* ❔ The names of the 42 unnamed hash-keyed fields (504 entries — see
  [idxd-unnamed-keys.md](idxd-unnamed-keys.md); shown to be unrecoverable from
  the hash alone).
* ❔ How much of the existing corpus the old string-pool reader got wrong. The
  legacy `get_f32`/`get_raw` path infers a field's value from *pool adjacency*
  (`<value>\0<key>\0`). That adjacency is a consequence of the field table, not a
  rule of the format, and it cannot represent a field whose value string is shared
  or reordered. Every number in `docs/re/` that came from it is now re-checkable
  against the true table, and has not yet been re-checked.
* ❔ Whether `IXUD`, the wide-string sibling, uses the same shape. Its offsets are
  in 16-bit chars and its hash is different ([ixud-localised-text](ixud-localised-text.md)).
* The type-identification question is now open rather than closed: with no schema
  field, an object's kind is known only from the caller that loads it.

## ✅ `Enumeration` — a record that indexes its own object (2026-08-28)

**360 of the disc's 7 750 IDXD objects carry a record named `Enumeration`.** Its
fields are all valueless (`(tag, name, "")`), and:

```
its field names == the object's other record names, EXACTLY   360 / 360
   a strict subset of them                                      0
   naming something not in the object                           0
```

The matching field/record counts run from 2 up past 25 — `(3,3) ×36`, `(9,9) ×24`,
`(13,13) ×24`, `(2,2) ×24`, `(25,25) ×18`, `(16,16) ×12` — and **every one is on
the diagonal**. A field list that merely overlapped would not do that at 360/360
across that range of sizes.

So `Enumeration` is a **table of contents for the object it lives in**. The kind
question above ("an object's kind is known only from the caller that loads it")
gets a partial answer for these 360: the object states its own contents.

⚠️ **Do not confuse it with `Enumerate`**, one letter shorter, which
[[archive-naming]] uses as the route-2 seed: an object whose *single* record is
named `Enumerate` and whose fields name **other objects** (resolving as
`name_hash("<name>.tbl")`). Same idea, opposite scope — self-index versus
declaration table — and the spelling is the only thing that distinguishes them.

[[mission-wave-arrivals]] counted 28 `Enumeration_*` records while filtering
non-route records out of a speed measurement; what it does not say is what they
are.

### ❌ The self-index is not a naming route — measured, 0 gained

Tempting next step: 360 objects state their own contents, so can that name the
objects themselves? **No.**

```
objects carrying an `Enumeration` record                       360
   key already named by archive-naming routes 1 + 2            348
   still unnamed                                                12
   newly named by name_hash(prefix + own record name + ext)      0
```

Sixteen path prefixes × seven extensions × every record name of every unnamed
object: **zero**. A self-index names an object's **records**; a TOC key is
`name_hash(path + file name)`. The two vocabularies never meet, and no amount of
record names reaches a file name. A measured negative.

✅ **The residual is exactly two objects, each repeated once per language pack,
and each is an unnamed variant of a named `_Test` table** — byte-identical across
all six packs, and *not* a duplicate of its named counterpart:

| key | size | records | its named counterpart |
|---|---:|---|---|
| `2390212806` | 1 493 B | `Placement_00…02`, `RouteTest_00…01` | `Stage\Null_Test.tbl` = `3121167452`, 27 545 B — **the same five records plus four `Frame_Test_Asteroid_*`** |
| `2719765792` | 14 124 B | `Formation_100`, `Formation_Fleet_01`, `Formation_Fleet_02` | `Formation_Test.tbl` = `3463590559`, **14 025 B — the same three record names, a different payload** |

So one is a trimmed `Null_Test`, the other a second `Formation_Test`-shaped file.
Neither name resolves under any of the sixteen prefixes with the eight candidate
spellings tried.

This also confirms [[stage-mission-tables]]'s attribution independently:
`name_hash(prefix + "Formation_Test.tbl")` → key `3463590559`, whose records are
exactly `Formation_100`, `Formation_Fleet_01`, `Formation_Fleet_02` — the two
frame-ratio exceptions really do live in the file it names.

❔ One control threw an unexplained miss: **`EnumSquadron_Test.tbl`**, named by the
`_Test` stage record, matches **no pak key** under any of the sixteen prefixes.
n = 1, not chased.