fabi/Sylpheed
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

re: decode the IDXD/IXUD record table — and there is no schema hash

Browse Source 
The binary region in front of the string pool was the parser's oldest open
note ("Not yet decoded"). It is a uniform 16-byte record array sorted by
name hash, a field count, a 12-byte field array sorted by key, a pool size,
and the pool. The trailing `pool_size == file_len - pool_base` identity makes
the layout self-checking, which is what caught the first wrong version.

Verified over the WHOLE disc with zero failures: 7750/7750 IDXD objects,
190782/190782 records reproducing their stored tag_hash, 1271462/1271462
named fields reproducing their key. IXUD is the same container with
ixud_hash, UTF-16BE and every offset in chars — 1104/1104 objects,
628165/628165 fields, checked with an independent parser.

Field names are stored on disc, so no preimage search is needed: a field's
middle word points at its own name. Only 504 fields disc-wide are hash-keyed
with no name; the other 1485073 nameless fields are positional, keyed by a
literal integer (line slots, movie ids).

Two long-held beliefs are WITHDRAWN:

* The word at 0x08 is not a schema hash. It is record 0's name_hash — the
  format has no type field at all, and an object's kind is known only from
  the caller that loads it. It survived as "schema" because tables of one
  kind share their lowest-hashed record name. Caught by a test asserting
  every movie id names a real record: 1005 -> STAGE10_PHASE01 failed because
  tag_hash("STAGE10_PHASE01") IS 0x067025B9, that table's supposed schema id.
* The field's middle word is not an always-0xFFFFFFFF flags word. It is
  0xFFFFFFFF for 54% of fields, enough to look constant in a small sample;
  the tell was that it is constant per key ACROSS records, which a per-record
  flag cannot be but a per-name pointer must.

`schema_hash` keeps its name rather than churn 33 call sites, with corrected
docs. The first sweep globbed dat/** and missed hidden/DefTables.pak (1425
objects); the test now walks the whole disc root.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-25 21:40:23 +00:00
parent fd113868bc
commit b411d03bd4
5 changed files with 685 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/re/BACKLOG.md
View File

@@ -1033,6 +1033,26 @@ premise was wrong.**
Candidates: the **7 `.embsec_` sections** (VAs 0x84D0000–0x86AC000, ~129 KB
total, executable) or a hashed record in `hidden/MiscBin.pak`. **Finding it
gives the actual per-phase clear condition for every stage.**
* ✅ **(2026-08-25) The IDXD/IXUD container is fully decoded** — the "binary
node/index region" in front of the string pool is a **uniform 16-byte record
array** `{name_hash, name_off, field_begin, field_end}` sorted by hash, then a
field count, **12-byte fields** `{key, name_off, value_off}` sorted by key, then
a pool size and the pool. Verified over the *whole* disc with zero failures:
IDXD **7 750/7 750** objects, **190 782/190 782** records, **1 271 462/1 271 462**
named fields; IXUD **1 104/1 104** objects, **628 165/628 165** fields (offsets in
chars). **Field names are stored on disc**, so no preimage search is needed —
only **504** fields disc-wide are hash-keyed with no name.
🔴 **Two corrections:** the header word at `0x08` is **not a schema hash**, it is
record 0's `name_hash` (7 750/7 750) — the format has no type field at all, so an
object's kind is known only from its loader; and the field's middle word is not
an `aux` flags word. See [`structures/idxd-container.md`](structures/idxd-container.md).
⚠️ My first disc sweep globbed `dat/**` and **missed `hidden/DefTables.pak`**
(1 425 objects); the test now walks the whole disc root.
▶️ **Follow-up now open:** the legacy value-before-key string-pool reader is an
*approximation* of the real table, and every number in this corpus that came out
of `get_f32`/`get_raw` is re-checkable against ground truth but **not yet
re-checked**. First step: diff the two readers across the disc and count
disagreements. Also open: recover the 504 unnamed hash keys.
* ✅ **(2026-08-25) Both guest hash routines located** — `sub_82447DF0` (IDXD)
and `sub_82447E70` (IXUD), transcribed instruction-for-instruction into Python
and Rust; `cargo test -p sylpheed-formats --lib hash` 10/10. **IXUD SOLVED:**

3
docs/re/INDEX.md
View File

@@ -12,7 +12,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
|--------|-------|-----------------------|-------|
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key — and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** — a field's middle word points at its own name — so nothing needs preimage search except the **504** fields disc-wide that are hash-keyed with no name; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) — the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟡 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked |
| XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ |
| T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) |
| RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** |
@@ -91,6 +91,7 @@ files, which is how the same ground got covered twice.
| [`ship-placement-capture-generalisation.md`](ship-placement-capture-generalisation.md) | Capital-ship placement — does the `e106` result generalise? (WIP, 2026-07-31) | 🚧 WIP, time-boxed session. Two results so far: a static audit across all |
| [`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) | Capital-ship part placement — runtime capture (ground truth) | ✅✅ STATIC ASSEMBLY IS EXACT — no captures needed anymore |
| [`structures/achievements.md`](structures/achievements.md) | Achievements — the 24-entry table, and where the earned state comes from | — |
| [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container — record/field table, and the two beliefs it withdraws | ✅ CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures |
| [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad — vtable `0x820B2A64` | ✅ CONFIRMED for the object layout and the atlas size, read live off |
| [`structures/mission-objective-counter.md`](structures/mission-objective-counter.md) | `REMAINING OB` — the mission's own objective counter, in RAM | ✅ CONFIRMED for one Stage 02 run: a big-endian u32 whose value |
| [`structures/movie-subtitles.md`](structures/movie-subtitles.md) | Movie subtitles & the movie ↔ mission ↔ text chain | — |

157
docs/re/structures/idxd-container.md Normal file
View File

@@ -0,0 +1,157 @@
# The IDXD container — record/field table
Status: ✅ **CONFIRMED**, decoded in full and verified over **the whole disc**
(not just `dat/` — `hidden/DefTables.pak` holds another 1425 objects, which the
first version of this sweep missed): **7750 / 7750**
objects parse, **190,782 / 190,782** records reproduce their stored name hash,
**1,271,462 / 1,271,462** named fields reproduce their key from their stored name.
Zero failures of any kind.
This closes the **"binary node/index region — not yet decoded"** note that stood
in `crates/sylpheed-formats/src/idxd.rs` for the whole life of the parser, and it
**demotes two beliefs the corpus was built on** (see *Two corrections* below).
Tests: `crates/sylpheed-formats/tests/idxd_records_disc.rs` —
`records_roundtrip_disc`, `first_header_word_is_record0_hash`,
`field_names_are_stored_disc`, `movie_table_ids_are_literal_field_keys`.
## ✅ The layout
All fields big-endian.
```text
Offset Size Field
0x00 4 "IDXD"
0x04 4 record_count n
0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
.. 4 field_count m (equals max(field_end))
.. 12*m fields { u32 key, u32 name_off, u32 value_off }
.. 4 pool_size (equals file_len - pool_base)
.. .. string pool (every *_off above is a byte offset from here)
```
* **Records are sorted ascending by `name_hash`** and the guest binary-searches
them with a 16-byte stride (`sub_82448AA0`). Verified: sorted in every object.
* `name_hash` is [`tag_hash`](idxd-tag-hash.md) of the record's **own name** —
not the pak TOC hash (different modulus, and not lowercased).
* A record owns the half-open field range `[field_begin, field_end)`. Ranges are
*not* contiguous in record order — records are in hash order while their fields
sit in name order, so the field array is shared, not partitioned by position.
* **Fields are sorted ascending by `key`** and lower-bounded (`sub_8244E338`).
* `field_count` and `pool_size` are what make the layout self-checking: a wrong
record stride lands on a `pool_size` that does not equal the remaining bytes.
That identity is what caught the first wrong version of this decode.
## ✅ Field names are on the disc
A field's middle word is **its own name's pool offset**, and then
`key == tag_hash(name)`. Field names never have to be recovered from their hash.
`name_off == 0xFFFF_FFFF` means the field has **no name**; its `key` is then a
literal positional integer — a line-slot index (`0,1,2,3`), a movie id (`105`,
`1303`). So a field is positional *exactly* when it stores no name; you do not
have to guess from the key's magnitude.
Disc-wide census of all **2,757,039** fields:
| kind | count |
|---|---|
| named, `key == tag_hash(name)` | 1,271,462 |
| unnamed, literal key (`< 0x10000`) | 1,485,073 |
| unnamed, hash-shaped key | 504 |
Those **504** are the entire remaining preimage problem on the disc. ❔ Their
names are unrecovered.
## Two corrections
### ❌ WITHDRAWN — "the word at `0x08` is a schema hash"
The corpus (and `IdxdObject::schema_hash`, and every `pak list` line printing
`schema 067025b9`) read the third header word as an object-type id whose preimage
was unknown. **It is not a schema id.** It is simply **record 0's `name_hash`**:
the record array is uniform 16-byte entries starting at `0x08`, and the header has
no type field at all.
Evidence: `tag_hash(records[0].name) == word@0x08` for **all 7750** objects on the
disc, zero exceptions (`first_header_word_is_record0_hash`).
How it was caught, which is the useful part: a test asserted that every movie id in
`BASE_INFO` names a real record, and one — `1005 -> "STAGE10_PHASE01"` — did not
resolve. The reason was that `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, the
movie table's supposed schema id. A "coincidence" at 1-in-2^32 is not a
coincidence; the record was being eaten by the header.
It still *works* as a type discriminator, because tables of one kind share their
lowest-hashed record name — which is exactly why it went unquestioned for so long.
`schema_hash` is therefore kept under its established name, with its docs corrected,
rather than renamed across 33 call sites. **Nothing on disc names an object's type.**
### ❌ WITHDRAWN — "the field's middle word is an `aux`/flags word, always `0xFFFFFFFF`"
It is `0xFFFFFFFF` for 54% of fields, which is enough to look constant in a small
sample. It is a name offset (above). The tell was that the word is constant *per
key across records* (`key=0x6c43a78d` always carried `1743`) — a per-record flag
cannot do that, but a per-name pointer must.
## Worked example — the movie table
`dat/tables.pak`, the object whose record 0 is `STAGE10_PHASE01` (105 records,
433 fields). Its `BASE_INFO` record mixes both field kinds:
```text
named: PATH = "dat\movie\" VERSION = "0x060329" SUBTITLE_FONT …
positional: 105 -> "STAGE01_PHASE01" 205 -> "STAGE02_PHASE01" 1005 -> "STAGE10_PHASE01"
```
Each positional value names another record in the same object, which carries the
real files:
```text
STAGE02_PHASE01: MOVIE = "RT02A.wmv" VOICETRACK = "VOICE_RT02A"
SUBTITLE = "…+SUBTITLE_RT02A.tbl" TELOP = "…+pwrt02.prt"
```
All 104 positional ids resolve to a real record — verified, no dangling entries.
This is the id space the mission script's cutscene request uses; see
[movie-subtitle-link](../movie-subtitle-link.md).
## ✅ `IXUD` is the same container
The wide-string sibling has an identical shape, with three substitutions: the hash
is `ixud_hash`, strings are UTF-16BE, and **every offset — record name, field name
and value alike — is in 16-bit chars**, so `pool_base + 2*off`. `pool_size` is
likewise a char count, which is the same identity as the already-known
`STR + 2*strsize == filesize` ([ixud-localised-text](ixud-localised-text.md)).
Verified independently over all **1104** IXUD objects on the disc: the header word
at `0x08` is record 0's `ixud_hash` (1104/1104), and `key == ixud_hash(field name)`
for **628,165 / 628,165** named fields, zero mismatches. Only 48 fields are
unnamed — 6 objects × 8, all in a `CATEGORY_DESC` record with keys `0..7`, a
positional array of weapon-category descriptions.
One extra rule shows up in IXUD's comparator (`sub_82447F38`) and is worth assuming
for IDXD too: `name_off == 0xFFFF_FFFF` is the **primary** sort key, so a record's
field slice is *unnamed fields first, then named fields*, each key-ascending
(1476/1476 slices). That is why there are two getters — lookup-by-integer searches
only the unnamed run, lookup-by-name only the named run.
🟡 The loader `sub_82448D00` also accepts legacy magics `IIDX`, `IDX2`, `IDX3`,
`IDXC` and rejects `IDXD` on that path with `"Old virsion binary table. Not
supported."` [sic]. **None of them occur on this disc** (magic census over every
pak entry: `IDXD` 7750, `T8aD` 4525, `RATC` 2985, `IXUD` 1104, `LSTA` 64, and zero
of the four legacy magics), so that path is read-only knowledge, untestable here.
## What this does *not* settle
* ❔ The names of the 504 unnamed hash-keyed fields.
* ❔ How much of the existing corpus the old string-pool reader got wrong. The
legacy `get_f32`/`get_raw` path infers a field's value from *pool adjacency*
(`<value>\0<key>\0`). That adjacency is a consequence of the field table, not a
rule of the format, and it cannot represent a field whose value string is shared
or reordered. Every number in `docs/re/` that came from it is now re-checkable
against the true table, and has not yet been re-checked.
* ❔ Whether `IXUD`, the wide-string sibling, uses the same shape. Its offsets are
in 16-bit chars and its hash is different ([ixud-localised-text](ixud-localised-text.md)).
* The type-identification question is now open rather than closed: with no schema
field, an object's kind is known only from the caller that loads it.