re: wire the IXUD record table into the crate — captions go 537 to 8800 of 8800
ixud.rs now has an IdxdObject-shaped reader, IxudObject, and build_caption_text reads captions as FIELDS instead of pairing them with whatever token follows in the pool. build_demo_text token adjacency 134 ids 537 lines build_caption_text token adjacency 3721 8074 build_caption_text record fields 4085 8800 = all of them Verified over the whole disc by tests/ixud_records_disc.rs: 1104/1104 objects parse, 1476/1476 records and 628165/628165 named fields reproduce their ixud_hash, 48 positional, zero failures. The header word at 0x08 is record 0's hash, asserted per object -- there is no schema field, exactly as for IDXD. The module doc described a 12-byte record directory and a "schema/type hash"; both were wrong and are corrected. I also have to correct my own number from the previous commit. "1.3% of the game's text" counted OCCURRENCES: each family lives in 24-45 IXUD blocks and the same key repeats across them. Distinct text-bearing MSG_* keys number 8800, not 44579, and every one has the <id>_<page>_<line> shape. So the real coverage was 537/8800 = 6.1%, and I overstated the gap about fivefold. Direction right, magnitude wrong. The DEMO control is the sharpest evidence for the change: token adjacency finds 537 lines there, the field reader 541. It was dropping lines even in the one family it was written for -- which is why the test now asserts "must not lose lines" rather than "must be identical". Same lesson twice in one session: pool adjacency is a consequence of how records are written, not a rule of the format. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
@@ -311,7 +311,7 @@ Message_044: voice = VOICE_E_012B
|
||||
`VOICE_E_012B` ↔ `MSG_VOICE_E_044`. Same family letter, **different index
|
||||
space**. Deriving one from the other will silently mis-pair audio and text.
|
||||
|
||||
## ❔ The crate reads 1.3 % of the game's text
|
||||
## ✅ The crate now reads ALL of the game's caption text (was 6 %)
|
||||
|
||||
`movie_subtitle` handles the `MSG_DEMO_*` family — the cutscene captions. That is
|
||||
the **smallest of eight** caption families in the English pak, and the rest have
|
||||
@@ -331,14 +331,23 @@ Counted over every IXUD block in `GP_MAIN_GAME_E.pak`:
|
||||
| **`MSG_DEMO`** | **1 252** | **560** | **cutscene captions — the only one read** |
|
||||
| **total** | **107 372** | **44 579** | |
|
||||
|
||||
**560 of 44 579 = 1.3 %.** (41.5 % of keys carry text; the rest are the empty
|
||||
line slots this container uses for padding, so the "with text" column is the
|
||||
honest denominator.)
|
||||
❌ **CORRECTED — that column counts OCCURRENCES, not lines.** Each family lives in
|
||||
24–45 IXUD blocks and the same key repeats across them. The distinct figures:
|
||||
|
||||
```
|
||||
44 579 text-bearing MSG_* field occurrences
|
||||
8 800 DISTINCT keys <- the honest denominator
|
||||
8 800 of those have the <id>_<page>_<line> shape (100 %)
|
||||
```
|
||||
|
||||
So the real coverage was **537 of 8800 = 6.1 %**, not 1.3 %. My earlier number
|
||||
used occurrences and overstated the gap about fivefold. The direction was right;
|
||||
the magnitude was not.
|
||||
|
||||
`MSG_VOICE_*` is the family the message tables reference — the dialogue whose
|
||||
voice bindings are analysed above — and nothing in `crates/` parses it.
|
||||
|
||||
### 🟡 Generalised — 15× more text, but the gap is NOT closed
|
||||
### ✅ Closed — 8800 of 8800, by reading fields instead of adjacent tokens
|
||||
|
||||
`movie_subtitle::build_caption_text` now reads all eight families. Key shapes are
|
||||
uniform and each family is 100 % consistent with its own:
|
||||
@@ -350,25 +359,25 @@ Recovered (`tests/caption_families_disc.rs`, `examples/caption_coverage.rs`):
|
||||
|
||||
| | ids | lines |
|
||||
|---|---|---|
|
||||
| `build_demo_text` (before) | 134 | **537** |
|
||||
| `build_caption_text` (now) | **3721** | **8074** |
|
||||
| `build_demo_text` — token adjacency | 134 | **537** |
|
||||
| `build_caption_text` — token adjacency | 3721 | 8074 |
|
||||
| `build_caption_text` — **record fields** | **4085** | **8800 = all of them** |
|
||||
|
||||
The `DEMO` family comes out **identical** through both readers — 537 lines either
|
||||
way — which is the control that generalising changed nothing that worked.
|
||||
Two steps, and the second is the one that mattered. Generalising the *key parser*
|
||||
took 537 → 8074; switching from **token adjacency to the record/field table**
|
||||
took 8074 → **8800, which is 8800 of 8800 distinct keys**.
|
||||
|
||||
❌ **But 8074 is still far short of the 44 579 text-bearing fields** the
|
||||
record-level scan counts. The new reader recovers about **18 %** of them.
|
||||
`ixud.rs` now has an `IdxdObject`-shaped reader — [`IxudObject`] — verified over
|
||||
the whole disc by `tests/ixud_records_disc.rs`: **1104/1104** objects parse,
|
||||
**1476/1476** records and **628 165/628 165** named fields reproduce their
|
||||
`ixud_hash`, 48 positional. The decode had been verified with a standalone
|
||||
parser hours earlier and was simply never wired in.
|
||||
|
||||
🔑 **Why, and it is the same lesson twice.** `ixud.rs` has **no record/field
|
||||
reader** — `build_caption_text` pairs a value with the key that happens to follow
|
||||
it in the raw UTF-16 token stream, exactly the adjacency heuristic that was wrong
|
||||
for IDXD. The IXUD record table *is* decoded and verified disc-wide (1104/1104
|
||||
objects, 628 165/628 165 fields reproducing their key) in
|
||||
[idxd-container](idxd-container.md) — it was simply never wired into the crate.
|
||||
|
||||
▶️ **Next:** give `ixud.rs` an `IdxdObject`-shaped record/field reader and read
|
||||
captions as *fields*, not adjacent tokens. The decode already exists; only the
|
||||
plumbing is missing.
|
||||
🔑 **The same lesson, twice in one session.** Pool adjacency is a *consequence* of
|
||||
how records are written, not a rule of the format — true for IDXD, true here. The
|
||||
`DEMO` control shows it plainly: the token reader finds **537** lines, the field
|
||||
reader **541**. Token adjacency was quietly dropping lines even in the one family
|
||||
it was written for.
|
||||
|
||||
▶️ Superseded first step: `movie_subtitle::build_demo_text` already pairs a text value
|
||||
with the `MSG_DEMO_<demo>_<page>_<line>` key that follows it; the other seven
|
||||
|
||||
Reference in New Issue
Block a user