re: read all eight caption families — 15x more text, and the same lesson twice

build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.

                       ids    lines
  build_demo_text      134      537
  build_caption_text  3721     8074

The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.

But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.

The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.

Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-26 02:47:13 +00:00
parent abe41d8c27
commit 1b295e3cd2
4 changed files with 215 additions and 1 deletions

View File

@@ -338,7 +338,39 @@ honest denominator.)
`MSG_VOICE_*` is the family the message tables reference — the dialogue whose
voice bindings are analysed above — and nothing in `crates/` parses it.
▶️ **First step:** `movie_subtitle::build_demo_text` already pairs a text value
### 🟡 Generalised — 15× more text, but the gap is NOT closed
`movie_subtitle::build_caption_text` now reads all eight families. Key shapes are
uniform and each family is 100 % consistent with its own:
* `ACRO` `ADAN` `ADPL` `BIRD` `DEMO` `RHIN` `TCAF``MSG_<FAM>_<id>_<page>_<line>`
* `VOICE` alone — `MSG_VOICE_<letter>_<id>_<page>_<line>`
Recovered (`tests/caption_families_disc.rs`, `examples/caption_coverage.rs`):
| | ids | lines |
|---|---|---|
| `build_demo_text` (before) | 134 | **537** |
| `build_caption_text` (now) | **3721** | **8074** |
The `DEMO` family comes out **identical** through both readers — 537 lines either
way — which is the control that generalising changed nothing that worked.
**But 8074 is still far short of the 44 579 text-bearing fields** the
record-level scan counts. The new reader recovers about **18 %** of them.
🔑 **Why, and it is the same lesson twice.** `ixud.rs` has **no record/field
reader** — `build_caption_text` pairs a value with the key that happens to follow
it in the raw UTF-16 token stream, exactly the adjacency heuristic that was wrong
for IDXD. The IXUD record table *is* decoded and verified disc-wide (1104/1104
objects, 628 165/628 165 fields reproducing their key) in
[idxd-container](idxd-container.md) — it was simply never wired into the crate.
▶️ **Next:** give `ixud.rs` an `IdxdObject`-shaped record/field reader and read
captions as *fields*, not adjacent tokens. The decode already exists; only the
plumbing is missing.
▶️ Superseded first step: `movie_subtitle::build_demo_text` already pairs a text value
with the `MSG_DEMO_<demo>_<page>_<line>` key that follows it; the other seven
families use the same `<id>_<page>_<line>` shape, so generalising the key parser
is most of the work. ⚠️ Do **not** assume the id spaces relate — the voice-bank