re: read all eight caption families — 15x more text, and the same lesson twice
build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.
ids lines
build_demo_text 134 537
build_caption_text 3721 8074
The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.
But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.
The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.
Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
@@ -338,7 +338,39 @@ honest denominator.)
|
||||
`MSG_VOICE_*` is the family the message tables reference — the dialogue whose
|
||||
voice bindings are analysed above — and nothing in `crates/` parses it.
|
||||
|
||||
▶️ **First step:** `movie_subtitle::build_demo_text` already pairs a text value
|
||||
### 🟡 Generalised — 15× more text, but the gap is NOT closed
|
||||
|
||||
`movie_subtitle::build_caption_text` now reads all eight families. Key shapes are
|
||||
uniform and each family is 100 % consistent with its own:
|
||||
|
||||
* `ACRO` `ADAN` `ADPL` `BIRD` `DEMO` `RHIN` `TCAF` — `MSG_<FAM>_<id>_<page>_<line>`
|
||||
* `VOICE` alone — `MSG_VOICE_<letter>_<id>_<page>_<line>`
|
||||
|
||||
Recovered (`tests/caption_families_disc.rs`, `examples/caption_coverage.rs`):
|
||||
|
||||
| | ids | lines |
|
||||
|---|---|---|
|
||||
| `build_demo_text` (before) | 134 | **537** |
|
||||
| `build_caption_text` (now) | **3721** | **8074** |
|
||||
|
||||
The `DEMO` family comes out **identical** through both readers — 537 lines either
|
||||
way — which is the control that generalising changed nothing that worked.
|
||||
|
||||
❌ **But 8074 is still far short of the 44 579 text-bearing fields** the
|
||||
record-level scan counts. The new reader recovers about **18 %** of them.
|
||||
|
||||
🔑 **Why, and it is the same lesson twice.** `ixud.rs` has **no record/field
|
||||
reader** — `build_caption_text` pairs a value with the key that happens to follow
|
||||
it in the raw UTF-16 token stream, exactly the adjacency heuristic that was wrong
|
||||
for IDXD. The IXUD record table *is* decoded and verified disc-wide (1104/1104
|
||||
objects, 628 165/628 165 fields reproducing their key) in
|
||||
[idxd-container](idxd-container.md) — it was simply never wired into the crate.
|
||||
|
||||
▶️ **Next:** give `ixud.rs` an `IdxdObject`-shaped record/field reader and read
|
||||
captions as *fields*, not adjacent tokens. The decode already exists; only the
|
||||
plumbing is missing.
|
||||
|
||||
▶️ Superseded first step: `movie_subtitle::build_demo_text` already pairs a text value
|
||||
with the `MSG_DEMO_<demo>_<page>_<line>` key that follows it; the other seven
|
||||
families use the same `<id>_<page>_<line>` shape, so generalising the key parser
|
||||
is most of the work. ⚠️ Do **not** assume the id spaces relate — the voice-bank
|
||||
|
||||
Reference in New Issue
Block a user