ixud.rs now has an IdxdObject-shaped reader, IxudObject, and build_caption_text
reads captions as FIELDS instead of pairing them with whatever token follows in
the pool.
build_demo_text token adjacency 134 ids 537 lines
build_caption_text token adjacency 3721 8074
build_caption_text record fields 4085 8800 = all of them
Verified over the whole disc by tests/ixud_records_disc.rs: 1104/1104 objects
parse, 1476/1476 records and 628165/628165 named fields reproduce their
ixud_hash, 48 positional, zero failures. The header word at 0x08 is record 0's
hash, asserted per object -- there is no schema field, exactly as for IDXD. The
module doc described a 12-byte record directory and a "schema/type hash"; both
were wrong and are corrected.
I also have to correct my own number from the previous commit. "1.3% of the
game's text" counted OCCURRENCES: each family lives in 24-45 IXUD blocks and
the same key repeats across them. Distinct text-bearing MSG_* keys number 8800,
not 44579, and every one has the <id>_<page>_<line> shape. So the real coverage
was 537/8800 = 6.1%, and I overstated the gap about fivefold. Direction right,
magnitude wrong.
The DEMO control is the sharpest evidence for the change: token adjacency finds
537 lines there, the field reader 541. It was dropping lines even in the one
family it was written for -- which is why the test now asserts "must not lose
lines" rather than "must be identical".
Same lesson twice in one session: pool adjacency is a consequence of how
records are written, not a rule of the format.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.
ids lines
build_demo_text 134 537
build_caption_text 3721 8074
The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.
But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.
The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.
Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
movie_subtitle handles MSG_DEMO_*, the cutscene captions. Counting every IXUD
block in GP_MAIN_GAME_E.pak, that is the SMALLEST of eight families:
MSG_ADAN 23236 keys 9801 with text ADAN combat chatter
MSG_RHIN 21196 8509 Rhino squadron
MSG_TCAF 17148 6728 TCAF
MSG_VOICE 13060 6776 in-mission scripted dialogue
MSG_BIRD 14036 5834 Bird squadron
MSG_ADPL 12640 4127 ADAN pilots
MSG_ACRO 4804 2244 Acropolis
MSG_DEMO 1252 560 cutscene captions <- the only one read
total 107372 44579
560 of 44579 text-bearing keys = 1.3%. I report the text-bearing column rather
than raw keys because only 41.5% of keys carry text -- the rest are the empty
line slots this container pads with, and counting those would flatter the
denominator.
MSG_VOICE_* is the family the message tables reference -- the dialogue whose
voice bindings this file now analyses in detail -- and nothing in crates/
parses it. So the corpus knows which bank plays for a line it cannot read.
First step recorded: build_demo_text already pairs a text value with the
MSG_DEMO_<demo>_<page>_<line> key that follows it, and the other seven
families use the same <id>_<page>_<line> shape, so generalising the key parser
is most of the work. With a warning attached: do NOT assume the id spaces
relate, since the voice-bank id and the caption id within one message page are
different numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The 60 tokens absent from BOTH voice languages, itemised:
45 VOICE_E_ family -- 44 numeric [0..43] plus the lettered VOICE_E_012B
13 VOICE_C_ at 421, 423-426, 430, 432, 447-450, 470, 471 -- INSIDE the
listed range [0..489], so interior gaps rather than a truncated tail
2 VOICE_D_182 and _183, adjacent
These lines were written and captioned; only the audio is missing. Their
caption keys resolve to real text in the IXUD blocks:
MSG_VOICE_C_355_000_00 "What are you doing? Quit wasting..."
MSG_VOICE_C_367_000_00 "The final defense weapon is..."
MSG_VOICE_C_347_000_00 (Japanese)
MSG_VOICE_D_152_000_00 (Japanese)
MSG_VOICE_E_044_000_00 (Japanese)
Three of the five sampled are still Japanese INSIDE the English pak --
captioned but never translated, matching the untranslated entries already
noted for the localised-text container.
Separately, a trap worth its own heading: within one message page the voice
bank token and the caption keys use DIFFERENT numbering.
Message_106 voice VOICE_C_468 lines MSG_VOICE_C_385_000_00..02
Message_129 voice VOICE_D_182 lines MSG_VOICE_D_152_000_00..02
Message_044 voice VOICE_E_012B ID MSG_VOICE_E_044
Same family letter, different index space. Deriving one id from the other
will silently mis-pair audio with text -- which is the same class of mistake
as the demo-id voice binding this corpus already had to reject in-game.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Last iteration I read the scattered missing runs as cut dialogue. Checking the
other languages refutes that for most of them.
Only two voice languages exist: tables.pak holds eng\sounds.tbl (4418 banks)
and jpn\sounds.tbl (5136), and NO fra/deu/esp/ita. The other four
localisations are subtitle-only, which is itself a fact the corpus did not
record.
The English set is a strict subset of the Japanese one:
eng jpn
listed 4418 5136
resolved 4175 4383
missing 268 60
lacked by
the other 0 718
Every id missing in Japanese is also missing in English, and NONE is missing
in Japanese but present in English. So the 268 split cleanly:
208 exist in Japanese, not in English -- recorded in the original and
never dubbed. A localisation gap, not cut content.
60 absent from both -- genuinely cut, including all 44 VOICE_E_ (listed
0 times in either language) plus 16 others.
Per family eng/jpn: A 683/771, B 383/423, C 366/474, D 387/546. The 718
Japanese-only banks include whole families English never had -- VOICE_ADAN_
(110), VOICE_ZZZZ_ (104), VOICE_TCAF_ (41), VOICE_RHIN_ (27).
So VOICE_E_ survives as the one genuinely-cut family, and my "short runs of
cut dialogue" reading was wrong: those lines are on the disc, in Japanese. The
previous text is kept above the correction.
This is why the "measured on English only" limit was worth writing down rather
than quietly generalising -- the second language changed the conclusion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Following up the incidental gap from the DEMO_* work. Resolving every VOICE_*
token the way the engine does -- token, then eng\sounds.tbl for its directory,
then sound.pak -- gives 4175 resolved and 268 unresolved of 4443 distinct.
The gap is in the manifest, not the archive: the unresolved tokens are not
listed in sounds.tbl at all, and of the 4175 that ARE listed, every one is
present in the pak. Nothing is listed-then-missing.
By family, script-used ids against sounds.tbl-listed ids:
VOICE_A_ 696 [0..772] listed 683 missing 14
VOICE_B_ 391 [0..422] listed 383 missing 8
VOICE_C_ 429 [0..489] listed 366 missing 63
VOICE_D_ 508 [0..598] listed 387 missing 126
VOICE_E_ 44 [0..43] listed 0 missing 44 <- the whole family
Two different shapes, and the distinction matters. A/B/C/D lose short runs
scattered through an otherwise dense range -- 11,12,13 / 56,57 / 181,182,183
/ 389-392 -- which is what cut dialogue looks like. VOICE_E_ is absent
entirely: 44 ids used, none listed, no VOICE_E_* anywhere in the manifest. A
whole speaker or category, cut.
For the port: a missing voice bank is a RETAIL condition, not a decode bug. A
reimplementation must tolerate an unresolvable VOICE_* reference, the same way
it must tolerate SUBTITLE_S12B.tbl resolving in no language.
Limit stated: measured on GP_MAIN_GAME_E.pak and eng\sounds.tbl only; whether
the other five languages drop the same ids is untested.
An earlier probe of mine checked only three sound.pak directories and would
have mis-attributed this; sounds.tbl lists five, including Briefing. Resolving
through the manifest rather than guessing directories is what makes the 268
trustworthy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
An open question from the game_data migration: 132 dialogue lines carry an
unprefixed DEMO_* token where a VOICE_* one is expected, and whether DEMO_*
names a bank was untested. It does not.
VOICE_* 4443 distinct, 4175 resolve in sound.pak
(2360 eng\Voice\, 1815 eng\etc\)
DEMO_* 296 distinct, 0 resolve
Zero resolve under any of the three directories, and looking them up through
the convention -- VOICE_DEMO_nnn -- resolves 0 of 296 as well. So the slot is
not "the voice bank"; it can hold either a bank reference or a demo one.
The tokens run DEMO_000 to DEMO_310, 287 distinct in [0, 310] -- a dense
sequential id space, which is the shape of the demo id this file already uses
in MSG_DEMO_<demo>_<page>_<line> caption keys. I did NOT confirm the overlap
and say so: those keys live in IXUD wide-string blocks while my scan walked
IDXD objects only, so it found 0 of them. That needs an IXUD-aware sweep.
Incidental and left open: 268 of the 4443 VOICE_* tokens do not resolve
either. That is a separate gap and the DEMO_* finding does not explain it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
movie_manifest::parse now reads BASE_INFO's positional field keys (the game's
own cutscene ids, stage*100 + slot) and follows each to its record, instead of
scraping the string pool. The pool stores each distinct string once, so a
REPEAT reference produced no token and read as "no binding".
That single cause explains every wrong cell: 13 later references to
VOICE_D_450..454, two to SUBTITLE_hokyu_LS_s11A.tbl, and MS01A's share of
pwterop_s01a.prt. All 18 hokyu movies are bound, not five.
Counts, verified independently by me against the disc before recording:
104 cutscene SLOTS binding 101 distinct MOVIES; 99 slots / 96 movies with a
voice track, 99 / 96 with a subtitle, 22 / 22 with a telop. The docs' old
94 / 83 / 21 are exactly the counts of DISTINCT POOL STRINGS -- not wrong
measurements, measurements of the wrong thing. Three denominators were being
conflated; the new test pins all three.
Two assertions in movie_manifest_disc.rs were false and are corrected:
hokyu_DS_s13A binds VOICE_D_452 and resolves to eng\etc\VOICE_D_452.slb. The
in-game verdict that rejected that value tested an INFERENCE from a shared
demo id, on a decoder that discards 85-87% of banks in this class -- see
voice-bank-leading-region.md, committed earlier today.
The ~104 script ids are no longer open: they are literal positional keys,
each naming its record, and all 104 resolve. The old "counts differ by three,
positional pairing does not work" has a concrete cause -- three resupply
movies are bound by TWO slots each.
Also corrected: the naming convention has 3 subtitle exceptions (s24A/s27A
borrow s11A's track) and 18 voice exceptions, not one and five.
The legacy scraper is kept as a fallback for blobs with no record table, so
the synthetic unit fixtures still exercise it.
Artifacts: examples/movie_map_csv.rs regenerates the CSV, now slot-keyed
(104 rows; the movie-keyed version silently dropped one slot of each
duplicate). Disc tests green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The record table gives a DIRECT binding hokyu_DS_s13A -> VOICE_D_452, where
the corpus records the movie as unbound and movie_manifest_disc.rs asserts
None, citing an in-game verdict that this exact value was "the wrong
recording". That is the only place on the disc where a runtime observation
disagrees with the record table, so it was worth settling.
First, shape: these banks are SHARED. Five slots bind VOICE_D_452, five bind
451, four 450, four 453, three 454 -- 21 hokyu slots over five banks, and the
movies repeat too. Generic resupply cutscenes, not per-stage recordings.
The recorded explanation for 453 decoding to 0.14 s and 454 to 0.43 s was
that the banks are "likely multi-subwave / not cleanly sliced". Refuted: the
count of RIFF magics EQUALS the number of sub-waves recovered in all five
banks, and the last data chunk ends exactly at EOF in four of them. Nothing
between or after sub-waves is being missed.
The real defect: slb::to_xma_riffs finds audio by searching for the RIFF
magic, and a large region PRECEDES it. 87% of VOICE_D_453 and 85% of
VOICE_D_454 sit in front of the first RIFF -- 21-27% zero over 256 distinct
byte values, i.e. content, not padding. VOICE_D_451 is the control: its
leading region is 100% zero, 1 distinct value, real padding.
So the in-game verdict listened to a decode that had discarded most of the
bank, for exactly this bank class. It is evidence about the decoder, not
about the mapping. Note also that what was rejected was a value INFERRED
from a shared demo id; the record table supplies the same value as a stored
field, and only the inference was ever tested.
This does NOT establish the binding is right -- it removes the only recorded
evidence against it. What the leading region actually holds is undecoded, and
confirming the binding needs a human listening.
Artifact: examples/voice_bank_shape.rs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
The movie cutscene subtitle + voice pipeline, driven by the ADVERTISE_MOVIE
manifest (the authoritative movie -> subtitle -> voice index). Also flushes
several sessions of local WIP (async viewer loading, grouped-pool XBG7/hero-ship
decode, drawlog tooling). See docs/HANDOFF-movie-voice-subtitles-2026-07-19.md.
Subtitles (movie_subtitle.rs):
- Full movie->track->text chain; join multi-line captions sharing one timing
(fixes S13A dropped "Look at it father" line); overlap-safe active_cues();
Latin-1 accents preserved.
Voice (slb.rs): XACT .slb -> XMA1 RIFF; take the FIRST sub-wave bounded by its
declared data size (fixes S10-S16 alternate-take garble); list_voice_clips.
Manifest (movie_manifest.rs): parse ADVERTISE_MOVIE (0x5B983A08) for the real
movie->voice binding (not always VOICE_<movie>; e.g. hokyu -> VOICE_D_* in etc\).
Resolve the token's sound.pak path via sounds.tbl. DIRECT bindings only — the
demo-id shared-clip fallback for unbound hokyu movies was verified WRONG in-game
and reverted (unbound hokyu stay unvoiced; correct join key is an OPEN problem).
Viewer: manifest-driven voice (movie player toggle + solo button), standalone
"Voice Lines" browser, stacked caption overlay.
Tests: 46 formats-lib + 11 viewer-lib + movie_manifest/movie_subtitle/slb disc
tests; full workspace green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
