re: read all eight caption families — 15x more text, and the same lesson twice
build_caption_text generalises the key parser from MSG_DEMO_* to all eight
families. The shapes are uniform and each family is 100% consistent with its
own: seven use MSG_<FAM>_<id>_<page>_<line>, and VOICE alone inserts a family
letter before the id.
ids lines
build_demo_text 134 537
build_caption_text 3721 8074
The DEMO family comes out identical through both readers -- 537 lines either
way -- which is the control that generalising changed nothing that already
worked. Pinned by tests/caption_families_disc.rs, along with VOICE ids keeping
their family letter.
But this does NOT close the gap, and the write-up says so: 8074 against the
44579 text-bearing fields the record-level scan counts is about 18%.
The reason is the same lesson this session already learned once.
build_caption_text pairs a value with the key that happens to follow it in the
raw UTF-16 token stream -- the adjacency heuristic that was wrong for IDXD and
is wrong here for the same reason. ixud.rs has no record/field reader at all.
The IXUD record table IS decoded and verified disc-wide (1104/1104 objects,
628165/628165 fields reproducing their key) and was simply never wired into
the crate.
Next step recorded: give ixud.rs an IdxdObject-shaped reader and read captions
as fields rather than adjacent tokens. The decode exists; only the plumbing is
missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
@@ -238,6 +238,71 @@ pub fn build_demo_text(text_pak: &PakArchive) -> BTreeMap<u32, Vec<String>> {
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Every caption family in the pack, not just the cutscene one.
|
||||
///
|
||||
/// `build_demo_text` reads `MSG_DEMO_*` — 560 text-bearing keys, the **smallest**
|
||||
/// of eight families. The other seven carry the combat chatter and the in-mission
|
||||
/// scripted dialogue: 44 019 more lines, or **98.7 %** of the game's text.
|
||||
///
|
||||
/// Key shapes, measured over every IXUD block in `GP_MAIN_GAME_E.pak`:
|
||||
///
|
||||
/// | family | shape | text-bearing keys |
|
||||
/// |---|---|---|
|
||||
/// | `ACRO` `ADAN` `ADPL` `BIRD` `DEMO` `RHIN` `TCAF` | `MSG_<FAM>_<id>_<page>_<line>` | 37 803 |
|
||||
/// | `VOICE` | `MSG_VOICE_<letter>_<id>_<page>_<line>` | 6 776 |
|
||||
///
|
||||
/// `VOICE` is the only family with a letter before the id, and every family is
|
||||
/// 100 % consistent with its own shape.
|
||||
///
|
||||
/// Returns `"<FAM>_<id>" → ordered lines`, e.g. `"ADAN_600"`, `"VOICE_A_150"`.
|
||||
///
|
||||
/// ⚠️ The id here is the **caption** id. It is *not* the voice-bank id: a message
|
||||
/// page binding `VOICE_C_468` carries lines keyed `MSG_VOICE_C_385_*`. Same
|
||||
/// family letter, different index space — do not derive one from the other.
|
||||
pub fn build_caption_text(text_pak: &PakArchive) -> BTreeMap<String, Vec<String>> {
|
||||
let mut by_id: BTreeMap<String, BTreeMap<(u32, u32), String>> = BTreeMap::new();
|
||||
for entry in text_pak.entries() {
|
||||
let Ok(bytes) = text_pak.read(entry) else {
|
||||
continue;
|
||||
};
|
||||
if !is_ixud(&bytes) {
|
||||
continue;
|
||||
}
|
||||
let toks = utf16le_tokens(&bytes);
|
||||
for w in toks.windows(2) {
|
||||
let Some((id, page, line)) = caption_key(&w[1]) else {
|
||||
continue;
|
||||
};
|
||||
// Same pairing rule as build_demo_text: the value is the token
|
||||
// immediately before the key, and only when it is real text rather
|
||||
// than another key (bare keys are serialized consecutively too).
|
||||
if caption_key(&w[0]).is_none() && !w[0].trim().is_empty() {
|
||||
by_id.entry(id).or_default().insert((page, line), clean(&w[0]));
|
||||
}
|
||||
}
|
||||
}
|
||||
by_id
|
||||
.into_iter()
|
||||
.map(|(d, m)| (d, m.into_values().collect()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `MSG_<FAM>_<id>_<page>_<line>` → `("<FAM>_<id>", page, line)`, with `VOICE`'s
|
||||
/// extra family letter folded into the id.
|
||||
fn caption_key(t: &str) -> Option<(String, u32, u32)> {
|
||||
let rest = t.strip_prefix("MSG_")?;
|
||||
let mut parts: Vec<&str> = rest.split('_').collect();
|
||||
// Trailing <page>_<line> are always numeric.
|
||||
let line: u32 = parts.pop()?.parse().ok()?;
|
||||
let page: u32 = parts.pop()?.parse().ok()?;
|
||||
// What remains is <FAM> or <FAM>_<letter>, then the numeric id.
|
||||
let id: u32 = parts.pop()?.parse().ok()?;
|
||||
if parts.is_empty() {
|
||||
return None;
|
||||
}
|
||||
Some((format!("{}_{id:03}", parts.join("_")), page, line))
|
||||
}
|
||||
|
||||
/// Parse a timing track's IXUD payload into `(token, start, end)` triples, where
|
||||
/// `token` is either a `MSG_DEMO_<d>` reference or an inline caption string.
|
||||
///
|
||||
|
||||
Reference in New Issue
Block a user