fabi/Syplheed-Reborn
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re: wire the IXUD record table into the crate — captions go 537 to 8800 of 8800

Browse Source 
ixud.rs now has an IdxdObject-shaped reader, IxudObject, and build_caption_text
reads captions as FIELDS instead of pairing them with whatever token follows in
the pool.

  build_demo_text      token adjacency   134 ids   537 lines
  build_caption_text   token adjacency  3721      8074
  build_caption_text   record fields    4085      8800  = all of them

Verified over the whole disc by tests/ixud_records_disc.rs: 1104/1104 objects
parse, 1476/1476 records and 628165/628165 named fields reproduce their
ixud_hash, 48 positional, zero failures. The header word at 0x08 is record 0's
hash, asserted per object -- there is no schema field, exactly as for IDXD. The
module doc described a 12-byte record directory and a "schema/type hash"; both
were wrong and are corrected.

I also have to correct my own number from the previous commit. "1.3% of the
game's text" counted OCCURRENCES: each family lives in 24-45 IXUD blocks and
the same key repeats across them. Distinct text-bearing MSG_* keys number 8800,
not 44579, and every one has the <id>_<page>_<line> shape. So the real coverage
was 537/8800 = 6.1%, and I overstated the gap about fivefold. Direction right,
magnitude wrong.

The DEMO control is the sharpest evidence for the change: token adjacency finds
537 lines there, the field reader 541. It was dropping lines even in the one
family it was written for -- which is why the test now asserts "must not lose
lines" rather than "must be identical".

Same lesson twice in one session: pool adjacency is a consequence of how
records are written, not a rule of the format.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-26 02:55:26 +00:00
parent 1b295e3cd2
commit 80a45bfbd7
6 changed files with 320 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
crates/sylpheed-formats/src/ixud.rs
View File

@@ -15,16 +15,35 @@
//! start timecode instead of inline text; the actual string then lives in a
//! separate global table (not resolved here).
//!
//! ## Binary layout (as far as needed)
//! ## Binary layout — ✅ decoded and verified disc-wide
//!
//! ```text
//! 0x00 4 Magic "IXUD"
//! 0x04 4 version (1)
//! 0x08 4 schema/type hash (constant 0x6CC83E70)
//! .. .. record directory { key_hash u32, offset u32, len u32 } × n
//! .. .. UTF-16BE string pool, NUL-terminated entries
//! 0x00 4 Magic "IXUD"
//! 0x04 4 record_count n
//! 0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
//! .. 4 field_count m
//! .. 12*m fields { u32 key, u32 name_off, u32 value_off }
//! .. 4 pool_size (in CHARS)
//! .. .. UTF-16BE string pool
//! ```
//! We don't need the directory to *present* the track — decoding the pool and
//! pairing tokens after the `SUBTITLE` header is enough and robust.
//!
//! **Every offset is in 16-bit chars, not bytes** — `pool_base + 2*off`. That
//! includes record and field *names*, not only values, and it is what makes the
//! `STR + 2*strsize == filesize` identity hold.
//!
//! `name_hash` is [`crate::hash::ixud_hash`] of the record's own name, and a
//! field's `key` is the same hash of its name; `name_off == 0xFFFF_FFFF` marks a
//! positional field whose key is a literal index. Verified over every IXUD object
//! on the disc: **1104/1104** objects parse, **1476/1476** records and
//! **628 165/628 165** named fields reproduce their stored hash.
//!
//! ❌ The layout this comment used to describe — a 12-byte record directory and a
//! "schema/type hash" at `0x08` — was wrong in the same way IDXD's was. There is
//! no schema field: the word at `0x08` is record 0's `name_hash`. See
//! `docs/re/structures/idxd-container.md`.
//!
//! The cue reader below still works on the string pool directly, which is fine
//! for presenting a track; [`IxudObject`] is the addressable route.
/// Magic at the start of every IXUD entry.
pub const IXUD_MAGIC: [u8; 4] = *b"IXUD";
@@ -185,3 +204,138 @@ mod tests {
assert!(parse(b"IDXD\0\0\0\0").is_none());
}
}
/// A parsed IXUD object: its records and their fields, addressable by name.
///
/// The counterpart of [`crate::IdxdObject`] for wide strings. Prefer this to the
/// token-pairing readers when you need a *specific* field: pool adjacency is a
/// consequence of how records are written, not a rule of the format.
#[derive(Debug, Clone)]
pub struct IxudObject {
/// The word at `0x08` — record 0's `name_hash`, not a schema id.
pub first_record_hash: u32,
records: Vec<IxudRecord>,
}
/// One named record of an [`IxudObject`].
#[derive(Debug, Clone)]
pub struct IxudRecord {
/// The record's name, decoded from UTF-16BE.
pub name: String,
/// `ixud_hash(name)`, as stored.
pub name_hash: u32,
/// This record's fields, in on-disc order.
pub fields: Vec<IxudField>,
}
/// One field of an [`IxudRecord`].
#[derive(Debug, Clone)]
pub struct IxudField {
/// `ixud_hash(name)` when named, else a literal positional index.
pub key: u32,
/// The field's own name, when it has one.
pub name: Option<String>,
/// The field's value.
pub value: String,
}
impl IxudRecord {
/// The field named `name` (matched by `ixud_hash`), if present.
pub fn field(&self, name: &str) -> Option<&IxudField> {
let key = crate::hash::ixud_hash_str(name);
self.fields.iter().find(|f| f.key == key)
}
/// The value of the field named `name`.
pub fn get(&self, name: &str) -> Option<&str> {
self.field(name).map(|f| f.value.as_str())
}
}
impl IxudObject {
/// Parse an IXUD object. `None` when the layout does not check out — the
/// trailing `pool_size` identity makes that self-verifying.
pub fn parse(b: &[u8]) -> Option<Self> {
if b.len() < 12 || b[0..4] != IXUD_MAGIC {
return None;
}
let be32 = |o: usize| u32::from_be_bytes([b[o], b[o + 1], b[o + 2], b[o + 3]]);
let n = be32(4) as usize;
if n == 0 || n > b.len() / 16 {
return None;
}
let field_count_at = 8usize.checked_add(n.checked_mul(16)?)?;
if field_count_at + 4 > b.len() {
return None;
}
let m = be32(field_count_at) as usize;
let fields_at = field_count_at + 4;
let pool_size_at = fields_at.checked_add(m.checked_mul(12)?)?;
let pool = pool_size_at.checked_add(4)?;
if pool > b.len() {
return None;
}
// pool_size counts CHARS, so this identity also proves the char scaling.
if (be32(pool_size_at) as usize).checked_mul(2)? != b.len() - pool {
return None;
}
let wstr = |off: u32| -> Option<String> {
if off == NO_NAME {
return None;
}
let start = pool.checked_add((off as usize).checked_mul(2)?)?;
let mut units = Vec::new();
let mut i = start;
while i + 1 < b.len() {
let u = u16::from_be_bytes([b[i], b[i + 1]]);
if u == 0 {
return Some(String::from_utf16_lossy(&units));
}
units.push(u);
i += 2;
}
None
};
let mut records = Vec::with_capacity(n);
for i in 0..n {
let r = 8 + 16 * i;
let (begin, end) = (be32(r + 8) as usize, be32(r + 12) as usize);
if begin > end || end > m {
return None;
}
let mut fields = Vec::with_capacity(end - begin);
for j in begin..end {
let f = fields_at + 12 * j;
fields.push(IxudField {
key: be32(f),
name: wstr(be32(f + 4)),
value: wstr(be32(f + 8))?,
});
}
records.push(IxudRecord {
name: wstr(be32(r + 4))?,
name_hash: be32(r),
fields,
});
}
Some(Self {
first_record_hash: be32(8),
records,
})
}
/// Every record, in on-disc order (ascending `name_hash`).
pub fn records(&self) -> &[IxudRecord] {
&self.records
}
/// The record named `name` (matched by `ixud_hash`).
pub fn record(&self, name: &str) -> Option<&IxudRecord> {
let key = crate::hash::ixud_hash_str(name);
self.records.iter().find(|r| r.name_hash == key)
}
}
/// A field `name_off` of `0xFFFF_FFFF` marks a positional field.
const NO_NAME: u32 = 0xFFFF_FFFF;

2
crates/sylpheed-formats/src/lib.rs
View File

@@ -85,7 +85,7 @@ pub mod ship_capture;
pub use audio::{AudioCodec, AudioInfo, GameAudio};
pub use font::FontInfo;
pub use idxd::{IdxdError, IdxdObject};
pub use ixud::{Cue, Subtitle};
pub use ixud::{Cue, IxudField, IxudObject, IxudRecord, Subtitle};
pub use movie_subtitle::{SubCue, SubLang};
pub use mesh::{GameMesh, Xbg7Model};
pub use ratc::RatcChild;

29
crates/sylpheed-formats/src/movie_subtitle.rs
View File

@@ -265,19 +265,24 @@ pub fn build_caption_text(text_pak: &PakArchive) -> BTreeMap<String, Vec<String>
let Ok(bytes) = text_pak.read(entry) else {
continue;
};
if !is_ixud(&bytes) {
let Some(obj) = crate::IxudObject::parse(&bytes) else {
continue;
}
let toks = utf16le_tokens(&bytes);
for w in toks.windows(2) {
let Some((id, page, line)) = caption_key(&w[1]) else {
continue;
};
// Same pairing rule as build_demo_text: the value is the token
// immediately before the key, and only when it is real text rather
// than another key (bare keys are serialized consecutively too).
if caption_key(&w[0]).is_none() && !w[0].trim().is_empty() {
by_id.entry(id).or_default().insert((page, line), clean(&w[0]));
};
// Read the caption as a FIELD -- its key is the `MSG_*` name and its
// value is the text. The token-adjacency version of this recovered 8074
// lines against the 44579 fields that actually carry text, because pool
// adjacency is a consequence of how records are written rather than a
// rule of the format. Same mistake the IDXD reader made.
for rec in obj.records() {
for f in &rec.fields {
let Some(key) = f.name.as_deref() else { continue };
let Some((id, page, line)) = caption_key(key) else {
continue;
};
if f.value.trim().is_empty() {
continue;
}
by_id.entry(id).or_default().insert((page, line), clean(&f.value));
}
}
}