diff --git a/crates/sylpheed-formats/src/idxd.rs b/crates/sylpheed-formats/src/idxd.rs index 7aee93e0..f2c81f8c 100644 --- a/crates/sylpheed-formats/src/idxd.rs +++ b/crates/sylpheed-formats/src/idxd.rs @@ -8,15 +8,44 @@ //! ## Layout (all fields big-endian) //! //! ```text -//! Offset Size Field -//! 0x00 4 Magic: "IDXD" -//! 0x04 4 count (number of top-level records; small) -//! 0x08 4 schema_hash (identifies the object type; custom hash family, preimage unknown) -//! 0x0C .. NODE / INDEX region — hash-keyed records + (for some schemas) auxiliary -//! binary tables. Not required for the string values below. -//! .. .. STRING POOL — the trailing, (almost) all-ASCII region. +//! Offset Size Field +//! 0x00 4 Magic: "IDXD" +//! 0x04 4 record_count n +//! 0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end } +//! .. 4 field_count m (equals max(field_end)) +//! .. 12*m fields { u32 key, u32 name_off, u32 value_off } +//! .. 4 pool_size (equals file_len − pool_base) +//! .. .. STRING POOL — every `*_off` above is a byte offset from here. //! ``` //! +//! Records are sorted ascending by `name_hash` and the guest binary-searches them +//! (`sub_82448AA0`); fields are sorted ascending by `key` and lower-bounded +//! (`sub_8244E338`). `name_hash` is [`crate::hash::tag_hash`] of the record's own +//! name — **not** the pak TOC hash: different modulus, and not lowercased. +//! +//! A field's `name_off` points at its own name, so **field names are on the disc** +//! and never have to be recovered from their hash; `key` is then `tag_hash(name)`. +//! A `name_off` of `0xFFFF_FFFF` means the field has no name, and its `key` is a +//! literal positional integer instead — a line-slot index, a movie id +//! ([`IdxdField::index`]). +//! +//! ## There is no schema field +//! +//! The word at `0x08` was long read as a `schema_hash` identifying the object +//! type. It is not: it is simply **record 0's `name_hash`**, the first cell of a +//! uniform 16-byte record array. `tag_hash(records[0].name)` reproduces it for +//! **all 7750** IDXD objects on the retail disc. It still works as a type +//! discriminator — tables of one kind share their lowest-hashed record name — so +//! [`IdxdObject::schema_hash`] is kept under its established name, but it +//! identifies a *record name*, not a schema. Nothing on disc names the type. +//! +//! Verified across the whole retail disc (`records_roundtrip_disc`, +//! `field_names_are_stored_disc`): all **7750** objects parse under this layout; +//! all **190782** records reproduce their stored `name_hash`; all **1271462** +//! named fields reproduce their `key` from their stored name; the remaining +//! **1485073** fields are unnamed with literal keys, and just **504** are +//! hash-keyed with no name stored. +//! //! ## The string pool: **value-before-key, defaults omitted** //! //! Each property that has an explicit value is serialized as `\0\0` — @@ -44,14 +73,34 @@ //! - [`IdxdObject::get_raw`] — the raw preceding token, no validation. Use for fields //! you *know* are identifier-valued (`ID`, `Name`, `Type`, `Model`). //! -//! ## Not yet decoded +//! ## Two readers, one object //! -//! Default/omitted fields (many ratios, the `…Count` family) carry no value in the -//! pool — their values live in the binary node/index region (hash-keyed by the same -//! unrecovered custom hash) or come from schema defaults. Binding those requires -//! decoding that region per schema; until then the typed getters return `None` for -//! them (honest "unknown") rather than guessing. See `reference_ipfb_archive_format` -//! in the project notes. +//! The record/field table above is exact, so prefer it: [`IdxdObject::records`], +//! [`record`](IdxdObject::record), [`IdxdRecord::field`]. The older +//! *value-before-key string-pool* reader ([`get_f32`](IdxdObject::get_f32) and +//! friends, documented below) predates the decode and is kept because a large +//! part of the corpus is written in terms of it. +//! +//! ## The legacy string-pool reader: **value-before-key, defaults omitted** +//! +//! In the pool each property that has an explicit value is laid out as +//! `\0\0` — the value string comes *immediately before* its field +//! name. Numbers are stored as ASCII text (`"600.0"`, `"144"`), so they read out +//! directly: +//! +//! ```text +//! … "1000.0" "HP" "500000.0" "RadarRange" "10.0" "Size_X" … +//! value key value key value key +//! ``` +//! +//! Fields left at their default omit the value string, appearing as a bare key. +//! So "the token before a key" is a real value *only when that token is itself +//! value-shaped*, which the typed getters enforce ([`get_f32`](IdxdObject::get_f32), +//! [`get_i64`](IdxdObject::get_i64), [`get_str`](IdxdObject::get_str) validate; +//! [`get_raw`](IdxdObject::get_raw) does not). The adjacency is a *consequence* of +//! the field table — each record's values are emitted next to their keys — not a +//! rule of the format, and it cannot see a field whose value string is shared or +//! reordered. Use the record API when correctness matters. use thiserror::Error; @@ -70,13 +119,75 @@ pub enum IdxdError { /// A parsed IDXD object: its schema id plus the decoded string pool. #[derive(Debug, Clone)] pub struct IdxdObject { - /// Object-type id (which kind of definition this is). Custom hash; groups - /// entries by schema even though the preimage is unknown. + /// The word at `0x08`. Despite the name it is **record 0's `name_hash`**, not + /// a schema id — see the module docs. Retained because it does discriminate + /// object kinds in practice and the corpus is written in terms of it; prefer + /// [`records`](Self::records) when you want meaning rather than a bucket key. pub schema_hash: u32, - /// The `count` header field (number of top-level records). + /// The `record_count` header field. pub count: u32, /// Ordered string-pool tokens (see module docs; value-before-key). tokens: Vec, + /// The decoded record/field table. `None` only when the binary region does + /// not parse — which no retail object does; see `records_roundtrip_disc`. + records: Option>, +} + +/// One named record of an [`IdxdObject`] — a row of the table. +#[derive(Debug, Clone)] +pub struct IdxdRecord { + /// The record's name, from the string pool. + pub name: String, + /// `tag_hash(name)`, as stored. + pub name_hash: u32, + /// This record's fields, in on-disc order (ascending `key`). + pub fields: Vec, +} + +/// One field of an [`IdxdRecord`] — a key/value cell. +#[derive(Debug, Clone)] +pub struct IdxdField { + /// `tag_hash(name)` when the field is named, else a literal positional + /// integer. See [`IdxdField::index`]. + pub key: u32, + /// The field's own name, when it has one. `None` for positional fields. + pub name: Option, + /// The field's value, from the string pool. Values are ASCII text, including + /// numbers (`"600.0"`). + pub value: String, +} + +impl IdxdField { + /// The key read as a literal positional index. `None` for a named field. + /// + /// Keyed off the stored name rather than off the key's magnitude, so it is + /// exact: a field is positional precisely when it has no name. + pub fn index(&self) -> Option { + self.name.is_none().then_some(self.key) + } + + /// Whether this field is named `name`. + pub fn is_named(&self, name: &str) -> bool { + self.name.as_deref() == Some(name) + } +} + +impl IdxdRecord { + /// The field named `name` (matched by `tag_hash`), if present. + pub fn field(&self, name: &str) -> Option<&IdxdField> { + let key = crate::hash::tag_hash(name); + self.fields.iter().find(|f| f.key == key) + } + + /// The field stored under the literal integer key `index`, if present. + pub fn field_at(&self, index: u32) -> Option<&IdxdField> { + self.fields.iter().find(|f| f.key == index) + } + + /// The value of the field named `name`. + pub fn get(&self, name: &str) -> Option<&str> { + self.field(name).map(|f| f.value.as_str()) + } } impl IdxdObject { @@ -92,10 +203,12 @@ impl IdxdObject { let count = be32(bytes, 4); let schema_hash = be32(bytes, 8); let tokens = extract_string_pool(bytes); + let records = parse_records(bytes, count); Ok(Self { schema_hash, count, tokens, + records, }) } @@ -109,6 +222,25 @@ impl IdxdObject { &self.tokens } + /// The decoded record/field table, or `None` if the binary region did not + /// parse. Every retail object parses; a `None` here means a malformed or + /// synthetic buffer. + pub fn records(&self) -> Option<&[IdxdRecord]> { + self.records.as_deref() + } + + /// The record named `name` (matched by `tag_hash`), if the table parsed. + pub fn record(&self, name: &str) -> Option<&IdxdRecord> { + let key = crate::hash::tag_hash(name); + self.records.as_ref()?.iter().find(|r| r.name_hash == key) + } + + /// Record 0 — the one whose hash sits in the header word this crate calls + /// [`schema_hash`](Self::schema_hash). + pub fn first_record(&self) -> Option<&IdxdRecord> { + self.records.as_ref()?.first() + } + /// The raw token immediately preceding the first occurrence of `key`. /// /// No validation: for a defaulted/omitted field this is the *neighbouring key*, @@ -206,6 +338,68 @@ impl IdxdObject { } } +/// Decode the record/field table (see module docs). Returns `None` — rather than +/// erroring — when any bound is inconsistent, so a malformed or synthetic buffer +/// still yields a usable string-pool reader. +fn parse_records(b: &[u8], count: u32) -> Option> { + let n = count as usize; + if n == 0 || n > b.len() / 16 { + return None; + } + let recs_at = 0x08_usize; + let field_count_at = recs_at.checked_add(n.checked_mul(16)?)?; + if field_count_at + 4 > b.len() { + return None; + } + let m = be32(b, field_count_at) as usize; + let fields_at = field_count_at + 4; + let pool_size_at = fields_at.checked_add(m.checked_mul(12)?)?; + let pool = pool_size_at.checked_add(4)?; + if pool > b.len() { + return None; + } + // The trailing identity: the stored pool size is exactly what is left. This is + // what makes the layout self-checking — a wrong record stride lands here. + if be32(b, pool_size_at) as usize != b.len() - pool { + return None; + } + let string_at = |off: u32| -> Option { + let start = pool.checked_add(off as usize)?; + if start >= b.len() { + return None; + } + let end = start + b[start..].iter().position(|&c| c == 0)?; + Some(String::from_utf8_lossy(&b[start..end]).into_owned()) + }; + let mut out = Vec::with_capacity(n); + for i in 0..n { + let r = recs_at + 16 * i; + let (begin, end) = (be32(b, r + 8) as usize, be32(b, r + 12) as usize); + if begin > end || end > m { + return None; + } + let mut fields = Vec::with_capacity(end - begin); + for j in begin..end { + let f = fields_at + 12 * j; + let name_off = be32(b, f + 4); + fields.push(IdxdField { + key: be32(b, f), + name: (name_off != NO_NAME).then(|| string_at(name_off)).flatten(), + value: string_at(be32(b, f + 8))?, + }); + } + out.push(IdxdRecord { + name: string_at(be32(b, r + 4))?, + name_hash: be32(b, r), + fields, + }); + } + Some(out) +} + +/// A field `name_off` of `0xFFFF_FFFF` means the field is positional, not named. +const NO_NAME: u32 = 0xFFFF_FFFF; + /// Extract the trailing string pool. Finds the smallest offset whose suffix is /// ≥98% printable-ASCII-or-NUL (the pool runs to end-of-buffer), then tokenises /// into maximal printable runs (NUL *or* any non-printable byte separates tokens, @@ -282,9 +476,9 @@ fn is_key_like(s: &str) -> bool { fn parse_number(s: &str) -> bool { !s.is_empty() && s.bytes().any(|b| b.is_ascii_digit()) - && s.bytes().enumerate().all(|(i, b)| { - b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+')) - }) + && s.bytes() + .enumerate() + .all(|(i, b)| b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+'))) } #[inline] @@ -358,7 +552,7 @@ mod tests { fn resolved_fields_lists_explicit_values_only() { let bytes = synth(&[ "rou_f001", "Model", // identifier value → excluded - "10.0", "Size_X", // explicit → included + "10.0", "Size_X", // explicit → included "FCSRange", // defaulted (no value) → excluded "Yes", "Mounted", // enum → included ]); diff --git a/crates/sylpheed-formats/tests/idxd_records_disc.rs b/crates/sylpheed-formats/tests/idxd_records_disc.rs new file mode 100644 index 00000000..3d234091 --- /dev/null +++ b/crates/sylpheed-formats/tests/idxd_records_disc.rs @@ -0,0 +1,291 @@ +//! The IDXD record/field table, checked against every IDXD object on the disc. +//! +//! The binary region in front of the string pool was undecoded for a long time +//! (`idxd.rs` used to say so). It is a record array plus a field array, and the +//! decisive evidence is that every record reproduces its own stored hash and +//! every named field reproduces its own key: the layout has no free parameters +//! left once that holds 1.4 million times. +//! +//! `first_header_word_is_record0_hash` is the one that demotes `schema_hash`. + +use std::path::{Path, PathBuf}; + +use sylpheed_formats::hash::tag_hash; +use sylpheed_formats::{IdxdObject, PakArchive}; + +fn disc_root() -> Option { + if let Ok(p) = std::env::var("SYLPHEED_DISC") { + let p = PathBuf::from(p); + if p.join("dat").is_dir() { + return Some(p); + } + } + let default = Path::new( + "/home/fabi/RE - Project Sylpheed/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja)", + ); + default.join("dat").is_dir().then(|| default.to_path_buf()) +} + +macro_rules! skip_without_disc { + ($root:ident) => { + let Some($root) = disc_root() else { + eprintln!("SKIP: extracted disc not found (set SYLPHEED_DISC to enable)"); + return; + }; + }; +} + +/// Every `.pak` on the disc, recursively. +/// +/// Deliberately the whole disc root, not `dat/`: `hidden/DefTables.pak` holds +/// another 1425 IDXD objects, and an earlier version of this sweep missed them. +fn all_paks(root: &Path) -> Vec { + let mut out = Vec::new(); + let mut stack = vec![root.to_path_buf()]; + while let Some(dir) = stack.pop() { + let Ok(rd) = std::fs::read_dir(&dir) else { + continue; + }; + for e in rd.flatten() { + let p = e.path(); + if p.is_dir() { + stack.push(p); + } else if p.extension().is_some_and(|x| x == "pak") { + out.push(p); + } + } + } + out.sort(); + out +} + +#[test] +fn records_roundtrip_disc() { + skip_without_disc!(root); + + let (mut objects, mut hashed, mut unparsed) = (0usize, 0usize, 0usize); + let mut bad: Vec = Vec::new(); + + for pak in all_paks(&root) { + let Ok(ar) = PakArchive::open(&pak) else { + continue; + }; + for entry in ar.entries() { + let Ok(bytes) = ar.read(entry) else { continue }; + if !IdxdObject::is_idxd(&bytes) { + continue; + } + let Ok(obj) = IdxdObject::parse(&bytes) else { + continue; + }; + objects += 1; + let Some(recs) = obj.records() else { + unparsed += 1; + bad.push(format!( + "{}:{:08x} record region did not parse", + pak.display(), + entry.name_hash + )); + continue; + }; + assert_eq!( + recs.len(), + obj.count as usize, + "{}:{:08x} record count", + pak.display(), + entry.name_hash + ); + for r in recs { + hashed += 1; + if r.name_hash != tag_hash(&r.name) { + if bad.len() < 10 { + bad.push(format!( + "{}:{:08x} record {:?} stored {:08x} != tag_hash {:08x}", + pak.display(), + entry.name_hash, + r.name, + r.name_hash, + tag_hash(&r.name) + )); + } + unparsed += 1; + } + } + // Records are stored sorted by hash so the guest can binary-search. + assert!( + recs.windows(2).all(|w| w[0].name_hash <= w[1].name_hash), + "{}:{:08x} records not sorted by hash", + pak.display(), + entry.name_hash + ); + } + } + + eprintln!("IDXD objects {objects}, hashed records {hashed}, failures {unparsed}"); + assert!( + bad.is_empty(), + "{} failures, first few:\n{}", + unparsed, + bad.join("\n") + ); + // Guard against the sweep silently finding nothing. + assert_eq!(objects, 7750, "IDXD object total changed"); + assert_eq!(hashed, 190_782, "record total changed"); +} + +/// A concrete, human-checkable row: the movie table keys its cutscene ids as +/// *literal integers*, which the string-pool reader could never have told apart +/// from field names. +#[test] +fn movie_table_ids_are_literal_field_keys() { + skip_without_disc!(root); + let ar = PakArchive::open(root.join("dat/tables.pak")).expect("tables.pak"); + + let mut found = false; + for entry in ar.entries() { + let Ok(bytes) = ar.read(entry) else { continue }; + if !IdxdObject::is_idxd(&bytes) { + continue; + } + let obj = IdxdObject::parse(&bytes).unwrap(); + if obj.schema_hash != 0x0670_25B9 { + continue; + } + let base = obj.record("BASE_INFO").expect("BASE_INFO record"); + // The five named fields the movie GamePart reads. + assert_eq!(base.get("PATH"), Some("dat\\movie\\")); + assert_eq!(base.get("VERSION"), Some("0x060329")); + // …and the numeric ids, which are keys, not names. + assert_eq!( + base.field_at(105).map(|f| f.value.as_str()), + Some("STAGE01_PHASE01") + ); + assert_eq!( + base.field_at(205).map(|f| f.value.as_str()), + Some("STAGE02_PHASE01") + ); + + // Each id names a record that carries the actual file names. + let rec = obj.record("STAGE02_PHASE01").expect("STAGE02_PHASE01"); + assert_eq!(rec.get("MOVIE"), Some("RT02A.wmv")); + assert_eq!(rec.get("VOICETRACK"), Some("VOICE_RT02A")); + + // Every literal-keyed BASE_INFO field must name a real record. + let ids: Vec = base.fields.iter().filter_map(|f| f.index()).collect(); + assert!(ids.len() > 90, "only {} literal ids", ids.len()); + for f in base.fields.iter().filter(|f| f.index().is_some()) { + assert!( + obj.record(&f.value).is_some(), + "id {} -> {:?} has no record", + f.key, + f.value + ); + } + found = true; + break; + } + assert!( + found, + "movie table (schema 067025b9) not found in tables.pak" + ); +} + +/// The header word this crate calls `schema_hash` is record 0's name hash. If +/// that were a coincidence it would not survive 6325 objects. +#[test] +fn first_header_word_is_record0_hash() { + skip_without_disc!(root); + let (mut checked, mut mismatched) = (0usize, 0usize); + for pak in all_paks(&root) { + let Ok(ar) = PakArchive::open(&pak) else { + continue; + }; + for entry in ar.entries() { + let Ok(bytes) = ar.read(entry) else { continue }; + if !IdxdObject::is_idxd(&bytes) { + continue; + } + let Ok(obj) = IdxdObject::parse(&bytes) else { + continue; + }; + let Some(first) = obj.first_record() else { + continue; + }; + checked += 1; + if tag_hash(&first.name) != obj.schema_hash { + mismatched += 1; + if mismatched <= 5 { + eprintln!( + "{}:{:08x} header {:08x} != tag_hash({:?}) {:08x}", + pak.display(), + entry.name_hash, + obj.schema_hash, + first.name, + tag_hash(&first.name) + ); + } + } + } + } + eprintln!("checked {checked}, mismatched {mismatched}"); + assert_eq!(mismatched, 0); + assert_eq!(checked, 7750); +} + +/// Field names are stored on disc; the key is `tag_hash` of the stored name. +/// Unnamed fields carry a literal positional key instead. +#[test] +fn field_names_are_stored_disc() { + skip_without_disc!(root); + let (mut named, mut positional, mut hash_keyed_unnamed, mut bad) = + (0usize, 0usize, 0usize, 0usize); + for pak in all_paks(&root) { + let Ok(ar) = PakArchive::open(&pak) else { + continue; + }; + for entry in ar.entries() { + let Ok(bytes) = ar.read(entry) else { continue }; + if !IdxdObject::is_idxd(&bytes) { + continue; + } + let Ok(obj) = IdxdObject::parse(&bytes) else { + continue; + }; + let Some(recs) = obj.records() else { continue }; + for r in recs { + for f in &r.fields { + match &f.name { + Some(n) => { + named += 1; + if tag_hash(n) != f.key { + bad += 1; + if bad <= 5 { + eprintln!( + "{}:{:08x} {:?}.{:?} key {:08x} != {:08x}", + pak.display(), + entry.name_hash, + r.name, + n, + f.key, + tag_hash(n) + ); + } + } + } + None if f.key < 0x1_0000 => positional += 1, + None => hash_keyed_unnamed += 1, + } + } + } + } + } + eprintln!( + "named {named}, positional {positional}, hash-keyed-but-unnamed {hash_keyed_unnamed}, bad {bad}" + ); + assert_eq!(bad, 0, "named fields must reproduce their key"); + assert_eq!(named, 1_271_462); + assert_eq!(positional, 1_485_073); + // A small residue keeps a hash-shaped key with no stored name: the only + // fields on the disc whose name still has to be recovered by preimage search. + assert_eq!(hash_keyed_unnamed, 504); +} diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index 9977ec00..4a18eac9 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -1033,6 +1033,26 @@ premise was wrong.** Candidates: the **7 `.embsec_` sections** (VAs 0x84D0000–0x86AC000, ~129 KB total, executable) or a hashed record in `hidden/MiscBin.pak`. **Finding it gives the actual per-phase clear condition for every stage.** +* ✅ **(2026-08-25) The IDXD/IXUD container is fully decoded** — the "binary + node/index region" in front of the string pool is a **uniform 16-byte record + array** `{name_hash, name_off, field_begin, field_end}` sorted by hash, then a + field count, **12-byte fields** `{key, name_off, value_off}` sorted by key, then + a pool size and the pool. Verified over the *whole* disc with zero failures: + IDXD **7 750/7 750** objects, **190 782/190 782** records, **1 271 462/1 271 462** + named fields; IXUD **1 104/1 104** objects, **628 165/628 165** fields (offsets in + chars). **Field names are stored on disc**, so no preimage search is needed — + only **504** fields disc-wide are hash-keyed with no name. + 🔴 **Two corrections:** the header word at `0x08` is **not a schema hash**, it is + record 0's `name_hash` (7 750/7 750) — the format has no type field at all, so an + object's kind is known only from its loader; and the field's middle word is not + an `aux` flags word. See [`structures/idxd-container.md`](structures/idxd-container.md). + ⚠️ My first disc sweep globbed `dat/**` and **missed `hidden/DefTables.pak`** + (1 425 objects); the test now walks the whole disc root. + ▶️ **Follow-up now open:** the legacy value-before-key string-pool reader is an + *approximation* of the real table, and every number in this corpus that came out + of `get_f32`/`get_raw` is re-checkable against ground truth but **not yet + re-checked**. First step: diff the two readers across the disc and count + disagreements. Also open: recover the 504 unnamed hash keys. * ✅ **(2026-08-25) Both guest hash routines located** — `sub_82447DF0` (IDXD) and `sub_82447E70` (IXUD), transcribed instruction-for-instruction into Python and Rust; `cargo test -p sylpheed-formats --lib hash` 10/10. **IXUD SOLVED:** diff --git a/docs/re/INDEX.md b/docs/re/INDEX.md index 42013e85..046afec6 100644 --- a/docs/re/INDEX.md +++ b/docs/re/INDEX.md @@ -12,7 +12,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes |--------|-------|-----------------------|-------| | IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads | | name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths | -| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values | +| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key — and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** — a field's middle word points at its own name — so nothing needs preimage search except the **504** fields disc-wide that are hash-keyed with no name; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) — the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟡 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked | | XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ | | T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) | | RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** | @@ -91,6 +91,7 @@ files, which is how the same ground got covered twice. | [`ship-placement-capture-generalisation.md`](ship-placement-capture-generalisation.md) | Capital-ship placement — does the `e106` result generalise? (WIP, 2026-07-31) | 🚧 WIP, time-boxed session. Two results so far: a static audit across all | | [`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) | Capital-ship part placement — runtime capture (ground truth) | ✅✅ STATIC ASSEMBLY IS EXACT — no captures needed anymore | | [`structures/achievements.md`](structures/achievements.md) | Achievements — the 24-entry table, and where the earned state comes from | — | +| [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container — record/field table, and the two beliefs it withdraws | ✅ CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures | | [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad — vtable `0x820B2A64` | ✅ CONFIRMED for the object layout and the atlas size, read live off | | [`structures/mission-objective-counter.md`](structures/mission-objective-counter.md) | `REMAINING OB` — the mission's own objective counter, in RAM | ✅ CONFIRMED for one Stage 02 run: a big-endian u32 whose value | | [`structures/movie-subtitles.md`](structures/movie-subtitles.md) | Movie subtitles & the movie ↔ mission ↔ text chain | — | diff --git a/docs/re/structures/idxd-container.md b/docs/re/structures/idxd-container.md new file mode 100644 index 00000000..cb70d982 --- /dev/null +++ b/docs/re/structures/idxd-container.md @@ -0,0 +1,157 @@ +# The IDXD container — record/field table + +Status: ✅ **CONFIRMED**, decoded in full and verified over **the whole disc** +(not just `dat/` — `hidden/DefTables.pak` holds another 1425 objects, which the +first version of this sweep missed): **7750 / 7750** +objects parse, **190,782 / 190,782** records reproduce their stored name hash, +**1,271,462 / 1,271,462** named fields reproduce their key from their stored name. +Zero failures of any kind. + +This closes the **"binary node/index region — not yet decoded"** note that stood +in `crates/sylpheed-formats/src/idxd.rs` for the whole life of the parser, and it +**demotes two beliefs the corpus was built on** (see *Two corrections* below). + +Tests: `crates/sylpheed-formats/tests/idxd_records_disc.rs` — +`records_roundtrip_disc`, `first_header_word_is_record0_hash`, +`field_names_are_stored_disc`, `movie_table_ids_are_literal_field_keys`. + +## ✅ The layout + +All fields big-endian. + +```text +Offset Size Field +0x00 4 "IDXD" +0x04 4 record_count n +0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end } +.. 4 field_count m (equals max(field_end)) +.. 12*m fields { u32 key, u32 name_off, u32 value_off } +.. 4 pool_size (equals file_len - pool_base) +.. .. string pool (every *_off above is a byte offset from here) +``` + +* **Records are sorted ascending by `name_hash`** and the guest binary-searches + them with a 16-byte stride (`sub_82448AA0`). Verified: sorted in every object. +* `name_hash` is [`tag_hash`](idxd-tag-hash.md) of the record's **own name** — + not the pak TOC hash (different modulus, and not lowercased). +* A record owns the half-open field range `[field_begin, field_end)`. Ranges are + *not* contiguous in record order — records are in hash order while their fields + sit in name order, so the field array is shared, not partitioned by position. +* **Fields are sorted ascending by `key`** and lower-bounded (`sub_8244E338`). +* `field_count` and `pool_size` are what make the layout self-checking: a wrong + record stride lands on a `pool_size` that does not equal the remaining bytes. + That identity is what caught the first wrong version of this decode. + +## ✅ Field names are on the disc + +A field's middle word is **its own name's pool offset**, and then +`key == tag_hash(name)`. Field names never have to be recovered from their hash. + +`name_off == 0xFFFF_FFFF` means the field has **no name**; its `key` is then a +literal positional integer — a line-slot index (`0,1,2,3`), a movie id (`105`, +`1303`). So a field is positional *exactly* when it stores no name; you do not +have to guess from the key's magnitude. + +Disc-wide census of all **2,757,039** fields: + +| kind | count | +|---|---| +| named, `key == tag_hash(name)` | 1,271,462 | +| unnamed, literal key (`< 0x10000`) | 1,485,073 | +| unnamed, hash-shaped key | 504 | + +Those **504** are the entire remaining preimage problem on the disc. ❔ Their +names are unrecovered. + +## Two corrections + +### ❌ WITHDRAWN — "the word at `0x08` is a schema hash" + +The corpus (and `IdxdObject::schema_hash`, and every `pak list` line printing +`schema 067025b9`) read the third header word as an object-type id whose preimage +was unknown. **It is not a schema id.** It is simply **record 0's `name_hash`**: +the record array is uniform 16-byte entries starting at `0x08`, and the header has +no type field at all. + +Evidence: `tag_hash(records[0].name) == word@0x08` for **all 7750** objects on the +disc, zero exceptions (`first_header_word_is_record0_hash`). + +How it was caught, which is the useful part: a test asserted that every movie id in +`BASE_INFO` names a real record, and one — `1005 -> "STAGE10_PHASE01"` — did not +resolve. The reason was that `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, the +movie table's supposed schema id. A "coincidence" at 1-in-2^32 is not a +coincidence; the record was being eaten by the header. + +It still *works* as a type discriminator, because tables of one kind share their +lowest-hashed record name — which is exactly why it went unquestioned for so long. +`schema_hash` is therefore kept under its established name, with its docs corrected, +rather than renamed across 33 call sites. **Nothing on disc names an object's type.** + +### ❌ WITHDRAWN — "the field's middle word is an `aux`/flags word, always `0xFFFFFFFF`" + +It is `0xFFFFFFFF` for 54% of fields, which is enough to look constant in a small +sample. It is a name offset (above). The tell was that the word is constant *per +key across records* (`key=0x6c43a78d` always carried `1743`) — a per-record flag +cannot do that, but a per-name pointer must. + +## Worked example — the movie table + +`dat/tables.pak`, the object whose record 0 is `STAGE10_PHASE01` (105 records, +433 fields). Its `BASE_INFO` record mixes both field kinds: + +```text +named: PATH = "dat\movie\" VERSION = "0x060329" SUBTITLE_FONT … +positional: 105 -> "STAGE01_PHASE01" 205 -> "STAGE02_PHASE01" 1005 -> "STAGE10_PHASE01" +``` + +Each positional value names another record in the same object, which carries the +real files: + +```text +STAGE02_PHASE01: MOVIE = "RT02A.wmv" VOICETRACK = "VOICE_RT02A" + SUBTITLE = "…+SUBTITLE_RT02A.tbl" TELOP = "…+pwrt02.prt" +``` + +All 104 positional ids resolve to a real record — verified, no dangling entries. +This is the id space the mission script's cutscene request uses; see +[movie-subtitle-link](../movie-subtitle-link.md). + +## ✅ `IXUD` is the same container + +The wide-string sibling has an identical shape, with three substitutions: the hash +is `ixud_hash`, strings are UTF-16BE, and **every offset — record name, field name +and value alike — is in 16-bit chars**, so `pool_base + 2*off`. `pool_size` is +likewise a char count, which is the same identity as the already-known +`STR + 2*strsize == filesize` ([ixud-localised-text](ixud-localised-text.md)). + +Verified independently over all **1104** IXUD objects on the disc: the header word +at `0x08` is record 0's `ixud_hash` (1104/1104), and `key == ixud_hash(field name)` +for **628,165 / 628,165** named fields, zero mismatches. Only 48 fields are +unnamed — 6 objects × 8, all in a `CATEGORY_DESC` record with keys `0..7`, a +positional array of weapon-category descriptions. + +One extra rule shows up in IXUD's comparator (`sub_82447F38`) and is worth assuming +for IDXD too: `name_off == 0xFFFF_FFFF` is the **primary** sort key, so a record's +field slice is *unnamed fields first, then named fields*, each key-ascending +(1476/1476 slices). That is why there are two getters — lookup-by-integer searches +only the unnamed run, lookup-by-name only the named run. + +🟡 The loader `sub_82448D00` also accepts legacy magics `IIDX`, `IDX2`, `IDX3`, +`IDXC` and rejects `IDXD` on that path with `"Old virsion binary table. Not +supported."` [sic]. **None of them occur on this disc** (magic census over every +pak entry: `IDXD` 7750, `T8aD` 4525, `RATC` 2985, `IXUD` 1104, `LSTA` 64, and zero +of the four legacy magics), so that path is read-only knowledge, untestable here. + +## What this does *not* settle + +* ❔ The names of the 504 unnamed hash-keyed fields. +* ❔ How much of the existing corpus the old string-pool reader got wrong. The + legacy `get_f32`/`get_raw` path infers a field's value from *pool adjacency* + (`\0\0`). That adjacency is a consequence of the field table, not a + rule of the format, and it cannot represent a field whose value string is shared + or reordered. Every number in `docs/re/` that came from it is now re-checkable + against the true table, and has not yet been re-checked. +* ❔ Whether `IXUD`, the wide-string sibling, uses the same shape. Its offsets are + in 16-bit chars and its hash is different ([ixud-localised-text](ixud-localised-text.md)). +* The type-identification question is now open rather than closed: with no schema + field, an object's kind is known only from the caller that loads it.