re: decode the IDXD/IXUD record table — and there is no schema hash
The binary region in front of the string pool was the parser's oldest open
note ("Not yet decoded"). It is a uniform 16-byte record array sorted by
name hash, a field count, a 12-byte field array sorted by key, a pool size,
and the pool. The trailing `pool_size == file_len - pool_base` identity makes
the layout self-checking, which is what caught the first wrong version.
Verified over the WHOLE disc with zero failures: 7750/7750 IDXD objects,
190782/190782 records reproducing their stored tag_hash, 1271462/1271462
named fields reproducing their key. IXUD is the same container with
ixud_hash, UTF-16BE and every offset in chars — 1104/1104 objects,
628165/628165 fields, checked with an independent parser.
Field names are stored on disc, so no preimage search is needed: a field's
middle word points at its own name. Only 504 fields disc-wide are hash-keyed
with no name; the other 1485073 nameless fields are positional, keyed by a
literal integer (line slots, movie ids).
Two long-held beliefs are WITHDRAWN:
* The word at 0x08 is not a schema hash. It is record 0's name_hash — the
format has no type field at all, and an object's kind is known only from
the caller that loads it. It survived as "schema" because tables of one
kind share their lowest-hashed record name. Caught by a test asserting
every movie id names a real record: 1005 -> STAGE10_PHASE01 failed because
tag_hash("STAGE10_PHASE01") IS 0x067025B9, that table's supposed schema id.
* The field's middle word is not an always-0xFFFFFFFF flags word. It is
0xFFFFFFFF for 54% of fields, enough to look constant in a small sample;
the tell was that it is constant per key ACROSS records, which a per-record
flag cannot be but a per-name pointer must.
`schema_hash` keeps its name rather than churn 33 call sites, with corrected
docs. The first sweep globbed dat/** and missed hidden/DefTables.pak (1425
objects); the test now walks the whole disc root.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
@@ -8,15 +8,44 @@
|
||||
//! ## Layout (all fields big-endian)
|
||||
//!
|
||||
//! ```text
|
||||
//! Offset Size Field
|
||||
//! 0x00 4 Magic: "IDXD"
|
||||
//! 0x04 4 count (number of top-level records; small)
|
||||
//! 0x08 4 schema_hash (identifies the object type; custom hash family, preimage unknown)
|
||||
//! 0x0C .. NODE / INDEX region — hash-keyed records + (for some schemas) auxiliary
|
||||
//! binary tables. Not required for the string values below.
|
||||
//! .. .. STRING POOL — the trailing, (almost) all-ASCII region.
|
||||
//! Offset Size Field
|
||||
//! 0x00 4 Magic: "IDXD"
|
||||
//! 0x04 4 record_count n
|
||||
//! 0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
|
||||
//! .. 4 field_count m (equals max(field_end))
|
||||
//! .. 12*m fields { u32 key, u32 name_off, u32 value_off }
|
||||
//! .. 4 pool_size (equals file_len − pool_base)
|
||||
//! .. .. STRING POOL — every `*_off` above is a byte offset from here.
|
||||
//! ```
|
||||
//!
|
||||
//! Records are sorted ascending by `name_hash` and the guest binary-searches them
|
||||
//! (`sub_82448AA0`); fields are sorted ascending by `key` and lower-bounded
|
||||
//! (`sub_8244E338`). `name_hash` is [`crate::hash::tag_hash`] of the record's own
|
||||
//! name — **not** the pak TOC hash: different modulus, and not lowercased.
|
||||
//!
|
||||
//! A field's `name_off` points at its own name, so **field names are on the disc**
|
||||
//! and never have to be recovered from their hash; `key` is then `tag_hash(name)`.
|
||||
//! A `name_off` of `0xFFFF_FFFF` means the field has no name, and its `key` is a
|
||||
//! literal positional integer instead — a line-slot index, a movie id
|
||||
//! ([`IdxdField::index`]).
|
||||
//!
|
||||
//! ## There is no schema field
|
||||
//!
|
||||
//! The word at `0x08` was long read as a `schema_hash` identifying the object
|
||||
//! type. It is not: it is simply **record 0's `name_hash`**, the first cell of a
|
||||
//! uniform 16-byte record array. `tag_hash(records[0].name)` reproduces it for
|
||||
//! **all 7750** IDXD objects on the retail disc. It still works as a type
|
||||
//! discriminator — tables of one kind share their lowest-hashed record name — so
|
||||
//! [`IdxdObject::schema_hash`] is kept under its established name, but it
|
||||
//! identifies a *record name*, not a schema. Nothing on disc names the type.
|
||||
//!
|
||||
//! Verified across the whole retail disc (`records_roundtrip_disc`,
|
||||
//! `field_names_are_stored_disc`): all **7750** objects parse under this layout;
|
||||
//! all **190782** records reproduce their stored `name_hash`; all **1271462**
|
||||
//! named fields reproduce their `key` from their stored name; the remaining
|
||||
//! **1485073** fields are unnamed with literal keys, and just **504** are
|
||||
//! hash-keyed with no name stored.
|
||||
//!
|
||||
//! ## The string pool: **value-before-key, defaults omitted**
|
||||
//!
|
||||
//! Each property that has an explicit value is serialized as `<value>\0<key>\0` —
|
||||
@@ -44,14 +73,34 @@
|
||||
//! - [`IdxdObject::get_raw`] — the raw preceding token, no validation. Use for fields
|
||||
//! you *know* are identifier-valued (`ID`, `Name`, `Type`, `Model`).
|
||||
//!
|
||||
//! ## Not yet decoded
|
||||
//! ## Two readers, one object
|
||||
//!
|
||||
//! Default/omitted fields (many ratios, the `…Count` family) carry no value in the
|
||||
//! pool — their values live in the binary node/index region (hash-keyed by the same
|
||||
//! unrecovered custom hash) or come from schema defaults. Binding those requires
|
||||
//! decoding that region per schema; until then the typed getters return `None` for
|
||||
//! them (honest "unknown") rather than guessing. See `reference_ipfb_archive_format`
|
||||
//! in the project notes.
|
||||
//! The record/field table above is exact, so prefer it: [`IdxdObject::records`],
|
||||
//! [`record`](IdxdObject::record), [`IdxdRecord::field`]. The older
|
||||
//! *value-before-key string-pool* reader ([`get_f32`](IdxdObject::get_f32) and
|
||||
//! friends, documented below) predates the decode and is kept because a large
|
||||
//! part of the corpus is written in terms of it.
|
||||
//!
|
||||
//! ## The legacy string-pool reader: **value-before-key, defaults omitted**
|
||||
//!
|
||||
//! In the pool each property that has an explicit value is laid out as
|
||||
//! `<value>\0<key>\0` — the value string comes *immediately before* its field
|
||||
//! name. Numbers are stored as ASCII text (`"600.0"`, `"144"`), so they read out
|
||||
//! directly:
|
||||
//!
|
||||
//! ```text
|
||||
//! … "1000.0" "HP" "500000.0" "RadarRange" "10.0" "Size_X" …
|
||||
//! value key value key value key
|
||||
//! ```
|
||||
//!
|
||||
//! Fields left at their default omit the value string, appearing as a bare key.
|
||||
//! So "the token before a key" is a real value *only when that token is itself
|
||||
//! value-shaped*, which the typed getters enforce ([`get_f32`](IdxdObject::get_f32),
|
||||
//! [`get_i64`](IdxdObject::get_i64), [`get_str`](IdxdObject::get_str) validate;
|
||||
//! [`get_raw`](IdxdObject::get_raw) does not). The adjacency is a *consequence* of
|
||||
//! the field table — each record's values are emitted next to their keys — not a
|
||||
//! rule of the format, and it cannot see a field whose value string is shared or
|
||||
//! reordered. Use the record API when correctness matters.
|
||||
|
||||
use thiserror::Error;
|
||||
|
||||
@@ -70,13 +119,75 @@ pub enum IdxdError {
|
||||
/// A parsed IDXD object: its schema id plus the decoded string pool.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IdxdObject {
|
||||
/// Object-type id (which kind of definition this is). Custom hash; groups
|
||||
/// entries by schema even though the preimage is unknown.
|
||||
/// The word at `0x08`. Despite the name it is **record 0's `name_hash`**, not
|
||||
/// a schema id — see the module docs. Retained because it does discriminate
|
||||
/// object kinds in practice and the corpus is written in terms of it; prefer
|
||||
/// [`records`](Self::records) when you want meaning rather than a bucket key.
|
||||
pub schema_hash: u32,
|
||||
/// The `count` header field (number of top-level records).
|
||||
/// The `record_count` header field.
|
||||
pub count: u32,
|
||||
/// Ordered string-pool tokens (see module docs; value-before-key).
|
||||
tokens: Vec<String>,
|
||||
/// The decoded record/field table. `None` only when the binary region does
|
||||
/// not parse — which no retail object does; see `records_roundtrip_disc`.
|
||||
records: Option<Vec<IdxdRecord>>,
|
||||
}
|
||||
|
||||
/// One named record of an [`IdxdObject`] — a row of the table.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IdxdRecord {
|
||||
/// The record's name, from the string pool.
|
||||
pub name: String,
|
||||
/// `tag_hash(name)`, as stored.
|
||||
pub name_hash: u32,
|
||||
/// This record's fields, in on-disc order (ascending `key`).
|
||||
pub fields: Vec<IdxdField>,
|
||||
}
|
||||
|
||||
/// One field of an [`IdxdRecord`] — a key/value cell.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IdxdField {
|
||||
/// `tag_hash(name)` when the field is named, else a literal positional
|
||||
/// integer. See [`IdxdField::index`].
|
||||
pub key: u32,
|
||||
/// The field's own name, when it has one. `None` for positional fields.
|
||||
pub name: Option<String>,
|
||||
/// The field's value, from the string pool. Values are ASCII text, including
|
||||
/// numbers (`"600.0"`).
|
||||
pub value: String,
|
||||
}
|
||||
|
||||
impl IdxdField {
|
||||
/// The key read as a literal positional index. `None` for a named field.
|
||||
///
|
||||
/// Keyed off the stored name rather than off the key's magnitude, so it is
|
||||
/// exact: a field is positional precisely when it has no name.
|
||||
pub fn index(&self) -> Option<u32> {
|
||||
self.name.is_none().then_some(self.key)
|
||||
}
|
||||
|
||||
/// Whether this field is named `name`.
|
||||
pub fn is_named(&self, name: &str) -> bool {
|
||||
self.name.as_deref() == Some(name)
|
||||
}
|
||||
}
|
||||
|
||||
impl IdxdRecord {
|
||||
/// The field named `name` (matched by `tag_hash`), if present.
|
||||
pub fn field(&self, name: &str) -> Option<&IdxdField> {
|
||||
let key = crate::hash::tag_hash(name);
|
||||
self.fields.iter().find(|f| f.key == key)
|
||||
}
|
||||
|
||||
/// The field stored under the literal integer key `index`, if present.
|
||||
pub fn field_at(&self, index: u32) -> Option<&IdxdField> {
|
||||
self.fields.iter().find(|f| f.key == index)
|
||||
}
|
||||
|
||||
/// The value of the field named `name`.
|
||||
pub fn get(&self, name: &str) -> Option<&str> {
|
||||
self.field(name).map(|f| f.value.as_str())
|
||||
}
|
||||
}
|
||||
|
||||
impl IdxdObject {
|
||||
@@ -92,10 +203,12 @@ impl IdxdObject {
|
||||
let count = be32(bytes, 4);
|
||||
let schema_hash = be32(bytes, 8);
|
||||
let tokens = extract_string_pool(bytes);
|
||||
let records = parse_records(bytes, count);
|
||||
Ok(Self {
|
||||
schema_hash,
|
||||
count,
|
||||
tokens,
|
||||
records,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -109,6 +222,25 @@ impl IdxdObject {
|
||||
&self.tokens
|
||||
}
|
||||
|
||||
/// The decoded record/field table, or `None` if the binary region did not
|
||||
/// parse. Every retail object parses; a `None` here means a malformed or
|
||||
/// synthetic buffer.
|
||||
pub fn records(&self) -> Option<&[IdxdRecord]> {
|
||||
self.records.as_deref()
|
||||
}
|
||||
|
||||
/// The record named `name` (matched by `tag_hash`), if the table parsed.
|
||||
pub fn record(&self, name: &str) -> Option<&IdxdRecord> {
|
||||
let key = crate::hash::tag_hash(name);
|
||||
self.records.as_ref()?.iter().find(|r| r.name_hash == key)
|
||||
}
|
||||
|
||||
/// Record 0 — the one whose hash sits in the header word this crate calls
|
||||
/// [`schema_hash`](Self::schema_hash).
|
||||
pub fn first_record(&self) -> Option<&IdxdRecord> {
|
||||
self.records.as_ref()?.first()
|
||||
}
|
||||
|
||||
/// The raw token immediately preceding the first occurrence of `key`.
|
||||
///
|
||||
/// No validation: for a defaulted/omitted field this is the *neighbouring key*,
|
||||
@@ -206,6 +338,68 @@ impl IdxdObject {
|
||||
}
|
||||
}
|
||||
|
||||
/// Decode the record/field table (see module docs). Returns `None` — rather than
|
||||
/// erroring — when any bound is inconsistent, so a malformed or synthetic buffer
|
||||
/// still yields a usable string-pool reader.
|
||||
fn parse_records(b: &[u8], count: u32) -> Option<Vec<IdxdRecord>> {
|
||||
let n = count as usize;
|
||||
if n == 0 || n > b.len() / 16 {
|
||||
return None;
|
||||
}
|
||||
let recs_at = 0x08_usize;
|
||||
let field_count_at = recs_at.checked_add(n.checked_mul(16)?)?;
|
||||
if field_count_at + 4 > b.len() {
|
||||
return None;
|
||||
}
|
||||
let m = be32(b, field_count_at) as usize;
|
||||
let fields_at = field_count_at + 4;
|
||||
let pool_size_at = fields_at.checked_add(m.checked_mul(12)?)?;
|
||||
let pool = pool_size_at.checked_add(4)?;
|
||||
if pool > b.len() {
|
||||
return None;
|
||||
}
|
||||
// The trailing identity: the stored pool size is exactly what is left. This is
|
||||
// what makes the layout self-checking — a wrong record stride lands here.
|
||||
if be32(b, pool_size_at) as usize != b.len() - pool {
|
||||
return None;
|
||||
}
|
||||
let string_at = |off: u32| -> Option<String> {
|
||||
let start = pool.checked_add(off as usize)?;
|
||||
if start >= b.len() {
|
||||
return None;
|
||||
}
|
||||
let end = start + b[start..].iter().position(|&c| c == 0)?;
|
||||
Some(String::from_utf8_lossy(&b[start..end]).into_owned())
|
||||
};
|
||||
let mut out = Vec::with_capacity(n);
|
||||
for i in 0..n {
|
||||
let r = recs_at + 16 * i;
|
||||
let (begin, end) = (be32(b, r + 8) as usize, be32(b, r + 12) as usize);
|
||||
if begin > end || end > m {
|
||||
return None;
|
||||
}
|
||||
let mut fields = Vec::with_capacity(end - begin);
|
||||
for j in begin..end {
|
||||
let f = fields_at + 12 * j;
|
||||
let name_off = be32(b, f + 4);
|
||||
fields.push(IdxdField {
|
||||
key: be32(b, f),
|
||||
name: (name_off != NO_NAME).then(|| string_at(name_off)).flatten(),
|
||||
value: string_at(be32(b, f + 8))?,
|
||||
});
|
||||
}
|
||||
out.push(IdxdRecord {
|
||||
name: string_at(be32(b, r + 4))?,
|
||||
name_hash: be32(b, r),
|
||||
fields,
|
||||
});
|
||||
}
|
||||
Some(out)
|
||||
}
|
||||
|
||||
/// A field `name_off` of `0xFFFF_FFFF` means the field is positional, not named.
|
||||
const NO_NAME: u32 = 0xFFFF_FFFF;
|
||||
|
||||
/// Extract the trailing string pool. Finds the smallest offset whose suffix is
|
||||
/// ≥98% printable-ASCII-or-NUL (the pool runs to end-of-buffer), then tokenises
|
||||
/// into maximal printable runs (NUL *or* any non-printable byte separates tokens,
|
||||
@@ -282,9 +476,9 @@ fn is_key_like(s: &str) -> bool {
|
||||
fn parse_number(s: &str) -> bool {
|
||||
!s.is_empty()
|
||||
&& s.bytes().any(|b| b.is_ascii_digit())
|
||||
&& s.bytes().enumerate().all(|(i, b)| {
|
||||
b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+'))
|
||||
})
|
||||
&& s.bytes()
|
||||
.enumerate()
|
||||
.all(|(i, b)| b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+')))
|
||||
}
|
||||
|
||||
#[inline]
|
||||
@@ -358,7 +552,7 @@ mod tests {
|
||||
fn resolved_fields_lists_explicit_values_only() {
|
||||
let bytes = synth(&[
|
||||
"rou_f001", "Model", // identifier value → excluded
|
||||
"10.0", "Size_X", // explicit → included
|
||||
"10.0", "Size_X", // explicit → included
|
||||
"FCSRange", // defaulted (no value) → excluded
|
||||
"Yes", "Mounted", // enum → included
|
||||
]);
|
||||
|
||||
291
crates/sylpheed-formats/tests/idxd_records_disc.rs
Normal file
291
crates/sylpheed-formats/tests/idxd_records_disc.rs
Normal file
@@ -0,0 +1,291 @@
|
||||
//! The IDXD record/field table, checked against every IDXD object on the disc.
|
||||
//!
|
||||
//! The binary region in front of the string pool was undecoded for a long time
|
||||
//! (`idxd.rs` used to say so). It is a record array plus a field array, and the
|
||||
//! decisive evidence is that every record reproduces its own stored hash and
|
||||
//! every named field reproduces its own key: the layout has no free parameters
|
||||
//! left once that holds 1.4 million times.
|
||||
//!
|
||||
//! `first_header_word_is_record0_hash` is the one that demotes `schema_hash`.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use sylpheed_formats::hash::tag_hash;
|
||||
use sylpheed_formats::{IdxdObject, PakArchive};
|
||||
|
||||
fn disc_root() -> Option<PathBuf> {
|
||||
if let Ok(p) = std::env::var("SYLPHEED_DISC") {
|
||||
let p = PathBuf::from(p);
|
||||
if p.join("dat").is_dir() {
|
||||
return Some(p);
|
||||
}
|
||||
}
|
||||
let default = Path::new(
|
||||
"/home/fabi/RE - Project Sylpheed/Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja)",
|
||||
);
|
||||
default.join("dat").is_dir().then(|| default.to_path_buf())
|
||||
}
|
||||
|
||||
macro_rules! skip_without_disc {
|
||||
($root:ident) => {
|
||||
let Some($root) = disc_root() else {
|
||||
eprintln!("SKIP: extracted disc not found (set SYLPHEED_DISC to enable)");
|
||||
return;
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
/// Every `.pak` on the disc, recursively.
|
||||
///
|
||||
/// Deliberately the whole disc root, not `dat/`: `hidden/DefTables.pak` holds
|
||||
/// another 1425 IDXD objects, and an earlier version of this sweep missed them.
|
||||
fn all_paks(root: &Path) -> Vec<PathBuf> {
|
||||
let mut out = Vec::new();
|
||||
let mut stack = vec![root.to_path_buf()];
|
||||
while let Some(dir) = stack.pop() {
|
||||
let Ok(rd) = std::fs::read_dir(&dir) else {
|
||||
continue;
|
||||
};
|
||||
for e in rd.flatten() {
|
||||
let p = e.path();
|
||||
if p.is_dir() {
|
||||
stack.push(p);
|
||||
} else if p.extension().is_some_and(|x| x == "pak") {
|
||||
out.push(p);
|
||||
}
|
||||
}
|
||||
}
|
||||
out.sort();
|
||||
out
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn records_roundtrip_disc() {
|
||||
skip_without_disc!(root);
|
||||
|
||||
let (mut objects, mut hashed, mut unparsed) = (0usize, 0usize, 0usize);
|
||||
let mut bad: Vec<String> = Vec::new();
|
||||
|
||||
for pak in all_paks(&root) {
|
||||
let Ok(ar) = PakArchive::open(&pak) else {
|
||||
continue;
|
||||
};
|
||||
for entry in ar.entries() {
|
||||
let Ok(bytes) = ar.read(entry) else { continue };
|
||||
if !IdxdObject::is_idxd(&bytes) {
|
||||
continue;
|
||||
}
|
||||
let Ok(obj) = IdxdObject::parse(&bytes) else {
|
||||
continue;
|
||||
};
|
||||
objects += 1;
|
||||
let Some(recs) = obj.records() else {
|
||||
unparsed += 1;
|
||||
bad.push(format!(
|
||||
"{}:{:08x} record region did not parse",
|
||||
pak.display(),
|
||||
entry.name_hash
|
||||
));
|
||||
continue;
|
||||
};
|
||||
assert_eq!(
|
||||
recs.len(),
|
||||
obj.count as usize,
|
||||
"{}:{:08x} record count",
|
||||
pak.display(),
|
||||
entry.name_hash
|
||||
);
|
||||
for r in recs {
|
||||
hashed += 1;
|
||||
if r.name_hash != tag_hash(&r.name) {
|
||||
if bad.len() < 10 {
|
||||
bad.push(format!(
|
||||
"{}:{:08x} record {:?} stored {:08x} != tag_hash {:08x}",
|
||||
pak.display(),
|
||||
entry.name_hash,
|
||||
r.name,
|
||||
r.name_hash,
|
||||
tag_hash(&r.name)
|
||||
));
|
||||
}
|
||||
unparsed += 1;
|
||||
}
|
||||
}
|
||||
// Records are stored sorted by hash so the guest can binary-search.
|
||||
assert!(
|
||||
recs.windows(2).all(|w| w[0].name_hash <= w[1].name_hash),
|
||||
"{}:{:08x} records not sorted by hash",
|
||||
pak.display(),
|
||||
entry.name_hash
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
eprintln!("IDXD objects {objects}, hashed records {hashed}, failures {unparsed}");
|
||||
assert!(
|
||||
bad.is_empty(),
|
||||
"{} failures, first few:\n{}",
|
||||
unparsed,
|
||||
bad.join("\n")
|
||||
);
|
||||
// Guard against the sweep silently finding nothing.
|
||||
assert_eq!(objects, 7750, "IDXD object total changed");
|
||||
assert_eq!(hashed, 190_782, "record total changed");
|
||||
}
|
||||
|
||||
/// A concrete, human-checkable row: the movie table keys its cutscene ids as
|
||||
/// *literal integers*, which the string-pool reader could never have told apart
|
||||
/// from field names.
|
||||
#[test]
|
||||
fn movie_table_ids_are_literal_field_keys() {
|
||||
skip_without_disc!(root);
|
||||
let ar = PakArchive::open(root.join("dat/tables.pak")).expect("tables.pak");
|
||||
|
||||
let mut found = false;
|
||||
for entry in ar.entries() {
|
||||
let Ok(bytes) = ar.read(entry) else { continue };
|
||||
if !IdxdObject::is_idxd(&bytes) {
|
||||
continue;
|
||||
}
|
||||
let obj = IdxdObject::parse(&bytes).unwrap();
|
||||
if obj.schema_hash != 0x0670_25B9 {
|
||||
continue;
|
||||
}
|
||||
let base = obj.record("BASE_INFO").expect("BASE_INFO record");
|
||||
// The five named fields the movie GamePart reads.
|
||||
assert_eq!(base.get("PATH"), Some("dat\\movie\\"));
|
||||
assert_eq!(base.get("VERSION"), Some("0x060329"));
|
||||
// …and the numeric ids, which are keys, not names.
|
||||
assert_eq!(
|
||||
base.field_at(105).map(|f| f.value.as_str()),
|
||||
Some("STAGE01_PHASE01")
|
||||
);
|
||||
assert_eq!(
|
||||
base.field_at(205).map(|f| f.value.as_str()),
|
||||
Some("STAGE02_PHASE01")
|
||||
);
|
||||
|
||||
// Each id names a record that carries the actual file names.
|
||||
let rec = obj.record("STAGE02_PHASE01").expect("STAGE02_PHASE01");
|
||||
assert_eq!(rec.get("MOVIE"), Some("RT02A.wmv"));
|
||||
assert_eq!(rec.get("VOICETRACK"), Some("VOICE_RT02A"));
|
||||
|
||||
// Every literal-keyed BASE_INFO field must name a real record.
|
||||
let ids: Vec<u32> = base.fields.iter().filter_map(|f| f.index()).collect();
|
||||
assert!(ids.len() > 90, "only {} literal ids", ids.len());
|
||||
for f in base.fields.iter().filter(|f| f.index().is_some()) {
|
||||
assert!(
|
||||
obj.record(&f.value).is_some(),
|
||||
"id {} -> {:?} has no record",
|
||||
f.key,
|
||||
f.value
|
||||
);
|
||||
}
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
assert!(
|
||||
found,
|
||||
"movie table (schema 067025b9) not found in tables.pak"
|
||||
);
|
||||
}
|
||||
|
||||
/// The header word this crate calls `schema_hash` is record 0's name hash. If
|
||||
/// that were a coincidence it would not survive 6325 objects.
|
||||
#[test]
|
||||
fn first_header_word_is_record0_hash() {
|
||||
skip_without_disc!(root);
|
||||
let (mut checked, mut mismatched) = (0usize, 0usize);
|
||||
for pak in all_paks(&root) {
|
||||
let Ok(ar) = PakArchive::open(&pak) else {
|
||||
continue;
|
||||
};
|
||||
for entry in ar.entries() {
|
||||
let Ok(bytes) = ar.read(entry) else { continue };
|
||||
if !IdxdObject::is_idxd(&bytes) {
|
||||
continue;
|
||||
}
|
||||
let Ok(obj) = IdxdObject::parse(&bytes) else {
|
||||
continue;
|
||||
};
|
||||
let Some(first) = obj.first_record() else {
|
||||
continue;
|
||||
};
|
||||
checked += 1;
|
||||
if tag_hash(&first.name) != obj.schema_hash {
|
||||
mismatched += 1;
|
||||
if mismatched <= 5 {
|
||||
eprintln!(
|
||||
"{}:{:08x} header {:08x} != tag_hash({:?}) {:08x}",
|
||||
pak.display(),
|
||||
entry.name_hash,
|
||||
obj.schema_hash,
|
||||
first.name,
|
||||
tag_hash(&first.name)
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
eprintln!("checked {checked}, mismatched {mismatched}");
|
||||
assert_eq!(mismatched, 0);
|
||||
assert_eq!(checked, 7750);
|
||||
}
|
||||
|
||||
/// Field names are stored on disc; the key is `tag_hash` of the stored name.
|
||||
/// Unnamed fields carry a literal positional key instead.
|
||||
#[test]
|
||||
fn field_names_are_stored_disc() {
|
||||
skip_without_disc!(root);
|
||||
let (mut named, mut positional, mut hash_keyed_unnamed, mut bad) =
|
||||
(0usize, 0usize, 0usize, 0usize);
|
||||
for pak in all_paks(&root) {
|
||||
let Ok(ar) = PakArchive::open(&pak) else {
|
||||
continue;
|
||||
};
|
||||
for entry in ar.entries() {
|
||||
let Ok(bytes) = ar.read(entry) else { continue };
|
||||
if !IdxdObject::is_idxd(&bytes) {
|
||||
continue;
|
||||
}
|
||||
let Ok(obj) = IdxdObject::parse(&bytes) else {
|
||||
continue;
|
||||
};
|
||||
let Some(recs) = obj.records() else { continue };
|
||||
for r in recs {
|
||||
for f in &r.fields {
|
||||
match &f.name {
|
||||
Some(n) => {
|
||||
named += 1;
|
||||
if tag_hash(n) != f.key {
|
||||
bad += 1;
|
||||
if bad <= 5 {
|
||||
eprintln!(
|
||||
"{}:{:08x} {:?}.{:?} key {:08x} != {:08x}",
|
||||
pak.display(),
|
||||
entry.name_hash,
|
||||
r.name,
|
||||
n,
|
||||
f.key,
|
||||
tag_hash(n)
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
None if f.key < 0x1_0000 => positional += 1,
|
||||
None => hash_keyed_unnamed += 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
eprintln!(
|
||||
"named {named}, positional {positional}, hash-keyed-but-unnamed {hash_keyed_unnamed}, bad {bad}"
|
||||
);
|
||||
assert_eq!(bad, 0, "named fields must reproduce their key");
|
||||
assert_eq!(named, 1_271_462);
|
||||
assert_eq!(positional, 1_485_073);
|
||||
// A small residue keeps a hash-shaped key with no stored name: the only
|
||||
// fields on the disc whose name still has to be recovered by preimage search.
|
||||
assert_eq!(hash_keyed_unnamed, 504);
|
||||
}
|
||||
@@ -1033,6 +1033,26 @@ premise was wrong.**
|
||||
Candidates: the **7 `.embsec_` sections** (VAs 0x84D0000–0x86AC000, ~129 KB
|
||||
total, executable) or a hashed record in `hidden/MiscBin.pak`. **Finding it
|
||||
gives the actual per-phase clear condition for every stage.**
|
||||
* ✅ **(2026-08-25) The IDXD/IXUD container is fully decoded** — the "binary
|
||||
node/index region" in front of the string pool is a **uniform 16-byte record
|
||||
array** `{name_hash, name_off, field_begin, field_end}` sorted by hash, then a
|
||||
field count, **12-byte fields** `{key, name_off, value_off}` sorted by key, then
|
||||
a pool size and the pool. Verified over the *whole* disc with zero failures:
|
||||
IDXD **7 750/7 750** objects, **190 782/190 782** records, **1 271 462/1 271 462**
|
||||
named fields; IXUD **1 104/1 104** objects, **628 165/628 165** fields (offsets in
|
||||
chars). **Field names are stored on disc**, so no preimage search is needed —
|
||||
only **504** fields disc-wide are hash-keyed with no name.
|
||||
🔴 **Two corrections:** the header word at `0x08` is **not a schema hash**, it is
|
||||
record 0's `name_hash` (7 750/7 750) — the format has no type field at all, so an
|
||||
object's kind is known only from its loader; and the field's middle word is not
|
||||
an `aux` flags word. See [`structures/idxd-container.md`](structures/idxd-container.md).
|
||||
⚠️ My first disc sweep globbed `dat/**` and **missed `hidden/DefTables.pak`**
|
||||
(1 425 objects); the test now walks the whole disc root.
|
||||
▶️ **Follow-up now open:** the legacy value-before-key string-pool reader is an
|
||||
*approximation* of the real table, and every number in this corpus that came out
|
||||
of `get_f32`/`get_raw` is re-checkable against ground truth but **not yet
|
||||
re-checked**. First step: diff the two readers across the disc and count
|
||||
disagreements. Also open: recover the 504 unnamed hash keys.
|
||||
* ✅ **(2026-08-25) Both guest hash routines located** — `sub_82447DF0` (IDXD)
|
||||
and `sub_82447E70` (IXUD), transcribed instruction-for-instruction into Python
|
||||
and Rust; `cargo test -p sylpheed-formats --lib hash` 10/10. **IXUD SOLVED:**
|
||||
|
||||
@@ -12,7 +12,7 @@ Promote to a prose `structures/…md` file when a format needs behavioural notes
|
||||
|--------|-------|-----------------------|-------|
|
||||
| IPFB `.pak` archive | ✅ | `sylpheed-formats/src/pak.rs` + `tests/pak_idxd_disc.rs` | header + 12-byte TOC, Z1/zlib payloads |
|
||||
| name-hash (TOC keys) | ✅ | `sylpheed-formats/src/hash.rs` | Barrett-reduction hash; recovers original paths |
|
||||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` | self-describing; ship/weapon stats verified vs known values |
|
||||
| IDXD object/table | ✅ | `sylpheed-formats/src/idxd.rs` + `tests/idxd_records_disc.rs` ([container](structures/idxd-container.md)) | **The binary record/index region in front of the string pool is DECODED** (2026-08-25), closing the parser's long-standing "not yet decoded" note. Uniform 16-byte records `{name_hash, name_off, field_begin, field_end}` sorted by hash and binary-searched, then a field count, 12-byte fields `{key, name_off, value_off}` sorted by key, a pool size, and the string pool; the trailing `pool_size == file_len - pool_base` identity makes the layout self-checking. Verified over the **whole disc** with **zero** failures: 7 750/7 750 objects, 190 782/190 782 records reproducing their stored `tag_hash`, 1 271 462/1 271 462 named fields reproducing their key — and `IXUD` is the same container with `ixud_hash`, UTF-16BE and all offsets in **chars** (1 104/1 104 objects, 628 165/628 165 fields). **Field names are stored on disc** — a field's middle word points at its own name — so nothing needs preimage search except the **504** fields disc-wide that are hash-keyed with no name; the other 1 485 073 nameless fields are *positional*, keyed by a literal integer (line slots, movie ids). ⚠️ **Two long-held beliefs WITHDRAWN**: the word at `0x08` is **not a schema hash**, it is record 0's `name_hash` (7 750/7 750) — the header has no type field at all, so an object's kind is known only from the caller that loads it; and the field's middle word is **not** an always-`0xFFFFFFFF` flags word. The first was caught by a test asserting that every movie id names a real record: `1005 -> STAGE10_PHASE01` failed because `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, that table's supposed schema id. 🟡 the legacy value-before-key string-pool reader is now known to be an *approximation* of the real table, and every number derived from it is re-checkable but not yet re-checked |
|
||||
| XPR2 texture + cubemap | 🟡/✅ | `sylpheed-formats/src/texture.rs` + [colour check](xpr2-colour-check.md) | de-tile + A8R8G8B8 and DXT1. **Channel order ✅ confirmed against the running game**: the Delta Saber's decoded atlas is orange-dominant (median saturated hue 23.3°, *zero* cool pixels) and the game renders the same hull at 9.3° — a red↔blue swap would sit at ≈200°. Exact fidelity (gamma/sRGB curve, premultiplied alpha, per-channel scale) is 🟡 untested, since a hue comparison cannot see it; cubemap face ordering ❔ |
|
||||
| T8aD 2D texture | ✅ | `sylpheed-formats/src/t8ad.rs` | **100 % of the disc decodes** (19 216/19 216, measured). The "~15 % deferred variants" were a wrong model, not a variant: a surface is a list of **arbitrary sub-rectangles**, each with a 16-byte header of `dst X, dst Y, width, height`, not a 256×256 grid — `0x1c` is the **rectangle count**. Uncovered area stays transparent. **Colours ✅ CONFIRMED** ([k8888](structures/texture-color-k8888.md)) |
|
||||
| RATC bundle | ✅ | `sylpheed-formats/src/ratc.rs` | child listing confirmed. **"One level deep" is not a limitation — there is nothing deeper**: 2 859 bundles hold 18 002 children at depth 1 and **0 at depth 2**, with no parse failures. Nested RATC blobs are **leaf records that reference siblings by name** (`opt `, the sprite name): 3 311 leaves, all embedding sibling names, **10 144 of 10 148 references resolve**. The 4 that do not are one dangling asset — `pmbase.rat` → `pmbase.t32` in `GP_STAGE_CLEAR.pak`'s four language builds, and `pmbase.t32` is **on the disc nowhere** |
|
||||
@@ -91,6 +91,7 @@ files, which is how the same ground got covered twice.
|
||||
| [`ship-placement-capture-generalisation.md`](ship-placement-capture-generalisation.md) | Capital-ship placement — does the `e106` result generalise? (WIP, 2026-07-31) | 🚧 WIP, time-boxed session. Two results so far: a static audit across all |
|
||||
| [`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) | Capital-ship part placement — runtime capture (ground truth) | ✅✅ STATIC ASSEMBLY IS EXACT — no captures needed anymore |
|
||||
| [`structures/achievements.md`](structures/achievements.md) | Achievements — the 24-entry table, and where the earned state comes from | — |
|
||||
| [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container — record/field table, and the two beliefs it withdraws | ✅ CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures |
|
||||
| [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad — vtable `0x820B2A64` | ✅ CONFIRMED for the object layout and the atlas size, read live off |
|
||||
| [`structures/mission-objective-counter.md`](structures/mission-objective-counter.md) | `REMAINING OB` — the mission's own objective counter, in RAM | ✅ CONFIRMED for one Stage 02 run: a big-endian u32 whose value |
|
||||
| [`structures/movie-subtitles.md`](structures/movie-subtitles.md) | Movie subtitles & the movie ↔ mission ↔ text chain | — |
|
||||
|
||||
157
docs/re/structures/idxd-container.md
Normal file
157
docs/re/structures/idxd-container.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# The IDXD container — record/field table
|
||||
|
||||
Status: ✅ **CONFIRMED**, decoded in full and verified over **the whole disc**
|
||||
(not just `dat/` — `hidden/DefTables.pak` holds another 1425 objects, which the
|
||||
first version of this sweep missed): **7750 / 7750**
|
||||
objects parse, **190,782 / 190,782** records reproduce their stored name hash,
|
||||
**1,271,462 / 1,271,462** named fields reproduce their key from their stored name.
|
||||
Zero failures of any kind.
|
||||
|
||||
This closes the **"binary node/index region — not yet decoded"** note that stood
|
||||
in `crates/sylpheed-formats/src/idxd.rs` for the whole life of the parser, and it
|
||||
**demotes two beliefs the corpus was built on** (see *Two corrections* below).
|
||||
|
||||
Tests: `crates/sylpheed-formats/tests/idxd_records_disc.rs` —
|
||||
`records_roundtrip_disc`, `first_header_word_is_record0_hash`,
|
||||
`field_names_are_stored_disc`, `movie_table_ids_are_literal_field_keys`.
|
||||
|
||||
## ✅ The layout
|
||||
|
||||
All fields big-endian.
|
||||
|
||||
```text
|
||||
Offset Size Field
|
||||
0x00 4 "IDXD"
|
||||
0x04 4 record_count n
|
||||
0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
|
||||
.. 4 field_count m (equals max(field_end))
|
||||
.. 12*m fields { u32 key, u32 name_off, u32 value_off }
|
||||
.. 4 pool_size (equals file_len - pool_base)
|
||||
.. .. string pool (every *_off above is a byte offset from here)
|
||||
```
|
||||
|
||||
* **Records are sorted ascending by `name_hash`** and the guest binary-searches
|
||||
them with a 16-byte stride (`sub_82448AA0`). Verified: sorted in every object.
|
||||
* `name_hash` is [`tag_hash`](idxd-tag-hash.md) of the record's **own name** —
|
||||
not the pak TOC hash (different modulus, and not lowercased).
|
||||
* A record owns the half-open field range `[field_begin, field_end)`. Ranges are
|
||||
*not* contiguous in record order — records are in hash order while their fields
|
||||
sit in name order, so the field array is shared, not partitioned by position.
|
||||
* **Fields are sorted ascending by `key`** and lower-bounded (`sub_8244E338`).
|
||||
* `field_count` and `pool_size` are what make the layout self-checking: a wrong
|
||||
record stride lands on a `pool_size` that does not equal the remaining bytes.
|
||||
That identity is what caught the first wrong version of this decode.
|
||||
|
||||
## ✅ Field names are on the disc
|
||||
|
||||
A field's middle word is **its own name's pool offset**, and then
|
||||
`key == tag_hash(name)`. Field names never have to be recovered from their hash.
|
||||
|
||||
`name_off == 0xFFFF_FFFF` means the field has **no name**; its `key` is then a
|
||||
literal positional integer — a line-slot index (`0,1,2,3`), a movie id (`105`,
|
||||
`1303`). So a field is positional *exactly* when it stores no name; you do not
|
||||
have to guess from the key's magnitude.
|
||||
|
||||
Disc-wide census of all **2,757,039** fields:
|
||||
|
||||
| kind | count |
|
||||
|---|---|
|
||||
| named, `key == tag_hash(name)` | 1,271,462 |
|
||||
| unnamed, literal key (`< 0x10000`) | 1,485,073 |
|
||||
| unnamed, hash-shaped key | 504 |
|
||||
|
||||
Those **504** are the entire remaining preimage problem on the disc. ❔ Their
|
||||
names are unrecovered.
|
||||
|
||||
## Two corrections
|
||||
|
||||
### ❌ WITHDRAWN — "the word at `0x08` is a schema hash"
|
||||
|
||||
The corpus (and `IdxdObject::schema_hash`, and every `pak list` line printing
|
||||
`schema 067025b9`) read the third header word as an object-type id whose preimage
|
||||
was unknown. **It is not a schema id.** It is simply **record 0's `name_hash`**:
|
||||
the record array is uniform 16-byte entries starting at `0x08`, and the header has
|
||||
no type field at all.
|
||||
|
||||
Evidence: `tag_hash(records[0].name) == word@0x08` for **all 7750** objects on the
|
||||
disc, zero exceptions (`first_header_word_is_record0_hash`).
|
||||
|
||||
How it was caught, which is the useful part: a test asserted that every movie id in
|
||||
`BASE_INFO` names a real record, and one — `1005 -> "STAGE10_PHASE01"` — did not
|
||||
resolve. The reason was that `tag_hash("STAGE10_PHASE01")` **is** `0x067025B9`, the
|
||||
movie table's supposed schema id. A "coincidence" at 1-in-2^32 is not a
|
||||
coincidence; the record was being eaten by the header.
|
||||
|
||||
It still *works* as a type discriminator, because tables of one kind share their
|
||||
lowest-hashed record name — which is exactly why it went unquestioned for so long.
|
||||
`schema_hash` is therefore kept under its established name, with its docs corrected,
|
||||
rather than renamed across 33 call sites. **Nothing on disc names an object's type.**
|
||||
|
||||
### ❌ WITHDRAWN — "the field's middle word is an `aux`/flags word, always `0xFFFFFFFF`"
|
||||
|
||||
It is `0xFFFFFFFF` for 54% of fields, which is enough to look constant in a small
|
||||
sample. It is a name offset (above). The tell was that the word is constant *per
|
||||
key across records* (`key=0x6c43a78d` always carried `1743`) — a per-record flag
|
||||
cannot do that, but a per-name pointer must.
|
||||
|
||||
## Worked example — the movie table
|
||||
|
||||
`dat/tables.pak`, the object whose record 0 is `STAGE10_PHASE01` (105 records,
|
||||
433 fields). Its `BASE_INFO` record mixes both field kinds:
|
||||
|
||||
```text
|
||||
named: PATH = "dat\movie\" VERSION = "0x060329" SUBTITLE_FONT …
|
||||
positional: 105 -> "STAGE01_PHASE01" 205 -> "STAGE02_PHASE01" 1005 -> "STAGE10_PHASE01"
|
||||
```
|
||||
|
||||
Each positional value names another record in the same object, which carries the
|
||||
real files:
|
||||
|
||||
```text
|
||||
STAGE02_PHASE01: MOVIE = "RT02A.wmv" VOICETRACK = "VOICE_RT02A"
|
||||
SUBTITLE = "…+SUBTITLE_RT02A.tbl" TELOP = "…+pwrt02.prt"
|
||||
```
|
||||
|
||||
All 104 positional ids resolve to a real record — verified, no dangling entries.
|
||||
This is the id space the mission script's cutscene request uses; see
|
||||
[movie-subtitle-link](../movie-subtitle-link.md).
|
||||
|
||||
## ✅ `IXUD` is the same container
|
||||
|
||||
The wide-string sibling has an identical shape, with three substitutions: the hash
|
||||
is `ixud_hash`, strings are UTF-16BE, and **every offset — record name, field name
|
||||
and value alike — is in 16-bit chars**, so `pool_base + 2*off`. `pool_size` is
|
||||
likewise a char count, which is the same identity as the already-known
|
||||
`STR + 2*strsize == filesize` ([ixud-localised-text](ixud-localised-text.md)).
|
||||
|
||||
Verified independently over all **1104** IXUD objects on the disc: the header word
|
||||
at `0x08` is record 0's `ixud_hash` (1104/1104), and `key == ixud_hash(field name)`
|
||||
for **628,165 / 628,165** named fields, zero mismatches. Only 48 fields are
|
||||
unnamed — 6 objects × 8, all in a `CATEGORY_DESC` record with keys `0..7`, a
|
||||
positional array of weapon-category descriptions.
|
||||
|
||||
One extra rule shows up in IXUD's comparator (`sub_82447F38`) and is worth assuming
|
||||
for IDXD too: `name_off == 0xFFFF_FFFF` is the **primary** sort key, so a record's
|
||||
field slice is *unnamed fields first, then named fields*, each key-ascending
|
||||
(1476/1476 slices). That is why there are two getters — lookup-by-integer searches
|
||||
only the unnamed run, lookup-by-name only the named run.
|
||||
|
||||
🟡 The loader `sub_82448D00` also accepts legacy magics `IIDX`, `IDX2`, `IDX3`,
|
||||
`IDXC` and rejects `IDXD` on that path with `"Old virsion binary table. Not
|
||||
supported."` [sic]. **None of them occur on this disc** (magic census over every
|
||||
pak entry: `IDXD` 7750, `T8aD` 4525, `RATC` 2985, `IXUD` 1104, `LSTA` 64, and zero
|
||||
of the four legacy magics), so that path is read-only knowledge, untestable here.
|
||||
|
||||
## What this does *not* settle
|
||||
|
||||
* ❔ The names of the 504 unnamed hash-keyed fields.
|
||||
* ❔ How much of the existing corpus the old string-pool reader got wrong. The
|
||||
legacy `get_f32`/`get_raw` path infers a field's value from *pool adjacency*
|
||||
(`<value>\0<key>\0`). That adjacency is a consequence of the field table, not a
|
||||
rule of the format, and it cannot represent a field whose value string is shared
|
||||
or reordered. Every number in `docs/re/` that came from it is now re-checkable
|
||||
against the true table, and has not yet been re-checked.
|
||||
* ❔ Whether `IXUD`, the wide-string sibling, uses the same shape. Its offsets are
|
||||
in 16-bit chars and its hash is different ([ixud-localised-text](ixud-localised-text.md)).
|
||||
* The type-identification question is now open rather than closed: with no schema
|
||||
field, an object's kind is known only from the caller that loads it.
|
||||
Reference in New Issue
Block a user