fabi/Syplheed-Reborn
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

re: decode the IDXD/IXUD record table — and there is no schema hash

Browse Source 
The binary region in front of the string pool was the parser's oldest open
note ("Not yet decoded"). It is a uniform 16-byte record array sorted by
name hash, a field count, a 12-byte field array sorted by key, a pool size,
and the pool. The trailing `pool_size == file_len - pool_base` identity makes
the layout self-checking, which is what caught the first wrong version.

Verified over the WHOLE disc with zero failures: 7750/7750 IDXD objects,
190782/190782 records reproducing their stored tag_hash, 1271462/1271462
named fields reproducing their key. IXUD is the same container with
ixud_hash, UTF-16BE and every offset in chars — 1104/1104 objects,
628165/628165 fields, checked with an independent parser.

Field names are stored on disc, so no preimage search is needed: a field's
middle word points at its own name. Only 504 fields disc-wide are hash-keyed
with no name; the other 1485073 nameless fields are positional, keyed by a
literal integer (line slots, movie ids).

Two long-held beliefs are WITHDRAWN:

* The word at 0x08 is not a schema hash. It is record 0's name_hash — the
  format has no type field at all, and an object's kind is known only from
  the caller that loads it. It survived as "schema" because tables of one
  kind share their lowest-hashed record name. Caught by a test asserting
  every movie id names a real record: 1005 -> STAGE10_PHASE01 failed because
  tag_hash("STAGE10_PHASE01") IS 0x067025B9, that table's supposed schema id.
* The field's middle word is not an always-0xFFFFFFFF flags word. It is
  0xFFFFFFFF for 54% of fields, enough to look constant in a small sample;
  the tell was that it is constant per key ACROSS records, which a per-record
  flag cannot be but a per-name pointer must.

`schema_hash` keeps its name rather than churn 33 call sites, with corrected
docs. The first sweep globbed dat/** and missed hidden/DefTables.pak (1425
objects); the test now walks the whole disc root.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-25 21:40:23 +00:00
parent 34c99a1568
commit af32540190
5 changed files with 685 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

236
crates/sylpheed-formats/src/idxd.rs
View File

@@ -8,15 +8,44 @@
//! ## Layout (all fields big-endian)
//!
//! ```text
//! Offset Size Field
//! 0x00 4 Magic: "IDXD"
//! 0x04 4 count (number of top-level records; small)
//! 0x08 4 schema_hash (identifies the object type; custom hash family, preimage unknown)
//! 0x0C .. NODE / INDEX region — hash-keyed records + (for some schemas) auxiliary
//! binary tables. Not required for the string values below.
//! .. .. STRING POOL — the trailing, (almost) all-ASCII region.
//! Offset Size Field
//! 0x00 4 Magic: "IDXD"
//! 0x04 4 record_count n
//! 0x08 16*n records { u32 name_hash, u32 name_off, u32 field_begin, u32 field_end }
//! .. 4 field_count m (equals max(field_end))
//! .. 12*m fields { u32 key, u32 name_off, u32 value_off }
//! .. 4 pool_size (equals file_len − pool_base)
//! .. .. STRING POOL — every `*_off` above is a byte offset from here.
//! ```
//!
//! Records are sorted ascending by `name_hash` and the guest binary-searches them
//! (`sub_82448AA0`); fields are sorted ascending by `key` and lower-bounded
//! (`sub_8244E338`). `name_hash` is [`crate::hash::tag_hash`] of the record's own
//! name — **not** the pak TOC hash: different modulus, and not lowercased.
//!
//! A field's `name_off` points at its own name, so **field names are on the disc**
//! and never have to be recovered from their hash; `key` is then `tag_hash(name)`.
//! A `name_off` of `0xFFFF_FFFF` means the field has no name, and its `key` is a
//! literal positional integer instead — a line-slot index, a movie id
//! ([`IdxdField::index`]).
//!
//! ## There is no schema field
//!
//! The word at `0x08` was long read as a `schema_hash` identifying the object
//! type. It is not: it is simply **record 0's `name_hash`**, the first cell of a
//! uniform 16-byte record array. `tag_hash(records[0].name)` reproduces it for
//! **all 7750** IDXD objects on the retail disc. It still works as a type
//! discriminator — tables of one kind share their lowest-hashed record name — so
//! [`IdxdObject::schema_hash`] is kept under its established name, but it
//! identifies a *record name*, not a schema. Nothing on disc names the type.
//!
//! Verified across the whole retail disc (`records_roundtrip_disc`,
//! `field_names_are_stored_disc`): all **7750** objects parse under this layout;
//! all **190782** records reproduce their stored `name_hash`; all **1271462**
//! named fields reproduce their `key` from their stored name; the remaining
//! **1485073** fields are unnamed with literal keys, and just **504** are
//! hash-keyed with no name stored.
//!
//! ## The string pool: **value-before-key, defaults omitted**
//!
//! Each property that has an explicit value is serialized as `<value>\0<key>\0` —
@@ -44,14 +73,34 @@
//! - [`IdxdObject::get_raw`] — the raw preceding token, no validation. Use for fields
//! you *know* are identifier-valued (`ID`, `Name`, `Type`, `Model`).
//!
//! ## Not yet decoded
//! ## Two readers, one object
//!
//! Default/omitted fields (many ratios, the `…Count` family) carry no value in the
//! pool — their values live in the binary node/index region (hash-keyed by the same
//! unrecovered custom hash) or come from schema defaults. Binding those requires
//! decoding that region per schema; until then the typed getters return `None` for
//! them (honest "unknown") rather than guessing. See `reference_ipfb_archive_format`
//! in the project notes.
//! The record/field table above is exact, so prefer it: [`IdxdObject::records`],
//! [`record`](IdxdObject::record), [`IdxdRecord::field`]. The older
//! *value-before-key string-pool* reader ([`get_f32`](IdxdObject::get_f32) and
//! friends, documented below) predates the decode and is kept because a large
//! part of the corpus is written in terms of it.
//!
//! ## The legacy string-pool reader: **value-before-key, defaults omitted**
//!
//! In the pool each property that has an explicit value is laid out as
//! `<value>\0<key>\0` — the value string comes *immediately before* its field
//! name. Numbers are stored as ASCII text (`"600.0"`, `"144"`), so they read out
//! directly:
//!
//! ```text
//! … "1000.0" "HP" "500000.0" "RadarRange" "10.0" "Size_X" …
//! value key value key value key
//! ```
//!
//! Fields left at their default omit the value string, appearing as a bare key.
//! So "the token before a key" is a real value *only when that token is itself
//! value-shaped*, which the typed getters enforce ([`get_f32`](IdxdObject::get_f32),
//! [`get_i64`](IdxdObject::get_i64), [`get_str`](IdxdObject::get_str) validate;
//! [`get_raw`](IdxdObject::get_raw) does not). The adjacency is a *consequence* of
//! the field table — each record's values are emitted next to their keys — not a
//! rule of the format, and it cannot see a field whose value string is shared or
//! reordered. Use the record API when correctness matters.
use thiserror::Error;
@@ -70,13 +119,75 @@ pub enum IdxdError {
/// A parsed IDXD object: its schema id plus the decoded string pool.
#[derive(Debug, Clone)]
pub struct IdxdObject {
/// Object-type id (which kind of definition this is). Custom hash; groups
/// entries by schema even though the preimage is unknown.
/// The word at `0x08`. Despite the name it is **record 0's `name_hash`**, not
/// a schema id — see the module docs. Retained because it does discriminate
/// object kinds in practice and the corpus is written in terms of it; prefer
/// [`records`](Self::records) when you want meaning rather than a bucket key.
pub schema_hash: u32,
/// The `count` header field (number of top-level records).
/// The `record_count` header field.
pub count: u32,
/// Ordered string-pool tokens (see module docs; value-before-key).
tokens: Vec<String>,
/// The decoded record/field table. `None` only when the binary region does
/// not parse — which no retail object does; see `records_roundtrip_disc`.
records: Option<Vec<IdxdRecord>>,
}
/// One named record of an [`IdxdObject`] — a row of the table.
#[derive(Debug, Clone)]
pub struct IdxdRecord {
/// The record's name, from the string pool.
pub name: String,
/// `tag_hash(name)`, as stored.
pub name_hash: u32,
/// This record's fields, in on-disc order (ascending `key`).
pub fields: Vec<IdxdField>,
}
/// One field of an [`IdxdRecord`] — a key/value cell.
#[derive(Debug, Clone)]
pub struct IdxdField {
/// `tag_hash(name)` when the field is named, else a literal positional
/// integer. See [`IdxdField::index`].
pub key: u32,
/// The field's own name, when it has one. `None` for positional fields.
pub name: Option<String>,
/// The field's value, from the string pool. Values are ASCII text, including
/// numbers (`"600.0"`).
pub value: String,
}
impl IdxdField {
/// The key read as a literal positional index. `None` for a named field.
///
/// Keyed off the stored name rather than off the key's magnitude, so it is
/// exact: a field is positional precisely when it has no name.
pub fn index(&self) -> Option<u32> {
self.name.is_none().then_some(self.key)
}
/// Whether this field is named `name`.
pub fn is_named(&self, name: &str) -> bool {
self.name.as_deref() == Some(name)
}
}
impl IdxdRecord {
/// The field named `name` (matched by `tag_hash`), if present.
pub fn field(&self, name: &str) -> Option<&IdxdField> {
let key = crate::hash::tag_hash(name);
self.fields.iter().find(|f| f.key == key)
}
/// The field stored under the literal integer key `index`, if present.
pub fn field_at(&self, index: u32) -> Option<&IdxdField> {
self.fields.iter().find(|f| f.key == index)
}
/// The value of the field named `name`.
pub fn get(&self, name: &str) -> Option<&str> {
self.field(name).map(|f| f.value.as_str())
}
}
impl IdxdObject {
@@ -92,10 +203,12 @@ impl IdxdObject {
let count = be32(bytes, 4);
let schema_hash = be32(bytes, 8);
let tokens = extract_string_pool(bytes);
let records = parse_records(bytes, count);
Ok(Self {
schema_hash,
count,
tokens,
records,
})
}
@@ -109,6 +222,25 @@ impl IdxdObject {
&self.tokens
}
/// The decoded record/field table, or `None` if the binary region did not
/// parse. Every retail object parses; a `None` here means a malformed or
/// synthetic buffer.
pub fn records(&self) -> Option<&[IdxdRecord]> {
self.records.as_deref()
}
/// The record named `name` (matched by `tag_hash`), if the table parsed.
pub fn record(&self, name: &str) -> Option<&IdxdRecord> {
let key = crate::hash::tag_hash(name);
self.records.as_ref()?.iter().find(|r| r.name_hash == key)
}
/// Record 0 — the one whose hash sits in the header word this crate calls
/// [`schema_hash`](Self::schema_hash).
pub fn first_record(&self) -> Option<&IdxdRecord> {
self.records.as_ref()?.first()
}
/// The raw token immediately preceding the first occurrence of `key`.
///
/// No validation: for a defaulted/omitted field this is the *neighbouring key*,
@@ -206,6 +338,68 @@ impl IdxdObject {
}
}
/// Decode the record/field table (see module docs). Returns `None` — rather than
/// erroring — when any bound is inconsistent, so a malformed or synthetic buffer
/// still yields a usable string-pool reader.
fn parse_records(b: &[u8], count: u32) -> Option<Vec<IdxdRecord>> {
let n = count as usize;
if n == 0 || n > b.len() / 16 {
return None;
}
let recs_at = 0x08_usize;
let field_count_at = recs_at.checked_add(n.checked_mul(16)?)?;
if field_count_at + 4 > b.len() {
return None;
}
let m = be32(b, field_count_at) as usize;
let fields_at = field_count_at + 4;
let pool_size_at = fields_at.checked_add(m.checked_mul(12)?)?;
let pool = pool_size_at.checked_add(4)?;
if pool > b.len() {
return None;
}
// The trailing identity: the stored pool size is exactly what is left. This is
// what makes the layout self-checking — a wrong record stride lands here.
if be32(b, pool_size_at) as usize != b.len() - pool {
return None;
}
let string_at = |off: u32| -> Option<String> {
let start = pool.checked_add(off as usize)?;
if start >= b.len() {
return None;
}
let end = start + b[start..].iter().position(|&c| c == 0)?;
Some(String::from_utf8_lossy(&b[start..end]).into_owned())
};
let mut out = Vec::with_capacity(n);
for i in 0..n {
let r = recs_at + 16 * i;
let (begin, end) = (be32(b, r + 8) as usize, be32(b, r + 12) as usize);
if begin > end || end > m {
return None;
}
let mut fields = Vec::with_capacity(end - begin);
for j in begin..end {
let f = fields_at + 12 * j;
let name_off = be32(b, f + 4);
fields.push(IdxdField {
key: be32(b, f),
name: (name_off != NO_NAME).then(|| string_at(name_off)).flatten(),
value: string_at(be32(b, f + 8))?,
});
}
out.push(IdxdRecord {
name: string_at(be32(b, r + 4))?,
name_hash: be32(b, r),
fields,
});
}
Some(out)
}
/// A field `name_off` of `0xFFFF_FFFF` means the field is positional, not named.
const NO_NAME: u32 = 0xFFFF_FFFF;
/// Extract the trailing string pool. Finds the smallest offset whose suffix is
/// ≥98% printable-ASCII-or-NUL (the pool runs to end-of-buffer), then tokenises
/// into maximal printable runs (NUL *or* any non-printable byte separates tokens,
@@ -282,9 +476,9 @@ fn is_key_like(s: &str) -> bool {
fn parse_number(s: &str) -> bool {
!s.is_empty()
&& s.bytes().any(|b| b.is_ascii_digit())
&& s.bytes().enumerate().all(|(i, b)| {
b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+'))
})
&& s.bytes()
.enumerate()
.all(|(i, b)| b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+')))
}
#[inline]
@@ -358,7 +552,7 @@ mod tests {
fn resolved_fields_lists_explicit_values_only() {
let bytes = synth(&[
"rou_f001", "Model", // identifier value → excluded
"10.0", "Size_X", // explicit → included
"10.0", "Size_X", // explicit → included
"FCSRange", // defaulted (no value) → excluded
"Yes", "Mounted", // enum → included
]);