feat(formats): IPFB archive + IDXD definition readers
Add readers for Project Sylpheed's on-disc data format so the reimplementation can load exact ship/weapon constants straight from the game files. - pak.rs: PakArchive — parse the IPFB `.pak` index + concatenated `.pNN` segments, binary-search the TOC by name-hash, and transparently inflate the per-entry "Z1" (zlib) container. Adds flate2. - idxd.rs: IdxdObject — parse the IDXD reflective object serialization. The string pool stores properties value-before-key with defaulted fields omitted; typed getters (get_f32/get_i64/get_str/get_bool) read explicit values reliably, get_raw exposes identifier fields, and resolved_fields() enumerates every explicit (key,value). Defaulted fields correctly return None rather than a neighbouring key. - sylpheed-cli: `pak list` (inventory entries + identity) and `pak dump <hash>` (full stat sheet for one object). Verified against the real disc (auto-skipped without it): DeltaSaber craft (HP=1000, Acceleration=600, velocity curve 100/700/1200, Turn=100, RadarRange=500000) and the DSaber missile (LoadingCount=144, Interval=3.00, Mass=0.77). 10 unit + 3 integration tests pass. Not yet decoded: defaulted fields (many ratios, the *Count family) whose values come from schema defaults or the hash-keyed binary node region. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
347
crates/sylpheed-formats/src/idxd.rs
Normal file
347
crates/sylpheed-formats/src/idxd.rs
Normal file
@@ -0,0 +1,347 @@
|
||||
//! `IDXD` reflective object reader — the game's data-definition serialization.
|
||||
//!
|
||||
//! Almost every gameplay definition (ships/craft, weapons, effects, menu configs)
|
||||
//! is an `IDXD` object embedded as a [`crate::pak`] entry. Each object is a small
|
||||
//! reflective property tree: a **binary node/index region** followed by a
|
||||
//! **string pool**.
|
||||
//!
|
||||
//! ## Layout (all fields big-endian)
|
||||
//!
|
||||
//! ```text
|
||||
//! Offset Size Field
|
||||
//! 0x00 4 Magic: "IDXD"
|
||||
//! 0x04 4 count (number of top-level records; small)
|
||||
//! 0x08 4 schema_hash (identifies the object type; custom hash family, preimage unknown)
|
||||
//! 0x0C .. NODE / INDEX region — hash-keyed records + (for some schemas) auxiliary
|
||||
//! binary tables. Not required for the string values below.
|
||||
//! .. .. STRING POOL — the trailing, (almost) all-ASCII region.
|
||||
//! ```
|
||||
//!
|
||||
//! ## The string pool: **value-before-key, defaults omitted**
|
||||
//!
|
||||
//! Each property that has an explicit value is serialized as `<value>\0<key>\0` —
|
||||
//! the value string comes *immediately before* its field name. Numbers are stored
|
||||
//! as ASCII text (`"600.0"`, `"144"`, `"0.3333"`), so they read out exactly:
|
||||
//!
|
||||
//! ```text
|
||||
//! … "1000.0" "HP" "500000.0" "RadarRange" "10.0" "Size_X" …
|
||||
//! value key value key value key
|
||||
//! ```
|
||||
//!
|
||||
//! Crucially, **fields left at their default value omit the value string** — they
|
||||
//! appear as a bare key with the *previous* field's territory ending before them
|
||||
//! (e.g. `… "RadarRange" "FCSRange" "YES" "MountedShieldGenerator" …`: `FCSRange`
|
||||
//! has no value). So "the token before a key" is a real value *only when that token
|
||||
//! is itself value-shaped*. The typed getters enforce this:
|
||||
//!
|
||||
//! - [`IdxdObject::get_f32`] / [`get_i64`](IdxdObject::get_i64) — parse the preceding
|
||||
//! token as a number; `None` if the field is absent or defaulted. **Reliable** —
|
||||
//! this is the API for numeric stats.
|
||||
//! - [`IdxdObject::get_str`] — the preceding token, but only when it is unambiguously
|
||||
//! a value (numeric, a `Yes/No`-style enum, or a delimited string like
|
||||
//! `"Vessel,Craft"`). `None` for bare identifiers, to avoid mistaking a neighbour
|
||||
//! key for a value.
|
||||
//! - [`IdxdObject::get_raw`] — the raw preceding token, no validation. Use for fields
|
||||
//! you *know* are identifier-valued (`ID`, `Name`, `Type`, `Model`).
|
||||
//!
|
||||
//! ## Not yet decoded
|
||||
//!
|
||||
//! Default/omitted fields (many ratios, the `…Count` family) carry no value in the
|
||||
//! pool — their values live in the binary node/index region (hash-keyed by the same
|
||||
//! unrecovered custom hash) or come from schema defaults. Binding those requires
|
||||
//! decoding that region per schema; until then the typed getters return `None` for
|
||||
//! them (honest "unknown") rather than guessing. See `reference_ipfb_archive_format`
|
||||
//! in the project notes.
|
||||
|
||||
use thiserror::Error;
|
||||
|
||||
/// Magic at the start of every IDXD object.
|
||||
pub const IDXD_MAGIC: [u8; 4] = *b"IDXD";
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum IdxdError {
|
||||
#[error("bad IDXD magic: expected {:?}, got {got:?}", IDXD_MAGIC)]
|
||||
BadMagic { got: [u8; 4] },
|
||||
|
||||
#[error("truncated IDXD object: {0} bytes")]
|
||||
Truncated(usize),
|
||||
}
|
||||
|
||||
/// A parsed IDXD object: its schema id plus the decoded string pool.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct IdxdObject {
|
||||
/// Object-type id (which kind of definition this is). Custom hash; groups
|
||||
/// entries by schema even though the preimage is unknown.
|
||||
pub schema_hash: u32,
|
||||
/// The `count` header field (number of top-level records).
|
||||
pub count: u32,
|
||||
/// Ordered string-pool tokens (see module docs; value-before-key).
|
||||
tokens: Vec<String>,
|
||||
}
|
||||
|
||||
impl IdxdObject {
|
||||
/// Parse an IDXD object from a decompressed [`crate::pak`] entry.
|
||||
pub fn parse(bytes: &[u8]) -> Result<Self, IdxdError> {
|
||||
if bytes.len() < 12 {
|
||||
return Err(IdxdError::Truncated(bytes.len()));
|
||||
}
|
||||
let magic: [u8; 4] = bytes[0..4].try_into().unwrap();
|
||||
if magic != IDXD_MAGIC {
|
||||
return Err(IdxdError::BadMagic { got: magic });
|
||||
}
|
||||
let count = be32(bytes, 4);
|
||||
let schema_hash = be32(bytes, 8);
|
||||
let tokens = extract_string_pool(bytes);
|
||||
Ok(Self {
|
||||
schema_hash,
|
||||
count,
|
||||
tokens,
|
||||
})
|
||||
}
|
||||
|
||||
/// Quick check for the IDXD magic without fully parsing.
|
||||
pub fn is_idxd(bytes: &[u8]) -> bool {
|
||||
bytes.len() >= 4 && bytes[0..4] == IDXD_MAGIC
|
||||
}
|
||||
|
||||
/// The ordered string-pool tokens (value-before-key interleaving).
|
||||
pub fn tokens(&self) -> &[String] {
|
||||
&self.tokens
|
||||
}
|
||||
|
||||
/// The raw token immediately preceding the first occurrence of `key`.
|
||||
///
|
||||
/// No validation: for a defaulted/omitted field this is the *neighbouring key*,
|
||||
/// not a value. Prefer [`get_str`](Self::get_str) / [`get_f32`](Self::get_f32)
|
||||
/// unless you know the field is identifier-valued (`ID`, `Name`, `Model`, …).
|
||||
pub fn get_raw(&self, key: &str) -> Option<&str> {
|
||||
let idx = self.tokens.iter().position(|t| t == key)?;
|
||||
if idx == 0 {
|
||||
return None;
|
||||
}
|
||||
Some(&self.tokens[idx - 1])
|
||||
}
|
||||
|
||||
/// The value of `key`, but only when the preceding token is unambiguously a
|
||||
/// value (numeric, a `Yes/No`-style enum, or a delimited string). Returns `None`
|
||||
/// for bare-identifier neighbours (which would usually be a defaulted field).
|
||||
pub fn get_str(&self, key: &str) -> Option<&str> {
|
||||
let v = self.get_raw(key)?;
|
||||
is_definite_value(v).then_some(v)
|
||||
}
|
||||
|
||||
/// `key`'s value parsed as `f32` (ASCII; a trailing `f` as in `"2.0f"` is
|
||||
/// tolerated). `None` if the field is missing, defaulted, or non-numeric.
|
||||
/// This is the reliable API for numeric stats.
|
||||
pub fn get_f32(&self, key: &str) -> Option<f32> {
|
||||
let v = self.get_raw(key)?;
|
||||
let v = v.strip_suffix(['f', 'F']).unwrap_or(v);
|
||||
parse_number(v).then(|| v.parse().ok()).flatten()
|
||||
}
|
||||
|
||||
/// `key`'s value parsed as `i64`. `None` if missing, defaulted, or non-integer.
|
||||
pub fn get_i64(&self, key: &str) -> Option<i64> {
|
||||
let v = self.get_raw(key)?;
|
||||
v.parse().ok()
|
||||
}
|
||||
|
||||
/// `key`'s value as a bool (`Yes`/`YES`/`On`/`1` → true; `No`/`NO`/`Off`/`0` → false).
|
||||
pub fn get_bool(&self, key: &str) -> Option<bool> {
|
||||
match self.get_raw(key)? {
|
||||
"Yes" | "YES" | "On" | "ON" | "1" => Some(true),
|
||||
"No" | "NO" | "Off" | "OFF" | "0" => Some(false),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every field that carries an explicit, value-shaped value, as `(key, value)`
|
||||
/// in on-disc order. Uses the value-before-key rule but only emits a pair when
|
||||
/// the key is identifier-shaped *and* the preceding token is unambiguously a
|
||||
/// value — so every returned pair is real, and defaulted fields are simply
|
||||
/// absent. Identifier-valued fields (e.g. `Model = rou_f001`) are excluded;
|
||||
/// read those with [`get_raw`](Self::get_raw). Ideal for a full-object dump.
|
||||
pub fn resolved_fields(&self) -> Vec<(&str, &str)> {
|
||||
let mut out = Vec::new();
|
||||
for i in 1..self.tokens.len() {
|
||||
let key = self.tokens[i].as_str();
|
||||
let val = self.tokens[i - 1].as_str();
|
||||
if is_key_like(key) && is_definite_value(val) {
|
||||
out.push((key, val));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
/// Extract the trailing string pool. Finds the smallest offset whose suffix is
|
||||
/// ≥98% printable-ASCII-or-NUL (the pool runs to end-of-buffer), then tokenises
|
||||
/// into maximal printable runs (NUL *or* any non-printable byte separates tokens,
|
||||
/// so a slightly-early boundary can't glue binary onto the first field).
|
||||
fn extract_string_pool(bytes: &[u8]) -> Vec<String> {
|
||||
let start = pool_start(bytes);
|
||||
let printable = |b: u8| (0x20..0x7f).contains(&b);
|
||||
let mut tokens = Vec::new();
|
||||
let mut i = start;
|
||||
let n = bytes.len();
|
||||
while i < n {
|
||||
if !printable(bytes[i]) {
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
let mut j = i;
|
||||
while j < n && printable(bytes[j]) {
|
||||
j += 1;
|
||||
}
|
||||
// Safe: bytes[i..j] verified all ASCII printable.
|
||||
tokens.push(String::from_utf8_lossy(&bytes[i..j]).into_owned());
|
||||
i = j;
|
||||
}
|
||||
tokens
|
||||
}
|
||||
|
||||
/// The string pool is the maximal trailing run of bytes that are all
|
||||
/// printable-ASCII-or-NUL. Since NUL-terminated ASCII strings never contain a
|
||||
/// byte outside `[0x20,0x7e] ∪ {0}`, the boundary is exactly one past the last
|
||||
/// non-printable byte — precise for both tiny and multi-KB node regions.
|
||||
///
|
||||
/// (Caveat: an IDXD value stored in raw Shift-JIS would contain high bytes and
|
||||
/// truncate the pool here; the gameplay-stat tables are ASCII, so this is safe
|
||||
/// for them. Localized text lives in separate `IXUD` entries, not IDXD.)
|
||||
fn pool_start(bytes: &[u8]) -> usize {
|
||||
let printable_or_nul = |b: u8| (0x20..0x7f).contains(&b) || b == 0;
|
||||
match bytes.iter().rposition(|&b| !printable_or_nul(b)) {
|
||||
Some(last_bin) => last_bin + 1,
|
||||
None => 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// A token that is unambiguously a *value* (not a field-name key): a number, a
|
||||
/// scalar enum literal, or a string containing a non-identifier char (comma,
|
||||
/// space, dot, slash, …).
|
||||
fn is_definite_value(s: &str) -> bool {
|
||||
if parse_number(s.strip_suffix(['f', 'F']).unwrap_or(s)) {
|
||||
return true;
|
||||
}
|
||||
if matches!(s, "Yes" | "No" | "YES" | "NO" | "On" | "Off" | "ON" | "OFF") {
|
||||
return true;
|
||||
}
|
||||
// A delimited string (comma/space/…) is a value, but only if it *starts* with
|
||||
// an alphanumeric — e.g. `Vessel,Craft`. Tokens beginning with punctuation
|
||||
// (`|Generic`, `{`, `.`) are node-table noise, not values.
|
||||
s.bytes().next().is_some_and(|b| b.is_ascii_alphanumeric())
|
||||
&& s.bytes().any(|b| !(b.is_ascii_alphanumeric() || b == b'_'))
|
||||
}
|
||||
|
||||
/// Whether `s` is plausibly a field-name key: an identifier that is not itself a
|
||||
/// value literal.
|
||||
fn is_key_like(s: &str) -> bool {
|
||||
let mut cs = s.chars();
|
||||
// Field names are identifiers of ≥3 chars (Size_X, Acceleration, …) or a
|
||||
// 2-char all-caps abbreviation (HP, ID, AV, AA). This rejects the short,
|
||||
// mixed-case junk tokens (`cV`, `nx`, `xb`) that leak from the node-table tail.
|
||||
matches!(cs.next(), Some(c) if c.is_ascii_alphabetic())
|
||||
&& s.chars().all(|c| c.is_ascii_alphanumeric() || c == '_')
|
||||
&& (s.len() >= 3 || s.chars().all(|c| c.is_ascii_uppercase()))
|
||||
&& !is_definite_value(s)
|
||||
}
|
||||
|
||||
/// Whether `s` is a plain decimal number (optionally signed, optionally fractional).
|
||||
fn parse_number(s: &str) -> bool {
|
||||
!s.is_empty()
|
||||
&& s.bytes().any(|b| b.is_ascii_digit())
|
||||
&& s.bytes().enumerate().all(|(i, b)| {
|
||||
b.is_ascii_digit() || b == b'.' || (i == 0 && (b == b'-' || b == b'+'))
|
||||
})
|
||||
}
|
||||
|
||||
#[inline]
|
||||
fn be32(b: &[u8], o: usize) -> u32 {
|
||||
u32::from_be_bytes([b[o], b[o + 1], b[o + 2], b[o + 3]])
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Build a synthetic IDXD: header + 32 bytes of binary filler + pool.
|
||||
/// `pool` is a flat token list already in on-disc (value-before-key) order.
|
||||
fn synth(pool: &[&str]) -> Vec<u8> {
|
||||
let mut b = Vec::new();
|
||||
b.extend_from_slice(&IDXD_MAGIC);
|
||||
b.extend_from_slice(&1u32.to_be_bytes());
|
||||
b.extend_from_slice(&0xDEAD_BEEFu32.to_be_bytes());
|
||||
b.extend_from_slice(&[0xFFu8; 32]); // binary node region (non-printable → skipped)
|
||||
for t in pool {
|
||||
b.extend_from_slice(t.as_bytes());
|
||||
b.push(0);
|
||||
}
|
||||
b
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_explicit_scalars() {
|
||||
// "1000.0" HP "500000.0" RadarRange "Vessel,Craft" TargetType "Yes" Mounted
|
||||
let bytes = synth(&[
|
||||
"1000.0",
|
||||
"HP",
|
||||
"500000.0",
|
||||
"RadarRange",
|
||||
"Vessel,Craft",
|
||||
"TargetType",
|
||||
"Yes",
|
||||
"Mounted",
|
||||
]);
|
||||
let o = IdxdObject::parse(&bytes).unwrap();
|
||||
assert_eq!(o.schema_hash, 0xDEAD_BEEF);
|
||||
assert_eq!(o.get_f32("HP"), Some(1000.0));
|
||||
assert_eq!(o.get_f32("RadarRange"), Some(500000.0));
|
||||
assert_eq!(o.get_i64("RadarRange"), None); // "500000.0" is not an integer
|
||||
assert_eq!(o.get_str("TargetType"), Some("Vessel,Craft"));
|
||||
assert_eq!(o.get_bool("Mounted"), Some(true));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn defaulted_field_yields_none_not_neighbour() {
|
||||
// RadarRange has value 500000; FCSRange is defaulted (no value) → its
|
||||
// preceding token is the *key* "RadarRange", which must NOT be returned.
|
||||
let bytes = synth(&["500000.0", "RadarRange", "FCSRange", "Yes", "Mounted"]);
|
||||
let o = IdxdObject::parse(&bytes).unwrap();
|
||||
assert_eq!(o.get_f32("RadarRange"), Some(500000.0));
|
||||
assert_eq!(o.get_f32("FCSRange"), None);
|
||||
assert_eq!(o.get_str("FCSRange"), None); // not the neighbour key
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn get_raw_exposes_identifier_values() {
|
||||
let bytes = synth(&["rou_f001", "Model", "Craft", "Type"]);
|
||||
let o = IdxdObject::parse(&bytes).unwrap();
|
||||
assert_eq!(o.get_raw("Model"), Some("rou_f001"));
|
||||
assert_eq!(o.get_raw("Type"), Some("Craft"));
|
||||
// get_str is conservative: bare identifiers are not "definite values".
|
||||
assert_eq!(o.get_str("Model"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resolved_fields_lists_explicit_values_only() {
|
||||
let bytes = synth(&[
|
||||
"rou_f001", "Model", // identifier value → excluded
|
||||
"10.0", "Size_X", // explicit → included
|
||||
"FCSRange", // defaulted (no value) → excluded
|
||||
"Yes", "Mounted", // enum → included
|
||||
]);
|
||||
let o = IdxdObject::parse(&bytes).unwrap();
|
||||
let fields = o.resolved_fields();
|
||||
assert!(fields.contains(&("Size_X", "10.0")));
|
||||
assert!(fields.contains(&("Mounted", "Yes")));
|
||||
assert!(!fields.iter().any(|(k, _)| *k == "Model"));
|
||||
assert!(!fields.iter().any(|(k, _)| *k == "FCSRange"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_bad_magic() {
|
||||
assert!(matches!(
|
||||
IdxdObject::parse(b"NOPE\0\0\0\0\0\0\0\0"),
|
||||
Err(IdxdError::BadMagic { .. })
|
||||
));
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user