re: partial field map inside the live definition object, and why the automation failed

Anchoring on a unit whose disc record sets a field locates it in the live object:
Size_X/Y/Z (+0x30/34/38) and HP (+0x54) confirmed across five capital ships and
the player fighter; +0x74 (0.8) and +0x84 (0.97) are probable ThrusterRatio and
ResistanceParalyze but rest on a single anchoring unit; +0x40 varies per unit and
is unidentified.

live_offsets.rs automates the correlation and currently produces one hit which is
false -- YawDragFactor 2.0 collided with the integer 2 at +0x0c stored as a
denormal. Causes recorded: get_f32 is unreliable on default-heavy IDXD records
because the value-before-key pairing shifts, and 96 words only reaches +0x180
while RadarRange/FCSRange sit beyond it. Next run dumps 512 words.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

crates/sylpheed-formats/examples/live_offsets.rs

@@ -0,0 +1,81 @@
+//! Where does each declared field sit inside the engine's live definition object?
+//!
+//! A live object carries no names, so the mapping has to be inferred: take a unit
+//! whose disc record SETS a field, look for that value in the dumped words of the
+//! matching live object, and keep the offsets that agree across several units.
+//!
+//! Run: live_offsets <disc-root> <live-dump.txt> <ID=0xVA> ...
+//!   e.g. … UN_f106_TCAF_Destroyer=0xbd3ee300 UN_f105_TCAF_Cruiser=0xbd40e200
+use std::collections::BTreeMap;
+use sylpheed_formats::idxd::IdxdObject;
+use sylpheed_formats::pak::PakArchive;
+
+fn main() {
+    let mut a = std::env::args().skip(1);
+    let disc = a.next().expect("disc root");
+    let dump = a.next().expect("live dump");
+    let pairs: Vec<(String, String)> = a
+        .filter_map(|s| s.split_once('=').map(|(i, v)| (i.to_string(), v.to_string())))
+        .collect();
+
+    // live: va -> offset -> f32
+    let mut live: BTreeMap<String, BTreeMap<usize, f32>> = BTreeMap::new();
+    let mut cur = String::new();
+    for line in std::fs::read_to_string(&dump).expect("dump").lines() {
+        if let Some(rest) = line.strip_prefix("=== ") {
+            cur = rest.trim().to_string();
+            continue;
+        }
+        let f: Vec<&str> = line.split_whitespace().collect();
+        if f.len() >= 4 && f[1].starts_with('+') {
+            if let (Ok(off), Ok(val)) =
+                (usize::from_str_radix(&f[1][1..], 16), f[3].parse::<f32>())
+            {
+                live.entry(cur.clone()).or_default().insert(off, val);
+            }
+        }
+    }
+
+    let pak = PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).expect("pak");
+    // key -> offset -> how many units agree
+    let mut votes: BTreeMap<String, BTreeMap<usize, usize>> = BTreeMap::new();
+    let mut units = 0usize;
+    for entry in pak.entries() {
+        let Ok(bytes) = pak.read(entry) else { continue };
+        let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
+        let Some(id) = obj.get_raw("ID") else { continue };
+        let Some((_, va)) = pairs.iter().find(|(i, _)| i == id) else { continue };
+        let Some(words) = live.get(va) else { continue };
+        units += 1;
+        for key in obj.tokens() {
+            let Some(v) = obj.get_f32(key) else { continue };
+            if !v.is_finite() || v == 0.0 {
+                continue; // zero matches everywhere and says nothing
+            }
+            for (off, w) in words {
+                if (*w - v).abs() <= v.abs() * 1e-6 {
+                    *votes.entry(key.clone()).or_default().entry(*off).or_default() += 1;
+                }
+            }
+        }
+    }
+    println!("{units} units correlated against their live objects\n");
+    println!("{:<28} {:>8}  offsets agreeing (votes)", "field", "unique?");
+    for (key, offs) in &votes {
+        let best = offs.iter().max_by_key(|(_, n)| **n).unwrap();
+        if *best.1 < units.max(2) {
+            continue; // needs every correlated unit to agree
+        }
+        let list: Vec<String> = offs
+            .iter()
+            .filter(|(_, n)| **n == *best.1)
+            .map(|(o, n)| format!("+{o:03x}×{n}"))
+            .collect();
+        println!(
+            "{:<28} {:>8}  {}",
+            key,
+            if list.len() == 1 { "unique" } else { "ambig" },
+            list.join(" ")
+        );
+    }
+}

crates/sylpheed-formats/examples/unit_fields.rs

@@ -0,0 +1,31 @@
+//! Every key/value a named unit record actually sets on disc.
+//!
+//! The live definition object in guest RAM has no field names; to find where a
+//! field like `HQRatio` sits inside it, anchor on a unit whose disc record DOES
+//! set that field and look for the value. This prints those anchors.
+//! Run: unit_fields <disc-root> <substring of ID>...
+use sylpheed_formats::idxd::IdxdObject;
+use sylpheed_formats::pak::PakArchive;
+
+fn main() {
+    let disc = std::env::args().nth(1).expect("disc root");
+    let want: Vec<String> = std::env::args().skip(2).collect();
+    let pak = PakArchive::open(format!("{disc}/dat/GP_MAIN_GAME_E.pak")).expect("pak");
+    for entry in pak.entries() {
+        let Ok(bytes) = pak.read(entry) else { continue };
+        let Ok(obj) = IdxdObject::parse(&bytes) else { continue };
+        let Some(id) = obj.get_raw("ID") else { continue };
+        if !want.iter().any(|w| id.contains(w.as_str())) {
+            continue;
+        }
+        println!("=== {id}   (schema {:08x})", obj.schema_hash);
+        // The pool interleaves VALUE before KEY (see the module docs), so the
+        // pairs read (t[i] = value, t[i+1] = key).
+        let t = obj.tokens();
+        let mut i = 0;
+        while i + 1 < t.len() {
+            println!("    {:<30} {}", t[i + 1], t[i]);
+            i += 2;
+        }
+    }
+}

docs/re/live-unit-definitions.md

@@ -114,3 +114,42 @@ not addressed. And this run only reaches the units Stage 02 instantiates: the
 **not** resolved here, because those fields' offsets in the live object are not
 yet known. Finding them is the next run: dump deeper than 96 words and look for
 the constants the disc *does* set on the records that set them.
+
+## Locating fields inside the live object
+
+The next step — reading the `…Ratio` / `…Count` family — needs to know *where*
+each field sits in the live object. The method is to anchor on a unit whose disc
+record **sets** a field and look for that value in its dumped words. Anchoring on
+`UN_f106_TCAF_Destroyer` (disc sets `Size_Radius 0.1`, `ThrusterRatio 0.8`,
+`ResistanceParalyze 0.97`) and then checking the same offsets across five capital
+ships and the player's fighter:
+
+| offset | reading | `f106` | `f105` | `e105` | `e106` | `f101` | Delta Saber |
+|---|---|---|---|---|---|---|---|
+| `+0x30` | `Size_X` ✅ | 200 | 700 | 600 | 300 | 400 | 10 |
+| `+0x34` | `Size_Y` ✅ | 200 | 700 | 600 | 300 | 400 | 7 |
+| `+0x38` | `Size_Z` ✅ | 2000 | 3800 | 3800 | 2100 | 1400 | 29 |
+| `+0x54` | `HP` ✅ | 10000 | 30000 | 30000 | 10000 | 25000 | 1500 |
+| `+0x74` | 🟡 `ThrusterRatio` | 0.8 | 0.8 | 0.8 | 0.8 | 0.8 | 1 |
+| `+0x84` | 🟡 `ResistanceParalyze` | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 | 0.97 |
+| `+0x40` | ❔ | 0.1 | 0.1 | 1 | 1 | 1 | 0.1 |
+
+`+0x74` and `+0x84` are marked 🟡 because they rest on one anchoring unit: the
+value is right for the Destroyer and constant across the others, which is
+consistent with a field the rest default, but a second anchoring unit that sets
+them to something *different* is what would prove it. `+0x40` is 0.1 on two units
+and 1 on three, so it is a real per-unit field — just not identified.
+
+⚠️ **The automated version of this does not work yet.** `examples/live_offsets.rs`
+correlates every `get_f32(key)` against every dumped word and keeps offsets all
+units agree on; over five units it produced exactly **one** hit
+(`YawDragFactor → +0x0c`), and that hit is **wrong** — `+0x0c` holds the integer
+2 (as a denormal float, `2.8e-45`), which collided with `YawDragFactor = 2.0`.
+Two causes, both fixable: the IDXD pool's value-before-key interleaving makes
+`get_f32` unreliable on records that default many fields (the pairing shifts), so
+most keys yield nothing; and 96 words only reaches `+0x180`, while `RadarRange
+60000` / `FCSRange 45000` — both set on the Destroyer — appear nowhere in that
+window, so much of the record lies beyond it.
+
+So the next run dumps **512 words** per object, and the disc side needs a reader
+that walks the pool with the default-aware rule rather than `get_f32` per key.