re: audit the legacy IDXD reader against the real field table — and fix a test that encoded its error

With the record table decoded there is finally a ground truth to check the
old string-pool reader against. It infers `key -> value` from pool adjacency,
which is a consequence of how records are written, not a rule of the format.

Verified by hand against the disc, with an independent parser:

* `FCSRange` = 500000.0 — the module docs' own canonical example of a field
  "left at its default" that "omits the value string".
* `ShieldRatio` = 1.0, where `tests/pak_idxd_disc.rs` asserted None with the
  comment "a defaulted/omitted field must be None". That test encoded the
  false belief; it now keeps the None as a deliberate characterisation of the
  legacy reader, with the true value asserted beside it.
* `get_raw("Model")` on GP_HANGAR_ARSENAL returns the first record's model for
  every record — silent corruption, not an absent value. New test pins four
  records that disagree with it.

The cause is the flat API having no way to name a record: only 548 of 6325
objects have one. `HP` on the DeltaSaber answers 1000.0, the hull, while 63
Turret_* records each carry their own 100.0 (measured — a first draft said 34,
taken from a report rather than from the disc).

Disc-wide rates are recorded as single-source and labelled as such: get_raw
52% wrong, typed getters 38% miss, but 100% correct on single-record objects.

Also records a negative result: the 504 unnamed field keys were NOT recovered.
A 572464-string dictionary and 73191 variants gave 0/42. The key deltas do
prove the preimage ends with the two decimal digits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE

crates/sylpheed-formats/tests/pak_idxd_disc.rs

@@ -80,8 +80,54 @@ fn deltasaber_craft_stats() {
     assert_eq!(obj.get_i64("Turn_AngularVelocity"), Some(100));
     assert_eq!(obj.get_f32("ChargeSpeed"), Some(800.0));
     assert_eq!(obj.get_f32("RadarRange"), Some(500000.0));
-    // a defaulted/omitted field must be None, not a neighbouring key
+    // ❌ NOT a defaulted field — this assertion used to claim it was. The record
+    // table says ShieldRatio = 1.0; the legacy string-pool reader simply cannot
+    // see it. Kept as a *characterisation* of that reader, with the truth beside
+    // it, so the blind spot stays visible instead of being read as "no value".
     assert_eq!(obj.get_f32("ShieldRatio"), None);
+    let generic = obj.record("Generic").expect("Generic record");
+    assert_eq!(generic.get("ShieldRatio"), Some("1.0"));
+    // Same for the module docs' canonical "field with no value".
+    assert_eq!(generic.get("FCSRange"), Some("500000.0"));
+
+    // The values above are real, but each lives in a *different* record, which
+    // the flat API cannot express: these three are Generic / Maneuver / Shield.
+    assert_eq!(generic.get("HP"), Some("1000.0"));
+    assert_eq!(obj.record("Maneuver").unwrap().get("Acceleration"), Some("600.0"));
+    assert_eq!(obj.record("Shield").unwrap().get("ChargeSpeed"), Some("800.0"));
+
+    // And `HP` is not one number: 63 turret records carry their own (all 100.0).
+    let turrets = obj
+        .records()
+        .unwrap()
+        .iter()
+        .filter(|r| r.name.starts_with("Turret_"))
+        .count();
+    assert_eq!(turrets, 63);
+    assert_eq!(obj.record("Turret_000").unwrap().get("HP"), Some("100.0"));
+}
+
+/// The legacy reader returns one object-wide answer for a per-record field, and
+/// it is silently *wrong* rather than absent. The hangar's weapon → model map is
+/// the clearest case: every record reads back as the first one.
+#[test]
+fn legacy_reader_flattens_per_record_fields() {
+    skip_without_disc!(root);
+    let arc = PakArchive::open(root.join("dat/GP_HANGAR_ARSENAL.pak")).unwrap();
+    let bytes = arc.read_by_hash(0x8f72_ddde).expect("entry present").unwrap();
+    let obj = IdxdObject::parse(&bytes).unwrap();
+
+    // One answer for the whole object …
+    assert_eq!(obj.get_raw("Model"), Some("rou_f001_wep_33_hangar"));
+    // … while the records disagree with it and with each other.
+    for (record, model) in [
+        ("Designator_LH", "rou_f001_wep_59_hangar"),
+        ("Laser_Mine_B9L", "rou_f001_wep_29_hangar"),
+        ("Saber_LG1", "rou_f001_wep_14_hangar"),
+        ("Chaff_Flare_Dispencer", "rou_f001_wep_71_hangar"),
+    ] {
+        assert_eq!(obj.record(record).unwrap().get("Model"), Some(model));
+    }
 }
 
 #[test]

docs/re/BACKLOG.md

@@ -1033,6 +1033,27 @@ premise was wrong.**
   Candidates: the **7 `.embsec_` sections** (VAs 0x84D0000–0x86AC000, ~129 KB
   total, executable) or a hashed record in `hidden/MiscBin.pak`. **Finding it
   gives the actual per-phase clear condition for every stage.**
+* 🟡 **(2026-08-25) The legacy IDXD string-pool reader is wrong far more often
+  than assumed** — now measurable for the first time, since the record table gives
+  a ground truth. Verified by hand: `FCSRange` (the module doc's own canonical
+  "field with no value") is really `500000.0`; `ShieldRatio` is `1.0` where a
+  **committed test asserted `None` and called it defaulted**; and
+  `get_raw("Model")` on the hangar table returns the *first* record's model for
+  every record — silent corruption, not an absent value. Single-source disc-wide
+  rates: `get_raw` 52 % wrong, typed getters 38 % miss — but **100 % correct on the
+  548 single-record objects**, so all the damage is the flat API having no way to
+  say *which* record it means. ▶️ **Open work:** re-read every per-record number in
+  this corpus through `IdxdObject::record`; highest value first — hangar models,
+  weapon `Power`/`Acceleration`/`MinimumVelocity`, turret and subsystem stats.
+  See [`idxd-legacy-reader-audit.md`](idxd-legacy-reader-audit.md).
+* ❌ **(2026-08-25) The 504 unnamed IDXD field keys were NOT recovered.** All sit
+  in `GP_READY_ROOM.pak`'s sound-bank table (6 identical objects × 2 records × 42
+  keys). A dictionary of 572 464 strings — every pool string disc-wide, PE ASCII
+  and UTF-16 runs, every identifier in this repo — plus 73 191 case/affix variants
+  gave **0/42**. The key deltas across `stage01…stage16` do prove the preimage
+  **ends with the two decimal digits**, and a meet-in-the-middle found nothing
+  word-like at ≤8 lowercase chars. 32 bits is not invertible without the right
+  wordlist; parked.
 * ✅ **(2026-08-25) The IDXD/IXUD container is fully decoded** — the "binary
   node/index region" in front of the string pool is a **uniform 16-byte record
   array** `{name_hash, name_off, field_begin, field_end}` sorted by hash, then a

docs/re/INDEX.md

@@ -91,6 +91,7 @@ files, which is how the same ground got covered twice.
 | [`ship-placement-capture-generalisation.md`](ship-placement-capture-generalisation.md) | Capital-ship placement — does the `e106` result generalise? (WIP, 2026-07-31) |  🚧 WIP, time-boxed session. Two results so far: a static audit across all |
 | [`ship-placement-runtime-capture.md`](ship-placement-runtime-capture.md) | Capital-ship part placement — runtime capture (ground truth) |  ✅✅ STATIC ASSEMBLY IS EXACT — no captures needed anymore |
 | [`structures/achievements.md`](structures/achievements.md) | Achievements — the 24-entry table, and where the earned state comes from | — |
+| [`idxd-legacy-reader-audit.md`](idxd-legacy-reader-audit.md) | The legacy IDXD string-pool reader vs the real field table — what the old numbers got wrong | 🟡 shape CONFIRMED by hand (`FCSRange`, `ShieldRatio`, hangar `Model`); disc-wide rates are single-source |
 | [`structures/idxd-container.md`](structures/idxd-container.md) | The IDXD/IXUD container — record/field table, and the two beliefs it withdraws | ✅ CONFIRMED disc-wide, 7 750/7 750 objects and 1 271 462/1 271 462 named fields, zero failures |
 | [`structures/hud-glyph-quad.md`](structures/hud-glyph-quad.md) | The HUD's glyph quad — vtable `0x820B2A64` | ✅ CONFIRMED for the object layout and the atlas size, read live off |
 | [`structures/mission-objective-counter.md`](structures/mission-objective-counter.md) | `REMAINING OB` — the mission's own objective counter, in RAM |  ✅ CONFIRMED for one Stage 02 run: a big-endian u32 whose value |

docs/re/idxd-legacy-reader-audit.md

@@ -0,0 +1,137 @@
+# The legacy IDXD string-pool reader is wrong far more often than assumed
+
+Status: 🟡 **the shape of the problem is CONFIRMED and reproduced by hand; the
+disc-wide percentages are a single-source measurement and are quoted as such.**
+
+Once the [record/field table](structures/idxd-container.md) was decoded there was,
+for the first time, a ground truth to check the old reader against. This note
+records what that comparison found. It is not a bug report against
+`crates/sylpheed-formats/src/idxd.rs` so much as against **the numbers already in
+this corpus that came out of it**.
+
+## What the old reader does
+
+It ignores the binary region and tokenises only the trailing string pool, then
+infers `key -> value` from **adjacency**: a field's value is the token immediately
+before its name (`<value>\0<key>\0`). `get_f32`/`get_i64`/`get_str` additionally
+require the preceding token to be *value-shaped* and return `None` otherwise;
+`get_raw` does not check, and the corpus uses `get_raw` for `ID`, `Name`, `Model`
+and `Type`.
+
+## ✅ The premise is false, and here is why
+
+Adjacency is a **consequence** of how the writer emits a record's fields, not a
+rule of the format. Two things break it:
+
+1. **A field's value can be absent from the pool as a distinct token** — the pool
+   deduplicates repeated strings, so a value shared with another field is stored
+   once and only one of them is adjacent to it.
+2. **An object has many records.** The flat API has no way to say *which* record's
+   `HP` is wanted, so it returns whichever name token it finds first.
+
+The second is the fatal one, and it is the common case: only **548 of 6325**
+`dat/` objects have a single record.
+
+### ✅ Verified by hand, against the disc
+
+Three claims I re-derived myself with an independent parser, because they are the
+load-bearing ones:
+
+| object | field | old reader | **truth (record table)** |
+|---|---|---|---|
+| `GP_MAIN_GAME_E.pak` `0x7c96296c` (DeltaSaber `rou_f001`), record `Generic` | `FCSRange` | `None` | **`500000.0`** |
+| same | `ShieldRatio` | `None` | **`1.0`** |
+| `GP_HANGAR_ARSENAL.pak` `0x8f72ddde` | `Model` | `rou_f001_wep_33_hangar` **for every record** | per record: `Designator_LH -> …_59_`, `Laser_Mine_B9L -> …_29_`, `Saber_LG1 -> …_14_`, `Chaff_Flare_Dispencer -> …_71_` |
+
+The first two matter beyond themselves:
+
+* **`FCSRange` is the module documentation's own canonical example** of a field
+  "left at its default" that "omits the value string". It has a value.
+* **`ShieldRatio` was asserted as `None` by a committed test**
+  (`tests/pak_idxd_disc.rs`) with the comment *"a defaulted/omitted field must be
+  None"*. That test encoded the false belief. It is now kept as a deliberate
+  **characterisation** of the legacy reader, with the true value asserted beside it
+  so the blind spot cannot be misread as "no value" again.
+
+The third is worse than a missing value: it is **silent corruption**. The hangar's
+entire weapon → model mapping reads back as the first record's model.
+
+`HP` in that same craft object is a good illustration of the flattening: the old
+reader answers `1000.0`, which is the hull — while **63 `Turret_*` records** each
+carry their own `HP = 100.0`, unreachable through the flat API. (I measured 63; a
+first draft of this note said 34, taken from a report rather than from the disc.)
+
+## 🟡 Disc-wide rates — one measurement, not yet independently reproduced
+
+Measured over all 6325 `dat/` objects by a Python port of the reader that
+reproduces all 21 assertions of the existing disc tests. I have **not** re-run
+these totals with a second implementation, so treat the percentages as indicative
+and the direction as certain.
+
+Of 1,244,919 named fields, the name follows its value adjacently in only
+**424,241 (34.1 %)**; 810,951 (65.1 %) are not adjacent at all.
+
+| API | lookups | agree | miss | **wrong** |
+|---|---|---|---|---|
+| `get_raw` (used for `ID`/`Name`/`Model`/`Type`) | 303,044 | 47.6 % | 0.2 % | **52.2 %** |
+| typed getters, truth is value-shaped | 186,443 | 61.6 % | 38.1 % | 0.3 % |
+| `resolved_fields()` emitted pairs | 131,493 | 87.2 % | — | 12.8 % |
+
+Split by record count, which is the whole story:
+
+| objects | typed agree | typed miss | typed wrong | `get_raw` wrong |
+|---|---|---|---|---|
+| 1 record (548) | **100.0 %** | 1 field | **0** | 31.0 % |
+| 2–9 records (2298) | 57.7 % | 42.1 % | 0.2 % | 46.8 % |
+| ≥10 records (3479) | 51.1 % | 48.4 % | 0.5 % | 61.6 % |
+
+So the typed getters are **exactly right on single-record objects** and degrade
+from there. `get_raw`'s 31 % on even those is one specific failure: for the 80,550
+fields whose true value is the **empty string**, it returns the neighbouring key
+100 % of the time.
+
+Per-field accuracy for fields this repo actually reads (agreement rate):
+`ScorePoint` 100 %, `BackGroundID` 100 %, `Name` 99.7 %, `Size_X` 99.1 %, `ID`
+99.1 %, `TargetType` 96.2 %, `Type` 82.5 %, `Model` 77.9 %, `RadarRange` 61.1 %,
+`HP` 52.8 %, `FCSRange` 47.8 %, `Acceleration` 25.6 %, `MinimumVelocity` 20.7 %,
+`ThrusterCount` 7.0 %, `ShieldGeneratorCount` 6.1 %.
+
+Two more findings from the same pass, ❔ unverified by me:
+
+* The old reader's `pool_start` heuristic is **early for 2340 objects (37 %)**,
+  swallowing tail bytes of the record array — which is where `resolved_fields()`'s
+  16,152 phantom keys come from — and **late for 6**: `GP_MAIN_GAME_*.pak` entry
+  `fbe15595` has non-printable bytes *inside* its real pool, hiding 189 of its 195
+  fields.
+* The `…Count` family that `idxd.rs` documented as having no on-disc value
+  ("their values live in the binary node/index region") **does** have values, in a
+  record named `StructureCount`. The old doc was right that the region held them
+  and wrong that they were unreachable.
+
+## ❌ The 504 unnamed hash keys were NOT recovered
+
+All 504 are in `dat/GP_READY_ROOM.pak`, in 6 byte-identical objects with records
+`FILE` and `OFFSET` sharing the same 42 keys (42 × 2 × 6 = 504). `FILE` values are
+sound banks — `stage01.isb … stage16.isb`, `challenge01-06.isb`, `tutorial*.isb`,
+`main.isb`, `function.isb`.
+
+Search was exhaustive and negative: the 10,462 named field names disc-wide, then
+572,464 strings (every pool string, PE ASCII and UTF-16 runs, every identifier in
+this repo), then 73,191 case/affix variants — **0 of 42** recovered.
+
+One structural fact did come out of it, and it is real: the key deltas across
+`stage01 … stage16` move by exactly `+1` per unit digit and `+256−9` per tens
+rollover, which means **the preimage ends with the two decimal digits and nothing
+follows them**. A meet-in-the-middle over the implied prefix state found no
+candidate ≤6 chars, and 1008 non-word-like collisions at ≤8 lowercase chars. A
+32-bit hash is not invertible without the right wordlist. ❔ Open.
+
+## What to do with this
+
+* **Do not** rip out the legacy reader. It is correct where it is used most
+  carefully, and 33 call sites plus a large body of recorded numbers depend on it.
+* **Do** treat any single number in `docs/re/` that came from `get_f32`/`get_raw`
+  on a multi-record object as unverified until re-read through
+  [`IdxdObject::record`](structures/idxd-container.md).
+* The highest-value re-check is anything per-record: hangar models, weapon
+  `Power`/`Acceleration`/`MinimumVelocity`, turret and subsystem stats.