723fc9b890
With the record table decoded there is finally a ground truth to check the
old string-pool reader against. It infers `key -> value` from pool adjacency,
which is a consequence of how records are written, not a rule of the format.
Verified by hand against the disc, with an independent parser:
* `FCSRange` = 500000.0 — the module docs' own canonical example of a field
"left at its default" that "omits the value string".
* `ShieldRatio` = 1.0, where `tests/pak_idxd_disc.rs` asserted None with the
comment "a defaulted/omitted field must be None". That test encoded the
false belief; it now keeps the None as a deliberate characterisation of the
legacy reader, with the true value asserted beside it.
* `get_raw("Model")` on GP_HANGAR_ARSENAL returns the first record's model for
every record — silent corruption, not an absent value. New test pins four
records that disagree with it.
The cause is the flat API having no way to name a record: only 548 of 6325
objects have one. `HP` on the DeltaSaber answers 1000.0, the hull, while 63
Turret_* records each carry their own 100.0 (measured — a first draft said 34,
taken from a report rather than from the disc).
Disc-wide rates are recorded as single-source and labelled as such: get_raw
52% wrong, typed getters 38% miss, but 100% correct on single-record objects.
Also records a negative result: the 504 unnamed field keys were NOT recovered.
A 572464-string dictionary and 73191 variants gave 0/42. The key deltas do
prove the preimage ends with the two decimal digits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
138 lines
7.2 KiB
Markdown
138 lines
7.2 KiB
Markdown
# The legacy IDXD string-pool reader is wrong far more often than assumed
|
||
|
||
Status: 🟡 **the shape of the problem is CONFIRMED and reproduced by hand; the
|
||
disc-wide percentages are a single-source measurement and are quoted as such.**
|
||
|
||
Once the [record/field table](structures/idxd-container.md) was decoded there was,
|
||
for the first time, a ground truth to check the old reader against. This note
|
||
records what that comparison found. It is not a bug report against
|
||
`crates/sylpheed-formats/src/idxd.rs` so much as against **the numbers already in
|
||
this corpus that came out of it**.
|
||
|
||
## What the old reader does
|
||
|
||
It ignores the binary region and tokenises only the trailing string pool, then
|
||
infers `key -> value` from **adjacency**: a field's value is the token immediately
|
||
before its name (`<value>\0<key>\0`). `get_f32`/`get_i64`/`get_str` additionally
|
||
require the preceding token to be *value-shaped* and return `None` otherwise;
|
||
`get_raw` does not check, and the corpus uses `get_raw` for `ID`, `Name`, `Model`
|
||
and `Type`.
|
||
|
||
## ✅ The premise is false, and here is why
|
||
|
||
Adjacency is a **consequence** of how the writer emits a record's fields, not a
|
||
rule of the format. Two things break it:
|
||
|
||
1. **A field's value can be absent from the pool as a distinct token** — the pool
|
||
deduplicates repeated strings, so a value shared with another field is stored
|
||
once and only one of them is adjacent to it.
|
||
2. **An object has many records.** The flat API has no way to say *which* record's
|
||
`HP` is wanted, so it returns whichever name token it finds first.
|
||
|
||
The second is the fatal one, and it is the common case: only **548 of 6325**
|
||
`dat/` objects have a single record.
|
||
|
||
### ✅ Verified by hand, against the disc
|
||
|
||
Three claims I re-derived myself with an independent parser, because they are the
|
||
load-bearing ones:
|
||
|
||
| object | field | old reader | **truth (record table)** |
|
||
|---|---|---|---|
|
||
| `GP_MAIN_GAME_E.pak` `0x7c96296c` (DeltaSaber `rou_f001`), record `Generic` | `FCSRange` | `None` | **`500000.0`** |
|
||
| same | `ShieldRatio` | `None` | **`1.0`** |
|
||
| `GP_HANGAR_ARSENAL.pak` `0x8f72ddde` | `Model` | `rou_f001_wep_33_hangar` **for every record** | per record: `Designator_LH -> …_59_`, `Laser_Mine_B9L -> …_29_`, `Saber_LG1 -> …_14_`, `Chaff_Flare_Dispencer -> …_71_` |
|
||
|
||
The first two matter beyond themselves:
|
||
|
||
* **`FCSRange` is the module documentation's own canonical example** of a field
|
||
"left at its default" that "omits the value string". It has a value.
|
||
* **`ShieldRatio` was asserted as `None` by a committed test**
|
||
(`tests/pak_idxd_disc.rs`) with the comment *"a defaulted/omitted field must be
|
||
None"*. That test encoded the false belief. It is now kept as a deliberate
|
||
**characterisation** of the legacy reader, with the true value asserted beside it
|
||
so the blind spot cannot be misread as "no value" again.
|
||
|
||
The third is worse than a missing value: it is **silent corruption**. The hangar's
|
||
entire weapon → model mapping reads back as the first record's model.
|
||
|
||
`HP` in that same craft object is a good illustration of the flattening: the old
|
||
reader answers `1000.0`, which is the hull — while **63 `Turret_*` records** each
|
||
carry their own `HP = 100.0`, unreachable through the flat API. (I measured 63; a
|
||
first draft of this note said 34, taken from a report rather than from the disc.)
|
||
|
||
## 🟡 Disc-wide rates — one measurement, not yet independently reproduced
|
||
|
||
Measured over all 6325 `dat/` objects by a Python port of the reader that
|
||
reproduces all 21 assertions of the existing disc tests. I have **not** re-run
|
||
these totals with a second implementation, so treat the percentages as indicative
|
||
and the direction as certain.
|
||
|
||
Of 1,244,919 named fields, the name follows its value adjacently in only
|
||
**424,241 (34.1 %)**; 810,951 (65.1 %) are not adjacent at all.
|
||
|
||
| API | lookups | agree | miss | **wrong** |
|
||
|---|---|---|---|---|
|
||
| `get_raw` (used for `ID`/`Name`/`Model`/`Type`) | 303,044 | 47.6 % | 0.2 % | **52.2 %** |
|
||
| typed getters, truth is value-shaped | 186,443 | 61.6 % | 38.1 % | 0.3 % |
|
||
| `resolved_fields()` emitted pairs | 131,493 | 87.2 % | — | 12.8 % |
|
||
|
||
Split by record count, which is the whole story:
|
||
|
||
| objects | typed agree | typed miss | typed wrong | `get_raw` wrong |
|
||
|---|---|---|---|---|
|
||
| 1 record (548) | **100.0 %** | 1 field | **0** | 31.0 % |
|
||
| 2–9 records (2298) | 57.7 % | 42.1 % | 0.2 % | 46.8 % |
|
||
| ≥10 records (3479) | 51.1 % | 48.4 % | 0.5 % | 61.6 % |
|
||
|
||
So the typed getters are **exactly right on single-record objects** and degrade
|
||
from there. `get_raw`'s 31 % on even those is one specific failure: for the 80,550
|
||
fields whose true value is the **empty string**, it returns the neighbouring key
|
||
100 % of the time.
|
||
|
||
Per-field accuracy for fields this repo actually reads (agreement rate):
|
||
`ScorePoint` 100 %, `BackGroundID` 100 %, `Name` 99.7 %, `Size_X` 99.1 %, `ID`
|
||
99.1 %, `TargetType` 96.2 %, `Type` 82.5 %, `Model` 77.9 %, `RadarRange` 61.1 %,
|
||
`HP` 52.8 %, `FCSRange` 47.8 %, `Acceleration` 25.6 %, `MinimumVelocity` 20.7 %,
|
||
`ThrusterCount` 7.0 %, `ShieldGeneratorCount` 6.1 %.
|
||
|
||
Two more findings from the same pass, ❔ unverified by me:
|
||
|
||
* The old reader's `pool_start` heuristic is **early for 2340 objects (37 %)**,
|
||
swallowing tail bytes of the record array — which is where `resolved_fields()`'s
|
||
16,152 phantom keys come from — and **late for 6**: `GP_MAIN_GAME_*.pak` entry
|
||
`fbe15595` has non-printable bytes *inside* its real pool, hiding 189 of its 195
|
||
fields.
|
||
* The `…Count` family that `idxd.rs` documented as having no on-disc value
|
||
("their values live in the binary node/index region") **does** have values, in a
|
||
record named `StructureCount`. The old doc was right that the region held them
|
||
and wrong that they were unreachable.
|
||
|
||
## ❌ The 504 unnamed hash keys were NOT recovered
|
||
|
||
All 504 are in `dat/GP_READY_ROOM.pak`, in 6 byte-identical objects with records
|
||
`FILE` and `OFFSET` sharing the same 42 keys (42 × 2 × 6 = 504). `FILE` values are
|
||
sound banks — `stage01.isb … stage16.isb`, `challenge01-06.isb`, `tutorial*.isb`,
|
||
`main.isb`, `function.isb`.
|
||
|
||
Search was exhaustive and negative: the 10,462 named field names disc-wide, then
|
||
572,464 strings (every pool string, PE ASCII and UTF-16 runs, every identifier in
|
||
this repo), then 73,191 case/affix variants — **0 of 42** recovered.
|
||
|
||
One structural fact did come out of it, and it is real: the key deltas across
|
||
`stage01 … stage16` move by exactly `+1` per unit digit and `+256−9` per tens
|
||
rollover, which means **the preimage ends with the two decimal digits and nothing
|
||
follows them**. A meet-in-the-middle over the implied prefix state found no
|
||
candidate ≤6 chars, and 1008 non-word-like collisions at ≤8 lowercase chars. A
|
||
32-bit hash is not invertible without the right wordlist. ❔ Open.
|
||
|
||
## What to do with this
|
||
|
||
* **Do not** rip out the legacy reader. It is correct where it is used most
|
||
carefully, and 33 call sites plus a large body of recorded numbers depend on it.
|
||
* **Do** treat any single number in `docs/re/` that came from `get_f32`/`get_raw`
|
||
on a multi-record object as unverified until re-read through
|
||
[`IdxdObject::record`](structures/idxd-container.md).
|
||
* The highest-value re-check is anything per-record: hangar models, weapon
|
||
`Power`/`Acceleration`/`MinimumVelocity`, turret and subsystem stats.
|