Syplheed-Reborn/docs/re/idxd-legacy-reader-audit.md

The legacy IDXD string-pool reader is wrong far more often than assumed

Status: 🟡 the shape of the problem is CONFIRMED and reproduced by hand; the disc-wide percentages are a single-source measurement and are quoted as such.

Once the record/field table was decoded there was, for the first time, a ground truth to check the old reader against. This note records what that comparison found. It is not a bug report against crates/sylpheed-formats/src/idxd.rs so much as against the numbers already in this corpus that came out of it.

What the old reader does

It ignores the binary region and tokenises only the trailing string pool, then infers key -> value from adjacency: a field's value is the token immediately before its name (<value>\0<key>\0). get_f32/get_i64/get_str additionally require the preceding token to be value-shaped and return None otherwise; get_raw does not check, and the corpus uses get_raw for ID, Name, Model and Type.

✅ The premise is false, and here is why

Adjacency is a consequence of how the writer emits a record's fields, not a rule of the format. Two things break it:

1. A field's value can be absent from the pool as a distinct token — the pool deduplicates repeated strings, so a value shared with another field is stored once and only one of them is adjacent to it.
2. An object has many records. The flat API has no way to say which record's HP is wanted, so it returns whichever name token it finds first.

The second is the fatal one, and it is the common case: only 548 of 6325 dat/ objects have a single record.

✅ Verified by hand, against the disc

Three claims I re-derived myself with an independent parser, because they are the load-bearing ones:

object	field	old reader	truth (record table)
GP_MAIN_GAME_E.pak 0x7c96296c (DeltaSaber rou_f001), record Generic	FCSRange	None	500000.0
same	ShieldRatio	None	1.0
GP_HANGAR_ARSENAL.pak 0x8f72ddde	Model	rou_f001_wep_33_hangar for every record	per record: Designator_LH -> …_59_, Laser_Mine_B9L -> …_29_, Saber_LG1 -> …_14_, Chaff_Flare_Dispencer -> …_71_

The first two matter beyond themselves:

• FCSRange is the module documentation's own canonical example of a field "left at its default" that "omits the value string". It has a value.
• ShieldRatio was asserted as None by a committed test (tests/pak_idxd_disc.rs) with the comment "a defaulted/omitted field must be None". That test encoded the false belief. It is now kept as a deliberate characterisation of the legacy reader, with the true value asserted beside it so the blind spot cannot be misread as "no value" again.

The third is worse than a missing value: it is silent corruption. The hangar's entire weapon → model mapping reads back as the first record's model.

HP in that same craft object is a good illustration of the flattening: the old reader answers 1000.0, which is the hull — while 63 Turret_* records each carry their own HP = 100.0, unreachable through the flat API. (I measured 63; a first draft of this note said 34, taken from a report rather than from the disc.)

🟡 Disc-wide rates — one measurement, not yet independently reproduced

Measured over all 6325 dat/ objects by a Python port of the reader that reproduces all 21 assertions of the existing disc tests. I have not re-run these totals with a second implementation, so treat the percentages as indicative and the direction as certain.

Of 1,244,919 named fields, the name follows its value adjacently in only 424,241 (34.1 %); 810,951 (65.1 %) are not adjacent at all.

API	lookups	agree	miss	wrong
get_raw (used for ID/Name/Model/Type)	303,044	47.6 %	0.2 %	52.2 %
typed getters, truth is value-shaped	186,443	61.6 %	38.1 %	0.3 %
resolved_fields() emitted pairs	131,493	87.2 %	—	12.8 %

Split by record count, which is the whole story:

objects	typed agree	typed miss	typed wrong	get_raw wrong
1 record (548)	100.0 %	1 field	0	31.0 %
2–9 records (2298)	57.7 %	42.1 %	0.2 %	46.8 %
≥10 records (3479)	51.1 %	48.4 %	0.5 %	61.6 %

So the typed getters are exactly right on single-record objects and degrade from there. get_raw's 31 % on even those is one specific failure: for the 80,550 fields whose true value is the empty string, it returns the neighbouring key 100 % of the time.

Per-field accuracy for fields this repo actually reads (agreement rate): ScorePoint 100 %, BackGroundID 100 %, Name 99.7 %, Size_X 99.1 %, ID 99.1 %, TargetType 96.2 %, Type 82.5 %, Model 77.9 %, RadarRange 61.1 %, HP 52.8 %, FCSRange 47.8 %, Acceleration 25.6 %, MinimumVelocity 20.7 %, ThrusterCount 7.0 %, ShieldGeneratorCount 6.1 %.

Two more findings from the same pass, ❔ unverified by me:

• The old reader's pool_start heuristic is early for 2340 objects (37 %), swallowing tail bytes of the record array — which is where resolved_fields()'s 16,152 phantom keys come from — and late for 6: GP_MAIN_GAME_*.pak entry fbe15595 has non-printable bytes inside its real pool, hiding 189 of its 195 fields.
• The …Count family that idxd.rs documented as having no on-disc value ("their values live in the binary node/index region") does have values, in a record named StructureCount. The old doc was right that the region held them and wrong that they were unreachable.

❌ The 504 unnamed hash keys were NOT recovered

All 504 are in dat/GP_READY_ROOM.pak, in 6 byte-identical objects with records FILE and OFFSET sharing the same 42 keys (42 × 2 × 6 = 504). FILE values are sound banks — stage01.isb … stage16.isb, challenge01-06.isb, tutorial*.isb, main.isb, function.isb.

Search was exhaustive and negative: the 10,462 named field names disc-wide, then 572,464 strings (every pool string, PE ASCII and UTF-16 runs, every identifier in this repo), then 73,191 case/affix variants — 0 of 42 recovered.

One structural fact did come out of it, and it is real: the key deltas across stage01 … stage16 move by exactly +1 per unit digit and +256−9 per tens rollover, which means the preimage ends with the two decimal digits and nothing follows them. A meet-in-the-middle over the implied prefix state found no candidate ≤6 chars, and 1008 non-word-like collisions at ≤8 lowercase chars. A 32-bit hash is not invertible without the right wordlist. ❔ Open.

What to do with this

• Do not rip out the legacy reader. It is correct where it is used most carefully, and 33 call sites plus a large body of recorded numbers depend on it.
• Do treat any single number in docs/re/ that came from get_f32/get_raw on a multi-record object as unverified until re-read through IdxdObject::record.
• The highest-value re-check is anything per-record: hangar models, weapon Power/Acceleration/MinimumVelocity, turret and subsystem stats.