re: tag_hash verified disc-wide; record keys follow a stronger own-name rule

Swept all 33 dat paks plus hidden/DefTables.pak: 7,750 IDXD objects, 190,782
records, 1,271,462 (name -> tag) pairs, 0 failures, 0 parse failures.

The key rule is stronger than first written: a record's key is tag_hash of its
own inline name, 190,782/190,782 -- no roster needed (roster-based recovery only
reaches ~30% of records). Verified independently here on UnitGroup_S02, 112/112.

The modulus is uniquely pinned, not just consistent: gcd of
poly256(name) - (tag & 0xFFFFFF) over all pairs is exactly 0x00FFFFDF, prime.

Case-sensitivity is load-bearing, not incidental: the disc has 17 name pairs
differing only in case (UNIT/Unit, TYPE/Type, STAGE/Stage, ...); name_hash
collides on all 17, tag_hash separates all 17.

Records the limit too: IXUD, the wide-string sibling, uses a DIFFERENT low-24
hash (0/19,808) though it keeps the same top-byte checksum (19,808/19,808). Its
offsets are counted in 16-bit chars (STR + 2*strsize == filesize) -- newly
decoded, all 534 parse. Its low bits are provably not a modular polynomial (gcd
= 1; exhaustive Barrett search over M in [2^20,2^25) finds nothing; fixed-position
deltas are non-constant), so it is lossy and needs the XEX.
This commit is contained in:
Sylpheed RE agent
2026-08-25 11:08:10 +00:00
parent 1772d94558
commit 55bfcc77d3

View File

@@ -1,8 +1,9 @@
# The IDXD record-key / field-tag hash
Status: ✅ recovered and verified 8643/8643 across `GP_MAIN_GAME_E.pak`;
🟡 the guest routine has not been located, so the implementation is exact
modular arithmetic rather than a transcribed op sequence.
Status: ✅ recovered and verified **1,271,462 / 1,271,462 disc-wide**;
✅ record keys follow a stronger rule than first thought (no roster needed);
🟡 the guest routine has not been located; 🔴 it does **not** cover the sibling
`IXUD` container.
This closes the **4-byte record key** that had been ❔ since the squadron roster
was decoded.
@@ -72,6 +73,75 @@ So a table's records can now be addressed **by name** without reading its roster
first. The roster is still the honest way to *enumerate* names — hashes do not
invert — but resolving a known name no longer needs it.
## ✅ Verified across the whole disc, and the rule is stronger
Sweeping all 33 `dat/*.pak` **plus** `hidden/DefTables.pak` (1425 more IDXD
objects — easy to miss): **7,750 IDXD objects, 190,782 records, 1,271,462
(name → tag) pairs, 0 failures.** Zero parse failures too: the
`STR + strsize == filesize` identity held on every object, and no object had a
duplicate record key.
**A record's key is `tag_hash` of its own inline name** — the string at the
record's second `u32` — for **190,782 / 190,782** records. The roster is not
needed at all for key recovery; it only helps *enumerate* names. (Checked here
independently on `UnitGroup_S02.tbl`: 112/112.)
Roster-based recovery, by contrast, only reaches about **30 %** of records
disc-wide, so prefer the own-name rule.
### ✅ The modulus is uniquely pinned, not merely consistent
Because the accumulator always stays below `M < 2²⁴`, the hash is an exact
Horner polynomial mod `M`, so any valid modulus must divide
`poly256(name) (tag & 0xFFFFFF)`. The **gcd of that quantity over all observed
pairs is exactly `0x00FFFFDF`**, which is prime. No other modulus fits.
If a Barrett form matching `name_hash`'s shape is ever wanted:
`RECIP == floor(2^55 / M) + 1` holds for the known pair
(`0x80031493 == floor(2^55/0x00FFF9D7)+1`), which gives `RECIP = 0x80001081`
for `M = 0x00FFFFDF`.
### ✅ Independent evidence that case-sensitivity is real, not an artefact
The disc carries **17 name pairs differing only in case**`UNIT`/`Unit`,
`TYPE`/`Type`, `STAGE`/`Stage`, `TIME`/`Time`, `MENU`/`Menu`, `VOLUME`/`Volume`,
`BackGround`/`Background`, `FONT_MAIN`/`Font_Main` and nine more. `name_hash`
**collides on all 17** (it lowercases); `tag_hash` separates all 17. The game
*needs* a case-sensitive hash here, which is why there are two.
There are **0 tag collisions** among the 13,451 distinct strings. Honest caveat:
the birthday expectation for 13,451 items in a 32-bit space is **0.021**
collisions, so zero is exactly what chance predicts — it is not evidence the
scheme was *designed* collision-free.
⚠️ Positional (unnamed) field entries carry a small **ordinal** in the tag slot
(0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse
lookup.
## 🔴 Where it breaks: the `IXUD` container
`IXUD` is IDXD's wide-string sibling — 534 entries, same header/record/property/
string-pool layout — and **`tag_hash` holds for 0 of its 19,808 names.**
Two things were learned rather than assumed:
***Its offsets are counted in 16-bit characters, not bytes.** The trailer
identity is `STR + 2·strsize == filesize`, and with that all 534 parse cleanly
(906 records, 624,440 named pairs). Strings are UTF-16BE.
***The top byte is still the same checksum** — `(sum of the ASCII bytes) & 0xFF`
matches **19,808 / 19,808**. Only the low 24 bits differ.
🔴 And its low 24 bits are **provably not a modular polynomial of any modulus**:
the gcd test that pins IDXD's modulus returns **1** here, for ASCII, UTF-16BE and
lowercased input alike; an exhaustive Barrett search over every `M` in
`[2²⁰, 2²⁵)` finds nothing. The reason is visible directly — single-character
deltas at a fixed position are constant within one name family
(`ExtraStage01/02/03Desc` all differ by `0x01005B71`) but **not** another
(`Extra01/02/03/04Requirement` alternate `0x0123B663` / `0x012395FC`). No pure
mod-`M` polynomial can do that. It is a **lossy** hash — the same family as
`name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the
accumulator exceeds 2²⁴. Recovering it will need the executable, not the data.
## 🟡 What is not settled
* **The guest routine has not been located.** `name_hash`'s low 24 bits come