re: tag_hash verified disc-wide; record keys follow a stronger own-name rule
Swept all 33 dat paks plus hidden/DefTables.pak: 7,750 IDXD objects, 190,782 records, 1,271,462 (name -> tag) pairs, 0 failures, 0 parse failures. The key rule is stronger than first written: a record's key is tag_hash of its own inline name, 190,782/190,782 -- no roster needed (roster-based recovery only reaches ~30% of records). Verified independently here on UnitGroup_S02, 112/112. The modulus is uniquely pinned, not just consistent: gcd of poly256(name) - (tag & 0xFFFFFF) over all pairs is exactly 0x00FFFFDF, prime. Case-sensitivity is load-bearing, not incidental: the disc has 17 name pairs differing only in case (UNIT/Unit, TYPE/Type, STAGE/Stage, ...); name_hash collides on all 17, tag_hash separates all 17. Records the limit too: IXUD, the wide-string sibling, uses a DIFFERENT low-24 hash (0/19,808) though it keeps the same top-byte checksum (19,808/19,808). Its offsets are counted in 16-bit chars (STR + 2*strsize == filesize) -- newly decoded, all 534 parse. Its low bits are provably not a modular polynomial (gcd = 1; exhaustive Barrett search over M in [2^20,2^25) finds nothing; fixed-position deltas are non-constant), so it is lossy and needs the XEX.
This commit is contained in:
@@ -1,8 +1,9 @@
|
||||
# The IDXD record-key / field-tag hash
|
||||
|
||||
Status: ✅ recovered and verified 8643/8643 across `GP_MAIN_GAME_E.pak`;
|
||||
🟡 the guest routine has not been located, so the implementation is exact
|
||||
modular arithmetic rather than a transcribed op sequence.
|
||||
Status: ✅ recovered and verified **1,271,462 / 1,271,462 disc-wide**;
|
||||
✅ record keys follow a stronger rule than first thought (no roster needed);
|
||||
🟡 the guest routine has not been located; 🔴 it does **not** cover the sibling
|
||||
`IXUD` container.
|
||||
|
||||
This closes the **4-byte record key** that had been ❔ since the squadron roster
|
||||
was decoded.
|
||||
@@ -72,6 +73,75 @@ So a table's records can now be addressed **by name** without reading its roster
|
||||
first. The roster is still the honest way to *enumerate* names — hashes do not
|
||||
invert — but resolving a known name no longer needs it.
|
||||
|
||||
## ✅ Verified across the whole disc, and the rule is stronger
|
||||
|
||||
Sweeping all 33 `dat/*.pak` **plus** `hidden/DefTables.pak` (1425 more IDXD
|
||||
objects — easy to miss): **7,750 IDXD objects, 190,782 records, 1,271,462
|
||||
(name → tag) pairs, 0 failures.** Zero parse failures too: the
|
||||
`STR + strsize == filesize` identity held on every object, and no object had a
|
||||
duplicate record key.
|
||||
|
||||
**A record's key is `tag_hash` of its own inline name** — the string at the
|
||||
record's second `u32` — for **190,782 / 190,782** records. The roster is not
|
||||
needed at all for key recovery; it only helps *enumerate* names. (Checked here
|
||||
independently on `UnitGroup_S02.tbl`: 112/112.)
|
||||
|
||||
Roster-based recovery, by contrast, only reaches about **30 %** of records
|
||||
disc-wide, so prefer the own-name rule.
|
||||
|
||||
### ✅ The modulus is uniquely pinned, not merely consistent
|
||||
|
||||
Because the accumulator always stays below `M < 2²⁴`, the hash is an exact
|
||||
Horner polynomial mod `M`, so any valid modulus must divide
|
||||
`poly256(name) − (tag & 0xFFFFFF)`. The **gcd of that quantity over all observed
|
||||
pairs is exactly `0x00FFFFDF`**, which is prime. No other modulus fits.
|
||||
|
||||
If a Barrett form matching `name_hash`'s shape is ever wanted:
|
||||
`RECIP == floor(2^55 / M) + 1` holds for the known pair
|
||||
(`0x80031493 == floor(2^55/0x00FFF9D7)+1`), which gives `RECIP = 0x80001081`
|
||||
for `M = 0x00FFFFDF`.
|
||||
|
||||
### ✅ Independent evidence that case-sensitivity is real, not an artefact
|
||||
|
||||
The disc carries **17 name pairs differing only in case** — `UNIT`/`Unit`,
|
||||
`TYPE`/`Type`, `STAGE`/`Stage`, `TIME`/`Time`, `MENU`/`Menu`, `VOLUME`/`Volume`,
|
||||
`BackGround`/`Background`, `FONT_MAIN`/`Font_Main` and nine more. `name_hash`
|
||||
**collides on all 17** (it lowercases); `tag_hash` separates all 17. The game
|
||||
*needs* a case-sensitive hash here, which is why there are two.
|
||||
|
||||
There are **0 tag collisions** among the 13,451 distinct strings. Honest caveat:
|
||||
the birthday expectation for 13,451 items in a 32-bit space is **0.021**
|
||||
collisions, so zero is exactly what chance predicts — it is not evidence the
|
||||
scheme was *designed* collision-free.
|
||||
|
||||
⚠️ Positional (unnamed) field entries carry a small **ordinal** in the tag slot
|
||||
(0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse
|
||||
lookup.
|
||||
|
||||
## 🔴 Where it breaks: the `IXUD` container
|
||||
|
||||
`IXUD` is IDXD's wide-string sibling — 534 entries, same header/record/property/
|
||||
string-pool layout — and **`tag_hash` holds for 0 of its 19,808 names.**
|
||||
|
||||
Two things were learned rather than assumed:
|
||||
|
||||
* ✅ **Its offsets are counted in 16-bit characters, not bytes.** The trailer
|
||||
identity is `STR + 2·strsize == filesize`, and with that all 534 parse cleanly
|
||||
(906 records, 624,440 named pairs). Strings are UTF-16BE.
|
||||
* ✅ **The top byte is still the same checksum** — `(sum of the ASCII bytes) & 0xFF`
|
||||
matches **19,808 / 19,808**. Only the low 24 bits differ.
|
||||
|
||||
🔴 And its low 24 bits are **provably not a modular polynomial of any modulus**:
|
||||
the gcd test that pins IDXD's modulus returns **1** here, for ASCII, UTF-16BE and
|
||||
lowercased input alike; an exhaustive Barrett search over every `M` in
|
||||
`[2²⁰, 2²⁵)` finds nothing. The reason is visible directly — single-character
|
||||
deltas at a fixed position are constant within one name family
|
||||
(`ExtraStage01/02/03Desc` all differ by `0x01005B71`) but **not** another
|
||||
(`Extra01/02/03/04Requirement` alternate `0x0123B663` / `0x012395FC`). No pure
|
||||
mod-`M` polynomial can do that. It is a **lossy** hash — the same family as
|
||||
`name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the
|
||||
accumulator exceeds 2²⁴. Recovering it will need the executable, not the data.
|
||||
|
||||
## 🟡 What is not settled
|
||||
|
||||
* **The guest routine has not been located.** `name_hash`'s low 24 bits come
|
||||
|
||||
Reference in New Issue
Block a user