From 5f03bc2f4a7ed5b95ef5f129747f31d227c60b4b Mon Sep 17 00:00:00 2001 From: Sylpheed RE agent Date: Tue, 25 Aug 2026 11:08:10 +0000 Subject: [PATCH] re: tag_hash verified disc-wide; record keys follow a stronger own-name rule Swept all 33 dat paks plus hidden/DefTables.pak: 7,750 IDXD objects, 190,782 records, 1,271,462 (name -> tag) pairs, 0 failures, 0 parse failures. The key rule is stronger than first written: a record's key is tag_hash of its own inline name, 190,782/190,782 -- no roster needed (roster-based recovery only reaches ~30% of records). Verified independently here on UnitGroup_S02, 112/112. The modulus is uniquely pinned, not just consistent: gcd of poly256(name) - (tag & 0xFFFFFF) over all pairs is exactly 0x00FFFFDF, prime. Case-sensitivity is load-bearing, not incidental: the disc has 17 name pairs differing only in case (UNIT/Unit, TYPE/Type, STAGE/Stage, ...); name_hash collides on all 17, tag_hash separates all 17. Records the limit too: IXUD, the wide-string sibling, uses a DIFFERENT low-24 hash (0/19,808) though it keeps the same top-byte checksum (19,808/19,808). Its offsets are counted in 16-bit chars (STR + 2*strsize == filesize) -- newly decoded, all 534 parse. Its low bits are provably not a modular polynomial (gcd = 1; exhaustive Barrett search over M in [2^20,2^25) finds nothing; fixed-position deltas are non-constant), so it is lossy and needs the XEX. --- docs/re/structures/idxd-tag-hash.md | 76 +++++++++++++++++++++++++++-- 1 file changed, 73 insertions(+), 3 deletions(-) diff --git a/docs/re/structures/idxd-tag-hash.md b/docs/re/structures/idxd-tag-hash.md index 9df7abd1..9aa544d1 100644 --- a/docs/re/structures/idxd-tag-hash.md +++ b/docs/re/structures/idxd-tag-hash.md @@ -1,8 +1,9 @@ # The IDXD record-key / field-tag hash -Status: βœ… recovered and verified 8643/8643 across `GP_MAIN_GAME_E.pak`; -🟑 the guest routine has not been located, so the implementation is exact -modular arithmetic rather than a transcribed op sequence. +Status: βœ… recovered and verified **1,271,462 / 1,271,462 disc-wide**; +βœ… record keys follow a stronger rule than first thought (no roster needed); +🟑 the guest routine has not been located; πŸ”΄ it does **not** cover the sibling +`IXUD` container. This closes the **4-byte record key** that had been ❔ since the squadron roster was decoded. @@ -72,6 +73,75 @@ So a table's records can now be addressed **by name** without reading its roster first. The roster is still the honest way to *enumerate* names β€” hashes do not invert β€” but resolving a known name no longer needs it. +## βœ… Verified across the whole disc, and the rule is stronger + +Sweeping all 33 `dat/*.pak` **plus** `hidden/DefTables.pak` (1425 more IDXD +objects β€” easy to miss): **7,750 IDXD objects, 190,782 records, 1,271,462 +(name β†’ tag) pairs, 0 failures.** Zero parse failures too: the +`STR + strsize == filesize` identity held on every object, and no object had a +duplicate record key. + +**A record's key is `tag_hash` of its own inline name** β€” the string at the +record's second `u32` β€” for **190,782 / 190,782** records. The roster is not +needed at all for key recovery; it only helps *enumerate* names. (Checked here +independently on `UnitGroup_S02.tbl`: 112/112.) + +Roster-based recovery, by contrast, only reaches about **30 %** of records +disc-wide, so prefer the own-name rule. + +### βœ… The modulus is uniquely pinned, not merely consistent + +Because the accumulator always stays below `M < 2²⁴`, the hash is an exact +Horner polynomial mod `M`, so any valid modulus must divide +`poly256(name) βˆ’ (tag & 0xFFFFFF)`. The **gcd of that quantity over all observed +pairs is exactly `0x00FFFFDF`**, which is prime. No other modulus fits. + +If a Barrett form matching `name_hash`'s shape is ever wanted: +`RECIP == floor(2^55 / M) + 1` holds for the known pair +(`0x80031493 == floor(2^55/0x00FFF9D7)+1`), which gives `RECIP = 0x80001081` +for `M = 0x00FFFFDF`. + +### βœ… Independent evidence that case-sensitivity is real, not an artefact + +The disc carries **17 name pairs differing only in case** β€” `UNIT`/`Unit`, +`TYPE`/`Type`, `STAGE`/`Stage`, `TIME`/`Time`, `MENU`/`Menu`, `VOLUME`/`Volume`, +`BackGround`/`Background`, `FONT_MAIN`/`Font_Main` and nine more. `name_hash` +**collides on all 17** (it lowercases); `tag_hash` separates all 17. The game +*needs* a case-sensitive hash here, which is why there are two. + +There are **0 tag collisions** among the 13,451 distinct strings. Honest caveat: +the birthday expectation for 13,451 items in a 32-bit space is **0.021** +collisions, so zero is exactly what chance predicts β€” it is not evidence the +scheme was *designed* collision-free. + +⚠️ Positional (unnamed) field entries carry a small **ordinal** in the tag slot +(0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse +lookup. + +## πŸ”΄ Where it breaks: the `IXUD` container + +`IXUD` is IDXD's wide-string sibling β€” 534 entries, same header/record/property/ +string-pool layout β€” and **`tag_hash` holds for 0 of its 19,808 names.** + +Two things were learned rather than assumed: + +* βœ… **Its offsets are counted in 16-bit characters, not bytes.** The trailer + identity is `STR + 2Β·strsize == filesize`, and with that all 534 parse cleanly + (906 records, 624,440 named pairs). Strings are UTF-16BE. +* βœ… **The top byte is still the same checksum** β€” `(sum of the ASCII bytes) & 0xFF` + matches **19,808 / 19,808**. Only the low 24 bits differ. + +πŸ”΄ And its low 24 bits are **provably not a modular polynomial of any modulus**: +the gcd test that pins IDXD's modulus returns **1** here, for ASCII, UTF-16BE and +lowercased input alike; an exhaustive Barrett search over every `M` in +`[2²⁰, 2²⁡)` finds nothing. The reason is visible directly β€” single-character +deltas at a fixed position are constant within one name family +(`ExtraStage01/02/03Desc` all differ by `0x01005B71`) but **not** another +(`Extra01/02/03/04Requirement` alternate `0x0123B663` / `0x012395FC`). No pure +mod-`M` polynomial can do that. It is a **lossy** hash β€” the same family as +`name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the +accumulator exceeds 2²⁴. Recovering it will need the executable, not the data. + ## 🟑 What is not settled * **The guest routine has not been located.** `name_hash`'s low 24 bits come