From 72a80d0d2fdffb6140f75afa492622d5dbf6eb8a Mon Sep 17 00:00:00 2001 From: Sylpheed RE agent Date: Tue, 25 Aug 2026 11:29:09 +0000 Subject: [PATCH] docs: IXUD solved, extsb correction, name_hash exactness retraction --- docs/re/structures/idxd-tag-hash.md | 111 +++++++++++++++++++++++----- 1 file changed, 91 insertions(+), 20 deletions(-) diff --git a/docs/re/structures/idxd-tag-hash.md b/docs/re/structures/idxd-tag-hash.md index 9aa544d1..11779cee 100644 --- a/docs/re/structures/idxd-tag-hash.md +++ b/docs/re/structures/idxd-tag-hash.md @@ -1,25 +1,46 @@ # The IDXD record-key / field-tag hash -Status: ✅ recovered and verified **1,271,462 / 1,271,462 disc-wide**; -✅ record keys follow a stronger rule than first thought (no roster needed); -🟡 the guest routine has not been located; 🔴 it does **not** cover the sibling -`IXUD` container. +Status: ✅ recovered, **guest routine located and transcribed**, verified +**1,271,462 / 1,271,462 disc-wide**; ✅ record keys follow a stronger rule (no +roster needed); ✅ **`IXUD` solved too** — a different, two-modulus hash. This closes the **4-byte record key** that had been ❔ since the squadron roster was decoded. ## ✅ The function -```python -TAG_MODULUS = (1 << 24) - 33 # 0x00FFFFDF, prime +**`sub_82447DF0`**, transcribed instruction for instruction: -def tag_hash(s): - b = s.encode(); lo = 0 - for c in b: - lo = (lo * 256 + c) % TAG_MODULUS - return ((sum(b) & 0xFF) << 24) | lo +```python +TAG_MODULUS = 0x00FFFFDF # 2^24 - 33, prime +TAG_MAGIC = 0x2101 # floor(2^56/M)+1, the guest's divide magic + +A = 0; B = 0 +for each byte of the name (NOT lowercased): + c = extsb(byte) # SIGN-EXTENDED -- see below + A = ((A << 8) + c) # 32-bit + B = B + c + hi = mulhwu(A, TAG_MAGIC) + q = (hi + ((A - hi) >>u 1)) >>u 23 # exact magic division, add-correction form + A = A - q * TAG_MODULUS +tag = ((B & 0xFF) << 24) | A ``` +The division is **exact**, so `A == A % 0x00FFFFDF`. + +### ⚠️ The bytes are SIGN-EXTENDED, and the disc could never have shown it + +The guest uses `extsb`. The version of this function recovered from the data +used **unsigned** bytes — and it matched **all 1,271,462** disc names, because +every one of them is ASCII. On random inputs containing a byte ≥ 0x80 the two +disagree **18,096 times in 20,000**; e.g. `4e3fcfa50c864c2b41cf` hashes to +`0x1AFF849A` in the guest and `0x1A65C73F` unsigned. + +This is the concrete answer to the "there may be inputs where the two disagree" +caveat this note used to carry — and a reminder that a corpus can validate a +function to six decimal places and still leave a whole branch untested. It took +the disassembly, not more data. + `tools/re-capture/unitgroup.py` (Python) and `sylpheed_formats::hash::tag_hash` (Rust) both implement it. @@ -118,7 +139,44 @@ scheme was *designed* collision-free. (0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse lookup. -## 🔴 Where it breaks: the `IXUD` container +## ✅ `IXUD` — solved: it chains TWO moduli + +**`sub_82447E70`** is the wide sibling — `lhz`, 16-bit code units, 64-bit +accumulator: + +```python +M1 = 0xFFFFFF67 # 2^32 - 153, the loop modulus (64-bit arithmetic) +M2 = 0x00FFFFDF # 2^24 - 33, the same final modulus as IDXD + +A = 0; B = 0 +for each big-endian UTF-16 code unit ch: + A = ((A << 16) + ch) % M1 + B = B + ch # the FULL code unit, not its low byte +tag = ((B & 0xFF) << 24) | (A % M2) +``` + +Verified independently against real data: **86/86 record keys** and +**108,261/108,261 field tags** in `GP_MAIN_GAME_E.pak`, `NoRecord` → +`0x1C6D9C96`. Disc-wide the sweep is **906/906 keys, 624,440/624,440 tags**. + +**Why every search for it failed** — and the earlier diagnosis in this note was +wrong. It is **not lossy**. It is a polynomial mod `M1` *then folded* through +`M2`, and that composition is not a polynomial modulo anything, so the gcd test +returns 1 and a Barrett sweep over single moduli cannot find it however wide. +The tell was there in the data: fixed-position character deltas are constant +within one name family but not another — which I read as "lossy" when it +actually means "a second reduction happened after the loop". + +⚠️ The checksum sums the **full 16-bit code units**. Every IXUD name on this +disc is ASCII, so this is indistinguishable here; for Japanese text it is not. + +### Its offsets are counted in 16-bit characters + +Same header/record/property/string-pool layout as IDXD, but `strsize` and every +string offset are in **characters, not bytes**: `STR + 2·strsize == filesize`. +All 534 objects parse under that rule. + +## 🔴 What this note used to say about `IXUD` `IXUD` is IDXD's wide-string sibling — 534 entries, same header/record/property/ string-pool layout — and **`tag_hash` holds for 0 of its 19,808 names.** @@ -142,13 +200,26 @@ mod-`M` polynomial can do that. It is a **lossy** hash — the same family as `name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the accumulator exceeds 2²⁴. Recovering it will need the executable, not the data. +## ✅ Correction: `name_hash`'s reduction is EXACT, not lossy + +This note (and `hash.rs`) claimed the missing trailing conditional subtract made +`sub_82455C78` something other than a true `%`. **That is wrong.** +`rlwinm r6,r6,9,23,31` is exactly `hi >> 23`, and with +`RECIP == floor(2^55/M) + 1` that is textbook Granlund–Montgomery magic +division. Checked at every quotient boundary (`k·M−1, k·M, k·M+1`) across the +whole 32-bit domain: **0 wrong of 770**. So `name_hash`'s low 24 bits really are +`A % 0x00FFF9D7`. + +Also corrected: `sub_82455C78` does **not** lowercase internally — callers pass +an already-normalised path. Where that normalisation happens is still ❔. + ## 🟡 What is not settled -* **The guest routine has not been located.** `name_hash`'s low 24 bits come - from a Barrett step with *no* final conditional subtract, which is not the - same function as `%` on every input. `tag_hash` is written here with exact - modular arithmetic because it matches all 8643 known pairs — but if the game - computes it the same Barrett way, there may be inputs where the two disagree. - Nothing outside those 8643 has been checked. Finding the routine in - `default.xex` would settle it. -* Whether the same hash keys IDXD tables in **other paks** is untested here. +* **Where IPFB path lowercasing happens.** `sub_82455C78` is case-sensitive and + no caller folds inline; it is somewhere upstream in the path library. +* **Non-ASCII behaviour is unexercised by the disc.** Both the `extsb` path + (IDXD) and the code-unit checksum (IXUD) are transcribed from the + disassembly, but no disc name reaches them. +* **Sibling magics exist and are unexamined:** the container dispatchers + reference `IDX2`, `IDX3`, `IDXC` and `IIDX` alongside `IDXD`/`IXUD`. Whether + any appear on disc is unchecked.