docs: IXUD solved, extsb correction, name_hash exactness retraction

docs/re/structures/idxd-tag-hash.md

@@ -1,25 +1,46 @@
 # The IDXD record-key / field-tag hash
 
-Status: ✅ recovered and verified **1,271,462 / 1,271,462 disc-wide**;
+Status: ✅ recovered, **guest routine located and transcribed**, verified
-✅ record keys follow a stronger rule than first thought (no roster needed);
+**1,271,462 / 1,271,462 disc-wide**; ✅ record keys follow a stronger rule (no
-🟡 the guest routine has not been located; 🔴 it does **not** cover the sibling
+roster needed); ✅ **`IXUD` solved too** — a different, two-modulus hash.
-`IXUD` container.
 
 This closes the **4-byte record key** that had been ❔ since the squadron roster
 was decoded.
 
 ## ✅ The function
 
-```python
+**`sub_82447DF0`**, transcribed instruction for instruction:
-TAG_MODULUS = (1 << 24) - 33          # 0x00FFFFDF, prime
 
-def tag_hash(s):
+```python
-    b = s.encode(); lo = 0
+TAG_MODULUS = 0x00FFFFDF   # 2^24 - 33, prime
-    for c in b:
+TAG_MAGIC   = 0x2101       # floor(2^56/M)+1, the guest's divide magic
-        lo = (lo * 256 + c) % TAG_MODULUS
+
-    return ((sum(b) & 0xFF) << 24) | lo
+A = 0; B = 0
+for each byte of the name (NOT lowercased):
+    c = extsb(byte)                       # SIGN-EXTENDED -- see below
+    A = ((A << 8) + c)                    # 32-bit
+    B = B + c
+    hi = mulhwu(A, TAG_MAGIC)
+    q  = (hi + ((A - hi) >>u 1)) >>u 23   # exact magic division, add-correction form
+    A  = A - q * TAG_MODULUS
+tag = ((B & 0xFF) << 24) | A
 ```
 
+The division is **exact**, so `A == A % 0x00FFFFDF`.
+
+### ⚠️ The bytes are SIGN-EXTENDED, and the disc could never have shown it
+
+The guest uses `extsb`. The version of this function recovered from the data
+used **unsigned** bytes — and it matched **all 1,271,462** disc names, because
+every one of them is ASCII. On random inputs containing a byte ≥ 0x80 the two
+disagree **18,096 times in 20,000**; e.g. `4e3fcfa50c864c2b41cf` hashes to
+`0x1AFF849A` in the guest and `0x1A65C73F` unsigned.
+
+This is the concrete answer to the "there may be inputs where the two disagree"
+caveat this note used to carry — and a reminder that a corpus can validate a
+function to six decimal places and still leave a whole branch untested. It took
+the disassembly, not more data.
+
 `tools/re-capture/unitgroup.py` (Python) and `sylpheed_formats::hash::tag_hash`
 (Rust) both implement it.
 
@@ -118,7 +139,44 @@ scheme was *designed* collision-free.
 (0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse
 lookup.
 
-## 🔴 Where it breaks: the `IXUD` container
+## ✅ `IXUD` — solved: it chains TWO moduli
+
+**`sub_82447E70`** is the wide sibling — `lhz`, 16-bit code units, 64-bit
+accumulator:
+
+```python
+M1 = 0xFFFFFF67   # 2^32 - 153, the loop modulus (64-bit arithmetic)
+M2 = 0x00FFFFDF   # 2^24 - 33, the same final modulus as IDXD
+
+A = 0; B = 0
+for each big-endian UTF-16 code unit ch:
+    A = ((A << 16) + ch) % M1
+    B = B + ch                    # the FULL code unit, not its low byte
+tag = ((B & 0xFF) << 24) | (A % M2)
+```
+
+Verified independently against real data: **86/86 record keys** and
+**108,261/108,261 field tags** in `GP_MAIN_GAME_E.pak`, `NoRecord` →
+`0x1C6D9C96`. Disc-wide the sweep is **906/906 keys, 624,440/624,440 tags**.
+
+**Why every search for it failed** — and the earlier diagnosis in this note was
+wrong. It is **not lossy**. It is a polynomial mod `M1` *then folded* through
+`M2`, and that composition is not a polynomial modulo anything, so the gcd test
+returns 1 and a Barrett sweep over single moduli cannot find it however wide.
+The tell was there in the data: fixed-position character deltas are constant
+within one name family but not another — which I read as "lossy" when it
+actually means "a second reduction happened after the loop".
+
+⚠️ The checksum sums the **full 16-bit code units**. Every IXUD name on this
+disc is ASCII, so this is indistinguishable here; for Japanese text it is not.
+
+### Its offsets are counted in 16-bit characters
+
+Same header/record/property/string-pool layout as IDXD, but `strsize` and every
+string offset are in **characters, not bytes**: `STR + 2·strsize == filesize`.
+All 534 objects parse under that rule.
+
+## 🔴 What this note used to say about `IXUD`
 
 `IXUD` is IDXD's wide-string sibling — 534 entries, same header/record/property/
 string-pool layout — and **`tag_hash` holds for 0 of its 19,808 names.**
@@ -142,13 +200,26 @@ mod-`M` polynomial can do that. It is a **lossy** hash — the same family as
 `name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the
 accumulator exceeds 2²⁴. Recovering it will need the executable, not the data.
 
+## ✅ Correction: `name_hash`'s reduction is EXACT, not lossy
+
+This note (and `hash.rs`) claimed the missing trailing conditional subtract made
+`sub_82455C78` something other than a true `%`. **That is wrong.**
+`rlwinm r6,r6,9,23,31` is exactly `hi >> 23`, and with
+`RECIP == floor(2^55/M) + 1` that is textbook Granlund–Montgomery magic
+division. Checked at every quotient boundary (`k·M−1, k·M, k·M+1`) across the
+whole 32-bit domain: **0 wrong of 770**. So `name_hash`'s low 24 bits really are
+`A % 0x00FFF9D7`.
+
+Also corrected: `sub_82455C78` does **not** lowercase internally — callers pass
+an already-normalised path. Where that normalisation happens is still ❔.
+
 ## 🟡 What is not settled
 
-* **The guest routine has not been located.** `name_hash`'s low 24 bits come
+* **Where IPFB path lowercasing happens.** `sub_82455C78` is case-sensitive and
-  from a Barrett step with *no* final conditional subtract, which is not the
+  no caller folds inline; it is somewhere upstream in the path library.
-  same function as `%` on every input. `tag_hash` is written here with exact
+* **Non-ASCII behaviour is unexercised by the disc.** Both the `extsb` path
-  modular arithmetic because it matches all 8643 known pairs — but if the game
+  (IDXD) and the code-unit checksum (IXUD) are transcribed from the
-  computes it the same Barrett way, there may be inputs where the two disagree.
+  disassembly, but no disc name reaches them.
-  Nothing outside those 8643 has been checked. Finding the routine in
+* **Sibling magics exist and are unexamined:** the container dispatchers
-  `default.xex` would settle it.
+  reference `IDX2`, `IDX3`, `IDXC` and `IIDX` alongside `IDXD`/`IXUD`. Whether
-* Whether the same hash keys IDXD tables in **other paks** is untested here.
+  any appear on disc is unchecked.