docs: IXUD solved, extsb correction, name_hash exactness retraction
This commit is contained in:
Sylpheed RE agent
@@ -1,25 +1,46 @@
|
||||
# The IDXD record-key / field-tag hash
|
||||
|
||||
Status: ✅ recovered and verified **1,271,462 / 1,271,462 disc-wide**;
|
||||
✅ record keys follow a stronger rule than first thought (no roster needed);
|
||||
🟡 the guest routine has not been located; 🔴 it does **not** cover the sibling
|
||||
`IXUD` container.
|
||||
Status: ✅ recovered, **guest routine located and transcribed**, verified
|
||||
**1,271,462 / 1,271,462 disc-wide**; ✅ record keys follow a stronger rule (no
|
||||
roster needed); ✅ **`IXUD` solved too** — a different, two-modulus hash.
|
||||
|
||||
This closes the **4-byte record key** that had been ❔ since the squadron roster
|
||||
was decoded.
|
||||
|
||||
## ✅ The function
|
||||
|
||||
```python
|
||||
TAG_MODULUS = (1 << 24) - 33 # 0x00FFFFDF, prime
|
||||
**`sub_82447DF0`**, transcribed instruction for instruction:
|
||||
|
||||
def tag_hash(s):
|
||||
b = s.encode(); lo = 0
|
||||
for c in b:
|
||||
lo = (lo * 256 + c) % TAG_MODULUS
|
||||
return ((sum(b) & 0xFF) << 24) | lo
|
||||
```python
|
||||
TAG_MODULUS = 0x00FFFFDF # 2^24 - 33, prime
|
||||
TAG_MAGIC = 0x2101 # floor(2^56/M)+1, the guest's divide magic
|
||||
|
||||
A = 0; B = 0
|
||||
for each byte of the name (NOT lowercased):
|
||||
c = extsb(byte) # SIGN-EXTENDED -- see below
|
||||
A = ((A << 8) + c) # 32-bit
|
||||
B = B + c
|
||||
hi = mulhwu(A, TAG_MAGIC)
|
||||
q = (hi + ((A - hi) >>u 1)) >>u 23 # exact magic division, add-correction form
|
||||
A = A - q * TAG_MODULUS
|
||||
tag = ((B & 0xFF) << 24) | A
|
||||
```
|
||||
|
||||
The division is **exact**, so `A == A % 0x00FFFFDF`.
|
||||
|
||||
### ⚠️ The bytes are SIGN-EXTENDED, and the disc could never have shown it
|
||||
|
||||
The guest uses `extsb`. The version of this function recovered from the data
|
||||
used **unsigned** bytes — and it matched **all 1,271,462** disc names, because
|
||||
every one of them is ASCII. On random inputs containing a byte ≥ 0x80 the two
|
||||
disagree **18,096 times in 20,000**; e.g. `4e3fcfa50c864c2b41cf` hashes to
|
||||
`0x1AFF849A` in the guest and `0x1A65C73F` unsigned.
|
||||
|
||||
This is the concrete answer to the "there may be inputs where the two disagree"
|
||||
caveat this note used to carry — and a reminder that a corpus can validate a
|
||||
function to six decimal places and still leave a whole branch untested. It took
|
||||
the disassembly, not more data.
|
||||
|
||||
`tools/re-capture/unitgroup.py` (Python) and `sylpheed_formats::hash::tag_hash`
|
||||
(Rust) both implement it.
|
||||
|
||||
@@ -118,7 +139,44 @@ scheme was *designed* collision-free.
|
||||
(0, 1, 2, …, the member index), **not** a hash. Never feed those to a reverse
|
||||
lookup.
|
||||
|
||||
## 🔴 Where it breaks: the `IXUD` container
|
||||
## ✅ `IXUD` — solved: it chains TWO moduli
|
||||
|
||||
**`sub_82447E70`** is the wide sibling — `lhz`, 16-bit code units, 64-bit
|
||||
accumulator:
|
||||
|
||||
```python
|
||||
M1 = 0xFFFFFF67 # 2^32 - 153, the loop modulus (64-bit arithmetic)
|
||||
M2 = 0x00FFFFDF # 2^24 - 33, the same final modulus as IDXD
|
||||
|
||||
A = 0; B = 0
|
||||
for each big-endian UTF-16 code unit ch:
|
||||
A = ((A << 16) + ch) % M1
|
||||
B = B + ch # the FULL code unit, not its low byte
|
||||
tag = ((B & 0xFF) << 24) | (A % M2)
|
||||
```
|
||||
|
||||
Verified independently against real data: **86/86 record keys** and
|
||||
**108,261/108,261 field tags** in `GP_MAIN_GAME_E.pak`, `NoRecord` →
|
||||
`0x1C6D9C96`. Disc-wide the sweep is **906/906 keys, 624,440/624,440 tags**.
|
||||
|
||||
**Why every search for it failed** — and the earlier diagnosis in this note was
|
||||
wrong. It is **not lossy**. It is a polynomial mod `M1` *then folded* through
|
||||
`M2`, and that composition is not a polynomial modulo anything, so the gcd test
|
||||
returns 1 and a Barrett sweep over single moduli cannot find it however wide.
|
||||
The tell was there in the data: fixed-position character deltas are constant
|
||||
within one name family but not another — which I read as "lossy" when it
|
||||
actually means "a second reduction happened after the loop".
|
||||
|
||||
⚠️ The checksum sums the **full 16-bit code units**. Every IXUD name on this
|
||||
disc is ASCII, so this is indistinguishable here; for Japanese text it is not.
|
||||
|
||||
### Its offsets are counted in 16-bit characters
|
||||
|
||||
Same header/record/property/string-pool layout as IDXD, but `strsize` and every
|
||||
string offset are in **characters, not bytes**: `STR + 2·strsize == filesize`.
|
||||
All 534 objects parse under that rule.
|
||||
|
||||
## 🔴 What this note used to say about `IXUD`
|
||||
|
||||
`IXUD` is IDXD's wide-string sibling — 534 entries, same header/record/property/
|
||||
string-pool layout — and **`tag_hash` holds for 0 of its 19,808 names.**
|
||||
@@ -142,13 +200,26 @@ mod-`M` polynomial can do that. It is a **lossy** hash — the same family as
|
||||
`name_hash`'s `rotl(a,8) & 0xFFFFFF00` step, which drops bits once the
|
||||
accumulator exceeds 2²⁴. Recovering it will need the executable, not the data.
|
||||
|
||||
## ✅ Correction: `name_hash`'s reduction is EXACT, not lossy
|
||||
|
||||
This note (and `hash.rs`) claimed the missing trailing conditional subtract made
|
||||
`sub_82455C78` something other than a true `%`. **That is wrong.**
|
||||
`rlwinm r6,r6,9,23,31` is exactly `hi >> 23`, and with
|
||||
`RECIP == floor(2^55/M) + 1` that is textbook Granlund–Montgomery magic
|
||||
division. Checked at every quotient boundary (`k·M−1, k·M, k·M+1`) across the
|
||||
whole 32-bit domain: **0 wrong of 770**. So `name_hash`'s low 24 bits really are
|
||||
`A % 0x00FFF9D7`.
|
||||
|
||||
Also corrected: `sub_82455C78` does **not** lowercase internally — callers pass
|
||||
an already-normalised path. Where that normalisation happens is still ❔.
|
||||
|
||||
## 🟡 What is not settled
|
||||
|
||||
* **The guest routine has not been located.** `name_hash`'s low 24 bits come
|
||||
from a Barrett step with *no* final conditional subtract, which is not the
|
||||
same function as `%` on every input. `tag_hash` is written here with exact
|
||||
modular arithmetic because it matches all 8643 known pairs — but if the game
|
||||
computes it the same Barrett way, there may be inputs where the two disagree.
|
||||
Nothing outside those 8643 has been checked. Finding the routine in
|
||||
`default.xex` would settle it.
|
||||
* Whether the same hash keys IDXD tables in **other paks** is untested here.
|
||||
* **Where IPFB path lowercasing happens.** `sub_82455C78` is case-sensitive and
|
||||
no caller folds inline; it is somewhere upstream in the path library.
|
||||
* **Non-ASCII behaviour is unexercised by the disc.** Both the `extsb` path
|
||||
(IDXD) and the code-unit checksum (IXUD) are transcribed from the
|
||||
disassembly, but no disc name reaches them.
|
||||
* **Sibling magics exist and are unexamined:** the container dispatchers
|
||||
reference `IDX2`, `IDX3`, `IDXC` and `IIDX` alongside `IDXD`/`IXUD`. Whether
|
||||
any appear on disc is unchecked.
|
||||
|
||||
Reference in New Issue
Block a user