re: recover the IDXD record-key / field-tag hash (8643/8643)
Closes the 4-byte record key. tag_hash is name_hash's shape -- byte-sum checksum in the top byte over a 24-bit modular polynomial -- with two different constants: modulus 0x00FFFFDF (2^24-33, prime) instead of 0x00FFF9D7, and no lowercasing, so tags are case-sensitive. name_hash explains 0 of 8643. Recovered from the tables rather than the executable: every inline field name is a known (name -> tag) pair, and comparing names differing in one character gives the per-position weights 1, 0x100, 0x10000, 0x21, 0x2100, ... -- a byte leaving bit 24 re-enters as 33, i.e. reduction mod 2^24-33. Holds where it is easy to get wrong (distance 8 and 9 carry correctly). A record's key is the tag of its own name: FormationSet rosters 362/362, UnitGroup rosters 281/281, S02 squadron names 111/111 -- so records can be addressed by name without reading the roster first. Implemented in Python (unitgroup.tag_hash) and Rust (sylpheed_formats::hash::tag_hash) with 3 new unit tests carrying disc-derived vectors; cargo test -p sylpheed-formats --lib hash is 8/8 green. Not settled: the guest routine is unlocated, so this uses exact modular arithmetic where the game may use a Barrett step without final fixup.
This commit is contained in:
22
docs/re/data/idxd-tag-hash.txt
Normal file
22
docs/re/data/idxd-tag-hash.txt
Normal file
@@ -0,0 +1,22 @@
|
||||
# IDXD record-key / field-tag hash — verification over the whole pak
|
||||
|
||||
tag_hash(s) = ((sum(bytes) & 0xff) << 24) | (base-256 polynomial mod 0x00FFFFDF)
|
||||
|
||||
named fields with a unique tag in GP_MAIN_GAME_E.pak : 8643
|
||||
tag_hash exact : 8643
|
||||
wrong : 0
|
||||
name_hash exact (the OTHER hash, for contrast) : 0
|
||||
|
||||
sample (name -> tag):
|
||||
AAGunShellID 31dd1a13
|
||||
AA_AxisMode_Max 80b0d604
|
||||
AA_AxisMode_Min 7eb0ddfa
|
||||
AA_PitchMinus_Max 6a133241
|
||||
AA_PitchMinus_Min 68133a37
|
||||
AA_PitchPlus_Max 028ab283
|
||||
AA_PitchPlus_Min 008aba79
|
||||
AA_Roll_Max ff8d842e
|
||||
AA_Roll_Min fd8d8c24
|
||||
AA_Yaw_Max 97182874
|
||||
AA_Yaw_Min 9518306a
|
||||
AB_AA_PitchMinus c78e8fdd
|
||||
84
docs/re/structures/idxd-tag-hash.md
Normal file
84
docs/re/structures/idxd-tag-hash.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# The IDXD record-key / field-tag hash
|
||||
|
||||
Status: ✅ recovered and verified 8643/8643 across `GP_MAIN_GAME_E.pak`;
|
||||
🟡 the guest routine has not been located, so the implementation is exact
|
||||
modular arithmetic rather than a transcribed op sequence.
|
||||
|
||||
This closes the **4-byte record key** that had been ❔ since the squadron roster
|
||||
was decoded.
|
||||
|
||||
## ✅ The function
|
||||
|
||||
```python
|
||||
TAG_MODULUS = (1 << 24) - 33 # 0x00FFFFDF, prime
|
||||
|
||||
def tag_hash(s):
|
||||
b = s.encode(); lo = 0
|
||||
for c in b:
|
||||
lo = (lo * 256 + c) % TAG_MODULUS
|
||||
return ((sum(b) & 0xFF) << 24) | lo
|
||||
```
|
||||
|
||||
`tools/re-capture/unitgroup.py` (Python) and `sylpheed_formats::hash::tag_hash`
|
||||
(Rust) both implement it.
|
||||
|
||||
## ✅ It is NOT `name_hash`, and the difference is two constants
|
||||
|
||||
The IPFB TOC hash and this one share a shape — an 8-bit additive checksum of the
|
||||
bytes in the top byte over a 24-bit modular polynomial in the low 24 — but
|
||||
differ in exactly two ways:
|
||||
|
||||
| | `name_hash` (IPFB TOC paths) | `tag_hash` (IDXD keys/tags) |
|
||||
|---|---|---|
|
||||
| modulus | `0x00FFF9D7` | **`0x00FFFFDF`** (2²⁴ − 33, prime) |
|
||||
| case | **lowercased** first | **case-sensitive** |
|
||||
|
||||
`name_hash` explains **0 of 8643** tags, so the two are not interchangeable.
|
||||
|
||||
## ✅ How it was recovered — from the tables, not the executable
|
||||
|
||||
Every IDXD record that carries an *inline* field name hands over a known
|
||||
(name → tag) pair. `GP_MAIN_GAME_E.pak` yields **8643** such pairs, all with
|
||||
distinct names.
|
||||
|
||||
1. **The top byte is the plain byte sum** — `(sum(bytes) & 0xff) == tag >> 24`
|
||||
for **8643 of 8643**. Same as `name_hash`.
|
||||
2. **The low 24 bits are a base-256 polynomial.** Comparing pairs of names that
|
||||
differ in a *single* character gives the weight of each position directly:
|
||||
|
||||
| distance from end | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| weight | `1` | `0x100` | `0x10000` | `0x21` | `0x2100` | `0x210000` | `0x441` | `0x44100` | `0x410084` |
|
||||
|
||||
Each step is a shift left by 8, and **a byte leaving bit 24 re-enters as 33**
|
||||
— `0x21`. That is reduction modulo `2²⁴ − 33`. It continues to hold where it
|
||||
would be easy to get wrong: at distance 8, `0x044100 << 8` overflows by
|
||||
`0x04`, and `0x410000 + 0x04·0x21 = 0x410084` ✓; at distance 9,
|
||||
`0x008400 + 0x41·0x21 = 0x008C61` ✓.
|
||||
|
||||
Result: **8643/8643 exact**, 0 wrong (`data/idxd-tag-hash.txt`).
|
||||
|
||||
## ✅ A record's key is the tag of its own name
|
||||
|
||||
Which is what makes it useful:
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| `FormationSet_S*` roster: `tag == tag_hash(name)` | **362 / 362** |
|
||||
| `UnitGroup_S*` roster: `tag == tag_hash(name)` | **281 / 281** |
|
||||
| S02 squadron names whose `tag_hash` is an actual record key | **111 / 111** |
|
||||
|
||||
So a table's records can now be addressed **by name** without reading its roster
|
||||
first. The roster is still the honest way to *enumerate* names — hashes do not
|
||||
invert — but resolving a known name no longer needs it.
|
||||
|
||||
## 🟡 What is not settled
|
||||
|
||||
* **The guest routine has not been located.** `name_hash`'s low 24 bits come
|
||||
from a Barrett step with *no* final conditional subtract, which is not the
|
||||
same function as `%` on every input. `tag_hash` is written here with exact
|
||||
modular arithmetic because it matches all 8643 known pairs — but if the game
|
||||
computes it the same Barrett way, there may be inputs where the two disagree.
|
||||
Nothing outside those 8643 has been checked. Finding the routine in
|
||||
`default.xex` would settle it.
|
||||
* Whether the same hash keys IDXD tables in **other paks** is untested here.
|
||||
Reference in New Issue
Block a user