Found the routines in the disassembly DB rather than guessing from data:
sub_82447DF0 IDXD tag hash (lbz+extsb, modulus 0x00FFFFDF, magic 0x2101)
sub_82447E70 IXUD tag hash (lhz, 64-bit, modulus 0xFFFFFF67 then 0x00FFFFDF)
Both transcribed instruction-for-instruction into Python and Rust.
IXUD SOLVED. It defeated every single-modulus search because it chains TWO
exact moduli -- the loop reduces mod 2^32-153 in 64-bit arithmetic and only the
result is folded mod 2^24-33. A polynomial mod M1 folded through M2 is not a
polynomial mod anything, which is exactly why the gcd test returned 1. Verified
independently: 86/86 record keys and 108,261/108,261 field tags in
GP_MAIN_GAME_E.pak, and NoRecord -> 0x1c6d9c96.
CORRECTION 1: tag_hash must SIGN-EXTEND each byte (extsb). My reconstruction
used unsigned bytes and matched all 1.27M disc names -- every one is ASCII --
while disagreeing on ~90% of random inputs with a byte >= 0x80 (verified:
18096/20000). The disc could never have caught this; only the disassembly did.
CORRECTION 2: name_hash's reduction is EXACT, not lossy. The module doc claimed
the missing conditional subtract made it something other than %. rlwinm r6,r6,
9,23,31 is just hi>>23, and with RECIP = floor(2^55/M)+1 that is Granlund-
Montgomery magic division -- 0 wrong at every quotient boundary across the full
32-bit domain. Retracted.
cargo test -p sylpheed-formats --lib hash: 10/10.
Closes the 4-byte record key. tag_hash is name_hash's shape -- byte-sum
checksum in the top byte over a 24-bit modular polynomial -- with two different
constants: modulus 0x00FFFFDF (2^24-33, prime) instead of 0x00FFF9D7, and no
lowercasing, so tags are case-sensitive. name_hash explains 0 of 8643.
Recovered from the tables rather than the executable: every inline field name
is a known (name -> tag) pair, and comparing names differing in one character
gives the per-position weights 1, 0x100, 0x10000, 0x21, 0x2100, ... -- a byte
leaving bit 24 re-enters as 33, i.e. reduction mod 2^24-33. Holds where it is
easy to get wrong (distance 8 and 9 carry correctly).
A record's key is the tag of its own name: FormationSet rosters 362/362,
UnitGroup rosters 281/281, S02 squadron names 111/111 -- so records can be
addressed by name without reading the roster first.
Implemented in Python (unitgroup.tag_hash) and Rust
(sylpheed_formats::hash::tag_hash) with 3 new unit tests carrying disc-derived
vectors; cargo test -p sylpheed-formats --lib hash is 8/8 green.
Not settled: the guest routine is unlocated, so this uses exact modular
arithmetic where the game may use a Barrett step without final fixup.
Add hash::recover_toc_name + TOC_NAME_SCHEMES (confirmed path schemes:
unit\<ID>.tbl, weapon\<ID>.tbl, message\<ID>.tbl, effect\<ID>.tbl,
<name>.tbl) that reproduce an entry's original backslash path from its
internal identity string via the recovered name-hash.
`pak list` now extracts identifier candidates from each IDXD entry
(ID/Name/Model + pool tokens) and prints the resolved path when a scheme
matches — e.g. unit\UN_f001_TCAF_DeltaSaber_T.tbl,
weapon\Weapon_…_Missile.tbl, message\CharacterCARL.tbl — plus a
name-resolved count. 308/1004 resolved on GP_MAIN_GAME_E; the remainder
use deeper cross-referenced paths (deferred).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pak TOC key was previously an unknown hash (crc32/fnv/djb2/… ruled
out), forcing content-only lookup. Recovered it by static RE of the
retail title:
sub_824609C8 (pak lookup-by-name) dups the requested path, lowercases
it (sub_825F4F90), hashes with sub_82455C78, then binary-searches the
sorted TOC. sub_82455C78 is a per-byte Barrett-reduced polynomial hash
(modulus 0x00FFF9D7, reciprocal 0x80031493): the low 24 bits are the
modular hash, the top byte an 8-bit additive checksum of the bytes.
New `hash` module reproduces it exactly (faithful op sequence, no
textbook %). Verified against the real disc: name_hash("files.tbl") ==
0x83421153 and name_hash("eng\\weapon.tbl") == 0x900C8DCD, both present
in retail TOCs. Add PakArchive::find_by_name / read_by_name and a disc
integration test resolving eng\weapon.tbl by path (→ valid IDXD).
Note: retail paths use backslash separators (eng\weapon.tbl).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
