re: locate both guest hash routines; IXUD solved; two corrections
Found the routines in the disassembly DB rather than guessing from data: sub_82447DF0 IDXD tag hash (lbz+extsb, modulus 0x00FFFFDF, magic 0x2101) sub_82447E70 IXUD tag hash (lhz, 64-bit, modulus 0xFFFFFF67 then 0x00FFFFDF) Both transcribed instruction-for-instruction into Python and Rust. IXUD SOLVED. It defeated every single-modulus search because it chains TWO exact moduli -- the loop reduces mod 2^32-153 in 64-bit arithmetic and only the result is folded mod 2^24-33. A polynomial mod M1 folded through M2 is not a polynomial mod anything, which is exactly why the gcd test returned 1. Verified independently: 86/86 record keys and 108,261/108,261 field tags in GP_MAIN_GAME_E.pak, and NoRecord -> 0x1c6d9c96. CORRECTION 1: tag_hash must SIGN-EXTEND each byte (extsb). My reconstruction used unsigned bytes and matched all 1.27M disc names -- every one is ASCII -- while disagreeing on ~90% of random inputs with a byte >= 0x80 (verified: 18096/20000). The disc could never have caught this; only the disassembly did. CORRECTION 2: name_hash's reduction is EXACT, not lossy. The module doc claimed the missing conditional subtract made it something other than %. rlwinm r6,r6, 9,23,31 is just hi>>23, and with RECIP = floor(2^55/M)+1 that is Granlund- Montgomery magic division -- 0 wrong at every quotient boundary across the full 32-bit domain. Retracted. cargo test -p sylpheed-formats --lib hash: 10/10.
This commit is contained in:
Sylpheed RE agent
@@ -20,9 +20,14 @@
|
||||
//! i.e. the low 24 bits are a modular polynomial hash and the top byte is an
|
||||
//! 8-bit additive checksum of the bytes. The reduction constant `0x8003_1493`
|
||||
//! is the reciprocal of the modulus `0x00FF_F9D7` used by the `mulhwu`/`mullw`
|
||||
//! Barrett step; there is **no** trailing conditional subtract, so the value is
|
||||
//! defined by the exact op sequence (faithfully reproduced below), not by a
|
||||
//! textbook `%`.
|
||||
//! Barrett step.
|
||||
//!
|
||||
//! **The reduction is EXACT, not lossy.** An earlier version of this note said
|
||||
//! the missing trailing conditional subtract made it something other than `%`.
|
||||
//! It does not: `rlwinm r6,r6,9,23,31` is exactly `hi >> 23`, and with
|
||||
//! `RECIP == floor(2^55/M) + 1` that is standard Granlund–Montgomery magic
|
||||
//! division. Checked at every quotient boundary (`k·M−1, k·M, k·M+1`) across the
|
||||
//! whole 32-bit domain: 0 wrong of 770. So the low 24 bits really are `A % M`.
|
||||
//!
|
||||
//! Verified against the real disc: `name_hash("files.tbl") == 0x8342_1153`
|
||||
//! and `name_hash("eng\\weapon.tbl") == 0x900C_8DCD`, both of which are present
|
||||
@@ -152,43 +157,82 @@ mod tests {
|
||||
}
|
||||
|
||||
/// Barrett modulus of the **record/field tag** hash — a different constant from
|
||||
/// [`MODULUS`], recovered separately (see below).
|
||||
const TAG_MODULUS: u32 = 0x00FF_FFDF; // 2^24 - 33
|
||||
/// [`MODULUS`], and the same one the IXUD hash folds down to.
|
||||
const TAG_MODULUS: u32 = 0x00FF_FFDF; // 2^24 - 33, prime
|
||||
/// The guest's divide magic for [`TAG_MODULUS`] (`floor(2^56/M) + 1`).
|
||||
const TAG_MAGIC: u32 = 0x2101;
|
||||
/// The IXUD loop modulus, applied in 64-bit arithmetic before [`TAG_MODULUS`].
|
||||
const IXUD_M1: u64 = 0xFFFF_FF67; // 2^32 - 153
|
||||
|
||||
/// Hash an IDXD **record key / field tag**.
|
||||
/// Hash an IDXD **record key / field tag** — `sub_82447DF0`.
|
||||
///
|
||||
/// This is *not* [`name_hash`]. IDXD tables key their records and name their
|
||||
/// fields with the same shape of hash — an 8-bit additive checksum in the top
|
||||
/// byte over a 24-bit modular polynomial — but with two differences:
|
||||
/// This is *not* [`name_hash`]. Same shape — an 8-bit additive checksum over a
|
||||
/// 24-bit modular polynomial — but two constants differ:
|
||||
///
|
||||
/// * the modulus is `0x00FF_FFDF` (= 2^24 − 33, prime), not `0x00FF_F9D7`;
|
||||
/// * the bytes are **not** lowercased, so tags are case-sensitive.
|
||||
/// * modulus `0x00FF_FFDF` (2^24 − 33, prime), not `0x00FF_F9D7`;
|
||||
/// * **no lowercasing**, so tags are case-sensitive. The disc relies on this:
|
||||
/// 17 name pairs differ only in case (`UNIT`/`Unit`, `TYPE`/`Type`, …) and
|
||||
/// `name_hash` collides on every one of them.
|
||||
///
|
||||
/// Recovered empirically rather than from the executable. Every IDXD record in
|
||||
/// `GP_MAIN_GAME_E.pak` that carries an inline field name gives a known
|
||||
/// (name → tag) pair; there are **8643** such pairs, all with distinct names,
|
||||
/// and `name_hash` explains none of them. Comparing pairs of names differing in
|
||||
/// a single character yields the per-position weights `1, 0x100, 0x10000,
|
||||
/// 0x21, 0x2100, 0x210000, 0x441, …` — i.e. a base-256 polynomial in which
|
||||
/// shifting a byte out of bit 24 re-enters as `33`, which is reduction modulo
|
||||
/// `2^24 − 33`. The top byte is the plain sum of the bytes, exactly as in
|
||||
/// `name_hash` (8643/8643).
|
||||
/// A record's key is the tag of its **own** name — 190,782/190,782 records
|
||||
/// disc-wide — so records are addressable by name without reading a roster.
|
||||
///
|
||||
/// This closes the IDXD record key: a record's key is the tag of its **name**,
|
||||
/// which each table also lists in an in-table roster record.
|
||||
///
|
||||
/// ⚠️ Implemented with exact modular arithmetic. The guest routine has **not**
|
||||
/// been located, so if it uses a Barrett step without a final fixup — as
|
||||
/// `sub_82455C78` does — there could be inputs where the two disagree. All 8643
|
||||
/// known pairs agree; nothing beyond them has been checked.
|
||||
/// The guest **sign-extends** each byte (`extsb`), and that is load-bearing:
|
||||
/// a version of this using unsigned bytes matched all 1.27M disc names, because
|
||||
/// every one is ASCII, while disagreeing on ~90% of random inputs containing a
|
||||
/// byte ≥ 0x80. Only the disassembly could catch that.
|
||||
pub fn tag_hash(name: &str) -> u32 {
|
||||
let mut lo: u32 = 0;
|
||||
let mut sum: u32 = 0;
|
||||
for &byte in name.as_bytes() {
|
||||
lo = ((lo as u64 * 256 + byte as u64) % TAG_MODULUS as u64) as u32;
|
||||
sum = sum.wrapping_add(byte as u32);
|
||||
tag_hash_bytes(name.as_bytes())
|
||||
}
|
||||
|
||||
/// [`tag_hash`] over raw bytes — the form that can express a non-UTF-8 name, and
|
||||
/// the only way to exercise the `extsb` path.
|
||||
pub fn tag_hash_bytes(bytes: &[u8]) -> u32 {
|
||||
let mut a: u32 = 0;
|
||||
let mut b: u32 = 0;
|
||||
for &byte in bytes {
|
||||
let c = byte as i8 as i32 as u32; // extsb
|
||||
a = (a << 8).wrapping_add(c);
|
||||
b = b.wrapping_add(c);
|
||||
// Exact magic division by TAG_MODULUS, in the guest's add-correction form.
|
||||
let hi = ((a as u64 * TAG_MAGIC as u64) >> 32) as u32;
|
||||
let q = hi.wrapping_add(a.wrapping_sub(hi) >> 1) >> 23;
|
||||
a = a.wrapping_sub(q.wrapping_mul(TAG_MODULUS));
|
||||
}
|
||||
((sum & 0xFF) << 24) | (lo & 0x00FF_FFFF)
|
||||
((b << 24) & 0xFF00_0000) | (a & 0x00FF_FFFF)
|
||||
}
|
||||
|
||||
/// Hash an **IXUD** record key / field tag — `sub_82447E70`.
|
||||
///
|
||||
/// IXUD is IDXD's wide-string sibling: identical container layout, but strings
|
||||
/// are UTF-16BE and `strsize` and every string offset are counted in **16-bit
|
||||
/// characters, not bytes** (`STR + 2·strsize == filesize`).
|
||||
///
|
||||
/// `units` is the name as big-endian UTF-16 code units.
|
||||
///
|
||||
/// It resisted every single-modulus search because it chains **two** exact
|
||||
/// moduli: the loop reduces mod `2^32 − 153` in 64-bit arithmetic, and only the
|
||||
/// final value is folded into 24 bits mod `2^24 − 33`. A polynomial mod `M1`
|
||||
/// folded through `M2` is not a polynomial mod anything, which is why a gcd over
|
||||
/// the observed pairs returns 1 and a Barrett sweep finds nothing.
|
||||
///
|
||||
/// The checksum byte sums the **full 16-bit code units**, not their low bytes —
|
||||
/// indistinguishable on this disc, where every IXUD name is ASCII, but not in
|
||||
/// general.
|
||||
pub fn ixud_hash(units: &[u16]) -> u32 {
|
||||
let mut a: u64 = 0;
|
||||
let mut b: u32 = 0;
|
||||
for &ch in units {
|
||||
a = ((a << 16) + ch as u64) % IXUD_M1;
|
||||
b = b.wrapping_add(ch as u32);
|
||||
}
|
||||
((b & 0xFF) << 24) | (a % TAG_MODULUS as u64) as u32
|
||||
}
|
||||
|
||||
/// Convenience: [`ixud_hash`] for an ASCII/UTF-8 name.
|
||||
pub fn ixud_hash_str(name: &str) -> u32 {
|
||||
let units: Vec<u16> = name.encode_utf16().collect();
|
||||
ixud_hash(&units)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
@@ -207,6 +251,22 @@ mod tag_tests {
|
||||
assert_eq!(tag_hash("Formation_ADAN_Turret07_30"), 0x30CE_86BE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ixud_uses_a_different_hash_entirely() {
|
||||
// Verified against real IXUD data: 86/86 record keys and 108,261/108,261
|
||||
// field tags in GP_MAIN_GAME_E.pak.
|
||||
assert_eq!(super::ixud_hash_str("NoRecord"), 0x1C6D_9C96);
|
||||
assert_ne!(super::ixud_hash_str("NoRecord"), tag_hash("NoRecord"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tag_hash_sign_extends_high_bytes() {
|
||||
// The guest uses extsb. Unsigned bytes agree on all-ASCII names but not
|
||||
// here -- this input is the concrete counterexample.
|
||||
let bytes = [0x4eu8, 0x3f, 0xcf, 0xa5, 0x0c, 0x86, 0x4c, 0x2b, 0x41, 0xcf];
|
||||
assert_eq!(super::tag_hash_bytes(&bytes), 0x1AFF_849A);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tags_are_case_sensitive_unlike_name_hash() {
|
||||
// name_hash lowercases first; tag_hash must not.
|
||||
|
||||
Reference in New Issue
Block a user