re: recover the IDXD record-key / field-tag hash (8643/8643)

Closes the 4-byte record key. tag_hash is name_hash's shape -- byte-sum
checksum in the top byte over a 24-bit modular polynomial -- with two different
constants: modulus 0x00FFFFDF (2^24-33, prime) instead of 0x00FFF9D7, and no
lowercasing, so tags are case-sensitive. name_hash explains 0 of 8643.

Recovered from the tables rather than the executable: every inline field name
is a known (name -> tag) pair, and comparing names differing in one character
gives the per-position weights 1, 0x100, 0x10000, 0x21, 0x2100, ... -- a byte
leaving bit 24 re-enters as 33, i.e. reduction mod 2^24-33. Holds where it is
easy to get wrong (distance 8 and 9 carry correctly).

A record's key is the tag of its own name: FormationSet rosters 362/362,
UnitGroup rosters 281/281, S02 squadron names 111/111 -- so records can be
addressed by name without reading the roster first.

Implemented in Python (unitgroup.tag_hash) and Rust
(sylpheed_formats::hash::tag_hash) with 3 new unit tests carrying disc-derived
vectors; cargo test -p sylpheed-formats --lib hash is 8/8 green.

Not settled: the guest routine is unlocated, so this uses exact modular
arithmetic where the game may use a Barrett step without final fixup.
This commit is contained in:
Sylpheed RE agent
2026-08-25 10:38:18 +00:00
parent 6903b19a27
commit cbf52ba9f9
6 changed files with 251 additions and 0 deletions

View File

@@ -150,3 +150,74 @@ mod tests {
assert_eq!(name_hash(""), 0);
}
}
/// Barrett modulus of the **record/field tag** hash — a different constant from
/// [`MODULUS`], recovered separately (see below).
const TAG_MODULUS: u32 = 0x00FF_FFDF; // 2^24 - 33
/// Hash an IDXD **record key / field tag**.
///
/// This is *not* [`name_hash`]. IDXD tables key their records and name their
/// fields with the same shape of hash — an 8-bit additive checksum in the top
/// byte over a 24-bit modular polynomial — but with two differences:
///
/// * the modulus is `0x00FF_FFDF` (= 2^24 33, prime), not `0x00FF_F9D7`;
/// * the bytes are **not** lowercased, so tags are case-sensitive.
///
/// Recovered empirically rather than from the executable. Every IDXD record in
/// `GP_MAIN_GAME_E.pak` that carries an inline field name gives a known
/// (name → tag) pair; there are **8643** such pairs, all with distinct names,
/// and `name_hash` explains none of them. Comparing pairs of names differing in
/// a single character yields the per-position weights `1, 0x100, 0x10000,
/// 0x21, 0x2100, 0x210000, 0x441, …` — i.e. a base-256 polynomial in which
/// shifting a byte out of bit 24 re-enters as `33`, which is reduction modulo
/// `2^24 33`. The top byte is the plain sum of the bytes, exactly as in
/// `name_hash` (8643/8643).
///
/// This closes the IDXD record key: a record's key is the tag of its **name**,
/// which each table also lists in an in-table roster record.
///
/// ⚠️ Implemented with exact modular arithmetic. The guest routine has **not**
/// been located, so if it uses a Barrett step without a final fixup — as
/// `sub_82455C78` does — there could be inputs where the two disagree. All 8643
/// known pairs agree; nothing beyond them has been checked.
pub fn tag_hash(name: &str) -> u32 {
let mut lo: u32 = 0;
let mut sum: u32 = 0;
for &byte in name.as_bytes() {
lo = ((lo as u64 * 256 + byte as u64) % TAG_MODULUS as u64) as u32;
sum = sum.wrapping_add(byte as u32);
}
((sum & 0xFF) << 24) | (lo & 0x00FF_FFFF)
}
#[cfg(test)]
mod tag_tests {
use super::tag_hash;
#[test]
fn known_tags_from_the_disc_tables() {
// Field names, from IDXD records in GP_MAIN_GAME_E.pak.
assert_eq!(tag_hash("SideID"), 0x1225_E093);
assert_eq!(tag_hash("ID"), 0x8D00_4944);
assert_eq!(tag_hash("Name"), 0x8161_7773);
// Formation record keys, from FormationSet_S02.tbl.
assert_eq!(tag_hash("Formation_4_Bird"), 0x22A5_EEED);
assert_eq!(tag_hash("Formation_1_only"), 0x6047_EECF);
assert_eq!(tag_hash("Formation_ADAN_Turret07_30"), 0x30CE_86BE);
}
#[test]
fn tags_are_case_sensitive_unlike_name_hash() {
// name_hash lowercases first; tag_hash must not.
assert_ne!(tag_hash("SideID"), tag_hash("sideid"));
}
#[test]
fn the_top_byte_is_the_byte_sum() {
for s in ["ID", "Formation_4_Bird", "SideID"] {
let sum = s.as_bytes().iter().map(|&b| b as u32).sum::<u32>() & 0xFF;
assert_eq!(tag_hash(s) >> 24, sum, "{s}");
}
}
}