re: recover the IDXD record-key / field-tag hash (8643/8643)

Closes the 4-byte record key. tag_hash is name_hash's shape -- byte-sum
checksum in the top byte over a 24-bit modular polynomial -- with two different
constants: modulus 0x00FFFFDF (2^24-33, prime) instead of 0x00FFF9D7, and no
lowercasing, so tags are case-sensitive. name_hash explains 0 of 8643.

Recovered from the tables rather than the executable: every inline field name
is a known (name -> tag) pair, and comparing names differing in one character
gives the per-position weights 1, 0x100, 0x10000, 0x21, 0x2100, ... -- a byte
leaving bit 24 re-enters as 33, i.e. reduction mod 2^24-33. Holds where it is
easy to get wrong (distance 8 and 9 carry correctly).

A record's key is the tag of its own name: FormationSet rosters 362/362,
UnitGroup rosters 281/281, S02 squadron names 111/111 -- so records can be
addressed by name without reading the roster first.

Implemented in Python (unitgroup.tag_hash) and Rust
(sylpheed_formats::hash::tag_hash) with 3 new unit tests carrying disc-derived
vectors; cargo test -p sylpheed-formats --lib hash is 8/8 green.

Not settled: the guest routine is unlocated, so this uses exact modular
arithmetic where the game may use a Barrett step without final fixup.

tools/re-capture/freeze_waitobj.sh

@@ -33,6 +33,15 @@
 #     freeze_waitobj.sh boot [fly_s]   boot, fly, capture `healthy`, leave running
 #     freeze_waitobj.sh watch [secs]   poll for the freeze, capture `frozen`
 #     freeze_waitobj.sh run [fly_s]    boot + healthy + watch, end to end
+#     freeze_waitobj.sh repeat [n] [gap]  N captures of a HEALTHY run, tags h1..hN
+#     freeze_waitobj.sh stable [n] [gap]  boot, then repeat
+#
+# `repeat`/`stable` exist because a one-sample-per-state diff cannot tell a
+# freeze transition from ordinary variation: run 1's "T74/T75 move off a
+# semaphore" did not reproduce, because the HEALTHY state varies between
+# instants too. Sample the healthy run several times first, and only treat a
+# frozen difference as a signature if it is not something healthy play does
+# anyway.
 #
 # `run` exists because the whole experiment does not fit one Bash call and a
 # `timeout` kills the process group -- it took the emulator down once. Launch it
@@ -100,6 +109,17 @@ PY
 MODE="${1:-boot}"
 FLY="${2:-}"
 
+if [ "$MODE" = repeat ]; then
+  N="${FLY:-5}"; GAP="${3:-45}"
+  pgrep -x xenia_canary >/dev/null || { echo "NO EMULATOR"; exit 1; }
+  for i in $(seq 1 "$N"); do
+    capture "h$i"
+    [ "$i" -lt "$N" ] && sleepfor "$GAP"
+  done
+  python3 "$SD/waitobj_report.py" --stability $(seq -f 'h%g' 1 "$N")
+  echo "STABILITY DONE"; exit 0
+fi
+
 if [ "$MODE" = watch ]; then
   SECS="${FLY:-500}"
   pgrep -x xenia_canary >/dev/null || { echo "NO EMULATOR"; exit 1; }
@@ -146,3 +166,7 @@ if [ "$MODE" = run ]; then
   echo "--- watching for the freeze ($(date +%T))"
   exec "$0" watch "${WATCH_S:-1500}"
 fi
+if [ "$MODE" = stable ]; then
+  echo "--- sampling the HEALTHY run ($(date +%T))"
+  exec "$0" repeat "${REPEAT_N:-6}" "${REPEAT_GAP:-45}"
+fi

tools/re-capture/unitgroup.py

@@ -48,6 +48,24 @@ def name_hash(s):
         a = (a - (q * MODULUS)) & 0xFFFFFFFF
     return (((b << 24) & 0xFF000000) | (a & 0x00FFFFFF)) & 0xFFFFFFFF
 
+TAG_MODULUS = (1 << 24) - 33      # 0x00FFFFDF, prime
+
+def tag_hash(s):
+    """IDXD record key / field tag -- NOT name_hash.
+
+    Same shape as name_hash (8-bit byte-sum checksum over a 24-bit modular
+    polynomial) but modulo 0x00FFFFDF instead of 0x00FFF9D7, and NOT
+    lowercased, so tags are case-sensitive. Recovered empirically from the 8643
+    (name -> tag) pairs the tables themselves carry; `unitgroup.py --checktags`
+    re-verifies all of them. A record's key is the tag of its own name, which
+    each table lists in an in-table roster record.
+    """
+    b = s.encode()
+    lo = 0
+    for c in b:
+        lo = (lo * 256 + c) % TAG_MODULUS
+    return ((sum(b) & 0xFF) << 24) | lo
+
 def read_entry(pak, h):
     idx = open(pak, 'rb').read()
     base = pak[:-4]

tools/re-capture/waitobj_report.py

@@ -91,7 +91,39 @@ def diff_threads(a, b):
         print('  T%-5d %-32s %-32s %s' % (t, fa, fb, '' if fa == fb else '  <-- CHANGED'))
 
 
+def stability(tags):
+    """Which thread states hold STILL across repeated samples of one healthy run?
+
+    Written after a frozen-vs-healthy diff was read as a signature and did not
+    reproduce: the healthy state varies between instants too, so a difference of
+    two samples is not yet a difference of two states. Anything that moves here
+    is disqualified as freeze evidence before it is ever used as such.
+    """
+    snaps = [(t, per_thread(t)) for t in tags]
+    snaps = [(t, d) for t, d in snaps if d]
+    if len(snaps) < 2:
+        print('need at least 2 usable captures, got %d' % len(snaps)); return
+    threads = sorted({t for _, d in snaps for t in d}, reverse=True)
+    print('=== stability across %d healthy captures: %s ===' % (
+        len(snaps), ', '.join(t for t, _ in snaps)))
+    stable = moved = 0
+    for th in threads:
+        vals = [d.get(th, '--') for _, d in snaps]
+        uniq = sorted(set(vals))
+        if len(uniq) == 1:
+            stable += 1
+            print('  T%-5d STABLE   %s' % (th, uniq[0]))
+        else:
+            moved += 1
+            print('  T%-5d VARIES   %s' % (th, '  |  '.join(vals)))
+    print('  --- %d stable, %d vary across healthy play' % (stable, moved))
+    print('  Only a thread in the STABLE set can carry a frozen-state signature;')
+    print('  a VARIES thread differing when frozen proves nothing.')
+
+
 if __name__ == '__main__':
+    if sys.argv[1:2] == ['--stability']:
+        stability(sys.argv[2:]); sys.exit(0)
     tallies = {t: report(t) for t in (sys.argv[1:] or ['healthy'])}
     if len(tallies) > 1:
         a, b = list(tallies)