re: the frozen wait-object capture, and screen_id was never a freeze test

Caught the freeze by waiting for the event (frozen.py + in_flight) instead of
sleeping a guessed interval; freeze_waitobj.sh splits into boot/watch so the
wait is not capped by one Bash call. Verified hard: a frame minutes later is
byte-identical to the capture.

Healthy vs frozen, same run: 20 -> 24 wait frames, XEvent 19 -> 23,
XSemaphore 8 -> 7. The signature is per-thread -- 17 of 24 threads sit on the
exact object they were on, four previously-running threads park, and T74/T75
move off a semaphore onto an event. So the freeze is not a whole-emulator stall.

Also corrects the previous entry's test: screen_id reads 'flight' during a
freeze by design, which is why frozen.py exists. Re-testing the saved frames
says that run was genuinely healthy, but it was right by luck.

heavy_read.py added to test whether the instrument provokes the freeze: I/O is
free (371 MB in 0.1s, page cache), the cost is Python-level CPU. One data point
-- 670s clean, then frozen 54s after the inducer started -- recorded as n=1, not
as causation.

tools/re-capture/freeze_waitobj.sh

@@ -18,8 +18,20 @@
 # is a vtable, so a value that does not resolve to a `vtable for ...` symbol is
 # discarded rather than interpreted.
 #
-# Captures TWICE in one run so the comparison is within-run: once while the
-# mission is healthy, once after the ~270s black-screen.
+# Captures TWICE so the comparison is within-run: once while the mission is
+# healthy, once frozen. The frozen one is reached by WAITING FOR THE EVENT, not
+# by a clock -- measured onsets are 27/45/83/183/255s (median ~83), only about
+# half of runs freeze at all, and the "~270s" figure this script used to sleep
+# for was never a real bound.
+#
+# "Frozen" is frozen.py's test -- two byte-identical frames while the flight HUD
+# is still up. It is NOT `screen_id == flight` being false: a frozen mission
+# still classifies as `flight`, which is the whole reason frozen.py exists.
+#
+# Subcommands, because the boot and the wait do not fit one Bash call but the
+# emulator survives BETWEEN calls in a turn:
+#     freeze_waitobj.sh boot [fly_s]   boot, fly, capture `healthy`, leave running
+#     freeze_waitobj.sh watch [secs]   poll for the freeze, capture `frozen`
 set -u
 export HOME=/sylph-home/re SDL_AUDIODRIVER=dummy DISPLAY=:98
 export PYTHONPATH=/sylph-home/.local/lib/python3.12/site-packages
@@ -76,20 +88,42 @@ PY
   echo "[$tag] captured"
 }
 
+MODE="${1:-boot}"
+FLY="${2:-}"
+
+if [ "$MODE" = watch ]; then
+  SECS="${FLY:-500}"
+  pgrep -x xenia_canary >/dev/null || { echo "NO EMULATOR"; exit 1; }
+  end=$((SECONDS + SECS))
+  while [ $SECONDS -lt $end ]; do
+    pgrep -x xenia_canary >/dev/null || { echo "EMULATOR GONE at ${SECONDS}s"; exit 4; }
+    if python3 "$SD/frozen.py" 5 >/dev/null 2>&1; then
+      if python3 -c "import sys;sys.path.insert(0,'$SD');import frozen;sys.exit(0 if frozen.in_flight() else 1)"; then
+        echo "FROZEN IN FLIGHT at ${SECONDS}s of this watch"
+        capture frozen
+        python3 "$SD/waitobj_report.py" healthy frozen
+        echo "FREEZE WAITOBJ DONE"; exit 0
+      fi
+      echo "frozen but NOT in flight (mission over) at ${SECONDS}s"; exit 5
+    fi
+    sleepfor 12
+  done
+  echo "NO FREEZE within ${SECS}s (still animating)"; exit 1
+fi
+
+# ---- boot mode ----
+FLY="${FLY:-150}"
 "$SD/launch_mission.sh" fly || { echo "BOOT FAILED"; exit 1; }
 CFG=/tmp/nav-fz.json
 for t in 1 2 3; do
   python3 "$SD/pad.py" set "rt=1" >/dev/null 2>&1 || true; sleepfor 3
   python3 "$SD/pad.py" clear >/dev/null 2>&1 || true
   if python3 "$SD/entities2.py" self 0x130 "$CFG" >/dev/null 2>&1; then
-    SYLPH_HUNT=1 SYLPH_KEEPOUT=1400 nohup python3 "$SD/pilot.py" "$CFG" $((FLY1+FLY2+60)) \
-      </dev/null >/tmp/fz-pilot.log 2>&1 & P=$!; echo "--- pilot flying"; break
+    SYLPH_HUNT=1 SYLPH_KEEPOUT=1400 nohup python3 "$SD/pilot.py" "$CFG" 3000 \
+      </dev/null >/tmp/fz-pilot.log 2>&1 & echo "--- pilot flying"; break
   fi
 done
-echo "--- flying ${FLY1}s to the healthy checkpoint"
-sleepfor "$FLY1"; capture healthy
-echo "--- flying ${FLY2}s more, past the ~270s black-screen"
-sleepfor "$FLY2"; capture frozen
-[ -n "${P:-}" ] && kill "$P" 2>/dev/null
-python3 "$SD/waitobj_report.py" healthy frozen
-echo "FREEZE WAITOBJ DONE"
+echo "--- flying ${FLY}s to the healthy control capture"
+sleepfor "$FLY"; capture healthy
+python3 "$SD/waitobj_report.py" healthy
+echo "BOOT PHASE DONE -- emulator left running; now: freeze_waitobj.sh watch"

tools/re-capture/heavy_read.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+"""Reproduce the EXPENSIVE part of the old probes: full-region guest scans.
+
+Not a probe -- it computes nothing. It exists to test whether the in-mission
+freeze is caused by the instrument rather than by the game. The tally in
+BACKLOG.md is suggestive but only correlational: the heavy probe (32 MB at
+startup + 32 MB every 90 s + two more for calibration) froze at 27/45/83/183/255
+s, while the cheap probe with the periodic rescan removed was clean past 200 s
+on 3 of 4 runs. This makes the comparison causal by adding the reads BACK to a
+run that is otherwise identical and demonstrably clean.
+
+MEASURED FIRST: the reads themselves are not the cost. A full uncapped walk of
+every allocated extent moves 371 MB in 0.1 s -- it is all page cache. What the
+old probes actually spent was CPU, unpacking and comparing every 4-byte word of
+that region in Python to build the witness candidate list. So `cpu` mode does
+that, and it is the mode that tests the hypothesis; `read` mode is kept only as
+the control that shows I/O is free.
+
+Usage: heavy_read.py [secs] [period_s] [read|cpu]
+"""
+import os, sys, time
+
+def shm():
+    d = [f for f in os.listdir('/dev/shm') if f.startswith('xenia_memory_')]
+    return '/dev/shm/' + d[0] if d else None
+
+def scan(path, cap=(1 << 62)):
+    """Walk EVERY allocated extent with SEEK_DATA. A 32 MB cap is page-cache cheap
+    (0.0 s measured), so the cap is off: the expensive thing the old probes did was
+    the FULL-region search, not a fixed-size read."""
+    got = 0
+    with open(path, 'rb', buffering=0) as f:
+        end = os.fstat(f.fileno()).st_size
+        off = 0
+        while off < end and got < cap:
+            try:
+                off = os.lseek(f.fileno(), off, os.SEEK_DATA)
+            except OSError:
+                break
+            n = min(1 << 20, cap - got)
+            b = f.read(n)
+            if not b:
+                break
+            got += len(b); off += len(b)
+    return got
+
+def cpu_scan(path):
+    """The expensive thing: touch every word in Python, as the witness search did."""
+    import struct
+    words = 0
+    with open(path, 'rb', buffering=0) as f:
+        end = os.fstat(f.fileno()).st_size
+        off = 0
+        while off < end:
+            try:
+                off = os.lseek(f.fileno(), off, os.SEEK_DATA)
+            except OSError:
+                break
+            b = f.read(1 << 20)
+            if not b:
+                break
+            n = len(b) // 4
+            for v in struct.unpack('>%dI' % n, b[:n * 4]):
+                if 0 < v < 1000:
+                    words += 1
+            off += len(b)
+    return words
+
+secs = float(sys.argv[1]) if len(sys.argv) > 1 else 600
+period = float(sys.argv[2]) if len(sys.argv) > 2 else 20
+mode = sys.argv[3] if len(sys.argv) > 3 else 'read' 
+p = shm()
+if not p:
+    print('no guest memory'); sys.exit(1)
+print('heavy reads on %s every %gs for %gs' % (p, period, secs), flush=True)
+t0 = time.time(); n = 0
+while time.time() - t0 < secs:
+    a = time.time(); n += 1
+    if mode == 'cpu':
+        got = cpu_scan(p)
+        print('  cpu-scan %d: %d hits in %.1fs' % (n, got, time.time() - a), flush=True)
+    else:
+        got = scan(p)
+        print('  scan %d: %.1f MB in %.1fs' % (n, got / 1048576, time.time() - a), flush=True)
+    time.sleep(period)

tools/re-capture/waitobj_report.py

@@ -70,6 +70,27 @@ def report(tag):
         print('      %-45s %d' % (k, c))
     return tally
 
+def per_thread(tag):
+    d = {}
+    for r in parse(tag):
+        live = []
+        for s_ in r['slots']:
+            if s_ is None: break
+            live.append(s_.replace('xe::kernel::', ''))
+        d[int(r['th'])] = '%s(%s)' % (r['kind'], ','.join(live))
+    return d
+
+
+def diff_threads(a, b):
+    """The tally alone hides the signature -- WHICH thread moved is the result."""
+    x, y = per_thread(a), per_thread(b)
+    print('=== per-thread %s -> %s ===' % (a, b))
+    print('  %-6s %-32s %-32s' % ('thread', a, b))
+    for t in sorted(set(x) | set(y), reverse=True):
+        fa, fb = x.get(t, '--'), y.get(t, '--')
+        print('  T%-5d %-32s %-32s %s' % (t, fa, fb, '' if fa == fb else '  <-- CHANGED'))
+
+
 if __name__ == '__main__':
     tallies = {t: report(t) for t in (sys.argv[1:] or ['healthy'])}
     if len(tallies) > 1:
@@ -79,3 +100,4 @@ if __name__ == '__main__':
         for k in sorted(keys):
             x, y = tallies[a][k], tallies[b][k]
             print('   %-45s %3d -> %-3d %s' % (k, x, y, '' if x == y else '  CHANGED'))
+        diff_threads(a, b)