From 2b98105ac03b15dd5e6b3c8fa0308ad886efc811 Mon Sep 17 00:00:00 2001
From: Sylpheed RE agent <agent@localhost>
Date: Mon, 24 Aug 2026 17:09:43 +0000
Subject: [PATCH] re: multi-witness vote works, its threshold did not, and
 freezes persist

The graded output is the improvement. Instead of a single bit it reports how
many witnesses agree, and the sequence tells a coherent story: 11, 9, 7, 4, 1,
then 0 of 31, with the drop to zero at t=183 s coinciding exactly with the last
loss and 106 s of nothing after it. That is a real freeze, identified.

The threshold was wrong though. Flagging a stall at "fewer than half" marked the
entire run stalled, including samples in which craft were destroyed, so 11 of 31
advancing is a healthy guest rather than a stalled one. The cause is the cluster
choice: the modal rate was 93/s, far above the ~16.5/s frame rate timer_probe
measured, and those are subsystem counters that tick in bursts and sit idle in
most 15 s windows even while the game runs. Picking the modal cluster was
convenient rather than principled.

Fixed to prefer the cluster whose rate falls in the frame-rate band of 8-40/s,
falling back to modal only if none exists, and to flag a stall only when zero
witnesses advance, which is the signal the data actually supports. Not yet run.

The uncomfortable part: this run used the cheap probe and still froze, at about
183 s. The previous iteration's "0 stalled samples" came from the unreliable
single-word witness and cannot stand as validation. What the evidence supports
now is that the no-probe control ran 300 s clean, the heavy probe froze at 27 to
255 s, and the cheap probe froze at 183 s -- one run on each arm. Cheap sampling
plausibly helps but does not remove the freeze, and it is equally possible the
freeze is stochastic and the control was lucky. Recorded as unresolved rather
than resolved in the probe's favour.

Practical consequence: the usable window is roughly three minutes per run,
sometimes less, whether or not the probe is cheap. Experiments needing longer
have to survive a freeze or be redesigned around one.
---
 docs/re/BACKLOG.md              | 15 ++++++--
 docs/re/guest-stalls.md         | 61 +++++++++++++++++++++++++++++++++
 tools/re-capture/wave7_probe.py | 21 +++++++++---
 3 files changed, 90 insertions(+), 7 deletions(-)

diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md
index 67b6edd8..fbe42559 100644
--- a/docs/re/BACKLOG.md
+++ b/docs/re/BACKLOG.md
@@ -428,8 +428,19 @@ search cannot find a *schedule*.
   first word in a 4 MB window with a plausible rate; intermittent counters pass
   that test. `timer_probe` had already solved this (286 candidates, modal rate
   ~17/s) and the lesson was not carried over. **Fixed to a majority vote over the
-  modal-rate cluster (≤32 witnesses, stall only if <half advance), plus explicit
-  `RUN UNVALIDATED` when no witness exists — not yet run.** ⚠️ The "0 stalled
+  modal-rate cluster, plus explicit `RUN UNVALIDATED` when no witness exists.**
+* ✅🔴 **Run 13 (2026-08-24): the vote is better, the threshold was wrong, and
+  freezes persist.** Graded output (11→9→7→4→1→**0** of 31) is coherent, and the
+  drop to 0/31 at t=183 s coincides exactly with the last loss — a real freeze,
+  106 s of nothing after. 🔴 But "<half = stalled" marked the WHOLE run stalled
+  including samples where craft died: **11/31 advancing is healthy**. Cause: the
+  modal cluster was **93/s**, not the ~16.5/s frame rate — bursty subsystem
+  counters. **Fixed: prefer the 8–40/s frame-rate band, stall only when ZERO
+  advance — not yet run.** 🔴 **The cheap probe froze too (t≈183 s)**, so the
+  earlier "0 stalled samples" validation is superseded; with n=1 per arm (control
+  clean 300 s, heavy 27–255 s, cheap 183 s) it is unresolved whether cheap
+  sampling helps or the freeze is stochastic. ⚠️ **Usable window ≈3 min per run
+  regardless** — design experiments to fit or to survive a freeze. ⚠️ The "0 stalled
   samples" that validated the cheap probe came from the unreliable witness and
   should be re-confirmed. Also fixed: bind now retries 3× and aborts rather than
   flying an unattended craft (one run was wasted that way). ❔ Multi-squadron threshold test still not run (zero losses that run).
diff --git a/docs/re/guest-stalls.md b/docs/re/guest-stalls.md
index 7765a75d..4b16df06 100644
--- a/docs/re/guest-stalls.md
+++ b/docs/re/guest-stalls.md
@@ -189,3 +189,64 @@ not from this witness, so they stand. But any run judged **only** by the
 single-word witness is unreliable in both directions, and the "0 stalled samples"
 that validated the cheap probe came from a run where the witness may equally have
 been a lucky pick. That validation should be repeated with the majority rule.
+
+---
+
+# The majority vote is better, the threshold was wrong — and freezes persist (2026-08-24)
+
+Status: ✅ the multi-witness vote produces a usable graded signal; 🔴 the
+"fewer than half" threshold is wrong; 🔴 **freezes still happen with the cheap
+probe**, so the earlier "0 stalled samples" validation is superseded.
+
+## The run
+
+```
+tick witnesses: 185 candidates, modal rate 93/s, using 31
+
+t= 15s  down=1  *** GUEST STALLED (11/31 witnesses moved) ***
+t= 45s  down=1  *** GUEST STALLED (11/31) ***
+t= 90s  down=1  *** GUEST STALLED ( 4/31) ***
+t=138s  down=1  *** GUEST STALLED ( 8/31) ***
+t=168s  down=0  *** GUEST STALLED ( 1/31) ***
+t=183s  down=1  *** GUEST STALLED ( 0/31) ***
+t=198s … t=289s  down=0   *** GUEST STALLED (0/31) ***  (every sample)
+TOTAL down=7  CONFIRMED arrivals=0
+```
+
+## ✅ Graded output is the improvement
+
+Unlike the single-word witness, this says *how many* agree, and the number tells
+a coherent story: 11 → 9 → 7 → 4 → 1 → **0**, with the drop to zero at t = 183 s
+coinciding exactly with the last loss. From t = 183 onward the guest really is
+frozen — 0 of 31 counters advance and nothing is destroyed for 106 s.
+
+## 🔴 The threshold was wrong
+
+Flagging a stall at "fewer than half" marked the entire run stalled, including
+samples where craft were being destroyed. **11 of 31 advancing is a healthy
+guest**, not a stalled one.
+
+The cause is the cluster choice. The modal rate here was **93 /s**, far above the
+~16.5 /s frame rate `timer_probe` measured — those are subsystem counters that
+tick in bursts, so most are idle in any given 15 s window even while the game
+runs. Selecting the modal cluster was convenient rather than principled.
+
+**Fixed:** prefer the cluster whose rate falls in the frame-rate band (8–40 /s),
+falling back to modal only if none exists, and flag a stall only when **zero**
+witnesses advance — the signal the data actually supports. Not yet run.
+
+## 🔴 Freezes are not eliminated by the cheap probe
+
+This run used the cheap probe and still froze, at t ≈ 183 s. So the previous
+iteration's "0 stalled samples" — which came from the unreliable single-word
+witness — cannot stand as validation.
+
+What the evidence now supports: the no-probe control ran 300 s clean (n = 1), the
+heavy probe froze at 27–255 s, and the cheap probe froze at 183 s (n = 1). Cheap
+sampling plausibly helps but **does not remove the freeze**, and with n = 1 on
+each arm it is equally possible the freeze is stochastic and the control was
+lucky. Recorded as unresolved rather than resolved in the probe's favour.
+
+**Consequence:** the usable window per run is roughly 3 minutes, sometimes less,
+whether or not the probe is cheap. Any experiment needing longer than that has to
+survive a freeze or be redesigned around one.
diff --git a/tools/re-capture/wave7_probe.py b/tools/re-capture/wave7_probe.py
index 62e300dd..ca739173 100755
--- a/tools/re-capture/wave7_probe.py
+++ b/tools/re-capture/wave7_probe.py
@@ -91,10 +91,18 @@ def main():
         va, vb = struct.unpack_from('>I', a, k)[0], struct.unpack_from('>I', b, k)[0]
         if va < vb and 5 < (vb - va) / 3.0 < 200:
             cands.append((lo + k, round((vb - va) / 3.0)))
-    modal = collections.Counter(r for _, r in cands).most_common(1)
-    ticks = [o for o, r in cands if modal and r == modal[0][0]][:32]
-    print('tick witnesses: %d candidates, modal rate %s/s, using %d'
-          % (len(cands), modal[0][0] if modal else '-', len(ticks))
+    # The MODAL cluster is not the frame counter. One run picked a modal rate of
+    # 93/s, and only 11 of 31 of those advanced during active combat -- they are
+    # subsystem counters that tick in bursts. timer_probe measured the frame-rate
+    # cluster at ~17/s, matching Canary's 14-19 fps on this box, so prefer a
+    # cluster in that band and fall back to modal only if none exists.
+    rates = collections.Counter(r for _, r in cands)
+    band = [r for r in rates if 8 <= r <= 40]
+    pick = max(band, key=lambda r: rates[r]) if band else (
+        rates.most_common(1)[0][0] if rates else None)
+    ticks = [o for o, r in cands if r == pick][:32]
+    print('tick witnesses: %d candidates, using %d at %s/s (frame-rate band)'
+          % (len(cands), len(ticks), pick)
           if ticks else 'tick witnesses: NONE -- RUN UNVALIDATED')
     last_ticks = [struct.unpack('>I', os.pread(fd, 4, o))[0] for o in ticks]
 
@@ -115,7 +123,10 @@ def main():
         if ticks:
             now = [struct.unpack('>I', os.pread(fd, 4, o))[0] for o in ticks]
             moved = sum(1 for x, y in zip(last_ticks, now) if y > x)
-            if moved * 2 < len(ticks):
+            # "Fewer than half" was too strict: 11 of 31 advanced while craft
+            # were being destroyed. The unambiguous signal in that run was
+            # 0 of 31, which coincided exactly with losses stopping.
+            if moved == 0:
                 st = '  *** GUEST STALLED (%d/%d witnesses moved) ***' % (moved, len(ticks))
                 stalls += 1
             last_ticks = now