From 1afe1b7134a64cb850fe402559ca630e2da1032b Mon Sep 17 00:00:00 2001 From: Sylpheed RE agent Date: Mon, 24 Aug 2026 17:30:03 +0000 Subject: [PATCH] re: fix the bind failure; witness search window was too narrow Two harness problems, one fixed and verified, one diagnosed. Bind: three consecutive attempts failed and the session aborted rather than flying an unattended craft, which is last iteration's guard working as intended. The cause is that entities2 self finds the player by motion between two samples, so a craft sitting still at mission start is invisible. The session now holds the throttle for three seconds before each attempt. Verified: the next run bound immediately and produced 11 losses with zero confirmed arrivals, making fourteen runs without one. One increase, 10 to 11, was surfaced and correctly not counted. Witness: the frame-rate-band selection is still unreliable. It found only five candidates in band and reported 0 of 5 moved in samples where craft were being destroyed, and a counter genuinely running at 24/s cannot fail to advance across a 15 s sample -- it would gain about 360. They are bursty counters that moved during the 3 s calibration and then stopped. The cause is mine. When the witness was bolted into wave7_probe the candidate search was narrowed to a 4 MB window, while timer_probe searched the whole 32 MB region and found 286 candidates with a clean cluster near 17/s. The narrowing was never justified, it was just cheaper to write. Now searches the full region once at startup, which is a one-off cost rather than per sample, and warns explicitly when fewer than eight witnesses are available so a weak vote is visible instead of silently trusted. Not yet run. Worth recording: this is the third attempt at this witness, and all three failures share one pattern -- a shortcut in selecting the witness rather than in the voting logic. Each was caught only because the flagged samples contradicted the loss events in the same output. That internal contradiction is the real check and should survive into any future version. --- docs/re/BACKLOG.md | 15 +++++++++- docs/re/guest-stalls.md | 46 +++++++++++++++++++++++++++++++ tools/re-capture/wave7_probe.py | 16 +++++++++-- tools/re-capture/wave7_session.sh | 7 +++++ 4 files changed, 81 insertions(+), 3 deletions(-) diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index fbe4255..df6e261 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -436,7 +436,20 @@ search cannot find a *schedule*. including samples where craft died: **11/31 advancing is healthy**. Cause: the modal cluster was **93/s**, not the ~16.5/s frame rate — bursty subsystem counters. **Fixed: prefer the 8–40/s frame-rate band, stall only when ZERO - advance — not yet run.** 🔴 **The cheap probe froze too (t≈183 s)**, so the + advance.** +* 🔴 **Witness attempt 3 still unreliable (2026-08-24)**: only **5** candidates + in band, and `0/5 moved` in samples where craft died — a real 24/s counter + cannot miss 15 s. Cause was mine: the candidate search had been narrowed to + **4 MB** when the witness was bolted in, while `timer_probe` searched the full + **32 MB** and found 286 with a clean ~17/s cluster. **Fixed: full-region search + once at startup + warn when <8 witnesses — not yet run.** All three witness + failures share one pattern: a shortcut in *selecting* the witness, each caught + only by the flagged samples contradicting losses in the same output. +* ✅ **Bind failure fixed + abort verified**: `entities2 self` finds the player by + MOTION, so a stationary craft at mission start is invisible; the session now + holds throttle 3 s before each attempt. One run correctly **aborted** after 3 + failures rather than flying unattended; the next bound immediately (11 losses, + 0 confirmed arrivals — 14 runs). 🔴 **The cheap probe froze too (t≈183 s)**, so the earlier "0 stalled samples" validation is superseded; with n=1 per arm (control clean 300 s, heavy 27–255 s, cheap 183 s) it is unresolved whether cheap sampling helps or the freeze is stochastic. ⚠️ **Usable window ≈3 min per run diff --git a/docs/re/guest-stalls.md b/docs/re/guest-stalls.md index 4b16df0..be985c2 100644 --- a/docs/re/guest-stalls.md +++ b/docs/re/guest-stalls.md @@ -250,3 +250,49 @@ lucky. Recorded as unresolved rather than resolved in the probe's favour. **Consequence:** the usable window per run is roughly 3 minutes, sometimes less, whether or not the probe is cheap. Any experiment needing longer than that has to survive a freeze or be redesigned around one. + +--- + +# Witness attempt 3: the search window was too narrow (2026-08-24) + +Status: 🔴 still unreliable; ✅ cause identified; ✅ bind failure separately +diagnosed and fixed. + +The frame-rate-band selection ran, and still flags stalls in samples where craft +are being destroyed: + +``` +tick witnesses: 96 candidates, using 5 at 24/s (frame-rate band) +t= 30s down=1 *** GUEST STALLED (0/5 witnesses moved) *** +t=156s down=1 *** GUEST STALLED (0/5) *** +t=236s down=1 *** GUEST STALLED (0/5) *** +TOTAL down=11 stalled samples=7 +``` + +**Five witnesses is not a vote.** And a counter genuinely running at 24 /s cannot +fail to advance across a 15 s sample — it would gain ~360. So these are bursty +counters again: they moved during the 3 s calibration and then stopped. + +The cause is mine: when the witness was bolted into `wave7_probe` the candidate +search was narrowed to a **4 MB** window, while `timer_probe` — which found a +clean 17 /s cluster — searched the **whole 32 MB** region and found 286 +candidates. Narrowing was never justified; it was just cheaper to write. + +**Fixed:** search the full region once at startup (a one-off cost, not per +sample), and warn explicitly when fewer than 8 witnesses are available so a weak +vote is visible rather than silently trusted. Not yet run. + +This is the third attempt at this witness. The pattern in all three failures is +the same — a shortcut in *selecting* the witness, not in the voting logic — and +each was caught only because the flagged samples contradicted the loss events in +the same output. That internal contradiction is the real check and is worth +keeping in any future version. + +## ✅ Separately: the bind failure is fixed + +Three consecutive bind attempts failed and the session **aborted rather than +flying unattended**, which is the behaviour added last iteration working as +intended. The cause: `entities2 self` finds the player by *motion* between two +samples, so a craft sitting still at mission start is invisible. The session now +holds the throttle for 3 s before each attempt. Verified — the next run bound +immediately and produced 11 losses. diff --git a/tools/re-capture/wave7_probe.py b/tools/re-capture/wave7_probe.py index ca73917..14f5489 100755 --- a/tools/re-capture/wave7_probe.py +++ b/tools/re-capture/wave7_probe.py @@ -84,8 +84,18 @@ def main(): # timer_probe's approach instead -- collect every candidate, keep the modal # rate cluster, and call a stall only when a MAJORITY of that cluster fails # to advance. - lo = gmem.va_to_off(0xBC000000) - a = os.pread(fd, 1 << 22, lo); time.sleep(3.0); b = os.pread(fd, 1 << 22, lo) + # 4 MB was too narrow: it yielded 5 candidates at a nominal 24/s that then + # failed to advance in 15 s windows where craft were being destroyed, i.e. + # they are bursty, not frame counters. timer_probe searched the WHOLE 32 MB + # region and found 286 with a clean cluster at ~17/s. Pay the one-off cost. + lo, hiw = gmem.va_to_off(0xBC000000), gmem.va_to_off(0xBE000000) + span = hiw - lo + def grab(): + out, pos = bytearray(), lo + while pos < hiw: + n = min(1 << 24, hiw - pos); out += os.pread(fd, n, pos); pos += n + return bytes(out) + a = grab(); time.sleep(3.0); b = grab() cands = [] for k in range(0, min(len(a), len(b)) - 3, 4): va, vb = struct.unpack_from('>I', a, k)[0], struct.unpack_from('>I', b, k)[0] @@ -101,6 +111,8 @@ def main(): pick = max(band, key=lambda r: rates[r]) if band else ( rates.most_common(1)[0][0] if rates else None) ticks = [o for o, r in cands if r == pick][:32] + if len(ticks) < 8: + print('WARNING: only %d witnesses; stall detection is weak' % len(ticks)) print('tick witnesses: %d candidates, using %d at %s/s (frame-rate band)' % (len(cands), len(ticks), pick) if ticks else 'tick witnesses: NONE -- RUN UNVALIDATED') diff --git a/tools/re-capture/wave7_session.sh b/tools/re-capture/wave7_session.sh index 6fb4003..ec495a6 100755 --- a/tools/re-capture/wave7_session.sh +++ b/tools/re-capture/wave7_session.sh @@ -9,8 +9,15 @@ CFG=/tmp/nav-live.json # The bind is intermittent and a failed bind means no pilot, no kills and a # completely uninformative run. Retry before giving up, and abort if it never # takes rather than silently flying an unattended craft. +# entities2 self finds the player by MOTION between two samples, so a craft that +# is sitting still at mission start is invisible and the bind fails -- three +# times in a row on the run that added this retry. Nudge the throttle first so +# there is something to see, then bind. BOUND=0 for try in 1 2 3; do + python3 "$SD/pad.py" set "rt=1" >/dev/null 2>&1 || true + sleep 3 + python3 "$SD/pad.py" clear >/dev/null 2>&1 || true if python3 "$SD/entities2.py" self 0x130 "$CFG" >/dev/null 2>&1; then BOUND=1; break; fi echo "--- bind attempt $try failed, retrying"; sleep 5 done