diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index fbe42559..df6e2610 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -436,7 +436,20 @@ search cannot find a *schedule*. including samples where craft died: **11/31 advancing is healthy**. Cause: the modal cluster was **93/s**, not the ~16.5/s frame rate — bursty subsystem counters. **Fixed: prefer the 8–40/s frame-rate band, stall only when ZERO - advance — not yet run.** 🔴 **The cheap probe froze too (t≈183 s)**, so the + advance.** +* 🔴 **Witness attempt 3 still unreliable (2026-08-24)**: only **5** candidates + in band, and `0/5 moved` in samples where craft died — a real 24/s counter + cannot miss 15 s. Cause was mine: the candidate search had been narrowed to + **4 MB** when the witness was bolted in, while `timer_probe` searched the full + **32 MB** and found 286 with a clean ~17/s cluster. **Fixed: full-region search + once at startup + warn when <8 witnesses — not yet run.** All three witness + failures share one pattern: a shortcut in *selecting* the witness, each caught + only by the flagged samples contradicting losses in the same output. +* ✅ **Bind failure fixed + abort verified**: `entities2 self` finds the player by + MOTION, so a stationary craft at mission start is invisible; the session now + holds throttle 3 s before each attempt. One run correctly **aborted** after 3 + failures rather than flying unattended; the next bound immediately (11 losses, + 0 confirmed arrivals — 14 runs). 🔴 **The cheap probe froze too (t≈183 s)**, so the earlier "0 stalled samples" validation is superseded; with n=1 per arm (control clean 300 s, heavy 27–255 s, cheap 183 s) it is unresolved whether cheap sampling helps or the freeze is stochastic. ⚠️ **Usable window ≈3 min per run diff --git a/docs/re/guest-stalls.md b/docs/re/guest-stalls.md index 4b16df06..be985c28 100644 --- a/docs/re/guest-stalls.md +++ b/docs/re/guest-stalls.md @@ -250,3 +250,49 @@ lucky. Recorded as unresolved rather than resolved in the probe's favour. **Consequence:** the usable window per run is roughly 3 minutes, sometimes less, whether or not the probe is cheap. Any experiment needing longer than that has to survive a freeze or be redesigned around one. + +--- + +# Witness attempt 3: the search window was too narrow (2026-08-24) + +Status: 🔴 still unreliable; ✅ cause identified; ✅ bind failure separately +diagnosed and fixed. + +The frame-rate-band selection ran, and still flags stalls in samples where craft +are being destroyed: + +``` +tick witnesses: 96 candidates, using 5 at 24/s (frame-rate band) +t= 30s down=1 *** GUEST STALLED (0/5 witnesses moved) *** +t=156s down=1 *** GUEST STALLED (0/5) *** +t=236s down=1 *** GUEST STALLED (0/5) *** +TOTAL down=11 stalled samples=7 +``` + +**Five witnesses is not a vote.** And a counter genuinely running at 24 /s cannot +fail to advance across a 15 s sample — it would gain ~360. So these are bursty +counters again: they moved during the 3 s calibration and then stopped. + +The cause is mine: when the witness was bolted into `wave7_probe` the candidate +search was narrowed to a **4 MB** window, while `timer_probe` — which found a +clean 17 /s cluster — searched the **whole 32 MB** region and found 286 +candidates. Narrowing was never justified; it was just cheaper to write. + +**Fixed:** search the full region once at startup (a one-off cost, not per +sample), and warn explicitly when fewer than 8 witnesses are available so a weak +vote is visible rather than silently trusted. Not yet run. + +This is the third attempt at this witness. The pattern in all three failures is +the same — a shortcut in *selecting* the witness, not in the voting logic — and +each was caught only because the flagged samples contradicted the loss events in +the same output. That internal contradiction is the real check and is worth +keeping in any future version. + +## ✅ Separately: the bind failure is fixed + +Three consecutive bind attempts failed and the session **aborted rather than +flying unattended**, which is the behaviour added last iteration working as +intended. The cause: `entities2 self` finds the player by *motion* between two +samples, so a craft sitting still at mission start is invisible. The session now +holds the throttle for 3 s before each attempt. Verified — the next run bound +immediately and produced 11 losses. diff --git a/tools/re-capture/wave7_probe.py b/tools/re-capture/wave7_probe.py index ca739173..14f54894 100755 --- a/tools/re-capture/wave7_probe.py +++ b/tools/re-capture/wave7_probe.py @@ -84,8 +84,18 @@ def main(): # timer_probe's approach instead -- collect every candidate, keep the modal # rate cluster, and call a stall only when a MAJORITY of that cluster fails # to advance. - lo = gmem.va_to_off(0xBC000000) - a = os.pread(fd, 1 << 22, lo); time.sleep(3.0); b = os.pread(fd, 1 << 22, lo) + # 4 MB was too narrow: it yielded 5 candidates at a nominal 24/s that then + # failed to advance in 15 s windows where craft were being destroyed, i.e. + # they are bursty, not frame counters. timer_probe searched the WHOLE 32 MB + # region and found 286 with a clean cluster at ~17/s. Pay the one-off cost. + lo, hiw = gmem.va_to_off(0xBC000000), gmem.va_to_off(0xBE000000) + span = hiw - lo + def grab(): + out, pos = bytearray(), lo + while pos < hiw: + n = min(1 << 24, hiw - pos); out += os.pread(fd, n, pos); pos += n + return bytes(out) + a = grab(); time.sleep(3.0); b = grab() cands = [] for k in range(0, min(len(a), len(b)) - 3, 4): va, vb = struct.unpack_from('>I', a, k)[0], struct.unpack_from('>I', b, k)[0] @@ -101,6 +111,8 @@ def main(): pick = max(band, key=lambda r: rates[r]) if band else ( rates.most_common(1)[0][0] if rates else None) ticks = [o for o, r in cands if r == pick][:32] + if len(ticks) < 8: + print('WARNING: only %d witnesses; stall detection is weak' % len(ticks)) print('tick witnesses: %d candidates, using %d at %s/s (frame-rate band)' % (len(cands), len(ticks), pick) if ticks else 'tick witnesses: NONE -- RUN UNVALIDATED') diff --git a/tools/re-capture/wave7_session.sh b/tools/re-capture/wave7_session.sh index 6fb40030..ec495a6d 100755 --- a/tools/re-capture/wave7_session.sh +++ b/tools/re-capture/wave7_session.sh @@ -9,8 +9,15 @@ CFG=/tmp/nav-live.json # The bind is intermittent and a failed bind means no pilot, no kills and a # completely uninformative run. Retry before giving up, and abort if it never # takes rather than silently flying an unattended craft. +# entities2 self finds the player by MOTION between two samples, so a craft that +# is sitting still at mission start is invisible and the bind fails -- three +# times in a row on the run that added this retry. Nudge the throttle first so +# there is something to see, then bind. BOUND=0 for try in 1 2 3; do + python3 "$SD/pad.py" set "rt=1" >/dev/null 2>&1 || true + sleep 3 + python3 "$SD/pad.py" clear >/dev/null 2>&1 || true if python3 "$SD/entities2.py" self 0x130 "$CFG" >/dev/null 2>&1; then BOUND=1; break; fi echo "--- bind attempt $try failed, retrying"; sleep 5 done