diff --git a/docs/re/BACKLOG.md b/docs/re/BACKLOG.md index 1e317a4..c8ed10e 100644 --- a/docs/re/BACKLOG.md +++ b/docs/re/BACKLOG.md @@ -391,8 +391,20 @@ search cannot find a *schedule*. objectives, specific squadrons untested). 🔴 **Guest stalled at ~t=255 s** — last 400 pilot lines have 1 distinct speed vs 236 in the first 400 — so trailing flat samples in ANY earlier run may be a frozen guest, not a quiet mission. - Probe now carries a frame-rate tick witness printing `*** GUEST STALLED ***` - (implemented, **not yet run**). + Probe now carries a frame-rate tick witness printing `*** GUEST STALLED ***`. +* ✅🔴 **The stall witness WORKS and is now mandatory** + ([`guest-stalls.md`](guest-stalls.md), 2026-08-24). Validated against the + pilot's independent telemetry (35 distinct speeds all run, 1 in the last 400 + lines; witness flagged t=27 s). It immediately **discarded** a run that would + otherwise have read as "no arrivals across 313 s" — worthless, the game was + frozen for 90 % of it. **Rule: a stalled run is discarded, and every write-up + must state the witness result.** 🔴 Stalls are frequent and early — last three + long runs stalled at ~255 s, 83 s and **~27 s**. 🟡 **Leading suspect: the probe + itself** — it reads the whole 32 MB heap plus ~300 `pread`s every 12 s while + lavapipe renders, and AGENT.md warns a full scan competes for every core. + **Next: run the pilot 300 s with NO probe**, judge from its log alone; if it + does not stall, the instrument is degrading what it measures and sampling must + get much cheaper. ❔ Multi-squadron threshold test did not run (zero losses). ⚠️ The ~210 s title movie at boot is the binding constraint on observable game time per turn. Earlier framing: diff --git a/docs/re/guest-stalls.md b/docs/re/guest-stalls.md new file mode 100644 index 0000000..ccbb6be --- /dev/null +++ b/docs/re/guest-stalls.md @@ -0,0 +1,73 @@ +# The guest stalls, often — and the probe now detects it + +Status: ✅ the stall witness works and is validated against independent +telemetry; ✅ it correctly discarded a run that would otherwise have been written +up as evidence; 🔴 stalls are frequent and early enough to threaten the whole +wave line of work; 🟡 the probe's own memory scanning is the leading suspect and +is untested. + +## ✅ The witness works + +`wave6_probe.py` locates a word that advances at frame rate and prints +`*** GUEST STALLED ***` on any sample where it fails to advance. Its first run: + +``` +tick witness at 0x11c225810, ~26.7/s +t= 14s craft=300 deployed=41 ARRIVALS=0 losses=0 +t= 27s craft=300 deployed=41 ARRIVALS=0 losses=0 *** GUEST STALLED *** + ... stalled for every remaining sample to t=313s +``` + +**Validated independently.** The pilot's own telemetry, which the probe never +sees, agrees: **35 distinct speed values across the entire log, and exactly 1 in +the last 400 lines**, against 236 in the first 400 lines of a healthy run. All +the variation is in the first ~50 s. The witness flagged the stall at t = 27 s and +the telemetry puts it in the same place. + +## ✅ It earned its keep immediately + +Without it, this run reads as *"no arrivals across 313 seconds with 300 craft +resident"* — a clean, quotable, and completely worthless result, since the game +was frozen for 90 % of it. That is exactly the failure the previous iteration +warned about, caught automatically on the first run after the fix. + +**Operational rule:** a run whose witness reports a stall is discarded, and every +future write-up must state the witness result. Flat samples are not evidence +unless the witness says the guest was advancing. + +## 🔴 Stalls are frequent and early + +| run | stall onset | usable window | +|---|---|---| +| keep-out 1400 (previous) | ~255 s | ~255 s | +| keep-out 600 | player died 83 s, frozen after | 83 s | +| this run | **~27 s** | ~27 s | + +Two of the last three long runs froze, one of them almost immediately. This is +consistent with the pre-existing freeze recorded elsewhere in the corpus, but its +frequency makes long observation windows unreliable — and long windows are +exactly what the arrival question needs. + +## 🟡 Leading suspect: the probe itself + +`docker/agent/AGENT.md` warns that a full memory scan "competes with the emulator +for every core under lavapipe", and the probes have grown heavier every +iteration. `wave6_probe` now reads the whole 32 MB entity heap **plus** a 1 KB +`pread` per craft — roughly 300 extra syscalls — every 12 seconds, while the +emulator is trying to render. + +That is a plausible cause of the freezes and it has never been tested. It would +also be an uncomfortable one: it would mean the instrument has been degrading the +thing it measures, and that the "no arrival" results were collected under +conditions the game was struggling with. + +**Test:** run the hunting pilot for 300 s with **no probe at all**, and read the +outcome only from the pilot's own log — distinct speed values over time. If it +does not stall, the probe is the cause and the sampling has to get much cheaper +(narrow the scan to the roster records' own region, sample less often, or read +only the craft bases already located rather than rescanning). + +## Not settled this iteration + +The multi-squadron kill-threshold test did not run: the guest froze before the +pilot destroyed anything, so there were zero losses and nothing to threshold.