re: the stall witness works, and stalls may be caused by the probe
First run with the tick witness. It flagged a stall from t=27 s and every sample after, and the pilot's own telemetry -- which the probe never reads -- agrees: 35 distinct speed values across the whole log and exactly 1 in the last 400 lines, against 236 in the first 400 of a healthy run. All variation is in the first ~50 s. The witness is validated. It earned its keep on that same run. Without it the output reads as "no arrivals across 313 seconds with 300 craft resident" -- clean, quotable and completely worthless, because the game was frozen for 90 % of it. Rule adopted: a run whose witness reports a stall is discarded, and every write-up states the witness result. Flat samples are not evidence unless the witness says the guest was advancing. Stalls are frequent and early. The last three long runs stalled at roughly 255 s, 83 s (after the player died) and 27 s. That makes long observation windows unreliable, and long windows are exactly what the arrival question needs. Leading suspect is the probe itself, and it is recorded because it is uncomfortable rather than despite it. AGENT.md warns that a full memory scan competes with the emulator for every core under lavapipe, and these probes have grown heavier each iteration: wave6 now reads the entire 32 MB entity heap plus about 300 extra preads every 12 seconds while the game renders. If that is the cause, the instrument has been degrading the thing it measures and the earlier "no arrival" results were collected under conditions the game was struggling with. Next is a control that needs no new decoding: run the hunting pilot for 300 s with no probe at all and judge from the pilot log alone. If it does not stall, sampling has to get much cheaper -- narrow the scan to the roster region, sample less often, or reread only the craft bases already located instead of rescanning the heap. The multi-squadron kill-threshold test did not run: the guest froze before anything was destroyed, so there were no losses to threshold.
This commit is contained in:
@@ -391,8 +391,20 @@ search cannot find a *schedule*.
|
||||
objectives, specific squadrons untested). 🔴 **Guest stalled at ~t=255 s** —
|
||||
last 400 pilot lines have 1 distinct speed vs 236 in the first 400 — so trailing
|
||||
flat samples in ANY earlier run may be a frozen guest, not a quiet mission.
|
||||
Probe now carries a frame-rate tick witness printing `*** GUEST STALLED ***`
|
||||
(implemented, **not yet run**).
|
||||
Probe now carries a frame-rate tick witness printing `*** GUEST STALLED ***`.
|
||||
* ✅🔴 **The stall witness WORKS and is now mandatory**
|
||||
([`guest-stalls.md`](guest-stalls.md), 2026-08-24). Validated against the
|
||||
pilot's independent telemetry (35 distinct speeds all run, 1 in the last 400
|
||||
lines; witness flagged t=27 s). It immediately **discarded** a run that would
|
||||
otherwise have read as "no arrivals across 313 s" — worthless, the game was
|
||||
frozen for 90 % of it. **Rule: a stalled run is discarded, and every write-up
|
||||
must state the witness result.** 🔴 Stalls are frequent and early — last three
|
||||
long runs stalled at ~255 s, 83 s and **~27 s**. 🟡 **Leading suspect: the probe
|
||||
itself** — it reads the whole 32 MB heap plus ~300 `pread`s every 12 s while
|
||||
lavapipe renders, and AGENT.md warns a full scan competes for every core.
|
||||
**Next: run the pilot 300 s with NO probe**, judge from its log alone; if it
|
||||
does not stall, the instrument is degrading what it measures and sampling must
|
||||
get much cheaper. ❔ Multi-squadron threshold test did not run (zero losses).
|
||||
⚠️ The ~210 s title movie at boot is the binding constraint on observable game
|
||||
time per turn.
|
||||
Earlier framing:
|
||||
|
||||
73
docs/re/guest-stalls.md
Normal file
73
docs/re/guest-stalls.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# The guest stalls, often — and the probe now detects it
|
||||
|
||||
Status: ✅ the stall witness works and is validated against independent
|
||||
telemetry; ✅ it correctly discarded a run that would otherwise have been written
|
||||
up as evidence; 🔴 stalls are frequent and early enough to threaten the whole
|
||||
wave line of work; 🟡 the probe's own memory scanning is the leading suspect and
|
||||
is untested.
|
||||
|
||||
## ✅ The witness works
|
||||
|
||||
`wave6_probe.py` locates a word that advances at frame rate and prints
|
||||
`*** GUEST STALLED ***` on any sample where it fails to advance. Its first run:
|
||||
|
||||
```
|
||||
tick witness at 0x11c225810, ~26.7/s
|
||||
t= 14s craft=300 deployed=41 ARRIVALS=0 losses=0
|
||||
t= 27s craft=300 deployed=41 ARRIVALS=0 losses=0 *** GUEST STALLED ***
|
||||
... stalled for every remaining sample to t=313s
|
||||
```
|
||||
|
||||
**Validated independently.** The pilot's own telemetry, which the probe never
|
||||
sees, agrees: **35 distinct speed values across the entire log, and exactly 1 in
|
||||
the last 400 lines**, against 236 in the first 400 lines of a healthy run. All
|
||||
the variation is in the first ~50 s. The witness flagged the stall at t = 27 s and
|
||||
the telemetry puts it in the same place.
|
||||
|
||||
## ✅ It earned its keep immediately
|
||||
|
||||
Without it, this run reads as *"no arrivals across 313 seconds with 300 craft
|
||||
resident"* — a clean, quotable, and completely worthless result, since the game
|
||||
was frozen for 90 % of it. That is exactly the failure the previous iteration
|
||||
warned about, caught automatically on the first run after the fix.
|
||||
|
||||
**Operational rule:** a run whose witness reports a stall is discarded, and every
|
||||
future write-up must state the witness result. Flat samples are not evidence
|
||||
unless the witness says the guest was advancing.
|
||||
|
||||
## 🔴 Stalls are frequent and early
|
||||
|
||||
| run | stall onset | usable window |
|
||||
|---|---|---|
|
||||
| keep-out 1400 (previous) | ~255 s | ~255 s |
|
||||
| keep-out 600 | player died 83 s, frozen after | 83 s |
|
||||
| this run | **~27 s** | ~27 s |
|
||||
|
||||
Two of the last three long runs froze, one of them almost immediately. This is
|
||||
consistent with the pre-existing freeze recorded elsewhere in the corpus, but its
|
||||
frequency makes long observation windows unreliable — and long windows are
|
||||
exactly what the arrival question needs.
|
||||
|
||||
## 🟡 Leading suspect: the probe itself
|
||||
|
||||
`docker/agent/AGENT.md` warns that a full memory scan "competes with the emulator
|
||||
for every core under lavapipe", and the probes have grown heavier every
|
||||
iteration. `wave6_probe` now reads the whole 32 MB entity heap **plus** a 1 KB
|
||||
`pread` per craft — roughly 300 extra syscalls — every 12 seconds, while the
|
||||
emulator is trying to render.
|
||||
|
||||
That is a plausible cause of the freezes and it has never been tested. It would
|
||||
also be an uncomfortable one: it would mean the instrument has been degrading the
|
||||
thing it measures, and that the "no arrival" results were collected under
|
||||
conditions the game was struggling with.
|
||||
|
||||
**Test:** run the hunting pilot for 300 s with **no probe at all**, and read the
|
||||
outcome only from the pilot's own log — distinct speed values over time. If it
|
||||
does not stall, the probe is the cause and the sampling has to get much cheaper
|
||||
(narrow the scan to the roster records' own region, sample less often, or read
|
||||
only the craft bases already located rather than rescanning).
|
||||
|
||||
## Not settled this iteration
|
||||
|
||||
The multi-squadron kill-threshold test did not run: the guest froze before the
|
||||
pilot destroyed anything, so there were zero losses and nothing to threshold.
|
||||
Reference in New Issue
Block a user