015fb7d21e
Run 12, cheap probe with a bound pilot: 16 losses over 290 s and zero confirmed arrivals, making twelve runs without one. The persistence rule earned its place immediately -- an increase of 13 to 15 was surfaced and correctly not counted, since it does not start from zero. Under the previous rule it would have been invisible, and a similar flicker straddling zero was nearly written up last iteration as the first arrival. The stall witness, on the other hand, is unreliable. Thirteen samples were flagged GUEST STALLED while recording losses in those same samples, and a frozen guest cannot destroy craft, so they are false positives and the run was healthy. The cause is the selection rule: it took the first word in a 4 MB window whose rate fell in a plausible band, and plenty of counters advance intermittently without saying anything about whether frames are being rendered. timer_probe had already solved this properly -- 286 candidates, a rate histogram with a dominant cluster near 17/s -- and that lesson was not carried over when the witness was bolted onto the probe. Now fixed to a majority vote: collect every candidate, keep the modal-rate cluster, sample up to 32 of them, and report a stall only when fewer than half advance. It also prints RUN UNVALIDATED when no witness is found, because an earlier run printed "stalled samples=0" alongside "tick witness: NONE", and a witness that does not exist cannot report zero stalls. Not yet run. Consequence worth flagging: the "0 stalled samples" that validated the cheap probe last iteration came from this same unreliable witness and should be re-confirmed under the majority rule. The pilot-log speed analysis that established the stalls in the first place is unaffected. Also fixed: the entity bind now retries three times and aborts if it never takes, instead of silently flying an unattended craft -- one run was wasted that way this iteration, producing no kills and no information.
192 lines
8.4 KiB
Markdown
192 lines
8.4 KiB
Markdown
# The guest stalls, often — and the probe now detects it
|
||
|
||
Status: ✅ the stall witness works and is validated against independent
|
||
telemetry; ✅ it correctly discarded a run that would otherwise have been written
|
||
up as evidence; 🔴 stalls are frequent and early enough to threaten the whole
|
||
wave line of work; 🟡 the probe's own memory scanning is the leading suspect and
|
||
is untested.
|
||
|
||
## ✅ The witness works
|
||
|
||
`wave6_probe.py` locates a word that advances at frame rate and prints
|
||
`*** GUEST STALLED ***` on any sample where it fails to advance. Its first run:
|
||
|
||
```
|
||
tick witness at 0x11c225810, ~26.7/s
|
||
t= 14s craft=300 deployed=41 ARRIVALS=0 losses=0
|
||
t= 27s craft=300 deployed=41 ARRIVALS=0 losses=0 *** GUEST STALLED ***
|
||
... stalled for every remaining sample to t=313s
|
||
```
|
||
|
||
**Validated independently.** The pilot's own telemetry, which the probe never
|
||
sees, agrees: **35 distinct speed values across the entire log, and exactly 1 in
|
||
the last 400 lines**, against 236 in the first 400 lines of a healthy run. All
|
||
the variation is in the first ~50 s. The witness flagged the stall at t = 27 s and
|
||
the telemetry puts it in the same place.
|
||
|
||
## ✅ It earned its keep immediately
|
||
|
||
Without it, this run reads as *"no arrivals across 313 seconds with 300 craft
|
||
resident"* — a clean, quotable, and completely worthless result, since the game
|
||
was frozen for 90 % of it. That is exactly the failure the previous iteration
|
||
warned about, caught automatically on the first run after the fix.
|
||
|
||
**Operational rule:** a run whose witness reports a stall is discarded, and every
|
||
future write-up must state the witness result. Flat samples are not evidence
|
||
unless the witness says the guest was advancing.
|
||
|
||
## 🔴 Stalls are frequent and early
|
||
|
||
| run | stall onset | usable window |
|
||
|---|---|---|
|
||
| keep-out 1400 (previous) | ~255 s | ~255 s |
|
||
| keep-out 600 | player died 83 s, frozen after | 83 s |
|
||
| this run | **~27 s** | ~27 s |
|
||
|
||
Two of the last three long runs froze, one of them almost immediately. This is
|
||
consistent with the pre-existing freeze recorded elsewhere in the corpus, but its
|
||
frequency makes long observation windows unreliable — and long windows are
|
||
exactly what the arrival question needs.
|
||
|
||
## 🟡 Leading suspect: the probe itself
|
||
|
||
`docker/agent/AGENT.md` warns that a full memory scan "competes with the emulator
|
||
for every core under lavapipe", and the probes have grown heavier every
|
||
iteration. `wave6_probe` now reads the whole 32 MB entity heap **plus** a 1 KB
|
||
`pread` per craft — roughly 300 extra syscalls — every 12 seconds, while the
|
||
emulator is trying to render.
|
||
|
||
That is a plausible cause of the freezes and it has never been tested. It would
|
||
also be an uncomfortable one: it would mean the instrument has been degrading the
|
||
thing it measures, and that the "no arrival" results were collected under
|
||
conditions the game was struggling with.
|
||
|
||
**Test:** run the hunting pilot for 300 s with **no probe at all**, and read the
|
||
outcome only from the pilot's own log — distinct speed values over time. If it
|
||
does not stall, the probe is the cause and the sampling has to get much cheaper
|
||
(narrow the scan to the roster records' own region, sample less often, or read
|
||
only the craft bases already located rather than rescanning).
|
||
|
||
## Not settled this iteration
|
||
|
||
The multi-squadron kill-threshold test did not run: the guest froze before the
|
||
pilot destroyed anything, so there were zero losses and nothing to threshold.
|
||
|
||
---
|
||
|
||
# ✅ Confirmed: the probe was causing the stalls (2026-08-24)
|
||
|
||
The control needed no new decoding — hunting pilot, `SYLPH_KEEPOUT=1400`, 300 s,
|
||
and **no memory probe at all**, judged only from the pilot's own log.
|
||
|
||
```
|
||
last pilot timestamp: 299.4s
|
||
t= 0- 60s distinct speeds: 211
|
||
t= 60-120s distinct speeds: 257
|
||
t=120-180s distinct speeds: 250
|
||
t=180-240s distinct speeds: 241
|
||
t=240-300s distinct speeds: 200
|
||
```
|
||
|
||
Healthy through the entire run, with no decline in the final window. Against the
|
||
probed runs, which stalled at ~255 s, ~83 s and ~27 s and ended with **1** distinct
|
||
speed value across 400 log lines.
|
||
|
||
**The instrument was degrading the thing it measured.** `AGENT.md` warned about
|
||
exactly this — "the full scan competes with the emulator for every core under
|
||
lavapipe" — and the probes grew heavier each iteration until `wave6_probe` was
|
||
reading the entire 32 MB entity heap plus ~300 extra `pread`s every 12 seconds
|
||
while the game rendered.
|
||
|
||
**Strength of the claim:** the contrast is stark and the mechanism was predicted
|
||
in advance, but the control is **n = 1**. It is recorded as confirmed-enough to
|
||
act on, not as proven; a second clean control would settle it.
|
||
|
||
## What this costs
|
||
|
||
Every "no arrival" result in
|
||
[mission-arrival-watch.md](mission-arrival-watch.md) was collected while the
|
||
probe was running, i.e. under conditions the game was struggling with, and in
|
||
some runs after it had frozen outright. That does not make them wrong — the
|
||
elimination test at t = 163 s happened in a window the witness later showed was
|
||
live — but it does mean **none of them is as strong as written**, and the whole
|
||
arrival question deserves re-running with cheap sampling before any conclusion
|
||
is drawn from silence.
|
||
|
||
## The fix: sample ~4 orders of magnitude less
|
||
|
||
The full rescan is nearly all waste. After the first scan the craft bases and
|
||
their roster links are already known, so a sample only needs the **hull word at
|
||
each known base** — 300 × 4 bytes ≈ 1.2 KB, against 32 MB.
|
||
|
||
`wave7_probe.py` does that: one full enumeration at the start, then cheap hull
|
||
polling, with a full rescan only every ~90 s to catch anything genuinely new.
|
||
That is 3–4 heavy scans per run instead of 25.
|
||
|
||
Implemented, **not yet run** — so the claim that it stops the stalling is
|
||
untested, and the next run must report the witness before anything else.
|
||
|
||
---
|
||
|
||
# ✅ The cheap probe does not stall the guest (2026-08-24)
|
||
|
||
First run of `wave7_probe.py` — one enumeration, then hull polling, full rescan
|
||
every 90 s:
|
||
|
||
```
|
||
TOTAL candidate-up=1 down=19 stalled samples=0
|
||
```
|
||
|
||
**Zero stalled samples across the whole run**, against three consecutive heavy-probe
|
||
runs that stalled at ~27, ~83 and ~255 s. The fix works, and the previous
|
||
iteration's confirmation is now supported from the other direction as well.
|
||
|
||
The guest is also visibly healthier: **19 losses against 8** in the heavy-probe
|
||
run of comparable length. Starving the emulator was suppressing the very activity
|
||
the probe existed to watch.
|
||
|
||
---
|
||
|
||
# The single-word witness gives false positives (2026-08-24)
|
||
|
||
Status: 🔴 the witness as first written is unreliable; ✅ the contradiction that
|
||
exposes it is in its own output; ✅ replaced with a majority vote, not yet run.
|
||
|
||
A run with the cheap probe and a bound pilot produced this:
|
||
|
||
```
|
||
t= 30s deployed=41 up=0 down=1 ... *** GUEST STALLED ***
|
||
t= 75s deployed=41 up=0 down=1 ... *** GUEST STALLED ***
|
||
t=151s deployed=40 up=0 down=2 ... *** GUEST STALLED ***
|
||
TOTAL down=16 stalled samples=13
|
||
```
|
||
|
||
**Thirteen samples flagged as stalled while recording losses in the same
|
||
samples.** A frozen guest cannot destroy craft, so those flags are false
|
||
positives, and the run was in fact healthy — 16 losses over 290 s.
|
||
|
||
The cause is the selection rule: it took **the first word** in a 4 MB window
|
||
whose rate fell in a plausible band. Plenty of counters advance intermittently —
|
||
a buffer index, a per-event tally — and one of those satisfies the test while
|
||
saying nothing about whether frames are being rendered.
|
||
|
||
`timer_probe.py` had already done this correctly and the lesson was not carried
|
||
over: it collected **286** linearly-advancing words and looked at the *rate
|
||
histogram*, where a dominant cluster sat at ~17 /s.
|
||
|
||
**Fixed:** `wave7_probe.py` now collects every candidate, keeps the modal-rate
|
||
cluster, samples up to 32 of them, and reports a stall only when **fewer than
|
||
half** advance. It also says `RUN UNVALIDATED` when no witness is found at all,
|
||
because an earlier run printed `stalled samples=0` with `tick witness: NONE` —
|
||
a witness that does not exist cannot report zero stalls.
|
||
|
||
Not yet run, so the majority rule is unverified.
|
||
|
||
## What this costs
|
||
|
||
The stall counts quoted for earlier runs came from the pilot-log speed analysis,
|
||
not from this witness, so they stand. But any run judged **only** by the
|
||
single-word witness is unreliable in both directions, and the "0 stalled samples"
|
||
that validated the cheap probe came from a run where the witness may equally have
|
||
been a lucky pick. That validation should be repeated with the majority rule.
|