This repository has been archived on 2026-09-16. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
Syplheed-Reborn/docs/re/guest-stalls.md
Sylpheed RE agent f92d60483a re: confirmed — the probe was causing the guest stalls
The control needed no new decoding: hunting pilot, keep-out 1400, 300 s, and no
memory probe at all, judged only from the pilot's own log. It ran healthy the
whole way -- 211, 257, 250, 241 and 200 distinct speed values in successive 60 s
windows, with no decline in the last one -- against probed runs that stalled at
roughly 27, 83 and 255 s and ended with a single distinct speed across 400 lines.

So the instrument was degrading the thing it measured. AGENT.md warned that a
full scan competes with the emulator for every core under lavapipe, and the
probes grew heavier each iteration until wave6 was reading the entire 32 MB
entity heap plus about 300 extra preads every 12 seconds while the game
rendered.

The control is n=1. The contrast is stark and the mechanism was predicted in
advance, so this is recorded as confirmed enough to act on rather than proven; a
second clean control would settle it.

What it costs is worth stating plainly. Every "no arrival" result so far was
collected while the probe was running, under conditions the game was struggling
with and in some runs after it had frozen. That does not make them wrong -- the
elimination test at t=163 s sits in a window the witness later showed was live --
but none of them is as strong as written, and the arrival question deserves
re-running with cheap sampling before silence is read as evidence.

wave7_probe.py is the fix: one full enumeration, then polling only the hull word
at each known craft base, about 1.2 KB per sample instead of 32 MB, with a full
rescan every 90 s to catch anything genuinely new. Three or four heavy scans per
run instead of twenty-five. Implemented but not yet run, so the claim that it
stops the stalling is untested and the next run must report the witness first.
2026-08-24 16:13:04 +00:00

5.7 KiB
Raw Blame History

The guest stalls, often — and the probe now detects it

Status: ✅ the stall witness works and is validated against independent telemetry; ✅ it correctly discarded a run that would otherwise have been written up as evidence; 🔴 stalls are frequent and early enough to threaten the whole wave line of work; 🟡 the probe's own memory scanning is the leading suspect and is untested.

✅ The witness works

wave6_probe.py locates a word that advances at frame rate and prints *** GUEST STALLED *** on any sample where it fails to advance. Its first run:

tick witness at 0x11c225810, ~26.7/s
t=  14s craft=300 deployed=41  ARRIVALS=0 losses=0
t=  27s craft=300 deployed=41  ARRIVALS=0 losses=0  *** GUEST STALLED ***
   ... stalled for every remaining sample to t=313s

Validated independently. The pilot's own telemetry, which the probe never sees, agrees: 35 distinct speed values across the entire log, and exactly 1 in the last 400 lines, against 236 in the first 400 lines of a healthy run. All the variation is in the first ~50 s. The witness flagged the stall at t = 27 s and the telemetry puts it in the same place.

✅ It earned its keep immediately

Without it, this run reads as "no arrivals across 313 seconds with 300 craft resident" — a clean, quotable, and completely worthless result, since the game was frozen for 90 % of it. That is exactly the failure the previous iteration warned about, caught automatically on the first run after the fix.

Operational rule: a run whose witness reports a stall is discarded, and every future write-up must state the witness result. Flat samples are not evidence unless the witness says the guest was advancing.

🔴 Stalls are frequent and early

run stall onset usable window
keep-out 1400 (previous) ~255 s ~255 s
keep-out 600 player died 83 s, frozen after 83 s
this run ~27 s ~27 s

Two of the last three long runs froze, one of them almost immediately. This is consistent with the pre-existing freeze recorded elsewhere in the corpus, but its frequency makes long observation windows unreliable — and long windows are exactly what the arrival question needs.

🟡 Leading suspect: the probe itself

docker/agent/AGENT.md warns that a full memory scan "competes with the emulator for every core under lavapipe", and the probes have grown heavier every iteration. wave6_probe now reads the whole 32 MB entity heap plus a 1 KB pread per craft — roughly 300 extra syscalls — every 12 seconds, while the emulator is trying to render.

That is a plausible cause of the freezes and it has never been tested. It would also be an uncomfortable one: it would mean the instrument has been degrading the thing it measures, and that the "no arrival" results were collected under conditions the game was struggling with.

Test: run the hunting pilot for 300 s with no probe at all, and read the outcome only from the pilot's own log — distinct speed values over time. If it does not stall, the probe is the cause and the sampling has to get much cheaper (narrow the scan to the roster records' own region, sample less often, or read only the craft bases already located rather than rescanning).

Not settled this iteration

The multi-squadron kill-threshold test did not run: the guest froze before the pilot destroyed anything, so there were zero losses and nothing to threshold.


✅ Confirmed: the probe was causing the stalls (2026-08-24)

The control needed no new decoding — hunting pilot, SYLPH_KEEPOUT=1400, 300 s, and no memory probe at all, judged only from the pilot's own log.

last pilot timestamp: 299.4s
  t=  0- 60s  distinct speeds: 211
  t= 60-120s  distinct speeds: 257
  t=120-180s  distinct speeds: 250
  t=180-240s  distinct speeds: 241
  t=240-300s  distinct speeds: 200

Healthy through the entire run, with no decline in the final window. Against the probed runs, which stalled at ~255 s, ~83 s and ~27 s and ended with 1 distinct speed value across 400 log lines.

The instrument was degrading the thing it measured. AGENT.md warned about exactly this — "the full scan competes with the emulator for every core under lavapipe" — and the probes grew heavier each iteration until wave6_probe was reading the entire 32 MB entity heap plus ~300 extra preads every 12 seconds while the game rendered.

Strength of the claim: the contrast is stark and the mechanism was predicted in advance, but the control is n = 1. It is recorded as confirmed-enough to act on, not as proven; a second clean control would settle it.

What this costs

Every "no arrival" result in mission-arrival-watch.md was collected while the probe was running, i.e. under conditions the game was struggling with, and in some runs after it had frozen outright. That does not make them wrong — the elimination test at t = 163 s happened in a window the witness later showed was live — but it does mean none of them is as strong as written, and the whole arrival question deserves re-running with cheap sampling before any conclusion is drawn from silence.

The fix: sample ~4 orders of magnitude less

The full rescan is nearly all waste. After the first scan the craft bases and their roster links are already known, so a sample only needs the hull word at each known base — 300 × 4 bytes ≈ 1.2 KB, against 32 MB.

wave7_probe.py does that: one full enumeration at the start, then cheap hull polling, with a full rescan only every ~90 s to catch anything genuinely new. That is 3–4 heavy scans per run instead of 25.

Implemented, not yet run — so the claim that it stops the stalling is untested, and the next run must report the witness before anything else.