Sylpheed/docs/re/guest-stalls.md

# The guest stalls, often — and the probe now detects it

Status: ✅ the stall witness works and is validated against independent
telemetry; ✅ it correctly discarded a run that would otherwise have been written
up as evidence; 🔴 stalls are frequent and early enough to threaten the whole
wave line of work; 🟡 the probe's own memory scanning is the leading suspect and
is untested.

## ✅ The witness works

`wave6_probe.py` locates a word that advances at frame rate and prints
`*** GUEST STALLED ***` on any sample where it fails to advance. Its first run:

```
tick witness at 0x11c225810, ~26.7/s
t=  14s craft=300 deployed=41  ARRIVALS=0 losses=0
t=  27s craft=300 deployed=41  ARRIVALS=0 losses=0  *** GUEST STALLED ***
   ... stalled for every remaining sample to t=313s
```

**Validated independently.** The pilot's own telemetry, which the probe never
sees, agrees: **35 distinct speed values across the entire log, and exactly 1 in
the last 400 lines**, against 236 in the first 400 lines of a healthy run. All
the variation is in the first ~50 s. The witness flagged the stall at t = 27 s and
the telemetry puts it in the same place.

## ✅ It earned its keep immediately

Without it, this run reads as *"no arrivals across 313 seconds with 300 craft
resident"* — a clean, quotable, and completely worthless result, since the game
was frozen for 90 % of it. That is exactly the failure the previous iteration
warned about, caught automatically on the first run after the fix.

**Operational rule:** a run whose witness reports a stall is discarded, and every
future write-up must state the witness result. Flat samples are not evidence
unless the witness says the guest was advancing.

## 🔴 Stalls are frequent and early

| run | stall onset | usable window |
|---|---|---|
| keep-out 1400 (previous) | ~255 s | ~255 s |
| keep-out 600 | player died 83 s, frozen after | 83 s |
| this run | **~27 s** | ~27 s |

Two of the last three long runs froze, one of them almost immediately. This is
consistent with the pre-existing freeze recorded elsewhere in the corpus, but its
frequency makes long observation windows unreliable — and long windows are
exactly what the arrival question needs.

## 🟡 Leading suspect: the probe itself

`docker/agent/AGENT.md` warns that a full memory scan "competes with the emulator
for every core under lavapipe", and the probes have grown heavier every
iteration. `wave6_probe` now reads the whole 32 MB entity heap **plus** a 1 KB
`pread` per craft — roughly 300 extra syscalls — every 12 seconds, while the
emulator is trying to render.

That is a plausible cause of the freezes and it has never been tested. It would
also be an uncomfortable one: it would mean the instrument has been degrading the
thing it measures, and that the "no arrival" results were collected under
conditions the game was struggling with.

**Test:** run the hunting pilot for 300 s with **no probe at all**, and read the
outcome only from the pilot's own log — distinct speed values over time. If it
does not stall, the probe is the cause and the sampling has to get much cheaper
(narrow the scan to the roster records' own region, sample less often, or read
only the craft bases already located rather than rescanning).

## Not settled this iteration

The multi-squadron kill-threshold test did not run: the guest froze before the
pilot destroyed anything, so there were zero losses and nothing to threshold.

---

# ✅ Confirmed: the probe was causing the stalls (2026-08-24)

The control needed no new decoding — hunting pilot, `SYLPH_KEEPOUT=1400`, 300 s,
and **no memory probe at all**, judged only from the pilot's own log.

```
last pilot timestamp: 299.4s
  t=  0- 60s  distinct speeds: 211
  t= 60-120s  distinct speeds: 257
  t=120-180s  distinct speeds: 250
  t=180-240s  distinct speeds: 241
  t=240-300s  distinct speeds: 200
```

Healthy through the entire run, with no decline in the final window. Against the
probed runs, which stalled at ~255 s, ~83 s and ~27 s and ended with **1** distinct
speed value across 400 log lines.

**The instrument was degrading the thing it measured.** `AGENT.md` warned about
exactly this — "the full scan competes with the emulator for every core under
lavapipe" — and the probes grew heavier each iteration until `wave6_probe` was
reading the entire 32 MB entity heap plus ~300 extra `pread`s every 12 seconds
while the game rendered.

**Strength of the claim:** the contrast is stark and the mechanism was predicted
in advance, but the control is **n = 1**. It is recorded as confirmed-enough to
act on, not as proven; a second clean control would settle it.

## What this costs

Every "no arrival" result in
[mission-arrival-watch.md](mission-arrival-watch.md) was collected while the
probe was running, i.e. under conditions the game was struggling with, and in
some runs after it had frozen outright. That does not make them wrong — the
elimination test at t = 163 s happened in a window the witness later showed was
live — but it does mean **none of them is as strong as written**, and the whole
arrival question deserves re-running with cheap sampling before any conclusion
is drawn from silence.

## The fix: sample ~4 orders of magnitude less

The full rescan is nearly all waste. After the first scan the craft bases and
their roster links are already known, so a sample only needs the **hull word at
each known base** — 300 × 4 bytes ≈ 1.2 KB, against 32 MB.

`wave7_probe.py` does that: one full enumeration at the start, then cheap hull
polling, with a full rescan only every ~90 s to catch anything genuinely new.
That is 3–4 heavy scans per run instead of 25.

Implemented, **not yet run** — so the claim that it stops the stalling is
untested, and the next run must report the witness before anything else.

---

# ✅ The cheap probe does not stall the guest (2026-08-24)

First run of `wave7_probe.py` — one enumeration, then hull polling, full rescan
every 90 s:

```
TOTAL candidate-up=1 down=19 stalled samples=0
```

**Zero stalled samples across the whole run**, against three consecutive heavy-probe
runs that stalled at ~27, ~83 and ~255 s. The fix works, and the previous
iteration's confirmation is now supported from the other direction as well.

The guest is also visibly healthier: **19 losses against 8** in the heavy-probe
run of comparable length. Starving the emulator was suppressing the very activity
the probe existed to watch.

---

# The single-word witness gives false positives (2026-08-24)

Status: 🔴 the witness as first written is unreliable; ✅ the contradiction that
exposes it is in its own output; ✅ replaced with a majority vote, not yet run.

A run with the cheap probe and a bound pilot produced this:

```
t= 30s deployed=41 up=0 down=1 ... *** GUEST STALLED ***
t= 75s deployed=41 up=0 down=1 ... *** GUEST STALLED ***
t=151s deployed=40 up=0 down=2 ... *** GUEST STALLED ***
TOTAL down=16  stalled samples=13
```

**Thirteen samples flagged as stalled while recording losses in the same
samples.** A frozen guest cannot destroy craft, so those flags are false
positives, and the run was in fact healthy — 16 losses over 290 s.

The cause is the selection rule: it took **the first word** in a 4 MB window
whose rate fell in a plausible band. Plenty of counters advance intermittently —
a buffer index, a per-event tally — and one of those satisfies the test while
saying nothing about whether frames are being rendered.

`timer_probe.py` had already done this correctly and the lesson was not carried
over: it collected **286** linearly-advancing words and looked at the *rate
histogram*, where a dominant cluster sat at ~17 /s.

**Fixed:** `wave7_probe.py` now collects every candidate, keeps the modal-rate
cluster, samples up to 32 of them, and reports a stall only when **fewer than
half** advance. It also says `RUN UNVALIDATED` when no witness is found at all,
because an earlier run printed `stalled samples=0` with `tick witness: NONE` —
a witness that does not exist cannot report zero stalls.

Not yet run, so the majority rule is unverified.

## What this costs

The stall counts quoted for earlier runs came from the pilot-log speed analysis,
not from this witness, so they stand. But any run judged **only** by the
single-word witness is unreliable in both directions, and the "0 stalled samples"
that validated the cheap probe came from a run where the witness may equally have
been a lucky pick. That validation should be repeated with the majority rule.

---

# The majority vote is better, the threshold was wrong — and freezes persist (2026-08-24)

Status: ✅ the multi-witness vote produces a usable graded signal; 🔴 the
"fewer than half" threshold is wrong; 🔴 **freezes still happen with the cheap
probe**, so the earlier "0 stalled samples" validation is superseded.

## The run

```
tick witnesses: 185 candidates, modal rate 93/s, using 31

t= 15s  down=1  *** GUEST STALLED (11/31 witnesses moved) ***
t= 45s  down=1  *** GUEST STALLED (11/31) ***
t= 90s  down=1  *** GUEST STALLED ( 4/31) ***
t=138s  down=1  *** GUEST STALLED ( 8/31) ***
t=168s  down=0  *** GUEST STALLED ( 1/31) ***
t=183s  down=1  *** GUEST STALLED ( 0/31) ***
t=198s … t=289s  down=0   *** GUEST STALLED (0/31) ***  (every sample)
TOTAL down=7  CONFIRMED arrivals=0
```

## ✅ Graded output is the improvement

Unlike the single-word witness, this says *how many* agree, and the number tells
a coherent story: 11 → 9 → 7 → 4 → 1 → **0**, with the drop to zero at t = 183 s
coinciding exactly with the last loss. From t = 183 onward the guest really is
frozen — 0 of 31 counters advance and nothing is destroyed for 106 s.

## 🔴 The threshold was wrong

Flagging a stall at "fewer than half" marked the entire run stalled, including
samples where craft were being destroyed. **11 of 31 advancing is a healthy
guest**, not a stalled one.

The cause is the cluster choice. The modal rate here was **93 /s**, far above the
~16.5 /s frame rate `timer_probe` measured — those are subsystem counters that
tick in bursts, so most are idle in any given 15 s window even while the game
runs. Selecting the modal cluster was convenient rather than principled.

**Fixed:** prefer the cluster whose rate falls in the frame-rate band (8–40 /s),
falling back to modal only if none exists, and flag a stall only when **zero**
witnesses advance — the signal the data actually supports. Not yet run.

## 🔴 Freezes are not eliminated by the cheap probe

This run used the cheap probe and still froze, at t ≈ 183 s. So the previous
iteration's "0 stalled samples" — which came from the unreliable single-word
witness — cannot stand as validation.

What the evidence now supports: the no-probe control ran 300 s clean (n = 1), the
heavy probe froze at 27–255 s, and the cheap probe froze at 183 s (n = 1). Cheap
sampling plausibly helps but **does not remove the freeze**, and with n = 1 on
each arm it is equally possible the freeze is stochastic and the control was
lucky. Recorded as unresolved rather than resolved in the probe's favour.

**Consequence:** the usable window per run is roughly 3 minutes, sometimes less,
whether or not the probe is cheap. Any experiment needing longer than that has to
survive a freeze or be redesigned around one.

---

# Witness attempt 3: the search window was too narrow (2026-08-24)

Status: 🔴 still unreliable; ✅ cause identified; ✅ bind failure separately
diagnosed and fixed.

The frame-rate-band selection ran, and still flags stalls in samples where craft
are being destroyed:

```
tick witnesses: 96 candidates, using 5 at 24/s (frame-rate band)
t= 30s down=1 *** GUEST STALLED (0/5 witnesses moved) ***
t=156s down=1 *** GUEST STALLED (0/5) ***
t=236s down=1 *** GUEST STALLED (0/5) ***
TOTAL down=11  stalled samples=7
```

**Five witnesses is not a vote.** And a counter genuinely running at 24 /s cannot
fail to advance across a 15 s sample — it would gain ~360. So these are bursty
counters again: they moved during the 3 s calibration and then stopped.

The cause is mine: when the witness was bolted into `wave7_probe` the candidate
search was narrowed to a **4 MB** window, while `timer_probe` — which found a
clean 17 /s cluster — searched the **whole 32 MB** region and found 286
candidates. Narrowing was never justified; it was just cheaper to write.

**Fixed:** search the full region once at startup (a one-off cost, not per
sample), and warn explicitly when fewer than 8 witnesses are available so a weak
vote is visible rather than silently trusted. Not yet run.

This is the third attempt at this witness. The pattern in all three failures is
the same — a shortcut in *selecting* the witness, not in the voting logic — and
each was caught only because the flagged samples contradicted the loss events in
the same output. That internal contradiction is the real check and is worth
keeping in any future version.

## ✅ Separately: the bind failure is fixed

Three consecutive bind attempts failed and the session **aborted rather than
flying unattended**, which is the behaviour added last iteration working as
intended. The cause: `entities2 self` finds the player by *motion* between two
samples, so a craft sitting still at mission start is invisible. The session now
holds the throttle for 3 s before each attempt. Verified — the next run bound
immediately and produced 11 losses.

---

# ✅ The witness is validated (2026-08-24, fourth attempt)

```
tick witnesses: 6500 candidates, using 32 at 21/s (frame-rate band)
t= 15s down=0
t= 30s down=1                       <- loss, NOT flagged
t= 45s down=0  *** GUEST STALLED (0/32 witnesses moved) ***
t= 60s … t=211s  down=0  *** GUEST STALLED (0/32) ***   (every sample)
TOTAL down=1  stalled samples=12
```

**Zero contradictions.** The single loss falls in an unflagged sample; every
flagged sample has no losses. That is the internal-consistency check the three
previous versions each failed, and it passes here.

What made the difference was searching the **whole 32 MB region** rather than a
4 MB window — 6500 candidates instead of 5, with a clean cluster at 21 /s
matching the measured frame rate.

## 🔴 But the freeze is now clearly the dominant problem

Onset across runs: **27, 45, 83, 183, 255 s** — median around 83 s. This run
froze at 45 s and produced exactly one loss, so it is useless for the arrival
question even though the instrument finally works.

The usable window is one to four minutes and unpredictable. Fourteen of fifteen
runs have gone into the arrival question and it still has no answer, largely
because of this.

## The probe is still not cheap

Worth being honest about: "cheap probe" was never that cheap, and the witness
made it worse. It still reads 32 MB to enumerate at startup, another 32 MB every
90 s to rescan, and the full-region calibration added **two more 32 MB reads**.
The freeze at 45 s came immediately after that calibration.

Trimmed: calibration now samples **8 spread windows of 512 KB** (4 MB total,
still covering the whole span) rather than reading everything twice, and the
rescan interval is doubled to 180 s. Not yet run.

---

# ✅ The periodic rescan was the freeze driver (2026-08-24)

Two changes, one failed and reverted, one that worked.

## 🔴 Trimming the calibration broke the witness

Sampling 8 spread 512 KB windows instead of the full region found **17**
candidates, none of them frame counters, and the witness then reported
`0/17 stalled` on **every sample of a run that recorded 13 losses** — a total
contradiction, caught by the same internal check as the previous three failures.
The frame-rate cluster is sparse and spread sampling misses it. Reverted.

Two 32 MB reads *once at startup* is the price of a witness that works.

## ✅ Disabling the periodic rescan removes the freeze

The recurring cost was the rescan — a 32 MB read every 90–180 s — and it exists
only to catch craft appearing from nowhere, which the roster work already showed
does not happen: every participant is allocated at mission load, so an arrival is
a state change on an existing craft, not a new allocation. Disabled.

Result, the first fully clean probed run:

```
tick witnesses: 3875 candidates, using 32 at 11/s (frame-rate band)
t=  0s … t=210s   no stall flag on any sample
8 losses recorded across the run
```

**Zero stalls over 210 s**, against freeze onsets of 27, 45, 83, 183 and 255 s in
every previous probed run. n = 1, so this is not proven — but combined with the
clean no-probe control it points clearly at recurring heavy reads rather than at
memory reading as such.

The run ended on the turn's timeout, not a freeze.

---

# Freezes are stochastic, not eliminated (2026-08-24)

The previous entry claimed disabling the periodic rescan "removes the freeze" on
n = 1. With more runs that is too strong. The tally:

| configuration | freeze onset |
|---|---|
| heavy probe | 27, 45, 83, 183, 255 s |
| cheap probe + 90 s rescan | 183 s |
| **cheap probe, no rescan** | clean 210 s, clean 240 s, clean 300 s, **frozen at 60 s** |

Three of four runs now survive past 200 s where essentially none did before, so
removing the rescan is a **real and large improvement** — but it is a change in
probability, not a fix, and the wording is corrected accordingly.

The witness handled the frozen run correctly: losses stop at t = 45 s, stalls are
flagged from t = 60 s onward, no contradiction.

## ✅ Calibration is no longer the bottleneck

The candidate search was the last Python loop over 8 million words. Replaced with
two numpy vector operations (`AGENT.md` notes numpy is installed, and its absence
used to look like a logic bug). 7250 candidates found, and the startup cost is
now small enough that **boot dominates**: ~190 s of title movie plus ~35 s to
flight, against a 595 s turn cap, leaves roughly 350 s of observation.

Boot is now the binding constraint on how much game time one turn can see, and
it is the only remaining lever worth pulling if longer windows are needed.

## Practical consequence

Do not treat a single frozen run as evidence of anything. Run, check the witness,
discard the frozen ones, and keep the clean runs — about three in four are usable
now, which is workable where it previously was not.

---

# ✅ What actually ends these runs: the mission black-screens at ~4.5 min (2026-08-25)

A two-segment OB watch made the mechanism visible for the first time.

**Segment 1** ran clean — witness reporting **0 stalled samples of 50**, OB steady
at 4 for 250 s. **Segment 2**, attaching to the same emulator, got:

```
NOT IN A MISSION (no unit definitions)
```

The emulator was still alive. The *mission* was gone. Looking at why:

* the pilot's **hull is 1500 in every line** — the player never died;
* the pilot log stops at **t = 267 s** with consecutive byte-identical lines
  (`spd=78`, same aim) — the freeze signature;
* the screen is now **entirely black**: `mean=(0.0, 0.0, 0.0)`.

This is the **pre-existing mission-end black-screen freeze** already recorded in
the corpus, caught in the act. Segment 1's clean witness is not a contradiction:
it sampled to t ≈ 250 s and the freeze came at ≈ 267 s.

## 🔴 Correction: chained attaches do NOT extend a mission

[mission-arrival-watch.md](mission-arrival-watch.md) records "BLOCKER REMOVED:
the emulator survives between calls", and that is true — but it was stated as if
observation could be extended indefinitely within a turn. It cannot. **The
emulator surviving is not the mission surviving.** The binding limit is the
game's own black-screen at roughly four and a half minutes, and no amount of
chaining crosses it.

The earlier chained run that reached 435 s of cumulative verified-live
observation did so across *two* segments of one mission — consistent with this,
since its second segment froze at t = 135 s.

## What it means for the open questions

* Any experiment needing more than ~4.5 minutes of one continuous mission is
  **not doable** on this build, chained or not. That includes clearing 16 marked
  fighters at two kills per five minutes.
* The freeze is now the highest-value target in its own right: fixing it would
  unblock the arrival question, the phase-advance question and the OB series at
  once. It is also, per the corpus, a long-standing known defect rather than
  something this work introduced.