docs: the mystery SIGKILL was this project's own Stop hook

/work/.claude/settings.json defines a Stop hook that kill -9s every xenia_canary
at the end of each agent turn and prints "Stop hook killed N stale xenia
process(es)". So every run that "died mysteriously" died at a turn boundary,
which is why the timings looked random from inside the run (810 s, 54 s, 486 s)
and why nothing in the guest, the cgroup or the host explained them.

What survives is the measurement and not the story: the memory readings were real
and did refute memory pressure - cgroup at 6.5 of 7.0 GB, host 12 GB free at the
moment of a kill - but they were refuting a cause that was never in play.

The method lesson is recorded because it cost three iterations: when a process
dies at a SESSION boundary, check the harness before instrumenting the guest. A
failure mode was documented, a hypothesis raised, a per-poll memory sampler
written and committed, and host and cgroup counters read - all downstream of the
assumption that the kill came from outside the agent. The hook's own message had
been printed after every turn.

The operational rule that follows: an emulator experiment must COMPLETE INSIDE A
SINGLE TURN. Nothing survives the boundary, so runs cannot be left for the next
tick, and experiments whose evidence arrives in the first minutes are the ones
that work - which is exactly why the two-pass bit-level test succeeded where the
long freeze-watches did not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
Sylpheed RE agent
2026-08-24 09:48:23 +00:00
parent 12e1d9453a
commit ce0461a858

View File

@@ -583,3 +583,52 @@ in here attributes a SIGKILL to its sender (no `dmesg`, no audit, and
`ptrace_scope=1` limits `strace` to my own descendants). The practical response
is to stop designing experiments that need a *long* run: the counter transitions
the OB work needs all happen in the first minutes of flight.
---
# ✅ 2026-08-24 — THE KILL IS SOLVED, and it was never the game: a Stop hook
**Everything above about the `EMULATOR GONE` / SIGKILL failure mode has a
one-line cause**, and it is not in the emulator, the container runtime or the
kernel. It is in this project's own Claude Code configuration —
`/work/.claude/settings.json`:
```json
"Stop": [{ "hooks": [{ "type": "command", "command":
"for name in xenia_canary xenia-rs; do pids=$(pgrep -x \"$name\"); ...
kill $pids; sleep 0.2; kill -9 $pids; ... 'Stop hook killed %d stale xenia process(es)'"
}]}]
```
A `Stop` hook fires at the **end of every agent turn** and `kill -9`s any running
`xenia_canary`. So every run that "died mysteriously" died at a turn boundary,
which is exactly why the timings looked random from inside the run (810 s, 54 s,
486 s) and why nothing in the guest, the cgroup or the host explained them.
**What survives from the investigation** is only the measurement, not the story:
the memory readings were real and did refute memory pressure as a cause — the
cgroup was at 6.5 GB of 7.0 and the host had 12 GB free at the moment of a kill.
That refutation was correct; it just was not pointing at anything.
## 🔑 The method lesson, which is the part worth keeping
**When a process dies at a session boundary, check the harness before
instrumenting the guest.** Three iterations went into this: a failure mode
documented, a memory hypothesis raised, a per-poll memory sampler written and
committed, host and cgroup counters read — all of it downstream of an assumption
that the kill came from *outside the agent*. The one place not looked at was the
agent's own configuration, and that is where it was. The `systemMessage` the hook
prints ("Stop hook killed 1 stale xenia process(es)") is surfaced to the user
after each turn, so the answer had been on screen the whole time.
## The rule that follows
**An emulator experiment must complete inside a single turn.** Nothing survives
the turn boundary, so:
* do not "leave a run going for the next tick" — it will be killed;
* prefer experiments whose evidence arrives in the first minutes of flight (the
`REMAINING OB` transitions all do — that is why the two-pass bit-level test
succeeded where the long freeze-watches did not);
* a watcher armed for 1 500 s can only ever watch for the rest of *this* turn.