docs+tools: the silent death is a SIGKILL from outside, and it is still not OOM
Run 8 died 54 s into its boot and this time bash named it: "176880 Killed" on the run-canary line, which is SIGKILL. So the third failure mode is not an internal fault - something outside the process is killing it. And it is still not the OOM killer. Checked immediately after: oom_kill remained 0 and the allocation-stall counter did not move from 4421, so during run 8 the cgroup never reached its limit (5.35 GB of 7 GiB), and the host had 13.8 GB available. Two kills, no OOM evidence either time. Rather than keep guessing after the fact, freeze_watch.sh now samples host MemAvailable, cgroup memory.current and oom_kill on every poll and dumps the last five samples when the process disappears - so the next occurrence carries its own contemporaneous reading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
This commit is contained in:
@@ -485,6 +485,16 @@ w> F8000254 XThread::Resume: host resume was refused for thread F80001E8 <- en
|
||||
`Fatal` or `abort` in the whole 1.1 MB log. No shutdown line either. The process
|
||||
is just gone.
|
||||
|
||||
**And on the next occurrence the shell named it: `Killed`.** Run 8 died 54 s into
|
||||
its *boot*, and `launch_mission.sh` printed
|
||||
|
||||
```
|
||||
line 74: 176880 Killed nohup run-canary --apu=sdl --log_mask=13 ...
|
||||
```
|
||||
|
||||
which is bash reporting **SIGKILL**. So this is not an internal fault at all —
|
||||
something outside the process is killing it.
|
||||
|
||||
### Memory pressure is a suspect, and only a suspect
|
||||
|
||||
The container's cgroup, read immediately after, with **no emulator running**:
|
||||
@@ -505,6 +515,17 @@ disc-wide format sweeps, which read every `.pak` — was most of it. 🔴 **But
|
||||
*what* did. Recorded as an unexplained third failure mode rather than an OOM
|
||||
story, because the counter that would have proved OOM says zero.
|
||||
|
||||
🔴 **Checked again immediately after run 8's SIGKILL, and it is still not OOM.**
|
||||
`oom_kill` remained **0** and the `max` (allocation-stall) counter did **not
|
||||
move** from 4 421 — so during run 8 the cgroup never even reached its limit,
|
||||
`memory.current` being 5.35 GB of 7 GiB. The host had **13.8 GB available** when
|
||||
checked. Two kills, no OOM evidence either time.
|
||||
|
||||
**So the cause is genuinely unidentified**, and the next occurrence is now
|
||||
instrumented rather than reconstructed: `freeze_watch.sh` samples host
|
||||
`MemAvailable`, the cgroup's `memory.current` and its `oom_kill` counter on every
|
||||
poll, and dumps the last five samples when it sees the process disappear.
|
||||
|
||||
**Hygiene that follows either way:** `/dev/shm/xenia_memory_*` survives a dead
|
||||
run (342 MB resident here) and `run-canary` only clears it at *launch*; and
|
||||
`vm.drop_caches` is not writable in the container (read-only `/proc/sys`), so
|
||||
|
||||
Reference in New Issue
Block a user