diff --git a/docker/ci/Dockerfile b/docker/ci/Dockerfile new file mode 100644 index 00000000..e43cb4dd --- /dev/null +++ b/docker/ci/Dockerfile @@ -0,0 +1,33 @@ +# The CI image, reproduced locally. +# +# 🔴 WHY THIS FILE EXISTS. This image was built once, by hand, from a Dockerfile +# in a scratch directory under /tmp — and /tmp was swept. The image survived on +# one host and its recipe did not, which is the same shape as every other thing +# this consolidation found: something correct that exists in exactly one place. +# +# The apt list below is COPIED from `.github/workflows/ci.yml`'s "Install Linux +# system dependencies" step. If that list changes, this must change with it, or +# a local run stops predicting the runner. +# +# docker build -t sylph-ci:local docker/ci +# +# ⚠️ The Rust version is PINNED and the runner's is NOT — `dtolnay/rust-toolchain@stable` +# floats. That difference is issue #15: `collapsible_else_if` is `warn` on 1.92 +# and `allow` on 1.98.1, so a local clippy pass and a runner clippy pass can +# legitimately disagree. When they do, the runner is the authority. +FROM rust:1.98.1-bookworm + +RUN apt-get update && apt-get install -y --no-install-recommends \ + libasound2-dev \ + libudev-dev \ + libwayland-dev \ + libxkbcommon-dev \ + libx11-dev \ + libxi-dev \ + pkg-config \ + && rm -rf /var/lib/apt/lists/* + +RUN rustup component add clippy rustfmt \ + && rustup target add wasm32-unknown-unknown + +WORKDIR /work diff --git a/docker/ci/Dockerfile.ffmpeg b/docker/ci/Dockerfile.ffmpeg new file mode 100644 index 00000000..54117979 --- /dev/null +++ b/docker/ci/Dockerfile.ffmpeg @@ -0,0 +1,11 @@ +# `sylph-ci:ffmpeg` — the CI image plus ffmpeg, for `sylpheed-export`. +# +# The exporter shells out to `ffprobe`/`ffmpeg` to transcode the two movies, so +# `cargo run -p sylpheed-export -- export` fails in the plain CI image with +# "run ffprobe -- is it on PATH?". CI itself never runs the exporter, which is +# why its own image does not carry this. +# +# docker build -t sylph-ci:ffmpeg -f docker/ci/Dockerfile.ffmpeg docker/ci +FROM sylph-ci:local +RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg \ + && rm -rf /var/lib/apt/lists/* diff --git a/docker/ci/run b/docker/ci/run new file mode 100755 index 00000000..bd7c007c --- /dev/null +++ b/docker/ci/run @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +# Run a command against this repository inside the capped CI container. +# +# docker/ci/run cargo check --workspace --all-targets +# docker/ci/run cargo clippy --workspace --all-targets --keep-going -- -D warnings +# docker/ci/run bash -c 'cargo fmt --all; cargo fmt --all -- --check | grep -c "^Diff in"' +# +# 🔴 EVERY HEAVY COMMAND GOES THROUGH HERE. A bare `cargo build` on the host is +# unbounded: `CARGO_BUILD_JOBS` caps codegen units, not rustc's own threads, not +# the linker, not the test harness. A full-parallel build has OOM-crashed this +# box, and unbounded host runs have frozen it since. The container is the cap +# that actually holds. +# +# Half the machine, no swap. `--memory-swap` equal to `--memory` means a build +# that would swap is killed instead, which is a fast failure rather than an +# hour of thrashing that takes the desktop with it. +set -euo pipefail + +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +IMAGE="${SYLPH_CI_IMAGE:-sylph-ci:local}" +CPUS="${SYLPH_CI_CPUS:-6}" +MEM_GB="${SYLPH_CI_MEM_GB:-7}" + +args=( + --rm + --cpus "$CPUS" + --memory "${MEM_GB}g" + --memory-swap "${MEM_GB}g" + --pids-limit 2048 + -v "$REPO:/work" + # Named volumes, not bind mounts: the host tree keeps a 32 GB `target/` from + # earlier host-side builds, and mixing the two produces rebuilds that look + # like cache misses and are actually two toolchains fighting over one directory. + -v sylph-ci-cargo:/cargo -e CARGO_HOME=/cargo + -v sylph-ci-target:/target -e CARGO_TARGET_DIR=/target + -w /work +) + +# The disc, read-only, when a disc-backed test or the exporter needs it. +DISC="${SYLPHEED_DISC:-$REPO/../sylph_extract}" +[ -d "$DISC" ] && args+=(-v "$DISC:/disc:ro" -e SYLPHEED_DISC=/disc) + +exec docker run "${args[@]}" "$IMAGE" "$@" diff --git a/docs/agents/CONTAINERS-AND-AGENTS.md b/docs/agents/CONTAINERS-AND-AGENTS.md new file mode 100644 index 00000000..2c4ccfc4 --- /dev/null +++ b/docs/agents/CONTAINERS-AND-AGENTS.md @@ -0,0 +1,157 @@ +# Containers and agents — how this is actually set up + +**Written 2026-09-13 for the other machine.** Everything below was read off the +running host, not remembered. Paths are as they exist on the desktop; the Pi +differs where it says so. + +## 1. The rule that governs all of it + +🔴 **Every heavy command runs in a capped container. No exceptions, including +one-off checks.** + +```bash +docker/ci/run cargo check --workspace --all-targets +``` + +A bare `cargo build` on the host is **unbounded**. `CARGO_BUILD_JOBS` caps +codegen units — not rustc's own threads, not the linker, not the test harness. +A full-parallel build has OOM-crashed this box, and unbounded host runs have +frozen it repeatedly since. `--memory-swap` equal to `--memory` means a build +that *would* swap is killed instead: a fast failure beats an hour of thrashing +that takes the desktop with it. + +What may run on the host: `git`, `grep`, `python3` over text, API calls. Anything +that compiles, links, transcodes or boots an emulator does not. + +## 2. Three kinds of container + +| | image | what it is | +|---|---|---| +| **CI / build** | `sylph-ci:local` | the runner, reproduced locally — `docker/ci/Dockerfile` | +| **Decoder agent** | `sylpheed-agent:latest` | an autonomous Claude Code session doing RE — `docker/decoder/` | +| **Port agent** | `sylpheed-port:latest` | the same, building the Godot port — `docker/port/` | + +### 2.1 The CI image + +`docker/ci/Dockerfile` — `rust:1.98.1-bookworm`, **the apt list copied verbatim +from `.github/workflows/ci.yml`**, plus `clippy`, `rustfmt` and the +`wasm32-unknown-unknown` target. + +```bash +docker build -t sylph-ci:local docker/ci +docker build -t sylph-ci:ffmpeg -f docker/ci/Dockerfile.ffmpeg docker/ci # exporter only +``` + +⚠️ **The Rust version is pinned here and floats on the runner** +(`dtolnay/rust-toolchain@stable`). That is issue #15, and it is not theoretical: +`collapsible_else_if` is `warn` on 1.92 and `allow` on 1.98.1. When a local +clippy pass and a runner pass disagree, **the runner is the authority.** + +⚠️ **`cargo clippy` stops at the first failing compilation unit.** Without +`--keep-going` the list looks short and is not. On the corpus branch the honest +count was **80**, and the first run showed **14**. + +`docker/ci/run` gives 6 CPUs / 7 GB / no swap, mounts the repo at `/work`, the +extracted disc read-only at `/disc`, and uses **named volumes** for `CARGO_HOME` +and `CARGO_TARGET_DIR` — never the host `target/`, which still holds 32 GB from +older host-side builds and produces rebuilds that look like cache misses. + +### 2.2 The agent containers + +Launched from the host by `docker/decoder/sylph-decoder` and +`docker/port/sylph-port`. Each starts a Claude Code session inside a container +with the repo, the disc, Canary's source and a warm build tree. + +**The budget is split deliberately, not halved per container** — there are two +agents and a Referee is planned, so each claiming half a box it shares would +oversubscribe it: + +| | CPUs | memory | `/dev/shm` | +|---|---|---|---| +| decoder | 5 | 6 GB | 2 GB | +| port | 3 | 4 GB | — | + +Overridable with `SYLPH_CPUS` / `SYLPH_MEM_GB` and `SYLPH_PORT_CPUS` / +`SYLPH_PORT_MEM_GB`. `/dev/shm` is an **exec-capable tmpfs**, not `--shm-size`: +Docker mounts the default `noexec` and Xenia maps its JIT code cache out of a +shm file. Its pages count against the memory cap, hence a third of it and no +more. + +**Volumes** (`docker volume ls`): + +``` +sylpheed-decoder-repo sylpheed-port-repo the agent's own clone +sylpheed-decoder-claude sylpheed-port-claude Claude Code state + transcripts +sylph-agent-cargo sylpheed-port-cargo CARGO_HOME +sylph-agent-target sylpheed-port-target CARGO_TARGET_DIR +sylph-agent-canary-build the warm 235 MB Canary build tree +sylpheed-exchange agent -> agent files, read-only in +``` + +**Credentials** — three files on the host, `chmod 600`, mounted read-only: + +``` +~/.sylph-claude-token Claude Code OAuth (a subscription, never the API) +~/.sylph-git-credentials the agents' PUSH credential — write:repository ONLY +~/.sylph-gitea-token-decoder the decoder's own Gitea account, for issues/PRs +~/.sylph-gitea-token-port the port's +~/.sylph-gitea-token-fabi the HUMAN's, full grant — mounted into NOTHING +``` + +🔴 **The narrow scope on `~/.sylph-git-credentials` is load-bearing.** Every +issue endpoint refuses it, which is what stops an agent reaching the Gitea API +as the human. Putting a broader token there would hand both agents the human's +identity, including merge. The human's own token lives in a path no launcher +reads and no container mounts — verified, not assumed. + +The Gitea MCP server is passed the token **by path**, not by value: +`GITEA_TOKEN_FILE=/sylph-home/re/.sylph-gitea-token`. + +## 3. How an agent iteration actually works + +Both briefs — `decoder-loop.md`, `port-loop.md` — are read **from the host tree +at container launch**. The loop is: read notifications → pick one approved issue +→ do the smallest experiment → classify → refute something → write it down → +`push-work`, open the PR, label `state/needs-human`, stop. + +⚠️ **Notifications are POLLED. Nothing is pushed.** An `@mention` or a PR comment +reaches an agent only on its next iteration. + +### 🔴 Two defects the other machine should know about + +**A brief change does not reach a running agent.** The entrypoint resumes the +newest transcript in the container's Claude volume, and the only guard is a +`<120 s` restart-loop test. Everything older resumes unconditionally — so a +container restarted after a workflow change comes back **still following the old +brief**, on the old branch. Seen: a decoder resumed a 7-day-old pre-migration +session and had to be stopped. Until the entrypoint learns to test transcript +age (or take a `SYLPH_FRESH=1`), adopting a brief change means **archiving the +transcripts** in `sylpheed-decoder-claude/projects/` and starting fresh. + +**Nothing brought an agent back to its own red PR.** It opened one, labelled the +issue, stopped — correctly, per the brief — and the next firing started a new +question on top. It pushed over a red CI six times. Fixed in both briefs (#24), +which is itself subject to the defect above: the fix only reaches a *fresh* +session. + +## 4. The other machine's half + +The Pi runs the Gitea instance (proxied to `git.mc02.dev` via a VPS, reachable +from the LAN and the internet) and the Actions runner. Its +`/var/lib/docker` is still on the **117 GB SD card** beside an idle 916 GB SSD; +moving `data-root` is outstanding. + +⚠️ **The runner serialises.** Three jobs per push, one at a time — a PR's checks +sat *pending* for 40+ minutes behind a failed native job. A pending check there +is not necessarily a slow one. + +## 5. Standing constraints that are not about containers + +* **One emulator process at a time**, ours or Canary, never both — a lockfile. +* **Canary runs muted.** Point it at the real ISO, not the `sylpheed.iso` symlink + (Wine cannot resolve it). +* **Never judge a crash or a hang from a Bash-launched emulator run.** A SIGKILL + that looked like our binary was the editor's process supervisor. Ask the human + to run it natively. +* **Never screenshot the port or the emulator yourself**; ask the human. +* The oracle is the real game in Xenia Canary — not our renderer, not the port.