Nothing anywhere executes the production `Caddyfile` before the real deploy —
the e2e stack mounts `e2e/Caddyfile.test` — and a syntax error there is total:
Caddy exits, `restart: unless-stopped` loops, 443 is dead for the whole event,
and `docker compose up -d --force-recreate caddy` still exits 0 while it
crash-loops. Step zero now validates it. I ran it against the current file
(which I changed last commit, unexercised): "Valid configuration", and the new
`read_body 30m` adapts to `read_timeout: 1800000000000`ns as intended.
And a warning §9 needed: a down migration is not a standalone repair. Roll the
IMAGE back first. `Upload::create` sends an `ON CONFLICT ... WHERE` predicate
that must match the live partial index exactly and is not compile-checked, so
running 026's or 031's down against the current binary turns every upload
carrying a client_upload_id — i.e. every upload from the shipped client — into
a runtime 500. 026's down can also fail outright on any database where a guest
deleted and re-uploaded a photo; it rolls back cleanly, but you cannot go below
it. Both verified against a live Postgres.
* memswap_limit is now IN docker-compose.yml on all four services. Compose
sets Memory but leaves MemorySwap unset, and Docker then permits swap equal
to the memory limit — so following §5's "add 2 GB of swap" silently DOUBLED
every ceiling, to ~5 GiB on a 3.82 GiB box. Nothing OOMs; instead Postgres's
working set becomes swap-eligible on a shared-tenancy SSD, turning a bounded
OOM-kill that restarts in seconds into unbounded latency with no signal but
"everything is slow". The runbook told the operator to hand-add it, which
also broke §0's own gate that docker-compose.yml must be unmodified.
Verified rather than assumed: service-level memswap_limit does compose with
deploy.resources.limits.memory (docker inspect → Memory=1073741824
MemorySwap=1207959552).
* DATABASE_MAX_CONNECTIONS pinned in compose. It is the one env var that is
now boot-FATAL when unparseable — the right call, but it means a stray quote
or a trailing inline comment in .env crash-loops the app behind a live
Caddy. MEDIA_PATH, EXPORT_PATH and APP_PORT are pinned for weaker reasons.
* .env.example's quota narrative was sized for a CX33: "~30 GB of a fresh
70 GB" on a box with 40 GB. And on THIS box the fixed point never binds at
all — ~210 MB/guest is below the 500 MiB floor, so everyone gets the floor
and the per-user quota stops bounding aggregate growth. What actually stops
uploads is the keepsake preflight at ~8 GB of media. That paragraph is what
an operator reads when a guest is blocked, and it pointed at the wrong knob.
* The emergency card gains the one disk symptom that can appear mid-event,
where `df -h` — its only disk instruction — actively misleads: the gate
fires ~10 GB + 2.2x media BEFORE the disk is full, so df shows ~20 GB free
at the moment uploads are being refused.
* Two code comments that now assert the opposite of the code: claim_job
promised that "the update_progress liveness check bails such a worker out
early" — it cannot, its predicate is on the job row, which a reopen does not
touch, so a mid-export reopen grinds the whole gallery to completion on a
2-vCPU box during the live event. And prune_superseded_archives still argued
"deleted bytes cannot be rolled back" as an invariant, after the reclaim
path was changed to prune even when that will not close the shortfall.
Both now describe what the code does.
* Smaller corrections: runbook §3's "two 48 MP photos ≈ 800 MB" scenario is
unreachable (compression.rs takes an exclusive heavy permit, so they
serialise) and contradicted .env.example; "all four healthy" is wrong since
caddy has no healthcheck; a README line reference pointed at a comment added
by the same commit that broke it.
* Caddy had no read_body, on the reasoning that "a slow body still has to
actually send bytes". That is an argument about disk, and disk is not the
scarce resource: upload_admission budgets concurrent bodies at 4096 MiB and
reserves the DECLARED cap, so a video/* upload reserves 500 MiB. Eight
connections that stall mid-body hold the whole budget, every other guest
waits 20s and gets a 503, and it never recovers on its own — the permit is
held until the handler returns. No attacker needed: eight guests starting
real videos and walking out of AP range does it, and TCP will not reap
those sockets for hours. 30m carries a 500 MB upload at ~2.2 Mbit/s, so it
does not fail the uploads this product exists to collect.
* APP_PORT is presented in .env.example as an ordinary editable line, while
the healthcheck hardcodes 127.0.0.1:3000 and the Caddyfile hardcodes
app:3000. Change it and the app boots and serves happily on the new port,
the healthcheck fails forever, app never turns healthy — and because caddy
is gated on service_healthy, CADDY NEVER STARTS. Port 443 dead for the
whole event, sole diagnostic "dependency failed to start". Pinned in
compose beside MEDIA_PATH and EXPORT_PATH, which are there for this reason.
* Runbook §12's recovery commands do not run as written: unwrapped
"$POSTGRES_USER" is expanded by the operator's shell, which does not have
it, so psql answers `FATAL: role "" does not exist`. §9 documents that trap
two hundred lines earlier and wraps its own calls in sh -c; §12 did not.
This is the block you run with the app crash-looping behind a live Caddy.
Its DELETE also hard-coded versions 21,22,23 as if to be copied verbatim,
on a tree that now has 31 migrations — now explicitly an example, with the
instruction to take the numbers from the actual boot error.
* Migration counts corrected across the runbook and .env.example (22 -> 31,
commit count 154 -> 196). All four were presented as literal command output
the operator is invited to reproduce.
`deploy.resources.reservations.cpus` was doing nothing. Outside Swarm, `docker compose up`
silently drops it — verified by inspecting a running container, where CpuShares, CpuQuota
and CpusetCpus were all unset while `limits.cpus` and `reservations.memory` came through as
NanoCpus and MemoryReservation. So the comment calling it "the piece that actually protects
the database" described a guarantee the box never had.
It matters on the CX22 the runbook targets: the ceilings sum to 1.2 + 0.6 + 0.5 = 2.3 on
2 vCPU, so the other services can oversubscribe the machine, and with every container on the
default weight Postgres competed on equal footing with two image resizes and an ffmpeg
poster. Replaced with `cpu_shares`, which does survive the translation — db 2048, caddy
1024, app 512, frontend 256 — so the weighting only binds when the CPU is actually
saturated, which is the moment the database must not lose.
The Caddyfile gains a 10s header-read timeout: there was no read timeout anywhere, so a
client could hold a connection, a tokio task and a `.tmp` file open indefinitely by sending
one byte a minute, and the upload sweeper is keyed on mtime precisely so a live upload never
ages out. Body reads stay unbounded — a 500 MB video over cellular legitimately takes
minutes, and a body timeout would fail exactly the uploads this product exists to collect.
.env.example documents that estimated_guest_count is a live input to the quota divisor
rather than the inert setting both it and the runbook previously implied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The production compose still carried `build:` keys and no `image:` keys, so a
`git clone` onto the CX22 followed by `docker compose up -d` would have started a
fat-LTO release build of 427 crates on a 2-vCPU/4 GB box — the outcome the whole
build-on-the-Mac decision exists to avoid, reached silently because `pull` skips a
service it is told to build rather than failing.
Both services now pull `registry.mc02.dev/eventsnap/*:${EVENTSNAP_VERSION}` with the
`:?` form, so a missing tag fails the command instead of resolving to an empty one.
`docker-compose.build.yml` restores the `build:` keys for the workstation that
produces the images, from the same context paths.
Also here:
- `DOMAIN` gets the same `:?` guard. Blank did not fail — it produced `https://` for
the frontend's ORIGIN and collapsed the Caddyfile's site block into a malformed
global block, so the stack came up with no TLS and no site.
- `stop_grace_period: 20s` on the app. Docker's default stop timeout is 10s, exactly
the app's own drain budget, so a redeploy could SIGKILL the process at the moment it
was finishing — truncating the in-flight upload the graceful shutdown protects.
- `COMMENTS_ENABLED` is pinned "false" alongside MEDIA_PATH. It is a product decision
for this event, and `.env.example` ships the generic `true`; pinning it means an
operator who copies the example and edits only the secrets cannot ship comments on.
- The frontend runtime stage now copies the lockfile and uses `npm ci`. Without it the
three `^`-ranged deps re-resolved at build time, so an image rebuilt days later could
differ from the one that was tested. Image also drops 120 MB -> 65 MB.
- `docker-compose.dev.yml` told the operator that production had the same `$`-eating
bug and to escape the hash as `$$` in `.env`. That is wrong and it breaks a working
deployment: Compose uses single-quoted env_file values literally, and doubling
produces a 74-character string `looks_bcrypt` rejects. Verified with `printenv`.
The runbook's rollback pointed at `v0.12.0`, which has 6 migrations against HEAD's 22
and was never built or pushed — running the emergency card's rollback line would have
crash-looped the app with `VersionMissing` during the event. §9 now has you tag one
build twice so the rollback target is bit-identical, and says plainly what that can and
cannot fix. Every `$DOMAIN` command gained the `set -a; . ./.env` it needs, the
down-migration psql commands are wrapped in `sh -c` so the container expands the
credentials rather than sending `-U ""`, and the advice to lower `max_video_size_mb`
is withdrawn: the client guard it was premised on does exist, but is pinned to a
compile-time constant, so lowering the DB value only moves failures later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
