* Caddy had no read_body, on the reasoning that "a slow body still has to
actually send bytes". That is an argument about disk, and disk is not the
scarce resource: upload_admission budgets concurrent bodies at 4096 MiB and
reserves the DECLARED cap, so a video/* upload reserves 500 MiB. Eight
connections that stall mid-body hold the whole budget, every other guest
waits 20s and gets a 503, and it never recovers on its own — the permit is
held until the handler returns. No attacker needed: eight guests starting
real videos and walking out of AP range does it, and TCP will not reap
those sockets for hours. 30m carries a 500 MB upload at ~2.2 Mbit/s, so it
does not fail the uploads this product exists to collect.
* APP_PORT is presented in .env.example as an ordinary editable line, while
the healthcheck hardcodes 127.0.0.1:3000 and the Caddyfile hardcodes
app:3000. Change it and the app boots and serves happily on the new port,
the healthcheck fails forever, app never turns healthy — and because caddy
is gated on service_healthy, CADDY NEVER STARTS. Port 443 dead for the
whole event, sole diagnostic "dependency failed to start". Pinned in
compose beside MEDIA_PATH and EXPORT_PATH, which are there for this reason.
* Runbook §12's recovery commands do not run as written: unwrapped
"$POSTGRES_USER" is expanded by the operator's shell, which does not have
it, so psql answers `FATAL: role "" does not exist`. §9 documents that trap
two hundred lines earlier and wraps its own calls in sh -c; §12 did not.
This is the block you run with the app crash-looping behind a live Caddy.
Its DELETE also hard-coded versions 21,22,23 as if to be copied verbatim,
on a tree that now has 31 migrations — now explicitly an example, with the
instruction to take the numbers from the actual boot error.
* Migration counts corrected across the runbook and .env.example (22 -> 31,
commit count 154 -> 196). All four were presented as literal command output
the operator is invited to reproduce.
`deploy.resources.reservations.cpus` was doing nothing. Outside Swarm, `docker compose up`
silently drops it — verified by inspecting a running container, where CpuShares, CpuQuota
and CpusetCpus were all unset while `limits.cpus` and `reservations.memory` came through as
NanoCpus and MemoryReservation. So the comment calling it "the piece that actually protects
the database" described a guarantee the box never had.
It matters on the CX22 the runbook targets: the ceilings sum to 1.2 + 0.6 + 0.5 = 2.3 on
2 vCPU, so the other services can oversubscribe the machine, and with every container on the
default weight Postgres competed on equal footing with two image resizes and an ffmpeg
poster. Replaced with `cpu_shares`, which does survive the translation — db 2048, caddy
1024, app 512, frontend 256 — so the weighting only binds when the CPU is actually
saturated, which is the moment the database must not lose.
The Caddyfile gains a 10s header-read timeout: there was no read timeout anywhere, so a
client could hold a connection, a tokio task and a `.tmp` file open indefinitely by sending
one byte a minute, and the upload sweeper is keyed on mtime precisely so a live upload never
ages out. Body reads stay unbounded — a 500 MB video over cellular legitimately takes
minutes, and a body timeout would fail exactly the uploads this product exists to collect.
.env.example documents that estimated_guest_count is a live input to the quota divisor
rather than the inert setting both it and the runbook previously implied.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The production compose still carried `build:` keys and no `image:` keys, so a
`git clone` onto the CX22 followed by `docker compose up -d` would have started a
fat-LTO release build of 427 crates on a 2-vCPU/4 GB box — the outcome the whole
build-on-the-Mac decision exists to avoid, reached silently because `pull` skips a
service it is told to build rather than failing.
Both services now pull `registry.mc02.dev/eventsnap/*:${EVENTSNAP_VERSION}` with the
`:?` form, so a missing tag fails the command instead of resolving to an empty one.
`docker-compose.build.yml` restores the `build:` keys for the workstation that
produces the images, from the same context paths.
Also here:
- `DOMAIN` gets the same `:?` guard. Blank did not fail — it produced `https://` for
the frontend's ORIGIN and collapsed the Caddyfile's site block into a malformed
global block, so the stack came up with no TLS and no site.
- `stop_grace_period: 20s` on the app. Docker's default stop timeout is 10s, exactly
the app's own drain budget, so a redeploy could SIGKILL the process at the moment it
was finishing — truncating the in-flight upload the graceful shutdown protects.
- `COMMENTS_ENABLED` is pinned "false" alongside MEDIA_PATH. It is a product decision
for this event, and `.env.example` ships the generic `true`; pinning it means an
operator who copies the example and edits only the secrets cannot ship comments on.
- The frontend runtime stage now copies the lockfile and uses `npm ci`. Without it the
three `^`-ranged deps re-resolved at build time, so an image rebuilt days later could
differ from the one that was tested. Image also drops 120 MB -> 65 MB.
- `docker-compose.dev.yml` told the operator that production had the same `$`-eating
bug and to escape the hash as `$$` in `.env`. That is wrong and it breaks a working
deployment: Compose uses single-quoted env_file values literally, and doubling
produces a 74-character string `looks_bcrypt` rejects. Verified with `printenv`.
The runbook's rollback pointed at `v0.12.0`, which has 6 migrations against HEAD's 22
and was never built or pushed — running the emergency card's rollback line would have
crash-looped the app with `VersionMissing` during the event. §9 now has you tag one
build twice so the rollback target is bit-identical, and says plainly what that can and
cannot fix. Every `$DOMAIN` command gained the `set -a; . ./.env` it needs, the
down-migration psql commands are wrapped in `sh -c` so the container expands the
credentials rather than sending `-U ""`, and the advice to lower `max_video_size_mb`
is withdrawn: the client guard it was premised on does exist, but is pinned to a
compile-time constant, so lowering the DB value only moves failures later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
