fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(runbook): validate the Caddyfile before the freeze, and pair down-migrations with a rollback

Browse Source 
Nothing anywhere executes the production `Caddyfile` before the real deploy —
the e2e stack mounts `e2e/Caddyfile.test` — and a syntax error there is total:
Caddy exits, `restart: unless-stopped` loops, 443 is dead for the whole event,
and `docker compose up -d --force-recreate caddy` still exits 0 while it
crash-loops. Step zero now validates it. I ran it against the current file
(which I changed last commit, unexercised): "Valid configuration", and the new
`read_body 30m` adapts to `read_timeout: 1800000000000`ns as intended.

And a warning §9 needed: a down migration is not a standalone repair. Roll the
IMAGE back first. `Upload::create` sends an `ON CONFLICT ... WHERE` predicate
that must match the live partial index exactly and is not compile-checked, so
running 026's or 031's down against the current binary turns every upload
carrying a client_upload_id — i.e. every upload from the shipped client — into
a runtime 500. 026's down can also fail outright on any database where a guest
deleted and re-uploaded a photo; it rolls back cleanly, but you cannot go below
it. Both verified against a live Postgres.
This commit is contained in:
fabi
2026-08-12 20:00:52 +02:00
parent 301e6636a5
commit 8af8c4fab7
1 changed files with 19 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
DEPLOYMENT_RUNBOOK.md
View File

@@ -50,6 +50,13 @@ git show HEAD:docker-compose.yml | grep -cE '^[[:space:]]*build:' # must be 0
# Every migration in the tree is committed — a build from a dirty tree bakes in extras. # Every migration in the tree is committed — a build from a dirty tree bakes in extras.
git status --porcelain -- backend/migrations/ # must print nothing git status --porcelain -- backend/migrations/ # must print nothing
# The Caddyfile PARSES. Nothing else checks it: the e2e stack mounts `e2e/Caddyfile.test`,
# so the production file is never executed until the real deploy — and a syntax error there
# is total. Caddy exits, `restart: unless-stopped` loops, 443 is dead for the whole event,
# and `docker compose up -d --force-recreate caddy` still exits 0 while it crash-loops.
docker run --rm -v "$PWD/Caddyfile:/etc/caddy/Caddyfile:ro" -e DOMAIN=example.com \
caddy:2-alpine caddy validate --config /etc/caddy/Caddyfile # must end "Valid configuration"
``` ```
| When | What | | When | What |
@@ -623,6 +630,18 @@ docker compose exec -T db sh -c \
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"' 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"'
``` ```
> **A down migration is only valid PAIRED WITH A CODE ROLLBACK — it is not a standalone repair.**
> `Upload::create` sends an `ON CONFLICT ... WHERE` predicate that must match the live partial
> index exactly, and these queries are not compile-checked. Run **026**'s or **031**'s down against
> the current binary and every upload carrying a `client_upload_id` — i.e. every upload from the
> shipped client — becomes a runtime 500. Roll the image back first, then the migration.
>
> **026's down can also fail outright, and that is expected.** It restores a wider unique index, so
> it aborts with `could not create unique index ... is duplicated` on any database where a guest
> ever deleted a photo and re-uploaded it. The transaction rolls back cleanly and the narrow index
> survives intact — no half-state — but you cannot go below 026 on a database that has seen real
> use. Verified against a live Postgres.
Migration **014** is the only destructive one on the way up, and the only one with a rehearsal Migration **014** is the only destructive one on the way up, and the only one with a rehearsal
harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump before the event. harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump before the event.