diff --git a/.env.example b/.env.example index 039164d..ecc9fc1 100644 --- a/.env.example +++ b/.env.example @@ -19,7 +19,7 @@ DOMAIN=my-event.example.com # Copying this file and starting the stack without that step fails with `manifest unknown`. # # Do NOT "fix" this by dropping back to v0.12.0: no image was ever built for it, and a -# 6-migration tree booting against a 22-migration database returns VersionMissing and +# 6-migration tree booting against a 31-migration database returns VersionMissing and # crash-loops forever behind a live Caddy. §9 covers this in full. EVENTSNAP_VERSION=v0.13.0 diff --git a/Caddyfile b/Caddyfile index 226f5dc..881c7b1 100644 --- a/Caddyfile +++ b/Caddyfile @@ -10,10 +10,19 @@ # # read_header is tight: a legitimate client sends its headers in one go. read_header 10s - # read_body is NOT set, and idle is generous: a guest pushing a 500 MB video over - # cellular legitimately takes many minutes, and a body timeout would fail exactly the - # uploads this product exists to collect. The header timeout is what stops the cheap - # attack; a slow *body* still has to actually send bytes. + # read_body is GENEROUS but present. It was omitted on the reasoning that "a slow body + # still has to actually send bytes" — which is an argument about disk, and disk is not + # the scarce resource here. `upload_admission` budgets concurrent bodies at 4096 MiB and + # reserves the DECLARED cap, so a `video/*` upload reserves 500 MiB: eight connections + # that stall mid-body hold the entire budget, every other guest waits 20s and gets a + # 503, and it never recovers on its own because the permit is held until the handler + # returns. That needs no attacker — eight guests starting real videos and then walking + # out of AP range does it, and TCP will not reap those sockets for hours. + # + # 30m carries a 500 MB video at ~2.2 Mbit/s sustained, which is well under venue wifi + # and under most cellular, so it does not fail the uploads this product exists to + # collect. It does bound the leak to something that drains. + read_body 30m idle 5m } } diff --git a/DEPLOYMENT_RUNBOOK.md b/DEPLOYMENT_RUNBOOK.md index 2318c3f..5089bbb 100644 --- a/DEPLOYMENT_RUNBOOK.md +++ b/DEPLOYMENT_RUNBOOK.md @@ -537,9 +537,9 @@ explanation: ``` $ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l -12 # 6 migrations. HEAD has 22. +12 # 6 migrations. HEAD has 31. $ git rev-list --count v0.12.0..HEAD -154 +196 ``` `db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration @@ -591,7 +591,7 @@ covers both. It exists so that the rollback line in the emergency card is safe t catastrophic. **If you genuinely need to undo a code change during the event, you cannot; freeze early enough that you never have to.** -**Across a migration boundary — avoid by freezing.** If you must: all 22 migrations have paired +**Across a migration boundary — avoid by freezing.** If you must: all 31 migrations have paired `.down.sql` files, but **none of them removes its own `_sqlx_migrations` row**, so that second step is mandatory and undocumented: @@ -799,23 +799,29 @@ differently, the next boot aborts with *"migration 21 was previously applied but modified"*, `main` exits non-zero, and `restart: unless-stopped` makes it **permanent** — with Caddy still routing traffic to the dead container. -Main-line `021`–`023` are byte-identical to what shipped, so a box that only ever ran tagged +Every main-line migration is byte-identical to what shipped, so a box that only ever ran tagged releases is unaffected. **Verify rather than assume** — run this against the server before any -deploy: +deploy. + +Note the `sh -c` wrapping, for the same reason as §9: `POSTGRES_USER` and `POSTGRES_DB` live in +`.env`, which Compose reads and **your shell does not**. Unwrapped, `-U "$POSTGRES_USER"` sends +`-U ""` and psql answers `FATAL: role "" does not exist` — at 11pm, with the app crash-looping. ```bash -docker compose exec -T db psql -U "$POSTGRES_USER" "$POSTGRES_DB" -c \ - "SELECT version, description, success FROM _sqlx_migrations ORDER BY version;" +docker compose exec -T db sh -c \ + 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version;"' ``` If the app is already crash-looping on a renumbered migration, and **only** if you have confirmed -the SQL in the new file is equivalent to what was actually applied: +the SQL in the new file is equivalent to what was actually applied. Take the version numbers from +the crash message and the query above — do **not** copy the ones below, which are an example: ```bash docker compose stop app -docker compose exec -T db psql -U "$POSTGRES_USER" "$POSTGRES_DB" -c \ - "DELETE FROM _sqlx_migrations WHERE version IN (21,22,23);" -docker compose start app # re-applies 021-023, then continues +# Replace 21,22,23 with the versions the boot error actually named. +docker compose exec -T db sh -c \ + 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version IN (21,22,23);"' +docker compose start app # re-applies exactly those, then continues ``` This re-runs those migrations. They must be idempotent (`IF NOT EXISTS` / `IF EXISTS`) or this diff --git a/docker-compose.yml b/docker-compose.yml index 20c2b1f..c035945 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -104,6 +104,14 @@ services: # create it and every upload 500s with EACCES. `environment` overrides `env_file`, # so this is authoritative for the container. MEDIA_PATH: /media + # Third member of the MEDIA_PATH / EXPORT_PATH family, and the nastiest of the three because + # the app itself reports nothing wrong. The healthcheck below hardcodes 127.0.0.1:3000 and + # the Caddyfile hardcodes app:3000, while `.env.example` presents APP_PORT as an ordinary + # editable line. Change it there and the app boots and serves happily on the new port, the + # healthcheck fails forever, `app` never turns healthy — and because caddy is gated on + # `service_healthy`, CADDY NEVER STARTS AT ALL. Port 443 is dead for the whole event and the + # only diagnostic is `dependency failed to start`. + APP_PORT: "3000" # Pinned for the same reason as MEDIA_PATH: `environment` beats `env_file`, so this cannot # be lost by an operator who copies `.env.example` and edits only the secrets — which is # the likely path, and `.env.example` ships the generic default of `true`.