`/health` returned the literal string "ok" and touched nothing. Every request in this
app needs the database, so that answered a question nobody asked: the container
reported healthy while every request 500'd, and with no operator watching during the
event there was no signal at all. It now runs `SELECT 1` with a 2s timeout.
Verified live against the production stack: 200, stop Postgres, 503 "database timeout",
start Postgres, 200 again — with no app restart, because sqlx revalidates on acquire.
Deliberately NOT wired to automatic recovery. Compose's `restart: unless-stopped` does
not react to healthcheck state anyway, and an autoheal sidecar would be actively wrong
here: it would truncate every in-flight upload to "fix" an outage that, as the test
above shows, clears on its own. This is a diagnostic — including for the runbook's
event-day `curl`.
The pool had only `max_connections` set. Three additions:
- `acquire_timeout(5s)`. sqlx defaults to 30, so a DB blip parked every request AND all
~100 SSE session revalidations for half a minute before erroring — the app looked
hung rather than degraded, and the backlog outlived the blip.
- `min_connections(2)`, so the first request after the setup-to-guests-arriving gap
doesn't pay TCP + auth.
- `statement_timeout=15s` / `lock_timeout=5s` per connection. Without them a single
pathological query holds a pool slot indefinitely and no client-side timeout can take
it back, because the slot is only released when Postgres finishes.
Those two SETs are sent as two statements. `sqlx::query` uses the extended query
protocol, which permits exactly one per call — as `SET a; SET b` every new connection
failed, which surfaced as the pool never opening one and `create_pool` reporting a
connect timeout. Caught by booting against an empty database rather than a warm one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
faea555967