fix(ops): make /health a readiness probe, and bound how long the pool waits
`/health` returned the literal string "ok" and touched nothing. Every request in this app needs the database, so that answered a question nobody asked: the container reported healthy while every request 500'd, and with no operator watching during the event there was no signal at all. It now runs `SELECT 1` with a 2s timeout. Verified live against the production stack: 200, stop Postgres, 503 "database timeout", start Postgres, 200 again — with no app restart, because sqlx revalidates on acquire. Deliberately NOT wired to automatic recovery. Compose's `restart: unless-stopped` does not react to healthcheck state anyway, and an autoheal sidecar would be actively wrong here: it would truncate every in-flight upload to "fix" an outage that, as the test above shows, clears on its own. This is a diagnostic — including for the runbook's event-day `curl`. The pool had only `max_connections` set. Three additions: - `acquire_timeout(5s)`. sqlx defaults to 30, so a DB blip parked every request AND all ~100 SSE session revalidations for half a minute before erroring — the app looked hung rather than degraded, and the backlog outlived the blip. - `min_connections(2)`, so the first request after the setup-to-guests-arriving gap doesn't pay TCP + auth. - `statement_timeout=15s` / `lock_timeout=5s` per connection. Without them a single pathological query holds a pool slot indefinitely and no client-side timeout can take it back, because the slot is only released when Postgres finishes. Those two SETs are sent as two statements. `sqlx::query` uses the extended query protocol, which permits exactly one per call — as `SET a; SET b` every new connection failed, which surfaced as the pool never opening one and `create_pool` reporting a connect timeout. Caught by booting against an empty database rather than a warm one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -54,6 +54,31 @@ pub async fn create_pool(database_url: &str) -> Result<PgPool> {
|
||||
|
||||
let pool = match PgPoolOptions::new()
|
||||
.max_connections(max_connections)
|
||||
// Fail fast instead of parking. sqlx's default is 30s, which on a DB blip means every
|
||||
// request AND all ~100 SSE session revalidations sit on the pool for half a minute
|
||||
// before erroring — the app looks hung rather than degraded, and the backlog outlives
|
||||
// the blip. Five seconds is far longer than a healthy acquire ever takes.
|
||||
.acquire_timeout(std::time::Duration::from_secs(5))
|
||||
// Keep a couple of connections warm so the first request after an idle stretch (the gap
|
||||
// between setting the venue up and the guests arriving) doesn't pay TCP + auth.
|
||||
.min_connections(2)
|
||||
// Bound every statement server-side. Without this a single pathological query holds a
|
||||
// pool slot indefinitely and no client-side timeout can take it back — the slot is only
|
||||
// released when Postgres finishes. `lock_timeout` covers the same hazard for a row lock
|
||||
// contended by, say, a release running against an in-flight upload.
|
||||
.after_connect(|conn, _meta| {
|
||||
Box::pin(async move {
|
||||
// Two statements, two round-trips, deliberately. `sqlx::query` uses the extended
|
||||
// query protocol, which permits exactly ONE statement per call — sending them as
|
||||
// `SET a; SET b` makes every new connection fail, which surfaces as the pool
|
||||
// never opening one at all and `create_pool` reporting a connect timeout.
|
||||
sqlx::query("SET statement_timeout = '15s'")
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
sqlx::query("SET lock_timeout = '5s'").execute(conn).await?;
|
||||
Ok(())
|
||||
})
|
||||
})
|
||||
.connect(database_url)
|
||||
.await
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user