fix(deploy): ship the swap ceilings, pin the last boot-fatal env var, and correct docs that misdirect

* memswap_limit is now IN docker-compose.yml on all four services. Compose
  sets Memory but leaves MemorySwap unset, and Docker then permits swap equal
  to the memory limit — so following §5's "add 2 GB of swap" silently DOUBLED
  every ceiling, to ~5 GiB on a 3.82 GiB box. Nothing OOMs; instead Postgres's
  working set becomes swap-eligible on a shared-tenancy SSD, turning a bounded
  OOM-kill that restarts in seconds into unbounded latency with no signal but
  "everything is slow". The runbook told the operator to hand-add it, which
  also broke §0's own gate that docker-compose.yml must be unmodified.
  Verified rather than assumed: service-level memswap_limit does compose with
  deploy.resources.limits.memory (docker inspect → Memory=1073741824
  MemorySwap=1207959552).

* DATABASE_MAX_CONNECTIONS pinned in compose. It is the one env var that is
  now boot-FATAL when unparseable — the right call, but it means a stray quote
  or a trailing inline comment in .env crash-loops the app behind a live
  Caddy. MEDIA_PATH, EXPORT_PATH and APP_PORT are pinned for weaker reasons.

* .env.example's quota narrative was sized for a CX33: "~30 GB of a fresh
  70 GB" on a box with 40 GB. And on THIS box the fixed point never binds at
  all — ~210 MB/guest is below the 500 MiB floor, so everyone gets the floor
  and the per-user quota stops bounding aggregate growth. What actually stops
  uploads is the keepsake preflight at ~8 GB of media. That paragraph is what
  an operator reads when a guest is blocked, and it pointed at the wrong knob.

* The emergency card gains the one disk symptom that can appear mid-event,
  where `df -h` — its only disk instruction — actively misleads: the gate
  fires ~10 GB + 2.2x media BEFORE the disk is full, so df shows ~20 GB free
  at the moment uploads are being refused.

* Two code comments that now assert the opposite of the code: claim_job
  promised that "the update_progress liveness check bails such a worker out
  early" — it cannot, its predicate is on the job row, which a reopen does not
  touch, so a mid-export reopen grinds the whole gallery to completion on a
  2-vCPU box during the live event. And prune_superseded_archives still argued
  "deleted bytes cannot be rolled back" as an invariant, after the reclaim
  path was changed to prune even when that will not close the shortfall.
  Both now describe what the code does.

* Smaller corrections: runbook §3's "two 48 MP photos ≈ 800 MB" scenario is
  unreachable (compression.rs takes an exclusive heavy permit, so they
  serialise) and contradicted .env.example; "all four healthy" is wrong since
  caddy has no healthcheck; a README line reference pointed at a comment added
  by the same commit that broke it.
This commit is contained in:
fabi
2026-08-12 19:10:45 +02:00
parent 182e712a0e
commit 4916eed436
5 changed files with 144 additions and 31 deletions

View File

@@ -64,6 +64,22 @@ services:
# Relative CPU weight under contention (Docker default is 1024). Only consulted when the
# CPU is actually saturated, which is exactly the moment the database must not lose.
cpu_shares: 2048
# Caps memory+swap together, so the `memory` limit above stays the real ceiling.
#
# Compose sets `Memory` but leaves `MemorySwap` unset, and Docker then permits swap EQUAL to
# the memory limit — so following the runbook's "add 2 GB of swap" step silently DOUBLES every
# container ceiling, to 5 GiB of ceilings on a 3.82 GiB box. Nothing OOMs; instead Postgres's
# working set becomes swap-eligible on a shared-tenancy VPS SSD, turning a bounded OOM-kill
# (which restarts in seconds) into unbounded latency everywhere with no signal but "it's slow".
#
# The runbook used to tell the operator to add this BY HAND, which also broke its own §0 gate
# requiring docker-compose.yml to be unmodified. Shipped here instead. 1152m against a 1G limit
# leaves 128 MB of swap — enough to absorb a spike, not enough to hide one.
#
# Verified rather than assumed: service-level `memswap_limit` DOES compose with
# `deploy.resources.limits.memory` — `docker inspect` reports Memory=1073741824
# MemorySwap=1207959552.
memswap_limit: 1152m
app:
# Production PULLS a prebuilt image; it never compiles. A release build of this crate is
@@ -112,6 +128,16 @@ services:
# `service_healthy`, CADDY NEVER STARTS AT ALL. Port 443 is dead for the whole event and the
# only diagnostic is `dependency failed to start`.
APP_PORT: "3000"
# Fourth member of the family, pinned for a reason the other three don't have: this one is
# boot-FATAL. `db.rs` rejects an unparseable value with `bail!` rather than falling back to
# the default (right call — an operator tuning a knob that silently never applied is worse),
# which means a stray quote, a trailing inline comment, or a smart-quote pasted into `.env`
# no longer degrades anything: it exits 1, and `restart: unless-stopped` crash-loops the app
# behind a live Caddy. `.trim()` covers whitespace and CRLF; it cannot cover those.
#
# Sized to the 2 vCPU this box has, not to the guest count — see `.env.example` and the
# `db` memory limit, which must be raised together with this.
DATABASE_MAX_CONNECTIONS: "15"
# Pinned for the same reason as MEDIA_PATH: `environment` beats `env_file`, so this cannot
# be lost by an operator who copies `.env.example` and edits only the secrets — which is
# the likely path, and `.env.example` ships the generic default of `true`.
@@ -161,6 +187,9 @@ services:
# Compression is throughput work with no guest waiting on it, so it yields to Postgres —
# which every request path, including the app's own, is blocked on.
cpu_shares: 512
# See the `db` service for why this is shipped rather than hand-added: without it, the
# runbook's swap step doubles this ceiling. 1152m against a 1G limit.
memswap_limit: 1152m
frontend:
# Pulled, not built — see the note on `app` above.
@@ -207,6 +236,9 @@ services:
# and then guests talk to `app` directly. A slow shell delays a reload; a slow database
# breaks the event.
cpu_shares: 256
# See the `db` service for why this is shipped rather than hand-added: without it, the
# runbook's swap step doubles this ceiling. 320m against a 256M limit.
memswap_limit: 320m
caddy:
image: caddy:2-alpine
@@ -239,6 +271,9 @@ services:
# Left at the Docker default (1024). Caddy is cheap but sits in front of everything, so
# it must not be the bottleneck; it is capped at 0.5 vCPU regardless.
cpu_shares: 1024
# See the `db` service for why this is shipped rather than hand-added: without it, the
# runbook's swap step doubles this ceiling. 320m against a 256M limit.
memswap_limit: 320m
volumes:
postgres_data: