fix(deploy): a permanent upload outage, a dead-on-arrival Caddy, and 11pm commands that don't run

* Caddy had no read_body, on the reasoning that "a slow body still has to
  actually send bytes". That is an argument about disk, and disk is not the
  scarce resource: upload_admission budgets concurrent bodies at 4096 MiB and
  reserves the DECLARED cap, so a video/* upload reserves 500 MiB. Eight
  connections that stall mid-body hold the whole budget, every other guest
  waits 20s and gets a 503, and it never recovers on its own — the permit is
  held until the handler returns. No attacker needed: eight guests starting
  real videos and walking out of AP range does it, and TCP will not reap
  those sockets for hours. 30m carries a 500 MB upload at ~2.2 Mbit/s, so it
  does not fail the uploads this product exists to collect.

* APP_PORT is presented in .env.example as an ordinary editable line, while
  the healthcheck hardcodes 127.0.0.1:3000 and the Caddyfile hardcodes
  app:3000. Change it and the app boots and serves happily on the new port,
  the healthcheck fails forever, app never turns healthy — and because caddy
  is gated on service_healthy, CADDY NEVER STARTS. Port 443 dead for the
  whole event, sole diagnostic "dependency failed to start". Pinned in
  compose beside MEDIA_PATH and EXPORT_PATH, which are there for this reason.

* Runbook §12's recovery commands do not run as written: unwrapped
  "$POSTGRES_USER" is expanded by the operator's shell, which does not have
  it, so psql answers `FATAL: role "" does not exist`. §9 documents that trap
  two hundred lines earlier and wraps its own calls in sh -c; §12 did not.
  This is the block you run with the app crash-looping behind a live Caddy.
  Its DELETE also hard-coded versions 21,22,23 as if to be copied verbatim,
  on a tree that now has 31 migrations — now explicitly an example, with the
  instruction to take the numbers from the actual boot error.

* Migration counts corrected across the runbook and .env.example (22 -> 31,
  commit count 154 -> 196). All four were presented as literal command output
  the operator is invited to reproduce.
This commit is contained in:
fabi
2026-08-12 09:15:55 +02:00
parent 4b61f4552b
commit f403222200
4 changed files with 39 additions and 16 deletions

View File

@@ -19,7 +19,7 @@ DOMAIN=my-event.example.com
# Copying this file and starting the stack without that step fails with `manifest unknown`.
#
# Do NOT "fix" this by dropping back to v0.12.0: no image was ever built for it, and a
# 6-migration tree booting against a 22-migration database returns VersionMissing and
# 6-migration tree booting against a 31-migration database returns VersionMissing and
# crash-loops forever behind a live Caddy. §9 covers this in full.
EVENTSNAP_VERSION=v0.13.0