fix(deploy): a permanent upload outage, a dead-on-arrival Caddy, and 11pm commands that don't run
* Caddy had no read_body, on the reasoning that "a slow body still has to actually send bytes". That is an argument about disk, and disk is not the scarce resource: upload_admission budgets concurrent bodies at 4096 MiB and reserves the DECLARED cap, so a video/* upload reserves 500 MiB. Eight connections that stall mid-body hold the whole budget, every other guest waits 20s and gets a 503, and it never recovers on its own — the permit is held until the handler returns. No attacker needed: eight guests starting real videos and walking out of AP range does it, and TCP will not reap those sockets for hours. 30m carries a 500 MB upload at ~2.2 Mbit/s, so it does not fail the uploads this product exists to collect. * APP_PORT is presented in .env.example as an ordinary editable line, while the healthcheck hardcodes 127.0.0.1:3000 and the Caddyfile hardcodes app:3000. Change it and the app boots and serves happily on the new port, the healthcheck fails forever, app never turns healthy — and because caddy is gated on service_healthy, CADDY NEVER STARTS. Port 443 dead for the whole event, sole diagnostic "dependency failed to start". Pinned in compose beside MEDIA_PATH and EXPORT_PATH, which are there for this reason. * Runbook §12's recovery commands do not run as written: unwrapped "$POSTGRES_USER" is expanded by the operator's shell, which does not have it, so psql answers `FATAL: role "" does not exist`. §9 documents that trap two hundred lines earlier and wraps its own calls in sh -c; §12 did not. This is the block you run with the app crash-looping behind a live Caddy. Its DELETE also hard-coded versions 21,22,23 as if to be copied verbatim, on a tree that now has 31 migrations — now explicitly an example, with the instruction to take the numbers from the actual boot error. * Migration counts corrected across the runbook and .env.example (22 -> 31, commit count 154 -> 196). All four were presented as literal command output the operator is invited to reproduce.
This commit is contained in:
17
Caddyfile
17
Caddyfile
@@ -10,10 +10,19 @@
|
||||
#
|
||||
# read_header is tight: a legitimate client sends its headers in one go.
|
||||
read_header 10s
|
||||
# read_body is NOT set, and idle is generous: a guest pushing a 500 MB video over
|
||||
# cellular legitimately takes many minutes, and a body timeout would fail exactly the
|
||||
# uploads this product exists to collect. The header timeout is what stops the cheap
|
||||
# attack; a slow *body* still has to actually send bytes.
|
||||
# read_body is GENEROUS but present. It was omitted on the reasoning that "a slow body
|
||||
# still has to actually send bytes" — which is an argument about disk, and disk is not
|
||||
# the scarce resource here. `upload_admission` budgets concurrent bodies at 4096 MiB and
|
||||
# reserves the DECLARED cap, so a `video/*` upload reserves 500 MiB: eight connections
|
||||
# that stall mid-body hold the entire budget, every other guest waits 20s and gets a
|
||||
# 503, and it never recovers on its own because the permit is held until the handler
|
||||
# returns. That needs no attacker — eight guests starting real videos and then walking
|
||||
# out of AP range does it, and TCP will not reap those sockets for hours.
|
||||
#
|
||||
# 30m carries a 500 MB video at ~2.2 Mbit/s sustained, which is well under venue wifi
|
||||
# and under most cellular, so it does not fail the uploads this product exists to
|
||||
# collect. It does bound the leak to something that drains.
|
||||
read_body 30m
|
||||
idle 5m
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user