fix(deploy): ship the swap ceilings, pin the last boot-fatal env var, and correct docs that misdirect

* memswap_limit is now IN docker-compose.yml on all four services. Compose
  sets Memory but leaves MemorySwap unset, and Docker then permits swap equal
  to the memory limit — so following §5's "add 2 GB of swap" silently DOUBLED
  every ceiling, to ~5 GiB on a 3.82 GiB box. Nothing OOMs; instead Postgres's
  working set becomes swap-eligible on a shared-tenancy SSD, turning a bounded
  OOM-kill that restarts in seconds into unbounded latency with no signal but
  "everything is slow". The runbook told the operator to hand-add it, which
  also broke §0's own gate that docker-compose.yml must be unmodified.
  Verified rather than assumed: service-level memswap_limit does compose with
  deploy.resources.limits.memory (docker inspect → Memory=1073741824
  MemorySwap=1207959552).

* DATABASE_MAX_CONNECTIONS pinned in compose. It is the one env var that is
  now boot-FATAL when unparseable — the right call, but it means a stray quote
  or a trailing inline comment in .env crash-loops the app behind a live
  Caddy. MEDIA_PATH, EXPORT_PATH and APP_PORT are pinned for weaker reasons.

* .env.example's quota narrative was sized for a CX33: "~30 GB of a fresh
  70 GB" on a box with 40 GB. And on THIS box the fixed point never binds at
  all — ~210 MB/guest is below the 500 MiB floor, so everyone gets the floor
  and the per-user quota stops bounding aggregate growth. What actually stops
  uploads is the keepsake preflight at ~8 GB of media. That paragraph is what
  an operator reads when a guest is blocked, and it pointed at the wrong knob.

* The emergency card gains the one disk symptom that can appear mid-event,
  where `df -h` — its only disk instruction — actively misleads: the gate
  fires ~10 GB + 2.2x media BEFORE the disk is full, so df shows ~20 GB free
  at the moment uploads are being refused.

* Two code comments that now assert the opposite of the code: claim_job
  promised that "the update_progress liveness check bails such a worker out
  early" — it cannot, its predicate is on the job row, which a reopen does not
  touch, so a mid-export reopen grinds the whole gallery to completion on a
  2-vCPU box during the live event. And prune_superseded_archives still argued
  "deleted bytes cannot be rolled back" as an invariant, after the reclaim
  path was changed to prune even when that will not close the shortfall.
  Both now describe what the code does.

* Smaller corrections: runbook §3's "two 48 MP photos ≈ 800 MB" scenario is
  unreachable (compression.rs takes an exclusive heavy permit, so they
  serialise) and contradicted .env.example; "all four healthy" is wrong since
  caddy has no healthcheck; a README line reference pointed at a comment added
  by the same commit that broke it.

.env.example

@@ -45,8 +45,12 @@ DATABASE_URL=postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/events
 POSTGRES_USER=eventsnap
 POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
 POSTGRES_DB=eventsnap
-# Connection pool size. The code default is 10 (backend/src/db.rs) — set it explicitly,
+# Connection pool size. The code default is 15 (DEFAULT_MAX_CONNECTIONS in backend/src/db.rs),
-# because a `.env` written by hand from this file's secrets is otherwise silently on 10.
+# and docker-compose.yml pins this value in `app.environment` so an edit here cannot reach the
+# container. That pin is deliberate: since the value became boot-FATAL when unparseable — so an
+# operator tuning a knob that never took effect gets told, instead of silently staying on the
+# default — a stray quote or a trailing inline comment in `.env` would crash-loop the app behind
+# a live Caddy. Change the pin in compose, not this line.
 #
 # SIZE IT TO THE CORES, NOT TO THE GUESTS. The earlier advice here was ~30, reasoned from
 # "~100 guests polling the feed at once" back when a feed page cost ~449 ms and connections
@@ -116,15 +120,26 @@ EXPORT_PATH=/exports
 # (upload::quota_limit_bytes). Earlier drafts of this file and the runbook both omitted
 # it and told operators it was inert; it is not.
 #
-# It is recomputed against LIVE free space on every upload, so it self-throttles: guests
+# It is recomputed against LIVE free space on every upload, so in principle it self-
-# converge on a fixed point at tolerance/(1+tolerance) of the free space you started
+# throttles: guests converge on a fixed point at tolerance/(1+tolerance) of the free space
-# with — 43% at 0.75, i.e. ~30 GB of a fresh 70 GB.
+# you started with — 43% at 0.75.
 #
-# Raising it therefore AUTHORISES GUESTS TO FILL MORE OF THE DISK. Setting 0.95 in the
+# ON THIS BOX THAT FIXED POINT NEVER BINDS, and it is worth knowing which knob actually
-# belief that it means "warn me later" moves the fixed point to ~49% and eats the
+# stops the disk filling. The arithmetic above used to be quoted as "~30 GB of a fresh
-# headroom the keepsake needs — and the keepsake needs a lot, because Gallery.zip and
+# 70 GB", which is an 80 GB CX33; this deploys to a CX22 with 40 GB. At ~28 GB free and
-# Memories.zip are each roughly a second copy of every original (both store media
+# estimated_guest_count = 100 flooring the divisor, the formula yields ~210 MB per guest —
-# uncompressed). Budget for media + 2x media, or move exports to their own volume.
+# BELOW the 500 MiB floor — so every guest is granted the floor and the per-user quota
+# stops bounding aggregate growth at all.
+#
+# What actually bounds it is the keepsake preflight in upload.rs: uploads are refused once
+# free < media x 1.1 x 2 + 10 GB, which on 40 GB lands at ~8 GB of media (README, "Sizing
+# the disk"). So if a guest reports being blocked, the number to look at is total media,
+# not this one.
+#
+# Raising this still AUTHORISES GUESTS TO FILL MORE OF THE DISK on a larger box, and it
+# still eats the headroom the keepsake needs — Gallery.zip and Memories.zip are each
+# roughly a second copy of every original (both store media uncompressed). Budget for
+# media + 2x media, or move exports to their own volume.
 #
 # 0.75 is the tested default. Lower it if the box is tight; raise it only if you have
 # provisioned export headroom separately.

DEPLOYMENT_RUNBOOK.md

@@ -222,9 +222,15 @@ guard in `imaging::decode_limits` does not cover. Estimated peak per photo:
 | 24 MP (iPhone Pro default) | ~223 MB |
 | 48 MP ("Max" mode) | ~354 MB |
 
-At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4
+Those are per-photo peaks, and the "two 48 MP photos at once" pair this limit used to be sized
-the same pair is ~1.5 GB → **OOM**. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
+against **is no longer reachable**: `compression.rs` takes an EXCLUSIVE `heavy` permit for a large
-against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives.
+decode, so two giants serialise no matter what `COMPRESSION_WORKER_CONCURRENCY` is set to (see
+`.env.example`, which makes the same point). The binding case is now one giant (~354 MB) plus the
+ordinary working set against the 1 GiB cap, which is comfortable.
+
+What has not changed is the reason to keep concurrency at 2 and `app` at 1G: at concurrency 4 the
+memory arithmetic stops working (app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
+against ~3910 MiB MemTotal — oversubscribed before a single photo arrives).
 
 **`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.**
 See §4.
@@ -367,10 +373,19 @@ echo '/swapfile none swap sw 0 0' >> /etc/fstab
 sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf
 ```
 
-> **Gotcha:** Compose sets each container's `Memory` limit but leaves `MemorySwap` unset, and Docker
+> **Already handled — do not hand-edit compose.** Compose sets each container's `Memory` limit but
-> then allows swap equal to the memory limit — so adding host swap silently **doubles** every
+> leaves `MemorySwap` unset, and Docker then allows swap equal to the memory limit, so adding host
-> container ceiling. If you add swap, also add `memswap_limit: 1152m` to `app` and `db`, and
+> swap would silently **double** every container ceiling (to ~5 GiB of ceilings on a 3.82 GiB box).
-> `memswap_limit: 320m` to `frontend` and `caddy` (service-level, not under `deploy:`).
+> `docker-compose.yml` now ships `memswap_limit` on all four services — 1152m on `app` and `db`,
+> 320m on `frontend` and `caddy` — so this step is safe as written.
+>
+> This used to say "add it yourself", which also broke §0's own gate that
+> `git status --porcelain -- docker-compose.yml` must print nothing. Confirm it is still there:
+>
+> ```bash
+> docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'
+> # 1073741824 1207959552 — the second number MUST be larger than the first but not double it.
+> ```
 
 ---
 
@@ -508,7 +523,8 @@ the cost of checking is 10 seconds; the cost of being wrong is the whole event.
 ```bash
 set -a; . ./.env; set +a                                     # $DOMAIN comes from .env, not your shell
 
-docker compose ps                                            # all four healthy
+docker compose ps                                            # db, app, frontend healthy; caddy has
+                                                             # no healthcheck and shows only "running"
 docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1   # must be 1073741824, not 0
 curl -fsS https://$DOMAIN/health                             # ok — now a real DB check, not a constant
 docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
@@ -873,6 +889,24 @@ sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker com
 df -h /var/lib/docker
 ```
 
+### "Der Speicher des Events ist fast voll" — guests cannot upload
+
+**`df -h` will look fine, and that is not a contradiction.** The upload gate refuses long before the
+disk fills: it reserves room for the keepsake, which is roughly a second copy of every original, plus
+a 10 GB floor. Uploads stop at **~8 GB of media** on a 40 GB box, when `df` still shows ~20 GB free.
+
+Check the number that actually binds, not free space:
+
+```bash
+docker compose exec -T db sh -c \
+  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT pg_size_pretty(sum(original_size_bytes)) FROM upload WHERE deleted_at IS NULL;"'
+```
+
+Mid-event, in order of preference: delete the largest videos from the host dashboard (each frees its
+own bytes immediately), or move `exports_data` to a separate volume. Raising `quota_tolerance` will
+**not** help — on this box every guest is already on the 500 MiB floor, so that knob is not what is
+refusing them (see §4 and `.env.example`).
+
 **NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS
 certificate. There is no undo.
 

README.md

@@ -151,7 +151,7 @@ Caddy automatically obtains a Let's Encrypt certificate on first start. The app
 ### Updating an existing deployment
 
 > **The event server never compiles.** `app` and `frontend` have **no `build:` key** — they
-> pull an immutable tag from the registry (`docker-compose.yml:53` says so explicitly, so that
+> pull an immutable tag from the registry (the `app` service in `docker-compose.yml` says so explicitly, so that
 > a wrong tag fails instantly with `manifest unknown` instead of silently starting a 45-minute
 > compile on the box guests are using). A `git pull` therefore deploys **nothing** on its own,
 > and `docker compose up -d --build` **errors** — there is nothing to build. Deploying means

backend/src/services/export.rs

@@ -674,13 +674,16 @@ async fn ensure_export_space_reclaiming(
             "pruning the previous {prefix} keepsake will not free the full shortfall on its own; \
              reclaiming anyway — it is already unservable, and the sibling half frees the rest"
         );
-    }
+    } else {
-
+        // `else`, not a second unconditional line: both used to fire in the shortfall case, and
+        // they read as contradicting each other ("will not free the shortfall" / "reclaiming it
+        // first") to whoever is reading logs at 2am.
         tracing::warn!(
             deficit,
             reclaimable,
             "not enough room to rebuild {prefix} alongside the previous keepsake; reclaiming it first"
         );
+    }
     prune_superseded_archives(pool, export_path, prefix, event_id, epoch).await;
     ensure_export_space(pool, event_id, export_path).await
 }
@@ -1566,9 +1569,21 @@ async fn query_hashtags(pool: &PgPool, event_id: Uuid) -> Result<Vec<(Uuid, Stri
 /// `export_job` (that is the point of the design: one write retires everything). So after a reopen
 /// the row is still `pending` at our epoch and this claim SUCCEEDS: the worker will build an archive
 /// nobody can ever see, because retirement is enforced at READ time (`export_current` requires
-/// `j.epoch = e.export_epoch`), not at write time. That is wasted work, not incorrectness — and the
+/// `j.epoch = e.export_epoch`), not at write time. That is wasted work, not incorrectness. Do not
-/// `update_progress` liveness check bails such a worker out early. Do not "optimise" this into a
+/// "optimise" this into a cross-table check: that is exactly the unsound guard we removed.
-/// cross-table check: that is exactly the unsound guard we removed.
+///
+/// This used to claim that "the `update_progress` liveness check bails such a worker out early".
+/// IT DOES NOT, and it cannot: `update_progress`'s predicate is `epoch = ours AND status =
+/// 'running'` on the JOB ROW, which a reopen does not touch — so the check returns true on every
+/// tick and the worker grinds the whole gallery to completion, every ffmpeg poster and every
+/// Lanczos3 resize, before its `finalize_job` writes `done` at an epoch nothing reads.
+///
+/// The cost is real on a 2-vCPU box: a host reopening the event mid-export — the documented
+/// "oops, one more photo" path — leaves a full export burning CPU and the heavy-image semaphore
+/// DURING the live event, and lands a full-gallery-sized orphan that nothing reclaims until the
+/// next successful build at a higher epoch. Bounded and not corrupting, so it is left as is; but
+/// the mitigation the old comment promised was never there, and anyone sizing this box should
+/// know that.
 ///
 /// Errors are distinguished from a lost claim: silently treating a pool timeout as "someone else
 /// owns it" left the row `pending` at 0% with no live worker and no error — a spinner forever.
@@ -1669,9 +1684,23 @@ async fn protected_files(pool: &PgPool, event_id: Uuid) -> Vec<String> {
 /// a hung ffmpeg — left the event with NO archive at all, which is the one outcome the product
 /// exists to prevent, at the one moment nobody is watching.
 ///
-/// The single exception is phase 2 of `ensure_export_space_reclaiming`, where the previous
+/// The single exception is phase 2 of `ensure_export_space_reclaiming`, and BE PRECISE ABOUT WHAT
-/// generation's bytes are the only way the rebuild can fit at all. Both call sites carry the full
+/// THAT EXCEPTION NOW COSTS, because the guarantee above is weaker than it reads. That phase used
-/// reasoning; do not "restore" a pre-build prune on the strength of this function's convenience.
+/// to prune only when the reclaimed bytes would actually close the shortfall. It no longer does:
+/// `reclaimable` is scoped to one prefix while `deficit` covers both halves plus the reserve, so on
+/// a tight disk each worker measured its own share as insufficient, neither pruned, and every "Neu
+/// erzeugen" refused identically — permanently stuck, with dead archives on the volume that nothing
+/// would reclaim and nothing could serve. It now prunes anyway and lets the re-check decide.
+///
+/// The trade that buys: a rebuild can now delete the last physical copy and THEN fail, which is
+/// precisely the "no archive at all" outcome this doc argues against. It is accepted because the
+/// refusal it replaces was unrecoverable — deterministic across retries — whereas this failure
+/// converges once the sibling worker frees its share. But "an epoch can be rolled back, deleted
+/// bytes cannot" is no longer a guarantee this module provides end to end, and a manual
+/// `UPDATE event SET export_epoch = <n>` can no longer rescue that case.
+///
+/// Both call sites carry the full reasoning; do not "restore" a pre-build prune on the strength of
+/// this function's convenience.
 ///
 /// Narrower than [`prune_stale_export_files`] on purpose: FINAL archives only. Those are inert — a
 /// superseded worker either already renamed its file (and will delete it itself when its guarded

docker-compose.yml

@@ -64,6 +64,22 @@ services:
     # Relative CPU weight under contention (Docker default is 1024). Only consulted when the
     # CPU is actually saturated, which is exactly the moment the database must not lose.
     cpu_shares: 2048
+    # Caps memory+swap together, so the `memory` limit above stays the real ceiling.
+    #
+    # Compose sets `Memory` but leaves `MemorySwap` unset, and Docker then permits swap EQUAL to
+    # the memory limit — so following the runbook's "add 2 GB of swap" step silently DOUBLES every
+    # container ceiling, to 5 GiB of ceilings on a 3.82 GiB box. Nothing OOMs; instead Postgres's
+    # working set becomes swap-eligible on a shared-tenancy VPS SSD, turning a bounded OOM-kill
+    # (which restarts in seconds) into unbounded latency everywhere with no signal but "it's slow".
+    #
+    # The runbook used to tell the operator to add this BY HAND, which also broke its own §0 gate
+    # requiring docker-compose.yml to be unmodified. Shipped here instead. 1152m against a 1G limit
+    # leaves 128 MB of swap — enough to absorb a spike, not enough to hide one.
+    #
+    # Verified rather than assumed: service-level `memswap_limit` DOES compose with
+    # `deploy.resources.limits.memory` — `docker inspect` reports Memory=1073741824
+    # MemorySwap=1207959552.
+    memswap_limit: 1152m
 
   app:
     # Production PULLS a prebuilt image; it never compiles. A release build of this crate is
@@ -112,6 +128,16 @@ services:
       # `service_healthy`, CADDY NEVER STARTS AT ALL. Port 443 is dead for the whole event and the
       # only diagnostic is `dependency failed to start`.
       APP_PORT: "3000"
+      # Fourth member of the family, pinned for a reason the other three don't have: this one is
+      # boot-FATAL. `db.rs` rejects an unparseable value with `bail!` rather than falling back to
+      # the default (right call — an operator tuning a knob that silently never applied is worse),
+      # which means a stray quote, a trailing inline comment, or a smart-quote pasted into `.env`
+      # no longer degrades anything: it exits 1, and `restart: unless-stopped` crash-loops the app
+      # behind a live Caddy. `.trim()` covers whitespace and CRLF; it cannot cover those.
+      #
+      # Sized to the 2 vCPU this box has, not to the guest count — see `.env.example` and the
+      # `db` memory limit, which must be raised together with this.
+      DATABASE_MAX_CONNECTIONS: "15"
       # Pinned for the same reason as MEDIA_PATH: `environment` beats `env_file`, so this cannot
       # be lost by an operator who copies `.env.example` and edits only the secrets — which is
       # the likely path, and `.env.example` ships the generic default of `true`.
@@ -161,6 +187,9 @@ services:
     # Compression is throughput work with no guest waiting on it, so it yields to Postgres —
     # which every request path, including the app's own, is blocked on.
     cpu_shares: 512
+    # See the `db` service for why this is shipped rather than hand-added: without it, the
+    # runbook's swap step doubles this ceiling. 1152m against a 1G limit.
+    memswap_limit: 1152m
 
   frontend:
     # Pulled, not built — see the note on `app` above.
@@ -207,6 +236,9 @@ services:
     # and then guests talk to `app` directly. A slow shell delays a reload; a slow database
     # breaks the event.
     cpu_shares: 256
+    # See the `db` service for why this is shipped rather than hand-added: without it, the
+    # runbook's swap step doubles this ceiling. 320m against a 256M limit.
+    memswap_limit: 320m
 
   caddy:
     image: caddy:2-alpine
@@ -239,6 +271,9 @@ services:
     # Left at the Docker default (1024). Caddy is cheap but sits in front of everything, so
     # it must not be the bottleneck; it is capped at 0.5 vCPU regardless.
     cpu_shares: 1024
+    # See the `db` service for why this is shipped rather than hand-added: without it, the
+    # runbook's swap step doubles this ceiling. 320m against a 256M limit.
+    memswap_limit: 320m
 
 volumes:
   postgres_data: