fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy): ship the swap ceilings, pin the last boot-fatal env var, and correct docs that misdirect

Browse Source 
* memswap_limit is now IN docker-compose.yml on all four services. Compose
  sets Memory but leaves MemorySwap unset, and Docker then permits swap equal
  to the memory limit — so following §5's "add 2 GB of swap" silently DOUBLED
  every ceiling, to ~5 GiB on a 3.82 GiB box. Nothing OOMs; instead Postgres's
  working set becomes swap-eligible on a shared-tenancy SSD, turning a bounded
  OOM-kill that restarts in seconds into unbounded latency with no signal but
  "everything is slow". The runbook told the operator to hand-add it, which
  also broke §0's own gate that docker-compose.yml must be unmodified.
  Verified rather than assumed: service-level memswap_limit does compose with
  deploy.resources.limits.memory (docker inspect → Memory=1073741824
  MemorySwap=1207959552).

* DATABASE_MAX_CONNECTIONS pinned in compose. It is the one env var that is
  now boot-FATAL when unparseable — the right call, but it means a stray quote
  or a trailing inline comment in .env crash-loops the app behind a live
  Caddy. MEDIA_PATH, EXPORT_PATH and APP_PORT are pinned for weaker reasons.

* .env.example's quota narrative was sized for a CX33: "~30 GB of a fresh
  70 GB" on a box with 40 GB. And on THIS box the fixed point never binds at
  all — ~210 MB/guest is below the 500 MiB floor, so everyone gets the floor
  and the per-user quota stops bounding aggregate growth. What actually stops
  uploads is the keepsake preflight at ~8 GB of media. That paragraph is what
  an operator reads when a guest is blocked, and it pointed at the wrong knob.

* The emergency card gains the one disk symptom that can appear mid-event,
  where `df -h` — its only disk instruction — actively misleads: the gate
  fires ~10 GB + 2.2x media BEFORE the disk is full, so df shows ~20 GB free
  at the moment uploads are being refused.

* Two code comments that now assert the opposite of the code: claim_job
  promised that "the update_progress liveness check bails such a worker out
  early" — it cannot, its predicate is on the job row, which a reopen does not
  touch, so a mid-export reopen grinds the whole gallery to completion on a
  2-vCPU box during the live event. And prune_superseded_archives still argued
  "deleted bytes cannot be rolled back" as an invariant, after the reclaim
  path was changed to prune even when that will not close the shortfall.
  Both now describe what the code does.

* Smaller corrections: runbook §3's "two 48 MP photos ≈ 800 MB" scenario is
  unreachable (compression.rs takes an exclusive heavy permit, so they
  serialise) and contradicted .env.example; "all four healthy" is wrong since
  caddy has no healthcheck; a README line reference pointed at a comment added
  by the same commit that broke it.
This commit is contained in:
fabi
2026-08-12 19:10:45 +02:00
parent 182e712a0e
commit 4916eed436
5 changed files with 144 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
DEPLOYMENT_RUNBOOK.md
View File

@@ -222,9 +222,15 @@ guard in `imaging::decode_limits` does not cover. Estimated peak per photo:
| 24 MP (iPhone Pro default) | ~223 MB |
| 48 MP ("Max" mode) | ~354 MB |
At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4
the same pair is ~1.5 GB → **OOM**. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives.
Those are per-photo peaks, and the "two 48 MP photos at once" pair this limit used to be sized
against **is no longer reachable**: `compression.rs` takes an EXCLUSIVE `heavy` permit for a large
decode, so two giants serialise no matter what `COMPRESSION_WORKER_CONCURRENCY` is set to (see
`.env.example`, which makes the same point). The binding case is now one giant (~354 MB) plus the
ordinary working set against the 1 GiB cap, which is comfortable.
What has not changed is the reason to keep concurrency at 2 and `app` at 1G: at concurrency 4 the
memory arithmetic stops working (app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
against ~3910 MiB MemTotal — oversubscribed before a single photo arrives).
**`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.**
See §4.
@@ -367,10 +373,19 @@ echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf
```
> **Gotcha:** Compose sets each container's `Memory` limit but leaves `MemorySwap` unset, and Docker
> then allows swap equal to the memory limit — so adding host swap silently **doubles** every
> container ceiling. If you add swap, also add `memswap_limit: 1152m` to `app` and `db`, and
> `memswap_limit: 320m` to `frontend` and `caddy` (service-level, not under `deploy:`).
> **Already handled — do not hand-edit compose.** Compose sets each container's `Memory` limit but
> leaves `MemorySwap` unset, and Docker then allows swap equal to the memory limit, so adding host
> swap would silently **double** every container ceiling (to ~5 GiB of ceilings on a 3.82 GiB box).
> `docker-compose.yml` now ships `memswap_limit` on all four services — 1152m on `app` and `db`,
> 320m on `frontend` and `caddy` — so this step is safe as written.
>
> This used to say "add it yourself", which also broke §0's own gate that
> `git status --porcelain -- docker-compose.yml` must print nothing. Confirm it is still there:
>
> ```bash
> docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'
> # 1073741824 1207959552 — the second number MUST be larger than the first but not double it.
> ```
---
@@ -508,7 +523,8 @@ the cost of checking is 10 seconds; the cost of being wrong is the whole event.
```bash
set -a; . ./.env; set +a # $DOMAIN comes from .env, not your shell
docker compose ps # all four healthy
docker compose ps # db, app, frontend healthy; caddy has
# no healthcheck and shows only "running"
docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1 # must be 1073741824, not 0
curl -fsS https://$DOMAIN/health # ok — now a real DB check, not a constant
docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
@@ -873,6 +889,24 @@ sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker com
df -h /var/lib/docker
```
### "Der Speicher des Events ist fast voll" — guests cannot upload
**`df -h` will look fine, and that is not a contradiction.** The upload gate refuses long before the
disk fills: it reserves room for the keepsake, which is roughly a second copy of every original, plus
a 10 GB floor. Uploads stop at **~8 GB of media** on a 40 GB box, when `df` still shows ~20 GB free.
Check the number that actually binds, not free space:
```bash
docker compose exec -T db sh -c \
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT pg_size_pretty(sum(original_size_bytes)) FROM upload WHERE deleted_at IS NULL;"'
```
Mid-event, in order of preference: delete the largest videos from the host dashboard (each frees its
own bytes immediately), or move `exports_data` to a separate volume. Raising `quota_tolerance` will
**not** help — on this box every guest is already on the 500 MiB floor, so that knob is not what is
refusing them (see §4 and `.env.example`).
**NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS
certificate. There is no undo.