diff --git a/.env.example b/.env.example index ecc9fc1..01d5e6e 100644 --- a/.env.example +++ b/.env.example @@ -45,8 +45,12 @@ DATABASE_URL=postgres://eventsnap:CHANGE_ME_use_a_strong_password@db:5432/events POSTGRES_USER=eventsnap POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password POSTGRES_DB=eventsnap -# Connection pool size. The code default is 10 (backend/src/db.rs) — set it explicitly, -# because a `.env` written by hand from this file's secrets is otherwise silently on 10. +# Connection pool size. The code default is 15 (DEFAULT_MAX_CONNECTIONS in backend/src/db.rs), +# and docker-compose.yml pins this value in `app.environment` so an edit here cannot reach the +# container. That pin is deliberate: since the value became boot-FATAL when unparseable — so an +# operator tuning a knob that never took effect gets told, instead of silently staying on the +# default — a stray quote or a trailing inline comment in `.env` would crash-loop the app behind +# a live Caddy. Change the pin in compose, not this line. # # SIZE IT TO THE CORES, NOT TO THE GUESTS. The earlier advice here was ~30, reasoned from # "~100 guests polling the feed at once" back when a feed page cost ~449 ms and connections @@ -116,15 +120,26 @@ EXPORT_PATH=/exports # (upload::quota_limit_bytes). Earlier drafts of this file and the runbook both omitted # it and told operators it was inert; it is not. # -# It is recomputed against LIVE free space on every upload, so it self-throttles: guests -# converge on a fixed point at tolerance/(1+tolerance) of the free space you started -# with — 43% at 0.75, i.e. ~30 GB of a fresh 70 GB. +# It is recomputed against LIVE free space on every upload, so in principle it self- +# throttles: guests converge on a fixed point at tolerance/(1+tolerance) of the free space +# you started with — 43% at 0.75. # -# Raising it therefore AUTHORISES GUESTS TO FILL MORE OF THE DISK. Setting 0.95 in the -# belief that it means "warn me later" moves the fixed point to ~49% and eats the -# headroom the keepsake needs — and the keepsake needs a lot, because Gallery.zip and -# Memories.zip are each roughly a second copy of every original (both store media -# uncompressed). Budget for media + 2x media, or move exports to their own volume. +# ON THIS BOX THAT FIXED POINT NEVER BINDS, and it is worth knowing which knob actually +# stops the disk filling. The arithmetic above used to be quoted as "~30 GB of a fresh +# 70 GB", which is an 80 GB CX33; this deploys to a CX22 with 40 GB. At ~28 GB free and +# estimated_guest_count = 100 flooring the divisor, the formula yields ~210 MB per guest — +# BELOW the 500 MiB floor — so every guest is granted the floor and the per-user quota +# stops bounding aggregate growth at all. +# +# What actually bounds it is the keepsake preflight in upload.rs: uploads are refused once +# free < media x 1.1 x 2 + 10 GB, which on 40 GB lands at ~8 GB of media (README, "Sizing +# the disk"). So if a guest reports being blocked, the number to look at is total media, +# not this one. +# +# Raising this still AUTHORISES GUESTS TO FILL MORE OF THE DISK on a larger box, and it +# still eats the headroom the keepsake needs — Gallery.zip and Memories.zip are each +# roughly a second copy of every original (both store media uncompressed). Budget for +# media + 2x media, or move exports to their own volume. # # 0.75 is the tested default. Lower it if the box is tight; raise it only if you have # provisioned export headroom separately. diff --git a/DEPLOYMENT_RUNBOOK.md b/DEPLOYMENT_RUNBOOK.md index 5089bbb..4bdf963 100644 --- a/DEPLOYMENT_RUNBOOK.md +++ b/DEPLOYMENT_RUNBOOK.md @@ -222,9 +222,15 @@ guard in `imaging::decode_limits` does not cover. Estimated peak per photo: | 24 MP (iPhone Pro default) | ~223 MB | | 48 MP ("Max" mode) | ~354 MB | -At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4 -the same pair is ~1.5 GB → **OOM**. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB -against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives. +Those are per-photo peaks, and the "two 48 MP photos at once" pair this limit used to be sized +against **is no longer reachable**: `compression.rs` takes an EXCLUSIVE `heavy` permit for a large +decode, so two giants serialise no matter what `COMPRESSION_WORKER_CONCURRENCY` is set to (see +`.env.example`, which makes the same point). The binding case is now one giant (~354 MB) plus the +ordinary working set against the 1 GiB cap, which is comfortable. + +What has not changed is the reason to keep concurrency at 2 and `app` at 1G: at concurrency 4 the +memory arithmetic stops working (app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB +against ~3910 MiB MemTotal — oversubscribed before a single photo arrives). **`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.** See §4. @@ -367,10 +373,19 @@ echo '/swapfile none swap sw 0 0' >> /etc/fstab sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf ``` -> **Gotcha:** Compose sets each container's `Memory` limit but leaves `MemorySwap` unset, and Docker -> then allows swap equal to the memory limit — so adding host swap silently **doubles** every -> container ceiling. If you add swap, also add `memswap_limit: 1152m` to `app` and `db`, and -> `memswap_limit: 320m` to `frontend` and `caddy` (service-level, not under `deploy:`). +> **Already handled — do not hand-edit compose.** Compose sets each container's `Memory` limit but +> leaves `MemorySwap` unset, and Docker then allows swap equal to the memory limit, so adding host +> swap would silently **double** every container ceiling (to ~5 GiB of ceilings on a 3.82 GiB box). +> `docker-compose.yml` now ships `memswap_limit` on all four services — 1152m on `app` and `db`, +> 320m on `frontend` and `caddy` — so this step is safe as written. +> +> This used to say "add it yourself", which also broke §0's own gate that +> `git status --porcelain -- docker-compose.yml` must print nothing. Confirm it is still there: +> +> ```bash +> docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}' +> # 1073741824 1207959552 — the second number MUST be larger than the first but not double it. +> ``` --- @@ -508,7 +523,8 @@ the cost of checking is 10 seconds; the cost of being wrong is the whole event. ```bash set -a; . ./.env; set +a # $DOMAIN comes from .env, not your shell -docker compose ps # all four healthy +docker compose ps # db, app, frontend healthy; caddy has + # no healthcheck and shows only "running" docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1 # must be 1073741824, not 0 curl -fsS https://$DOMAIN/health # ok — now a real DB check, not a constant docker compose exec app printenv COMMENTS_ENABLED RUST_LOG @@ -873,6 +889,24 @@ sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker com df -h /var/lib/docker ``` +### "Der Speicher des Events ist fast voll" — guests cannot upload + +**`df -h` will look fine, and that is not a contradiction.** The upload gate refuses long before the +disk fills: it reserves room for the keepsake, which is roughly a second copy of every original, plus +a 10 GB floor. Uploads stop at **~8 GB of media** on a 40 GB box, when `df` still shows ~20 GB free. + +Check the number that actually binds, not free space: + +```bash +docker compose exec -T db sh -c \ + 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT pg_size_pretty(sum(original_size_bytes)) FROM upload WHERE deleted_at IS NULL;"' +``` + +Mid-event, in order of preference: delete the largest videos from the host dashboard (each frees its +own bytes immediately), or move `exports_data` to a separate volume. Raising `quota_tolerance` will +**not** help — on this box every guest is already on the 500 MiB floor, so that knob is not what is +refusing them (see §4 and `.env.example`). + **NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS certificate. There is no undo. diff --git a/README.md b/README.md index 9397860..c7ac6e8 100644 --- a/README.md +++ b/README.md @@ -151,7 +151,7 @@ Caddy automatically obtains a Let's Encrypt certificate on first start. The app ### Updating an existing deployment > **The event server never compiles.** `app` and `frontend` have **no `build:` key** — they -> pull an immutable tag from the registry (`docker-compose.yml:53` says so explicitly, so that +> pull an immutable tag from the registry (the `app` service in `docker-compose.yml` says so explicitly, so that > a wrong tag fails instantly with `manifest unknown` instead of silently starting a 45-minute > compile on the box guests are using). A `git pull` therefore deploys **nothing** on its own, > and `docker compose up -d --build` **errors** — there is nothing to build. Deploying means diff --git a/backend/src/services/export.rs b/backend/src/services/export.rs index fe1f503..2034cdc 100644 --- a/backend/src/services/export.rs +++ b/backend/src/services/export.rs @@ -674,13 +674,16 @@ async fn ensure_export_space_reclaiming( "pruning the previous {prefix} keepsake will not free the full shortfall on its own; \ reclaiming anyway — it is already unservable, and the sibling half frees the rest" ); + } else { + // `else`, not a second unconditional line: both used to fire in the shortfall case, and + // they read as contradicting each other ("will not free the shortfall" / "reclaiming it + // first") to whoever is reading logs at 2am. + tracing::warn!( + deficit, + reclaimable, + "not enough room to rebuild {prefix} alongside the previous keepsake; reclaiming it first" + ); } - - tracing::warn!( - deficit, - reclaimable, - "not enough room to rebuild {prefix} alongside the previous keepsake; reclaiming it first" - ); prune_superseded_archives(pool, export_path, prefix, event_id, epoch).await; ensure_export_space(pool, event_id, export_path).await } @@ -1566,9 +1569,21 @@ async fn query_hashtags(pool: &PgPool, event_id: Uuid) -> Result Vec { /// a hung ffmpeg — left the event with NO archive at all, which is the one outcome the product /// exists to prevent, at the one moment nobody is watching. /// -/// The single exception is phase 2 of `ensure_export_space_reclaiming`, where the previous -/// generation's bytes are the only way the rebuild can fit at all. Both call sites carry the full -/// reasoning; do not "restore" a pre-build prune on the strength of this function's convenience. +/// The single exception is phase 2 of `ensure_export_space_reclaiming`, and BE PRECISE ABOUT WHAT +/// THAT EXCEPTION NOW COSTS, because the guarantee above is weaker than it reads. That phase used +/// to prune only when the reclaimed bytes would actually close the shortfall. It no longer does: +/// `reclaimable` is scoped to one prefix while `deficit` covers both halves plus the reserve, so on +/// a tight disk each worker measured its own share as insufficient, neither pruned, and every "Neu +/// erzeugen" refused identically — permanently stuck, with dead archives on the volume that nothing +/// would reclaim and nothing could serve. It now prunes anyway and lets the re-check decide. +/// +/// The trade that buys: a rebuild can now delete the last physical copy and THEN fail, which is +/// precisely the "no archive at all" outcome this doc argues against. It is accepted because the +/// refusal it replaces was unrecoverable — deterministic across retries — whereas this failure +/// converges once the sibling worker frees its share. But "an epoch can be rolled back, deleted +/// bytes cannot" is no longer a guarantee this module provides end to end, and a manual +/// `UPDATE event SET export_epoch = ` can no longer rescue that case. +/// +/// Both call sites carry the full reasoning; do not "restore" a pre-build prune on the strength of +/// this function's convenience. /// /// Narrower than [`prune_stale_export_files`] on purpose: FINAL archives only. Those are inert — a /// superseded worker either already renamed its file (and will delete it itself when its guarded diff --git a/docker-compose.yml b/docker-compose.yml index c035945..e41a8b6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -64,6 +64,22 @@ services: # Relative CPU weight under contention (Docker default is 1024). Only consulted when the # CPU is actually saturated, which is exactly the moment the database must not lose. cpu_shares: 2048 + # Caps memory+swap together, so the `memory` limit above stays the real ceiling. + # + # Compose sets `Memory` but leaves `MemorySwap` unset, and Docker then permits swap EQUAL to + # the memory limit — so following the runbook's "add 2 GB of swap" step silently DOUBLES every + # container ceiling, to 5 GiB of ceilings on a 3.82 GiB box. Nothing OOMs; instead Postgres's + # working set becomes swap-eligible on a shared-tenancy VPS SSD, turning a bounded OOM-kill + # (which restarts in seconds) into unbounded latency everywhere with no signal but "it's slow". + # + # The runbook used to tell the operator to add this BY HAND, which also broke its own §0 gate + # requiring docker-compose.yml to be unmodified. Shipped here instead. 1152m against a 1G limit + # leaves 128 MB of swap — enough to absorb a spike, not enough to hide one. + # + # Verified rather than assumed: service-level `memswap_limit` DOES compose with + # `deploy.resources.limits.memory` — `docker inspect` reports Memory=1073741824 + # MemorySwap=1207959552. + memswap_limit: 1152m app: # Production PULLS a prebuilt image; it never compiles. A release build of this crate is @@ -112,6 +128,16 @@ services: # `service_healthy`, CADDY NEVER STARTS AT ALL. Port 443 is dead for the whole event and the # only diagnostic is `dependency failed to start`. APP_PORT: "3000" + # Fourth member of the family, pinned for a reason the other three don't have: this one is + # boot-FATAL. `db.rs` rejects an unparseable value with `bail!` rather than falling back to + # the default (right call — an operator tuning a knob that silently never applied is worse), + # which means a stray quote, a trailing inline comment, or a smart-quote pasted into `.env` + # no longer degrades anything: it exits 1, and `restart: unless-stopped` crash-loops the app + # behind a live Caddy. `.trim()` covers whitespace and CRLF; it cannot cover those. + # + # Sized to the 2 vCPU this box has, not to the guest count — see `.env.example` and the + # `db` memory limit, which must be raised together with this. + DATABASE_MAX_CONNECTIONS: "15" # Pinned for the same reason as MEDIA_PATH: `environment` beats `env_file`, so this cannot # be lost by an operator who copies `.env.example` and edits only the secrets — which is # the likely path, and `.env.example` ships the generic default of `true`. @@ -161,6 +187,9 @@ services: # Compression is throughput work with no guest waiting on it, so it yields to Postgres — # which every request path, including the app's own, is blocked on. cpu_shares: 512 + # See the `db` service for why this is shipped rather than hand-added: without it, the + # runbook's swap step doubles this ceiling. 1152m against a 1G limit. + memswap_limit: 1152m frontend: # Pulled, not built — see the note on `app` above. @@ -207,6 +236,9 @@ services: # and then guests talk to `app` directly. A slow shell delays a reload; a slow database # breaks the event. cpu_shares: 256 + # See the `db` service for why this is shipped rather than hand-added: without it, the + # runbook's swap step doubles this ceiling. 320m against a 256M limit. + memswap_limit: 320m caddy: image: caddy:2-alpine @@ -239,6 +271,9 @@ services: # Left at the Docker default (1024). Caddy is cheap but sits in front of everything, so # it must not be the bottleneck; it is capped at 0.5 vCPU regardless. cpu_shares: 1024 + # See the `db` service for why this is shipped rather than hand-added: without it, the + # runbook's swap step doubles this ceiling. 320m against a 256M limit. + memswap_limit: 320m volumes: postgres_data: