EventSnap/DEPLOYMENT_RUNBOOK.md

EventSnap — Production Deployment Runbook

Target: Hetzner CX22 (2 vCPU, 4 GB RAM, 40 GB disk), single host, docker compose. Event profile: ~100 guests, ~100 photos + a few videos, one evening, operator attending and unavailable to troubleshoot.

Everything here is written for that last constraint. Where a choice trades throughput for "cannot need attention on the night", it takes the stable option.

Note on this repo's own docs. README.md:62 and PROJECT.md:359 specify a CX33 (4 vCPU / 8 GB / 80 GB). You are deploying to half of that on every axis. Two pieces of tuning advice in .env.example are calibrated for the CX33 and are actively wrong for a CX22 — they are called out in §3. Trust this file over .env.example for sizing.

---

0. Timeline — the single most important control

Step zero: commit everything, before you build anything

The production deployment does not exist in git yet. At the time of writing, docker-compose.yml and .env.example are modified but uncommitted, and DEPLOYMENT_RUNBOOK.md, docker-compose.build.yml, both .dockerignore files and migrations 021/022 are untracked. None of them is gitignored — they are simply not committed.

That is not a tidiness problem, it is the deployment failing in two ways at once:

• §7 tells you to git clone onto the server. git show HEAD:docker-compose.yml still has build: keys and no image: keys, so on that clone docker compose pull skips both services and docker compose up -d starts a fat-LTO release build of 427 crates on the CX22 — the exact scenario §1 rules out as an expected OOM. The committed file also has no log rotation.
• sqlx::migrate!() embeds ./migrations at compile time. A build from the working tree bakes in 021 and 022 and applies them on first boot; any later rebuild from a clean clone produces an image that lacks them and crash-loops with VersionMissing against its own database.

git add docker-compose.yml docker-compose.dev.yml docker-compose.build.yml .env.example \
        backend/.dockerignore frontend/.dockerignore frontend/Dockerfile \
        backend/migrations/021_*.sql backend/migrations/022_*.sql \
        DEPLOYMENT_RUNBOOK.md
git commit -m "chore: production compose, ignore files and runbook"
git push

# Prove it landed — this must print two `image:` lines and nothing about `build:`
git show HEAD:docker-compose.yml | grep -E 'image:|build:'
git show HEAD --stat | grep -c migrations/02   # must be 4

When	What
T‑7 days	Commit and push everything above. Registry + DNS pre-flight (§5). Build and push images (§6).
T‑5 days	First deploy to the server (§7). Verify admin login. Leave it running.
T‑3 days	Freeze migrations. No further code deploys unless something is broken.
T‑2 days	Pre-pull current and previous image tags (§9). Run the backup rehearsal (§10).
Event day	Change nothing. Configuration tweaks via the admin dashboard only (§4).

Why the freeze matters more than anything else here. Migrations run automatically at boot (backend/src/db.rs, create_pool) and sqlx::migrate!() is used without set_ignore_missing. An older image booted against a newer schema does not degrade — it crash-loops with VersionMissing. The repo documents this itself in backend/migrations/014_export_epoch.up.sql:3-8. Once the migration set is frozen, rollback is a one-line .env edit; before it is frozen, rollback means hand-running down-SQL under pressure.

---

1. Deployment strategy — build on the Mac, push, pull

Decision: build linux/amd64 images on the M3 Pro, push to registry.mc02.dev, pull on the server.

Rejected alternatives, with the reason each loses:

Option	Why not
Build on the CX22	backend/Cargo.toml sets lto = true + codegen-units = 1 over 427 crates. Fat LTO links the whole program in one single-threaded process; peak RSS is estimated at 2.5–4 GB against ~1.5–2.5 GB free with the stack running. OOM is the expected outcome, not a tail risk. And rollback would also be a build — 35–60 min under pressure.
CI (Gitea on Pi 5)	Ruled out by you, and correct: a Pi 5 is strictly worse than the CX22 for a fat-LTO link.
True cross-compilation (--target x86_64-unknown-linux-musl)	The dependency graph has four C-compiling crates — ring (per-arch assembly), zstd-sys, libdeflate-sys, libsqlite3-sys — so it needs a real musl cross-toolchain. Alpine has no such package for an arm64 host; the working answer is cargo-zigbuild, which means a new base image and a new toolchain days before an unrepeatable event. Slow but certain beats fast but novel.

Emulated build is the right call because the cost lands where it is free: on your laptop, days early, with nothing depending on it. Estimated wall-clock for the backend is 25–60 min with Rosetta enabled (hours without it) — start it and walk away.

Escape hatch if emulation is unbearable: spin up a temporary Hetzner CPX41 (8 vCPU/16 GB) in the same account, build natively, push, destroy it. Costs cents, same commands, no repo change.

---

2. Repo changes — ALREADY APPLIED

Status: these are done, in the working tree. They are documented here so you know what changed and why, not as work to repeat. Run git diff to review before committing.

2.1 .dockerignore files (were absent)

backend/.dockerignore:

target/

frontend/.dockerignore:

node_modules/
.svelte-kit/
build/

Without these, the first time you run cargo build or npm install locally, every image build ships a multi-GB context to an emulated builder. Worse: frontend/Dockerfile:9 does COPY . . after npm ci, so a macOS node_modules/ would be merged over the container's Linux one.

2.2 Switch compose from build: to image:

In docker-compose.yml, replace the build: block on app and frontend:

  app:
    image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}

  frontend:
    image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}

Two deliberate properties:

• The :? form fails loudly on an unset variable instead of resolving to an empty tag.
• Removing build: entirely is a safety feature. With no build: key on the server, a wrong tag is an instant manifest unknown — never a surprise 45-minute compile on the production box.

New docker-compose.build.yml (opt-in, Mac only — follows the convention stated in the header of docker-compose.dev.yml that overlays are never auto-loaded):

# Build overlay. NOT loaded automatically. Used only where images are BUILT — never on the
# production server, which pulls.
#   docker compose -f docker-compose.yml -f docker-compose.build.yml build
services:
  app:
    build: { context: ./backend, dockerfile: Dockerfile }
  frontend:
    build: { context: ./frontend, dockerfile: Dockerfile }

e2e/docker-compose.test.yml keeps its own build: blocks, so the e2e gate is unaffected.

2.3 Add log rotation to every service

docker-compose.yml currently sets no logging config, so the default json-file driver keeps logs forever, on the same filesystem as Postgres and the media. Add to each of the four services:

    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }

(Equivalently, set it once in /etc/docker/daemon.json — but that restarts the Docker daemon, so do it before the stack is live, not after.)

---

3. .env — production values

# ── Identity ──────────────────────────────────────────────────────────────
DOMAIN=<your domain>
EVENT_NAME=<...>
EVENT_SLUG=<...>

# ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
EVENTSNAP_VERSION=v0.13.0

# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64>
POSTGRES_PASSWORD=<openssl rand -hex 24>
DATABASE_URL=postgres://eventsnap:<SAME PASSWORD>@db:5432/eventsnap
ADMIN_PASSWORD_HASH='<docker run --rm caddy:2-alpine caddy hash-password --plaintext "pw">'

# ── Paths — must match the volume mounts ──────────────────────────────────
MEDIA_PATH=/media          # pinned by compose anyway, but keep consistent
EXPORT_PATH=/exports       # NOT pinned by compose — see the trap in §7.3

# ── Sizing (see the two corrections below) ────────────────────────────────
DATABASE_MAX_CONNECTIONS=30
COMPRESSION_WORKER_CONCURRENCY=2

# ── Comments off, likes + captions on ─────────────────────────────────────
COMMENTS_ENABLED=false

# ── Logging (NEW — production currently defaults to DEBUG) ────────────────
RUST_LOG=eventsnap_backend=info,tower_http=warn

Two corrections to the repo's own advice — do not follow .env.example here

COMPRESSION_WORKER_CONCURRENCY: keep 2. Do NOT raise to 4, and do NOT raise the app memory limit to 2G. .env.example:92-98 justifies both on the premise that "each worker can run an ffmpeg transcode". There is no transcode anywhere in this codebase. services/video.rs::run_ffmpeg runs ffmpeg -ss <t> -i <src> -vframes 1 -vf scale=… — a single poster frame. Video originals are stored and served byte-for-byte.

The real memory consumer is the image path: image 0.25's resize builds an Rgba32F intermediate at 16 bytes/px, sized source_width × target_height, which the 256 MiB decode guard in imaging::decode_limits does not cover. Estimated peak per photo:

Source	Peak (decode + display resize)
12 MP (typical phone)	~145 MB
24 MP (iPhone Pro default)	~223 MB
48 MP ("Max" mode)	~354 MB

At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4 the same pair is ~1.5 GB → OOM. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives.

quota_tolerance: keep 0.75. Raising it does not make anything more generous for a real guest. See §4.

---

4. Admin dashboard configuration

These live in the DB config table, not .env. Changes take effect on the next request — patch_config invalidates the cache synchronously (admin::patch_config). No restart needed. Set them after the first deploy, before the event.

Key	Default	Set to	Why
upload_rate_per_hour	100	1000	A guest multi-selecting 100 photos hits exactly 100. Client backs off and resumes, but this makes it a non-issue.
feed_rate_per_min	60	240	Headroom for reconnect bursts.
social_rate_per_min	120	600	This is the likes limiter — the interaction you're keeping.
export_rate_per_day	3	20	One shared bucket across both archives — enforce_export_rate keys on export:{user_id} regardless of which archive is being fetched. Downloading Gallery.zip + Memories.zip costs 2 of 3; one retry locks a guest out for 24 h. The limit is now charged when the download ticket is minted, so being over it produces a visible German error instead of a tap that silently does nothing.
max_video_size_mb	500	leave at 500	See below.
quota_tolerance	0.75	leave at 0.75	See below.
quota_enabled, storage_quota_enabled, rate_limits_enabled	true	leave on	This is the only disk-full safety net.

Ignore estimated_guest_count and upload_count_quota_enabled. Both are seeded, validated and rendered in the admin UI — and read by no code at all (verified by grep across backend/src). Changing them does nothing. estimated_guest_count in particular does not feed the quota formula.

Why quotas are already as generous as you want

per_user_limit = floor(free_disk × quota_tolerance / max(active_uploaders, 1))

active_uploaders is SELECT COUNT(DISTINCT user_id) FROM upload WHERE deleted_at IS NULL (upload::compute_storage_quota) — people who actually uploaded, not guests who joined. The 70 guests who never upload are not in the denominator; their share flows to the photographers automatically. The redistribution you asked for is already the design.

With ~28 GB free and a realistic 30 people actually uploading, each gets ~700 MB — against an expected ~1.25 GB for the entire event. Nobody will be blocked.

Raising quota_tolerance would only raise the saturation ceiling (media converges to t/(1+t) of free space: 43% at 0.75, 50% at 1.0). It does nothing for a real guest at your volume, and it eats the headroom the keepsake needs — the export preflight reserves media × 1.10 × 2 because Gallery.zip and Memories.zip each store media uncompressed (export.rs — both archives store media uncompressed).

Why max_video_size_mb stays at 500. An earlier draft of this runbook said to lower it to 250, on the grounds that there is no client-side size check. That was wrong on both halves:

• A client guard does exist — frontend/src/routes/upload/+page.svelte rejects anything over HARD_MAX_UPLOAD_BYTES (576 MiB) before a byte leaves the phone, alongside the HEIC reject.
• That guard is a compile-time constant, not the DB value. Lowering max_video_size_mb therefore changes nothing on the client: the picker still accepts the clip, the phone still starts sending it, and the server aborts it partway (stream_field_to_file does stop at the cap, so the AP is not saturated for the full file — but the guest still gets a mid-upload failure where 500 MB would simply have worked).

So lowering it is a pure capacity decision with no UX upside, and at your volume there is no capacity problem to solve. Leave it. If you ever do want a smaller ceiling to bite on the phone, the constant above has to move with it.

---

5. Pre-flight — T‑7 days (do NOT leave this to event week)

Registry

From both the Mac and the server, as the user who will deploy (root's docker login does not help a non-root deploy — credentials go to ~/.docker/config.json):

getent hosts registry.mc02.dev                 # DNS resolves from the server
curl -fsSI https://registry.mc02.dev/v2/       # TLS must be PUBLICLY trusted; Docker rejects
                                               # self-signed without extra daemon config
docker login registry.mc02.dev

Also confirm: the eventsnap repository/namespace exists if your registry requires pre-creation (Harbor does; plain registry:2 does not), and the registry host has ~500 MB free per release.

DNS and TLS — get this wrong and you are locked out for an hour

The DNS A record must point at the CX22 before the first docker compose up -d, because Caddy attempts certificate issuance on boot. Let's Encrypt allows only 5 failed validations per hostname per hour (refilling 1 per 12 min). A misconfigured DNS record plus a few impatient restarts will rate-limit you out of getting a certificate at all.

• Deploy days early so issuance happens with no time pressure.
• Never delete the caddy_data volume — it holds the certificate and the ACME account key.
• Never run docker compose down -v. It destroys postgres_data, media_data, exports_data and caddy_data.

Host preparation

docker compose version          # must be v2.x — the deploy.resources limits need it
free -h && swapon --show        # Hetzner images ship no swap
df -h /var/lib/docker           # want ≥ 25 GB free

Add 2 GB of swap as an OOM cushion — a compression spike that would otherwise kill the container instead swaps out cold pages and merely runs slowly:

fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf

Gotcha: Compose sets each container's Memory limit but leaves MemorySwap unset, and Docker then allows swap equal to the memory limit — so adding host swap silently doubles every container ceiling. If you add swap, also add memswap_limit: 1152m to app and db, and memswap_limit: 320m to frontend and caddy (service-level, not under deploy:).

---

6. Build and push — on the Mac

Enable Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation", and set Resources → Memory ≥ 8 GB (the fat-LTO step will OOM inside the VM otherwise, even with 18 GB on the host).

docker run --rm --platform linux/amd64 alpine uname -m     # must print x86_64

cd /Users/fabianhammprivat/Projects/EventSnap
VERSION=v0.13.0                          # latest existing tag is v0.12.0 — see §9 before reusing it
ROLLBACK=v0.13.0-a                       # the SAME source, tagged twice; §9 explains why
SHA=$(git rev-parse --short HEAD)

docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/app:$SHA \
  --push ./backend

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/frontend:$VERSION \
  -t registry.mc02.dev/eventsnap/frontend:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/frontend:$SHA \
  --push ./frontend

Verify the architecture — never skip this

An arm64 image pulls fine and then dies with exec format error. Catch it here, not on the server:

docker buildx imagetools inspect registry.mc02.dev/eventsnap/app:$VERSION
docker buildx imagetools inspect registry.mc02.dev/eventsnap/frontend:$VERSION
# Both MUST report  Platform: linux/amd64

Then prove the binary actually executes. AppConfig::from_env() runs before any DB connection (main.rs builds AppConfig before touching the pool), so this needs no database:

docker run --rm --platform linux/amd64 \
  -e APP_ENV=production -e JWT_SECRET=x -e DATABASE_URL=x -e EVENT_SLUG=x \
  registry.mc02.dev/eventsnap/app:$VERSION

Expect the "Refusing to start in production … placeholder" message. That message means the amd64 binary ran. exec format error means the architecture is wrong.

Tag policy: never latest in production. With latest you cannot tell what is running, rollback becomes a registry re-push (impossible if the registry is down — exactly when you need it), and restart: unless-stopped after a reboot is ambiguous. Two immutable tags per build: semver and short SHA. Deploy by semver.

---

7. First deploy — on the server

# Directory name matters: volumes are prefixed with it, and README's backup commands
# hardcode the `eventsnap_` prefix.
git clone <repo> eventsnap && cd eventsnap
cp .env.example .env && nano .env          # every value from §3
docker login registry.mc02.dev
docker compose pull                        # must fully succeed before anything starts

7.1 The secret pre-flight — run this before the first up -d, always

docker compose run --rm --no-deps app

--no-deps means db never starts, so no Postgres data directory is initialised. AppConfig::from_env() runs first and reports every placeholder at once (config::validate_secrets). When the only remaining complaint is a database connection failure, the secrets are good.

Why this step exists. POSTGRES_PASSWORD is applied only at initdb. docker-compose.yml starts db in the same command as app, so a single up -d with a placeholder bakes the wrong password in permanently — the app then loops on password authentication failed, and the only exits are ALTER ROLE or down -v, which deletes the database, the media and the exports. The repo describes this trap at backend/src/db.rs (explain_auth_failure); this command is what avoids it.

7.2 Bring it up

# `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN.
set -a; . ./.env; set +a

docker compose up -d
docker compose logs -f app          # wait for "database connected and migrations applied"
curl -fsS https://$DOMAIN/health    # → ok  (503 means the app is up but the DB is not)

7.3 Verify the three things compose does not pin

MEDIA_PATH is pinned to /media on the app service in docker-compose.yml. Its siblings are not:

docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH

1. DATABASE_URL must contain @db:5432. A dev .env points it at @localhost, which inside the container is the app itself.
2. EXPORT_PATH must be /exports. Anywhere else and the keepsake archives are written to the container's writable layer and vanish on the next up -d — including on a rollback.
3. ADMIN_PASSWORD_HASH must match .env byte for byte.

Then actually log in to /admin with the real password. This is not optional politeness:

• The production secret guard only rejects placeholders (config::looks_placeholder). A hash corrupted by shell or Compose escaping is not a placeholder — the app boots green, /health says ok, and every admin login 401s.
• The Admin user row is created by a successful admin login (auth::handlers::admin_login), and promoting anyone to Host requires an Admin or Host (auth::middleware's role guard). No admin login ⇒ no host, ever ⇒ you cannot close the event, release the gallery, ban anyone, reset a PIN, or change any limit — for the whole event.

Per the Compose spec, single-quoted env_file values are used literally and the quotes are stripped, so the single-quote form in .env.example is correct. The comment in docker-compose.dev.yml claiming production has the same bug is stale. Verify anyway — the cost of checking is 10 seconds; the cost of being wrong is the whole event.

Promote a second person to Host once you are in, so a single lost session is not fatal.

---

8. Post-deploy verification

set -a; . ./.env; set +a                                     # $DOMAIN comes from .env, not your shell

docker compose ps                                            # all four healthy
docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1   # must be 1073741824, not 0
curl -fsS https://$DOMAIN/health                             # ok — now a real DB check, not a constant
docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
docker builder prune -af && docker image prune -f            # reclaim build cache
df -h /var/lib/docker

Then, from a phone on cellular (not the office wifi):

• Join as a guest with a PIN
• Upload a photo → appears in the feed within a few seconds
• Upload a video → plays back (iOS Safari range requests)
• Add a caption, add a like — no comment UI anywhere
• Admin login works; host dashboard reachable
• Release the gallery on a test event and download both archives

---

9. Rollback

There is no older image you can roll back to. Build the rollback target yourself.

Read this before the event, not during it. The obvious move — drop EVENTSNAP_VERSION back to the previous released tag — takes the app down permanently and looks like a crash loop with no explanation:

$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
12          # 6 migrations. HEAD has 22.
$ git rev-list --count v0.12.0..HEAD
154

db.rs runs sqlx::migrate!() with no set_ignore_missing, so an image built from a 6-migration tree, booting against a database that already carries versions 007–022, returns VersionMissing. create_pool errors, main exits 1, and restart: unless-stopped restarts it forever — with Caddy still routing traffic to it. (014_export_epoch.up.sql documents this failure mode; §0 restates it.) No v0.12.0 image was ever built or pushed either, so the pre-pull would fail with manifest unknown before you ever got that far.

So: at build time, tag the SAME frozen commit twice. Two identical images, two names. The rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical migration set, which makes it a genuine no-op rather than a gamble:

# In §6, push both tags from the one build:
VERSION=v0.13.0
ROLLBACK=v0.13.0-a          # same source, different name — the rollback target

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  --push ./backend
# ...and the same two tags for ./frontend

Rolling back — ~30 seconds, no network:

sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env
docker compose up -d app frontend

This only works offline if both are already resident. Pre-pull all four at T‑2:

docker pull registry.mc02.dev/eventsnap/app:v0.13.0
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a
docker image ls | grep eventsnap        # confirm all four

Once resident, up -d, reboots, restarts and rollbacks need zero registry contact. That single step makes a registry outage on event day irrelevant.

Be clear-eyed about what this buys you: an identical image cannot undo a bad release, only an image that got corrupted or a container that wedged — and docker compose restart app already covers both. It exists so that the rollback line in the emergency card is safe to run rather than catastrophic. If you genuinely need to undo a code change during the event, you cannot; freeze early enough that you never have to.

Across a migration boundary — avoid by freezing. If you must: all 22 migrations have paired .down.sql files, but none of them removes its own _sqlx_migrations row, so that second step is mandatory and undocumented:

docker compose stop app
# `sh -c` so the CONTAINER expands the credentials. They live in the container's environment
# and in .env — not in your shell — so an unwrapped `-U "$POSTGRES_USER"` sends `-U ""` and
# psql answers `FATAL: role "" does not exist`.
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' \
  < backend/migrations/0NN_x.down.sql
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"'

Migration 014 is the only destructive one on the way up, and the only one with a rehearsal harness — backend/scripts/rehearse-014.sh. Run it once against a real dump before the event.

Registry-down transport fallback (layers are already compressed — do not add gzip):

docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
            registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load'

---

10. Backup

Full commands are in README.md:315-435 and are correct — pg_dump --clean --if-exists, plus alpine tar out of eventsnap_media_data and eventsnap_exports_data, mounted at /src (not /media), with chown -R 100:101 on restore because the app runs non-root and BusyBox tar has no --same-owner. There is deliberately no script.

Two gaps the README does not cover:

1. Nothing backs up .env, which holds the only copy of POSTGRES_PASSWORD. A dump you cannot authenticate against is not a backup. Copy .env off the box, encrypted, once it is final.
2. Timing. Do not use nightly cron — every irreplaceable byte is created inside one evening. Take the dump and the media tarball back-to-back the night of the event, after locking uploads from the host dashboard, so the pair is consistent.

---

11. Disk — the numbers for 40 GB

Expected event (100 photos @ ~4 MB, 5 videos @ ~150 MB). Originals are always kept and never lossily recompressed; derivatives are a ≤800 px preview and a ≤2048 px display JPEG.

media (originals + derivatives)                    ~1.25 GB
peak during keepsake build (+ Gallery + Memories)  ~3.5 GB
OS + Docker images + Postgres + logs               ~4.5 GB
────────────────────────────────────────────────────────────
total                             ~11–13 GB of ~36 GB usable

40 GB fits with roughly 3× headroom, provided you build elsewhere (a server-side build adds 3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against live free space on every upload).

The README.md:295-299 "ENOSPC" projection models guests saturating the quota (~12 GB of media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully: the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard warns when free space drops below 10 GB or below the keepsake requirement (handlers::host's low-disk thresholds).

---

12. Known issues you are shipping with

None of these has a fix in this runbook; they are listed so nothing is a surprise. Severity is scored purely by "would this interrupt you during the event".

Fixed in this pass

Was	Fix
Queued uploads never rehydrated — loadQueue() had one call site, so a guest whose PWA was evicted mid-upload and reopened onto /feed had pending photos in IndexedDB that nothing ever read. Silent photo loss.	loadQueue() now runs on every authenticated boot in +layout.svelte.
A corrupted ADMIN_PASSWORD_HASH booted green and only failed at admin login — unrecoverable mid-event.	config.rs now validates the bcrypt shape, not just placeholder-ness, and refuses to start.
SSE reconnect thundering herd — flat 500 ms jitter regardless of backoff.	Jitter now scales with the delay; the feed's in-place refresh is spread over 800–2800 ms.
No client-side size/HEIC pre-check — a doomed 600 MB upload saturated the venue AP before being rejected.	Certain-rejects are refused before any bytes leave the phone.
Upload-queue UI unreachable — the badged FAB opened the picker, not the queue, so a failed upload had no retry button.	The upload sheet now shows a queue entry whenever the badge is non-zero.
A mid-event 401 left a dead screen with no nav and no URL bar.	api.ts now returns the guest to /join (skipping the auth routes themselves).
Rate limiter could panic while holding its mutex (timestamps[0] with max == 0), poisoning it process-wide.	Uses first() with a fail-open guard.
Hashtag deadlock — edit_upload and add_comment upserted tags in client/text order.	Both now sort + dedup on the normalised key, matching the upload path.
Unbounded Docker logs + debug-level app logging.	logging: caps every service at 30 MB; RUST_LOG documented in .env.example.

Fixed in the readiness pass — behaviour you should know about

These change what you will observe on the night, so they are listed separately from the table above.

Was	Now
Any ffmpeg-level failure DESTROYED the video. A poster-frame error — ffmpeg hanging on a truncated .mov, an ENOSPC on thumbnails/, a DB blip — propagated into the give-up path, which soft-deleted the upload. Reproduced live: on a host with no ffmpeg the clip was gone ~6 s after its 201 Created.	No failure in the video branch can fail the upload. The clip stays in the feed and plays from its original; only the poster is missing.
A lost response duplicated the photo. Every retry minted a fresh upload id, so a phone that lost the reply and re-sent — manually, or automatically on reconnect — stored the same photo two or three times and paid quota for each.	The client sends a client_upload_id; migration 022 makes it unique. A retry returns 200 with the original row. Verified live: three sends → one row, quota charged once.
/health returned a constant "ok". The container reported healthy while every request 500'd.	Runs SELECT 1 with a 2 s timeout: 200 ok / 503. Verified live: 200 → stop Postgres → 503 → start Postgres → 200, with no app restart (the pool revalidates on acquire). Note that Compose does not restart on an unhealthy probe — this is a diagnostic, deliberately not wired to automatic recovery, because a restart would truncate in-flight uploads to "fix" an outage that clears on its own.
Abandoned .tmp uploads were never reclaimed (see the old issue #2 below — it is now fixed, not deferred).	Swept at boot and hourly, keyed on modification time so a live upload can never age into it.
A full disk made itself worse. ENOSPC was retried three times, then refunded the quota, soft-deleted the row and kept the bytes — freeing nothing and inviting an immediate re-upload into the same full disk.	ENOSPC is classified separately: no retry, no refund, no delete. The row stays live and the photo is served from its original until there is room to compress it.
The keepsake download failed invisibly. The rate limit was enforced inside the iframe navigation, so an over-limit guest tapped and nothing happened, forever.	Charged when the ticket is minted — a normal fetch — so it surfaces as a German message naming the daily window. Raise export_rate_per_day per §4 anyway.
/host and /admin subscribed to SSE but never opened the connection, so the keepsake progress bar froze after a release.	Both connect on mount and disconnect on destroy, as /export already did.
?limit=-5 on the feed returned a 500.	Clamped at both ends.
All recurring hygiene lived in one unsupervised task — one panic silently stopped session pruning, media reclamation and the temp sweep for the rest of the event.	Supervised and re-spawned, with an error log.
No pool acquire or statement timeout. A DB blip parked every request for 30 s.	5 s acquire, statement_timeout=15s, lock_timeout=5s.

Still open — accepted for this event

#	Issue	Impact
1	Guest delete/caption-edit after release invalidates both keepsake archives (upload.rs), forcing a rebuild with a 20 s debounce. No-op before release, so it cannot bite during the event.	Post-event only. Left alone because blocking guest deletes has real privacy downsides — that is a product call, not a bug fix.
2	Video poster frames do not regenerate after a restart mid-compression (the backfill filters mime_type LIKE 'image/%').	Cosmetic — the video still plays; only its poster is missing.
3	SSE keep-alives are sent as SSE comments (:ping), which the browser's EventSource parser discards without dispatching. A client therefore cannot implement a pure silence timer to detect a half-open socket.	Worked around client-side: the feed runs a jittered 60–120 s /feed/delta backstop and reconnects when a poll returns content the stream never delivered. A cleaner fix is to emit keep-alives as a named event; that is a coordinated backend+frontend change, not worth making during a freeze.
4	The lightbox stops at the end of the loaded page — stepping past the last loaded photo does not fetch the next one.	The guest scrolls the feed (which does page) and re-opens.

---

13. Event-day emergency card

Assume you have a phone and two minutes.

cd ~/eventsnap

# FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this,
# every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage
# when the site is fine.
set -a; . ./.env; set +a

# Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not)
curl -fsS https://$DOMAIN/health

# What is broken?
docker compose ps
docker compose logs --tail=100 app

# Nuclear option that is SAFE (keeps all data):
docker compose restart app

# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
# Do NOT substitute an older release tag here; it will crash-loop on the migration set.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend

# Disk check
df -h /var/lib/docker

NEVER run docker compose down -v. It deletes the database, all media, all exports and the TLS certificate. There is no undo.

Most limits are changeable from the admin dashboard without a restart — reach for that before touching the shell.