EventSnap/DEPLOYMENT_RUNBOOK.md

EventSnap — Production Deployment Runbook

Target: Hetzner CX22 (2 vCPU, 4 GB RAM, 40 GB disk), single host, docker compose. Event profile: ~100 guests, ~100 photos + a few videos, one evening, operator attending and unavailable to troubleshoot.

Everything here is written for that last constraint. Where a choice trades throughput for "cannot need attention on the night", it takes the stable option.

Note on this repo's own docs. README.md:62 and PROJECT.md:359 specify a CX33 (4 vCPU / 8 GB / 80 GB). You are deploying to half of that on every axis. Two pieces of tuning advice in .env.example are calibrated for the CX33 and are actively wrong for a CX22 — they are called out in §3. Trust this file over .env.example for sizing.

---

0. Timeline — the single most important control

Step zero: verify the deployment files are committed, before you build anything

Everything the server clones must be in git — §7 tells you to git clone onto the box, so anything living only in your working tree is not part of the deployment. Two failure modes if it is not:

• If the committed docker-compose.yml still carried build: keys and no image: keys, then on that clone docker compose pull would skip both services and docker compose up -d would start a fat-LTO release build of 427 crates on the CX22 — the exact scenario §1 rules out as an expected OOM.
• sqlx::migrate!() embeds ./migrations at compile time. An image built from a working tree with uncommitted migrations bakes them in and applies them on first boot; any later rebuild from a clean clone produces an image that lacks them and crash-loops with VersionMissing against its own database.

As of this writing all of these are committed and the check below passes. Run it anyway — it costs a second and it is the difference between finding this now and finding it at T‑5.

# Every deployment file must be tracked. Prints nothing and exits 0 when correct;
# names the offender and exits non-zero otherwise.
git ls-files --error-unmatch \
  docker-compose.yml docker-compose.dev.yml docker-compose.build.yml .env.example \
  backend/.dockerignore frontend/.dockerignore frontend/Dockerfile \
  DEPLOYMENT_RUNBOOK.md Caddyfile >/dev/null

# No uncommitted edits to them.
git status --porcelain -- docker-compose.yml .env.example Caddyfile DEPLOYMENT_RUNBOOK.md

# The COMMITTED compose must pull, not build: 4 `image:` lines, zero `build:` lines.
git show HEAD:docker-compose.yml | grep -cE '^[[:space:]]*image:'   # must be 4
git show HEAD:docker-compose.yml | grep -cE '^[[:space:]]*build:'   # must be 0

# Every migration in the tree is committed — a build from a dirty tree bakes in extras.
git status --porcelain -- backend/migrations/                       # must print nothing

# The Caddyfile PARSES. Nothing else checks it: the e2e stack mounts `e2e/Caddyfile.test`,
# so the production file is never executed until the real deploy — and a syntax error there
# is total. Caddy exits, `restart: unless-stopped` loops, 443 is dead for the whole event,
# and `docker compose up -d --force-recreate caddy` still exits 0 while it crash-loops.
docker run --rm -v "$PWD/Caddyfile:/etc/caddy/Caddyfile:ro" -e DOMAIN=example.com \
  caddy:2-alpine caddy validate --config /etc/caddy/Caddyfile   # must end "Valid configuration"

When	What
T‑7 days	Commit and push everything above. Registry + DNS pre-flight (§5). Build and push images (§6).
T‑5 days	First deploy to the server (§7). Verify admin login. Leave it running.
T‑5 days	⚠ Enable Hetzner automated snapshots (§10.1) and point an uptime monitor at /health (§10.4). Two console checkboxes, ~10 minutes total. Without them a failure during the event is both total and unnoticed.
T‑3 days	Freeze migrations. No further code deploys unless something is broken.
T‑2 days	Pre-pull current and previous image tags (§9). Install the hourly DB dump and prove it runs (§10.2). Run the backup rehearsal (§10.3).
Event day	Change nothing. Configuration tweaks via the admin dashboard only (§4).

Why the freeze matters more than anything else here. Migrations run automatically at boot (backend/src/db.rs, create_pool) and sqlx::migrate!() is used without set_ignore_missing. An older image booted against a newer schema does not degrade — it crash-loops with VersionMissing. The repo documents this itself in backend/migrations/014_export_epoch.up.sql:3-8. Once the migration set is frozen, rollback is a one-line .env edit; before it is frozen, rollback means hand-running down-SQL under pressure.

---

1. Deployment strategy — build on the Mac, push, pull

Decision: build linux/amd64 images on the M3 Pro, push to registry.mc02.dev, pull on the server.

Rejected alternatives, with the reason each loses:

Option	Why not
Build on the CX22	backend/Cargo.toml sets lto = true + codegen-units = 1 over 427 crates. Fat LTO links the whole program in one single-threaded process; peak RSS is estimated at 2.5–4 GB against ~1.5–2.5 GB free with the stack running. OOM is the expected outcome, not a tail risk. And rollback would also be a build — 35–60 min under pressure.
CI (Gitea on Pi 5)	Ruled out by you, and correct: a Pi 5 is strictly worse than the CX22 for a fat-LTO link.
True cross-compilation (--target x86_64-unknown-linux-musl)	The dependency graph has four C-compiling crates — ring (per-arch assembly), zstd-sys, libdeflate-sys, libsqlite3-sys — so it needs a real musl cross-toolchain. Alpine has no such package for an arm64 host; the working answer is cargo-zigbuild, which means a new base image and a new toolchain days before an unrepeatable event. Slow but certain beats fast but novel.

Emulated build is the right call because the cost lands where it is free: on your laptop, days early, with nothing depending on it. Estimated wall-clock for the backend is 25–60 min with Rosetta enabled (hours without it) — start it and walk away.

Escape hatch if emulation is unbearable: spin up a temporary Hetzner CPX41 (8 vCPU/16 GB) in the same account, build natively, push, destroy it. Costs cents, same commands, no repo change.

---

2. Repo changes — ALREADY APPLIED

Status: these are done, in the working tree. They are documented here so you know what changed and why, not as work to repeat. Run git diff to review before committing.

2.1 .dockerignore files (were absent)

backend/.dockerignore:

target/

frontend/.dockerignore:

node_modules/
.svelte-kit/
build/

Without these, the first time you run cargo build or npm install locally, every image build ships a multi-GB context to an emulated builder. Worse: frontend/Dockerfile:9 does COPY . . after npm ci, so a macOS node_modules/ would be merged over the container's Linux one.

2.2 Switch compose from build: to image:

In docker-compose.yml, replace the build: block on app and frontend:

  app:
    image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}

  frontend:
    image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}

Two deliberate properties:

• The :? form fails loudly on an unset variable instead of resolving to an empty tag.
• Removing build: entirely is a safety feature. With no build: key on the server, a wrong tag is an instant manifest unknown — never a surprise 45-minute compile on the production box.

New docker-compose.build.yml (opt-in, Mac only — follows the convention stated in the header of docker-compose.dev.yml that overlays are never auto-loaded):

# Build overlay. NOT loaded automatically. Used only where images are BUILT — never on the
# production server, which pulls.
#   docker compose -f docker-compose.yml -f docker-compose.build.yml build
services:
  app:
    build: { context: ./backend, dockerfile: Dockerfile }
  frontend:
    build: { context: ./frontend, dockerfile: Dockerfile }

e2e/docker-compose.test.yml keeps its own build: blocks, so the e2e gate is unaffected.

2.3 Add log rotation to every service

docker-compose.yml currently sets no logging config, so the default json-file driver keeps logs forever, on the same filesystem as Postgres and the media. Add to each of the four services:

    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }

(Equivalently, set it once in /etc/docker/daemon.json — but that restarts the Docker daemon, so do it before the stack is live, not after.)

---

3. .env — production values

# ── Identity ──────────────────────────────────────────────────────────────
DOMAIN=<your domain>
EVENT_NAME=<...>
EVENT_SLUG=<...>

# ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
EVENTSNAP_VERSION=v0.13.0

# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64>
POSTGRES_PASSWORD=<openssl rand -hex 24>
DATABASE_URL=postgres://eventsnap:<SAME PASSWORD>@db:5432/eventsnap
ADMIN_PASSWORD_HASH='<docker run --rm caddy:2-alpine caddy hash-password --plaintext "pw">'

# ── Paths — must match the volume mounts ──────────────────────────────────
# All four of these are PINNED in docker-compose.yml under `app.environment`, which overrides
# `env_file`. Keep them consistent here for readability, but understand that editing them in
# `.env` changes nothing — the pin is what the container gets. Change the pin.
MEDIA_PATH=/media
EXPORT_PATH=/exports
APP_PORT=3000

# ── Sizing (see the two corrections below) ────────────────────────────────
# 15, matching .env.example, the `db` sizing comment in docker-compose.yml and the code
# default. An earlier draft of this runbook said 30: that does not fit the 1G memory limit
# compose allots `db`, and 30 simultaneous queries cannot run on 2 vCPU anyway — they queue
# on the CPU instead of on the pool. Raise it only alongside more cores AND a bigger limit.
#
# ALSO PINNED IN COMPOSE (see above), and pinned for a sharper reason than the paths: an
# unparseable value here is boot-FATAL rather than falling back to the default, so a stray
# quote or a trailing inline comment in `.env` would crash-loop the app behind a live Caddy.
DATABASE_MAX_CONNECTIONS=15
COMPRESSION_WORKER_CONCURRENCY=2

# ── Comments off, likes + captions on ─────────────────────────────────────
COMMENTS_ENABLED=false

# ── Logging (NEW — production currently defaults to DEBUG) ────────────────
RUST_LOG=eventsnap_backend=info,tower_http=warn

Two sizing decisions worth understanding before you touch them

.env.example now agrees with this section on both — it carries the same reasoning inline and self-corrects the old advice. Kept here because these are the two knobs an operator is most tempted to raise under pressure.

COMPRESSION_WORKER_CONCURRENCY: keep 2. Do NOT raise to 4, and do NOT raise the app memory limit to 2G. An earlier draft justified both on the premise that "each worker can run an ffmpeg transcode". There is no transcode anywhere in this codebase. services/video.rs::run_ffmpeg runs ffmpeg -ss <t> -i <src> -vframes 1 -vf scale=… — a single poster frame. Video originals are stored and served byte-for-byte.

The real memory consumer is the image path: image 0.25's resize builds an Rgba32F intermediate at 16 bytes/px, sized source_width × target_height, which the 256 MiB decode guard in imaging::decode_limits does not cover. Estimated peak per photo:

Source	Peak (decode + display resize)
12 MP (typical phone)	~145 MB
24 MP (iPhone Pro default)	~223 MB
48 MP ("Max" mode)	~354 MB

Those are per-photo peaks, and the "two 48 MP photos at once" pair this limit used to be sized against is no longer reachable: compression.rs takes an EXCLUSIVE heavy permit for a large decode, so two giants serialise no matter what COMPRESSION_WORKER_CONCURRENCY is set to (see .env.example, which makes the same point). The binding case is now one giant (~354 MB) plus the ordinary working set against the 1 GiB cap, which is comfortable.

What has not changed is the reason to keep concurrency at 2 and app at 1G: at concurrency 4 the memory arithmetic stops working (app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB against ~3910 MiB MemTotal — oversubscribed before a single photo arrives).

quota_tolerance: keep 0.75. Raising it does not make anything more generous for a real guest. See §4.

---

4. Admin dashboard configuration

These live in the DB config table, not .env. Changes take effect on the next request — patch_config invalidates the cache synchronously (admin::patch_config). No restart needed. Set them after the first deploy, before the event.

Key	Default	Set to	Why
upload_rate_per_hour	100	1000	A guest multi-selecting 100 photos hits exactly 100. Client backs off and resumes, but this makes it a non-issue.
feed_rate_per_min	60	240	Headroom for reconnect bursts.
social_rate_per_min	120	600	This is the likes limiter — the interaction you're keeping.
export_rate_per_day	3	20	One shared bucket across both archives — enforce_export_rate keys on export:{user_id} regardless of which archive is being fetched. Downloading Gallery.zip + Memories.zip costs 2 of 3; one retry locks a guest out for 24 h. The limit is now charged when the download ticket is minted, so being over it produces a visible German error instead of a tap that silently does nothing.
max_video_size_mb	500	leave at 500	See below.
quota_tolerance	0.75	leave at 0.75	See below.
quota_enabled, storage_quota_enabled, rate_limits_enabled	true	leave on	This is the only disk-full safety net.

Correction (was wrong in an earlier draft). This section used to say "Ignore estimated_guest_count … read by no code at all". That is false — it is a live tuning knob and it is the dominant term in the quota divisor for a normal event. An operator who believed the old text and changed it would have moved every guest's ceiling. upload_count_quota_enabled genuinely is inert.

Why quotas are already as generous as you want

divisor        = max(active_uploaders, estimated_guest_count, 1)
per_user_limit = max(floor(free_disk × quota_tolerance / divisor), 500 MiB)

(upload::quota_limit_bytes. The 500 MiB floor applies only when the whole budget can back it — below that the divided value stands, so the quota cannot promise space the disk does not have.)

active_uploaders is SELECT COUNT(DISTINCT user_id) FROM upload WHERE deleted_at IS NULL — people who actually uploaded, not guests who joined. But it is a max, not the sole divisor: estimated_guest_count (default 100) acts as a floor on the divisor, so the ceiling settles at its final value early instead of sliding down all evening as guests arrive. It also blunts the abuse case where the divisor was attacker-controlled — ~1000 throwaway accounts once drove every real guest's ceiling to ~52 MB.

With ~28 GB free, quota_tolerance 0.75 and a realistic 30 people actually uploading, the divisor is 100 (not 30, because estimated_guest_count floors it), giving 28 GB × 0.75 / 100 ≈ 210 MB — which is below the floor, so every guest is granted the 500 MiB minimum. Against an expected ~1.25 GB for the entire event, nobody will be blocked. An earlier draft computed "~700 MB each" by dividing by 30; that ignored the floor on the divisor and was wrong.

Raising quota_tolerance would only raise the saturation ceiling (media converges to t/(1+t) of free space: 43% at 0.75, 50% at 1.0). It does nothing for a real guest at your volume, and it eats the headroom the keepsake needs — the export preflight reserves media × 1.10 × 2 because Gallery.zip and Memories.zip each store media uncompressed (export.rs — both archives store media uncompressed).

Why max_video_size_mb stays at 500. An earlier draft of this runbook said to lower it to 250, on the grounds that there is no client-side size check. That was wrong on both halves:

• A client guard does exist — frontend/src/routes/upload/+page.svelte rejects anything over HARD_MAX_UPLOAD_BYTES (576 MiB) before a byte leaves the phone, alongside the HEIC reject.
• That guard is a compile-time constant, not the DB value. Lowering max_video_size_mb therefore changes nothing on the client: the picker still accepts the clip, the phone still starts sending it, and the server aborts it partway (stream_field_to_file does stop at the cap, so the AP is not saturated for the full file — but the guest still gets a mid-upload failure where 500 MB would simply have worked).

So lowering it is a pure capacity decision with no UX upside, and at your volume there is no capacity problem to solve. Leave it. If you ever do want a smaller ceiling to bite on the phone, the constant above has to move with it.

---

5. Pre-flight — T‑7 days (do NOT leave this to event week)

Registry

From both the Mac and the server, as the user who will deploy (root's docker login does not help a non-root deploy — credentials go to ~/.docker/config.json):

getent hosts registry.mc02.dev                 # DNS resolves from the server
curl -fsSI https://registry.mc02.dev/v2/       # TLS must be PUBLICLY trusted; Docker rejects
                                               # self-signed without extra daemon config
docker login registry.mc02.dev

Also confirm: the eventsnap repository/namespace exists if your registry requires pre-creation (Harbor does; plain registry:2 does not), and the registry host has ~500 MB free per release.

DNS and TLS — get this wrong and you are locked out for an hour

The DNS A record must point at the CX22 before the first docker compose up -d, because Caddy attempts certificate issuance on boot. Let's Encrypt allows only 5 failed validations per hostname per hour (refilling 1 per 12 min). A misconfigured DNS record plus a few impatient restarts will rate-limit you out of getting a certificate at all.

• Deploy days early so issuance happens with no time pressure.
• Never delete the caddy_data volume — it holds the certificate and the ACME account key.
• Never run docker compose down -v. It destroys postgres_data, media_data, exports_data and caddy_data.

Host preparation

docker compose version          # must be v2.x — see below, this one is not optional
free -h && swapon --show        # Hetzner images ship no swap
df -h /var/lib/docker           # want ≥ 25 GB free

If docker compose version reports v1 (or docker-compose is a separate Python binary), STOP and install the v2 plugin before deploying. This check previously had no failure action, which made it decorative — and it is the single check that the whole sizing argument rests on.

On Compose v1, deploy.resources.limits is silently ignored outside Swarm: no warning, no error, up -d exits 0. Every memory and CPU limit in docker-compose.yml evaporates, and §1's arithmetic (app 1G + db 1G + 256M + 256M inside ~3910 MiB) becomes fiction — the first 48 MP photo takes the box out via the OOM killer instead of being bounded. On v2 the limits are real (verified empirically: memory: 1G produces HostConfig.Memory=1073741824).

# Debian/Ubuntu, with Docker's official repo already configured:
apt-get update && apt-get install -y docker-compose-plugin
docker compose version   # must now print v2.x

Verify the limits actually landed, once the stack is up — this is the check that matters, not the version string:

docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}'
# Must print two NON-ZERO numbers. `0 0` means the limits were dropped.

Add 2 GB of swap as an OOM cushion — a compression spike that would otherwise kill the container instead swaps out cold pages and merely runs slowly:

fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf

Already handled — do not hand-edit compose. Compose sets each container's Memory limit but leaves MemorySwap unset, and Docker then allows swap equal to the memory limit, so adding host swap would silently double every container ceiling (to ~5 GiB of ceilings on a 3.82 GiB box). docker-compose.yml now ships memswap_limit on all four services — 1152m on app and db, 320m on frontend and caddy — so this step is safe as written.

This used to say "add it yourself", which also broke §0's own gate that git status --porcelain -- docker-compose.yml must print nothing. Confirm it is still there:

docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'
# 1073741824 1207959552 — the second number MUST be larger than the first but not double it.

---

6. Build and push — on the Mac

Enable Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation", and set Resources → Memory ≥ 8 GB (the fat-LTO step will OOM inside the VM otherwise, even with 18 GB on the host).

docker run --rm --platform linux/amd64 alpine uname -m     # must print x86_64

cd /Users/fabianhammprivat/Projects/EventSnap
VERSION=v0.13.0                          # latest existing tag is v0.12.0 — see §9 before reusing it
ROLLBACK=v0.13.0-a                       # the SAME source, tagged twice; §9 explains why
SHA=$(git rev-parse --short HEAD)

docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/app:$SHA \
  --push ./backend

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/frontend:$VERSION \
  -t registry.mc02.dev/eventsnap/frontend:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/frontend:$SHA \
  --push ./frontend

Verify the architecture — never skip this

An arm64 image pulls fine and then dies with exec format error. Catch it here, not on the server:

docker buildx imagetools inspect registry.mc02.dev/eventsnap/app:$VERSION
docker buildx imagetools inspect registry.mc02.dev/eventsnap/frontend:$VERSION
# Both MUST report  Platform: linux/amd64

Then prove the binary actually executes. AppConfig::from_env() runs before any DB connection (main.rs builds AppConfig before touching the pool), so this needs no database:

docker run --rm --platform linux/amd64 \
  -e APP_ENV=production -e JWT_SECRET=x -e DATABASE_URL=x -e EVENT_SLUG=x \
  registry.mc02.dev/eventsnap/app:$VERSION

Expect the "Refusing to start in production … placeholder" message. That message means the amd64 binary ran. exec format error means the architecture is wrong.

Tag policy: never latest in production. With latest you cannot tell what is running, rollback becomes a registry re-push (impossible if the registry is down — exactly when you need it), and restart: unless-stopped after a reboot is ambiguous. Two immutable tags per build: semver and short SHA. Deploy by semver.

---

7. First deploy — on the server

# Directory name matters: volumes are prefixed with it, and README's backup commands
# hardcode the `eventsnap_` prefix.
git clone <repo> eventsnap && cd eventsnap
cp .env.example .env && nano .env          # every value from §3
docker login registry.mc02.dev
docker compose pull                        # must fully succeed before anything starts

7.1 The secret pre-flight — run this before the first up -d, always

docker compose run --rm --no-deps app

--no-deps means db never starts, so no Postgres data directory is initialised. AppConfig::from_env() runs first and reports every placeholder at once (config::validate_secrets). When the only remaining complaint is a database connection failure, the secrets are good.

Why this step exists. POSTGRES_PASSWORD is applied only at initdb. docker-compose.yml starts db in the same command as app, so a single up -d with a placeholder bakes the wrong password in permanently — the app then loops on password authentication failed, and the only exits are ALTER ROLE or down -v, which deletes the database, the media and the exports. The repo describes this trap at backend/src/db.rs (explain_auth_failure); this command is what avoids it.

7.2 Bring it up

# `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN.
set -a; . ./.env; set +a

docker compose up -d
docker compose logs -f app          # wait for "database connected and migrations applied"
curl -fsS https://$DOMAIN/health    # → ok  (503 means the app is up but the DB is not)

7.3 Verify the three things compose does not pin

MEDIA_PATH is pinned to /media on the app service in docker-compose.yml. Its siblings are not:

docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH

1. DATABASE_URL must contain @db:5432. A dev .env points it at @localhost, which inside the container is the app itself.
2. EXPORT_PATH must be /exports. Anywhere else and the keepsake archives are written to the container's writable layer and vanish on the next up -d — including on a rollback.
3. ADMIN_PASSWORD_HASH must match .env byte for byte.

Then actually log in to /admin with the real password. This is not optional politeness:

• The production secret guard only rejects placeholders (config::looks_placeholder). A hash corrupted by shell or Compose escaping is not a placeholder — the app boots green, /health says ok, and every admin login 401s.
• The Admin user row is created by a successful admin login (auth::handlers::admin_login), and promoting anyone to Host requires an Admin or Host (auth::middleware's role guard). No admin login ⇒ no host, ever ⇒ you cannot close the event, release the gallery, ban anyone, reset a PIN, or change any limit — for the whole event.

Per the Compose spec, single-quoted env_file values are used literally and the quotes are stripped, so the single-quote form in .env.example is correct. The comment in docker-compose.dev.yml claiming production has the same bug is stale. Verify anyway — the cost of checking is 10 seconds; the cost of being wrong is the whole event.

Promote a second person to Host once you are in, so a single lost session is not fatal.

---

8. Post-deploy verification

set -a; . ./.env; set +a                                     # $DOMAIN comes from .env, not your shell

docker compose ps                                            # db, app, frontend healthy; caddy has
                                                             # no healthcheck and shows only "running"
docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1   # must be 1073741824, not 0
curl -fsS https://$DOMAIN/health                             # ok — now a real DB check, not a constant
docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
docker builder prune -af && docker image prune -f            # reclaim build cache
df -h /var/lib/docker

Then, from a phone on cellular (not the office wifi):

• Join as a guest with a PIN
• Upload a photo → appears in the feed within a few seconds
• Upload a video → plays back (iOS Safari range requests)
• Add a caption, add a like — no comment UI anywhere
• Admin login works; host dashboard reachable
• Release the gallery on a test event and download both archives

---

9. Rollback

There is no older image you can roll back to. Build the rollback target yourself.

Read this before the event, not during it. The obvious move — drop EVENTSNAP_VERSION back to the previous released tag — takes the app down permanently and looks like a crash loop with no explanation:

$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
12          # 6 migrations. HEAD has 31.
$ git rev-list --count v0.12.0..HEAD
196

db.rs runs sqlx::migrate!() with no set_ignore_missing, so an image built from a 6-migration tree, booting against a database that already carries versions 007–022, returns VersionMissing. create_pool errors, main exits 1, and restart: unless-stopped restarts it forever — with Caddy still routing traffic to it. (014_export_epoch.up.sql documents this failure mode; §0 restates it.) No v0.12.0 image was ever built or pushed either, so the pre-pull would fail with manifest unknown before you ever got that far.

So: at build time, tag the SAME frozen commit twice. Two identical images, two names. The rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical migration set, which makes it a genuine no-op rather than a gamble:

# In §6, push both tags from the one build:
VERSION=v0.13.0
ROLLBACK=v0.13.0-a          # same source, different name — the rollback target

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  --push ./backend
# ...and the same two tags for ./frontend

Rolling back — ~30 seconds, no network:

sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env
docker compose up -d app frontend

This only works offline if both are already resident. Pre-pull all four at T‑2:

docker pull registry.mc02.dev/eventsnap/app:v0.13.0
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a
docker image ls | grep eventsnap        # confirm all four

Once resident, up -d, reboots, restarts and rollbacks need zero registry contact. That single step makes a registry outage on event day irrelevant.

Be clear-eyed about what this buys you: an identical image cannot undo a bad release, only an image that got corrupted or a container that wedged — and docker compose restart app already covers both. It exists so that the rollback line in the emergency card is safe to run rather than catastrophic. If you genuinely need to undo a code change during the event, you cannot; freeze early enough that you never have to.

Across a migration boundary — avoid by freezing. If you must: all 31 migrations have paired .down.sql files, but none of them removes its own _sqlx_migrations row, so that second step is mandatory and undocumented:

docker compose stop app
# `sh -c` so the CONTAINER expands the credentials. They live in the container's environment
# and in .env — not in your shell — so an unwrapped `-U "$POSTGRES_USER"` sends `-U ""` and
# psql answers `FATAL: role "" does not exist`.
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' \
  < backend/migrations/0NN_x.down.sql
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"'

A down migration is only valid PAIRED WITH A CODE ROLLBACK — it is not a standalone repair. Upload::create sends an ON CONFLICT ... WHERE predicate that must match the live partial index exactly, and these queries are not compile-checked. Run 026's or 031's down against the current binary and every upload carrying a client_upload_id — i.e. every upload from the shipped client — becomes a runtime 500. Roll the image back first, then the migration.

026's down can also fail outright, and that is expected. It restores a wider unique index, so it aborts with could not create unique index ... is duplicated on any database where a guest ever deleted a photo and re-uploaded it. The transaction rolls back cleanly and the narrow index survives intact — no half-state — but you cannot go below 026 on a database that has seen real use. Verified against a live Postgres.

Migration 014 is the only destructive one on the way up, and the only one with a rehearsal harness — backend/scripts/rehearse-014.sh. Run it once against a real dump before the event.

Registry-down transport fallback (layers are already compressed — do not add gzip):

docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
            registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load'

---

10. Backup

Full commands are in README.md:315-435 and are correct — pg_dump --clean --if-exists, plus alpine tar out of eventsnap_media_data and eventsnap_exports_data, mounted at /src (not /media), with chown -R 100:101 on restore because the app runs non-root and BusyBox tar has no --same-owner. There is deliberately no script.

Three gaps the README does not cover:

1. Nothing backs up .env, which holds the only copy of POSTGRES_PASSWORD. A dump you cannot authenticate against is not a backup. Copy .env off the box, encrypted, once it is final.
2. The final dump is not a backup — it is an archive. Taking it after locking uploads gives you a consistent pair, and that is the right way to archive the finished event. But it means that until the host locks uploads there is no copy of anything anywhere. All four volumes sit on the same 40 GB filesystem, on one VPS, with no redundancy. A disk or host failure at 23:00 — the fullest the gallery will ever be — loses 100% of the event, permanently, with the guests still in the room. Both of the mitigations below are required.
3. Nothing is watching. See §10.2.

10.1 Snapshots — do this once, before the event

⚠ ACTION REQUIRED — Hetzner Cloud console, ~5 minutes, one checkbox. Server → Backups → enable. Costs ~20% of the server price and needs no operator action ever again.

This is the single highest-value item in this runbook. It converts "total, permanent loss" into "lose at most the hours since the last snapshot", automatically, with nobody awake. It covers the whole volume set at once — database, media, exports and .env — which the pg_dump path does not.

It does not replace §10's archive: snapshots are whole-disk and crash-consistent, so restoring one gives you the box back, not a portable copy of the photos. Do both.

10.2 A mid-event database dump — cheap, and the only thing cron should do

The database is small (a few MB — it holds rows, not pixels) and it is the part that cannot be reconstructed: media files on disk without their upload rows are anonymous UUIDs with no uploader, caption, hashtag or timestamp. Dumping it hourly costs essentially nothing and is safe while uploads are live, because a pg_dump is transactionally consistent on its own.

Media is the bulk and is recoverable from guests' phones in the worst case, so it stays on the event-night schedule below.

# On the server, before the event. Hourly DB-only dump, keeping the last 48.
mkdir -p /root/eventsnap-dumps
cat >/root/eventsnap-dump.sh <<'SH'
#!/bin/sh
set -eu
cd /root/eventsnap
set -a; . ./.env; set +a
OUT="/root/eventsnap-dumps/db-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
docker compose exec -T db sh -c \
  'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' | gzip >"$OUT.tmp"
mv "$OUT.tmp" "$OUT"          # atomic: never leave a truncated dump looking complete
ls -1t /root/eventsnap-dumps/db-*.sql.gz | tail -n +49 | xargs -r rm
SH
chmod +x /root/eventsnap-dump.sh
( crontab -l 2>/dev/null; echo '17 * * * * /root/eventsnap-dump.sh >>/var/log/eventsnap-dump.log 2>&1' ) | crontab -

# Prove it works NOW, not at 23:00:
/root/eventsnap-dump.sh && ls -lh /root/eventsnap-dumps/

These land on the same filesystem, so they do not survive a disk loss — that is what §10.1 is for. They protect against the far more likely failure: a bad migration, an accidental host action, or a corrupted table.

10.3 The event-night archive — unchanged

Take the dump and the media tarball back-to-back the night of the event, after locking uploads from the host dashboard, so the pair is consistent. Copy both off the box before you sleep.

10.4 Monitoring — something has to be able to wake you

⚠ ACTION REQUIRED — external uptime monitor, ~5 minutes. Point any free monitor (UptimeRobot, Better Stack, Healthchecks.io — all have free tiers with SMS or push) at https://$DOMAIN/health, 1–5 minute interval, alerting to a phone that will be on you during the event.

There is otherwise no metrics collection, no alerting, no log shipping and no external check anywhere in this deployment. Without this step, none of the following reaches a human: a crash loop, a full disk, a dead database, an expired certificate, or the box being off. The host is at a party and is not watching a dashboard.

/health is already built for exactly this and nothing currently consumes it:

Response	Meaning	Action
200 ok	App and database are answering	—
503 database timeout / database unavailable	App is up, Postgres is not	§13 emergency card
Connection refused / TLS error	App container or Caddy is down	docker compose ps, then §13
Timeout	Box is gone, or the disk is full enough to wedge it	§10.1 snapshot restore

It runs a real SELECT 1 against the pool with a 2 s timeout — a green check means the request path guests use is genuinely working, not merely that a process is listening.

The one signal this does not give you is disk. The low-disk banner on /host requires the host to open a dashboard during their own party and does not refresh without a manual reload, so treat it as a pre-event check, not an alert. Before the event, confirm headroom with §11's numbers; the export preflight and the upload quota are the automated backstops.

---

11. Disk — the numbers for 40 GB

Expected event (100 photos @ ~4 MB, 5 videos @ ~150 MB). Originals are always kept and never lossily recompressed; derivatives are a ≤800 px preview and a ≤2048 px display JPEG.

media (originals + derivatives)                    ~1.25 GB
peak during keepsake build (+ Gallery + Memories)  ~3.5 GB
OS + Docker images + Postgres + logs               ~4.5 GB
────────────────────────────────────────────────────────────
total                             ~11–13 GB of ~36 GB usable

40 GB fits with roughly 3× headroom, provided you build elsewhere (a server-side build adds 3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against live free space on every upload).

The README.md:295-299 "ENOSPC" projection models guests saturating the quota (~12 GB of media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully: the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard warns when free space drops below 10 GB or below the keepsake requirement (handlers::host's low-disk thresholds).

---

12. Known issues you are shipping with

None of these has a fix in this runbook; they are listed so nothing is a surprise. Severity is scored purely by "would this interrupt you during the event".

Fixed in this pass

Was	Fix
Queued uploads never rehydrated — loadQueue() had one call site, so a guest whose PWA was evicted mid-upload and reopened onto /feed had pending photos in IndexedDB that nothing ever read. Silent photo loss.	loadQueue() now runs on every authenticated boot in +layout.svelte.
A corrupted ADMIN_PASSWORD_HASH booted green and only failed at admin login — unrecoverable mid-event.	config.rs now validates the bcrypt shape, not just placeholder-ness, and refuses to start.
SSE reconnect thundering herd — flat 500 ms jitter regardless of backoff.	Jitter now scales with the delay; the feed's in-place refresh is spread over 800–2800 ms.
No client-side size/HEIC pre-check — a doomed 600 MB upload saturated the venue AP before being rejected.	Certain-rejects are refused before any bytes leave the phone.
Upload-queue UI unreachable — the badged FAB opened the picker, not the queue, so a failed upload had no retry button.	The upload sheet now shows a queue entry whenever the badge is non-zero.
A mid-event 401 left a dead screen with no nav and no URL bar.	api.ts now returns the guest to /join (skipping the auth routes themselves).
Rate limiter could panic while holding its mutex (timestamps[0] with max == 0), poisoning it process-wide.	Uses first() with a fail-open guard.
Hashtag deadlock — edit_upload and add_comment upserted tags in client/text order.	Both now sort + dedup on the normalised key, matching the upload path.
Unbounded Docker logs + debug-level app logging.	logging: caps every service at 30 MB; RUST_LOG documented in .env.example.

Fixed in the readiness pass — behaviour you should know about

These change what you will observe on the night, so they are listed separately from the table above.

Was	Now
Any ffmpeg-level failure DESTROYED the video. A poster-frame error — ffmpeg hanging on a truncated .mov, an ENOSPC on thumbnails/, a DB blip — propagated into the give-up path, which soft-deleted the upload. Reproduced live: on a host with no ffmpeg the clip was gone ~6 s after its 201 Created.	No failure in the video branch can fail the upload. The clip stays in the feed and plays from its original; only the poster is missing.
A lost response duplicated the photo. Every retry minted a fresh upload id, so a phone that lost the reply and re-sent — manually, or automatically on reconnect — stored the same photo two or three times and paid quota for each.	The client sends a client_upload_id; migration 022 makes it unique. A retry returns 200 with the original row. Verified live: three sends → one row, quota charged once.
/health returned a constant "ok". The container reported healthy while every request 500'd.	Runs SELECT 1 with a 2 s timeout: 200 ok / 503. Verified live: 200 → stop Postgres → 503 → start Postgres → 200, with no app restart (the pool revalidates on acquire). Note that Compose does not restart on an unhealthy probe — this is a diagnostic, deliberately not wired to automatic recovery, because a restart would truncate in-flight uploads to "fix" an outage that clears on its own.
Abandoned .tmp uploads were never reclaimed (see the old issue #2 below — it is now fixed, not deferred).	Swept at boot and hourly, keyed on modification time so a live upload can never age into it.
A full disk made itself worse. ENOSPC was retried three times, then refunded the quota, soft-deleted the row and kept the bytes — freeing nothing and inviting an immediate re-upload into the same full disk.	ENOSPC is classified separately: no retry, no refund, no delete. The row stays live and the photo is served from its original until there is room to compress it.
The keepsake download failed invisibly. The rate limit was enforced inside the iframe navigation, so an over-limit guest tapped and nothing happened, forever.	Charged when the ticket is minted — a normal fetch — so it surfaces as a German message naming the daily window. Raise export_rate_per_day per §4 anyway.
/host and /admin subscribed to SSE but never opened the connection, so the keepsake progress bar froze after a release.	Both connect on mount and disconnect on destroy, as /export already did.
?limit=-5 on the feed returned a 500.	Clamped at both ends.
All recurring hygiene lived in one unsupervised task — one panic silently stopped session pruning, media reclamation and the temp sweep for the rest of the event.	Supervised and re-spawned, with an error log.
No pool acquire or statement timeout. A DB blip parked every request for 30 s.	5 s acquire, statement_timeout=15s, lock_timeout=5s.

Still open — accepted for this event

#	Issue	Impact
1	Guest delete/caption-edit after release invalidates both keepsake archives (upload.rs), forcing a rebuild with a 20 s debounce. No-op before release, so it cannot bite during the event.	Post-event only. Left alone because blocking guest deletes has real privacy downsides — that is a product call, not a bug fix.
2	Video poster frames do not regenerate after a restart mid-compression (the backfill filters mime_type LIKE 'image/%').	Cosmetic — the video still plays; only its poster is missing.
3	SSE keep-alives are sent as SSE comments (:ping), which the browser's EventSource parser discards without dispatching. A client therefore cannot implement a pure silence timer to detect a half-open socket.	Worked around client-side: the feed runs a jittered 60–120 s /feed/delta backstop and reconnects when a poll returns content the stream never delivered. A cleaner fix is to emit keep-alives as a named event; that is a coordinated backend+frontend change, not worth making during a freeze.
4	The lightbox stops at the end of the loaded page — stepping past the last loaded photo does not fetch the next one.	The guest scrolls the feed (which does page) and re-opens.

Migration checksum mismatch — VersionMissing / "previously applied but has been modified"

sqlx compares checksums, so renaming or renumbering a migration file is indistinguishable from editing one. If a box ever booted an image built from a branch that numbered migrations differently, the next boot aborts with "migration 21 was previously applied but has been modified", main exits non-zero, and restart: unless-stopped makes it permanent — with Caddy still routing traffic to the dead container.

Every main-line migration is byte-identical to what shipped, so a box that only ever ran tagged releases is unaffected. Verify rather than assume — run this against the server before any deploy.

Note the sh -c wrapping, for the same reason as §9: POSTGRES_USER and POSTGRES_DB live in .env, which Compose reads and your shell does not. Unwrapped, -U "$POSTGRES_USER" sends -U "" and psql answers FATAL: role "" does not exist — at 11pm, with the app crash-looping.

docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version;"'

If the app is already crash-looping on a renumbered migration, and only if you have confirmed the SQL in the new file is equivalent to what was actually applied. Take the version numbers from the crash message and the query above — do not copy the ones below, which are an example:

docker compose stop app
# Replace 21,22,23 with the versions the boot error actually named.
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version IN (21,22,23);"'
docker compose start app     # re-applies exactly those, then continues

This re-runs those migrations. They must be idempotent (IF NOT EXISTS / IF EXISTS) or this fails differently. Take a pg_dump first — see §10.

Schema changes are not compile-time checked

All ~120 queries use the runtime sqlx::query() API. There are no query! macros and no .sqlx cache, and the backend compiles with no DATABASE_URL at all. Consequences:

• A migration that renames or drops a column compiles clean and fails in production as a runtime 500 on whichever request path touches it first.
• cargo build succeeding tells you nothing about schema/query agreement. Only cargo test (which runs against a real Postgres) and manual exercise of the affected route do.

So: after any migration that touches an existing column, exercise the routes that read it before you consider the deploy done. README and PROJECT previously claimed compile-time checking; they have been corrected.

---

13. Event-day emergency card

Assume you have a phone and two minutes.

cd ~/eventsnap

# FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this,
# every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage
# when the site is fine.
set -a; . ./.env; set +a

# Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not)
curl -fsS https://$DOMAIN/health

# What is broken?
docker compose ps
docker compose logs --tail=100 app

# Nuclear option that is SAFE (keeps all data):
docker compose restart app

# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
# Do NOT substitute an older release tag here; it will crash-loop on the migration set.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend

# Disk check
df -h /var/lib/docker

"Der Speicher des Events ist fast voll" — guests cannot upload

df -h will look fine, and that is not a contradiction. The upload gate refuses long before the disk fills: it reserves room for the keepsake, which is roughly a second copy of every original, plus a 10 GB floor. Uploads stop at ~8 GB of media on a 40 GB box, when df still shows ~20 GB free.

Check the number that actually binds, not free space:

docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT pg_size_pretty(sum(original_size_bytes)) FROM upload WHERE deleted_at IS NULL;"'

Mid-event, in order of preference: delete the largest videos from the host dashboard (each frees its own bytes immediately), or move exports_data to a separate volume. Raising quota_tolerance will not help — on this box every guest is already on the 500 MiB floor, so that knob is not what is refusing them (see §4 and .env.example).

NEVER run docker compose down -v. It deletes the database, all media, all exports and the TLS certificate. There is no undo.

Most limits are changeable from the admin dashboard without a restart — reach for that before touching the shell.