EventSnap/DEPLOYMENT_RUNBOOK.md

# EventSnap — Production Deployment Runbook

Target: **Hetzner CX22** (2 vCPU, 4 GB RAM, 40 GB disk), single host, docker compose.
Event profile: ~100 guests, ~100 photos + a few videos, one evening, **operator attending and unavailable to troubleshoot**.

Everything here is written for that last constraint. Where a choice trades throughput for
"cannot need attention on the night", it takes the stable option.

> **Note on this repo's own docs.** `README.md:62` and `PROJECT.md:359` specify a **CX33
> (4 vCPU / 8 GB / 80 GB)**. You are deploying to half of that on every axis. Two pieces of
> tuning advice in `.env.example` are calibrated for the CX33 and are actively wrong for a
> CX22 — they are called out in §3. Trust this file over `.env.example` for sizing.

---

## 0. Timeline — the single most important control

### Step zero: verify the deployment files are committed, before you build anything

Everything the server clones must be in git — §7 tells you to `git clone` onto the box, so
anything living only in your working tree is not part of the deployment. Two failure modes if it
is not:

- If the committed `docker-compose.yml` still carried `build:` keys and no `image:` keys, then on
  that clone `docker compose pull` would skip both services and `docker compose up -d` would start
  **a fat-LTO release build of 427 crates on the CX22** — the exact scenario §1 rules out as an
  expected OOM.
- `sqlx::migrate!()` embeds `./migrations` **at compile time**. An image built from a working tree
  with uncommitted migrations bakes them in and applies them on first boot; any later rebuild from
  a clean clone produces an image that lacks them and crash-loops with `VersionMissing` against its
  own database.

**As of this writing all of these are committed and the check below passes.** Run it anyway — it
costs a second and it is the difference between finding this now and finding it at T‑5.

```bash
# Every deployment file must be tracked. Prints nothing and exits 0 when correct;
# names the offender and exits non-zero otherwise.
git ls-files --error-unmatch \
  docker-compose.yml docker-compose.dev.yml docker-compose.build.yml .env.example \
  backend/.dockerignore frontend/.dockerignore frontend/Dockerfile \
  DEPLOYMENT_RUNBOOK.md Caddyfile >/dev/null

# No uncommitted edits to them.
git status --porcelain -- docker-compose.yml .env.example Caddyfile DEPLOYMENT_RUNBOOK.md

# The COMMITTED compose must pull, not build: 4 `image:` lines, zero `build:` lines.
git show HEAD:docker-compose.yml | grep -cE '^[[:space:]]*image:'   # must be 4
git show HEAD:docker-compose.yml | grep -cE '^[[:space:]]*build:'   # must be 0

# Every migration in the tree is committed — a build from a dirty tree bakes in extras.
git status --porcelain -- backend/migrations/                       # must print nothing
```

| When | What |
|---|---|
| **T‑7 days** | Commit and push everything above. Registry + DNS pre-flight (§5). Build and push images (§6). |
| **T‑5 days** | First deploy to the server (§7). Verify admin login. Leave it running. |
| **T‑5 days** | ⚠ **Enable Hetzner automated snapshots** (§10.1) and **point an uptime monitor at `/health`** (§10.4). Two console checkboxes, ~10 minutes total. Without them a failure during the event is both total and unnoticed. |
| **T‑3 days** | **Freeze migrations.** No further code deploys unless something is broken. |
| **T‑2 days** | Pre-pull current *and* previous image tags (§9). Install the hourly DB dump and prove it runs (§10.2). Run the backup rehearsal (§10.3). |
| **Event day** | Change nothing. Configuration tweaks via the admin dashboard only (§4). |

**Why the freeze matters more than anything else here.** Migrations run automatically at boot
(`backend/src/db.rs`, `create_pool`) and `sqlx::migrate!()` is used *without* `set_ignore_missing`. An older
image booted against a newer schema does not degrade — it **crash-loops** with `VersionMissing`.
The repo documents this itself in `backend/migrations/014_export_epoch.up.sql:3-8`. Once the
migration set is frozen, rollback is a one-line `.env` edit; before it is frozen, rollback means
hand-running down-SQL under pressure.

---

## 1. Deployment strategy — build on the Mac, push, pull

**Decision: build `linux/amd64` images on the M3 Pro, push to `registry.mc02.dev`, pull on the server.**

Rejected alternatives, with the reason each loses:

| Option | Why not |
|---|---|
| **Build on the CX22** | `backend/Cargo.toml` sets `lto = true` + `codegen-units = 1` over **427 crates**. Fat LTO links the whole program in one single-threaded process; peak RSS is estimated at 2.5–4 GB against ~1.5–2.5 GB free with the stack running. OOM is the expected outcome, not a tail risk. And *rollback would also be a build* — 35–60 min under pressure. |
| **CI (Gitea on Pi 5)** | Ruled out by you, and correct: a Pi 5 is strictly worse than the CX22 for a fat-LTO link. |
| **True cross-compilation** (`--target x86_64-unknown-linux-musl`) | The dependency graph has **four C-compiling crates** — `ring` (per-arch assembly), `zstd-sys`, `libdeflate-sys`, `libsqlite3-sys` — so it needs a real musl cross-toolchain. Alpine has no such package for an arm64 host; the working answer is `cargo-zigbuild`, which means a new base image and a new toolchain days before an unrepeatable event. **Slow but certain beats fast but novel.** |

Emulated build is the right call because the cost lands where it is free: on your laptop, days
early, with nothing depending on it. Estimated wall-clock for the backend is **25–60 min with
Rosetta enabled** (hours without it) — start it and walk away.

> **Escape hatch if emulation is unbearable:** spin up a temporary Hetzner CPX41 (8 vCPU/16 GB) in
> the same account, build natively, push, destroy it. Costs cents, same commands, no repo change.

---

## 2. Repo changes — ALREADY APPLIED

> **Status: these are done, in the working tree.** They are documented here so you know what
> changed and why, not as work to repeat. Run `git diff` to review before committing.

### 2.1 `.dockerignore` files (were absent)

`backend/.dockerignore`:
```
target/
```

`frontend/.dockerignore`:
```
node_modules/
.svelte-kit/
build/
```

Without these, the first time you run `cargo build` or `npm install` locally, every image build
ships a multi-GB context to an *emulated* builder. Worse: `frontend/Dockerfile:9` does `COPY . .`
**after** `npm ci`, so a macOS `node_modules/` would be merged over the container's Linux one.

### 2.2 Switch compose from `build:` to `image:`

In `docker-compose.yml`, replace the `build:` block on `app` and `frontend`:

```yaml
  app:
    image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
```
```yaml
  frontend:
    image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
```

Two deliberate properties:
- The `:?` form **fails loudly** on an unset variable instead of resolving to an empty tag.
- **Removing `build:` entirely is a safety feature.** With no `build:` key on the server, a wrong
  tag is an instant `manifest unknown` — never a surprise 45-minute compile on the production box.

New `docker-compose.build.yml` (opt-in, Mac only — follows the convention stated in
the header of `docker-compose.dev.yml` that overlays are never auto-loaded):

```yaml
# Build overlay. NOT loaded automatically. Used only where images are BUILT — never on the
# production server, which pulls.
#   docker compose -f docker-compose.yml -f docker-compose.build.yml build
services:
  app:
    build: { context: ./backend, dockerfile: Dockerfile }
  frontend:
    build: { context: ./frontend, dockerfile: Dockerfile }
```

`e2e/docker-compose.test.yml` keeps its own `build:` blocks, so the e2e gate is unaffected.

### 2.3 Add log rotation to every service

`docker-compose.yml` currently sets **no logging config**, so the default `json-file` driver keeps
logs forever, on the same filesystem as Postgres and the media. Add to each of the four services:

```yaml
    logging:
      driver: json-file
      options: { max-size: "10m", max-file: "3" }
```

(Equivalently, set it once in `/etc/docker/daemon.json` — but that restarts the Docker daemon,
so do it *before* the stack is live, not after.)

---

## 3. `.env` — production values

```bash
# ── Identity ──────────────────────────────────────────────────────────────
DOMAIN=<your domain>
EVENT_NAME=<...>
EVENT_SLUG=<...>

# ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
EVENTSNAP_VERSION=v0.13.0

# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64>
POSTGRES_PASSWORD=<openssl rand -hex 24>
DATABASE_URL=postgres://eventsnap:<SAME PASSWORD>@db:5432/eventsnap
ADMIN_PASSWORD_HASH='<docker run --rm caddy:2-alpine caddy hash-password --plaintext "pw">'

# ── Paths — must match the volume mounts ──────────────────────────────────
MEDIA_PATH=/media          # pinned by compose anyway, but keep consistent
EXPORT_PATH=/exports       # NOT pinned by compose — see the trap in §7.3

# ── Sizing (see the two corrections below) ────────────────────────────────
# 15, matching .env.example, the `db` sizing comment in docker-compose.yml and the code
# default. An earlier draft of this runbook said 30: that does not fit the 1G memory limit
# compose allots `db`, and 30 simultaneous queries cannot run on 2 vCPU anyway — they queue
# on the CPU instead of on the pool. Raise it only alongside more cores AND a bigger limit.
DATABASE_MAX_CONNECTIONS=15
COMPRESSION_WORKER_CONCURRENCY=2

# ── Comments off, likes + captions on ─────────────────────────────────────
COMMENTS_ENABLED=false

# ── Logging (NEW — production currently defaults to DEBUG) ────────────────
RUST_LOG=eventsnap_backend=info,tower_http=warn
```

### Two sizing decisions worth understanding before you touch them

`.env.example` now agrees with this section on both — it carries the same reasoning inline and
self-corrects the old advice. Kept here because these are the two knobs an operator is most
tempted to raise under pressure.

**`COMPRESSION_WORKER_CONCURRENCY`: keep `2`. Do NOT raise to 4, and do NOT raise the app memory
limit to 2G.** An earlier draft justified both on the premise that "each worker can run an
ffmpeg transcode". **There is no transcode anywhere in this codebase.** `services/video.rs::run_ffmpeg`
runs `ffmpeg -ss <t> -i <src> -vframes 1 -vf scale=…` — a single poster frame. Video originals are
stored and served byte-for-byte.

The real memory consumer is the **image** path: `image` 0.25's `resize` builds an `Rgba32F`
intermediate at **16 bytes/px**, sized `source_width × target_height`, which the 256 MiB decode
guard in `imaging::decode_limits` does not cover. Estimated peak per photo:

| Source | Peak (decode + display resize) |
|---|---|
| 12 MP (typical phone) | ~145 MB |
| 24 MP (iPhone Pro default) | ~223 MB |
| 48 MP ("Max" mode) | ~354 MB |

Those are per-photo peaks, and the "two 48 MP photos at once" pair this limit used to be sized
against **is no longer reachable**: `compression.rs` takes an EXCLUSIVE `heavy` permit for a large
decode, so two giants serialise no matter what `COMPRESSION_WORKER_CONCURRENCY` is set to (see
`.env.example`, which makes the same point). The binding case is now one giant (~354 MB) plus the
ordinary working set against the 1 GiB cap, which is comfortable.

What has not changed is the reason to keep concurrency at 2 and `app` at 1G: at concurrency 4 the
memory arithmetic stops working (app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
against ~3910 MiB MemTotal — oversubscribed before a single photo arrives).

**`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.**
See §4.

---

## 4. Admin dashboard configuration

These live in the DB `config` table, not `.env`. Changes take effect on the **next request** —
`patch_config` invalidates the cache synchronously (`admin::patch_config`). No restart needed. Set them
after the first deploy, before the event.

| Key | Default | **Set to** | Why |
|---|---|---|---|
| `upload_rate_per_hour` | 100 | **1000** | A guest multi-selecting 100 photos hits exactly 100. Client backs off and resumes, but this makes it a non-issue. |
| `feed_rate_per_min` | 60 | **240** | Headroom for reconnect bursts. |
| `social_rate_per_min` | 120 | **600** | This is the **likes** limiter — the interaction you're keeping. |
| `export_rate_per_day` | 3 | **20** | **One shared bucket across both archives** — `enforce_export_rate` keys on `export:{user_id}` regardless of which archive is being fetched. Downloading Gallery.zip + Memories.zip costs 2 of 3; one retry locks a guest out for 24 h. The limit is now charged when the download *ticket* is minted, so being over it produces a visible German error instead of a tap that silently does nothing. |
| `max_video_size_mb` | 500 | **leave at 500** | See below. |
| `quota_tolerance` | 0.75 | **leave at 0.75** | See below. |
| `quota_enabled`, `storage_quota_enabled`, `rate_limits_enabled` | true | **leave on** | This is the only disk-full safety net. |

> **Correction (was wrong in an earlier draft).** This section used to say "**Ignore
> `estimated_guest_count`** … read by no code at all". That is **false** — it is a live tuning
> knob and it is the dominant term in the quota divisor for a normal event. An operator who
> believed the old text and changed it would have moved every guest's ceiling. `upload_count_quota_enabled`
> genuinely is inert.

### Why quotas are already as generous as you want

```
divisor        = max(active_uploaders, estimated_guest_count, 1)
per_user_limit = max(floor(free_disk × quota_tolerance / divisor), 500 MiB)
```
(`upload::quota_limit_bytes`. The 500 MiB floor applies only when the whole budget can back it —
below that the divided value stands, so the quota cannot promise space the disk does not have.)

`active_uploaders` is `SELECT COUNT(DISTINCT user_id) FROM upload WHERE deleted_at IS NULL` —
**people who actually uploaded**, not guests who joined. But it is a `max`, not the sole divisor:
`estimated_guest_count` (default **100**) acts as a **floor on the divisor**, so the ceiling settles
at its final value early instead of sliding down all evening as guests arrive. It also blunts the
abuse case where the divisor was attacker-controlled — ~1000 throwaway accounts once drove every
real guest's ceiling to ~52 MB.

With ~28 GB free, `quota_tolerance` 0.75 and a realistic 30 people actually uploading, the divisor
is **100** (not 30, because `estimated_guest_count` floors it), giving 28 GB × 0.75 / 100 ≈ 210 MB —
which is below the floor, so **every guest is granted the 500 MiB minimum**. Against an expected
~1.25 GB for the *entire event*, nobody will be blocked. An earlier draft computed "~700 MB each"
by dividing by 30; that ignored the floor on the divisor and was wrong.

Raising `quota_tolerance` would only raise the **saturation ceiling** (media converges to
`t/(1+t)` of free space: 43% at 0.75, 50% at 1.0). It does nothing for a real guest at your volume,
and it eats the headroom the keepsake needs — the export preflight reserves `media × 1.10 × 2`
because `Gallery.zip` and `Memories.zip` each store media **uncompressed** (`export.rs` — both archives store media uncompressed).

**Why `max_video_size_mb` stays at 500.** An earlier draft of this runbook said to lower it to 250,
on the grounds that there is no client-side size check. That was wrong on both halves:

- A client guard **does** exist — `frontend/src/routes/upload/+page.svelte` rejects anything over
  `HARD_MAX_UPLOAD_BYTES` (576 MiB) before a byte leaves the phone, alongside the HEIC reject.
- That guard is a **compile-time constant**, not the DB value. Lowering `max_video_size_mb` therefore
  changes nothing on the client: the picker still accepts the clip, the phone still starts sending
  it, and the *server* aborts it partway (`stream_field_to_file` does stop at the cap, so the AP is
  not saturated for the full file — but the guest still gets a mid-upload failure where 500 MB would
  simply have worked).

So lowering it is a pure capacity decision with no UX upside, and at your volume there is no capacity
problem to solve. Leave it. If you ever do want a smaller ceiling to bite on the phone, the constant
above has to move with it.

---

## 5. Pre-flight — T‑7 days (do NOT leave this to event week)

### Registry

From **both** the Mac and the server, as **the user who will deploy** (root's `docker login` does
not help a non-root deploy — credentials go to `~/.docker/config.json`):

```bash
getent hosts registry.mc02.dev                 # DNS resolves from the server
curl -fsSI https://registry.mc02.dev/v2/       # TLS must be PUBLICLY trusted; Docker rejects
                                               # self-signed without extra daemon config
docker login registry.mc02.dev
```

Also confirm: the `eventsnap` repository/namespace exists if your registry requires pre-creation
(Harbor does; plain `registry:2` does not), and the registry host has ~500 MB free per release.

### DNS and TLS — get this wrong and you are locked out for an hour

The DNS **A record must point at the CX22 before the first `docker compose up -d`**, because Caddy
attempts certificate issuance on boot. Let's Encrypt allows only **5 failed validations per hostname
per hour** (refilling 1 per 12 min). A misconfigured DNS record plus a few impatient restarts will
rate-limit you out of getting a certificate at all.

- Deploy days early so issuance happens with no time pressure.
- **Never delete the `caddy_data` volume** — it holds the certificate and the ACME account key.
- Never run `docker compose down -v`. It destroys `postgres_data`, `media_data`, `exports_data`
  *and* `caddy_data`.

### Host preparation

```bash
docker compose version          # must be v2.x — see below, this one is not optional
free -h && swapon --show        # Hetzner images ship no swap
df -h /var/lib/docker           # want ≥ 25 GB free
```

> **If `docker compose version` reports v1 (or `docker-compose` is a separate Python binary), STOP
> and install the v2 plugin before deploying.** This check previously had no failure action, which
> made it decorative — and it is the single check that the whole sizing argument rests on.
>
> On Compose v1, `deploy.resources.limits` is **silently ignored** outside Swarm: no warning, no
> error, `up -d` exits 0. Every memory and CPU limit in `docker-compose.yml` evaporates, and §1's
> arithmetic (`app` 1G + `db` 1G + 256M + 256M inside ~3910 MiB) becomes fiction — the first 48 MP
> photo takes the box out via the OOM killer instead of being bounded. On v2 the limits are real
> (verified empirically: `memory: 1G` produces `HostConfig.Memory=1073741824`).
>
> ```bash
> # Debian/Ubuntu, with Docker's official repo already configured:
> apt-get update && apt-get install -y docker-compose-plugin
> docker compose version   # must now print v2.x
> ```
>
> Verify the limits actually landed, once the stack is up — this is the check that matters, not the
> version string:
>
> ```bash
> docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}'
> # Must print two NON-ZERO numbers. `0 0` means the limits were dropped.
> ```

**Add 2 GB of swap** as an OOM cushion — a compression spike that would otherwise kill the container
instead swaps out cold pages and merely runs slowly:

```bash
fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab
sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf
```

> **Already handled — do not hand-edit compose.** Compose sets each container's `Memory` limit but
> leaves `MemorySwap` unset, and Docker then allows swap equal to the memory limit, so adding host
> swap would silently **double** every container ceiling (to ~5 GiB of ceilings on a 3.82 GiB box).
> `docker-compose.yml` now ships `memswap_limit` on all four services — 1152m on `app` and `db`,
> 320m on `frontend` and `caddy` — so this step is safe as written.
>
> This used to say "add it yourself", which also broke §0's own gate that
> `git status --porcelain -- docker-compose.yml` must print nothing. Confirm it is still there:
>
> ```bash
> docker inspect eventsnap-app-1 --format '{{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}'
> # 1073741824 1207959552 — the second number MUST be larger than the first but not double it.
> ```

---

## 6. Build and push — on the Mac

Enable **Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation"**, and set
**Resources → Memory ≥ 8 GB** (the fat-LTO step will OOM inside the VM otherwise, even with 18 GB on
the host).

```bash
docker run --rm --platform linux/amd64 alpine uname -m     # must print x86_64

cd /Users/fabianhammprivat/Projects/EventSnap
VERSION=v0.13.0                          # latest existing tag is v0.12.0 — see §9 before reusing it
ROLLBACK=v0.13.0-a                       # the SAME source, tagged twice; §9 explains why
SHA=$(git rev-parse --short HEAD)

docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/app:$SHA \
  --push ./backend

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/frontend:$VERSION \
  -t registry.mc02.dev/eventsnap/frontend:$ROLLBACK \
  -t registry.mc02.dev/eventsnap/frontend:$SHA \
  --push ./frontend
```

### Verify the architecture — never skip this

An arm64 image pulls fine and then dies with `exec format error`. Catch it here, not on the server:

```bash
docker buildx imagetools inspect registry.mc02.dev/eventsnap/app:$VERSION
docker buildx imagetools inspect registry.mc02.dev/eventsnap/frontend:$VERSION
# Both MUST report  Platform: linux/amd64
```

Then prove the binary actually executes. `AppConfig::from_env()` runs before any DB connection
(`main.rs` builds `AppConfig` before touching the pool), so this needs no database:

```bash
docker run --rm --platform linux/amd64 \
  -e APP_ENV=production -e JWT_SECRET=x -e DATABASE_URL=x -e EVENT_SLUG=x \
  registry.mc02.dev/eventsnap/app:$VERSION
```
Expect the "Refusing to start in production … placeholder" message. **That message means the amd64
binary ran.** `exec format error` means the architecture is wrong.

**Tag policy: never `latest` in production.** With `latest` you cannot tell what is running,
rollback becomes a registry re-push (impossible if the registry is down — exactly when you need it),
and `restart: unless-stopped` after a reboot is ambiguous. Two immutable tags per build: semver and
short SHA. Deploy by semver.

---

## 7. First deploy — on the server

```bash
# Directory name matters: volumes are prefixed with it, and README's backup commands
# hardcode the `eventsnap_` prefix.
git clone <repo> eventsnap && cd eventsnap
cp .env.example .env && nano .env          # every value from §3
docker login registry.mc02.dev
docker compose pull                        # must fully succeed before anything starts
```

### 7.1 The secret pre-flight — run this before the first `up -d`, always

```bash
docker compose run --rm --no-deps app
```

`--no-deps` means `db` never starts, so **no Postgres data directory is initialised**.
`AppConfig::from_env()` runs first and reports *every* placeholder at once (`config::validate_secrets`).
When the only remaining complaint is a database *connection* failure, the secrets are good.

**Why this step exists.** `POSTGRES_PASSWORD` is applied **only at initdb**. `docker-compose.yml`
starts `db` in the same command as `app`, so a single `up -d` with a placeholder bakes the wrong
password in permanently — the app then loops on `password authentication failed`, and the only exits
are `ALTER ROLE` or `down -v`, which deletes the database, the media and the exports. The repo
describes this trap at `backend/src/db.rs` (`explain_auth_failure`); this command is what avoids it.

### 7.2 Bring it up

```bash
# `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN.
set -a; . ./.env; set +a

docker compose up -d
docker compose logs -f app          # wait for "database connected and migrations applied"
curl -fsS https://$DOMAIN/health    # → ok  (503 means the app is up but the DB is not)
```

### 7.3 Verify the three things compose does *not* pin

`MEDIA_PATH` is pinned to `/media` on the `app` service in `docker-compose.yml`. Its siblings
are not:

```bash
docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH
```

1. **`DATABASE_URL`** must contain `@db:5432`. A dev `.env` points it at `@localhost`, which inside
   the container is the app itself.
2. **`EXPORT_PATH`** must be `/exports`. Anywhere else and the keepsake archives are written to the
   container's writable layer and **vanish on the next `up -d`** — including on a rollback.
3. **`ADMIN_PASSWORD_HASH`** must match `.env` **byte for byte.**

**Then actually log in to `/admin` with the real password.** This is not optional politeness:

- The production secret guard only rejects *placeholders* (`config::looks_placeholder`). A hash **corrupted**
  by shell or Compose escaping is not a placeholder — the app boots green, `/health` says `ok`, and
  every admin login 401s.
- The Admin user row is created **by a successful admin login** (`auth::handlers::admin_login`), and
  promoting anyone to Host requires an Admin or Host (`auth::middleware`'s role guard). **No admin login
  ⇒ no host, ever** ⇒ you cannot close the event, release the gallery, ban anyone, reset a PIN, or
  change any limit — for the whole event.

Per the Compose spec, single-quoted `env_file` values *are* used literally and the quotes are
stripped, so the single-quote form in `.env.example` is correct. The comment in
`docker-compose.dev.yml` claiming production has the same bug is **stale**. Verify anyway —
the cost of checking is 10 seconds; the cost of being wrong is the whole event.

**Promote a second person to Host** once you are in, so a single lost session is not fatal.

---

## 8. Post-deploy verification

```bash
set -a; . ./.env; set +a                                     # $DOMAIN comes from .env, not your shell

docker compose ps                                            # db, app, frontend healthy; caddy has
                                                             # no healthcheck and shows only "running"
docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1   # must be 1073741824, not 0
curl -fsS https://$DOMAIN/health                             # ok — now a real DB check, not a constant
docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
docker builder prune -af && docker image prune -f            # reclaim build cache
df -h /var/lib/docker
```

Then, from a phone on cellular (not the office wifi):

- [ ] Join as a guest with a PIN
- [ ] Upload a photo → appears in the feed within a few seconds
- [ ] Upload a video → plays back (iOS Safari range requests)
- [ ] Add a caption, add a like — **no comment UI anywhere**
- [ ] Admin login works; host dashboard reachable
- [ ] Release the gallery on a test event and download both archives

---

## 9. Rollback

### There is no older image you can roll back to. Build the rollback target yourself.

Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the
previous released tag — **takes the app down permanently** and looks like a crash loop with no
explanation:

```
$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
12          # 6 migrations. HEAD has 31.
$ git rev-list --count v0.12.0..HEAD
196
```

`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration
tree, booting against a database that already carries versions 007–022, returns `VersionMissing`.
`create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy
still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.)
No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with
`manifest unknown` before you ever got that far.

**So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The
rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical
migration set, which makes it a genuine no-op rather than a gamble:

```bash
# In §6, push both tags from the one build:
VERSION=v0.13.0
ROLLBACK=v0.13.0-a          # same source, different name — the rollback target

docker buildx build --platform linux/amd64 \
  -t registry.mc02.dev/eventsnap/app:$VERSION \
  -t registry.mc02.dev/eventsnap/app:$ROLLBACK \
  --push ./backend
# ...and the same two tags for ./frontend
```

**Rolling back — ~30 seconds, no network:**

```bash
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env
docker compose up -d app frontend
```

This only works offline if both are already resident. **Pre-pull all four at T‑2:**

```bash
docker pull registry.mc02.dev/eventsnap/app:v0.13.0
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a
docker image ls | grep eventsnap        # confirm all four
```

Once resident, `up -d`, reboots, restarts and rollbacks need **zero** registry contact. That single
step makes a registry outage on event day irrelevant.

Be clear-eyed about what this buys you: an identical image cannot undo a bad *release*, only an
image that got corrupted or a container that wedged — and `docker compose restart app` already
covers both. It exists so that the rollback line in the emergency card is safe to run rather than
catastrophic. **If you genuinely need to undo a code change during the event, you cannot; freeze
early enough that you never have to.**

**Across a migration boundary — avoid by freezing.** If you must: all 31 migrations have paired
`.down.sql` files, but **none of them removes its own `_sqlx_migrations` row**, so that second step
is mandatory and undocumented:

```bash
docker compose stop app
# `sh -c` so the CONTAINER expands the credentials. They live in the container's environment
# and in .env — not in your shell — so an unwrapped `-U "$POSTGRES_USER"` sends `-U ""` and
# psql answers `FATAL: role "" does not exist`.
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' \
  < backend/migrations/0NN_x.down.sql
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"'
```

Migration **014** is the only destructive one on the way up, and the only one with a rehearsal
harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump before the event.

**Registry-down transport fallback** (layers are already compressed — do not add gzip):

```bash
docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
            registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load'
```

---

## 10. Backup

Full commands are in `README.md:315-435` and are correct — `pg_dump --clean --if-exists`, plus
`alpine tar` out of `eventsnap_media_data` and `eventsnap_exports_data`, mounted at `/src` (not
`/media`), with `chown -R 100:101` on restore because the app runs non-root and BusyBox tar has no
`--same-owner`. There is deliberately no script.

Three gaps the README does not cover:

1. **Nothing backs up `.env`**, which holds the only copy of `POSTGRES_PASSWORD`. A dump you cannot
   authenticate against is not a backup. Copy `.env` off the box, encrypted, once it is final.
2. **The final dump is not a backup — it is an archive.** Taking it *after* locking uploads gives
   you a consistent pair, and that is the right way to archive the finished event. But it means
   that until the host locks uploads there is **no copy of anything anywhere**. All four volumes
   sit on the same 40 GB filesystem, on one VPS, with no redundancy. A disk or host failure at
   23:00 — the fullest the gallery will ever be — loses **100% of the event**, permanently, with
   the guests still in the room. Both of the mitigations below are required.
3. **Nothing is watching.** See §10.2.

### 10.1 Snapshots — do this once, before the event

> ⚠ **ACTION REQUIRED — Hetzner Cloud console, ~5 minutes, one checkbox.**
> Server → **Backups** → enable. Costs ~20% of the server price and needs no operator action
> ever again.

This is the single highest-value item in this runbook. It converts "total, permanent loss" into
"lose at most the hours since the last snapshot", automatically, with nobody awake. It covers the
whole volume set at once — database, media, exports and `.env` — which the `pg_dump` path does not.

It does **not** replace §10's archive: snapshots are whole-disk and crash-consistent, so restoring
one gives you the box back, not a portable copy of the photos. Do both.

### 10.2 A mid-event database dump — cheap, and the only thing cron should do

The database is small (a few MB — it holds rows, not pixels) and it is the part that cannot be
reconstructed: media files on disk without their `upload` rows are anonymous UUIDs with no
uploader, caption, hashtag or timestamp. Dumping it hourly costs essentially nothing and is safe
while uploads are live, because a `pg_dump` is transactionally consistent on its own.

Media is the bulk and *is* recoverable from guests' phones in the worst case, so it stays on the
event-night schedule below.

```bash
# On the server, before the event. Hourly DB-only dump, keeping the last 48.
mkdir -p /root/eventsnap-dumps
cat >/root/eventsnap-dump.sh <<'SH'
#!/bin/sh
set -eu
cd /root/eventsnap
set -a; . ./.env; set +a
OUT="/root/eventsnap-dumps/db-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
docker compose exec -T db sh -c \
  'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' | gzip >"$OUT.tmp"
mv "$OUT.tmp" "$OUT"          # atomic: never leave a truncated dump looking complete
ls -1t /root/eventsnap-dumps/db-*.sql.gz | tail -n +49 | xargs -r rm
SH
chmod +x /root/eventsnap-dump.sh
( crontab -l 2>/dev/null; echo '17 * * * * /root/eventsnap-dump.sh >>/var/log/eventsnap-dump.log 2>&1' ) | crontab -

# Prove it works NOW, not at 23:00:
/root/eventsnap-dump.sh && ls -lh /root/eventsnap-dumps/
```

These land on the same filesystem, so they do **not** survive a disk loss — that is what §10.1 is
for. They protect against the far more likely failure: a bad migration, an accidental host action,
or a corrupted table.

### 10.3 The event-night archive — unchanged

Take the dump and the media tarball back-to-back **the night of the event, after locking uploads
from the host dashboard**, so the pair is consistent. Copy both off the box before you sleep.

### 10.4 Monitoring — something has to be able to wake you

> ⚠ **ACTION REQUIRED — external uptime monitor, ~5 minutes.**
> Point any free monitor (UptimeRobot, Better Stack, Healthchecks.io — all have free tiers with
> SMS or push) at `https://$DOMAIN/health`, 1–5 minute interval, **alerting to a phone that will
> be on you during the event.**

There is otherwise **no** metrics collection, no alerting, no log shipping and no external check
anywhere in this deployment. Without this step, none of the following reaches a human: a crash
loop, a full disk, a dead database, an expired certificate, or the box being off. The host is at
a party and is not watching a dashboard.

`/health` is already built for exactly this and nothing currently consumes it:

| Response | Meaning | Action |
|---|---|---|
| `200 ok` | App **and** database are answering | — |
| `503 database timeout` / `database unavailable` | App is up, Postgres is not | §13 emergency card |
| Connection refused / TLS error | App container or Caddy is down | `docker compose ps`, then §13 |
| Timeout | Box is gone, or the disk is full enough to wedge it | §10.1 snapshot restore |

It runs a real `SELECT 1` against the pool with a 2 s timeout — a green check means the request
path guests use is genuinely working, not merely that a process is listening.

**The one signal this does not give you is disk.** The low-disk banner on `/host` requires the
host to open a dashboard during their own party and does not refresh without a manual reload, so
treat it as a pre-event check, not an alert. Before the event, confirm headroom with §11's
numbers; the export preflight and the upload quota are the automated backstops.

---

## 11. Disk — the numbers for 40 GB

Expected event (100 photos @ ~4 MB, 5 videos @ ~150 MB). Originals are **always kept** and never
lossily recompressed; derivatives are a ≤800 px preview and a ≤2048 px display JPEG.

```
media (originals + derivatives)                    ~1.25 GB
peak during keepsake build (+ Gallery + Memories)  ~3.5 GB
OS + Docker images + Postgres + logs               ~4.5 GB
────────────────────────────────────────────────────────────
total                             ~11–13 GB of ~36 GB usable
```

**40 GB fits with roughly 3× headroom**, provided you build elsewhere (a server-side build adds
3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against
*live* free space on every upload).

The `README.md:295-299` "ENOSPC" projection models guests **saturating the quota** (~12 GB of
media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully:
the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard
warns when free space drops below 10 GB or below the keepsake requirement (`handlers::host`'s low-disk thresholds).

---

## 12. Known issues you are shipping with

None of these has a fix in this runbook; they are listed so nothing is a surprise. Severity is
scored purely by "would this interrupt you during the event".

### Fixed in this pass

| Was | Fix |
|---|---|
| **Queued uploads never rehydrated** — `loadQueue()` had one call site, so a guest whose PWA was evicted mid-upload and reopened onto `/feed` had pending photos in IndexedDB that nothing ever read. Silent photo loss. | `loadQueue()` now runs on every authenticated boot in `+layout.svelte`. |
| **A corrupted `ADMIN_PASSWORD_HASH` booted green** and only failed at admin login — unrecoverable mid-event. | `config.rs` now validates the bcrypt *shape*, not just placeholder-ness, and refuses to start. |
| **SSE reconnect thundering herd** — flat 500 ms jitter regardless of backoff. | Jitter now scales with the delay; the feed's in-place refresh is spread over 800–2800 ms. |
| **No client-side size/HEIC pre-check** — a doomed 600 MB upload saturated the venue AP before being rejected. | Certain-rejects are refused before any bytes leave the phone. |
| **Upload-queue UI unreachable** — the badged FAB opened the picker, not the queue, so a failed upload had no retry button. | The upload sheet now shows a queue entry whenever the badge is non-zero. |
| **A mid-event 401 left a dead screen** with no nav and no URL bar. | `api.ts` now returns the guest to `/join` (skipping the auth routes themselves). |
| **Rate limiter could panic while holding its mutex** (`timestamps[0]` with `max == 0`), poisoning it process-wide. | Uses `first()` with a fail-open guard. |
| **Hashtag deadlock** — `edit_upload` and `add_comment` upserted tags in client/text order. | Both now sort + dedup on the normalised key, matching the upload path. |
| **Unbounded Docker logs + debug-level app logging.** | `logging:` caps every service at 30 MB; `RUST_LOG` documented in `.env.example`. |

### Fixed in the readiness pass — behaviour you should know about

These change what you will observe on the night, so they are listed separately from the table above.

| Was | Now |
|---|---|
| **Any ffmpeg-level failure DESTROYED the video.** A poster-frame error — ffmpeg hanging on a truncated `.mov`, an ENOSPC on `thumbnails/`, a DB blip — propagated into the give-up path, which soft-deleted the upload. Reproduced live: on a host with no ffmpeg the clip was gone ~6 s after its `201 Created`. | No failure in the video branch can fail the upload. The clip stays in the feed and plays from its original; only the poster is missing. |
| **A lost response duplicated the photo.** Every retry minted a fresh upload id, so a phone that lost the reply and re-sent — manually, or automatically on reconnect — stored the same photo two or three times and paid quota for each. | The client sends a `client_upload_id`; migration 022 makes it unique. A retry returns `200` with the original row. Verified live: three sends → one row, quota charged once. |
| **`/health` returned a constant `"ok"`.** The container reported healthy while every request 500'd. | Runs `SELECT 1` with a 2 s timeout: `200 ok` / `503`. Verified live: 200 → stop Postgres → 503 → start Postgres → 200, **with no app restart** (the pool revalidates on acquire). Note that Compose does not restart on an unhealthy probe — this is a diagnostic, deliberately not wired to automatic recovery, because a restart would truncate in-flight uploads to "fix" an outage that clears on its own. |
| **Abandoned `.tmp` uploads were never reclaimed** (see the old issue #2 below — it is now fixed, not deferred). | Swept at boot and hourly, keyed on modification time so a live upload can never age into it. |
| **A full disk made itself worse.** ENOSPC was retried three times, then refunded the quota, soft-deleted the row and *kept* the bytes — freeing nothing and inviting an immediate re-upload into the same full disk. | ENOSPC is classified separately: no retry, no refund, no delete. The row stays live and the photo is served from its original until there is room to compress it. |
| **The keepsake download failed invisibly.** The rate limit was enforced inside the iframe navigation, so an over-limit guest tapped and *nothing happened*, forever. | Charged when the ticket is minted — a normal `fetch` — so it surfaces as a German message naming the daily window. Raise `export_rate_per_day` per §4 anyway. |
| **`/host` and `/admin` subscribed to SSE but never opened the connection**, so the keepsake progress bar froze after a release. | Both connect on mount and disconnect on destroy, as `/export` already did. |
| **`?limit=-5` on the feed returned a 500.** | Clamped at both ends. |
| **All recurring hygiene lived in one unsupervised task** — one panic silently stopped session pruning, media reclamation and the temp sweep for the rest of the event. | Supervised and re-spawned, with an error log. |
| **No pool acquire or statement timeout.** A DB blip parked every request for 30 s. | 5 s acquire, `statement_timeout=15s`, `lock_timeout=5s`. |

### Still open — accepted for this event

| # | Issue | Impact |
|---|---|---|
| 1 | **Guest delete/caption-edit after release invalidates both keepsake archives** (`upload.rs`), forcing a rebuild with a 20 s debounce. **No-op before release**, so it cannot bite during the event. | Post-event only. Left alone because blocking guest deletes has real privacy downsides — that is a product call, not a bug fix. |
| 2 | **Video poster frames do not regenerate after a restart** mid-compression (the backfill filters `mime_type LIKE 'image/%'`). | Cosmetic — the video still plays; only its poster is missing. |
| 3 | **SSE keep-alives are sent as SSE comments** (`:ping`), which the browser's EventSource parser discards without dispatching. A client therefore cannot implement a pure silence timer to detect a half-open socket. | Worked around client-side: the feed runs a jittered 60–120 s `/feed/delta` backstop and reconnects when a poll returns content the stream never delivered. A cleaner fix is to emit keep-alives as a *named* event; that is a coordinated backend+frontend change, not worth making during a freeze. |
| 4 | **The lightbox stops at the end of the loaded page** — stepping past the last loaded photo does not fetch the next one. | The guest scrolls the feed (which does page) and re-opens. |

### Migration checksum mismatch — `VersionMissing` / "previously applied but has been modified"

`sqlx` compares **checksums**, so renaming or renumbering a migration file is indistinguishable
from editing one. If a box ever booted an image built from a branch that numbered migrations
differently, the next boot aborts with *"migration 21 was previously applied but has been
modified"*, `main` exits non-zero, and `restart: unless-stopped` makes it **permanent** — with
Caddy still routing traffic to the dead container.

Every main-line migration is byte-identical to what shipped, so a box that only ever ran tagged
releases is unaffected. **Verify rather than assume** — run this against the server before any
deploy.

Note the `sh -c` wrapping, for the same reason as §9: `POSTGRES_USER` and `POSTGRES_DB` live in
`.env`, which Compose reads and **your shell does not**. Unwrapped, `-U "$POSTGRES_USER"` sends
`-U ""` and psql answers `FATAL: role "" does not exist` — at 11pm, with the app crash-looping.

```bash
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT version, description, success FROM _sqlx_migrations ORDER BY version;"'
```

If the app is already crash-looping on a renumbered migration, and **only** if you have confirmed
the SQL in the new file is equivalent to what was actually applied. Take the version numbers from
the crash message and the query above — do **not** copy the ones below, which are an example:

```bash
docker compose stop app
# Replace 21,22,23 with the versions the boot error actually named.
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version IN (21,22,23);"'
docker compose start app     # re-applies exactly those, then continues
```

This re-runs those migrations. They must be idempotent (`IF NOT EXISTS` / `IF EXISTS`) or this
fails differently. Take a `pg_dump` first — see §10.

### Schema changes are **not** compile-time checked

All ~120 queries use the runtime `sqlx::query()` API. There are no `query!` macros and no `.sqlx`
cache, and the backend **compiles with no `DATABASE_URL` at all**. Consequences:

- A migration that renames or drops a column **compiles clean** and fails in production as a
  runtime 500 on whichever request path touches it first.
- `cargo build` succeeding tells you nothing about schema/query agreement. Only `cargo test`
  (which runs against a real Postgres) and manual exercise of the affected route do.

So: after any migration that touches an existing column, exercise the routes that read it before
you consider the deploy done. README and PROJECT previously claimed compile-time checking; they
have been corrected.

---

## 13. Event-day emergency card

Assume you have a phone and two minutes.

```bash
cd ~/eventsnap

# FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this,
# every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage
# when the site is fine.
set -a; . ./.env; set +a

# Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not)
curl -fsS https://$DOMAIN/health

# What is broken?
docker compose ps
docker compose logs --tail=100 app

# Nuclear option that is SAFE (keeps all data):
docker compose restart app

# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
# Do NOT substitute an older release tag here; it will crash-loop on the migration set.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend

# Disk check
df -h /var/lib/docker
```

### "Der Speicher des Events ist fast voll" — guests cannot upload

**`df -h` will look fine, and that is not a contradiction.** The upload gate refuses long before the
disk fills: it reserves room for the keepsake, which is roughly a second copy of every original, plus
a 10 GB floor. Uploads stop at **~8 GB of media** on a 40 GB box, when `df` still shows ~20 GB free.

Check the number that actually binds, not free space:

```bash
docker compose exec -T db sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -tAc "SELECT pg_size_pretty(sum(original_size_bytes)) FROM upload WHERE deleted_at IS NULL;"'
```

Mid-event, in order of preference: delete the largest videos from the host dashboard (each frees its
own bytes immediately), or move `exports_data` to a separate volume. Raising `quota_tolerance` will
**not** help — on this box every guest is already on the 500 MiB floor, so that knob is not what is
refusing them (see §4 and `.env.example`).

**NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS
certificate. There is no undo.

Most limits are changeable from the **admin dashboard without a restart** — reach for that before
touching the shell.