# EventSnap — Production Deployment Runbook Target: **Hetzner CX22** (2 vCPU, 4 GB RAM, 40 GB disk), single host, docker compose. Event profile: ~100 guests, ~100 photos + a few videos, one evening, **operator attending and unavailable to troubleshoot**. Everything here is written for that last constraint. Where a choice trades throughput for "cannot need attention on the night", it takes the stable option. > **Note on this repo's own docs.** `README.md:62` and `PROJECT.md:359` specify a **CX33 > (4 vCPU / 8 GB / 80 GB)**. You are deploying to half of that on every axis. Two pieces of > tuning advice in `.env.example` are calibrated for the CX33 and are actively wrong for a > CX22 — they are called out in §3. Trust this file over `.env.example` for sizing. --- ## 0. Timeline — the single most important control ### Step zero: commit everything, before you build anything **The production deployment does not exist in git yet.** At the time of writing, `docker-compose.yml` and `.env.example` are modified but uncommitted, and `DEPLOYMENT_RUNBOOK.md`, `docker-compose.build.yml`, both `.dockerignore` files and migrations `021`/`022` are untracked. None of them is gitignored — they are simply not committed. That is not a tidiness problem, it is the deployment failing in two ways at once: - §7 tells you to `git clone` onto the server. `git show HEAD:docker-compose.yml` still has `build:` keys and **no `image:` keys**, so on that clone `docker compose pull` skips both services and `docker compose up -d` starts **a fat-LTO release build of 427 crates on the CX22** — the exact scenario §1 rules out as an expected OOM. The committed file also has no log rotation. - `sqlx::migrate!()` embeds `./migrations` **at compile time**. A build from the working tree bakes in 021 and 022 and applies them on first boot; any later rebuild from a clean clone produces an image that lacks them and crash-loops with `VersionMissing` against its own database. ```bash git add docker-compose.yml docker-compose.dev.yml docker-compose.build.yml .env.example \ backend/.dockerignore frontend/.dockerignore frontend/Dockerfile \ backend/migrations/021_*.sql backend/migrations/022_*.sql \ DEPLOYMENT_RUNBOOK.md git commit -m "chore: production compose, ignore files and runbook" git push # Prove it landed — this must print two `image:` lines and nothing about `build:` git show HEAD:docker-compose.yml | grep -E 'image:|build:' git show HEAD --stat | grep -c migrations/02 # must be 4 ``` | When | What | |---|---| | **T‑7 days** | Commit and push everything above. Registry + DNS pre-flight (§5). Build and push images (§6). | | **T‑5 days** | First deploy to the server (§7). Verify admin login. Leave it running. | | **T‑3 days** | **Freeze migrations.** No further code deploys unless something is broken. | | **T‑2 days** | Pre-pull current *and* previous image tags (§9). Run the backup rehearsal (§10). | | **Event day** | Change nothing. Configuration tweaks via the admin dashboard only (§4). | **Why the freeze matters more than anything else here.** Migrations run automatically at boot (`backend/src/db.rs`, `create_pool`) and `sqlx::migrate!()` is used *without* `set_ignore_missing`. An older image booted against a newer schema does not degrade — it **crash-loops** with `VersionMissing`. The repo documents this itself in `backend/migrations/014_export_epoch.up.sql:3-8`. Once the migration set is frozen, rollback is a one-line `.env` edit; before it is frozen, rollback means hand-running down-SQL under pressure. --- ## 1. Deployment strategy — build on the Mac, push, pull **Decision: build `linux/amd64` images on the M3 Pro, push to `registry.mc02.dev`, pull on the server.** Rejected alternatives, with the reason each loses: | Option | Why not | |---|---| | **Build on the CX22** | `backend/Cargo.toml` sets `lto = true` + `codegen-units = 1` over **427 crates**. Fat LTO links the whole program in one single-threaded process; peak RSS is estimated at 2.5–4 GB against ~1.5–2.5 GB free with the stack running. OOM is the expected outcome, not a tail risk. And *rollback would also be a build* — 35–60 min under pressure. | | **CI (Gitea on Pi 5)** | Ruled out by you, and correct: a Pi 5 is strictly worse than the CX22 for a fat-LTO link. | | **True cross-compilation** (`--target x86_64-unknown-linux-musl`) | The dependency graph has **four C-compiling crates** — `ring` (per-arch assembly), `zstd-sys`, `libdeflate-sys`, `libsqlite3-sys` — so it needs a real musl cross-toolchain. Alpine has no such package for an arm64 host; the working answer is `cargo-zigbuild`, which means a new base image and a new toolchain days before an unrepeatable event. **Slow but certain beats fast but novel.** | Emulated build is the right call because the cost lands where it is free: on your laptop, days early, with nothing depending on it. Estimated wall-clock for the backend is **25–60 min with Rosetta enabled** (hours without it) — start it and walk away. > **Escape hatch if emulation is unbearable:** spin up a temporary Hetzner CPX41 (8 vCPU/16 GB) in > the same account, build natively, push, destroy it. Costs cents, same commands, no repo change. --- ## 2. Repo changes — ALREADY APPLIED > **Status: these are done, in the working tree.** They are documented here so you know what > changed and why, not as work to repeat. Run `git diff` to review before committing. ### 2.1 `.dockerignore` files (were absent) `backend/.dockerignore`: ``` target/ ``` `frontend/.dockerignore`: ``` node_modules/ .svelte-kit/ build/ ``` Without these, the first time you run `cargo build` or `npm install` locally, every image build ships a multi-GB context to an *emulated* builder. Worse: `frontend/Dockerfile:9` does `COPY . .` **after** `npm ci`, so a macOS `node_modules/` would be merged over the container's Linux one. ### 2.2 Switch compose from `build:` to `image:` In `docker-compose.yml`, replace the `build:` block on `app` and `frontend`: ```yaml app: image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env} ``` ```yaml frontend: image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env} ``` Two deliberate properties: - The `:?` form **fails loudly** on an unset variable instead of resolving to an empty tag. - **Removing `build:` entirely is a safety feature.** With no `build:` key on the server, a wrong tag is an instant `manifest unknown` — never a surprise 45-minute compile on the production box. New `docker-compose.build.yml` (opt-in, Mac only — follows the convention stated in the header of `docker-compose.dev.yml` that overlays are never auto-loaded): ```yaml # Build overlay. NOT loaded automatically. Used only where images are BUILT — never on the # production server, which pulls. # docker compose -f docker-compose.yml -f docker-compose.build.yml build services: app: build: { context: ./backend, dockerfile: Dockerfile } frontend: build: { context: ./frontend, dockerfile: Dockerfile } ``` `e2e/docker-compose.test.yml` keeps its own `build:` blocks, so the e2e gate is unaffected. ### 2.3 Add log rotation to every service `docker-compose.yml` currently sets **no logging config**, so the default `json-file` driver keeps logs forever, on the same filesystem as Postgres and the media. Add to each of the four services: ```yaml logging: driver: json-file options: { max-size: "10m", max-file: "3" } ``` (Equivalently, set it once in `/etc/docker/daemon.json` — but that restarts the Docker daemon, so do it *before* the stack is live, not after.) --- ## 3. `.env` — production values ```bash # ── Identity ────────────────────────────────────────────────────────────── DOMAIN= EVENT_NAME=<...> EVENT_SLUG=<...> # ── Image version (NEW — drives the image: tags in docker-compose.yml) ──── EVENTSNAP_VERSION=v0.13.0 # ── Secrets — ALL of them, before the first `up -d` ─────────────────────── JWT_SECRET= POSTGRES_PASSWORD= DATABASE_URL=postgres://eventsnap:@db:5432/eventsnap ADMIN_PASSWORD_HASH='' # ── Paths — must match the volume mounts ────────────────────────────────── MEDIA_PATH=/media # pinned by compose anyway, but keep consistent EXPORT_PATH=/exports # NOT pinned by compose — see the trap in §7.3 # ── Sizing (see the two corrections below) ──────────────────────────────── DATABASE_MAX_CONNECTIONS=30 COMPRESSION_WORKER_CONCURRENCY=2 # ── Comments off, likes + captions on ───────────────────────────────────── COMMENTS_ENABLED=false # ── Logging (NEW — production currently defaults to DEBUG) ──────────────── RUST_LOG=eventsnap_backend=info,tower_http=warn ``` ### Two corrections to the repo's own advice — do not follow `.env.example` here **`COMPRESSION_WORKER_CONCURRENCY`: keep `2`. Do NOT raise to 4, and do NOT raise the app memory limit to 2G.** `.env.example:92-98` justifies both on the premise that "each worker can run an ffmpeg transcode". **There is no transcode anywhere in this codebase.** `services/video.rs::run_ffmpeg` runs `ffmpeg -ss -i -vframes 1 -vf scale=…` — a single poster frame. Video originals are stored and served byte-for-byte. The real memory consumer is the **image** path: `image` 0.25's `resize` builds an `Rgba32F` intermediate at **16 bytes/px**, sized `source_width × target_height`, which the 256 MiB decode guard in `imaging::decode_limits` does not cover. Estimated peak per photo: | Source | Peak (decode + display resize) | |---|---| | 12 MP (typical phone) | ~145 MB | | 24 MP (iPhone Pro default) | ~223 MB | | 48 MP ("Max" mode) | ~354 MB | At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4 the same pair is ~1.5 GB → **OOM**. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives. **`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.** See §4. --- ## 4. Admin dashboard configuration These live in the DB `config` table, not `.env`. Changes take effect on the **next request** — `patch_config` invalidates the cache synchronously (`admin::patch_config`). No restart needed. Set them after the first deploy, before the event. | Key | Default | **Set to** | Why | |---|---|---|---| | `upload_rate_per_hour` | 100 | **1000** | A guest multi-selecting 100 photos hits exactly 100. Client backs off and resumes, but this makes it a non-issue. | | `feed_rate_per_min` | 60 | **240** | Headroom for reconnect bursts. | | `social_rate_per_min` | 120 | **600** | This is the **likes** limiter — the interaction you're keeping. | | `export_rate_per_day` | 3 | **20** | **One shared bucket across both archives** — `enforce_export_rate` keys on `export:{user_id}` regardless of which archive is being fetched. Downloading Gallery.zip + Memories.zip costs 2 of 3; one retry locks a guest out for 24 h. The limit is now charged when the download *ticket* is minted, so being over it produces a visible German error instead of a tap that silently does nothing. | | `max_video_size_mb` | 500 | **leave at 500** | See below. | | `quota_tolerance` | 0.75 | **leave at 0.75** | See below. | | `quota_enabled`, `storage_quota_enabled`, `rate_limits_enabled` | true | **leave on** | This is the only disk-full safety net. | **Ignore `estimated_guest_count` and `upload_count_quota_enabled`.** Both are seeded, validated and rendered in the admin UI — and **read by no code at all** (verified by grep across `backend/src`). Changing them does nothing. `estimated_guest_count` in particular does *not* feed the quota formula. ### Why quotas are already as generous as you want ``` per_user_limit = floor(free_disk × quota_tolerance / max(active_uploaders, 1)) ``` `active_uploaders` is `SELECT COUNT(DISTINCT user_id) FROM upload WHERE deleted_at IS NULL` (`upload::compute_storage_quota`) — **people who actually uploaded**, not guests who joined. The 70 guests who never upload are not in the denominator; their share flows to the photographers automatically. **The redistribution you asked for is already the design.** With ~28 GB free and a realistic 30 people actually uploading, each gets **~700 MB** — against an expected ~1.25 GB for the *entire event*. Nobody will be blocked. Raising `quota_tolerance` would only raise the **saturation ceiling** (media converges to `t/(1+t)` of free space: 43% at 0.75, 50% at 1.0). It does nothing for a real guest at your volume, and it eats the headroom the keepsake needs — the export preflight reserves `media × 1.10 × 2` because `Gallery.zip` and `Memories.zip` each store media **uncompressed** (`export.rs` — both archives store media uncompressed). **Why `max_video_size_mb` stays at 500.** An earlier draft of this runbook said to lower it to 250, on the grounds that there is no client-side size check. That was wrong on both halves: - A client guard **does** exist — `frontend/src/routes/upload/+page.svelte` rejects anything over `HARD_MAX_UPLOAD_BYTES` (576 MiB) before a byte leaves the phone, alongside the HEIC reject. - That guard is a **compile-time constant**, not the DB value. Lowering `max_video_size_mb` therefore changes nothing on the client: the picker still accepts the clip, the phone still starts sending it, and the *server* aborts it partway (`stream_field_to_file` does stop at the cap, so the AP is not saturated for the full file — but the guest still gets a mid-upload failure where 500 MB would simply have worked). So lowering it is a pure capacity decision with no UX upside, and at your volume there is no capacity problem to solve. Leave it. If you ever do want a smaller ceiling to bite on the phone, the constant above has to move with it. --- ## 5. Pre-flight — T‑7 days (do NOT leave this to event week) ### Registry From **both** the Mac and the server, as **the user who will deploy** (root's `docker login` does not help a non-root deploy — credentials go to `~/.docker/config.json`): ```bash getent hosts registry.mc02.dev # DNS resolves from the server curl -fsSI https://registry.mc02.dev/v2/ # TLS must be PUBLICLY trusted; Docker rejects # self-signed without extra daemon config docker login registry.mc02.dev ``` Also confirm: the `eventsnap` repository/namespace exists if your registry requires pre-creation (Harbor does; plain `registry:2` does not), and the registry host has ~500 MB free per release. ### DNS and TLS — get this wrong and you are locked out for an hour The DNS **A record must point at the CX22 before the first `docker compose up -d`**, because Caddy attempts certificate issuance on boot. Let's Encrypt allows only **5 failed validations per hostname per hour** (refilling 1 per 12 min). A misconfigured DNS record plus a few impatient restarts will rate-limit you out of getting a certificate at all. - Deploy days early so issuance happens with no time pressure. - **Never delete the `caddy_data` volume** — it holds the certificate and the ACME account key. - Never run `docker compose down -v`. It destroys `postgres_data`, `media_data`, `exports_data` *and* `caddy_data`. ### Host preparation ```bash docker compose version # must be v2.x — the deploy.resources limits need it free -h && swapon --show # Hetzner images ship no swap df -h /var/lib/docker # want ≥ 25 GB free ``` **Add 2 GB of swap** as an OOM cushion — a compression spike that would otherwise kill the container instead swaps out cold pages and merely runs slowly: ```bash fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile echo '/swapfile none swap sw 0 0' >> /etc/fstab sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf ``` > **Gotcha:** Compose sets each container's `Memory` limit but leaves `MemorySwap` unset, and Docker > then allows swap equal to the memory limit — so adding host swap silently **doubles** every > container ceiling. If you add swap, also add `memswap_limit: 1152m` to `app` and `db`, and > `memswap_limit: 320m` to `frontend` and `caddy` (service-level, not under `deploy:`). --- ## 6. Build and push — on the Mac Enable **Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation"**, and set **Resources → Memory ≥ 8 GB** (the fat-LTO step will OOM inside the VM otherwise, even with 18 GB on the host). ```bash docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64 cd /Users/fabianhammprivat/Projects/EventSnap VERSION=v0.13.0 # latest existing tag is v0.12.0 — see §9 before reusing it ROLLBACK=v0.13.0-a # the SAME source, tagged twice; §9 explains why SHA=$(git rev-parse --short HEAD) docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap docker buildx build --platform linux/amd64 \ -t registry.mc02.dev/eventsnap/app:$VERSION \ -t registry.mc02.dev/eventsnap/app:$ROLLBACK \ -t registry.mc02.dev/eventsnap/app:$SHA \ --push ./backend docker buildx build --platform linux/amd64 \ -t registry.mc02.dev/eventsnap/frontend:$VERSION \ -t registry.mc02.dev/eventsnap/frontend:$ROLLBACK \ -t registry.mc02.dev/eventsnap/frontend:$SHA \ --push ./frontend ``` ### Verify the architecture — never skip this An arm64 image pulls fine and then dies with `exec format error`. Catch it here, not on the server: ```bash docker buildx imagetools inspect registry.mc02.dev/eventsnap/app:$VERSION docker buildx imagetools inspect registry.mc02.dev/eventsnap/frontend:$VERSION # Both MUST report Platform: linux/amd64 ``` Then prove the binary actually executes. `AppConfig::from_env()` runs before any DB connection (`main.rs` builds `AppConfig` before touching the pool), so this needs no database: ```bash docker run --rm --platform linux/amd64 \ -e APP_ENV=production -e JWT_SECRET=x -e DATABASE_URL=x -e EVENT_SLUG=x \ registry.mc02.dev/eventsnap/app:$VERSION ``` Expect the "Refusing to start in production … placeholder" message. **That message means the amd64 binary ran.** `exec format error` means the architecture is wrong. **Tag policy: never `latest` in production.** With `latest` you cannot tell what is running, rollback becomes a registry re-push (impossible if the registry is down — exactly when you need it), and `restart: unless-stopped` after a reboot is ambiguous. Two immutable tags per build: semver and short SHA. Deploy by semver. --- ## 7. First deploy — on the server ```bash # Directory name matters: volumes are prefixed with it, and README's backup commands # hardcode the `eventsnap_` prefix. git clone eventsnap && cd eventsnap cp .env.example .env && nano .env # every value from §3 docker login registry.mc02.dev docker compose pull # must fully succeed before anything starts ``` ### 7.1 The secret pre-flight — run this before the first `up -d`, always ```bash docker compose run --rm --no-deps app ``` `--no-deps` means `db` never starts, so **no Postgres data directory is initialised**. `AppConfig::from_env()` runs first and reports *every* placeholder at once (`config::validate_secrets`). When the only remaining complaint is a database *connection* failure, the secrets are good. **Why this step exists.** `POSTGRES_PASSWORD` is applied **only at initdb**. `docker-compose.yml` starts `db` in the same command as `app`, so a single `up -d` with a placeholder bakes the wrong password in permanently — the app then loops on `password authentication failed`, and the only exits are `ALTER ROLE` or `down -v`, which deletes the database, the media and the exports. The repo describes this trap at `backend/src/db.rs` (`explain_auth_failure`); this command is what avoids it. ### 7.2 Bring it up ```bash # `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN. set -a; . ./.env; set +a docker compose up -d docker compose logs -f app # wait for "database connected and migrations applied" curl -fsS https://$DOMAIN/health # → ok (503 means the app is up but the DB is not) ``` ### 7.3 Verify the three things compose does *not* pin `MEDIA_PATH` is pinned to `/media` on the `app` service in `docker-compose.yml`. Its siblings are not: ```bash docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH ``` 1. **`DATABASE_URL`** must contain `@db:5432`. A dev `.env` points it at `@localhost`, which inside the container is the app itself. 2. **`EXPORT_PATH`** must be `/exports`. Anywhere else and the keepsake archives are written to the container's writable layer and **vanish on the next `up -d`** — including on a rollback. 3. **`ADMIN_PASSWORD_HASH`** must match `.env` **byte for byte.** **Then actually log in to `/admin` with the real password.** This is not optional politeness: - The production secret guard only rejects *placeholders* (`config::looks_placeholder`). A hash **corrupted** by shell or Compose escaping is not a placeholder — the app boots green, `/health` says `ok`, and every admin login 401s. - The Admin user row is created **by a successful admin login** (`auth::handlers::admin_login`), and promoting anyone to Host requires an Admin or Host (`auth::middleware`'s role guard). **No admin login ⇒ no host, ever** ⇒ you cannot close the event, release the gallery, ban anyone, reset a PIN, or change any limit — for the whole event. Per the Compose spec, single-quoted `env_file` values *are* used literally and the quotes are stripped, so the single-quote form in `.env.example` is correct. The comment in `docker-compose.dev.yml` claiming production has the same bug is **stale**. Verify anyway — the cost of checking is 10 seconds; the cost of being wrong is the whole event. **Promote a second person to Host** once you are in, so a single lost session is not fatal. --- ## 8. Post-deploy verification ```bash set -a; . ./.env; set +a # $DOMAIN comes from .env, not your shell docker compose ps # all four healthy docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1 # must be 1073741824, not 0 curl -fsS https://$DOMAIN/health # ok — now a real DB check, not a constant docker compose exec app printenv COMMENTS_ENABLED RUST_LOG docker builder prune -af && docker image prune -f # reclaim build cache df -h /var/lib/docker ``` Then, from a phone on cellular (not the office wifi): - [ ] Join as a guest with a PIN - [ ] Upload a photo → appears in the feed within a few seconds - [ ] Upload a video → plays back (iOS Safari range requests) - [ ] Add a caption, add a like — **no comment UI anywhere** - [ ] Admin login works; host dashboard reachable - [ ] Release the gallery on a test event and download both archives --- ## 9. Rollback ### There is no older image you can roll back to. Build the rollback target yourself. Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the previous released tag — **takes the app down permanently** and looks like a crash loop with no explanation: ``` $ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l 12 # 6 migrations. HEAD has 22. $ git rev-list --count v0.12.0..HEAD 154 ``` `db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration tree, booting against a database that already carries versions 007–022, returns `VersionMissing`. `create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.) No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with `manifest unknown` before you ever got that far. **So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical migration set, which makes it a genuine no-op rather than a gamble: ```bash # In §6, push both tags from the one build: VERSION=v0.13.0 ROLLBACK=v0.13.0-a # same source, different name — the rollback target docker buildx build --platform linux/amd64 \ -t registry.mc02.dev/eventsnap/app:$VERSION \ -t registry.mc02.dev/eventsnap/app:$ROLLBACK \ --push ./backend # ...and the same two tags for ./frontend ``` **Rolling back — ~30 seconds, no network:** ```bash sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env docker compose up -d app frontend ``` This only works offline if both are already resident. **Pre-pull all four at T‑2:** ```bash docker pull registry.mc02.dev/eventsnap/app:v0.13.0 docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0 docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a docker image ls | grep eventsnap # confirm all four ``` Once resident, `up -d`, reboots, restarts and rollbacks need **zero** registry contact. That single step makes a registry outage on event day irrelevant. Be clear-eyed about what this buys you: an identical image cannot undo a bad *release*, only an image that got corrupted or a container that wedged — and `docker compose restart app` already covers both. It exists so that the rollback line in the emergency card is safe to run rather than catastrophic. **If you genuinely need to undo a code change during the event, you cannot; freeze early enough that you never have to.** **Across a migration boundary — avoid by freezing.** If you must: all 22 migrations have paired `.down.sql` files, but **none of them removes its own `_sqlx_migrations` row**, so that second step is mandatory and undocumented: ```bash docker compose stop app # `sh -c` so the CONTAINER expands the credentials. They live in the container's environment # and in .env — not in your shell — so an unwrapped `-U "$POSTGRES_USER"` sends `-U ""` and # psql answers `FATAL: role "" does not exist`. docker compose exec -T db sh -c \ 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' \ < backend/migrations/0NN_x.down.sql docker compose exec -T db sh -c \ 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"' ``` Migration **014** is the only destructive one on the way up, and the only one with a rehearsal harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump before the event. **Registry-down transport fallback** (layers are already compressed — do not add gzip): ```bash docker save registry.mc02.dev/eventsnap/app:v0.13.0 \ registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load' ``` --- ## 10. Backup Full commands are in `README.md:315-435` and are correct — `pg_dump --clean --if-exists`, plus `alpine tar` out of `eventsnap_media_data` and `eventsnap_exports_data`, mounted at `/src` (not `/media`), with `chown -R 100:101` on restore because the app runs non-root and BusyBox tar has no `--same-owner`. There is deliberately no script. Two gaps the README does not cover: 1. **Nothing backs up `.env`**, which holds the only copy of `POSTGRES_PASSWORD`. A dump you cannot authenticate against is not a backup. Copy `.env` off the box, encrypted, once it is final. 2. **Timing.** Do not use nightly cron — every irreplaceable byte is created inside one evening. Take the dump and the media tarball back-to-back **the night of the event, after locking uploads from the host dashboard**, so the pair is consistent. --- ## 11. Disk — the numbers for 40 GB Expected event (100 photos @ ~4 MB, 5 videos @ ~150 MB). Originals are **always kept** and never lossily recompressed; derivatives are a ≤800 px preview and a ≤2048 px display JPEG. ``` media (originals + derivatives) ~1.25 GB peak during keepsake build (+ Gallery + Memories) ~3.5 GB OS + Docker images + Postgres + logs ~4.5 GB ──────────────────────────────────────────────────────────── total ~11–13 GB of ~36 GB usable ``` **40 GB fits with roughly 3× headroom**, provided you build elsewhere (a server-side build adds 3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against *live* free space on every upload). The `README.md:295-299` "ENOSPC" projection models guests **saturating the quota** (~12 GB of media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully: the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard warns when free space drops below 10 GB or below the keepsake requirement (`handlers::host`'s low-disk thresholds). --- ## 12. Known issues you are shipping with None of these has a fix in this runbook; they are listed so nothing is a surprise. Severity is scored purely by "would this interrupt you during the event". ### Fixed in this pass | Was | Fix | |---|---| | **Queued uploads never rehydrated** — `loadQueue()` had one call site, so a guest whose PWA was evicted mid-upload and reopened onto `/feed` had pending photos in IndexedDB that nothing ever read. Silent photo loss. | `loadQueue()` now runs on every authenticated boot in `+layout.svelte`. | | **A corrupted `ADMIN_PASSWORD_HASH` booted green** and only failed at admin login — unrecoverable mid-event. | `config.rs` now validates the bcrypt *shape*, not just placeholder-ness, and refuses to start. | | **SSE reconnect thundering herd** — flat 500 ms jitter regardless of backoff. | Jitter now scales with the delay; the feed's in-place refresh is spread over 800–2800 ms. | | **No client-side size/HEIC pre-check** — a doomed 600 MB upload saturated the venue AP before being rejected. | Certain-rejects are refused before any bytes leave the phone. | | **Upload-queue UI unreachable** — the badged FAB opened the picker, not the queue, so a failed upload had no retry button. | The upload sheet now shows a queue entry whenever the badge is non-zero. | | **A mid-event 401 left a dead screen** with no nav and no URL bar. | `api.ts` now returns the guest to `/join` (skipping the auth routes themselves). | | **Rate limiter could panic while holding its mutex** (`timestamps[0]` with `max == 0`), poisoning it process-wide. | Uses `first()` with a fail-open guard. | | **Hashtag deadlock** — `edit_upload` and `add_comment` upserted tags in client/text order. | Both now sort + dedup on the normalised key, matching the upload path. | | **Unbounded Docker logs + debug-level app logging.** | `logging:` caps every service at 30 MB; `RUST_LOG` documented in `.env.example`. | ### Fixed in the readiness pass — behaviour you should know about These change what you will observe on the night, so they are listed separately from the table above. | Was | Now | |---|---| | **Any ffmpeg-level failure DESTROYED the video.** A poster-frame error — ffmpeg hanging on a truncated `.mov`, an ENOSPC on `thumbnails/`, a DB blip — propagated into the give-up path, which soft-deleted the upload. Reproduced live: on a host with no ffmpeg the clip was gone ~6 s after its `201 Created`. | No failure in the video branch can fail the upload. The clip stays in the feed and plays from its original; only the poster is missing. | | **A lost response duplicated the photo.** Every retry minted a fresh upload id, so a phone that lost the reply and re-sent — manually, or automatically on reconnect — stored the same photo two or three times and paid quota for each. | The client sends a `client_upload_id`; migration 022 makes it unique. A retry returns `200` with the original row. Verified live: three sends → one row, quota charged once. | | **`/health` returned a constant `"ok"`.** The container reported healthy while every request 500'd. | Runs `SELECT 1` with a 2 s timeout: `200 ok` / `503`. Verified live: 200 → stop Postgres → 503 → start Postgres → 200, **with no app restart** (the pool revalidates on acquire). Note that Compose does not restart on an unhealthy probe — this is a diagnostic, deliberately not wired to automatic recovery, because a restart would truncate in-flight uploads to "fix" an outage that clears on its own. | | **Abandoned `.tmp` uploads were never reclaimed** (see the old issue #2 below — it is now fixed, not deferred). | Swept at boot and hourly, keyed on modification time so a live upload can never age into it. | | **A full disk made itself worse.** ENOSPC was retried three times, then refunded the quota, soft-deleted the row and *kept* the bytes — freeing nothing and inviting an immediate re-upload into the same full disk. | ENOSPC is classified separately: no retry, no refund, no delete. The row stays live and the photo is served from its original until there is room to compress it. | | **The keepsake download failed invisibly.** The rate limit was enforced inside the iframe navigation, so an over-limit guest tapped and *nothing happened*, forever. | Charged when the ticket is minted — a normal `fetch` — so it surfaces as a German message naming the daily window. Raise `export_rate_per_day` per §4 anyway. | | **`/host` and `/admin` subscribed to SSE but never opened the connection**, so the keepsake progress bar froze after a release. | Both connect on mount and disconnect on destroy, as `/export` already did. | | **`?limit=-5` on the feed returned a 500.** | Clamped at both ends. | | **All recurring hygiene lived in one unsupervised task** — one panic silently stopped session pruning, media reclamation and the temp sweep for the rest of the event. | Supervised and re-spawned, with an error log. | | **No pool acquire or statement timeout.** A DB blip parked every request for 30 s. | 5 s acquire, `statement_timeout=15s`, `lock_timeout=5s`. | ### Still open — accepted for this event | # | Issue | Impact | |---|---|---| | 1 | **Guest delete/caption-edit after release invalidates both keepsake archives** (`upload.rs`), forcing a rebuild with a 20 s debounce. **No-op before release**, so it cannot bite during the event. | Post-event only. Left alone because blocking guest deletes has real privacy downsides — that is a product call, not a bug fix. | | 2 | **Video poster frames do not regenerate after a restart** mid-compression (the backfill filters `mime_type LIKE 'image/%'`). | Cosmetic — the video still plays; only its poster is missing. | | 3 | **SSE keep-alives are sent as SSE comments** (`:ping`), which the browser's EventSource parser discards without dispatching. A client therefore cannot implement a pure silence timer to detect a half-open socket. | Worked around client-side: the feed runs a jittered 60–120 s `/feed/delta` backstop and reconnects when a poll returns content the stream never delivered. A cleaner fix is to emit keep-alives as a *named* event; that is a coordinated backend+frontend change, not worth making during a freeze. | | 4 | **The lightbox stops at the end of the loaded page** — stepping past the last loaded photo does not fetch the next one. | The guest scrolls the feed (which does page) and re-opens. | --- ## 13. Event-day emergency card Assume you have a phone and two minutes. ```bash cd ~/eventsnap # FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this, # every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage # when the site is fine. set -a; . ./.env; set +a # Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not) curl -fsS https://$DOMAIN/health # What is broken? docker compose ps docker compose logs --tail=100 app # Nuclear option that is SAFE (keeps all data): docker compose restart app # Roll back to the identically-built sibling image — see §9 for what this can and cannot fix. # Do NOT substitute an older release tag here; it will crash-loop on the migration set. sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend # Disk check df -h /var/lib/docker ``` **NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS certificate. There is no undo. Most limits are changeable from the **admin dashboard without a restart** — reach for that before touching the shell.