deploy: pull prebuilt images instead of building on the event server
The production compose still carried `build:` keys and no `image:` keys, so a
`git clone` onto the CX22 followed by `docker compose up -d` would have started a
fat-LTO release build of 427 crates on a 2-vCPU/4 GB box — the outcome the whole
build-on-the-Mac decision exists to avoid, reached silently because `pull` skips a
service it is told to build rather than failing.
Both services now pull `registry.mc02.dev/eventsnap/*:${EVENTSNAP_VERSION}` with the
`:?` form, so a missing tag fails the command instead of resolving to an empty one.
`docker-compose.build.yml` restores the `build:` keys for the workstation that
produces the images, from the same context paths.
Also here:
- `DOMAIN` gets the same `:?` guard. Blank did not fail — it produced `https://` for
the frontend's ORIGIN and collapsed the Caddyfile's site block into a malformed
global block, so the stack came up with no TLS and no site.
- `stop_grace_period: 20s` on the app. Docker's default stop timeout is 10s, exactly
the app's own drain budget, so a redeploy could SIGKILL the process at the moment it
was finishing — truncating the in-flight upload the graceful shutdown protects.
- `COMMENTS_ENABLED` is pinned "false" alongside MEDIA_PATH. It is a product decision
for this event, and `.env.example` ships the generic `true`; pinning it means an
operator who copies the example and edits only the secrets cannot ship comments on.
- The frontend runtime stage now copies the lockfile and uses `npm ci`. Without it the
three `^`-ranged deps re-resolved at build time, so an image rebuilt days later could
differ from the one that was tested. Image also drops 120 MB -> 65 MB.
- `docker-compose.dev.yml` told the operator that production had the same `$`-eating
bug and to escape the hash as `$$` in `.env`. That is wrong and it breaks a working
deployment: Compose uses single-quoted env_file values literally, and doubling
produces a 74-character string `looks_bcrypt` rejects. Verified with `printenv`.
The runbook's rollback pointed at `v0.12.0`, which has 6 migrations against HEAD's 22
and was never built or pushed — running the emergency card's rollback line would have
crash-looped the app with `VersionMissing` during the event. §9 now has you tag one
build twice so the rollback target is bit-identical, and says plainly what that can and
cannot fix. Every `$DOMAIN` command gained the `set -a; . ./.env` it needs, the
down-migration psql commands are wrapped in `sh -c` so the container expands the
credentials rather than sending `-U ""`, and the advice to lower `max_video_size_mb`
is withdrawn: the client guard it was premised on does exist, but is pinned to a
compile-time constant, so lowering the DB value only moves failures later.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
67
.env.example
67
.env.example
@@ -1,7 +1,19 @@
|
|||||||
# ── Domain ────────────────────────────────────────────────────────────────────
|
# ── Domain ────────────────────────────────────────────────────────────────────
|
||||||
# Public domain Caddy will serve and obtain a TLS certificate for.
|
# Public domain Caddy will serve and obtain a TLS certificate for.
|
||||||
|
#
|
||||||
|
# The DNS A record must already point at this server BEFORE the first `up -d`: Caddy
|
||||||
|
# requests a certificate on boot, and Let's Encrypt allows only 5 failed validations per
|
||||||
|
# hostname per hour. Never delete the caddy_data volume — it holds the certificate and
|
||||||
|
# the ACME account key.
|
||||||
DOMAIN=my-event.example.com
|
DOMAIN=my-event.example.com
|
||||||
|
|
||||||
|
# ── Image version ─────────────────────────────────────────────────────────────
|
||||||
|
# Tag pulled for the `app` and `frontend` services (docker-compose.yml). Production runs
|
||||||
|
# prebuilt images from the registry and never compiles — see DEPLOYMENT_RUNBOOK.md.
|
||||||
|
# Always an immutable tag, never `latest`: rollback is `EVENTSNAP_VERSION=<previous>`
|
||||||
|
# + `docker compose up -d`, which works offline if that image is still resident locally.
|
||||||
|
EVENTSNAP_VERSION=v0.13.0
|
||||||
|
|
||||||
# ── App server ────────────────────────────────────────────────────────────────
|
# ── App server ────────────────────────────────────────────────────────────────
|
||||||
APP_PORT=3000
|
APP_PORT=3000
|
||||||
# Set to `production` in real deployments. This activates the secret guard that
|
# Set to `production` in real deployments. This activates the secret guard that
|
||||||
@@ -90,10 +102,53 @@ EXPORT_PATH=/exports
|
|||||||
# provisioned export headroom separately.
|
# provisioned export headroom separately.
|
||||||
|
|
||||||
# ── Workers ───────────────────────────────────────────────────────────────────
|
# ── Workers ───────────────────────────────────────────────────────────────────
|
||||||
# Number of parallel image/video compression workers. Default 2. This is the main
|
# Number of parallel media compression workers. Default 2. Boot-time only.
|
||||||
# throughput bottleneck: with 2 workers a burst of uploads can take ~5s to appear.
|
#
|
||||||
# For a large event (100+ guests) 4 is a good target — but each worker can run an
|
# CORRECTION TO EARLIER GUIDANCE: this used to say "each worker can run an ffmpeg
|
||||||
# ffmpeg transcode, so if you raise this ALSO raise the app container's memory limit
|
# transcode, so raise the app memory limit to ~2G if you set 4". There is NO video
|
||||||
# in docker-compose.yml (`app.deploy.resources.limits.memory`) from 1G to ~2G, or a
|
# transcode anywhere in this codebase — services/video.rs runs
|
||||||
# burst of large videos can OOM the box and take Postgres down with it.
|
# `ffmpeg -ss <t> -i <src> -vframes 1 -vf scale=...`, a single poster frame, and video
|
||||||
|
# originals are stored and served byte-for-byte. Poster extraction costs ~150-250 MB
|
||||||
|
# for a moment; it is not the constraint.
|
||||||
|
#
|
||||||
|
# The real memory consumer is the IMAGE path. `image` 0.25's resize builds an Rgba32F
|
||||||
|
# intermediate at 16 BYTES PER PIXEL, sized (source_width x target_height) — which the
|
||||||
|
# 256 MiB decode guard in imaging.rs does NOT cover. Peak per photo, decode + the 2048px
|
||||||
|
# display resize: ~145 MB at 12 MP, ~223 MB at 24 MP, ~354 MB at 48 MP.
|
||||||
|
#
|
||||||
|
# So on a 2 vCPU / 4 GB box (e.g. Hetzner CX22) KEEP THIS AT 2:
|
||||||
|
# * concurrency 2, two 48 MP photos ≈ 800 MB against the 1G app limit — ~25% margin.
|
||||||
|
# * concurrency 4, the same pair ≈ 1.5 GB — OOM.
|
||||||
|
# * and app=2G + db=1G + frontend/caddy 256M each + ~370 MB of OS/Docker exceeds the
|
||||||
|
# ~3910 MiB a "4 GB" VM actually reports. Raising the limit oversubscribes the host.
|
||||||
|
# 4 is only reasonable on the 4 vCPU / 8 GB box README.md documents.
|
||||||
|
#
|
||||||
|
# Throughput at 2 is not the bottleneck anyone thinks it is: ~2.5s per 12 MP photo, so
|
||||||
|
# 100 photos is ~250 CPU-seconds spread over an entire evening.
|
||||||
COMPRESSION_WORKER_CONCURRENCY=2
|
COMPRESSION_WORKER_CONCURRENCY=2
|
||||||
|
|
||||||
|
# ── Comments ──────────────────────────────────────────────────────────────────
|
||||||
|
# Master switch for the comment feature. Boot-time only (NOT in the admin UI), so it
|
||||||
|
# needs a `docker compose up -d` to apply. Anything other than false/0/no/off is on.
|
||||||
|
#
|
||||||
|
# When false the backend rejects NEW comments with 403 and the frontend hides the whole
|
||||||
|
# comment UI, including in the offline keepsake viewer. Likes and captions are entirely
|
||||||
|
# separate features and are unaffected. Existing comments stay in the database (hidden),
|
||||||
|
# so flipping it back restores them.
|
||||||
|
#
|
||||||
|
# Note it gates POSTING only: GET /upload/{id}/comments still serves already-existing
|
||||||
|
# comments, and the keepsake's data.json still embeds their text. Irrelevant if the flag
|
||||||
|
# is off from the first boot, since no comment can ever have been written.
|
||||||
|
# NOTE for the current deployment: `docker-compose.yml` PINS this to "false" on the app
|
||||||
|
# service, and `environment` overrides `env_file` — so changing it here has no effect in
|
||||||
|
# production. Remove that line from the compose file first if you want comments back.
|
||||||
|
COMMENTS_ENABLED=true
|
||||||
|
|
||||||
|
# ── Logging ───────────────────────────────────────────────────────────────────
|
||||||
|
# SET THIS IN PRODUCTION. Without it the app falls back to
|
||||||
|
# `eventsnap_backend=debug,tower_http=debug` (see main.rs), and with TraceLayer that is a
|
||||||
|
# debug line per HTTP request — including every preview and thumbnail fetch. Combined with
|
||||||
|
# Docker's json-file driver it writes to the same filesystem as the database and the media.
|
||||||
|
# docker-compose.yml caps each service's logs at 30 MB; this keeps the volume sane in the
|
||||||
|
# first place. The e2e stack has always used exactly this value.
|
||||||
|
RUST_LOG=eventsnap_backend=info,tower_http=warn
|
||||||
|
|||||||
693
DEPLOYMENT_RUNBOOK.md
Normal file
693
DEPLOYMENT_RUNBOOK.md
Normal file
@@ -0,0 +1,693 @@
|
|||||||
|
# EventSnap — Production Deployment Runbook
|
||||||
|
|
||||||
|
Target: **Hetzner CX22** (2 vCPU, 4 GB RAM, 40 GB disk), single host, docker compose.
|
||||||
|
Event profile: ~100 guests, ~100 photos + a few videos, one evening, **operator attending and unavailable to troubleshoot**.
|
||||||
|
|
||||||
|
Everything here is written for that last constraint. Where a choice trades throughput for
|
||||||
|
"cannot need attention on the night", it takes the stable option.
|
||||||
|
|
||||||
|
> **Note on this repo's own docs.** `README.md:62` and `PROJECT.md:359` specify a **CX33
|
||||||
|
> (4 vCPU / 8 GB / 80 GB)**. You are deploying to half of that on every axis. Two pieces of
|
||||||
|
> tuning advice in `.env.example` are calibrated for the CX33 and are actively wrong for a
|
||||||
|
> CX22 — they are called out in §3. Trust this file over `.env.example` for sizing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 0. Timeline — the single most important control
|
||||||
|
|
||||||
|
### Step zero: commit everything, before you build anything
|
||||||
|
|
||||||
|
**The production deployment does not exist in git yet.** At the time of writing, `docker-compose.yml`
|
||||||
|
and `.env.example` are modified but uncommitted, and `DEPLOYMENT_RUNBOOK.md`,
|
||||||
|
`docker-compose.build.yml`, both `.dockerignore` files and migrations `021`/`022` are untracked.
|
||||||
|
None of them is gitignored — they are simply not committed.
|
||||||
|
|
||||||
|
That is not a tidiness problem, it is the deployment failing in two ways at once:
|
||||||
|
|
||||||
|
- §7 tells you to `git clone` onto the server. `git show HEAD:docker-compose.yml` still has
|
||||||
|
`build:` keys and **no `image:` keys**, so on that clone `docker compose pull` skips both services
|
||||||
|
and `docker compose up -d` starts **a fat-LTO release build of 427 crates on the CX22** — the
|
||||||
|
exact scenario §1 rules out as an expected OOM. The committed file also has no log rotation.
|
||||||
|
- `sqlx::migrate!()` embeds `./migrations` **at compile time**. A build from the working tree bakes
|
||||||
|
in 021 and 022 and applies them on first boot; any later rebuild from a clean clone produces an
|
||||||
|
image that lacks them and crash-loops with `VersionMissing` against its own database.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add docker-compose.yml docker-compose.dev.yml docker-compose.build.yml .env.example \
|
||||||
|
backend/.dockerignore frontend/.dockerignore frontend/Dockerfile \
|
||||||
|
backend/migrations/021_*.sql backend/migrations/022_*.sql \
|
||||||
|
DEPLOYMENT_RUNBOOK.md
|
||||||
|
git commit -m "chore: production compose, ignore files and runbook"
|
||||||
|
git push
|
||||||
|
|
||||||
|
# Prove it landed — this must print two `image:` lines and nothing about `build:`
|
||||||
|
git show HEAD:docker-compose.yml | grep -E 'image:|build:'
|
||||||
|
git show HEAD --stat | grep -c migrations/02 # must be 4
|
||||||
|
```
|
||||||
|
|
||||||
|
| When | What |
|
||||||
|
|---|---|
|
||||||
|
| **T‑7 days** | Commit and push everything above. Registry + DNS pre-flight (§5). Build and push images (§6). |
|
||||||
|
| **T‑5 days** | First deploy to the server (§7). Verify admin login. Leave it running. |
|
||||||
|
| **T‑3 days** | **Freeze migrations.** No further code deploys unless something is broken. |
|
||||||
|
| **T‑2 days** | Pre-pull current *and* previous image tags (§9). Run the backup rehearsal (§10). |
|
||||||
|
| **Event day** | Change nothing. Configuration tweaks via the admin dashboard only (§4). |
|
||||||
|
|
||||||
|
**Why the freeze matters more than anything else here.** Migrations run automatically at boot
|
||||||
|
(`backend/src/db.rs`, `create_pool`) and `sqlx::migrate!()` is used *without* `set_ignore_missing`. An older
|
||||||
|
image booted against a newer schema does not degrade — it **crash-loops** with `VersionMissing`.
|
||||||
|
The repo documents this itself in `backend/migrations/014_export_epoch.up.sql:3-8`. Once the
|
||||||
|
migration set is frozen, rollback is a one-line `.env` edit; before it is frozen, rollback means
|
||||||
|
hand-running down-SQL under pressure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Deployment strategy — build on the Mac, push, pull
|
||||||
|
|
||||||
|
**Decision: build `linux/amd64` images on the M3 Pro, push to `registry.mc02.dev`, pull on the server.**
|
||||||
|
|
||||||
|
Rejected alternatives, with the reason each loses:
|
||||||
|
|
||||||
|
| Option | Why not |
|
||||||
|
|---|---|
|
||||||
|
| **Build on the CX22** | `backend/Cargo.toml` sets `lto = true` + `codegen-units = 1` over **427 crates**. Fat LTO links the whole program in one single-threaded process; peak RSS is estimated at 2.5–4 GB against ~1.5–2.5 GB free with the stack running. OOM is the expected outcome, not a tail risk. And *rollback would also be a build* — 35–60 min under pressure. |
|
||||||
|
| **CI (Gitea on Pi 5)** | Ruled out by you, and correct: a Pi 5 is strictly worse than the CX22 for a fat-LTO link. |
|
||||||
|
| **True cross-compilation** (`--target x86_64-unknown-linux-musl`) | The dependency graph has **four C-compiling crates** — `ring` (per-arch assembly), `zstd-sys`, `libdeflate-sys`, `libsqlite3-sys` — so it needs a real musl cross-toolchain. Alpine has no such package for an arm64 host; the working answer is `cargo-zigbuild`, which means a new base image and a new toolchain days before an unrepeatable event. **Slow but certain beats fast but novel.** |
|
||||||
|
|
||||||
|
Emulated build is the right call because the cost lands where it is free: on your laptop, days
|
||||||
|
early, with nothing depending on it. Estimated wall-clock for the backend is **25–60 min with
|
||||||
|
Rosetta enabled** (hours without it) — start it and walk away.
|
||||||
|
|
||||||
|
> **Escape hatch if emulation is unbearable:** spin up a temporary Hetzner CPX41 (8 vCPU/16 GB) in
|
||||||
|
> the same account, build natively, push, destroy it. Costs cents, same commands, no repo change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Repo changes — ALREADY APPLIED
|
||||||
|
|
||||||
|
> **Status: these are done, in the working tree.** They are documented here so you know what
|
||||||
|
> changed and why, not as work to repeat. Run `git diff` to review before committing.
|
||||||
|
|
||||||
|
### 2.1 `.dockerignore` files (were absent)
|
||||||
|
|
||||||
|
`backend/.dockerignore`:
|
||||||
|
```
|
||||||
|
target/
|
||||||
|
```
|
||||||
|
|
||||||
|
`frontend/.dockerignore`:
|
||||||
|
```
|
||||||
|
node_modules/
|
||||||
|
.svelte-kit/
|
||||||
|
build/
|
||||||
|
```
|
||||||
|
|
||||||
|
Without these, the first time you run `cargo build` or `npm install` locally, every image build
|
||||||
|
ships a multi-GB context to an *emulated* builder. Worse: `frontend/Dockerfile:9` does `COPY . .`
|
||||||
|
**after** `npm ci`, so a macOS `node_modules/` would be merged over the container's Linux one.
|
||||||
|
|
||||||
|
### 2.2 Switch compose from `build:` to `image:`
|
||||||
|
|
||||||
|
In `docker-compose.yml`, replace the `build:` block on `app` and `frontend`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
app:
|
||||||
|
image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
|
||||||
|
```
|
||||||
|
```yaml
|
||||||
|
frontend:
|
||||||
|
image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
|
||||||
|
```
|
||||||
|
|
||||||
|
Two deliberate properties:
|
||||||
|
- The `:?` form **fails loudly** on an unset variable instead of resolving to an empty tag.
|
||||||
|
- **Removing `build:` entirely is a safety feature.** With no `build:` key on the server, a wrong
|
||||||
|
tag is an instant `manifest unknown` — never a surprise 45-minute compile on the production box.
|
||||||
|
|
||||||
|
New `docker-compose.build.yml` (opt-in, Mac only — follows the convention stated in
|
||||||
|
the header of `docker-compose.dev.yml` that overlays are never auto-loaded):
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# Build overlay. NOT loaded automatically. Used only where images are BUILT — never on the
|
||||||
|
# production server, which pulls.
|
||||||
|
# docker compose -f docker-compose.yml -f docker-compose.build.yml build
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
build: { context: ./backend, dockerfile: Dockerfile }
|
||||||
|
frontend:
|
||||||
|
build: { context: ./frontend, dockerfile: Dockerfile }
|
||||||
|
```
|
||||||
|
|
||||||
|
`e2e/docker-compose.test.yml` keeps its own `build:` blocks, so the e2e gate is unaffected.
|
||||||
|
|
||||||
|
### 2.3 Add log rotation to every service
|
||||||
|
|
||||||
|
`docker-compose.yml` currently sets **no logging config**, so the default `json-file` driver keeps
|
||||||
|
logs forever, on the same filesystem as Postgres and the media. Add to each of the four services:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
logging:
|
||||||
|
driver: json-file
|
||||||
|
options: { max-size: "10m", max-file: "3" }
|
||||||
|
```
|
||||||
|
|
||||||
|
(Equivalently, set it once in `/etc/docker/daemon.json` — but that restarts the Docker daemon,
|
||||||
|
so do it *before* the stack is live, not after.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. `.env` — production values
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# ── Identity ──────────────────────────────────────────────────────────────
|
||||||
|
DOMAIN=<your domain>
|
||||||
|
EVENT_NAME=<...>
|
||||||
|
EVENT_SLUG=<...>
|
||||||
|
|
||||||
|
# ── Image version (NEW — drives the image: tags in docker-compose.yml) ────
|
||||||
|
EVENTSNAP_VERSION=v0.13.0
|
||||||
|
|
||||||
|
# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
|
||||||
|
JWT_SECRET=<openssl rand -hex 64>
|
||||||
|
POSTGRES_PASSWORD=<openssl rand -hex 24>
|
||||||
|
DATABASE_URL=postgres://eventsnap:<SAME PASSWORD>@db:5432/eventsnap
|
||||||
|
ADMIN_PASSWORD_HASH='<docker run --rm caddy:2-alpine caddy hash-password --plaintext "pw">'
|
||||||
|
|
||||||
|
# ── Paths — must match the volume mounts ──────────────────────────────────
|
||||||
|
MEDIA_PATH=/media # pinned by compose anyway, but keep consistent
|
||||||
|
EXPORT_PATH=/exports # NOT pinned by compose — see the trap in §7.3
|
||||||
|
|
||||||
|
# ── Sizing (see the two corrections below) ────────────────────────────────
|
||||||
|
DATABASE_MAX_CONNECTIONS=30
|
||||||
|
COMPRESSION_WORKER_CONCURRENCY=2
|
||||||
|
|
||||||
|
# ── Comments off, likes + captions on ─────────────────────────────────────
|
||||||
|
COMMENTS_ENABLED=false
|
||||||
|
|
||||||
|
# ── Logging (NEW — production currently defaults to DEBUG) ────────────────
|
||||||
|
RUST_LOG=eventsnap_backend=info,tower_http=warn
|
||||||
|
```
|
||||||
|
|
||||||
|
### Two corrections to the repo's own advice — do not follow `.env.example` here
|
||||||
|
|
||||||
|
**`COMPRESSION_WORKER_CONCURRENCY`: keep `2`. Do NOT raise to 4, and do NOT raise the app memory
|
||||||
|
limit to 2G.** `.env.example:92-98` justifies both on the premise that "each worker can run an
|
||||||
|
ffmpeg transcode". **There is no transcode anywhere in this codebase.** `services/video.rs::run_ffmpeg`
|
||||||
|
runs `ffmpeg -ss <t> -i <src> -vframes 1 -vf scale=…` — a single poster frame. Video originals are
|
||||||
|
stored and served byte-for-byte.
|
||||||
|
|
||||||
|
The real memory consumer is the **image** path: `image` 0.25's `resize` builds an `Rgba32F`
|
||||||
|
intermediate at **16 bytes/px**, sized `source_width × target_height`, which the 256 MiB decode
|
||||||
|
guard in `imaging::decode_limits` does not cover. Estimated peak per photo:
|
||||||
|
|
||||||
|
| Source | Peak (decode + display resize) |
|
||||||
|
|---|---|
|
||||||
|
| 12 MP (typical phone) | ~145 MB |
|
||||||
|
| 24 MP (iPhone Pro default) | ~223 MB |
|
||||||
|
| 48 MP ("Max" mode) | ~354 MB |
|
||||||
|
|
||||||
|
At concurrency 2, two 48 MP photos ≈ 800 MB against the 1 GiB cap — ~25% margin. At concurrency 4
|
||||||
|
the same pair is ~1.5 GB → **OOM**. And app=2G + db=1G + 256M + 256M + ~370 MB OS/Docker ≈ 3954 MiB
|
||||||
|
against ~3910 MiB MemTotal — the box is oversubscribed before a single photo arrives.
|
||||||
|
|
||||||
|
**`quota_tolerance`: keep `0.75`. Raising it does not make anything more generous for a real guest.**
|
||||||
|
See §4.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Admin dashboard configuration
|
||||||
|
|
||||||
|
These live in the DB `config` table, not `.env`. Changes take effect on the **next request** —
|
||||||
|
`patch_config` invalidates the cache synchronously (`admin::patch_config`). No restart needed. Set them
|
||||||
|
after the first deploy, before the event.
|
||||||
|
|
||||||
|
| Key | Default | **Set to** | Why |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `upload_rate_per_hour` | 100 | **1000** | A guest multi-selecting 100 photos hits exactly 100. Client backs off and resumes, but this makes it a non-issue. |
|
||||||
|
| `feed_rate_per_min` | 60 | **240** | Headroom for reconnect bursts. |
|
||||||
|
| `social_rate_per_min` | 120 | **600** | This is the **likes** limiter — the interaction you're keeping. |
|
||||||
|
| `export_rate_per_day` | 3 | **20** | **One shared bucket across both archives** — `enforce_export_rate` keys on `export:{user_id}` regardless of which archive is being fetched. Downloading Gallery.zip + Memories.zip costs 2 of 3; one retry locks a guest out for 24 h. The limit is now charged when the download *ticket* is minted, so being over it produces a visible German error instead of a tap that silently does nothing. |
|
||||||
|
| `max_video_size_mb` | 500 | **leave at 500** | See below. |
|
||||||
|
| `quota_tolerance` | 0.75 | **leave at 0.75** | See below. |
|
||||||
|
| `quota_enabled`, `storage_quota_enabled`, `rate_limits_enabled` | true | **leave on** | This is the only disk-full safety net. |
|
||||||
|
|
||||||
|
**Ignore `estimated_guest_count` and `upload_count_quota_enabled`.** Both are seeded, validated and
|
||||||
|
rendered in the admin UI — and **read by no code at all** (verified by grep across `backend/src`).
|
||||||
|
Changing them does nothing. `estimated_guest_count` in particular does *not* feed the quota formula.
|
||||||
|
|
||||||
|
### Why quotas are already as generous as you want
|
||||||
|
|
||||||
|
```
|
||||||
|
per_user_limit = floor(free_disk × quota_tolerance / max(active_uploaders, 1))
|
||||||
|
```
|
||||||
|
`active_uploaders` is `SELECT COUNT(DISTINCT user_id) FROM upload WHERE deleted_at IS NULL`
|
||||||
|
(`upload::compute_storage_quota`) — **people who actually uploaded**, not guests who joined. The 70 guests who never
|
||||||
|
upload are not in the denominator; their share flows to the photographers automatically. **The
|
||||||
|
redistribution you asked for is already the design.**
|
||||||
|
|
||||||
|
With ~28 GB free and a realistic 30 people actually uploading, each gets **~700 MB** — against an
|
||||||
|
expected ~1.25 GB for the *entire event*. Nobody will be blocked.
|
||||||
|
|
||||||
|
Raising `quota_tolerance` would only raise the **saturation ceiling** (media converges to
|
||||||
|
`t/(1+t)` of free space: 43% at 0.75, 50% at 1.0). It does nothing for a real guest at your volume,
|
||||||
|
and it eats the headroom the keepsake needs — the export preflight reserves `media × 1.10 × 2`
|
||||||
|
because `Gallery.zip` and `Memories.zip` each store media **uncompressed** (`export.rs` — both archives store media uncompressed).
|
||||||
|
|
||||||
|
**Why `max_video_size_mb` stays at 500.** An earlier draft of this runbook said to lower it to 250,
|
||||||
|
on the grounds that there is no client-side size check. That was wrong on both halves:
|
||||||
|
|
||||||
|
- A client guard **does** exist — `frontend/src/routes/upload/+page.svelte` rejects anything over
|
||||||
|
`HARD_MAX_UPLOAD_BYTES` (576 MiB) before a byte leaves the phone, alongside the HEIC reject.
|
||||||
|
- That guard is a **compile-time constant**, not the DB value. Lowering `max_video_size_mb` therefore
|
||||||
|
changes nothing on the client: the picker still accepts the clip, the phone still starts sending
|
||||||
|
it, and the *server* aborts it partway (`stream_field_to_file` does stop at the cap, so the AP is
|
||||||
|
not saturated for the full file — but the guest still gets a mid-upload failure where 500 MB would
|
||||||
|
simply have worked).
|
||||||
|
|
||||||
|
So lowering it is a pure capacity decision with no UX upside, and at your volume there is no capacity
|
||||||
|
problem to solve. Leave it. If you ever do want a smaller ceiling to bite on the phone, the constant
|
||||||
|
above has to move with it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Pre-flight — T‑7 days (do NOT leave this to event week)
|
||||||
|
|
||||||
|
### Registry
|
||||||
|
|
||||||
|
From **both** the Mac and the server, as **the user who will deploy** (root's `docker login` does
|
||||||
|
not help a non-root deploy — credentials go to `~/.docker/config.json`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
getent hosts registry.mc02.dev # DNS resolves from the server
|
||||||
|
curl -fsSI https://registry.mc02.dev/v2/ # TLS must be PUBLICLY trusted; Docker rejects
|
||||||
|
# self-signed without extra daemon config
|
||||||
|
docker login registry.mc02.dev
|
||||||
|
```
|
||||||
|
|
||||||
|
Also confirm: the `eventsnap` repository/namespace exists if your registry requires pre-creation
|
||||||
|
(Harbor does; plain `registry:2` does not), and the registry host has ~500 MB free per release.
|
||||||
|
|
||||||
|
### DNS and TLS — get this wrong and you are locked out for an hour
|
||||||
|
|
||||||
|
The DNS **A record must point at the CX22 before the first `docker compose up -d`**, because Caddy
|
||||||
|
attempts certificate issuance on boot. Let's Encrypt allows only **5 failed validations per hostname
|
||||||
|
per hour** (refilling 1 per 12 min). A misconfigured DNS record plus a few impatient restarts will
|
||||||
|
rate-limit you out of getting a certificate at all.
|
||||||
|
|
||||||
|
- Deploy days early so issuance happens with no time pressure.
|
||||||
|
- **Never delete the `caddy_data` volume** — it holds the certificate and the ACME account key.
|
||||||
|
- Never run `docker compose down -v`. It destroys `postgres_data`, `media_data`, `exports_data`
|
||||||
|
*and* `caddy_data`.
|
||||||
|
|
||||||
|
### Host preparation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose version # must be v2.x — the deploy.resources limits need it
|
||||||
|
free -h && swapon --show # Hetzner images ship no swap
|
||||||
|
df -h /var/lib/docker # want ≥ 25 GB free
|
||||||
|
```
|
||||||
|
|
||||||
|
**Add 2 GB of swap** as an OOM cushion — a compression spike that would otherwise kill the container
|
||||||
|
instead swaps out cold pages and merely runs slowly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
|
||||||
|
echo '/swapfile none swap sw 0 0' >> /etc/fstab
|
||||||
|
sysctl -w vm.swappiness=10 && echo 'vm.swappiness=10' > /etc/sysctl.d/99-swap.conf
|
||||||
|
```
|
||||||
|
|
||||||
|
> **Gotcha:** Compose sets each container's `Memory` limit but leaves `MemorySwap` unset, and Docker
|
||||||
|
> then allows swap equal to the memory limit — so adding host swap silently **doubles** every
|
||||||
|
> container ceiling. If you add swap, also add `memswap_limit: 1152m` to `app` and `db`, and
|
||||||
|
> `memswap_limit: 320m` to `frontend` and `caddy` (service-level, not under `deploy:`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Build and push — on the Mac
|
||||||
|
|
||||||
|
Enable **Docker Desktop → Settings → General → "Use Rosetta for x86_64/amd64 emulation"**, and set
|
||||||
|
**Resources → Memory ≥ 8 GB** (the fat-LTO step will OOM inside the VM otherwise, even with 18 GB on
|
||||||
|
the host).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm --platform linux/amd64 alpine uname -m # must print x86_64
|
||||||
|
|
||||||
|
cd /Users/fabianhammprivat/Projects/EventSnap
|
||||||
|
VERSION=v0.13.0 # latest existing tag is v0.12.0 — see §9 before reusing it
|
||||||
|
ROLLBACK=v0.13.0-a # the SAME source, tagged twice; §9 explains why
|
||||||
|
SHA=$(git rev-parse --short HEAD)
|
||||||
|
|
||||||
|
docker buildx create --name eventsnap --use 2>/dev/null || docker buildx use eventsnap
|
||||||
|
|
||||||
|
docker buildx build --platform linux/amd64 \
|
||||||
|
-t registry.mc02.dev/eventsnap/app:$VERSION \
|
||||||
|
-t registry.mc02.dev/eventsnap/app:$ROLLBACK \
|
||||||
|
-t registry.mc02.dev/eventsnap/app:$SHA \
|
||||||
|
--push ./backend
|
||||||
|
|
||||||
|
docker buildx build --platform linux/amd64 \
|
||||||
|
-t registry.mc02.dev/eventsnap/frontend:$VERSION \
|
||||||
|
-t registry.mc02.dev/eventsnap/frontend:$ROLLBACK \
|
||||||
|
-t registry.mc02.dev/eventsnap/frontend:$SHA \
|
||||||
|
--push ./frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify the architecture — never skip this
|
||||||
|
|
||||||
|
An arm64 image pulls fine and then dies with `exec format error`. Catch it here, not on the server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker buildx imagetools inspect registry.mc02.dev/eventsnap/app:$VERSION
|
||||||
|
docker buildx imagetools inspect registry.mc02.dev/eventsnap/frontend:$VERSION
|
||||||
|
# Both MUST report Platform: linux/amd64
|
||||||
|
```
|
||||||
|
|
||||||
|
Then prove the binary actually executes. `AppConfig::from_env()` runs before any DB connection
|
||||||
|
(`main.rs` builds `AppConfig` before touching the pool), so this needs no database:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run --rm --platform linux/amd64 \
|
||||||
|
-e APP_ENV=production -e JWT_SECRET=x -e DATABASE_URL=x -e EVENT_SLUG=x \
|
||||||
|
registry.mc02.dev/eventsnap/app:$VERSION
|
||||||
|
```
|
||||||
|
Expect the "Refusing to start in production … placeholder" message. **That message means the amd64
|
||||||
|
binary ran.** `exec format error` means the architecture is wrong.
|
||||||
|
|
||||||
|
**Tag policy: never `latest` in production.** With `latest` you cannot tell what is running,
|
||||||
|
rollback becomes a registry re-push (impossible if the registry is down — exactly when you need it),
|
||||||
|
and `restart: unless-stopped` after a reboot is ambiguous. Two immutable tags per build: semver and
|
||||||
|
short SHA. Deploy by semver.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. First deploy — on the server
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Directory name matters: volumes are prefixed with it, and README's backup commands
|
||||||
|
# hardcode the `eventsnap_` prefix.
|
||||||
|
git clone <repo> eventsnap && cd eventsnap
|
||||||
|
cp .env.example .env && nano .env # every value from §3
|
||||||
|
docker login registry.mc02.dev
|
||||||
|
docker compose pull # must fully succeed before anything starts
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.1 The secret pre-flight — run this before the first `up -d`, always
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose run --rm --no-deps app
|
||||||
|
```
|
||||||
|
|
||||||
|
`--no-deps` means `db` never starts, so **no Postgres data directory is initialised**.
|
||||||
|
`AppConfig::from_env()` runs first and reports *every* placeholder at once (`config::validate_secrets`).
|
||||||
|
When the only remaining complaint is a database *connection* failure, the secrets are good.
|
||||||
|
|
||||||
|
**Why this step exists.** `POSTGRES_PASSWORD` is applied **only at initdb**. `docker-compose.yml`
|
||||||
|
starts `db` in the same command as `app`, so a single `up -d` with a placeholder bakes the wrong
|
||||||
|
password in permanently — the app then loops on `password authentication failed`, and the only exits
|
||||||
|
are `ALTER ROLE` or `down -v`, which deletes the database, the media and the exports. The repo
|
||||||
|
describes this trap at `backend/src/db.rs` (`explain_auth_failure`); this command is what avoids it.
|
||||||
|
|
||||||
|
### 7.2 Bring it up
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN.
|
||||||
|
set -a; . ./.env; set +a
|
||||||
|
|
||||||
|
docker compose up -d
|
||||||
|
docker compose logs -f app # wait for "database connected and migrations applied"
|
||||||
|
curl -fsS https://$DOMAIN/health # → ok (503 means the app is up but the DB is not)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.3 Verify the three things compose does *not* pin
|
||||||
|
|
||||||
|
`MEDIA_PATH` is pinned to `/media` on the `app` service in `docker-compose.yml`. Its siblings
|
||||||
|
are not:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH
|
||||||
|
```
|
||||||
|
|
||||||
|
1. **`DATABASE_URL`** must contain `@db:5432`. A dev `.env` points it at `@localhost`, which inside
|
||||||
|
the container is the app itself.
|
||||||
|
2. **`EXPORT_PATH`** must be `/exports`. Anywhere else and the keepsake archives are written to the
|
||||||
|
container's writable layer and **vanish on the next `up -d`** — including on a rollback.
|
||||||
|
3. **`ADMIN_PASSWORD_HASH`** must match `.env` **byte for byte.**
|
||||||
|
|
||||||
|
**Then actually log in to `/admin` with the real password.** This is not optional politeness:
|
||||||
|
|
||||||
|
- The production secret guard only rejects *placeholders* (`config::looks_placeholder`). A hash **corrupted**
|
||||||
|
by shell or Compose escaping is not a placeholder — the app boots green, `/health` says `ok`, and
|
||||||
|
every admin login 401s.
|
||||||
|
- The Admin user row is created **by a successful admin login** (`auth::handlers::admin_login`), and
|
||||||
|
promoting anyone to Host requires an Admin or Host (`auth::middleware`'s role guard). **No admin login
|
||||||
|
⇒ no host, ever** ⇒ you cannot close the event, release the gallery, ban anyone, reset a PIN, or
|
||||||
|
change any limit — for the whole event.
|
||||||
|
|
||||||
|
Per the Compose spec, single-quoted `env_file` values *are* used literally and the quotes are
|
||||||
|
stripped, so the single-quote form in `.env.example` is correct. The comment in
|
||||||
|
`docker-compose.dev.yml` claiming production has the same bug is **stale**. Verify anyway —
|
||||||
|
the cost of checking is 10 seconds; the cost of being wrong is the whole event.
|
||||||
|
|
||||||
|
**Promote a second person to Host** once you are in, so a single lost session is not fatal.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Post-deploy verification
|
||||||
|
|
||||||
|
```bash
|
||||||
|
set -a; . ./.env; set +a # $DOMAIN comes from .env, not your shell
|
||||||
|
|
||||||
|
docker compose ps # all four healthy
|
||||||
|
docker inspect -f '{{.HostConfig.Memory}}' eventsnap-app-1 # must be 1073741824, not 0
|
||||||
|
curl -fsS https://$DOMAIN/health # ok — now a real DB check, not a constant
|
||||||
|
docker compose exec app printenv COMMENTS_ENABLED RUST_LOG
|
||||||
|
docker builder prune -af && docker image prune -f # reclaim build cache
|
||||||
|
df -h /var/lib/docker
|
||||||
|
```
|
||||||
|
|
||||||
|
Then, from a phone on cellular (not the office wifi):
|
||||||
|
|
||||||
|
- [ ] Join as a guest with a PIN
|
||||||
|
- [ ] Upload a photo → appears in the feed within a few seconds
|
||||||
|
- [ ] Upload a video → plays back (iOS Safari range requests)
|
||||||
|
- [ ] Add a caption, add a like — **no comment UI anywhere**
|
||||||
|
- [ ] Admin login works; host dashboard reachable
|
||||||
|
- [ ] Release the gallery on a test event and download both archives
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Rollback
|
||||||
|
|
||||||
|
### There is no older image you can roll back to. Build the rollback target yourself.
|
||||||
|
|
||||||
|
Read this before the event, not during it. The obvious move — drop `EVENTSNAP_VERSION` back to the
|
||||||
|
previous released tag — **takes the app down permanently** and looks like a crash loop with no
|
||||||
|
explanation:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
|
||||||
|
12 # 6 migrations. HEAD has 22.
|
||||||
|
$ git rev-list --count v0.12.0..HEAD
|
||||||
|
154
|
||||||
|
```
|
||||||
|
|
||||||
|
`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration
|
||||||
|
tree, booting against a database that already carries versions 007–022, returns `VersionMissing`.
|
||||||
|
`create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy
|
||||||
|
still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.)
|
||||||
|
No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with
|
||||||
|
`manifest unknown` before you ever got that far.
|
||||||
|
|
||||||
|
**So: at build time, tag the SAME frozen commit twice.** Two identical images, two names. The
|
||||||
|
rollback then swaps to a binary that is bit-for-bit what you tested and carries the identical
|
||||||
|
migration set, which makes it a genuine no-op rather than a gamble:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# In §6, push both tags from the one build:
|
||||||
|
VERSION=v0.13.0
|
||||||
|
ROLLBACK=v0.13.0-a # same source, different name — the rollback target
|
||||||
|
|
||||||
|
docker buildx build --platform linux/amd64 \
|
||||||
|
-t registry.mc02.dev/eventsnap/app:$VERSION \
|
||||||
|
-t registry.mc02.dev/eventsnap/app:$ROLLBACK \
|
||||||
|
--push ./backend
|
||||||
|
# ...and the same two tags for ./frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rolling back — ~30 seconds, no network:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env
|
||||||
|
docker compose up -d app frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
This only works offline if both are already resident. **Pre-pull all four at T‑2:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker pull registry.mc02.dev/eventsnap/app:v0.13.0
|
||||||
|
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0
|
||||||
|
docker pull registry.mc02.dev/eventsnap/app:v0.13.0-a
|
||||||
|
docker pull registry.mc02.dev/eventsnap/frontend:v0.13.0-a
|
||||||
|
docker image ls | grep eventsnap # confirm all four
|
||||||
|
```
|
||||||
|
|
||||||
|
Once resident, `up -d`, reboots, restarts and rollbacks need **zero** registry contact. That single
|
||||||
|
step makes a registry outage on event day irrelevant.
|
||||||
|
|
||||||
|
Be clear-eyed about what this buys you: an identical image cannot undo a bad *release*, only an
|
||||||
|
image that got corrupted or a container that wedged — and `docker compose restart app` already
|
||||||
|
covers both. It exists so that the rollback line in the emergency card is safe to run rather than
|
||||||
|
catastrophic. **If you genuinely need to undo a code change during the event, you cannot; freeze
|
||||||
|
early enough that you never have to.**
|
||||||
|
|
||||||
|
**Across a migration boundary — avoid by freezing.** If you must: all 22 migrations have paired
|
||||||
|
`.down.sql` files, but **none of them removes its own `_sqlx_migrations` row**, so that second step
|
||||||
|
is mandatory and undocumented:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose stop app
|
||||||
|
# `sh -c` so the CONTAINER expands the credentials. They live in the container's environment
|
||||||
|
# and in .env — not in your shell — so an unwrapped `-U "$POSTGRES_USER"` sends `-U ""` and
|
||||||
|
# psql answers `FATAL: role "" does not exist`.
|
||||||
|
docker compose exec -T db sh -c \
|
||||||
|
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' \
|
||||||
|
< backend/migrations/0NN_x.down.sql
|
||||||
|
docker compose exec -T db sh -c \
|
||||||
|
'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "DELETE FROM _sqlx_migrations WHERE version = NN;"'
|
||||||
|
```
|
||||||
|
|
||||||
|
Migration **014** is the only destructive one on the way up, and the only one with a rehearsal
|
||||||
|
harness — `backend/scripts/rehearse-014.sh`. Run it once against a real dump before the event.
|
||||||
|
|
||||||
|
**Registry-down transport fallback** (layers are already compressed — do not add gzip):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
|
||||||
|
registry.mc02.dev/eventsnap/frontend:v0.13.0 | ssh root@SERVER 'docker load'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Backup
|
||||||
|
|
||||||
|
Full commands are in `README.md:315-435` and are correct — `pg_dump --clean --if-exists`, plus
|
||||||
|
`alpine tar` out of `eventsnap_media_data` and `eventsnap_exports_data`, mounted at `/src` (not
|
||||||
|
`/media`), with `chown -R 100:101` on restore because the app runs non-root and BusyBox tar has no
|
||||||
|
`--same-owner`. There is deliberately no script.
|
||||||
|
|
||||||
|
Two gaps the README does not cover:
|
||||||
|
|
||||||
|
1. **Nothing backs up `.env`**, which holds the only copy of `POSTGRES_PASSWORD`. A dump you cannot
|
||||||
|
authenticate against is not a backup. Copy `.env` off the box, encrypted, once it is final.
|
||||||
|
2. **Timing.** Do not use nightly cron — every irreplaceable byte is created inside one evening.
|
||||||
|
Take the dump and the media tarball back-to-back **the night of the event, after locking uploads
|
||||||
|
from the host dashboard**, so the pair is consistent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Disk — the numbers for 40 GB
|
||||||
|
|
||||||
|
Expected event (100 photos @ ~4 MB, 5 videos @ ~150 MB). Originals are **always kept** and never
|
||||||
|
lossily recompressed; derivatives are a ≤800 px preview and a ≤2048 px display JPEG.
|
||||||
|
|
||||||
|
```
|
||||||
|
media (originals + derivatives) ~1.25 GB
|
||||||
|
peak during keepsake build (+ Gallery + Memories) ~3.5 GB
|
||||||
|
OS + Docker images + Postgres + logs ~4.5 GB
|
||||||
|
────────────────────────────────────────────────────────────
|
||||||
|
total ~11–13 GB of ~36 GB usable
|
||||||
|
```
|
||||||
|
|
||||||
|
**40 GB fits with roughly 3× headroom**, provided you build elsewhere (a server-side build adds
|
||||||
|
3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against
|
||||||
|
*live* free space on every upload).
|
||||||
|
|
||||||
|
The `README.md:295-299` "ENOSPC" projection models guests **saturating the quota** (~12 GB of
|
||||||
|
media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully:
|
||||||
|
the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard
|
||||||
|
warns when free space drops below 10 GB or below the keepsake requirement (`handlers::host`'s low-disk thresholds).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Known issues you are shipping with
|
||||||
|
|
||||||
|
None of these has a fix in this runbook; they are listed so nothing is a surprise. Severity is
|
||||||
|
scored purely by "would this interrupt you during the event".
|
||||||
|
|
||||||
|
### Fixed in this pass
|
||||||
|
|
||||||
|
| Was | Fix |
|
||||||
|
|---|---|
|
||||||
|
| **Queued uploads never rehydrated** — `loadQueue()` had one call site, so a guest whose PWA was evicted mid-upload and reopened onto `/feed` had pending photos in IndexedDB that nothing ever read. Silent photo loss. | `loadQueue()` now runs on every authenticated boot in `+layout.svelte`. |
|
||||||
|
| **A corrupted `ADMIN_PASSWORD_HASH` booted green** and only failed at admin login — unrecoverable mid-event. | `config.rs` now validates the bcrypt *shape*, not just placeholder-ness, and refuses to start. |
|
||||||
|
| **SSE reconnect thundering herd** — flat 500 ms jitter regardless of backoff. | Jitter now scales with the delay; the feed's in-place refresh is spread over 800–2800 ms. |
|
||||||
|
| **No client-side size/HEIC pre-check** — a doomed 600 MB upload saturated the venue AP before being rejected. | Certain-rejects are refused before any bytes leave the phone. |
|
||||||
|
| **Upload-queue UI unreachable** — the badged FAB opened the picker, not the queue, so a failed upload had no retry button. | The upload sheet now shows a queue entry whenever the badge is non-zero. |
|
||||||
|
| **A mid-event 401 left a dead screen** with no nav and no URL bar. | `api.ts` now returns the guest to `/join` (skipping the auth routes themselves). |
|
||||||
|
| **Rate limiter could panic while holding its mutex** (`timestamps[0]` with `max == 0`), poisoning it process-wide. | Uses `first()` with a fail-open guard. |
|
||||||
|
| **Hashtag deadlock** — `edit_upload` and `add_comment` upserted tags in client/text order. | Both now sort + dedup on the normalised key, matching the upload path. |
|
||||||
|
| **Unbounded Docker logs + debug-level app logging.** | `logging:` caps every service at 30 MB; `RUST_LOG` documented in `.env.example`. |
|
||||||
|
|
||||||
|
### Fixed in the readiness pass — behaviour you should know about
|
||||||
|
|
||||||
|
These change what you will observe on the night, so they are listed separately from the table above.
|
||||||
|
|
||||||
|
| Was | Now |
|
||||||
|
|---|---|
|
||||||
|
| **Any ffmpeg-level failure DESTROYED the video.** A poster-frame error — ffmpeg hanging on a truncated `.mov`, an ENOSPC on `thumbnails/`, a DB blip — propagated into the give-up path, which soft-deleted the upload. Reproduced live: on a host with no ffmpeg the clip was gone ~6 s after its `201 Created`. | No failure in the video branch can fail the upload. The clip stays in the feed and plays from its original; only the poster is missing. |
|
||||||
|
| **A lost response duplicated the photo.** Every retry minted a fresh upload id, so a phone that lost the reply and re-sent — manually, or automatically on reconnect — stored the same photo two or three times and paid quota for each. | The client sends a `client_upload_id`; migration 022 makes it unique. A retry returns `200` with the original row. Verified live: three sends → one row, quota charged once. |
|
||||||
|
| **`/health` returned a constant `"ok"`.** The container reported healthy while every request 500'd. | Runs `SELECT 1` with a 2 s timeout: `200 ok` / `503`. Verified live: 200 → stop Postgres → 503 → start Postgres → 200, **with no app restart** (the pool revalidates on acquire). Note that Compose does not restart on an unhealthy probe — this is a diagnostic, deliberately not wired to automatic recovery, because a restart would truncate in-flight uploads to "fix" an outage that clears on its own. |
|
||||||
|
| **Abandoned `.tmp` uploads were never reclaimed** (see the old issue #2 below — it is now fixed, not deferred). | Swept at boot and hourly, keyed on modification time so a live upload can never age into it. |
|
||||||
|
| **A full disk made itself worse.** ENOSPC was retried three times, then refunded the quota, soft-deleted the row and *kept* the bytes — freeing nothing and inviting an immediate re-upload into the same full disk. | ENOSPC is classified separately: no retry, no refund, no delete. The row stays live and the photo is served from its original until there is room to compress it. |
|
||||||
|
| **The keepsake download failed invisibly.** The rate limit was enforced inside the iframe navigation, so an over-limit guest tapped and *nothing happened*, forever. | Charged when the ticket is minted — a normal `fetch` — so it surfaces as a German message naming the daily window. Raise `export_rate_per_day` per §4 anyway. |
|
||||||
|
| **`/host` and `/admin` subscribed to SSE but never opened the connection**, so the keepsake progress bar froze after a release. | Both connect on mount and disconnect on destroy, as `/export` already did. |
|
||||||
|
| **`?limit=-5` on the feed returned a 500.** | Clamped at both ends. |
|
||||||
|
| **All recurring hygiene lived in one unsupervised task** — one panic silently stopped session pruning, media reclamation and the temp sweep for the rest of the event. | Supervised and re-spawned, with an error log. |
|
||||||
|
| **No pool acquire or statement timeout.** A DB blip parked every request for 30 s. | 5 s acquire, `statement_timeout=15s`, `lock_timeout=5s`. |
|
||||||
|
|
||||||
|
### Still open — accepted for this event
|
||||||
|
|
||||||
|
| # | Issue | Impact |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | **Guest delete/caption-edit after release invalidates both keepsake archives** (`upload.rs`), forcing a rebuild with a 20 s debounce. **No-op before release**, so it cannot bite during the event. | Post-event only. Left alone because blocking guest deletes has real privacy downsides — that is a product call, not a bug fix. |
|
||||||
|
| 2 | **Video poster frames do not regenerate after a restart** mid-compression (the backfill filters `mime_type LIKE 'image/%'`). | Cosmetic — the video still plays; only its poster is missing. |
|
||||||
|
| 3 | **SSE keep-alives are sent as SSE comments** (`:ping`), which the browser's EventSource parser discards without dispatching. A client therefore cannot implement a pure silence timer to detect a half-open socket. | Worked around client-side: the feed runs a jittered 60–120 s `/feed/delta` backstop and reconnects when a poll returns content the stream never delivered. A cleaner fix is to emit keep-alives as a *named* event; that is a coordinated backend+frontend change, not worth making during a freeze. |
|
||||||
|
| 4 | **The lightbox stops at the end of the loaded page** — stepping past the last loaded photo does not fetch the next one. | The guest scrolls the feed (which does page) and re-opens. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Event-day emergency card
|
||||||
|
|
||||||
|
Assume you have a phone and two minutes.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/eventsnap
|
||||||
|
|
||||||
|
# FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this,
|
||||||
|
# every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage
|
||||||
|
# when the site is fine.
|
||||||
|
set -a; . ./.env; set +a
|
||||||
|
|
||||||
|
# Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not)
|
||||||
|
curl -fsS https://$DOMAIN/health
|
||||||
|
|
||||||
|
# What is broken?
|
||||||
|
docker compose ps
|
||||||
|
docker compose logs --tail=100 app
|
||||||
|
|
||||||
|
# Nuclear option that is SAFE (keeps all data):
|
||||||
|
docker compose restart app
|
||||||
|
|
||||||
|
# Roll back to the identically-built sibling image — see §9 for what this can and cannot fix.
|
||||||
|
# Do NOT substitute an older release tag here; it will crash-loop on the migration set.
|
||||||
|
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.0-a/' .env && docker compose up -d app frontend
|
||||||
|
|
||||||
|
# Disk check
|
||||||
|
df -h /var/lib/docker
|
||||||
|
```
|
||||||
|
|
||||||
|
**NEVER** run `docker compose down -v`. It deletes the database, all media, all exports and the TLS
|
||||||
|
certificate. There is no undo.
|
||||||
|
|
||||||
|
Most limits are changeable from the **admin dashboard without a restart** — reach for that before
|
||||||
|
touching the shell.
|
||||||
12
backend/.dockerignore
Normal file
12
backend/.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
# Build context exclusions.
|
||||||
|
#
|
||||||
|
# `target/` does not exist on a clean checkout, which is why builds have worked without this
|
||||||
|
# file — but the moment anyone runs `cargo build` locally it becomes multi-GB, and every
|
||||||
|
# `docker build` would ship all of it to the daemon for nothing (the Dockerfile only COPYs
|
||||||
|
# Cargo.toml, Cargo.lock, src, static and migrations). On an EMULATED amd64 builder that
|
||||||
|
# transfer is the slowest part of the build.
|
||||||
|
target/
|
||||||
|
|
||||||
|
# Never let a real .env reach an image layer.
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
24
docker-compose.build.yml
Normal file
24
docker-compose.build.yml
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
# Build overlay. NOT loaded automatically (same convention as docker-compose.dev.yml — this
|
||||||
|
# repo deliberately never uses the auto-loaded `docker-compose.override.yml` name).
|
||||||
|
#
|
||||||
|
# Used ONLY on a workstation that builds and pushes images. Never on the production server,
|
||||||
|
# which pulls: see the note on the `app` service in docker-compose.yml.
|
||||||
|
#
|
||||||
|
# Restores the `build:` keys that production omits, so `docker compose build` still works from
|
||||||
|
# a single source of truth for the build context paths:
|
||||||
|
#
|
||||||
|
# docker compose -f docker-compose.yml -f docker-compose.build.yml build
|
||||||
|
#
|
||||||
|
# For the actual release build use buildx directly instead — it is what produces linux/amd64
|
||||||
|
# images from an arm64 Mac and pushes them in one step (see DEPLOYMENT_RUNBOOK.md §6):
|
||||||
|
#
|
||||||
|
# docker buildx build --platform linux/amd64 -t registry.mc02.dev/eventsnap/app:vX.Y.Z --push ./backend
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
build:
|
||||||
|
context: ./backend
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
frontend:
|
||||||
|
build:
|
||||||
|
context: ./frontend
|
||||||
|
dockerfile: Dockerfile
|
||||||
@@ -21,11 +21,15 @@ services:
|
|||||||
# app user can't create it from `/`, so every upload 500s with EACCES. The media
|
# app user can't create it from `/`, so every upload 500s with EACCES. The media
|
||||||
# volume is mounted at /media (see docker-compose.yml) — point the app there.
|
# volume is mounted at /media (see docker-compose.yml) — point the app there.
|
||||||
MEDIA_PATH: /media
|
MEDIA_PATH: /media
|
||||||
# Recent Docker Compose interpolates env_file values, so the `$` segments of the
|
# Set here purely so local dev needs no `.env` at all. The `$` are doubled because
|
||||||
# bcrypt ADMIN_PASSWORD_HASH in .env get eaten (the salt reads as an unset var) —
|
# THIS is a `docker-compose.yml` `environment:` value, where Compose does interpolate.
|
||||||
# every admin login then 401s. Re-supply it here with `$` doubled to `$$` so Compose
|
#
|
||||||
# passes the literal hash. NOTE: production (docker-compose.yml + .env) has the SAME
|
# DO NOT COPY THE DOUBLING INTO `.env`. An earlier version of this comment claimed
|
||||||
# bug — escape the hash as `$$` in .env, or set it via `environment:` there too.
|
# production had "the same bug" and told you to escape the hash as `$$` there — that is
|
||||||
|
# wrong and it breaks a working deployment. Compose uses SINGLE-QUOTED `env_file` values
|
||||||
|
# literally, which is the form `.env.example` ships, so the `$` segments survive intact;
|
||||||
|
# doubling them produces a 74-character string that `looks_bcrypt` rejects and the app
|
||||||
|
# refuses to boot on. Verified with `docker compose exec app printenv`.
|
||||||
ADMIN_PASSWORD_HASH: "$$2b$$12$$PAteqCNpsbm6d0HTJcywfOaUovjAU.iNVlsL7EDYaRC/z4P/xv7ye"
|
ADMIN_PASSWORD_HASH: "$$2b$$12$$PAteqCNpsbm6d0HTJcywfOaUovjAU.iNVlsL7EDYaRC/z4P/xv7ye"
|
||||||
# Smoke-testing the comment kill-switch: boot-time flag, so it needs a restart
|
# Smoke-testing the comment kill-switch: boot-time flag, so it needs a restart
|
||||||
# (not an admin-UI toggle). Backend rejects new comments (403) and the frontend
|
# (not an admin-UI toggle). Backend rejects new comments (403) and the frontend
|
||||||
@@ -33,9 +37,10 @@ services:
|
|||||||
COMMENTS_ENABLED: "false"
|
COMMENTS_ENABLED: "false"
|
||||||
|
|
||||||
caddy:
|
caddy:
|
||||||
# The prod caddy service has no env_file, so the Caddyfile's `{$DOMAIN}` expands
|
# The caddy service has no env_file, so the Caddyfile's `{$DOMAIN}` would expand to empty
|
||||||
# to empty and the site block collapses into a malformed global block. Supply it
|
# and the site block would collapse into a malformed global block. Supply it for local dev
|
||||||
# for local dev (from .env → localhost, which Caddy serves with a local self-signed
|
# (from .env → localhost, which Caddy serves with a local self-signed cert). Production
|
||||||
# cert). NOTE: the prod compose likely needs DOMAIN wired to caddy too.
|
# already wires DOMAIN into caddy's `environment:`, guarded with `:?` so an unset value
|
||||||
|
# fails the command instead of silently producing a site with no address.
|
||||||
environment:
|
environment:
|
||||||
DOMAIN: ${DOMAIN}
|
DOMAIN: ${DOMAIN}
|
||||||
|
|||||||
@@ -1,7 +1,21 @@
|
|||||||
|
# Docker's default json-file driver has NO rotation at all, and every container writes to the
|
||||||
|
# same filesystem as postgres_data, media_data and exports_data. Filling that filesystem does
|
||||||
|
# not degrade one subsystem — Postgres stops being able to write and the whole event goes down
|
||||||
|
# (see README "Sizing the disk"). This caps logs at 30 MB per service, permanently.
|
||||||
|
#
|
||||||
|
# Paired with RUST_LOG in .env: without it the app falls back to `eventsnap_backend=debug,
|
||||||
|
# tower_http=debug` (main.rs), which is a debug line per HTTP request including every preview.
|
||||||
|
x-logging: &default-logging
|
||||||
|
driver: json-file
|
||||||
|
options:
|
||||||
|
max-size: "10m"
|
||||||
|
max-file: "3"
|
||||||
|
|
||||||
services:
|
services:
|
||||||
db:
|
db:
|
||||||
image: postgres:16-alpine
|
image: postgres:16-alpine
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
logging: *default-logging
|
||||||
env_file: .env
|
env_file: .env
|
||||||
environment:
|
environment:
|
||||||
POSTGRES_USER: ${POSTGRES_USER}
|
POSTGRES_USER: ${POSTGRES_USER}
|
||||||
@@ -28,10 +42,18 @@ services:
|
|||||||
memory: 1G
|
memory: 1G
|
||||||
|
|
||||||
app:
|
app:
|
||||||
build:
|
# Production PULLS a prebuilt image; it never compiles. A release build of this crate is
|
||||||
context: ./backend
|
# fat-LTO over 427 dependencies (see backend/Cargo.toml [profile.release]) and peaks well
|
||||||
dockerfile: Dockerfile
|
# above the RAM a 4 GB box has spare with the stack running — and a rollback would be a
|
||||||
|
# second build under pressure. Images are built on a workstation and pushed; see
|
||||||
|
# docker-compose.build.yml and DEPLOYMENT_RUNBOOK.md.
|
||||||
|
#
|
||||||
|
# There is deliberately NO `build:` key here: without one, a wrong tag fails instantly with
|
||||||
|
# "manifest unknown" instead of silently starting a 45-minute compile on the event server.
|
||||||
|
# The `:?` form fails loudly on an unset variable rather than resolving to an empty tag.
|
||||||
|
image: registry.mc02.dev/eventsnap/app:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
logging: *default-logging
|
||||||
env_file: .env
|
env_file: .env
|
||||||
environment:
|
environment:
|
||||||
# Activates the production secret guard in config.rs — refuses to boot with
|
# Activates the production secret guard in config.rs — refuses to boot with
|
||||||
@@ -43,9 +65,23 @@ services:
|
|||||||
# create it and every upload 500s with EACCES. `environment` overrides `env_file`,
|
# create it and every upload 500s with EACCES. `environment` overrides `env_file`,
|
||||||
# so this is authoritative for the container.
|
# so this is authoritative for the container.
|
||||||
MEDIA_PATH: /media
|
MEDIA_PATH: /media
|
||||||
|
# Pinned for the same reason as MEDIA_PATH: `environment` beats `env_file`, so this cannot
|
||||||
|
# be lost by an operator who copies `.env.example` and edits only the secrets — which is
|
||||||
|
# the likely path, and `.env.example` ships the generic default of `true`.
|
||||||
|
#
|
||||||
|
# This is a product decision for this event, not a technical one: guests should be present
|
||||||
|
# at the party, not in a comment thread. Likes and captions stay on and are unaffected.
|
||||||
|
# Boot-time only, so changing it means `docker compose up -d`, not an admin toggle.
|
||||||
|
# To re-enable comments, delete this line and set COMMENTS_ENABLED in .env.
|
||||||
|
COMMENTS_ENABLED: "false"
|
||||||
depends_on:
|
depends_on:
|
||||||
db:
|
db:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
# Longer than the app's own 10s shutdown backstop (main.rs SHUTDOWN_GRACE), because Docker's
|
||||||
|
# default stop timeout is ALSO 10s — so a redeploy raced the graceful drain and could SIGKILL
|
||||||
|
# the process at the exact moment it was finishing, truncating the in-flight upload the
|
||||||
|
# graceful shutdown exists to protect. The app always exits well before 20s.
|
||||||
|
stop_grace_period: 20s
|
||||||
volumes:
|
volumes:
|
||||||
- media_data:/media
|
- media_data:/media
|
||||||
# Export archives live OUTSIDE /media so the public media ServeDir can't
|
# Export archives live OUTSIDE /media so the public media ServeDir can't
|
||||||
@@ -71,15 +107,18 @@ services:
|
|||||||
memory: 1G
|
memory: 1G
|
||||||
|
|
||||||
frontend:
|
frontend:
|
||||||
build:
|
# Pulled, not built — see the note on `app` above.
|
||||||
context: ./frontend
|
image: registry.mc02.dev/eventsnap/frontend:${EVENTSNAP_VERSION:?set EVENTSNAP_VERSION in .env}
|
||||||
dockerfile: Dockerfile
|
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
logging: *default-logging
|
||||||
env_file: .env
|
env_file: .env
|
||||||
environment:
|
environment:
|
||||||
# adapter-node behind Caddy TLS needs the public origin for CSRF checks on
|
# adapter-node behind Caddy TLS needs the public origin for CSRF checks on
|
||||||
# POST form actions — without it they fail only in production.
|
# POST form actions — without it they fail only in production.
|
||||||
ORIGIN: "https://${DOMAIN}"
|
# `:?` for the same reason EVENTSNAP_VERSION uses it. A blank DOMAIN doesn't fail — it
|
||||||
|
# produces `https://` here and collapses the Caddyfile's site block below, so the stack
|
||||||
|
# comes up with no TLS and no site and the only symptom is a browser error.
|
||||||
|
ORIGIN: "https://${DOMAIN:?set DOMAIN in .env}"
|
||||||
depends_on:
|
depends_on:
|
||||||
- app
|
- app
|
||||||
expose:
|
expose:
|
||||||
@@ -100,12 +139,13 @@ services:
|
|||||||
caddy:
|
caddy:
|
||||||
image: caddy:2-alpine
|
image: caddy:2-alpine
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
logging: *default-logging
|
||||||
environment:
|
environment:
|
||||||
# The Caddyfile's site address is `{$DOMAIN}`, read from THIS container's env.
|
# The Caddyfile's site address is `{$DOMAIN}`, read from THIS container's env.
|
||||||
# Without it, `{$DOMAIN}` expands to empty, the site block collapses, and Caddy
|
# Without it, `{$DOMAIN}` expands to empty, the site block collapses, and Caddy
|
||||||
# serves nothing / fails to obtain a TLS cert. `env_file` alone wouldn't help —
|
# serves nothing / fails to obtain a TLS cert. `env_file` alone wouldn't help —
|
||||||
# Caddy needs it in `environment`, and this keeps the Caddyfile the single source.
|
# Caddy needs it in `environment`, and this keeps the Caddyfile the single source.
|
||||||
DOMAIN: ${DOMAIN}
|
DOMAIN: ${DOMAIN:?set DOMAIN in .env}
|
||||||
ports:
|
ports:
|
||||||
- "80:80"
|
- "80:80"
|
||||||
- "443:443"
|
- "443:443"
|
||||||
|
|||||||
18
frontend/.dockerignore
Normal file
18
frontend/.dockerignore
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
# Build context exclusions.
|
||||||
|
#
|
||||||
|
# This one is a correctness fix, not just a speed one. The Dockerfile runs `npm ci` and THEN
|
||||||
|
# `COPY . .`, so a macOS `node_modules/` in the build context is copied OVER the Linux one the
|
||||||
|
# container just installed. npm's platform-specific binaries (@rollup/rollup-linux-x64-musl,
|
||||||
|
# @esbuild/linux-x64, @tailwindcss/oxide) live in separate optional packages, so the result is
|
||||||
|
# a two-platform node_modules with a host-authored .package-lock.json — sometimes it works,
|
||||||
|
# sometimes the build fails with a missing native binding, and which one you get depends on
|
||||||
|
# whether someone ran `npm install` locally.
|
||||||
|
node_modules/
|
||||||
|
|
||||||
|
# Build outputs — regenerated inside the image; stale copies only confuse the layer cache.
|
||||||
|
.svelte-kit/
|
||||||
|
build/
|
||||||
|
|
||||||
|
# Never let a real .env reach an image layer.
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
@@ -13,8 +13,12 @@ FROM node:22-alpine
|
|||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY --from=builder /app/build ./build
|
COPY --from=builder /app/build ./build
|
||||||
COPY --from=builder /app/package.json ./
|
# The lockfile comes along so the runtime deps are the ones that were tested. With only
|
||||||
RUN npm install --omit=dev
|
# package.json here, `npm install` re-resolved the `^`-ranged dependencies at build time, so an
|
||||||
|
# image rebuilt days later could ship different code than the one that passed the checks — the
|
||||||
|
# kind of difference that only shows up on the event server.
|
||||||
|
COPY --from=builder /app/package.json /app/package-lock.json ./
|
||||||
|
RUN npm ci --omit=dev
|
||||||
|
|
||||||
# Run as the image's built-in non-root `node` user.
|
# Run as the image's built-in non-root `node` user.
|
||||||
RUN chown -R node:node /app
|
RUN chown -R node:node /app
|
||||||
|
|||||||
Reference in New Issue
Block a user