fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy): give Postgres a CPU floor that Docker actually honours

Browse Source 
`deploy.resources.reservations.cpus` was doing nothing. Outside Swarm, `docker compose up`
silently drops it — verified by inspecting a running container, where CpuShares, CpuQuota
and CpusetCpus were all unset while `limits.cpus` and `reservations.memory` came through as
NanoCpus and MemoryReservation. So the comment calling it "the piece that actually protects
the database" described a guarantee the box never had.

It matters on the CX22 the runbook targets: the ceilings sum to 1.2 + 0.6 + 0.5 = 2.3 on
2 vCPU, so the other services can oversubscribe the machine, and with every container on the
default weight Postgres competed on equal footing with two image resizes and an ffmpeg
poster. Replaced with `cpu_shares`, which does survive the translation — db 2048, caddy
1024, app 512, frontend 256 — so the weighting only binds when the CPU is actually
saturated, which is the moment the database must not lose.

The Caddyfile gains a 10s header-read timeout: there was no read timeout anywhere, so a
client could hold a connection, a tokio task and a `.tmp` file open indefinitely by sending
one byte a minute, and the upload sweeper is keyed on mtime precisely so a live upload never
ages out. Body reads stay unbounded — a 500 MB video over cellular legitimately takes
minutes, and a body timeout would fail exactly the uploads this product exists to collect.

.env.example documents that estimated_guest_count is a live input to the quota divisor
rather than the inert setting both it and the runbook previously implied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-08-11 22:44:48 +02:00
parent 32dfe6874a
commit 0c0d5d5981
6 changed files with 344 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
README.md
View File

@@ -49,7 +49,7 @@ A guest scans the QR code on their way in, types their name, and is immediately
| Styling | Tailwind CSS v4 |
| Backend | Rust + Axum |
| Async | Tokio |
| Database | PostgreSQL 16 via SQLx (compile-time query checking) |
| Database | PostgreSQL 16 via SQLx (runtime query API; migrations embedded at compile time) |
| Auth | Custom JWT (`jsonwebtoken`) + bcrypt PINs |
| Image processing | `image` crate + `oxipng` (lossless compression) |
| Video processing | ffmpeg via `tokio::process::Command` |
@@ -150,45 +150,63 @@ Caddy automatically obtains a Let's Encrypt certificate on first start. The app
### Updating an existing deployment
> **`docker compose up -d` alone will NOT deploy your changes.** `app` and `frontend` are
> `build:` services with no published image tag, and Compose has no source-change detection:
> if an image with that name already exists it is reused. After a `git pull` the command
> reports `Container … Running`, changes nothing, and **exits 0** — so a deploy that shipped
> nothing looks exactly like a successful one. `--build` is what makes it real.
> **The event server never compiles.** `app` and `frontend` have **no `build:` key** — they
> pull an immutable tag from the registry (`docker-compose.yml:53` says so explicitly, so that
> a wrong tag fails instantly with `manifest unknown` instead of silently starting a 45-minute
> compile on the box guests are using). A `git pull` therefore deploys **nothing** on its own,
> and `docker compose up -d --build` **errors** — there is nothing to build. Deploying means
> pushing a new tag from a workstation and pointing `EVENTSNAP_VERSION` at it.
```bash
# ── On your workstation: build and push the new tag ───────────────────────────
# Push the rollback twin at the same time, from the same source — see
# DEPLOYMENT_RUNBOOK.md §9 for why an identical second tag is the rollback target.
VERSION=v0.13.1
docker buildx build --platform linux/amd64 \
-t registry.mc02.dev/eventsnap/app:$VERSION \
-t registry.mc02.dev/eventsnap/app:$VERSION-a --push ./backend
docker buildx build --platform linux/amd64 \
-t registry.mc02.dev/eventsnap/frontend:$VERSION \
-t registry.mc02.dev/eventsnap/frontend:$VERSION-a --push ./frontend
# ── On the server ─────────────────────────────────────────────────────────────
cd /path/to/eventsnap
# 1. Back up first — migrations run automatically on boot and are not reversible in place.
# (See "Backup" below; the database dump is the one that matters here.)
# 2. Fetch the new code.
# 2. Fetch the new compose/Caddyfile. This does NOT change which image runs.
git pull
# 3. Rebuild and restart the application services. --build is NOT optional.
docker compose up -d --build
# 3. Point the stack at the new tag.
sed -i 's/^EVENTSNAP_VERSION=.*/EVENTSNAP_VERSION=v0.13.1/' .env
# 4. Apply any Caddyfile change. Step 3 does NOT do this — see the warning below.
# 4. Pull explicitly, BEFORE restarting. A failure here (bad tag, registry down) leaves the
# running stack untouched; letting `up -d` discover it takes the app down first.
docker compose pull app frontend
# 5. Restart onto the new images.
docker compose up -d app frontend
# 6. Apply any Caddyfile change. Step 5 does NOT do this — see the warning below.
docker compose up -d --force-recreate caddy
# 5. Confirm the app came back up. Anything other than "ok" means check the logs.
# 7. Confirm the app came back up. Anything other than "ok" means check the logs.
curl -fsS https://DOMAIN/health && echo
# 6. Confirm a NEW image was actually built. Note the IMAGE ID before you start and
# compare — it must have changed. (Ignore the CREATED column; it reports the base
# layer's age, not this build's.) An unchanged ID means step 3 ran without --build
# and you are still serving the old code.
# 8. Confirm the running containers are actually on the new tag.
docker compose images app frontend
```
Migrations are applied by the backend on startup, so step 3 covers them. If `app` stays
Migrations are applied by the backend on startup, so step 5 covers them. If `app` stays
unhealthy afterwards, `docker compose logs app` will name the failing migration — and note
that a migration applied by a *newer* build is not removed by checking out an older commit,
so rolling back code without restoring the database snapshot from step 1 leaves the schema
ahead of the binary and the app refusing to boot.
that a migration applied by a *newer* build is not removed by rolling the tag back, so
reverting `EVENTSNAP_VERSION` without restoring the database snapshot from step 1 leaves the
schema ahead of the binary and the app refusing to boot. **This is why the rollback target is
an identical twin tag rather than an older release** — see `DEPLOYMENT_RUNBOOK.md` §9.
> **Why step 4 exists.** `--build` only rebuilds services that have a `build:` section, and
> `caddy` is a pinned upstream image. Compose decides whether to recreate a container from its
> **Why step 6 exists.** Steps 4–5 only touch `app` and `frontend`; `caddy` is a separate
> pinned upstream image. Compose decides whether to recreate a container from its
> *config hash*, which covers the mount **specification** (`./Caddyfile:/etc/caddy/Caddyfile:ro`)
> but **not the file's contents** — so a `git pull` that changes `./Caddyfile` produces no
> delta, Compose reports `Running`, and Caddy keeps serving its old config indefinitely. Exit
@@ -196,7 +214,7 @@ ahead of the binary and the app refusing to boot.
>
> That is not hypothetical: the fix that made the keepsake download work on iOS
> (`137c4ee`) touched the Caddyfile and four e2e files and nothing else, so **all** of its
> production effect lives in that one file. Without step 4 you deploy it, watch both image IDs
> production effect lives in that one file. Without step 6 you deploy it, watch both image IDs
> change, and iOS downloads stay broken.
>
> `--force-recreate` rather than `restart` or `caddy reload`: the bind mount is resolved to an