fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ops): the hourly backup never ran, and an unset POSTGRES_USER crash-loops silently

Browse Source 
**The backup that did not exist.**
`.env.example` ships `EVENT_NAME=Max & Maria's Wedding` and the runbook tells you
to `cp .env.example .env`. Compose's env_file parser reads that fine. POSIX `sh`
does not: `. ./.env` aborts with "Unterminated quoted string" (verified, rc=2),
and every variable defined after that line is left unset.

The §10.2 cron script is `#!/bin/sh` + `set -eu` + `. ./.env`, so it exited
before `pg_dump` — every hour, into a log nobody reads. The only automated backup
of the one thing the runbook calls irreconstructible produced nothing, and §10.2's
own "prove it works NOW" only catches it if `.env` is already final at that
moment.

The script reads NOTHING from `.env` — `POSTGRES_USER`/`POSTGRES_DB` are expanded
inside the db container by the single-quoted `sh -c`. The source line was pure
liability and is gone. `EVENT_NAME` is now double-quoted in `.env.example`, which
both parsers read identically (verified), and the three interactive sourcing
sites now read just `$DOMAIN` instead of sourcing the whole file. The verify step
also proves the dump is a non-empty valid gzip containing tables, rather than
that a file exists.

Also fixes the script's `cd /root/eventsnap`, which contradicts §5's non-root
deploy and §13's `~/eventsnap` — under a non-root deploy it failed the same way,
silently.

**The crash loop with no message.**
`docker-compose.yml` interpolated `POSTGRES_USER`/`POSTGRES_DB` with no default
and no `:?` guard, into `environment:`, which OVERRIDES `env_file`. Unset does
not fall back — it resolves to the empty string, initdb creates a role and
database named "", `DATABASE_URL` still says `eventsnap`, and the app hits
`FATAL: role "eventsnap" does not exist` forever. `pg_isready -U "" -d ""` never
passes, so `app` never turns healthy and Caddy — gated on `service_healthy` —
never starts: port 443 dead for the whole event, exit only via `down -v`.

Both now carry `:?` guards (verified they fire), and §3's ".env template — ALL of
them" list, which omitted both, now includes them.

Other runbook corrections: the backup/restore pointer named a line range that had
drifted into an unrelated section and stopped mid-restore, before the media
restore and the mandatory `chown` — now referenced by heading, which cannot go
stale. §7.3 told you to verify that `EXPORT_PATH` is not pinned when §3 correctly
says it is. Stale counts: rev-list 196 -> 217, "versions 007–022" -> 007–031,
`frontend/Dockerfile:9` -> :8, and the low-disk description now matches the code
(the 10 GB absolute floor was removed as unreachable).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
fabi
2026-08-13 19:46:21 +02:00
parent cfc8bd0016
commit 9759c7c669
3 changed files with 85 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

95
DEPLOYMENT_RUNBOOK.md
View File

@@ -118,7 +118,7 @@ build/
```
Without these, the first time you run `cargo build` or `npm install` locally, every image build
ships a multi-GB context to an *emulated* builder. Worse: `frontend/Dockerfile:9` does `COPY . .`
ships a multi-GB context to an *emulated* builder. Worse: `frontend/Dockerfile:8` does `COPY . .`
**after** `npm ci`, so a macOS `node_modules/` would be merged over the container's Linux one.
### 2.2 Switch compose from `build:` to `image:`
@@ -184,6 +184,13 @@ EVENTSNAP_VERSION=v0.13.0
# ── Secrets — ALL of them, before the first `up -d` ───────────────────────
JWT_SECRET=<openssl rand -hex 64>
# These two are NOT optional and have no defaults. docker-compose.yml interpolates them into
# `environment:`, which overrides `env_file`, so leaving them out does not fall back — it creates
# a Postgres role and database named "" while DATABASE_URL still says `eventsnap`. The result is
# a permanent crash loop whose only clean exit is `down -v`. Compose now refuses to start without
# them, but write them here anyway: the three values below must agree with each other.
POSTGRES_USER=eventsnap
POSTGRES_DB=eventsnap
POSTGRES_PASSWORD=<openssl rand -hex 24>
DATABASE_URL=postgres://eventsnap:<SAME PASSWORD>@db:5432/eventsnap
ADMIN_PASSWORD_HASH='<docker run --rm caddy:2-alpine caddy hash-password --plaintext "pw">'
@@ -491,28 +498,33 @@ describes this trap at `backend/src/db.rs` (`explain_auth_failure`); this comman
### 7.2 Bring it up
```bash
# `.env` is consumed by docker compose, not by your shell — load it before using $DOMAIN.
set -a; . ./.env; set +a
# `.env` is consumed by docker compose, not by your shell — read $DOMAIN out of it first.
# Reads that ONE variable rather than sourcing the file: `.env` legitimately holds values with
# apostrophes (EVENT_NAME), and `. ./.env` aborts on one with "Unterminated quoted string".
DOMAIN=$(sed -n 's/^DOMAIN=//p' .env | tr -d "\"'")
docker compose up -d
docker compose logs -f app # wait for "database connected and migrations applied"
curl -fsS https://$DOMAIN/health # → ok (503 means the app is up but the DB is not)
```
### 7.3 Verify the three things compose does *not* pin
### 7.3 Verify what the container actually received
`MEDIA_PATH` is pinned to `/media` on the `app` service in `docker-compose.yml`. Its siblings
are not:
`MEDIA_PATH`, `EXPORT_PATH` and `APP_PORT` are all **pinned** on the `app` service in
`docker-compose.yml`, exactly as §3 says — editing them in `.env` changes nothing. This step is
not about whether they are pinned; it is about confirming the container got the values you think
it did, including the two that genuinely do come from `.env`:
```bash
docker compose exec app printenv DATABASE_URL EXPORT_PATH ADMIN_PASSWORD_HASH
```
1. **`DATABASE_URL`** must contain `@db:5432`. A dev `.env` points it at `@localhost`, which inside
the container is the app itself.
2. **`EXPORT_PATH`** must be `/exports`. Anywhere else and the keepsake archives are written to the
container's writable layer and **vanish on the next `up -d`** — including on a rollback.
3. **`ADMIN_PASSWORD_HASH`** must match `.env` **byte for byte.**
1. **`DATABASE_URL`** (from `.env`) must contain `@db:5432`. A dev `.env` points it at
`@localhost`, which inside the container is the app itself.
2. **`EXPORT_PATH`** (pinned) must read `/exports`. If it does not, the pin has been edited —
anywhere else and the keepsake archives are written to the container's writable layer and
**vanish on the next `up -d`**, including on a rollback.
3. **`ADMIN_PASSWORD_HASH`** (from `.env`) must match `.env` **byte for byte.**
**Then actually log in to `/admin` with the real password.** This is not optional politeness:
@@ -536,7 +548,7 @@ the cost of checking is 10 seconds; the cost of being wrong is the whole event.
## 8. Post-deploy verification
```bash
set -a; . ./.env; set +a # $DOMAIN comes from .env, not your shell
DOMAIN=$(sed -n 's/^DOMAIN=//p' .env | tr -d "\"'") # $DOMAIN comes from .env, not your shell
docker compose ps # db, app, frontend healthy; caddy has
# no healthcheck and shows only "running"
@@ -570,11 +582,11 @@ explanation:
$ git ls-tree --name-only v0.12.0 backend/migrations/ | wc -l
12 # 6 migrations. HEAD has 31.
$ git rev-list --count v0.12.0..HEAD
196
217
```
`db.rs` runs `sqlx::migrate!()` with no `set_ignore_missing`, so an image built from a 6-migration
tree, booting against a database that already carries versions 007–022, returns `VersionMissing`.
tree, booting against a database that already carries versions 007–031, returns `VersionMissing`.
`create_pool` errors, `main` exits 1, and `restart: unless-stopped` restarts it forever — with Caddy
still routing traffic to it. (`014_export_epoch.up.sql` documents this failure mode; §0 restates it.)
No `v0.12.0` image was ever built or pushed either, so the pre-pull would fail with
@@ -664,7 +676,11 @@ docker save registry.mc02.dev/eventsnap/app:v0.13.0 \
## 10. Backup
Full commands are in `README.md:315-435` and are correct — `pg_dump --clean --if-exists`, plus
Full commands are in README's **`## Backup`** and **`## Restore`** sections — read `## Restore` to
its END (through the media *and* exports restore, and the `chown` that follows), not just the
database step. Referenced by heading, not by line number: the previous pointer named a line range
that had drifted to the middle of an unrelated section and stopped mid-way through restore step 2,
which would have restored the database and no media. They are correct — `pg_dump --clean --if-exists`, plus
`alpine tar` out of `eventsnap_media_data` and `eventsnap_exports_data`, mounted at `/src` (not
`/media`), with `chown -R 100:101` on restore because the app runs non-root and BusyBox tar has no
`--same-owner`. There is deliberately no script.
@@ -704,25 +720,37 @@ while uploads are live, because a `pg_dump` is transactionally consistent on its
Media is the bulk and *is* recoverable from guests' phones in the worst case, so it stays on the
event-night schedule below.
Install this as **the same user you deployed as** (§5) — not root. `docker compose` needs that
user's docker group membership and its compose project, and a root crontab has neither.
```bash
# On the server, before the event. Hourly DB-only dump, keeping the last 48.
mkdir -p /root/eventsnap-dumps
cat >/root/eventsnap-dump.sh <<'SH'
mkdir -p ~/eventsnap-dumps
cat >~/eventsnap-dump.sh <<'SH'
#!/bin/sh
set -eu
cd /root/eventsnap
set -a; . ./.env; set +a
OUT="/root/eventsnap-dumps/db-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
# Must match your deploy directory from §5. cron starts in $HOME, so this cannot be relative.
cd "$HOME/eventsnap"
# NOTE: deliberately does NOT source .env. Nothing here reads it — POSTGRES_USER and POSTGRES_DB
# are expanded INSIDE the db container by the single-quoted sh -c below, using the values compose
# already injected. Sourcing it was actively harmful: `.env` legitimately contains values with
# apostrophes (EVENT_NAME="Max & Maria's Wedding"), and POSIX sh aborts on one with
# "Unterminated quoted string". Under `set -eu` this script would exit before pg_dump — every
# hour, silently, leaving the only automated backup of the irreplaceable table permanently empty.
OUT="$HOME/eventsnap-dumps/db-$(date -u +%Y%m%dT%H%M%SZ).sql.gz"
docker compose exec -T db sh -c \
'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' | gzip >"$OUT.tmp"
mv "$OUT.tmp" "$OUT" # atomic: never leave a truncated dump looking complete
ls -1t /root/eventsnap-dumps/db-*.sql.gz | tail -n +49 | xargs -r rm
ls -1t "$HOME"/eventsnap-dumps/db-*.sql.gz | tail -n +49 | xargs -r rm
SH
chmod +x /root/eventsnap-dump.sh
( crontab -l 2>/dev/null; echo '17 * * * * /root/eventsnap-dump.sh >>/var/log/eventsnap-dump.log 2>&1' ) | crontab -
chmod +x ~/eventsnap-dump.sh
( crontab -l 2>/dev/null; echo "17 * * * * $HOME/eventsnap-dump.sh >>$HOME/eventsnap-dumps/dump.log 2>&1" ) | crontab -
# Prove it works NOW, not at 23:00:
/root/eventsnap-dump.sh && ls -lh /root/eventsnap-dumps/
# Prove it works NOW, not at 23:00 — and prove it produced a NON-EMPTY dump, since the failure
# this replaces produced a zero-byte file and a clean exit code.
~/eventsnap-dump.sh && ls -lh ~/eventsnap-dumps/
gzip -t ~/eventsnap-dumps/db-*.sql.gz && echo "dump is a valid gzip"
zcat ~/eventsnap-dumps/db-*.sql.gz | grep -c 'CREATE TABLE' # must be > 0, not just "a file exists"
```
These land on the same filesystem, so they do **not** survive a disk loss — that is what §10.1 is
@@ -782,10 +810,13 @@ total ~11–13 GB of ~36 GB usable
3–5 GB of cache that permanently shrinks the guest quota, because the quota is recomputed against
*live* free space on every upload).
The `README.md:295-299` "ENOSPC" projection models guests **saturating the quota** (~12 GB of
media), not 100 photos. That scenario needs ~10× your expected volume — and it degrades gracefully:
the export preflight refuses up front rather than hitting ENOSPC mid-write, and the host dashboard
warns when free space drops below 10 GB or below the keepsake requirement (`handlers::host`'s low-disk thresholds).
The "ENOSPC" projection in README's **`### Sizing the disk`** discussion models guests
**saturating the quota** (~12 GB of media), not 100 photos. That scenario needs ~10× your expected
volume — and it degrades gracefully: the export preflight refuses up front rather than hitting
ENOSPC mid-write, and the host dashboard warns while free space is still **1.25× above the level at
which uploads stop** (`handlers::host::disk_is_low`). Note that is the only trigger: the separate
10 GB absolute floor this used to describe was removed as unreachable, because the derived
threshold is always higher.
---
@@ -895,8 +926,10 @@ cd ~/eventsnap
# FIRST LINE, ALWAYS. `.env` is read by docker compose, NOT by your shell — without this,
# every `$DOMAIN` below expands to nothing and `curl https:///health` reads like an outage
# when the site is fine.
set -a; . ./.env; set +a
# when the site is fine. Reads the one variable instead of sourcing the file, because `.env`
# legitimately contains an apostrophe (EVENT_NAME) and `. ./.env` dies on it — which at 11pm
# looks exactly like the outage you came here to diagnose.
DOMAIN=$(sed -n 's/^DOMAIN=//p' .env | tr -d "\"'")
# Is it alive? (200 = app AND database are answering; 503 = the app is up, the DB is not)
curl -fsS https://$DOMAIN/health