From 27e4004cc8e014369d2b6c818b51c7764186df8d Mon Sep 17 00:00:00 2001 From: fabi Date: Tue, 28 Jul 2026 07:49:52 +0200 Subject: [PATCH] docs(backup): make the backup commands work; fix the e2e/prod divergences MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backup. Both documented commands failed on the shipped stack, and the sentence explaining them was wrong too: - `pg_dump $DATABASE_URL` — `DATABASE_URL` is only ever in the compose environment, never an operator's shell, and it points at `db:5432`, which is compose-internal DNS. The app image has no postgres client either. - `> /media/backups/…` — `/media` is a named volume mounted inside the app container, not a host path, and nothing ever creates a `backups` subdirectory. - `rsync /opt/eventsnap/media/` — that path does not exist anywhere. - "a single path to back up" — false, and dangerously so: exports were moved to their own `exports_data` volume precisely so a keepsake (which contains every photo in the event) can't be served off the media tree. Backing up only `media_data` silently loses every generated keepsake. Rewritten as three commands — db via `docker compose exec -T db pg_dump`, and one `docker run … tar` per volume — all verified against the running stack. The volume mounts use `/src`, not `/media`: I hit the footgun while testing this. Docker pre-populates an EMPTY volume from the image's own directory and chowns it to match, so `-v media_data:/media alpine` tars alpine's cdrom/floppy/usb, writes them into the volume, and leaves it root-owned so the non-root app can no longer write. Mounting where the image has nothing avoids all of it. Documented inline so the next person doesn't rediscover it. Also correct the architecture notes: `/media/*` no longer routes to the backend (that static tree was removed as a gating bypass), and `exports_data` was missing from the volume list — the one volume an operator most needs to know about. e2e stack: add the `EXPORT_PATH` + `/exports` volume it was missing. The file says "mirrors production layout"; without these, exports landed on the container's writable layer at the default path, so export-leak and export-video wrote real archives into ephemeral storage and the "exports live outside media" invariant was never actually exercised. Pre-existing red test, unrelated to the audit: all four 02-upload/quota tests have been failing since 4464147 "stop /me/quota leaking raw disk to guests" (2026-07-19), which post-dates the spec's last edit. `setLimitTo` calibrated `quota_tolerance` from `free_disk_bytes` read through the GUEST's token — a field that commit deliberately zeroes for non-staff. Dividing by it yields a NaN tolerance, so every test in the block died in the helper. Read the calibration inputs through a staff token and keep reading the ceiling back through the guest, whose limit is the thing under test. Co-Authored-By: Claude Opus 5 (1M context) --- PROJECT.md | 19 +++++++----- README.md | 51 ++++++++++++++++++++++++++----- e2e/docker-compose.test.yml | 10 ++++++ e2e/specs/02-upload/quota.spec.ts | 22 ++++++++++--- 4 files changed, 81 insertions(+), 21 deletions(-) diff --git a/PROJECT.md b/PROJECT.md index 7095f83..034a906 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -1133,16 +1133,19 @@ eventsnap/ ### Backup Strategy -```bash -# Daily (e.g. as a separate Compose service or cron on the VPS) -pg_dump $DATABASE_URL | gzip > /media/backups/db_$(date +%Y-%m-%d).sql.gz +Three artefacts in three places: the database, the `media_data` volume +(originals + derivatives), and the **separate** `exports_data` volume. See +[README.md](README.md#backup) for the exact commands. -# Weekly: rsync /media volume to Hetzner Storage Box -rsync -az /opt/eventsnap/media/ \ - user@u123456.your-storagebox.de:backup/eventsnap/ -``` +Everything runs through `docker compose` / `docker run`, because `DATABASE_URL` +and the `/media` and `/exports` paths only exist inside the compose network — +they are not host paths, and `DATABASE_URL` is never exported into an operator's +shell. -The `/media` volume contains originals, previews, thumbnails, generated exports, and DB backups — a single volume to back up. +Export archives are deliberately outside `MEDIA_PATH` (`EXPORT_PATH=/exports`): a +keepsake contains every photo in the event, and keeping it off the media tree is +what stops it being reachable except through the ticket-gated handler. A backup +of the media volume alone silently loses every generated keepsake. --- diff --git a/README.md b/README.md index f4f1751..73d880c 100644 --- a/README.md +++ b/README.md @@ -162,23 +162,58 @@ See [.env.example](.env.example) for the full list with descriptions and default └────────┘ ``` -- `/api/*` and `/media/*` → Rust backend +- `/api/*` → Rust backend - Everything else → SvelteKit frontend (`adapter-node`) -- Named volumes: `postgres_data`, `media_data`, `caddy_data` +- Named volumes: `postgres_data`, `media_data`, `exports_data`, `caddy_data` + +Media is **not** served as static files. Every image goes through a +visibility-checked alias (`/api/v1/upload/{id}/{preview,display,thumbnail,original}`) +so a host takedown or a ban actually revokes access to the bytes. --- ## Backup -```bash -# Database snapshot -pg_dump $DATABASE_URL | gzip > /media/backups/db_$(date +%Y-%m-%d).sql.gz +There are **three** things to back up, and they live in three different places. +`DATABASE_URL` and the container paths (`/media`, `/exports`) are meaningful only +*inside* the compose network — they are not host paths, and `DATABASE_URL` is +never exported into an operator's shell — so every command below runs through +`docker compose` from the repo directory. -# Weekly offsite sync (Hetzner Storage Box or similar) -rsync -az /opt/eventsnap/media/ user@storagebox.example.com:backup/eventsnap/ +```bash +# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no +# postgres client), reading credentials from the compose environment. +mkdir -p ./backups +docker compose exec -T db \ + sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \ + | gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz + +# 2. Uploaded media (originals + derivatives) out of the named volume. +# NOTE the mountpoint is /src, not /media: if the volume is ever empty, Docker +# pre-populates a fresh mount from the image's own directory, and alpine ships a +# /media containing cdrom/floppy/usb. Mounting somewhere the image has nothing +# avoids silently tarring (and polluting the volume with) those. +docker run --rm \ + -v eventsnap_media_data:/src:ro -v "$PWD/backups":/backup \ + alpine tar czf /backup/media_$(date +%Y-%m-%d).tar.gz -C /src . + +# 3. Export archives — a SEPARATE volume (see the security note below). +docker run --rm \ + -v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \ + alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src . + +# Weekly offsite sync of the three artefacts above. +rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/ ``` -The `/media` volume holds originals, previews, thumbnails, exports, and DB backups — a single path to back up. +Volume names are prefixed with the compose project name — `eventsnap_` if you run +from a directory called `eventsnap`. Confirm yours with `docker volume ls`. + +> **Exports are deliberately NOT under `/media`.** They live on their own +> `exports_data` volume (`EXPORT_PATH=/exports`) because a keepsake archive +> contains every photo in the event; keeping it outside the media tree is what +> stops it being reachable except through the ticket-gated download handler. +> Backing up only the media volume therefore loses every generated keepsake. --- diff --git a/e2e/docker-compose.test.yml b/e2e/docker-compose.test.yml index adf77d5..15a0440 100644 --- a/e2e/docker-compose.test.yml +++ b/e2e/docker-compose.test.yml @@ -42,11 +42,20 @@ services: EVENT_NAME: E2E Test Event APP_PORT: '3000' MEDIA_PATH: /media + # Exports MUST live outside MEDIA_PATH — see the note on the volume below and + # config.rs::validate. Omitting this left exports on the container's writable + # layer at the /exports default, so the test stack diverged from the prod layout + # it claims to mirror, and export-leak/export-video wrote real archives into + # ephemeral storage. + EXPORT_PATH: /exports SESSION_EXPIRY_DAYS: '30' EVENTSNAP_TEST_MODE: '1' # ENABLES /admin/__truncate — never set in prod RUST_LOG: eventsnap_backend=info,tower_http=warn volumes: - media_data:/media + # Separate volume, exactly as in production: a keepsake archive contains every + # photo in the event, so it is kept off the media tree. + - exports_data:/exports expose: - '3000' @@ -75,3 +84,4 @@ services: volumes: media_data: + exports_data: diff --git a/e2e/specs/02-upload/quota.spec.ts b/e2e/specs/02-upload/quota.spec.ts index f054d7b..34126eb 100644 --- a/e2e/specs/02-upload/quota.spec.ts +++ b/e2e/specs/02-upload/quota.spec.ts @@ -56,14 +56,21 @@ function upload(jwt: string, name: string) { /** * Pick a `quota_tolerance` that makes the per-user ceiling land on `targetBytes`. * limit = floor(free_disk * tolerance / max(active, 1)) ⇒ tolerance = target * active / free. + * + * `staffJwt` reads the calibration inputs, `jwt` is the guest the limit is being aimed at. + * They must be different tokens: `free_disk_bytes` and `active_uploaders` are raw server + * telemetry and `/me/quota` zeroes both for non-staff (handlers/me.rs — "must never reach a + * guest"). Calibrating off the guest's own response divides by zero and yields a NaN + * tolerance, which is what silently broke this whole describe block. */ async function setLimitTo( api: any, adminToken: string, + staffJwt: string, jwt: string, targetBytes: number ): Promise { - const q = await quotaOf(jwt); + const q = await quotaOf(staffJwt); expect( q.free_disk_bytes, 'the disk must be readable, else quota fails OPEN and proves nothing' @@ -72,6 +79,7 @@ async function setLimitTo( const tolerance = (targetBytes * active) / (q.free_disk_bytes as number); await api.patchConfig(adminToken, { quota_tolerance: tolerance.toExponential(12) }); + // Read back through the GUEST, whose ceiling is the one under test. const after = await quotaOf(jwt); expect(after.enabled).toBe(true); return after.limit_bytes as number; @@ -89,10 +97,11 @@ test.describe('Upload — storage quota enforcement', () => { api, adminToken, guest, + host, }) => { const g = await guest('QuotaOver'); // Ceiling below one file: the very first upload must be refused. - const limit = await setLimitTo(api, adminToken, g.jwt, Math.floor(SIZE / 2)); + const limit = await setLimitTo(api, adminToken, host.jwt, g.jwt, Math.floor(SIZE / 2)); expect(limit).toBeLessThan(SIZE); const res = await upload(g.jwt, 'too-big.jpg'); @@ -111,9 +120,10 @@ test.describe('Upload — storage quota enforcement', () => { api, adminToken, guest, + host, }) => { const g = await guest('QuotaUnder'); - await setLimitTo(api, adminToken, g.jwt, SIZE * 4); + await setLimitTo(api, adminToken, host.jwt, g.jwt, SIZE * 4); expect((await upload(g.jwt, 'fine.jpg')).status).toBe(201); expect((await quotaOf(g.jwt)).used_bytes).toBe(SIZE); @@ -123,11 +133,12 @@ test.describe('Upload — storage quota enforcement', () => { api, adminToken, guest, + host, }) => { const g = await guest('QuotaRacer'); // Room for exactly ONE file. - const limit = await setLimitTo(api, adminToken, g.jwt, Math.floor(SIZE * 1.5)); + const limit = await setLimitTo(api, adminToken, host.jwt, g.jwt, Math.floor(SIZE * 1.5)); expect(limit).toBeGreaterThanOrEqual(SIZE); expect(limit).toBeLessThan(SIZE * 2); @@ -170,10 +181,11 @@ test.describe('Upload — storage quota enforcement', () => { api, adminToken, guest, + host, }) => { // Zero test hits before this — and it is the source of the "X von Y MB genutzt" widget. const g = await guest('QuotaWidget'); - await setLimitTo(api, adminToken, g.jwt, SIZE * 10); + await setLimitTo(api, adminToken, host.jwt, g.jwt, SIZE * 10); const before = await quotaOf(g.jwt); expect(before.enabled).toBe(true);