docs: add a restore procedure, fix the backup cadence, and correct quota_tolerance
Four things, all found by the same question: what does an operator standing at the
venue actually need?
A RESTORE PROCEDURE. There was none anywhere, and a backup you have never restored
isn't a backup. Two hazards worth writing down: media must be extracted preserving
ownership (the app runs as uid 100 / gid 101, and a root-owned restore makes every
upload fail with EACCES surfacing as a generic 500), and the app must be STOPPED
first, because migrations run on boot and a live pool will fight the restore.
Both the backup and the restore commands were run against the real stack before being
written down, which caught two that would have failed:
- The plain `pg_dump` did not restore: `psql` aborted on `ERROR: schema
"_sqlx_test" already exists`. pg_dump emits no DROPs without --clean --if-exists,
so the documented dump could only ever be restored into an empty database. Fixed
at the source (the dump is now self-cleaning) and verified end to end: 16 tables
back, exit 0.
- `--same-owner` does not exist in BusyBox tar, which is what `alpine` ships, so
the extract aborted before unpacking anything. `--numeric-owner` plus the
explicit chown, verified to land 100:101.
BACKUP CADENCE. "Weekly offsite" is the wrong shape when every irreplaceable byte is
created in one eight-hour window and nobody can retake a wedding. The backup that
matters runs that night, and again after the release so the keepsake is captured.
Also: take the DB dump and the media tarball back to back, or you get rows pointing
at files the dump doesn't know about.
quota_tolerance WAS DOCUMENTED AS SOMETHING IT ISN'T. .env.example called it "fraction
of disk that triggers the low-storage warning". It is the multiplier in
`floor(free_disk * tolerance / active_uploaders)` -- so an operator who wants "warn me
later" and sets 0.95 is actually authorising guests to fill 95% of the disk, moving
the fixed point from 43% to ~49% and eating the export headroom. The admin UI labelled
it "Toleranz (0-1)" with no explanation at all, which invites exactly that reading;
it is now "Speicher-Anteil für Gäste" with the formula in the hint. Wrong docs on a
tuning knob are worse than no docs.
SIZING. New section with the arithmetic: three volumes on one filesystem, the quota
fixed point at tolerance/(1+tolerance), and the fact the 80 GB baseline does not cover
the keepsake -- both archives are built concurrently and each is roughly a second copy
of every original. Provision ~3x expected media, or give exports its own volume.
Also ticks the low-disk alert off the roadmap, since it now exists.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
20
.env.example
20
.env.example
@@ -54,8 +54,26 @@ EXPORT_PATH=/exports
|
|||||||
# max image size 20 MB
|
# max image size 20 MB
|
||||||
# max video size 500 MB
|
# max video size 500 MB
|
||||||
# estimated guests 100
|
# estimated guests 100
|
||||||
# quota tolerance 0.75 (fraction of disk that triggers the low-storage warning)
|
# quota tolerance 0.75 (see below — NOT a warning threshold)
|
||||||
# Adjust these in the admin UI before the event if needed.
|
# Adjust these in the admin UI before the event if needed.
|
||||||
|
#
|
||||||
|
# quota_tolerance is the MULTIPLIER IN THE PER-USER QUOTA FORMULA, not the point at
|
||||||
|
# which anything warns you:
|
||||||
|
#
|
||||||
|
# per_user_limit = floor(free_disk * quota_tolerance / active_uploaders)
|
||||||
|
#
|
||||||
|
# It is recomputed against LIVE free space on every upload, so it self-throttles: guests
|
||||||
|
# converge on a fixed point at tolerance/(1+tolerance) of the free space you started
|
||||||
|
# with — 43% at 0.75, i.e. ~30 GB of a fresh 70 GB.
|
||||||
|
#
|
||||||
|
# Raising it therefore AUTHORISES GUESTS TO FILL MORE OF THE DISK. Setting 0.95 in the
|
||||||
|
# belief that it means "warn me later" moves the fixed point to ~49% and eats the
|
||||||
|
# headroom the keepsake needs — and the keepsake needs a lot, because Gallery.zip and
|
||||||
|
# Memories.zip are each roughly a second copy of every original (both store media
|
||||||
|
# uncompressed). Budget for media + 2x media, or move exports to their own volume.
|
||||||
|
#
|
||||||
|
# 0.75 is the tested default. Lower it if the box is tight; raise it only if you have
|
||||||
|
# provisioned export headroom separately.
|
||||||
|
|
||||||
# ── Workers ───────────────────────────────────────────────────────────────────
|
# ── Workers ───────────────────────────────────────────────────────────────────
|
||||||
# Number of parallel image/video compression workers. Default 2. This is the main
|
# Number of parallel image/video compression workers. Default 2. This is the main
|
||||||
|
|||||||
125
README.md
125
README.md
@@ -34,7 +34,6 @@ A guest scans the QR code on their way in, types their name, and is immediately
|
|||||||
### Planned (v1.x)
|
### Planned (v1.x)
|
||||||
|
|
||||||
- Individual file download button
|
- Individual file download button
|
||||||
- Low-disk alert (< 10 GB free)
|
|
||||||
- Event banner / cover image
|
- Event banner / cover image
|
||||||
- Chunked resumable upload for large videos
|
- Chunked resumable upload for large videos
|
||||||
- Host-curated story highlights
|
- Host-curated story highlights
|
||||||
@@ -231,6 +230,45 @@ so a host takedown or a ban actually revokes access to the bytes.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Sizing the disk
|
||||||
|
|
||||||
|
`postgres_data`, `media_data` and `exports_data` are all Docker named volumes under
|
||||||
|
`/var/lib/docker/volumes`, so **they share one filesystem**. Filling it does not
|
||||||
|
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
||||||
|
down.
|
||||||
|
|
||||||
|
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
|
||||||
|
active_uploaders` is recomputed against live free space on every upload, so guests
|
||||||
|
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
|
||||||
|
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
|
||||||
|
the OS and images, media settles at ~30 GB and stops.
|
||||||
|
|
||||||
|
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
|
||||||
|
`Memories.zip` are built concurrently and each is roughly a second copy of every
|
||||||
|
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
|
||||||
|
untouched original for every video and for every image at or under 5 MB. So a release
|
||||||
|
wants room for **two more copies of the gallery** on top of the gallery itself.
|
||||||
|
|
||||||
|
| Stage | Used | Free (80 GB box) |
|
||||||
|
|---|---|---|
|
||||||
|
| Fresh box (OS + images) | ~10 GB | ~70 GB |
|
||||||
|
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
|
||||||
|
| Host releases → both archives | ~100 GB | **ENOSPC** |
|
||||||
|
|
||||||
|
Two ways to size for it:
|
||||||
|
|
||||||
|
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
||||||
|
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
||||||
|
size that one at ~2× expected media.
|
||||||
|
|
||||||
|
This is no longer silent. The export refuses up front with the two numbers rather than
|
||||||
|
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
|
||||||
|
generation before it starts (so peak is one generation, not two), and the host
|
||||||
|
dashboard warns as soon as the keepsake would not fit — which is the only point at
|
||||||
|
which anyone can still do something about it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Backup
|
## Backup
|
||||||
|
|
||||||
There are **three** things to back up, and they live in three different places.
|
There are **three** things to back up, and they live in three different places.
|
||||||
@@ -242,9 +280,12 @@ never exported into an operator's shell — so every command below runs through
|
|||||||
```bash
|
```bash
|
||||||
# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no
|
# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no
|
||||||
# postgres client), reading credentials from the compose environment.
|
# postgres client), reading credentials from the compose environment.
|
||||||
|
# --clean --if-exists makes the dump SELF-CLEANING: without it the restore below
|
||||||
|
# aborts on the first "already exists" against a database that has ever booted,
|
||||||
|
# which is every database you would actually want to restore over.
|
||||||
mkdir -p ./backups
|
mkdir -p ./backups
|
||||||
docker compose exec -T db \
|
docker compose exec -T db \
|
||||||
sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
sh -c 'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
||||||
| gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz
|
| gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz
|
||||||
|
|
||||||
# 2. Uploaded media (originals + derivatives) out of the named volume.
|
# 2. Uploaded media (originals + derivatives) out of the named volume.
|
||||||
@@ -261,7 +302,7 @@ docker run --rm \
|
|||||||
-v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \
|
-v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \
|
||||||
alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src .
|
alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src .
|
||||||
|
|
||||||
# Weekly offsite sync of the three artefacts above.
|
# Offsite sync of the three artefacts above.
|
||||||
rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/
|
rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -274,6 +315,82 @@ from a directory called `eventsnap`. Confirm yours with `docker volume ls`.
|
|||||||
> stops it being reachable except through the ticket-gated download handler.
|
> stops it being reachable except through the ticket-gated download handler.
|
||||||
> Backing up only the media volume therefore loses every generated keepsake.
|
> Backing up only the media volume therefore loses every generated keepsake.
|
||||||
|
|
||||||
|
### When to run it
|
||||||
|
|
||||||
|
**A nightly cron is the wrong shape for this app.** Every irreplaceable byte is
|
||||||
|
created inside one eight-hour window, and nobody can retake a wedding. Run the three
|
||||||
|
commands above:
|
||||||
|
|
||||||
|
1. **The night of the event**, once uploads have stopped. This is the backup that
|
||||||
|
matters; everything else is a formality.
|
||||||
|
2. **After the host releases the gallery**, so the generated keepsake is captured too.
|
||||||
|
3. Weekly thereafter, until the event is archived and torn down.
|
||||||
|
|
||||||
|
Take the DB dump and the media tarball **back to back**, without uploads in flight
|
||||||
|
between them. Upload rows reference files by path — a database from 22:00 and a media
|
||||||
|
volume from 23:00 gives you rows pointing at files the dump doesn't know about, and
|
||||||
|
rows whose files aren't in the tarball. Locking uploads from the host dashboard first
|
||||||
|
(**Uploads sperren**) makes the pair genuinely consistent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Restore
|
||||||
|
|
||||||
|
An untested backup is not a backup. Run this once against a scratch host **before**
|
||||||
|
the event — it is roughly ten minutes, and it is the only way to find out that your
|
||||||
|
tarball is empty or your dump is truncated while that is still a small problem.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 0. Stop the app FIRST. Migrations run on boot and a live pool will fight the
|
||||||
|
# restore — a booting app against a half-restored schema can leave the migration
|
||||||
|
# table and the schema disagreeing, which is its own recovery problem.
|
||||||
|
# Leave `db` running: the dump is restored through it.
|
||||||
|
docker compose stop app caddy
|
||||||
|
|
||||||
|
# 1. Database. The dump carries its own DROPs (step 1 of Backup), so this replaces
|
||||||
|
# rather than collides. A dump taken WITHOUT --clean --if-exists will abort here
|
||||||
|
# on the first "already exists" — restore that one into a fresh empty database
|
||||||
|
# instead.
|
||||||
|
gunzip -c ./backups/db_2026-07-29.sql.gz \
|
||||||
|
| docker compose exec -T db \
|
||||||
|
sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" --set ON_ERROR_STOP=1'
|
||||||
|
|
||||||
|
# 2. Media. NOTE the `--numeric-owner` and the chown: the app runs as a
|
||||||
|
# NON-ROOT user (uid 100, gid 101 — `addgroup -S app && adduser -S app`), and a
|
||||||
|
# restore that lands root-owned files makes every upload fail with EACCES deep in
|
||||||
|
# the write path, surfacing to the guest as a generic 500 with nothing in the UI
|
||||||
|
# to suggest permissions. The explicit chown is what guarantees it — BusyBox tar
|
||||||
|
# (which is what `alpine` ships) has no --same-owner, and restores ownership only
|
||||||
|
# because it runs as root here.
|
||||||
|
docker run --rm \
|
||||||
|
-v eventsnap_media_data:/dst -v "$PWD/backups":/backup:ro \
|
||||||
|
alpine sh -c 'tar xzf /backup/media_2026-07-29.tar.gz -C /dst \
|
||||||
|
--numeric-owner && chown -R 100:101 /dst'
|
||||||
|
|
||||||
|
# 3. Exports. Same volume-name caveat, same ownership rules.
|
||||||
|
docker run --rm \
|
||||||
|
-v eventsnap_exports_data:/dst -v "$PWD/backups":/backup:ro \
|
||||||
|
alpine sh -c 'tar xzf /backup/exports_2026-07-29.tar.gz -C /dst \
|
||||||
|
--numeric-owner && chown -R 100:101 /dst'
|
||||||
|
|
||||||
|
# 4. Back up. Migrations run, then export recovery re-arms any keepsake whose file
|
||||||
|
# didn't come back with the volume.
|
||||||
|
docker compose up -d app caddy
|
||||||
|
docker compose logs -f app # watch for "migrations applied"
|
||||||
|
|
||||||
|
# 5. Verify — all three, not just the first.
|
||||||
|
curl -fsS https://DOMAIN/health && echo # → ok
|
||||||
|
# … then sign in as host and confirm the feed renders images (proves the media
|
||||||
|
# volume restored AND is readable by uid 100), and that the keepsake downloads.
|
||||||
|
```
|
||||||
|
|
||||||
|
If the media volume restored but images 404 while the feed lists them, the paths are
|
||||||
|
there and the bytes aren't — check `docker compose exec app ls -ln /media/originals`
|
||||||
|
and confirm both the files and the `100:101` ownership.
|
||||||
|
|
||||||
|
The restore is deliberately **not** automated. It is rare, destructive, and the one
|
||||||
|
operation where a script that half-works is worse than a checklist someone reads.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Running the backend test suite
|
## Running the backend test suite
|
||||||
@@ -348,7 +465,7 @@ Open:
|
|||||||
- [ ] SSE delta-fetch on foreground reconnect (scaffolded in [sse.ts](frontend/src/lib/sse.ts), not wired)
|
- [ ] SSE delta-fetch on foreground reconnect (scaffolded in [sse.ts](frontend/src/lib/sse.ts), not wired)
|
||||||
- [ ] Live diashow / slideshow mode — see [docs/CONCEPT_DIASHOW.md](docs/CONCEPT_DIASHOW.md)
|
- [ ] Live diashow / slideshow mode — see [docs/CONCEPT_DIASHOW.md](docs/CONCEPT_DIASHOW.md)
|
||||||
- [ ] Individual file download button per post
|
- [ ] Individual file download button per post
|
||||||
- [ ] Low-disk alert (< 10 GB free)
|
- [x] Low-disk alert — host dashboard warns below 10 GB free, or whenever the keepsake would not fit
|
||||||
- [ ] Event banner / cover image
|
- [ ] Event banner / cover image
|
||||||
- [ ] Chunked resumable upload for files > 100 MB
|
- [ ] Chunked resumable upload for files > 100 MB
|
||||||
- [ ] Shared Tailwind config between main app and export-viewer
|
- [ ] Shared Tailwind config between main app and export-viewer
|
||||||
|
|||||||
@@ -105,7 +105,20 @@
|
|||||||
kind: 'bool',
|
kind: 'bool',
|
||||||
hint: 'Reserviert für künftige Anzahl-Limits.'
|
hint: 'Reserviert für künftige Anzahl-Limits.'
|
||||||
},
|
},
|
||||||
{ key: 'quota_tolerance', label: 'Toleranz (0–1)', kind: 'number' },
|
{
|
||||||
|
key: 'quota_tolerance',
|
||||||
|
label: 'Speicher-Anteil für Gäste (0–1)',
|
||||||
|
kind: 'number',
|
||||||
|
// "Toleranz (0–1)" with no hint invited exactly the wrong reading — that a higher
|
||||||
|
// number means "warn me later". It is the multiplier in
|
||||||
|
// `floor(freier Speicher × Anteil / aktive Uploader)`, so raising it authorises
|
||||||
|
// guests to fill MORE of the disk, not less.
|
||||||
|
hint:
|
||||||
|
'Anteil des freien Speichers, den alle Gäste zusammen belegen dürfen: ' +
|
||||||
|
'Limit = freier Speicher × Anteil ÷ aktive Uploader. Kein Warnschwellenwert — ' +
|
||||||
|
'ein höherer Wert gibt MEHR Speicher frei. Das Keepsake braucht zusätzlich ' +
|
||||||
|
'etwa das Doppelte der Mediengröße; 0,75 ist der getestete Standard.'
|
||||||
|
},
|
||||||
{ key: 'estimated_guest_count', label: 'Geschätzte Gästezahl', kind: 'number' }
|
{ key: 'estimated_guest_count', label: 'Geschätzte Gästezahl', kind: 'number' }
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|||||||
Reference in New Issue
Block a user