Compare commits
13 Commits
feat/low-d
...
backup/pre
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4a9cf878ea | ||
|
|
1e5953a566 | ||
|
|
50d1b5b06d | ||
|
|
bac30404e3 | ||
|
|
5d0c7cd949 | ||
|
|
a20b96d893 | ||
|
|
24ac862f81 | ||
|
|
a3d8ae72e3 | ||
|
|
e1653cc54e | ||
|
|
35390800c7 | ||
|
|
14ebe1e543 | ||
|
|
a4a4e46c53 | ||
|
|
e6e8a52d87 |
24
.env.example
24
.env.example
@@ -18,6 +18,10 @@ POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
|
||||
POSTGRES_DB=eventsnap
|
||||
# Connection pool size. Default 10. For a busy event (~100 guests polling the feed
|
||||
# + SSE + uploads at once) raise to ~30 so requests don't queue on a pool permit.
|
||||
# PAIRED WITH THE DB CONTAINER'S MEMORY LIMIT: 30 backends plus Postgres 16's default
|
||||
# shared_buffers is already snug in the 1G that docker-compose.yml allots the `db`
|
||||
# service. If you raise this, raise `db.deploy.resources.limits.memory` with it — an
|
||||
# OOM in Postgres doesn't degrade one feature, it takes the whole event down.
|
||||
DATABASE_MAX_CONNECTIONS=30
|
||||
|
||||
# ── Authentication ────────────────────────────────────────────────────────────
|
||||
@@ -54,8 +58,26 @@ EXPORT_PATH=/exports
|
||||
# max image size 20 MB
|
||||
# max video size 500 MB
|
||||
# estimated guests 100
|
||||
# quota tolerance 0.75 (fraction of disk that triggers the low-storage warning)
|
||||
# quota tolerance 0.75 (see below — NOT a warning threshold)
|
||||
# Adjust these in the admin UI before the event if needed.
|
||||
#
|
||||
# quota_tolerance is the MULTIPLIER IN THE PER-USER QUOTA FORMULA, not the point at
|
||||
# which anything warns you:
|
||||
#
|
||||
# per_user_limit = floor(free_disk * quota_tolerance / active_uploaders)
|
||||
#
|
||||
# It is recomputed against LIVE free space on every upload, so it self-throttles: guests
|
||||
# converge on a fixed point at tolerance/(1+tolerance) of the free space you started
|
||||
# with — 43% at 0.75, i.e. ~30 GB of a fresh 70 GB.
|
||||
#
|
||||
# Raising it therefore AUTHORISES GUESTS TO FILL MORE OF THE DISK. Setting 0.95 in the
|
||||
# belief that it means "warn me later" moves the fixed point to ~49% and eats the
|
||||
# headroom the keepsake needs — and the keepsake needs a lot, because Gallery.zip and
|
||||
# Memories.zip are each roughly a second copy of every original (both store media
|
||||
# uncompressed). Budget for media + 2x media, or move exports to their own volume.
|
||||
#
|
||||
# 0.75 is the tested default. Lower it if the box is tight; raise it only if you have
|
||||
# provisioned export headroom separately.
|
||||
|
||||
# ── Workers ───────────────────────────────────────────────────────────────────
|
||||
# Number of parallel image/video compression workers. Default 2. This is the main
|
||||
|
||||
125
README.md
125
README.md
@@ -34,7 +34,6 @@ A guest scans the QR code on their way in, types their name, and is immediately
|
||||
### Planned (v1.x)
|
||||
|
||||
- Individual file download button
|
||||
- Low-disk alert (< 10 GB free)
|
||||
- Event banner / cover image
|
||||
- Chunked resumable upload for large videos
|
||||
- Host-curated story highlights
|
||||
@@ -231,6 +230,45 @@ so a host takedown or a ban actually revokes access to the bytes.
|
||||
|
||||
---
|
||||
|
||||
## Sizing the disk
|
||||
|
||||
`postgres_data`, `media_data` and `exports_data` are all Docker named volumes under
|
||||
`/var/lib/docker/volumes`, so **they share one filesystem**. Filling it does not
|
||||
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
||||
down.
|
||||
|
||||
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
|
||||
active_uploaders` is recomputed against live free space on every upload, so guests
|
||||
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
|
||||
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
|
||||
the OS and images, media settles at ~30 GB and stops.
|
||||
|
||||
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
|
||||
`Memories.zip` are built concurrently and each is roughly a second copy of every
|
||||
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
|
||||
untouched original for every video and for every image at or under 5 MB. So a release
|
||||
wants room for **two more copies of the gallery** on top of the gallery itself.
|
||||
|
||||
| Stage | Used | Free (80 GB box) |
|
||||
|---|---|---|
|
||||
| Fresh box (OS + images) | ~10 GB | ~70 GB |
|
||||
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
|
||||
| Host releases → both archives | ~100 GB | **ENOSPC** |
|
||||
|
||||
Two ways to size for it:
|
||||
|
||||
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
||||
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
||||
size that one at ~2× expected media.
|
||||
|
||||
This is no longer silent. The export refuses up front with the two numbers rather than
|
||||
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
|
||||
generation before it starts (so peak is one generation, not two), and the host
|
||||
dashboard warns as soon as the keepsake would not fit — which is the only point at
|
||||
which anyone can still do something about it.
|
||||
|
||||
---
|
||||
|
||||
## Backup
|
||||
|
||||
There are **three** things to back up, and they live in three different places.
|
||||
@@ -242,9 +280,12 @@ never exported into an operator's shell — so every command below runs through
|
||||
```bash
|
||||
# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no
|
||||
# postgres client), reading credentials from the compose environment.
|
||||
# --clean --if-exists makes the dump SELF-CLEANING: without it the restore below
|
||||
# aborts on the first "already exists" against a database that has ever booted,
|
||||
# which is every database you would actually want to restore over.
|
||||
mkdir -p ./backups
|
||||
docker compose exec -T db \
|
||||
sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
||||
sh -c 'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
||||
| gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz
|
||||
|
||||
# 2. Uploaded media (originals + derivatives) out of the named volume.
|
||||
@@ -261,7 +302,7 @@ docker run --rm \
|
||||
-v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \
|
||||
alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src .
|
||||
|
||||
# Weekly offsite sync of the three artefacts above.
|
||||
# Offsite sync of the three artefacts above.
|
||||
rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/
|
||||
```
|
||||
|
||||
@@ -274,6 +315,82 @@ from a directory called `eventsnap`. Confirm yours with `docker volume ls`.
|
||||
> stops it being reachable except through the ticket-gated download handler.
|
||||
> Backing up only the media volume therefore loses every generated keepsake.
|
||||
|
||||
### When to run it
|
||||
|
||||
**A nightly cron is the wrong shape for this app.** Every irreplaceable byte is
|
||||
created inside one eight-hour window, and nobody can retake a wedding. Run the three
|
||||
commands above:
|
||||
|
||||
1. **The night of the event**, once uploads have stopped. This is the backup that
|
||||
matters; everything else is a formality.
|
||||
2. **After the host releases the gallery**, so the generated keepsake is captured too.
|
||||
3. Weekly thereafter, until the event is archived and torn down.
|
||||
|
||||
Take the DB dump and the media tarball **back to back**, without uploads in flight
|
||||
between them. Upload rows reference files by path — a database from 22:00 and a media
|
||||
volume from 23:00 gives you rows pointing at files the dump doesn't know about, and
|
||||
rows whose files aren't in the tarball. Locking uploads from the host dashboard first
|
||||
(**Uploads sperren**) makes the pair genuinely consistent.
|
||||
|
||||
---
|
||||
|
||||
## Restore
|
||||
|
||||
An untested backup is not a backup. Run this once against a scratch host **before**
|
||||
the event — it is roughly ten minutes, and it is the only way to find out that your
|
||||
tarball is empty or your dump is truncated while that is still a small problem.
|
||||
|
||||
```bash
|
||||
# 0. Stop the app FIRST. Migrations run on boot and a live pool will fight the
|
||||
# restore — a booting app against a half-restored schema can leave the migration
|
||||
# table and the schema disagreeing, which is its own recovery problem.
|
||||
# Leave `db` running: the dump is restored through it.
|
||||
docker compose stop app caddy
|
||||
|
||||
# 1. Database. The dump carries its own DROPs (step 1 of Backup), so this replaces
|
||||
# rather than collides. A dump taken WITHOUT --clean --if-exists will abort here
|
||||
# on the first "already exists" — restore that one into a fresh empty database
|
||||
# instead.
|
||||
gunzip -c ./backups/db_2026-07-29.sql.gz \
|
||||
| docker compose exec -T db \
|
||||
sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" --set ON_ERROR_STOP=1'
|
||||
|
||||
# 2. Media. NOTE the `--numeric-owner` and the chown: the app runs as a
|
||||
# NON-ROOT user (uid 100, gid 101 — `addgroup -S app && adduser -S app`), and a
|
||||
# restore that lands root-owned files makes every upload fail with EACCES deep in
|
||||
# the write path, surfacing to the guest as a generic 500 with nothing in the UI
|
||||
# to suggest permissions. The explicit chown is what guarantees it — BusyBox tar
|
||||
# (which is what `alpine` ships) has no --same-owner, and restores ownership only
|
||||
# because it runs as root here.
|
||||
docker run --rm \
|
||||
-v eventsnap_media_data:/dst -v "$PWD/backups":/backup:ro \
|
||||
alpine sh -c 'tar xzf /backup/media_2026-07-29.tar.gz -C /dst \
|
||||
--numeric-owner && chown -R 100:101 /dst'
|
||||
|
||||
# 3. Exports. Same volume-name caveat, same ownership rules.
|
||||
docker run --rm \
|
||||
-v eventsnap_exports_data:/dst -v "$PWD/backups":/backup:ro \
|
||||
alpine sh -c 'tar xzf /backup/exports_2026-07-29.tar.gz -C /dst \
|
||||
--numeric-owner && chown -R 100:101 /dst'
|
||||
|
||||
# 4. Back up. Migrations run, then export recovery re-arms any keepsake whose file
|
||||
# didn't come back with the volume.
|
||||
docker compose up -d app caddy
|
||||
docker compose logs -f app # watch for "migrations applied"
|
||||
|
||||
# 5. Verify — all three, not just the first.
|
||||
curl -fsS https://DOMAIN/health && echo # → ok
|
||||
# … then sign in as host and confirm the feed renders images (proves the media
|
||||
# volume restored AND is readable by uid 100), and that the keepsake downloads.
|
||||
```
|
||||
|
||||
If the media volume restored but images 404 while the feed lists them, the paths are
|
||||
there and the bytes aren't — check `docker compose exec app ls -ln /media/originals`
|
||||
and confirm both the files and the `100:101` ownership.
|
||||
|
||||
The restore is deliberately **not** automated. It is rare, destructive, and the one
|
||||
operation where a script that half-works is worse than a checklist someone reads.
|
||||
|
||||
---
|
||||
|
||||
## Running the backend test suite
|
||||
@@ -348,7 +465,7 @@ Open:
|
||||
- [ ] SSE delta-fetch on foreground reconnect (scaffolded in [sse.ts](frontend/src/lib/sse.ts), not wired)
|
||||
- [ ] Live diashow / slideshow mode — see [docs/CONCEPT_DIASHOW.md](docs/CONCEPT_DIASHOW.md)
|
||||
- [ ] Individual file download button per post
|
||||
- [ ] Low-disk alert (< 10 GB free)
|
||||
- [x] Low-disk alert — host dashboard warns below 10 GB free, or whenever the keepsake would not fit
|
||||
- [ ] Event banner / cover image
|
||||
- [ ] Chunked resumable upload for files > 100 MB
|
||||
- [ ] Shared Tailwind config between main app and export-viewer
|
||||
|
||||
1
backend/migrations/020_social_rate.down.sql
Normal file
1
backend/migrations/020_social_rate.down.sql
Normal file
@@ -0,0 +1 @@
|
||||
DELETE FROM config WHERE key IN ('social_rate_per_min', 'social_rate_enabled');
|
||||
16
backend/migrations/020_social_rate.up.sql
Normal file
16
backend/migrations/020_social_rate.up.sql
Normal file
@@ -0,0 +1,16 @@
|
||||
-- Per-user rate limit for social writes (likes, comments, comment deletions).
|
||||
--
|
||||
-- These were the only writes in the app with no limit at all. Every other mutating
|
||||
-- path -- upload, join, recover, export, admin login -- carries one; social.rs
|
||||
-- carried none, so the coverage was asymmetric rather than deliberately open.
|
||||
--
|
||||
-- Severity is genuinely low for an invited-guest event, and the amplification worry
|
||||
-- turned out to be contained: a like fans an SSE broadcast to ~100 clients, but the
|
||||
-- export regeneration it could otherwise trigger is debounced (REGEN_DEBOUNCE 20s)
|
||||
-- and superseded workers are inert. So this closes the gap for symmetry, not urgency,
|
||||
-- and the ceiling is set high enough that no real guest will ever meet it -- a
|
||||
-- double-tapping enthusiast at a wedding is not the thing being defended against.
|
||||
INSERT INTO config (key, value) VALUES
|
||||
('social_rate_per_min', '120'),
|
||||
('social_rate_enabled', 'true')
|
||||
ON CONFLICT (key) DO NOTHING;
|
||||
@@ -127,6 +127,9 @@ pub async fn patch_config(
|
||||
// Same shape for /recover: the per-(ip, name) bucket is the anti-guessing control,
|
||||
// this only bounds a name-cycling flood in front of a cost-12 bcrypt (migration 019).
|
||||
("recover_ip_rate_per_min", true, 1.0, 100_000.0),
|
||||
// Aggregate ceiling on likes + comments + comment deletions, per user per minute.
|
||||
// These were the only mutating endpoints with no limit at all (migration 020).
|
||||
("social_rate_per_min", true, 1.0, 100_000.0),
|
||||
("quota_tolerance", false, 0.0, 1.0),
|
||||
("estimated_guest_count", true, 1.0, 1_000_000.0),
|
||||
];
|
||||
@@ -141,6 +144,7 @@ pub async fn patch_config(
|
||||
// missing from this allowlist — so the switch existed in code and could never be flipped.
|
||||
"admin_login_rate_enabled",
|
||||
"recover_rate_enabled",
|
||||
"social_rate_enabled",
|
||||
"quota_enabled",
|
||||
"storage_quota_enabled",
|
||||
"upload_count_quota_enabled",
|
||||
@@ -193,6 +197,23 @@ pub async fn patch_config(
|
||||
"Wert für {key} liegt außerhalb des zulässigen Bereichs ({min}–{max})."
|
||||
)));
|
||||
}
|
||||
// Zero is in range and catastrophic. `quota_tolerance` is the multiplier in
|
||||
// `free_disk * tolerance / active_uploaders`, so 0 makes every per-user limit 0 and
|
||||
// refuses EVERY upload — mid-event, with "Du hast dein Upload-Limit für dieses Event
|
||||
// erreicht", an error naming the wrong cause entirely. `storage_quota_enabled` is the
|
||||
// intended off-switch.
|
||||
//
|
||||
// Rejecting the value rather than raising the floor: very small tolerances are
|
||||
// legitimate (they are how a large disk is throttled down to a sensible per-guest
|
||||
// ceiling, and how the e2e quota tests steer it — around 1e-5 on a 174 GB volume), so
|
||||
// a floor of, say, 0.01 would forbid real configurations to prevent one typo.
|
||||
if key_str == "quota_tolerance" && n == 0.0 {
|
||||
return Err(AppError::BadRequest(
|
||||
"quota_tolerance = 0 würde jeden Upload blockieren. Zum Abschalten der \
|
||||
Speicher-Quote stattdessen „Speicher-Quote aktiv“ ausschalten."
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
} else if BOOL_KEYS.contains(&key_str) {
|
||||
match value.trim().to_ascii_lowercase().as_str() {
|
||||
"true" | "false" | "1" | "0" | "yes" | "no" | "on" | "off" => {}
|
||||
|
||||
@@ -10,8 +10,40 @@ use crate::error::AppError;
|
||||
use crate::models::comment::{Comment, CommentDto};
|
||||
use crate::models::hashtag::{self, Hashtag};
|
||||
use crate::models::upload::Upload;
|
||||
use crate::services::config;
|
||||
use crate::state::AppState;
|
||||
|
||||
/// Throttle a social write. Keyed PER USER, like the feed and upload limits and for the same
|
||||
/// reason: at a venue every guest sits behind one NAT, so an IP key hands the whole party a
|
||||
/// single bucket and the most active guest starves everyone else.
|
||||
///
|
||||
/// These were the only mutating endpoints in the app with no limit at all — the coverage was
|
||||
/// asymmetric, not deliberately open. The ceiling is set well above anything a real guest
|
||||
/// produces; this bounds a script, not an enthusiastic double-tapper.
|
||||
async fn check_social_rate(state: &AppState, user_id: Uuid) -> Result<(), AppError> {
|
||||
let rate_limits_on = config::get_bool(&state.config_cache, "rate_limits_enabled", true).await;
|
||||
let social_rate_on = config::get_bool(&state.config_cache, "social_rate_enabled", true).await;
|
||||
if !(rate_limits_on && social_rate_on) {
|
||||
return Ok(());
|
||||
}
|
||||
let rate_limit = config::get_usize(&state.config_cache, "social_rate_per_min", 120).await;
|
||||
// ONE bucket across likes, comments and comment deletions. Separate buckets would let a
|
||||
// caller triple the aggregate write rate just by alternating between them.
|
||||
state
|
||||
.rate_limiter
|
||||
.check_with_retry(
|
||||
format!("social:{user_id}"),
|
||||
rate_limit,
|
||||
std::time::Duration::from_secs(60),
|
||||
)
|
||||
.map_err(|retry_after_secs| {
|
||||
AppError::TooManyRequests(
|
||||
"Zu viele Aktionen. Bitte warte kurz und versuche es erneut.".into(),
|
||||
Some(retry_after_secs),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct LikeResponse {
|
||||
/// The caller's like state *after* this toggle. The client sets `liked_by_me` from
|
||||
@@ -35,6 +67,7 @@ pub async fn toggle_like(
|
||||
if user.is_banned {
|
||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||
}
|
||||
check_social_rate(&state, auth.user_id).await?;
|
||||
|
||||
// Event-scope: the upload must belong to the caller's event (404 otherwise),
|
||||
// matching the host handlers' find_by_id_and_event pattern.
|
||||
@@ -141,6 +174,7 @@ pub async fn add_comment(
|
||||
if user.is_banned {
|
||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||
}
|
||||
check_social_rate(&state, auth.user_id).await?;
|
||||
|
||||
// Event-scope: only comment on an upload that belongs to the caller's event.
|
||||
Upload::find_by_id_and_event(&state.pool, upload_id, auth.event_id)
|
||||
@@ -216,6 +250,7 @@ pub async fn delete_comment(
|
||||
if auth.is_banned {
|
||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||
}
|
||||
check_social_rate(&state, auth.user_id).await?;
|
||||
let comment = Comment::find_by_id(&state.pool, comment_id)
|
||||
.await?
|
||||
.ok_or_else(|| AppError::NotFound("Kommentar nicht gefunden.".into()))?;
|
||||
|
||||
@@ -63,6 +63,7 @@ pub async fn truncate_all(
|
||||
('export_rate_per_day', '3'),
|
||||
('join_ip_rate_per_min', '60'),
|
||||
('recover_ip_rate_per_min', '30'),
|
||||
('social_rate_per_min', '120'),
|
||||
('quota_tolerance', '0.75'),
|
||||
('estimated_guest_count', '100'),
|
||||
('compression_concurrency', '2'),
|
||||
@@ -71,6 +72,7 @@ pub async fn truncate_all(
|
||||
('feed_rate_enabled', 'false'),
|
||||
('export_rate_enabled', 'false'),
|
||||
('join_rate_enabled', 'false'),
|
||||
('social_rate_enabled', 'false'),
|
||||
('admin_login_rate_enabled', 'false'),
|
||||
('quota_enabled', 'false'),
|
||||
('storage_quota_enabled', 'false'),
|
||||
|
||||
@@ -20,6 +20,29 @@ use crate::state::SseEvent;
|
||||
|
||||
static VIEWER_DIR: Dir<'_> = include_dir!("$CARGO_MANIFEST_DIR/static/export-viewer");
|
||||
|
||||
// ── Shared visibility filter ─────────────────────────────────────────────────
|
||||
|
||||
/// The predicate that decides what lands in a keepsake, as ONE definition.
|
||||
///
|
||||
/// Two queries have to agree on it: [`query_uploads`], which selects the rows the archives are
|
||||
/// built from, and [`estimate_export_bytes`], which sizes them for the disk preflight. They used
|
||||
/// to state it separately, and the direction of drift matters — an estimate that misses rows the
|
||||
/// archive writes UNDER-reserves, which is the exact ENOSPC the preflight exists to prevent.
|
||||
///
|
||||
/// A `SRC:`-marked copy in the integration tests cannot catch that: drift means production moved
|
||||
/// and the copy didn't, so both sides of such a test sit still and it keeps passing. Sharing the
|
||||
/// fragment removes the failure by construction instead, and leaves the test doing what it is
|
||||
/// actually good at — pinning the behaviour.
|
||||
///
|
||||
/// CONTRACT: callers must alias `upload` as `u` and join `"user"` as `usr`, and bind the event id
|
||||
/// as `$1`.
|
||||
macro_rules! export_visibility_where {
|
||||
() => {
|
||||
"WHERE u.event_id = $1 AND u.deleted_at IS NULL
|
||||
AND usr.uploads_hidden = FALSE AND usr.is_banned = FALSE"
|
||||
};
|
||||
}
|
||||
|
||||
// ── DB query rows ────────────────────────────────────────────────────────────
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
@@ -557,7 +580,7 @@ async fn run_zip_export_inner(
|
||||
};
|
||||
let entry_name = format!("{folder}/{date}_{name_safe}_{}.{ext}", row.id);
|
||||
|
||||
let builder = ZipEntryBuilder::new(entry_name.into(), Compression::Stored);
|
||||
let builder = keepsake_entry(entry_name, Compression::Stored);
|
||||
|
||||
// Open BEFORE writing the entry header. A missing source is skipped (the media file
|
||||
// was deleted, or its processing failed) — but it must be skipped without aborting the
|
||||
@@ -950,7 +973,7 @@ async fn run_html_export_inner(
|
||||
|
||||
// Write data.json
|
||||
{
|
||||
let builder = ZipEntryBuilder::new("data.json".into(), Compression::Deflate);
|
||||
let builder = keepsake_entry("data.json".into(), Compression::Deflate);
|
||||
let mut entry = zip.write_entry_stream(builder).await?;
|
||||
let mut cursor = AllowStdIo::new(std::io::Cursor::new(data_json.as_bytes()));
|
||||
fcopy(&mut cursor, &mut entry).await?;
|
||||
@@ -959,7 +982,7 @@ async fn run_html_export_inner(
|
||||
|
||||
// Write README.txt
|
||||
{
|
||||
let builder = ZipEntryBuilder::new("README.txt".into(), Compression::Deflate);
|
||||
let builder = keepsake_entry("README.txt".into(), Compression::Deflate);
|
||||
let mut entry = zip.write_entry_stream(builder).await?;
|
||||
let mut cursor = AllowStdIo::new(std::io::Cursor::new(README_TEXT.as_bytes()));
|
||||
fcopy(&mut cursor, &mut entry).await?;
|
||||
@@ -992,7 +1015,7 @@ async fn run_html_export_inner(
|
||||
};
|
||||
|
||||
let entry_name = format!("media/{name}");
|
||||
let builder = ZipEntryBuilder::new(entry_name.into(), Compression::Stored);
|
||||
let builder = keepsake_entry(entry_name, Compression::Stored);
|
||||
let mut zip_entry = zip.write_entry_stream(builder).await?;
|
||||
let mut f = src_file.compat();
|
||||
fcopy(&mut f, &mut zip_entry).await?;
|
||||
@@ -1051,7 +1074,7 @@ async fn run_html_export_inner(
|
||||
// ── DB helpers ───────────────────────────────────────────────────────────────
|
||||
|
||||
async fn query_uploads(pool: &PgPool, event_id: Uuid) -> Result<Vec<ExportUploadRow>> {
|
||||
Ok(sqlx::query_as::<_, ExportUploadRow>(
|
||||
Ok(sqlx::query_as::<_, ExportUploadRow>(concat!(
|
||||
"SELECT u.id, u.original_path, u.mime_type, u.caption,
|
||||
usr.display_name AS uploader_name,
|
||||
COUNT(DISTINCT l.user_id) AS like_count,
|
||||
@@ -1059,11 +1082,12 @@ async fn query_uploads(pool: &PgPool, event_id: Uuid) -> Result<Vec<ExportUpload
|
||||
FROM upload u
|
||||
JOIN \"user\" usr ON usr.id = u.user_id
|
||||
LEFT JOIN \"like\" l ON l.upload_id = u.id
|
||||
WHERE u.event_id = $1 AND u.deleted_at IS NULL
|
||||
AND usr.uploads_hidden = FALSE AND usr.is_banned = FALSE
|
||||
",
|
||||
export_visibility_where!(),
|
||||
"
|
||||
GROUP BY u.id, usr.display_name
|
||||
ORDER BY u.created_at ASC",
|
||||
)
|
||||
))
|
||||
.bind(event_id)
|
||||
.fetch_all(pool)
|
||||
.await?)
|
||||
@@ -1267,15 +1291,16 @@ fn is_superseded_archive(
|
||||
/// downscaled to 2000px first, which only makes this estimate more conservative — the direction we
|
||||
/// want, since being wrong low means ENOSPC halfway through.
|
||||
///
|
||||
/// Matches [`query_uploads`]' visibility filter exactly, so hidden/banned uploads aren't counted.
|
||||
/// Shares [`query_uploads`]' visibility filter via [`export_visibility_where`], so hidden/banned
|
||||
/// uploads can't be counted here but skipped there (or the reverse, which under-reserves).
|
||||
pub async fn estimate_export_bytes(pool: &PgPool, event_id: Uuid) -> Result<u64> {
|
||||
let (bytes,): (i64,) = sqlx::query_as(
|
||||
let (bytes,): (i64,) = sqlx::query_as(concat!(
|
||||
"SELECT COALESCE(SUM(u.original_size_bytes), 0)::bigint
|
||||
FROM upload u
|
||||
JOIN \"user\" usr ON usr.id = u.user_id
|
||||
WHERE u.event_id = $1 AND u.deleted_at IS NULL
|
||||
AND usr.uploads_hidden = FALSE AND usr.is_banned = FALSE",
|
||||
)
|
||||
",
|
||||
export_visibility_where!(),
|
||||
))
|
||||
.bind(event_id)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
@@ -1527,6 +1552,36 @@ async fn maybe_broadcast_complete(
|
||||
/// double-clicking `index.html` (file://), where browsers block a cross-origin
|
||||
/// `fetch()` of a sibling `data.json` — so the data is inlined into the page.
|
||||
/// (`data.json` is still written separately for the http-served case.)
|
||||
/// Permissions stamped on every entry in both archives: `rw-r--r--`.
|
||||
///
|
||||
/// `ZipEntryBuilder::new` leaves the external file attribute at zero, and the host compatibility
|
||||
/// defaults to Unix — so every entry was written with a stored mode of **0000**. Windows Explorer
|
||||
/// ignores Unix modes and was fine, which is exactly why this survived: on Linux and macOS
|
||||
/// `unzip` faithfully applies what the archive asks for, and the guest gets a directory of files
|
||||
/// none of which they can open. `?---------` on every line of `unzip -Z`.
|
||||
///
|
||||
/// That is the keepsake — the artifact the whole event exists to produce — arriving unreadable,
|
||||
/// after distribution, with no server-side symptom at all.
|
||||
/// `S_IFREG | 0644`. The type bits are included because the mode is written whole into the high
|
||||
/// half of the external file attribute: without them extractors see a file of type "unknown"
|
||||
/// (`unzip -Z` renders `?rw-r--r--`), which works but is not what the archive means to say.
|
||||
const KEEPSAKE_ENTRY_MODE: u16 = 0o100_644;
|
||||
|
||||
/// Build a ZIP entry for the keepsake. ALL entries in both archives go through here so the mode
|
||||
/// can't be forgotten at one of the six call sites.
|
||||
fn keepsake_entry(name: String, compression: Compression) -> ZipEntryBuilder {
|
||||
ZipEntryBuilder::new(name.into(), compression).unix_permissions(KEEPSAKE_ENTRY_MODE)
|
||||
}
|
||||
|
||||
/// Escape a JSON payload for inlining inside a `<script>` element.
|
||||
///
|
||||
/// See the call site in [`write_viewer_with_data`] for why this is every `<` and not just `</`.
|
||||
/// Kept separate so the property that matters — no `<` survives, and the value still decodes to
|
||||
/// the original — can be asserted without building a ZIP.
|
||||
fn escape_json_for_script(data_json: &str) -> String {
|
||||
data_json.replace('<', "\\u003c")
|
||||
}
|
||||
|
||||
async fn write_viewer_with_data(
|
||||
dir: &include_dir::Dir<'_>,
|
||||
zip: &mut ZipFileWriter<tokio::fs::File>,
|
||||
@@ -1538,8 +1593,31 @@ async fn write_viewer_with_data(
|
||||
if path == "index.html" {
|
||||
let html = std::str::from_utf8(file.contents())
|
||||
.context("export-viewer index.html is not valid UTF-8")?;
|
||||
// Escape `</` so a caption containing `</script>` can't break out of the tag.
|
||||
let safe = data_json.replace("</", "<\\/");
|
||||
// Escape EVERY `<`, not just `</`.
|
||||
//
|
||||
// `</` -> `<\/` stops the obvious break-out (`</script><img onerror=…>`) and is inert
|
||||
// against XSS. It does not stop the caption steering the HTML TOKENIZER. A caption
|
||||
// containing `<!--<script` with no later `-->` puts the parser into
|
||||
// script-data-double-escaped state; from there the template's own `</script>` only
|
||||
// steps back to script-data-escaped instead of closing the element, and the rest of the
|
||||
// document — including the viewer bundle — is swallowed as script data. Nothing
|
||||
// executes and nothing leaks; `window.__EXPORT_DATA__` is simply never assigned and the
|
||||
// keepsake opens blank.
|
||||
//
|
||||
// That failure is silent and POST-DISTRIBUTION: the export succeeds, the ZIP is
|
||||
// well-formed, the job writes `done`, /export/status is green, and the host hands out a
|
||||
// file that only fails when a guest double-clicks index.html — in every copy, with no
|
||||
// way to fix it after the fact. Reachable from any guest-authored caption or comment,
|
||||
// since both are embedded in the viewer.
|
||||
//
|
||||
// `<` never appears in JSON structural syntax — only inside string values — so a global
|
||||
// replace is sound, and `<` is valid in both JSON and a JS string literal. One
|
||||
// rule covers `</script`, `<!--` and `<script` together, which is the point: the
|
||||
// previous escape was named for the single case it did handle.
|
||||
//
|
||||
// NOTE this is deliberately only for the INLINED copy. `data.json` is written
|
||||
// separately, in no HTML context, and must stay literal.
|
||||
let safe = escape_json_for_script(data_json);
|
||||
// Match the live app's colour theme: inject the same `:root:root{…}` override
|
||||
// the app builds at runtime so a rose/sage/custom event exports a rose/sage
|
||||
// keepsake (not the embedded default gold). The CSS is generated purely from
|
||||
@@ -1553,13 +1631,13 @@ async fn write_viewer_with_data(
|
||||
Some(idx) => format!("{}{}{}", &html[..idx], head_inject, &html[idx..]),
|
||||
None => format!("{head_inject}{html}"),
|
||||
};
|
||||
let builder = ZipEntryBuilder::new(path.into(), Compression::Deflate);
|
||||
let builder = keepsake_entry(path, Compression::Deflate);
|
||||
let mut entry = zip.write_entry_stream(builder).await?;
|
||||
let mut cursor = AllowStdIo::new(std::io::Cursor::new(injected.as_bytes()));
|
||||
fcopy(&mut cursor, &mut entry).await?;
|
||||
entry.close().await?;
|
||||
} else {
|
||||
let builder = ZipEntryBuilder::new(path.into(), Compression::Deflate);
|
||||
let builder = keepsake_entry(path, Compression::Deflate);
|
||||
let mut entry = zip.write_entry_stream(builder).await?;
|
||||
let mut cursor = AllowStdIo::new(std::io::Cursor::new(file.contents()));
|
||||
fcopy(&mut cursor, &mut entry).await?;
|
||||
@@ -1790,6 +1868,62 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
/// Every `<` is escaped, whatever it is part of.
|
||||
///
|
||||
/// PREVENTS the regression to `</` -> `<\\/`, which is named for the one case it handles.
|
||||
/// `<!--<script` with no later `-->` drives the HTML tokenizer into
|
||||
/// script-data-double-escaped state, where the template's own `</script>` no longer closes
|
||||
/// the element — the viewer bundle is swallowed as script data, `__EXPORT_DATA__` is never
|
||||
/// assigned, and the keepsake opens blank in every copy the host has already handed out.
|
||||
#[test]
|
||||
fn no_left_angle_bracket_survives_inlining() {
|
||||
for payload in [
|
||||
r#"{"caption":"<!--<script"}"#,
|
||||
r#"{"caption":"</script><img src=x onerror=alert(1)>"}"#,
|
||||
r#"{"caption":"<!--"}"#,
|
||||
r#"{"caption":"<script>"}"#,
|
||||
r#"{"caption":"a < b"}"#,
|
||||
] {
|
||||
let escaped = escape_json_for_script(payload);
|
||||
assert!(
|
||||
!escaped.contains('<'),
|
||||
"a surviving `<` can still steer the tokenizer: {escaped}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// The escape must not change what the viewer READS — it is a transport encoding, not a
|
||||
/// sanitiser. A caption is guest-authored text that has to render back exactly.
|
||||
#[test]
|
||||
fn the_payload_still_decodes_to_the_original_value() {
|
||||
// `<` appears only inside JSON string values, never in structural syntax, so a global
|
||||
// replace is sound — this is the assertion that says so.
|
||||
for caption in [
|
||||
"<!--<script",
|
||||
"</script><img src=x onerror=alert(1)>",
|
||||
"a < b und c > d",
|
||||
"ganz normale Bildunterschrift",
|
||||
"Herz <3",
|
||||
] {
|
||||
let json = serde_json::json!({ "posts": [{ "caption": caption }] }).to_string();
|
||||
let escaped = escape_json_for_script(&json);
|
||||
let back: serde_json::Value =
|
||||
serde_json::from_str(&escaped).expect("the escaped form must still be valid JSON");
|
||||
assert_eq!(
|
||||
back["posts"][0]["caption"].as_str(),
|
||||
Some(caption),
|
||||
"the caption must survive the round trip unchanged"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Nothing else in the document is touched.
|
||||
#[test]
|
||||
fn a_payload_with_no_angle_brackets_is_unchanged() {
|
||||
let json = r#"{"posts":[{"caption":"schönes Foto"}]}"#;
|
||||
assert_eq!(escape_json_for_script(json), json);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_lone_armed_job_reserves_for_one_archive() {
|
||||
// A ViewerOnly regeneration re-arms only the HTML half — reserving for two would refuse
|
||||
|
||||
@@ -293,6 +293,10 @@ pub async fn set_user_moderation(pool: &PgPool, user_id: Uuid, banned: bool, hid
|
||||
|
||||
/// SRC: `services/export.rs::query_uploads` — the visibility filter, verbatim, projected down to
|
||||
/// `(id, original_size_bytes)`. This is the row set that ACTUALLY lands in the archives.
|
||||
///
|
||||
/// Production builds this WHERE from `export_visibility_where!()`, shared with
|
||||
/// `estimate_export_bytes`. A copy here can pin the behaviour but CANNOT detect production moving
|
||||
/// away from it — that is what sharing the fragment is for, not this.
|
||||
pub async fn export_visible_uploads(pool: &PgPool, event_id: Uuid) -> Vec<(Uuid, i64)> {
|
||||
sqlx::query_as(
|
||||
"SELECT u.id, u.original_size_bytes
|
||||
@@ -309,7 +313,9 @@ pub async fn export_visible_uploads(pool: &PgPool, event_id: Uuid) -> Vec<(Uuid,
|
||||
.expect("export_visible_uploads")
|
||||
}
|
||||
|
||||
/// SRC: `services/export.rs::estimate_export_bytes` — verbatim.
|
||||
/// SRC: `services/export.rs::estimate_export_bytes` — verbatim. Same caveat as above: production
|
||||
/// shares its WHERE with `query_uploads` via `export_visibility_where!()`, so these two copies
|
||||
/// agreeing proves the behaviour, not the absence of drift.
|
||||
pub async fn estimate_export_bytes(pool: &PgPool, event_id: Uuid) -> i64 {
|
||||
let (bytes,): (i64,) = sqlx::query_as(
|
||||
"SELECT COALESCE(SUM(u.original_size_bytes), 0)::bigint
|
||||
|
||||
@@ -16,10 +16,18 @@
|
||||
//!
|
||||
//! What these tests pin is the ESTIMATE — the part that decides. The arithmetic on top of it lives
|
||||
//! in `services/export.rs`'s unit tests; the filesystem selection lives in `is_superseded_archive`.
|
||||
//! The risk here is drift: if `query_uploads` ever gains or loses a visibility predicate and
|
||||
//! `estimate_export_bytes` doesn't, the preflight silently sizes the wrong gallery. So rather than
|
||||
//! restating the filter, these assert the estimate against the row set the archive actually
|
||||
//! contains.
|
||||
//!
|
||||
//! ON DRIFT, precisely, because it is easy to overclaim here. The hazard is that `query_uploads`
|
||||
//! (which selects the rows the archives are built from) and `estimate_export_bytes` (which sizes
|
||||
//! them) could disagree — and an estimate missing rows the archive writes UNDER-reserves, the one
|
||||
//! direction that reintroduces the ENOSPC. **These tests cannot catch that**, and neither can any
|
||||
//! test in this harness: both sides here are `SRC:`-marked hand-copies in `tests/common/mod.rs`,
|
||||
//! so if production moved and the copies didn't, they would sit still and keep passing.
|
||||
//!
|
||||
//! That is fixed where it can be — the two queries now share one `export_visibility_where!()`
|
||||
//! fragment in `services/export.rs`, so they cannot diverge by construction. What is left for
|
||||
//! these tests is what the convention is genuinely good at: pinning the BEHAVIOUR, so a change
|
||||
//! that deliberately alters the filter has to come here and say so.
|
||||
|
||||
mod common;
|
||||
|
||||
@@ -29,9 +37,10 @@ use sqlx::PgPool;
|
||||
/// The estimate must equal the sum over EXACTLY the rows `query_uploads` returns — computed from
|
||||
/// that row set, not from a restatement of its WHERE clause.
|
||||
///
|
||||
/// PREVENTS: the two queries drifting apart. An estimate that counts rows the archive skips is
|
||||
/// merely pessimistic; one that MISSES rows the archive writes under-reserves, which is the whole
|
||||
/// failure being guarded against.
|
||||
/// PINS: which uploads the preflight is allowed to count. Each excluded row below is excluded by a
|
||||
/// DIFFERENT predicate, so a change that drops or weakens any one of them fails here and has to be
|
||||
/// argued for. (It does not detect production drifting away from these copies — see the file
|
||||
/// header; `export_visibility_where!()` is what makes that impossible.)
|
||||
#[sqlx::test]
|
||||
async fn the_estimate_sums_exactly_the_rows_the_archive_will_contain(pool: PgPool) {
|
||||
let event_id = seed_event(&pool, "wedding").await;
|
||||
|
||||
@@ -17,7 +17,15 @@ services:
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
# 1G, not 512M. DATABASE_MAX_CONNECTIONS defaults to 30 for a ~100-guest event
|
||||
# (feed polling + SSE + uploads at once), and 30 backends plus Postgres 16's
|
||||
# default shared_buffers leaves very little headroom at 512M. An OOM here does
|
||||
# not degrade one feature — it takes the event down, because every request
|
||||
# path touches the database. Memory is the cheaper knob than shrinking the
|
||||
# pool back and reintroducing the queueing it was raised to fix.
|
||||
#
|
||||
# Raising DATABASE_MAX_CONNECTIONS further means raising this too.
|
||||
memory: 1G
|
||||
|
||||
app:
|
||||
build:
|
||||
|
||||
136
e2e/specs/03-feed/social-rate-limit.spec.ts
Normal file
136
e2e/specs/03-feed/social-rate-limit.spec.ts
Normal file
@@ -0,0 +1,136 @@
|
||||
/**
|
||||
* Regression guard — likes, comments and comment deletions are rate limited.
|
||||
*
|
||||
* These were the only mutating endpoints in the app with no limit at all. Every other write path
|
||||
* -- upload, join, recover, export, admin login -- carried one; `social.rs` carried none, so the
|
||||
* coverage was asymmetric rather than deliberately open.
|
||||
*
|
||||
* Severity is genuinely low for an invited-guest event, and the amplification worry is contained:
|
||||
* a like does fan an SSE broadcast to every connected client, but the export regeneration a
|
||||
* comment deletion triggers is debounced (REGEN_DEBOUNCE 20s) and superseded workers are inert. So
|
||||
* this closes the gap for symmetry, and the ceiling is set well above anything a real guest
|
||||
* produces -- it bounds a script, not an enthusiastic double-tapper.
|
||||
*
|
||||
* The bucket is shared across all three actions on purpose: separate buckets would let a caller
|
||||
* triple the aggregate write rate just by alternating between them. That is what the second test
|
||||
* pins, and it is the part most likely to be lost in a refactor.
|
||||
*
|
||||
* Keyed per USER, not per IP — at a venue every guest is behind one NAT, so an IP key would hand
|
||||
* the whole party one bucket. Third test.
|
||||
*/
|
||||
import { test, expect } from '../../fixtures/test';
|
||||
import { seedUpload } from '../../helpers/seed';
|
||||
import { BASE } from '../../helpers/env';
|
||||
|
||||
const like = (jwt: string, uploadId: string) =>
|
||||
fetch(`${BASE}/api/v1/upload/${uploadId}/like`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${jwt}` },
|
||||
});
|
||||
|
||||
const comment = (jwt: string, uploadId: string, body: string) =>
|
||||
fetch(`${BASE}/api/v1/upload/${uploadId}/comments`, {
|
||||
method: 'POST',
|
||||
headers: { Authorization: `Bearer ${jwt}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ body }),
|
||||
});
|
||||
|
||||
test.describe('Social — rate limit', () => {
|
||||
test('a burst of likes past the ceiling returns 429 with Retry-After', async ({
|
||||
api,
|
||||
adminToken,
|
||||
guest,
|
||||
}) => {
|
||||
await api.patchConfig(adminToken, {
|
||||
rate_limits_enabled: 'true',
|
||||
social_rate_enabled: 'true',
|
||||
social_rate_per_min: '3',
|
||||
});
|
||||
|
||||
const g = await guest('Tapper');
|
||||
const uploadId = await seedUpload(g.jwt);
|
||||
|
||||
// Sequential, not parallel: a toggle flips state, so ordering matters for the assertion.
|
||||
const statuses: number[] = [];
|
||||
for (let i = 0; i < 5; i++) statuses.push((await like(g.jwt, uploadId)).status);
|
||||
|
||||
expect(statuses.slice(0, 3), 'the first three are within the ceiling').toEqual([200, 200, 200]);
|
||||
expect(statuses.slice(3), 'everything past it is refused').toEqual([429, 429]);
|
||||
|
||||
const limited = await like(g.jwt, uploadId);
|
||||
expect(limited.status).toBe(429);
|
||||
expect(
|
||||
limited.headers.get('retry-after'),
|
||||
'a 429 without Retry-After tells the client nothing about when to come back'
|
||||
).toBeTruthy();
|
||||
});
|
||||
|
||||
test('likes and comments share one bucket', async ({ api, adminToken, guest }) => {
|
||||
// THE assertion. Per-action buckets would let a caller triple the aggregate write rate by
|
||||
// alternating, which defeats the point of having a ceiling at all.
|
||||
await api.patchConfig(adminToken, {
|
||||
rate_limits_enabled: 'true',
|
||||
social_rate_enabled: 'true',
|
||||
social_rate_per_min: '2',
|
||||
});
|
||||
|
||||
const g = await guest('Mixer');
|
||||
const uploadId = await seedUpload(g.jwt);
|
||||
|
||||
expect((await like(g.jwt, uploadId)).status).toBe(200);
|
||||
expect((await comment(g.jwt, uploadId, 'schön!')).status).toBe(201);
|
||||
// Two writes spent, whichever endpoints they went to.
|
||||
expect(
|
||||
(await comment(g.jwt, uploadId, 'noch eins')).status,
|
||||
'a comment must consume the same budget a like does'
|
||||
).toBe(429);
|
||||
expect((await like(g.jwt, uploadId)).status).toBe(429);
|
||||
});
|
||||
|
||||
test('one guest hitting the ceiling does not block another', async ({
|
||||
api,
|
||||
adminToken,
|
||||
guest,
|
||||
}) => {
|
||||
// Keyed per user, not per IP. Every request in this suite comes from one address, which is
|
||||
// exactly the venue-NAT shape that made the /join and /feed limits turn guests away.
|
||||
await api.patchConfig(adminToken, {
|
||||
rate_limits_enabled: 'true',
|
||||
social_rate_enabled: 'true',
|
||||
social_rate_per_min: '2',
|
||||
});
|
||||
|
||||
const noisy = await guest('Noisy');
|
||||
const quiet = await guest('Quiet');
|
||||
const uploadId = await seedUpload(noisy.jwt);
|
||||
|
||||
for (let i = 0; i < 3; i++) await like(noisy.jwt, uploadId);
|
||||
expect((await like(noisy.jwt, uploadId)).status).toBe(429);
|
||||
|
||||
expect(
|
||||
(await like(quiet.jwt, uploadId)).status,
|
||||
'a second guest behind the same IP must have their own budget'
|
||||
).toBe(200);
|
||||
});
|
||||
|
||||
test('flipping social_rate_enabled off bypasses the limit', async ({
|
||||
api,
|
||||
adminToken,
|
||||
guest,
|
||||
}) => {
|
||||
// The toggle has to actually be honoured, or the admin switch is decorative — the failure
|
||||
// mode two other per-area toggles already shipped with.
|
||||
await api.patchConfig(adminToken, {
|
||||
rate_limits_enabled: 'true',
|
||||
social_rate_enabled: 'false',
|
||||
social_rate_per_min: '2',
|
||||
});
|
||||
|
||||
const g = await guest('Unlimited');
|
||||
const uploadId = await seedUpload(g.jwt);
|
||||
|
||||
const statuses: number[] = [];
|
||||
for (let i = 0; i < 6; i++) statuses.push((await like(g.jwt, uploadId)).status);
|
||||
expect(statuses.every((s) => s === 200)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -40,10 +40,19 @@ test.describe('Video — the lightbox plays it', () => {
|
||||
page,
|
||||
guest,
|
||||
signIn,
|
||||
db,
|
||||
}) => {
|
||||
const g = await guest('VideoWatcher');
|
||||
const id = await seedVideo(g.jwt);
|
||||
|
||||
// The poster assertion below needs the ffmpeg thumbnail to EXIST — the lightbox binds
|
||||
// `poster={upload.thumbnail_url ?? undefined}`, so the attribute is simply absent until
|
||||
// compression finishes. Without this wait the test races the worker and fails against a
|
||||
// cold stack (first run after `stack:down -v`, cold ffmpeg), which is exactly when a suite
|
||||
// is least likely to be believed. The `src` assertion is unconditional; only the poster
|
||||
// needs the wait.
|
||||
await expect.poll(() => db.compressionStatus(id), { timeout: 60_000 }).toBe('done');
|
||||
|
||||
await signIn(page, g);
|
||||
await page.goto('/feed');
|
||||
|
||||
|
||||
@@ -68,6 +68,40 @@ test.describe('Admin — config API', () => {
|
||||
expect(cfg.privacy_note).toBe(note);
|
||||
await api.patchConfig(adminToken, { privacy_note: '' });
|
||||
});
|
||||
test('quota_tolerance = 0 is rejected, with a pointer to the real off-switch', async ({
|
||||
api,
|
||||
adminToken,
|
||||
}) => {
|
||||
// Zero is inside the documented 0–1 range and catastrophic: the per-user limit is
|
||||
// `free_disk * tolerance / active_uploaders`, so 0 refuses EVERY upload — mid-event, with
|
||||
// "Du hast dein Upload-Limit für dieses Event erreicht", which names the wrong cause
|
||||
// entirely. `storage_quota_enabled` is what an admin reaching for an off-switch wants.
|
||||
const res = await fetch(
|
||||
(process.env.E2E_FRONTEND_URL ?? 'http://localhost:3101') + '/api/v1/admin/config',
|
||||
{
|
||||
method: 'PATCH',
|
||||
headers: { Authorization: `Bearer ${adminToken}`, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ quota_tolerance: '0' }),
|
||||
}
|
||||
);
|
||||
expect(res.status).toBe(400);
|
||||
expect(
|
||||
(await res.text()).toLowerCase(),
|
||||
'the error must name the switch the admin actually wanted'
|
||||
).toContain('speicher-quote');
|
||||
|
||||
// The value is untouched — validation fully precedes any write.
|
||||
expect((await api.getConfig(adminToken)).quota_tolerance).toBe('0.75');
|
||||
});
|
||||
|
||||
test('a very small quota_tolerance is still accepted', async ({ api, adminToken }) => {
|
||||
// The mirror. Rejecting 0 must not become a floor: small tolerances are how a large disk is
|
||||
// throttled to a sensible per-guest ceiling, and how the quota specs steer it (~1e-5 on a
|
||||
// 174 GB volume). A floor of 0.01 would forbid real configurations to prevent one typo.
|
||||
await api.patchConfig(adminToken, { quota_tolerance: '0.00001' });
|
||||
expect((await api.getConfig(adminToken)).quota_tolerance).toBe('0.00001');
|
||||
await api.patchConfig(adminToken, { quota_tolerance: '0.75' });
|
||||
});
|
||||
});
|
||||
|
||||
test.describe('Admin — stats', () => {
|
||||
|
||||
103
e2e/specs/06-export/archive-permissions.spec.ts
Normal file
103
e2e/specs/06-export/archive-permissions.spec.ts
Normal file
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Regression guard — the keepsake extracts to files a guest can actually open.
|
||||
*
|
||||
* `ZipEntryBuilder::new` leaves the external file attribute at zero, and async_zip's host
|
||||
* compatibility defaults to Unix — so every entry in BOTH archives was written with a stored mode
|
||||
* of 0000. `unzip -Z` showed `?---------` on every line.
|
||||
*
|
||||
* Windows Explorer ignores Unix modes, which is exactly why this survived. On Linux and macOS,
|
||||
* `unzip` faithfully applies what the archive asks for, and the guest gets a folder of photos none
|
||||
* of which they can open — plus an index.html the browser refuses with ERR_ACCESS_DENIED.
|
||||
*
|
||||
* Unconditional: it affected every keepsake ever produced, no hostile input required. And it is
|
||||
* invisible server-side — the export succeeds, the ZIP is well-formed, the job writes `done`,
|
||||
* /export/status is green. The only way to see it is to extract the real artifact and try to read
|
||||
* it, which is what this does.
|
||||
*
|
||||
* Found while chasing an unrelated ERR_ACCESS_DENIED that looked like a Playwright quirk.
|
||||
*/
|
||||
import { test, expect } from '../../fixtures/test';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, writeFileSync, rmSync, readFileSync, statSync, readdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { seedUpload } from '../../helpers/seed';
|
||||
import { BASE } from '../../helpers/env';
|
||||
|
||||
/** Walk every file under `dir`, ignoring the archives we dropped there ourselves. */
|
||||
function walk(dir: string, skip: string[] = []): string[] {
|
||||
return readdirSync(dir, { withFileTypes: true }).flatMap((e) => {
|
||||
const p = join(dir, e.name);
|
||||
if (e.isDirectory()) return walk(p, skip);
|
||||
return skip.includes(e.name) ? [] : [p];
|
||||
});
|
||||
}
|
||||
|
||||
test.describe('Export — the archives extract to readable files', () => {
|
||||
test('every entry in both keepsake archives is owner-readable', async ({ host, guest, db }) => {
|
||||
test.setTimeout(120_000);
|
||||
const bearer = { Authorization: `Bearer ${host.jwt}` };
|
||||
|
||||
const g = await guest('Archivist');
|
||||
const id = await seedUpload(g.jwt, { caption: 'ein Foto' });
|
||||
await expect.poll(() => db.compressionStatus(id), { timeout: 30_000 }).toBe('done');
|
||||
|
||||
expect(
|
||||
(await fetch(`${BASE}/api/v1/host/gallery/release`, { method: 'POST', headers: bearer }))
|
||||
.status
|
||||
).toBe(204);
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const res = await fetch(`${BASE}/api/v1/export/status`, { headers: bearer });
|
||||
const s = await res.json();
|
||||
return s.zip?.status === 'done' && s.html?.status === 'done';
|
||||
},
|
||||
{ timeout: 90_000, intervals: [500] }
|
||||
)
|
||||
.toBe(true);
|
||||
|
||||
for (const kind of ['zip', 'html'] as const) {
|
||||
const ticketRes = await fetch(`${BASE}/api/v1/export/ticket`, {
|
||||
method: 'POST',
|
||||
headers: bearer,
|
||||
});
|
||||
const { ticket } = (await ticketRes.json()) as { ticket: string };
|
||||
const dl = await fetch(`${BASE}/api/v1/export/${kind}?ticket=${encodeURIComponent(ticket)}`);
|
||||
expect(dl.status, `downloading the ${kind} archive`).toBe(200);
|
||||
|
||||
const dir = mkdtempSync(join(tmpdir(), `eventsnap-perms-${kind}-`));
|
||||
try {
|
||||
const zipPath = join(dir, 'archive.zip');
|
||||
writeFileSync(zipPath, Buffer.from(await dl.arrayBuffer()));
|
||||
|
||||
// The mode as STORED in the archive — this is what a guest's unzip will apply. Reading it
|
||||
// from the central directory catches the defect even on a filesystem that would mask it.
|
||||
const listing = execFileSync('unzip', ['-Z', zipPath], { encoding: 'utf8' });
|
||||
const modes = listing
|
||||
.split('\n')
|
||||
.filter((l) => /^[?d-][rwx-]{9}\s/.test(l))
|
||||
.map((l) => l.slice(0, 10));
|
||||
expect(modes.length, `${kind}: no entries listed`).toBeGreaterThan(0);
|
||||
for (const m of modes) {
|
||||
expect(m, `${kind}: an entry is stored mode ${m} — the guest cannot open it`).toMatch(
|
||||
/^.r[w-]-/
|
||||
);
|
||||
}
|
||||
|
||||
// And extraction really does produce readable files.
|
||||
execFileSync('unzip', ['-qo', zipPath, '-d', dir]);
|
||||
const files = walk(dir, ['archive.zip']);
|
||||
expect(files.length, `${kind}: nothing extracted`).toBeGreaterThan(0);
|
||||
for (const f of files) {
|
||||
expect(statSync(f).mode & 0o400, `${f} is not owner-readable`).toBeTruthy();
|
||||
// The assertion that matters to a guest: the bytes actually come out.
|
||||
expect(() => readFileSync(f)).not.toThrow();
|
||||
}
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
125
e2e/specs/06-export/viewer-caption-injection.spec.ts
Normal file
125
e2e/specs/06-export/viewer-caption-injection.spec.ts
Normal file
@@ -0,0 +1,125 @@
|
||||
/**
|
||||
* Regression guard — a guest-authored caption cannot brick the offline keepsake.
|
||||
*
|
||||
* The viewer's data is inlined as `<script>window.__EXPORT_DATA__={…};</script>` (it must be:
|
||||
* guests open index.html over file://, where a cross-origin fetch of a sibling data.json is
|
||||
* blocked). Captions and comments are guest text and land in that payload.
|
||||
*
|
||||
* The escape used to be `</` → `<\/`. Against XSS that holds — `</script><img src=x onerror=…>`
|
||||
* round-trips inert. It does NOT stop the caption steering the HTML TOKENIZER: `<!--<script` with
|
||||
* no later `-->` drives the parser into script-data-double-escaped state, where the template's own
|
||||
* `</script>` steps back to script-data-escaped instead of closing the element. Everything after —
|
||||
* including the viewer bundle — is swallowed as script data. Nothing executes and nothing leaks;
|
||||
* `__EXPORT_DATA__` is never assigned and the keepsake renders blank.
|
||||
*
|
||||
* What makes it worth a browser-level test rather than a unit test alone: the failure is SILENT and
|
||||
* POST-DISTRIBUTION. The export succeeds, the ZIP is well-formed, the job writes `done`,
|
||||
* /export/status is green, and the host hands out a file that only fails when a guest
|
||||
* double-clicks it — in every copy, unfixably. It is not visible by reading the escape. It is only
|
||||
* visible by running a real parser over the real artifact, which is what this does: release, pull
|
||||
* the actual Memories.zip, extract index.html, open it over file:// in Chromium, and assert the
|
||||
* viewer actually booted.
|
||||
*
|
||||
* The near-miss worth recording: `<!--<script>alert(1)</script>-->` comes back CLEAN, because the
|
||||
* trailing `-->` returns the parser to script-data state. A probe using the terminated form
|
||||
* quietly repairs the very thing it is testing for. Only the unterminated variant exposes it.
|
||||
*/
|
||||
import { test, expect } from '../../fixtures/test';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { seedUpload } from '../../helpers/seed';
|
||||
import { BASE } from '../../helpers/env';
|
||||
|
||||
/** Unterminated on purpose — see the header. The terminated form self-repairs. */
|
||||
const TOKENIZER_PAYLOAD = '<!--<script';
|
||||
/** The classic break-out. Already handled, kept so the fix can never regress on it. */
|
||||
const BREAKOUT_PAYLOAD = '</script><img src=x onerror=window.__XSS__=1>';
|
||||
|
||||
test.describe('Export — a caption cannot brick the keepsake viewer', () => {
|
||||
test('the exported viewer boots with a tokenizer-hostile caption in it', async ({
|
||||
page,
|
||||
host,
|
||||
guest,
|
||||
db,
|
||||
}) => {
|
||||
test.setTimeout(120_000);
|
||||
const bearer = { Authorization: `Bearer ${host.jwt}` };
|
||||
|
||||
const g = await guest('Trickster');
|
||||
const a = await seedUpload(g.jwt, { caption: TOKENIZER_PAYLOAD });
|
||||
const b = await seedUpload(g.jwt, { caption: BREAKOUT_PAYLOAD });
|
||||
for (const id of [a, b]) {
|
||||
await expect.poll(() => db.compressionStatus(id), { timeout: 30_000 }).toBe('done');
|
||||
}
|
||||
|
||||
expect(
|
||||
(await fetch(`${BASE}/api/v1/host/gallery/release`, { method: 'POST', headers: bearer }))
|
||||
.status
|
||||
).toBe(204);
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const res = await fetch(`${BASE}/api/v1/export/status`, { headers: bearer });
|
||||
return (await res.json()).html?.status;
|
||||
},
|
||||
{ timeout: 90_000, intervals: [500] }
|
||||
)
|
||||
.toBe('done');
|
||||
|
||||
const ticketRes = await fetch(`${BASE}/api/v1/export/ticket`, {
|
||||
method: 'POST',
|
||||
headers: bearer,
|
||||
});
|
||||
const { ticket } = (await ticketRes.json()) as { ticket: string };
|
||||
const dl = await fetch(`${BASE}/api/v1/export/html?ticket=${encodeURIComponent(ticket)}`);
|
||||
expect(dl.status).toBe(200);
|
||||
|
||||
const dir = mkdtempSync(join(tmpdir(), 'eventsnap-viewer-'));
|
||||
try {
|
||||
const zipPath = join(dir, 'Memories.zip');
|
||||
writeFileSync(zipPath, Buffer.from(await dl.arrayBuffer()));
|
||||
// Extract the WHOLE archive: index.html pulls in the viewer's own JS/CSS, and the point of
|
||||
// this test is that those later resources are still reachable by the parser.
|
||||
execFileSync('unzip', ['-qo', zipPath, '-d', dir]);
|
||||
|
||||
// file://, not http://. That is how a guest actually opens the keepsake, and it is the
|
||||
// whole reason the data is inlined rather than fetched from a sibling data.json.
|
||||
let xss = false;
|
||||
page.on('dialog', (d) => {
|
||||
xss = true;
|
||||
void d.dismiss();
|
||||
});
|
||||
await page.goto('file://' + join(dir, 'index.html'));
|
||||
|
||||
// 1. The payload was assigned at all. This is the assertion that fails on the old escape —
|
||||
// the second script block is never reached, so the global stays undefined.
|
||||
const captions = await page.evaluate(() => {
|
||||
const d = (window as unknown as { __EXPORT_DATA__?: { posts?: { caption?: string }[] } })
|
||||
.__EXPORT_DATA__;
|
||||
return d?.posts?.map((p) => p.caption ?? '') ?? null;
|
||||
});
|
||||
expect(captions, '__EXPORT_DATA__ was never assigned — the viewer is bricked').not.toBeNull();
|
||||
|
||||
// 2. The captions survived verbatim. The escape is a transport encoding, not a sanitiser:
|
||||
// a guest's text has to come back exactly, or we have silently rewritten their words.
|
||||
expect(captions).toContain(TOKENIZER_PAYLOAD);
|
||||
expect(captions).toContain(BREAKOUT_PAYLOAD);
|
||||
|
||||
// 3. And nothing executed.
|
||||
expect(
|
||||
await page.evaluate(() => (window as unknown as { __XSS__?: number }).__XSS__ === 1),
|
||||
'the caption must be inert, not merely non-fatal'
|
||||
).toBe(false);
|
||||
expect(xss).toBe(false);
|
||||
|
||||
// 4. The viewer actually rendered — the whole document parsed, not just the head. If the
|
||||
// tokenizer had swallowed the bundle, the body would be empty of viewer output.
|
||||
await expect(page.locator('body')).not.toBeEmpty();
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -84,9 +84,16 @@
|
||||
{ key: 'feed_rate_enabled', label: 'Feed-Limit aktiv', kind: 'bool' },
|
||||
{ key: 'export_rate_enabled', label: 'Export-Limit aktiv', kind: 'bool' },
|
||||
{ key: 'join_rate_enabled', label: 'Join-Limit aktiv', kind: 'bool' },
|
||||
{ key: 'social_rate_enabled', label: 'Interaktions-Limit aktiv', kind: 'bool' },
|
||||
{ key: 'upload_rate_per_hour', label: 'Upload-Limit pro Stunde', kind: 'number' },
|
||||
{ key: 'feed_rate_per_min', label: 'Feed-Anfragen pro Minute', kind: 'number' },
|
||||
{ key: 'export_rate_per_day', label: 'Export-Downloads pro Tag', kind: 'number' }
|
||||
{ key: 'export_rate_per_day', label: 'Export-Downloads pro Tag', kind: 'number' },
|
||||
{
|
||||
key: 'social_rate_per_min',
|
||||
label: 'Interaktionen pro Minute',
|
||||
kind: 'number',
|
||||
hint: 'Likes, Kommentare und Kommentar-Löschungen zusammen, pro Gast. Bewusst hoch angesetzt — soll ein Skript bremsen, keinen begeisterten Gast.'
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -105,7 +112,20 @@
|
||||
kind: 'bool',
|
||||
hint: 'Reserviert für künftige Anzahl-Limits.'
|
||||
},
|
||||
{ key: 'quota_tolerance', label: 'Toleranz (0–1)', kind: 'number' },
|
||||
{
|
||||
key: 'quota_tolerance',
|
||||
label: 'Speicher-Anteil für Gäste (0–1)',
|
||||
kind: 'number',
|
||||
// "Toleranz (0–1)" with no hint invited exactly the wrong reading — that a higher
|
||||
// number means "warn me later". It is the multiplier in
|
||||
// `floor(freier Speicher × Anteil / aktive Uploader)`, so raising it authorises
|
||||
// guests to fill MORE of the disk, not less.
|
||||
hint:
|
||||
'Anteil des freien Speichers, den alle Gäste zusammen belegen dürfen: ' +
|
||||
'Limit = freier Speicher × Anteil ÷ aktive Uploader. Kein Warnschwellenwert — ' +
|
||||
'ein höherer Wert gibt MEHR Speicher frei. Das Keepsake braucht zusätzlich ' +
|
||||
'etwa das Doppelte der Mediengröße; 0,75 ist der getestete Standard.'
|
||||
},
|
||||
{ key: 'estimated_guest_count', label: 'Geschätzte Gästezahl', kind: 'number' }
|
||||
]
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user