Compare commits
10 Commits
fix/export
...
fix/video-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
a3d8ae72e3 | ||
|
|
e1653cc54e | ||
|
|
35390800c7 | ||
|
|
14ebe1e543 | ||
|
|
a4a4e46c53 | ||
|
|
e6e8a52d87 | ||
|
|
43c2a0d09c | ||
|
|
6818cabf91 | ||
|
|
f777764839 | ||
|
|
aeb958f6ba |
24
.env.example
24
.env.example
@@ -18,6 +18,10 @@ POSTGRES_PASSWORD=CHANGE_ME_use_a_strong_password
|
|||||||
POSTGRES_DB=eventsnap
|
POSTGRES_DB=eventsnap
|
||||||
# Connection pool size. Default 10. For a busy event (~100 guests polling the feed
|
# Connection pool size. Default 10. For a busy event (~100 guests polling the feed
|
||||||
# + SSE + uploads at once) raise to ~30 so requests don't queue on a pool permit.
|
# + SSE + uploads at once) raise to ~30 so requests don't queue on a pool permit.
|
||||||
|
# PAIRED WITH THE DB CONTAINER'S MEMORY LIMIT: 30 backends plus Postgres 16's default
|
||||||
|
# shared_buffers is already snug in the 1G that docker-compose.yml allots the `db`
|
||||||
|
# service. If you raise this, raise `db.deploy.resources.limits.memory` with it — an
|
||||||
|
# OOM in Postgres doesn't degrade one feature, it takes the whole event down.
|
||||||
DATABASE_MAX_CONNECTIONS=30
|
DATABASE_MAX_CONNECTIONS=30
|
||||||
|
|
||||||
# ── Authentication ────────────────────────────────────────────────────────────
|
# ── Authentication ────────────────────────────────────────────────────────────
|
||||||
@@ -54,8 +58,26 @@ EXPORT_PATH=/exports
|
|||||||
# max image size 20 MB
|
# max image size 20 MB
|
||||||
# max video size 500 MB
|
# max video size 500 MB
|
||||||
# estimated guests 100
|
# estimated guests 100
|
||||||
# quota tolerance 0.75 (fraction of disk that triggers the low-storage warning)
|
# quota tolerance 0.75 (see below — NOT a warning threshold)
|
||||||
# Adjust these in the admin UI before the event if needed.
|
# Adjust these in the admin UI before the event if needed.
|
||||||
|
#
|
||||||
|
# quota_tolerance is the MULTIPLIER IN THE PER-USER QUOTA FORMULA, not the point at
|
||||||
|
# which anything warns you:
|
||||||
|
#
|
||||||
|
# per_user_limit = floor(free_disk * quota_tolerance / active_uploaders)
|
||||||
|
#
|
||||||
|
# It is recomputed against LIVE free space on every upload, so it self-throttles: guests
|
||||||
|
# converge on a fixed point at tolerance/(1+tolerance) of the free space you started
|
||||||
|
# with — 43% at 0.75, i.e. ~30 GB of a fresh 70 GB.
|
||||||
|
#
|
||||||
|
# Raising it therefore AUTHORISES GUESTS TO FILL MORE OF THE DISK. Setting 0.95 in the
|
||||||
|
# belief that it means "warn me later" moves the fixed point to ~49% and eats the
|
||||||
|
# headroom the keepsake needs — and the keepsake needs a lot, because Gallery.zip and
|
||||||
|
# Memories.zip are each roughly a second copy of every original (both store media
|
||||||
|
# uncompressed). Budget for media + 2x media, or move exports to their own volume.
|
||||||
|
#
|
||||||
|
# 0.75 is the tested default. Lower it if the box is tight; raise it only if you have
|
||||||
|
# provisioned export headroom separately.
|
||||||
|
|
||||||
# ── Workers ───────────────────────────────────────────────────────────────────
|
# ── Workers ───────────────────────────────────────────────────────────────────
|
||||||
# Number of parallel image/video compression workers. Default 2. This is the main
|
# Number of parallel image/video compression workers. Default 2. This is the main
|
||||||
|
|||||||
125
README.md
125
README.md
@@ -34,7 +34,6 @@ A guest scans the QR code on their way in, types their name, and is immediately
|
|||||||
### Planned (v1.x)
|
### Planned (v1.x)
|
||||||
|
|
||||||
- Individual file download button
|
- Individual file download button
|
||||||
- Low-disk alert (< 10 GB free)
|
|
||||||
- Event banner / cover image
|
- Event banner / cover image
|
||||||
- Chunked resumable upload for large videos
|
- Chunked resumable upload for large videos
|
||||||
- Host-curated story highlights
|
- Host-curated story highlights
|
||||||
@@ -231,6 +230,45 @@ so a host takedown or a ban actually revokes access to the bytes.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Sizing the disk
|
||||||
|
|
||||||
|
`postgres_data`, `media_data` and `exports_data` are all Docker named volumes under
|
||||||
|
`/var/lib/docker/volumes`, so **they share one filesystem**. Filling it does not
|
||||||
|
degrade one subsystem — Postgres stops being able to write and the whole event goes
|
||||||
|
down.
|
||||||
|
|
||||||
|
Uploads are self-limiting. `per_user_limit = free_disk × quota_tolerance ÷
|
||||||
|
active_uploaders` is recomputed against live free space on every upload, so guests
|
||||||
|
converge on a fixed point at `tolerance / (1 + tolerance)` of the free space you
|
||||||
|
started with — **43%** at the default 0.75. On an 80 GB box with ~70 GB free after
|
||||||
|
the OS and images, media settles at ~30 GB and stops.
|
||||||
|
|
||||||
|
**The keepsake is what the 80 GB baseline does not cover.** `Gallery.zip` and
|
||||||
|
`Memories.zip` are built concurrently and each is roughly a second copy of every
|
||||||
|
original: both write their media `Compression::Stored`, and `Memories.zip` streams the
|
||||||
|
untouched original for every video and for every image at or under 5 MB. So a release
|
||||||
|
wants room for **two more copies of the gallery** on top of the gallery itself.
|
||||||
|
|
||||||
|
| Stage | Used | Free (80 GB box) |
|
||||||
|
|---|---|---|
|
||||||
|
| Fresh box (OS + images) | ~10 GB | ~70 GB |
|
||||||
|
| Guests reach the quota fixed point | ~40 GB | ~40 GB |
|
||||||
|
| Host releases → both archives | ~100 GB | **ENOSPC** |
|
||||||
|
|
||||||
|
Two ways to size for it:
|
||||||
|
|
||||||
|
- **Provision ~3× your expected media** on one volume (media + two archives), or
|
||||||
|
- **give `exports_data` its own volume** so a full export cannot reach Postgres, and
|
||||||
|
size that one at ~2× expected media.
|
||||||
|
|
||||||
|
This is no longer silent. The export refuses up front with the two numbers rather than
|
||||||
|
hitting ENOSPC halfway through a multi-GB write, a rebuild reclaims the superseded
|
||||||
|
generation before it starts (so peak is one generation, not two), and the host
|
||||||
|
dashboard warns as soon as the keepsake would not fit — which is the only point at
|
||||||
|
which anyone can still do something about it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Backup
|
## Backup
|
||||||
|
|
||||||
There are **three** things to back up, and they live in three different places.
|
There are **three** things to back up, and they live in three different places.
|
||||||
@@ -242,9 +280,12 @@ never exported into an operator's shell — so every command below runs through
|
|||||||
```bash
|
```bash
|
||||||
# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no
|
# 1. Database snapshot. Runs pg_dump inside the db container (the app image has no
|
||||||
# postgres client), reading credentials from the compose environment.
|
# postgres client), reading credentials from the compose environment.
|
||||||
|
# --clean --if-exists makes the dump SELF-CLEANING: without it the restore below
|
||||||
|
# aborts on the first "already exists" against a database that has ever booted,
|
||||||
|
# which is every database you would actually want to restore over.
|
||||||
mkdir -p ./backups
|
mkdir -p ./backups
|
||||||
docker compose exec -T db \
|
docker compose exec -T db \
|
||||||
sh -c 'pg_dump -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
sh -c 'pg_dump --clean --if-exists -U "$POSTGRES_USER" "$POSTGRES_DB"' \
|
||||||
| gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz
|
| gzip > ./backups/db_$(date +%Y-%m-%d).sql.gz
|
||||||
|
|
||||||
# 2. Uploaded media (originals + derivatives) out of the named volume.
|
# 2. Uploaded media (originals + derivatives) out of the named volume.
|
||||||
@@ -261,7 +302,7 @@ docker run --rm \
|
|||||||
-v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \
|
-v eventsnap_exports_data:/src:ro -v "$PWD/backups":/backup \
|
||||||
alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src .
|
alpine tar czf /backup/exports_$(date +%Y-%m-%d).tar.gz -C /src .
|
||||||
|
|
||||||
# Weekly offsite sync of the three artefacts above.
|
# Offsite sync of the three artefacts above.
|
||||||
rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/
|
rsync -az ./backups/ user@storagebox.example.com:backup/eventsnap/
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -274,6 +315,82 @@ from a directory called `eventsnap`. Confirm yours with `docker volume ls`.
|
|||||||
> stops it being reachable except through the ticket-gated download handler.
|
> stops it being reachable except through the ticket-gated download handler.
|
||||||
> Backing up only the media volume therefore loses every generated keepsake.
|
> Backing up only the media volume therefore loses every generated keepsake.
|
||||||
|
|
||||||
|
### When to run it
|
||||||
|
|
||||||
|
**A nightly cron is the wrong shape for this app.** Every irreplaceable byte is
|
||||||
|
created inside one eight-hour window, and nobody can retake a wedding. Run the three
|
||||||
|
commands above:
|
||||||
|
|
||||||
|
1. **The night of the event**, once uploads have stopped. This is the backup that
|
||||||
|
matters; everything else is a formality.
|
||||||
|
2. **After the host releases the gallery**, so the generated keepsake is captured too.
|
||||||
|
3. Weekly thereafter, until the event is archived and torn down.
|
||||||
|
|
||||||
|
Take the DB dump and the media tarball **back to back**, without uploads in flight
|
||||||
|
between them. Upload rows reference files by path — a database from 22:00 and a media
|
||||||
|
volume from 23:00 gives you rows pointing at files the dump doesn't know about, and
|
||||||
|
rows whose files aren't in the tarball. Locking uploads from the host dashboard first
|
||||||
|
(**Uploads sperren**) makes the pair genuinely consistent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Restore
|
||||||
|
|
||||||
|
An untested backup is not a backup. Run this once against a scratch host **before**
|
||||||
|
the event — it is roughly ten minutes, and it is the only way to find out that your
|
||||||
|
tarball is empty or your dump is truncated while that is still a small problem.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 0. Stop the app FIRST. Migrations run on boot and a live pool will fight the
|
||||||
|
# restore — a booting app against a half-restored schema can leave the migration
|
||||||
|
# table and the schema disagreeing, which is its own recovery problem.
|
||||||
|
# Leave `db` running: the dump is restored through it.
|
||||||
|
docker compose stop app caddy
|
||||||
|
|
||||||
|
# 1. Database. The dump carries its own DROPs (step 1 of Backup), so this replaces
|
||||||
|
# rather than collides. A dump taken WITHOUT --clean --if-exists will abort here
|
||||||
|
# on the first "already exists" — restore that one into a fresh empty database
|
||||||
|
# instead.
|
||||||
|
gunzip -c ./backups/db_2026-07-29.sql.gz \
|
||||||
|
| docker compose exec -T db \
|
||||||
|
sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" --set ON_ERROR_STOP=1'
|
||||||
|
|
||||||
|
# 2. Media. NOTE the `--numeric-owner` and the chown: the app runs as a
|
||||||
|
# NON-ROOT user (uid 100, gid 101 — `addgroup -S app && adduser -S app`), and a
|
||||||
|
# restore that lands root-owned files makes every upload fail with EACCES deep in
|
||||||
|
# the write path, surfacing to the guest as a generic 500 with nothing in the UI
|
||||||
|
# to suggest permissions. The explicit chown is what guarantees it — BusyBox tar
|
||||||
|
# (which is what `alpine` ships) has no --same-owner, and restores ownership only
|
||||||
|
# because it runs as root here.
|
||||||
|
docker run --rm \
|
||||||
|
-v eventsnap_media_data:/dst -v "$PWD/backups":/backup:ro \
|
||||||
|
alpine sh -c 'tar xzf /backup/media_2026-07-29.tar.gz -C /dst \
|
||||||
|
--numeric-owner && chown -R 100:101 /dst'
|
||||||
|
|
||||||
|
# 3. Exports. Same volume-name caveat, same ownership rules.
|
||||||
|
docker run --rm \
|
||||||
|
-v eventsnap_exports_data:/dst -v "$PWD/backups":/backup:ro \
|
||||||
|
alpine sh -c 'tar xzf /backup/exports_2026-07-29.tar.gz -C /dst \
|
||||||
|
--numeric-owner && chown -R 100:101 /dst'
|
||||||
|
|
||||||
|
# 4. Back up. Migrations run, then export recovery re-arms any keepsake whose file
|
||||||
|
# didn't come back with the volume.
|
||||||
|
docker compose up -d app caddy
|
||||||
|
docker compose logs -f app # watch for "migrations applied"
|
||||||
|
|
||||||
|
# 5. Verify — all three, not just the first.
|
||||||
|
curl -fsS https://DOMAIN/health && echo # → ok
|
||||||
|
# … then sign in as host and confirm the feed renders images (proves the media
|
||||||
|
# volume restored AND is readable by uid 100), and that the keepsake downloads.
|
||||||
|
```
|
||||||
|
|
||||||
|
If the media volume restored but images 404 while the feed lists them, the paths are
|
||||||
|
there and the bytes aren't — check `docker compose exec app ls -ln /media/originals`
|
||||||
|
and confirm both the files and the `100:101` ownership.
|
||||||
|
|
||||||
|
The restore is deliberately **not** automated. It is rare, destructive, and the one
|
||||||
|
operation where a script that half-works is worse than a checklist someone reads.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Running the backend test suite
|
## Running the backend test suite
|
||||||
@@ -348,7 +465,7 @@ Open:
|
|||||||
- [ ] SSE delta-fetch on foreground reconnect (scaffolded in [sse.ts](frontend/src/lib/sse.ts), not wired)
|
- [ ] SSE delta-fetch on foreground reconnect (scaffolded in [sse.ts](frontend/src/lib/sse.ts), not wired)
|
||||||
- [ ] Live diashow / slideshow mode — see [docs/CONCEPT_DIASHOW.md](docs/CONCEPT_DIASHOW.md)
|
- [ ] Live diashow / slideshow mode — see [docs/CONCEPT_DIASHOW.md](docs/CONCEPT_DIASHOW.md)
|
||||||
- [ ] Individual file download button per post
|
- [ ] Individual file download button per post
|
||||||
- [ ] Low-disk alert (< 10 GB free)
|
- [x] Low-disk alert — host dashboard warns below 10 GB free, or whenever the keepsake would not fit
|
||||||
- [ ] Event banner / cover image
|
- [ ] Event banner / cover image
|
||||||
- [ ] Chunked resumable upload for files > 100 MB
|
- [ ] Chunked resumable upload for files > 100 MB
|
||||||
- [ ] Shared Tailwind config between main app and export-viewer
|
- [ ] Shared Tailwind config between main app and export-viewer
|
||||||
|
|||||||
1
backend/migrations/020_social_rate.down.sql
Normal file
1
backend/migrations/020_social_rate.down.sql
Normal file
@@ -0,0 +1 @@
|
|||||||
|
DELETE FROM config WHERE key IN ('social_rate_per_min', 'social_rate_enabled');
|
||||||
16
backend/migrations/020_social_rate.up.sql
Normal file
16
backend/migrations/020_social_rate.up.sql
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
-- Per-user rate limit for social writes (likes, comments, comment deletions).
|
||||||
|
--
|
||||||
|
-- These were the only writes in the app with no limit at all. Every other mutating
|
||||||
|
-- path -- upload, join, recover, export, admin login -- carries one; social.rs
|
||||||
|
-- carried none, so the coverage was asymmetric rather than deliberately open.
|
||||||
|
--
|
||||||
|
-- Severity is genuinely low for an invited-guest event, and the amplification worry
|
||||||
|
-- turned out to be contained: a like fans an SSE broadcast to ~100 clients, but the
|
||||||
|
-- export regeneration it could otherwise trigger is debounced (REGEN_DEBOUNCE 20s)
|
||||||
|
-- and superseded workers are inert. So this closes the gap for symmetry, not urgency,
|
||||||
|
-- and the ceiling is set high enough that no real guest will ever meet it -- a
|
||||||
|
-- double-tapping enthusiast at a wedding is not the thing being defended against.
|
||||||
|
INSERT INTO config (key, value) VALUES
|
||||||
|
('social_rate_per_min', '120'),
|
||||||
|
('social_rate_enabled', 'true')
|
||||||
|
ON CONFLICT (key) DO NOTHING;
|
||||||
@@ -127,6 +127,9 @@ pub async fn patch_config(
|
|||||||
// Same shape for /recover: the per-(ip, name) bucket is the anti-guessing control,
|
// Same shape for /recover: the per-(ip, name) bucket is the anti-guessing control,
|
||||||
// this only bounds a name-cycling flood in front of a cost-12 bcrypt (migration 019).
|
// this only bounds a name-cycling flood in front of a cost-12 bcrypt (migration 019).
|
||||||
("recover_ip_rate_per_min", true, 1.0, 100_000.0),
|
("recover_ip_rate_per_min", true, 1.0, 100_000.0),
|
||||||
|
// Aggregate ceiling on likes + comments + comment deletions, per user per minute.
|
||||||
|
// These were the only mutating endpoints with no limit at all (migration 020).
|
||||||
|
("social_rate_per_min", true, 1.0, 100_000.0),
|
||||||
("quota_tolerance", false, 0.0, 1.0),
|
("quota_tolerance", false, 0.0, 1.0),
|
||||||
("estimated_guest_count", true, 1.0, 1_000_000.0),
|
("estimated_guest_count", true, 1.0, 1_000_000.0),
|
||||||
];
|
];
|
||||||
@@ -141,6 +144,7 @@ pub async fn patch_config(
|
|||||||
// missing from this allowlist — so the switch existed in code and could never be flipped.
|
// missing from this allowlist — so the switch existed in code and could never be flipped.
|
||||||
"admin_login_rate_enabled",
|
"admin_login_rate_enabled",
|
||||||
"recover_rate_enabled",
|
"recover_rate_enabled",
|
||||||
|
"social_rate_enabled",
|
||||||
"quota_enabled",
|
"quota_enabled",
|
||||||
"storage_quota_enabled",
|
"storage_quota_enabled",
|
||||||
"upload_count_quota_enabled",
|
"upload_count_quota_enabled",
|
||||||
|
|||||||
@@ -35,6 +35,32 @@ pub struct EventStatus {
|
|||||||
pub is_active: bool,
|
pub is_active: bool,
|
||||||
pub uploads_locked: bool,
|
pub uploads_locked: bool,
|
||||||
pub export_released: bool,
|
pub export_released: bool,
|
||||||
|
/// Free space on the volume the keepsake is written to. `None` when the mount can't be
|
||||||
|
/// resolved — the UI hides the widget rather than rendering a confident zero.
|
||||||
|
pub disk_free_bytes: Option<u64>,
|
||||||
|
/// What a full keepsake build would need right now (both halves).
|
||||||
|
pub keepsake_required_bytes: u64,
|
||||||
|
/// Whether the host should be warned. See [`disk_is_low`].
|
||||||
|
pub disk_low: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Absolute floor below which free space is worth surfacing regardless of gallery size — the
|
||||||
|
/// threshold the README has carried on the roadmap since v1.
|
||||||
|
const LOW_DISK_FLOOR_BYTES: u64 = 10_000_000_000;
|
||||||
|
|
||||||
|
/// Is free space low enough that the host needs to know?
|
||||||
|
///
|
||||||
|
/// Two triggers, because a fixed threshold answers the wrong question. `postgres_data`,
|
||||||
|
/// `media_data` and `exports_data` are all Docker named volumes on one filesystem, so a full disk
|
||||||
|
/// does not degrade one subsystem — it stops Postgres writing and takes the event down. That is
|
||||||
|
/// what the absolute floor is for.
|
||||||
|
///
|
||||||
|
/// The second trigger is the one that actually earns its place: the keepsake needs room for two
|
||||||
|
/// gallery-sized archives, and the only moment a host can do anything about that is BEFORE they
|
||||||
|
/// release. Warning at "you could not build the keepsake right now" turns a post-event dead end
|
||||||
|
/// into a decision someone can still make.
|
||||||
|
fn disk_is_low(free: u64, keepsake_required: u64) -> bool {
|
||||||
|
free < LOW_DISK_FLOOR_BYTES || free < keepsake_required
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Count non-banned hosts/admins in the event OTHER than `excluding` — the operators
|
/// Count non-banned hosts/admins in the event OTHER than `excluding` — the operators
|
||||||
@@ -72,11 +98,29 @@ pub async fn get_event_status(
|
|||||||
.await?
|
.await?
|
||||||
.ok_or_else(|| AppError::NotFound("Event nicht gefunden.".into()))?;
|
.ok_or_else(|| AppError::NotFound("Event nicht gefunden.".into()))?;
|
||||||
|
|
||||||
|
// Measured on the EXPORT volume, not the media one: that is where the cliff is, and it is a
|
||||||
|
// distinct mount point even when both are backed by the same filesystem. The cached reading is
|
||||||
|
// right here — this is advisory, polled on every dashboard load, and a 15s-stale number costs
|
||||||
|
// nothing (unlike the export preflight, which reads uncached because it is about to write).
|
||||||
|
let free = state
|
||||||
|
.disk_cache
|
||||||
|
.snapshot(&state.config.export_path)
|
||||||
|
.map(|d| d.free);
|
||||||
|
let keepsake_required_bytes =
|
||||||
|
crate::services::export::keepsake_space_required(&state.pool, event.id)
|
||||||
|
.await
|
||||||
|
.unwrap_or(0);
|
||||||
|
|
||||||
Ok(Json(EventStatus {
|
Ok(Json(EventStatus {
|
||||||
name: event.name,
|
name: event.name,
|
||||||
is_active: event.is_active,
|
is_active: event.is_active,
|
||||||
uploads_locked: event.uploads_locked_at.is_some(),
|
uploads_locked: event.uploads_locked_at.is_some(),
|
||||||
export_released: event.export_released_at.is_some(),
|
export_released: event.export_released_at.is_some(),
|
||||||
|
disk_free_bytes: free,
|
||||||
|
keepsake_required_bytes,
|
||||||
|
// Unknown free space is NOT low. Fails open, exactly as the upload quota and the export
|
||||||
|
// preflight do: a scary banner on an unreadable mount would train the host to ignore it.
|
||||||
|
disk_low: free.is_some_and(|f| disk_is_low(f, keepsake_required_bytes)),
|
||||||
}))
|
}))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -765,3 +809,45 @@ pub async fn release_gallery(
|
|||||||
|
|
||||||
Ok(StatusCode::NO_CONTENT)
|
Ok(StatusCode::NO_CONTENT)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::{LOW_DISK_FLOOR_BYTES, disk_is_low};
|
||||||
|
|
||||||
|
const GB: u64 = 1_000_000_000;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_healthy_disk_with_room_for_the_keepsake_is_not_low() {
|
||||||
|
assert!(!disk_is_low(40 * GB, 25 * GB));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_absolute_floor_fires_even_when_the_gallery_is_tiny() {
|
||||||
|
// All three volumes share one filesystem, so running out doesn't degrade one subsystem —
|
||||||
|
// Postgres stops being able to write and the event goes down. A 1 GB gallery would clear
|
||||||
|
// the keepsake test comfortably; the floor is what catches this.
|
||||||
|
assert!(disk_is_low(5 * GB, GB));
|
||||||
|
assert!(disk_is_low(LOW_DISK_FLOOR_BYTES - 1, 0));
|
||||||
|
assert!(!disk_is_low(LOW_DISK_FLOOR_BYTES, 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn plenty_of_space_is_still_low_when_the_keepsake_would_not_fit() {
|
||||||
|
// THE case the fixed threshold misses, and the one that matters: 30 GB free is nowhere near
|
||||||
|
// any floor, but a 30 GB gallery needs room for TWO archives. The host can act on this
|
||||||
|
// before releasing; after releasing, they cannot.
|
||||||
|
assert!(disk_is_low(30 * GB, 66 * GB));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_keepsake_trigger_is_exact_at_the_boundary() {
|
||||||
|
assert!(!disk_is_low(66 * GB, 66 * GB), "exactly enough is enough");
|
||||||
|
assert!(disk_is_low(66 * GB - 1, 66 * GB));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_empty_gallery_needs_nothing_and_only_the_floor_applies() {
|
||||||
|
assert!(!disk_is_low(11 * GB, 0));
|
||||||
|
assert!(disk_is_low(9 * GB, 0));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -10,8 +10,40 @@ use crate::error::AppError;
|
|||||||
use crate::models::comment::{Comment, CommentDto};
|
use crate::models::comment::{Comment, CommentDto};
|
||||||
use crate::models::hashtag::{self, Hashtag};
|
use crate::models::hashtag::{self, Hashtag};
|
||||||
use crate::models::upload::Upload;
|
use crate::models::upload::Upload;
|
||||||
|
use crate::services::config;
|
||||||
use crate::state::AppState;
|
use crate::state::AppState;
|
||||||
|
|
||||||
|
/// Throttle a social write. Keyed PER USER, like the feed and upload limits and for the same
|
||||||
|
/// reason: at a venue every guest sits behind one NAT, so an IP key hands the whole party a
|
||||||
|
/// single bucket and the most active guest starves everyone else.
|
||||||
|
///
|
||||||
|
/// These were the only mutating endpoints in the app with no limit at all — the coverage was
|
||||||
|
/// asymmetric, not deliberately open. The ceiling is set well above anything a real guest
|
||||||
|
/// produces; this bounds a script, not an enthusiastic double-tapper.
|
||||||
|
async fn check_social_rate(state: &AppState, user_id: Uuid) -> Result<(), AppError> {
|
||||||
|
let rate_limits_on = config::get_bool(&state.config_cache, "rate_limits_enabled", true).await;
|
||||||
|
let social_rate_on = config::get_bool(&state.config_cache, "social_rate_enabled", true).await;
|
||||||
|
if !(rate_limits_on && social_rate_on) {
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
let rate_limit = config::get_usize(&state.config_cache, "social_rate_per_min", 120).await;
|
||||||
|
// ONE bucket across likes, comments and comment deletions. Separate buckets would let a
|
||||||
|
// caller triple the aggregate write rate just by alternating between them.
|
||||||
|
state
|
||||||
|
.rate_limiter
|
||||||
|
.check_with_retry(
|
||||||
|
format!("social:{user_id}"),
|
||||||
|
rate_limit,
|
||||||
|
std::time::Duration::from_secs(60),
|
||||||
|
)
|
||||||
|
.map_err(|retry_after_secs| {
|
||||||
|
AppError::TooManyRequests(
|
||||||
|
"Zu viele Aktionen. Bitte warte kurz und versuche es erneut.".into(),
|
||||||
|
Some(retry_after_secs),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
#[derive(Serialize)]
|
#[derive(Serialize)]
|
||||||
pub struct LikeResponse {
|
pub struct LikeResponse {
|
||||||
/// The caller's like state *after* this toggle. The client sets `liked_by_me` from
|
/// The caller's like state *after* this toggle. The client sets `liked_by_me` from
|
||||||
@@ -35,6 +67,7 @@ pub async fn toggle_like(
|
|||||||
if user.is_banned {
|
if user.is_banned {
|
||||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||||
}
|
}
|
||||||
|
check_social_rate(&state, auth.user_id).await?;
|
||||||
|
|
||||||
// Event-scope: the upload must belong to the caller's event (404 otherwise),
|
// Event-scope: the upload must belong to the caller's event (404 otherwise),
|
||||||
// matching the host handlers' find_by_id_and_event pattern.
|
// matching the host handlers' find_by_id_and_event pattern.
|
||||||
@@ -141,6 +174,7 @@ pub async fn add_comment(
|
|||||||
if user.is_banned {
|
if user.is_banned {
|
||||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||||
}
|
}
|
||||||
|
check_social_rate(&state, auth.user_id).await?;
|
||||||
|
|
||||||
// Event-scope: only comment on an upload that belongs to the caller's event.
|
// Event-scope: only comment on an upload that belongs to the caller's event.
|
||||||
Upload::find_by_id_and_event(&state.pool, upload_id, auth.event_id)
|
Upload::find_by_id_and_event(&state.pool, upload_id, auth.event_id)
|
||||||
@@ -216,6 +250,7 @@ pub async fn delete_comment(
|
|||||||
if auth.is_banned {
|
if auth.is_banned {
|
||||||
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
return Err(AppError::Forbidden("Du bist gesperrt.".into()));
|
||||||
}
|
}
|
||||||
|
check_social_rate(&state, auth.user_id).await?;
|
||||||
let comment = Comment::find_by_id(&state.pool, comment_id)
|
let comment = Comment::find_by_id(&state.pool, comment_id)
|
||||||
.await?
|
.await?
|
||||||
.ok_or_else(|| AppError::NotFound("Kommentar nicht gefunden.".into()))?;
|
.ok_or_else(|| AppError::NotFound("Kommentar nicht gefunden.".into()))?;
|
||||||
|
|||||||
@@ -63,6 +63,7 @@ pub async fn truncate_all(
|
|||||||
('export_rate_per_day', '3'),
|
('export_rate_per_day', '3'),
|
||||||
('join_ip_rate_per_min', '60'),
|
('join_ip_rate_per_min', '60'),
|
||||||
('recover_ip_rate_per_min', '30'),
|
('recover_ip_rate_per_min', '30'),
|
||||||
|
('social_rate_per_min', '120'),
|
||||||
('quota_tolerance', '0.75'),
|
('quota_tolerance', '0.75'),
|
||||||
('estimated_guest_count', '100'),
|
('estimated_guest_count', '100'),
|
||||||
('compression_concurrency', '2'),
|
('compression_concurrency', '2'),
|
||||||
@@ -71,6 +72,7 @@ pub async fn truncate_all(
|
|||||||
('feed_rate_enabled', 'false'),
|
('feed_rate_enabled', 'false'),
|
||||||
('export_rate_enabled', 'false'),
|
('export_rate_enabled', 'false'),
|
||||||
('join_rate_enabled', 'false'),
|
('join_rate_enabled', 'false'),
|
||||||
|
('social_rate_enabled', 'false'),
|
||||||
('admin_login_rate_enabled', 'false'),
|
('admin_login_rate_enabled', 'false'),
|
||||||
('quota_enabled', 'false'),
|
('quota_enabled', 'false'),
|
||||||
('storage_quota_enabled', 'false'),
|
('storage_quota_enabled', 'false'),
|
||||||
|
|||||||
@@ -1268,7 +1268,7 @@ fn is_superseded_archive(
|
|||||||
/// want, since being wrong low means ENOSPC halfway through.
|
/// want, since being wrong low means ENOSPC halfway through.
|
||||||
///
|
///
|
||||||
/// Matches [`query_uploads`]' visibility filter exactly, so hidden/banned uploads aren't counted.
|
/// Matches [`query_uploads`]' visibility filter exactly, so hidden/banned uploads aren't counted.
|
||||||
async fn estimate_export_bytes(pool: &PgPool, event_id: Uuid) -> Result<u64> {
|
pub async fn estimate_export_bytes(pool: &PgPool, event_id: Uuid) -> Result<u64> {
|
||||||
let (bytes,): (i64,) = sqlx::query_as(
|
let (bytes,): (i64,) = sqlx::query_as(
|
||||||
"SELECT COALESCE(SUM(u.original_size_bytes), 0)::bigint
|
"SELECT COALESCE(SUM(u.original_size_bytes), 0)::bigint
|
||||||
FROM upload u
|
FROM upload u
|
||||||
@@ -1302,6 +1302,20 @@ fn required_free_bytes(media_bytes: u64, armed: i64) -> u64 {
|
|||||||
needed.min(u64::MAX as u128) as u64
|
needed.min(u64::MAX as u128) as u64
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Free bytes a full keepsake build would need RIGHT NOW, both halves included.
|
||||||
|
///
|
||||||
|
/// The same arithmetic the preflight uses, exposed so the host dashboard can warn BEFORE the
|
||||||
|
/// release rather than reporting a failure after it. The preflight can only ever say "this didn't
|
||||||
|
/// fit"; at that point the gallery is full, the event is over, and the remedies (ask guests to stop
|
||||||
|
/// uploading, grow the volume) are all much harder. Hard-codes both halves because that is what a
|
||||||
|
/// release arms.
|
||||||
|
pub async fn keepsake_space_required(pool: &PgPool, event_id: Uuid) -> Result<u64> {
|
||||||
|
Ok(required_free_bytes(
|
||||||
|
estimate_export_bytes(pool, event_id).await?,
|
||||||
|
2,
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
/// Refuse to start an export that cannot fit, with a reason the host can act on.
|
/// Refuse to start an export that cannot fit, with a reason the host can act on.
|
||||||
///
|
///
|
||||||
/// Without this the failure mode is ENOSPC halfway through a multi-GB write, and the wreckage
|
/// Without this the failure mode is ENOSPC halfway through a multi-GB write, and the wreckage
|
||||||
|
|||||||
@@ -11,8 +11,9 @@
|
|||||||
//! 2. **Periodic tasks** — pruning that should happen "every hour" rather than per
|
//! 2. **Periodic tasks** — pruning that should happen "every hour" rather than per
|
||||||
//! request: expired sessions (otherwise the table grows unboundedly), the
|
//! request: expired sessions (otherwise the table grows unboundedly), the
|
||||||
//! rate-limiter's in-memory windows (so keys for IPs that left long ago don't
|
//! rate-limiter's in-memory windows (so keys for IPs that left long ago don't
|
||||||
//! accumulate), and the originals of uploads whose compression permanently failed
|
//! accumulate), and the media of soft-deleted uploads — both the ones whose compression
|
||||||
//! (which are deliberately retained for a recovery window, then reclaimed).
|
//! permanently failed and the ones a guest or host deliberately removed — which are
|
||||||
|
//! retained for a recovery window and then reclaimed.
|
||||||
|
|
||||||
use std::path::PathBuf;
|
use std::path::PathBuf;
|
||||||
use std::time::Duration;
|
use std::time::Duration;
|
||||||
@@ -37,6 +38,27 @@ use crate::services::sse_tickets::SseTicketStore;
|
|||||||
/// failed upload still has the file, while the leak stays bounded.
|
/// failed upload still has the file, while the leak stays bounded.
|
||||||
const FAILED_ORIGINAL_RETENTION_DAYS: i64 = 14;
|
const FAILED_ORIGINAL_RETENTION_DAYS: i64 = 14;
|
||||||
|
|
||||||
|
/// How long a DELIBERATELY deleted upload's files are kept before they are reclaimed.
|
||||||
|
///
|
||||||
|
/// The same leak, reached by the ordinary path rather than the exceptional one.
|
||||||
|
/// `soft_delete_in_event` stamps `deleted_at` and refunds `total_upload_bytes`, but nothing ever
|
||||||
|
/// removed the bytes — so the quota stopped bounding the disk. Upload 500 MB, delete, quota is back
|
||||||
|
/// to zero, upload another 500 MB: not an attack, just a guest curating their camera roll, which is
|
||||||
|
/// what people do. The host then sees guests hitting "Du hast dein Upload-Limit erreicht" while the
|
||||||
|
/// admin widget shows a disk full of files no upload row points at, and the quota message is
|
||||||
|
/// actively misleading because the space really is gone — just not to anyone the accounting can
|
||||||
|
/// name.
|
||||||
|
///
|
||||||
|
/// Much shorter than the failure window on purpose. Fourteen days outlives the whole event, so a
|
||||||
|
/// deliberate delete would never reclaim anything while it mattered. A day still gives an operator
|
||||||
|
/// a recovery window for a mis-tap.
|
||||||
|
///
|
||||||
|
/// NOTE what this does NOT do: within the window the bytes are still spent and still unaccounted,
|
||||||
|
/// so a guest deleting and re-uploading through an eight-hour event can outrun the sweep. Bounding
|
||||||
|
/// that would mean holding the quota until the file is actually reclaimed rather than refunding at
|
||||||
|
/// `deleted_at` — a deliberate trade, and the reason the low-disk warning exists.
|
||||||
|
const DELETED_UPLOAD_RETENTION_HOURS: i64 = 24;
|
||||||
|
|
||||||
/// Reset rows left in flight by a previous crashed instance. Run once on startup,
|
/// Reset rows left in flight by a previous crashed instance. Run once on startup,
|
||||||
/// before the HTTP server starts taking requests, so users never observe the
|
/// before the HTTP server starts taking requests, so users never observe the
|
||||||
/// half-state.
|
/// half-state.
|
||||||
@@ -117,38 +139,59 @@ pub fn spawn_periodic_tasks(
|
|||||||
loop {
|
loop {
|
||||||
tick.tick().await;
|
tick.tick().await;
|
||||||
cleanup_sessions(&pool).await;
|
cleanup_sessions(&pool).await;
|
||||||
cleanup_failed_originals(&pool, &media_path).await;
|
cleanup_deleted_media(&pool, &media_path).await;
|
||||||
rate_limiter.prune();
|
rate_limiter.prune();
|
||||||
sse_tickets.prune();
|
sse_tickets.prune();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Reclaim the originals of uploads whose compression permanently failed, once they are
|
/// Reclaim the media of soft-deleted uploads once they are past their retention window.
|
||||||
/// past [`FAILED_ORIGINAL_RETENTION_DAYS`].
|
|
||||||
///
|
///
|
||||||
/// Deliberately narrow. It only touches rows that are BOTH `compression_status = 'failed'`
|
/// ONLY ever touches rows with `deleted_at IS NOT NULL`, so it can never reach a live upload. Two
|
||||||
/// AND soft-deleted — i.e. the exact state the compression worker's give-up path leaves
|
/// classes, two windows, because the two deletes mean different things:
|
||||||
/// behind — so it can never reach a live upload or one whose preview works. `original_path`
|
///
|
||||||
/// is cleared in the same pass, which makes the sweep idempotent and stops a later run
|
/// - a compression failure the guest didn't ask for and may want investigated —
|
||||||
/// re-reporting a file that is already gone. The row itself is kept: it is the audit trail
|
/// [`FAILED_ORIGINAL_RETENTION_DAYS`];
|
||||||
/// for the failure, and it costs a few hundred bytes.
|
/// - a deliberate removal by the guest or the host — [`DELETED_UPLOAD_RETENTION_HOURS`].
|
||||||
async fn cleanup_failed_originals(pool: &PgPool, media_path: &std::path::Path) {
|
///
|
||||||
let rows = sqlx::query_as::<_, (uuid::Uuid, String)>(
|
/// ALL FOUR paths are reclaimed, not just the original. The previous version cleared
|
||||||
"SELECT id, original_path FROM upload
|
/// `original_path` alone, which was right for its only case (a failed compression produces no
|
||||||
WHERE compression_status = 'failed'
|
/// derivatives) but wrong the moment the sweep reaches a successfully processed upload: preview,
|
||||||
AND deleted_at IS NOT NULL
|
/// display and thumbnail are each a separate file on disk, none of them counted in
|
||||||
AND deleted_at < NOW() - ($1 || ' days')::interval
|
/// `original_size_bytes`, and nothing else ever removed them.
|
||||||
AND original_path <> ''",
|
///
|
||||||
|
/// Every column is cleared in the same pass, which makes the sweep idempotent and stops a later run
|
||||||
|
/// re-reporting files that are already gone. The ROW is kept: it is the audit trail, it costs a few
|
||||||
|
/// hundred bytes, and `backfill_stale_derivatives` is guarded on `deleted_at IS NULL` so a nulled
|
||||||
|
/// `preview_path` can never make it regenerate what was just reclaimed.
|
||||||
|
async fn cleanup_deleted_media(pool: &PgPool, media_path: &std::path::Path) {
|
||||||
|
type Row = (
|
||||||
|
uuid::Uuid,
|
||||||
|
String,
|
||||||
|
Option<String>,
|
||||||
|
Option<String>,
|
||||||
|
Option<String>,
|
||||||
|
);
|
||||||
|
let rows = sqlx::query_as::<_, Row>(
|
||||||
|
"SELECT id, original_path, preview_path, display_path, thumbnail_path FROM upload
|
||||||
|
WHERE deleted_at IS NOT NULL
|
||||||
|
AND CASE WHEN compression_status = 'failed'
|
||||||
|
THEN deleted_at < NOW() - ($1 || ' days')::interval
|
||||||
|
ELSE deleted_at < NOW() - ($2 || ' hours')::interval
|
||||||
|
END
|
||||||
|
AND (original_path <> '' OR preview_path IS NOT NULL
|
||||||
|
OR display_path IS NOT NULL OR thumbnail_path IS NOT NULL)",
|
||||||
)
|
)
|
||||||
.bind(FAILED_ORIGINAL_RETENTION_DAYS.to_string())
|
.bind(FAILED_ORIGINAL_RETENTION_DAYS.to_string())
|
||||||
|
.bind(DELETED_UPLOAD_RETENTION_HOURS.to_string())
|
||||||
.fetch_all(pool)
|
.fetch_all(pool)
|
||||||
.await;
|
.await;
|
||||||
|
|
||||||
let rows = match rows {
|
let rows = match rows {
|
||||||
Ok(r) => r,
|
Ok(r) => r,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(error = ?e, "failed-original sweep query failed");
|
tracing::warn!(error = ?e, "deleted-media sweep query failed");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
@@ -157,32 +200,52 @@ async fn cleanup_failed_originals(pool: &PgPool, media_path: &std::path::Path) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let mut reclaimed = 0usize;
|
let mut reclaimed = 0usize;
|
||||||
for (id, original_path) in rows {
|
for (id, original, preview, display, thumbnail) in rows {
|
||||||
let absolute = media_path.join(&original_path);
|
let paths: Vec<String> = std::iter::once(original)
|
||||||
|
.filter(|p| !p.is_empty())
|
||||||
|
.chain([preview, display, thumbnail].into_iter().flatten())
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
// All-or-nothing per row: the columns are only cleared once every file for that upload is
|
||||||
|
// gone. Clearing after a partial success would strand the survivors with nothing pointing
|
||||||
|
// at them — the same unowned-bytes state this sweep exists to drain.
|
||||||
|
let mut all_gone = true;
|
||||||
|
for rel in &paths {
|
||||||
|
let absolute = media_path.join(rel);
|
||||||
match tokio::fs::remove_file(&absolute).await {
|
match tokio::fs::remove_file(&absolute).await {
|
||||||
Ok(()) => reclaimed += 1,
|
Ok(()) => reclaimed += 1,
|
||||||
// Already gone (manual cleanup, restored backup) — still clear the column so
|
// Already gone (manual cleanup, restored backup) — still counts as reclaimed for
|
||||||
// the row stops being re-selected every hour.
|
// the purpose of clearing the columns, or the row is re-selected every hour forever.
|
||||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
|
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
tracing::warn!(error = ?e, %id, path = %absolute.display(),
|
tracing::warn!(error = ?e, %id, path = %absolute.display(),
|
||||||
"could not reclaim failed original; leaving the row for the next sweep");
|
"could not reclaim deleted media; leaving the row for the next sweep");
|
||||||
|
all_gone = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !all_gone {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
|
||||||
if let Err(e) = sqlx::query("UPDATE upload SET original_path = '' WHERE id = $1")
|
if let Err(e) = sqlx::query(
|
||||||
|
"UPDATE upload SET original_path = '', preview_path = NULL,
|
||||||
|
display_path = NULL, thumbnail_path = NULL
|
||||||
|
WHERE id = $1",
|
||||||
|
)
|
||||||
.bind(id)
|
.bind(id)
|
||||||
.execute(pool)
|
.execute(pool)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
tracing::warn!(error = ?e, %id, "reclaimed the file but could not clear original_path");
|
tracing::warn!(error = ?e, %id, "reclaimed the files but could not clear the paths");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if reclaimed > 0 {
|
if reclaimed > 0 {
|
||||||
tracing::info!(
|
tracing::info!(
|
||||||
"reclaimed {reclaimed} original(s) from uploads that failed compression more than \
|
"reclaimed {reclaimed} file(s) from soft-deleted uploads (deliberate deletes after \
|
||||||
{FAILED_ORIGINAL_RETENTION_DAYS} days ago"
|
{DELETED_UPLOAD_RETENTION_HOURS}h, compression failures after \
|
||||||
|
{FAILED_ORIGINAL_RETENTION_DAYS}d)"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,27 @@
|
|||||||
//! DB-backed tests for the failed-original sweep (`services/maintenance.rs`).
|
//! DB-backed tests for the deleted-media sweep (`services/maintenance.rs`).
|
||||||
//!
|
//!
|
||||||
//! Context: the compression worker deliberately no longer deletes an upload's original when
|
//! Context, in two halves.
|
||||||
//! its transcode fails — a transient ENOSPC or a codec panic must never destroy the only
|
|
||||||
//! copy of a photo a guest cannot retake. But the row is soft-deleted and the uploader's
|
|
||||||
//! quota IS refunded, so those bytes become invisible, unowned and free. A guest hitting a
|
|
||||||
//! reproducible codec failure could accumulate orphans at no personal cost, and since
|
|
||||||
//! `active_uploaders` counts only users with non-deleted uploads, dropping out of that count
|
|
||||||
//! actually RAISES everyone's per-user ceiling while the disk fills.
|
|
||||||
//!
|
//!
|
||||||
//! The sweep reclaims them after a retention window. Its selection predicate is the whole
|
//! The compression worker deliberately no longer deletes an upload's original when its transcode
|
||||||
//! safety argument — it must reach the give-up path's leftovers and nothing else — so that
|
//! fails — a transient ENOSPC or a codec panic must never destroy the only copy of a photo a guest
|
||||||
//! is what these tests pin, following the same "reproduce the SQL verbatim" pattern as
|
//! cannot retake. But the row is soft-deleted and the uploader's quota IS refunded, so those bytes
|
||||||
//! `upload_concurrency.rs`.
|
//! become invisible, unowned and free.
|
||||||
//!
|
//!
|
||||||
//! `#[sqlx::test]` gives each test a fresh database with the real migrations applied.
|
//! The SAME hole was reachable by the ordinary path, and that one is not an edge case at all:
|
||||||
|
//! `soft_delete_in_event` refunds `total_upload_bytes` on every guest or host delete and nothing
|
||||||
|
//! removed the files, so the quota stopped bounding the disk. Upload 500 MB, delete, quota back to
|
||||||
|
//! zero, upload another 500 MB — a guest curating their camera roll, which is what people do. The
|
||||||
|
//! sweep used to reach only `compression_status = 'failed'`, so it never touched this case; the
|
||||||
|
//! test below that now asserts an owner-deleted upload IS reclaimed is the one that used to assert
|
||||||
|
//! the opposite.
|
||||||
|
//!
|
||||||
|
//! Two windows, because the two deletes mean different things: 14 days for a failure an operator
|
||||||
|
//! may want to investigate, 24 hours for a removal someone asked for (14 days outlives the whole
|
||||||
|
//! event, so a deliberate delete would never reclaim anything while it mattered).
|
||||||
|
//!
|
||||||
|
//! The selection predicate is the whole safety argument — it must reach both leftovers and never a
|
||||||
|
//! live upload — so that is what these pin, following the same "reproduce the SQL verbatim" pattern
|
||||||
|
//! as `upload_concurrency.rs`. `#[sqlx::test]` gives each test a fresh, migrated database.
|
||||||
|
|
||||||
mod common;
|
mod common;
|
||||||
|
|
||||||
@@ -21,195 +29,324 @@ use common::*;
|
|||||||
use sqlx::PgPool;
|
use sqlx::PgPool;
|
||||||
use uuid::Uuid;
|
use uuid::Uuid;
|
||||||
|
|
||||||
/// SRC: `services/maintenance.rs::cleanup_failed_originals` — the selection, verbatim.
|
const FAILED_DAYS: i64 = 14;
|
||||||
async fn sweep_selects(pool: &PgPool, retention_days: i64) -> Vec<Uuid> {
|
const DELETED_HOURS: i64 = 24;
|
||||||
sqlx::query_as::<_, (Uuid, String)>(
|
|
||||||
"SELECT id, original_path FROM upload
|
/// SRC: `services/maintenance.rs::cleanup_deleted_media` — the selection, verbatim.
|
||||||
WHERE compression_status = 'failed'
|
async fn sweep_selects(pool: &PgPool, failed_days: i64, deleted_hours: i64) -> Vec<Uuid> {
|
||||||
AND deleted_at IS NOT NULL
|
type Row = (Uuid, String, Option<String>, Option<String>, Option<String>);
|
||||||
AND deleted_at < NOW() - ($1 || ' days')::interval
|
sqlx::query_as::<_, Row>(
|
||||||
AND original_path <> ''",
|
"SELECT id, original_path, preview_path, display_path, thumbnail_path FROM upload
|
||||||
|
WHERE deleted_at IS NOT NULL
|
||||||
|
AND CASE WHEN compression_status = 'failed'
|
||||||
|
THEN deleted_at < NOW() - ($1 || ' days')::interval
|
||||||
|
ELSE deleted_at < NOW() - ($2 || ' hours')::interval
|
||||||
|
END
|
||||||
|
AND (original_path <> '' OR preview_path IS NOT NULL
|
||||||
|
OR display_path IS NOT NULL OR thumbnail_path IS NOT NULL)",
|
||||||
)
|
)
|
||||||
.bind(retention_days.to_string())
|
.bind(failed_days.to_string())
|
||||||
|
.bind(deleted_hours.to_string())
|
||||||
.fetch_all(pool)
|
.fetch_all(pool)
|
||||||
.await
|
.await
|
||||||
.expect("sweep query")
|
.expect("sweep query")
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.map(|(id, _)| id)
|
.map(|(id, ..)| id)
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
#[allow(clippy::too_many_arguments)]
|
/// Seed an upload aged `deleted_hours_ago` (None = live), with optional derivative paths.
|
||||||
async fn seed_upload(
|
async fn seed_aged_upload(
|
||||||
pool: &PgPool,
|
pool: &PgPool,
|
||||||
event_id: Uuid,
|
event_id: Uuid,
|
||||||
user_id: Uuid,
|
user_id: Uuid,
|
||||||
status: &str,
|
status: &str,
|
||||||
deleted_days_ago: Option<i64>,
|
deleted_hours_ago: Option<i64>,
|
||||||
original_path: &str,
|
original_path: &str,
|
||||||
|
derivatives: bool,
|
||||||
) -> Uuid {
|
) -> Uuid {
|
||||||
let id: Uuid = sqlx::query_scalar(
|
sqlx::query_scalar(
|
||||||
"INSERT INTO upload (event_id, user_id, original_path, mime_type, original_size_bytes,
|
"INSERT INTO upload (event_id, user_id, original_path, mime_type, original_size_bytes,
|
||||||
compression_status, deleted_at)
|
compression_status, deleted_at,
|
||||||
|
preview_path, display_path, thumbnail_path)
|
||||||
VALUES ($1, $2, $3, 'image/jpeg', 1000, $4,
|
VALUES ($1, $2, $3, 'image/jpeg', 1000, $4,
|
||||||
CASE WHEN $5::bigint IS NULL THEN NULL
|
CASE WHEN $5::bigint IS NULL THEN NULL
|
||||||
ELSE NOW() - ($5::text || ' days')::interval END)
|
ELSE NOW() - ($5::text || ' hours')::interval END,
|
||||||
|
CASE WHEN $6 THEN 'previews/p.jpg' END,
|
||||||
|
CASE WHEN $6 THEN 'displays/d.jpg' END,
|
||||||
|
CASE WHEN $6 THEN 'thumbs/t.jpg' END)
|
||||||
RETURNING id",
|
RETURNING id",
|
||||||
)
|
)
|
||||||
.bind(event_id)
|
.bind(event_id)
|
||||||
.bind(user_id)
|
.bind(user_id)
|
||||||
.bind(original_path)
|
.bind(original_path)
|
||||||
.bind(status)
|
.bind(status)
|
||||||
.bind(deleted_days_ago)
|
.bind(deleted_hours_ago)
|
||||||
|
.bind(derivatives)
|
||||||
.fetch_one(pool)
|
.fetch_one(pool)
|
||||||
.await
|
.await
|
||||||
.expect("seed upload");
|
.expect("seed upload")
|
||||||
id
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A live upload is untouchable no matter how the windows are configured.
|
||||||
|
///
|
||||||
|
/// PREVENTS: the catastrophic loosening. Everything else here is about reclaiming more; this is the
|
||||||
|
/// one assertion that must never bend.
|
||||||
#[sqlx::test]
|
#[sqlx::test]
|
||||||
async fn sweeps_only_long_failed_soft_deleted_uploads(pool: PgPool) {
|
async fn a_live_upload_is_never_selected(pool: PgPool) {
|
||||||
let event_id = seed_event(&pool, "sweep-event").await;
|
let event_id = seed_event(&pool, "sweep-live").await;
|
||||||
let user_id = seed_user(&pool, event_id, "Sweeper").await;
|
let user_id = seed_user(&pool, event_id, "Sweeper").await;
|
||||||
|
|
||||||
// The one and only thing the sweep may touch: the exact state the compression worker's
|
for status in ["done", "failed", "processing", "pending"] {
|
||||||
// give-up path leaves behind, aged past the window.
|
let live = seed_aged_upload(
|
||||||
let target = seed_upload(
|
|
||||||
&pool,
|
&pool,
|
||||||
event_id,
|
event_id,
|
||||||
user_id,
|
user_id,
|
||||||
"failed",
|
status,
|
||||||
Some(30),
|
|
||||||
"originals/e/target.jpg",
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
|
|
||||||
// Everything below is a near-miss that must survive.
|
|
||||||
// A healthy live upload — the catastrophic case if the predicate were ever loosened.
|
|
||||||
let live = seed_upload(
|
|
||||||
&pool,
|
|
||||||
event_id,
|
|
||||||
user_id,
|
|
||||||
"done",
|
|
||||||
None,
|
None,
|
||||||
"originals/e/live.jpg",
|
"originals/e/live.jpg",
|
||||||
|
true,
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
// Failed but still inside the retention window: the recovery window is the entire point
|
assert!(
|
||||||
// of keeping the file, so reclaiming it early would defeat the fix it protects.
|
!sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS)
|
||||||
let recent = seed_upload(
|
.await
|
||||||
&pool,
|
.contains(&live),
|
||||||
event_id,
|
"a non-deleted upload with status {status} must never be swept"
|
||||||
user_id,
|
);
|
||||||
"failed",
|
}
|
||||||
Some(1),
|
}
|
||||||
"originals/e/recent.jpg",
|
|
||||||
)
|
/// THE FIX. An upload a guest or host deliberately deleted is reclaimed once past 24 hours.
|
||||||
.await;
|
///
|
||||||
// Failed but NOT soft-deleted — not the give-up path; something else set this status.
|
/// PREVENTS: the regression back to a sweep scoped to `compression_status = 'failed'`, which is
|
||||||
let failed_live = seed_upload(
|
/// what let the quota stop bounding the disk. This assertion is the inverse of the one this file
|
||||||
&pool,
|
/// used to make.
|
||||||
event_id,
|
#[sqlx::test]
|
||||||
user_id,
|
async fn a_deliberately_deleted_upload_is_reclaimed_after_a_day(pool: PgPool) {
|
||||||
"failed",
|
let event_id = seed_event(&pool, "sweep-deleted").await;
|
||||||
None,
|
let user_id = seed_user(&pool, event_id, "Curator").await;
|
||||||
"originals/e/failed-live.jpg",
|
|
||||||
)
|
let deleted = seed_aged_upload(
|
||||||
.await;
|
|
||||||
// Soft-deleted by the OWNER, compression fine. Its file is already gone; this row must
|
|
||||||
// never be re-processed.
|
|
||||||
let owner_deleted = seed_upload(
|
|
||||||
&pool,
|
&pool,
|
||||||
event_id,
|
event_id,
|
||||||
user_id,
|
user_id,
|
||||||
"done",
|
"done",
|
||||||
Some(30),
|
Some(48),
|
||||||
"originals/e/owner.jpg",
|
"originals/e/owner.jpg",
|
||||||
|
true,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
// Still inside the window — a mis-tap is recoverable for a day.
|
||||||
|
let recent = seed_aged_upload(
|
||||||
|
&pool,
|
||||||
|
event_id,
|
||||||
|
user_id,
|
||||||
|
"done",
|
||||||
|
Some(2),
|
||||||
|
"originals/e/recent.jpg",
|
||||||
|
true,
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
// Already swept: `original_path` cleared. Re-selecting it every hour would log a
|
|
||||||
// phantom reclaim forever.
|
|
||||||
let already_swept = seed_upload(&pool, event_id, user_id, "failed", Some(30), "").await;
|
|
||||||
|
|
||||||
let selected = sweep_selects(&pool, 14).await;
|
let selected = sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS).await;
|
||||||
|
assert!(
|
||||||
assert_eq!(
|
selected.contains(&deleted),
|
||||||
selected,
|
"a deliberate delete past the window must be reclaimed — this is the leak"
|
||||||
vec![target],
|
);
|
||||||
"the sweep must select exactly the aged give-up-path leftovers"
|
assert!(
|
||||||
|
!selected.contains(&recent),
|
||||||
|
"a delete inside the window keeps its recovery grace"
|
||||||
);
|
);
|
||||||
for (id, what) in [
|
|
||||||
(live, "a live upload"),
|
|
||||||
(recent, "a failure still inside the retention window"),
|
|
||||||
(failed_live, "a failed but not soft-deleted upload"),
|
|
||||||
(owner_deleted, "an owner-deleted upload"),
|
|
||||||
(already_swept, "an already-swept row"),
|
|
||||||
] {
|
|
||||||
assert!(!selected.contains(&id), "the sweep must not touch {what}");
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The two windows are independent: a failure is retained far longer than a deliberate delete.
|
||||||
|
///
|
||||||
|
/// PREVENTS: collapsing them into one. Applying 24h to failures would destroy the recovery window
|
||||||
|
/// the retained-original fix exists to provide; applying 14 days to deliberate deletes would mean
|
||||||
|
/// nothing is ever reclaimed during an event.
|
||||||
#[sqlx::test]
|
#[sqlx::test]
|
||||||
async fn retention_window_is_honoured_at_the_boundary(pool: PgPool) {
|
async fn the_two_retention_windows_do_not_bleed_into_each_other(pool: PgPool) {
|
||||||
|
let event_id = seed_event(&pool, "sweep-windows").await;
|
||||||
|
let user_id = seed_user(&pool, event_id, "Windows").await;
|
||||||
|
|
||||||
|
// 48h old: past the deliberate window, nowhere near the failure window.
|
||||||
|
let failed_recent = seed_aged_upload(
|
||||||
|
&pool,
|
||||||
|
event_id,
|
||||||
|
user_id,
|
||||||
|
"failed",
|
||||||
|
Some(48),
|
||||||
|
"originals/e/f-recent.jpg",
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let deleted_same_age = seed_aged_upload(
|
||||||
|
&pool,
|
||||||
|
event_id,
|
||||||
|
user_id,
|
||||||
|
"done",
|
||||||
|
Some(48),
|
||||||
|
"originals/e/d-same.jpg",
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
// 30 days old: past both.
|
||||||
|
let failed_old = seed_aged_upload(
|
||||||
|
&pool,
|
||||||
|
event_id,
|
||||||
|
user_id,
|
||||||
|
"failed",
|
||||||
|
Some(30 * 24),
|
||||||
|
"originals/e/f-old.jpg",
|
||||||
|
false,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let selected = sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS).await;
|
||||||
|
assert!(
|
||||||
|
!selected.contains(&failed_recent),
|
||||||
|
"a 2-day-old compression failure is still inside its 14-day recovery window"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
selected.contains(&deleted_same_age),
|
||||||
|
"a deliberate delete of the same age is past its 24-hour window"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
selected.contains(&failed_old),
|
||||||
|
"a 30-day-old failure is past both windows"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Boundary behaviour on both windows.
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn retention_windows_are_honoured_at_the_boundary(pool: PgPool) {
|
||||||
let event_id = seed_event(&pool, "sweep-boundary").await;
|
let event_id = seed_event(&pool, "sweep-boundary").await;
|
||||||
let user_id = seed_user(&pool, event_id, "Boundary").await;
|
let user_id = seed_user(&pool, event_id, "Boundary").await;
|
||||||
|
|
||||||
let inside = seed_upload(
|
let cases = [
|
||||||
|
("failed", 13 * 24, false, "13 days"),
|
||||||
|
("failed", 15 * 24, true, "15 days"),
|
||||||
|
("done", 23, false, "23 hours"),
|
||||||
|
("done", 25, true, "25 hours"),
|
||||||
|
];
|
||||||
|
for (status, hours, expected, label) in cases {
|
||||||
|
let id = seed_aged_upload(
|
||||||
&pool,
|
&pool,
|
||||||
event_id,
|
event_id,
|
||||||
user_id,
|
user_id,
|
||||||
"failed",
|
status,
|
||||||
Some(13),
|
Some(hours),
|
||||||
"originals/e/inside.jpg",
|
"originals/e/b.jpg",
|
||||||
|
false,
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
let outside = seed_upload(
|
assert_eq!(
|
||||||
&pool,
|
sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS)
|
||||||
event_id,
|
.await
|
||||||
user_id,
|
.contains(&id),
|
||||||
"failed",
|
expected,
|
||||||
Some(15),
|
"a {status} upload deleted {label} ago: expected swept={expected}"
|
||||||
"originals/e/outside.jpg",
|
|
||||||
)
|
|
||||||
.await;
|
|
||||||
|
|
||||||
let selected = sweep_selects(&pool, 14).await;
|
|
||||||
assert!(
|
|
||||||
selected.contains(&outside),
|
|
||||||
"15 days old must be past a 14-day window"
|
|
||||||
);
|
|
||||||
assert!(
|
|
||||||
!selected.contains(&inside),
|
|
||||||
"13 days old must still be retained"
|
|
||||||
);
|
);
|
||||||
|
sqlx::query("DELETE FROM upload WHERE id = $1")
|
||||||
|
.bind(id)
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.expect("clean up");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A row is re-selected until EVERY one of its paths is cleared.
|
||||||
|
///
|
||||||
|
/// PREVENTS: two failures at once. The sweep used to clear `original_path` alone, which was right
|
||||||
|
/// for its only case (a failed compression produces no derivatives) but leaves preview, display and
|
||||||
|
/// thumbnail on disk the moment it reaches a successfully processed upload — three files per
|
||||||
|
/// upload, none of them counted in `original_size_bytes`, that nothing else ever removes. And a row
|
||||||
|
/// whose paths are all cleared must stop coming back, or every hourly tick logs a phantom reclaim
|
||||||
|
/// forever.
|
||||||
#[sqlx::test]
|
#[sqlx::test]
|
||||||
async fn clearing_original_path_makes_the_sweep_idempotent(pool: PgPool) {
|
async fn a_row_is_reselected_until_every_path_is_cleared(pool: PgPool) {
|
||||||
// The sweep clears `original_path` after reclaiming the file. Without that, a row whose
|
|
||||||
// file is already gone is re-selected on every hourly tick forever.
|
|
||||||
let event_id = seed_event(&pool, "sweep-idempotent").await;
|
let event_id = seed_event(&pool, "sweep-idempotent").await;
|
||||||
let user_id = seed_user(&pool, event_id, "Idem").await;
|
let user_id = seed_user(&pool, event_id, "Idem").await;
|
||||||
let id = seed_upload(
|
let id = seed_aged_upload(
|
||||||
&pool,
|
&pool,
|
||||||
event_id,
|
event_id,
|
||||||
user_id,
|
user_id,
|
||||||
"failed",
|
"done",
|
||||||
Some(30),
|
Some(48),
|
||||||
"originals/e/once.jpg",
|
"originals/e/once.jpg",
|
||||||
|
true,
|
||||||
)
|
)
|
||||||
.await;
|
.await;
|
||||||
|
|
||||||
assert_eq!(sweep_selects(&pool, 14).await, vec![id]);
|
assert_eq!(sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS).await, [id]);
|
||||||
|
|
||||||
|
// Clearing only the original is NOT enough — the derivatives are still on disk.
|
||||||
sqlx::query("UPDATE upload SET original_path = '' WHERE id = $1")
|
sqlx::query("UPDATE upload SET original_path = '' WHERE id = $1")
|
||||||
.bind(id)
|
.bind(id)
|
||||||
.execute(&pool)
|
.execute(&pool)
|
||||||
.await
|
.await
|
||||||
.expect("clear path");
|
.expect("clear original");
|
||||||
|
assert_eq!(
|
||||||
|
sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS).await,
|
||||||
|
[id],
|
||||||
|
"derivatives left behind must keep the row selected"
|
||||||
|
);
|
||||||
|
|
||||||
|
sqlx::query(
|
||||||
|
"UPDATE upload SET preview_path = NULL, display_path = NULL, thumbnail_path = NULL
|
||||||
|
WHERE id = $1",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.expect("clear derivatives");
|
||||||
assert!(
|
assert!(
|
||||||
sweep_selects(&pool, 14).await.is_empty(),
|
sweep_selects(&pool, FAILED_DAYS, DELETED_HOURS)
|
||||||
"a swept row must not come back"
|
.await
|
||||||
|
.is_empty(),
|
||||||
|
"a fully swept row must not come back"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The derivative backfill must never resurrect what the sweep just reclaimed.
|
||||||
|
///
|
||||||
|
/// PREVENTS: an interaction, not a bug in either piece. The sweep nulls `preview_path`, and
|
||||||
|
/// `backfill_stale_derivatives` selects on `display_path IS NULL AND preview_path IS NOT NULL` —
|
||||||
|
/// close enough that a future edit to either could have the backfill re-decode an original that is
|
||||||
|
/// no longer on disk, on every boot. `deleted_at IS NULL` is what keeps them apart.
|
||||||
|
#[sqlx::test]
|
||||||
|
async fn the_backfill_ignores_swept_rows(pool: PgPool) {
|
||||||
|
let event_id = seed_event(&pool, "sweep-backfill").await;
|
||||||
|
let user_id = seed_user(&pool, event_id, "Backfill").await;
|
||||||
|
seed_aged_upload(
|
||||||
|
&pool,
|
||||||
|
event_id,
|
||||||
|
user_id,
|
||||||
|
"done",
|
||||||
|
Some(48),
|
||||||
|
"originals/e/gone.jpg",
|
||||||
|
true,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
|
||||||
|
// SRC: `services/compression.rs::backfill_stale_derivatives` — the selection, verbatim.
|
||||||
|
let backfilled: Vec<(Uuid, String, String)> = sqlx::query_as(
|
||||||
|
"SELECT id, original_path, mime_type FROM upload
|
||||||
|
WHERE deleted_at IS NULL AND mime_type LIKE 'image/%'
|
||||||
|
AND original_path IS NOT NULL
|
||||||
|
AND (
|
||||||
|
(display_path IS NULL AND preview_path IS NOT NULL)
|
||||||
|
OR derivatives_rev < $1
|
||||||
|
)",
|
||||||
|
)
|
||||||
|
.bind(1i16)
|
||||||
|
.fetch_all(&pool)
|
||||||
|
.await
|
||||||
|
.expect("backfill query");
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
backfilled.is_empty(),
|
||||||
|
"a soft-deleted row must be invisible to the backfill, before or after sweeping"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,7 +17,15 @@ services:
|
|||||||
deploy:
|
deploy:
|
||||||
resources:
|
resources:
|
||||||
limits:
|
limits:
|
||||||
memory: 512M
|
# 1G, not 512M. DATABASE_MAX_CONNECTIONS defaults to 30 for a ~100-guest event
|
||||||
|
# (feed polling + SSE + uploads at once), and 30 backends plus Postgres 16's
|
||||||
|
# default shared_buffers leaves very little headroom at 512M. An OOM here does
|
||||||
|
# not degrade one feature — it takes the event down, because every request
|
||||||
|
# path touches the database. Memory is the cheaper knob than shrinking the
|
||||||
|
# pool back and reintroducing the queueing it was raised to fix.
|
||||||
|
#
|
||||||
|
# Raising DATABASE_MAX_CONNECTIONS further means raising this too.
|
||||||
|
memory: 1G
|
||||||
|
|
||||||
app:
|
app:
|
||||||
build:
|
build:
|
||||||
|
|||||||
@@ -105,6 +105,20 @@ export const db = {
|
|||||||
});
|
});
|
||||||
},
|
},
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Overstate an upload's recorded size.
|
||||||
|
*
|
||||||
|
* The keepsake size estimate and the low-disk threshold are pure SQL over
|
||||||
|
* `original_size_bytes` — no file is read — so this is the lever for driving "the keepsake
|
||||||
|
* would not fit" without a genuinely full disk. The bytes on disk are unchanged; only the
|
||||||
|
* accounting the warning reads from moves.
|
||||||
|
*/
|
||||||
|
async setUploadSizeBytes(uploadId: string, bytes: number) {
|
||||||
|
await withClient((c) =>
|
||||||
|
c.query(`UPDATE upload SET original_size_bytes = $2 WHERE id = $1`, [uploadId, bytes])
|
||||||
|
);
|
||||||
|
},
|
||||||
|
|
||||||
async setExportReleased(slug: string, released: boolean) {
|
async setExportReleased(slug: string, released: boolean) {
|
||||||
await withClient((c) =>
|
await withClient((c) =>
|
||||||
c.query(`UPDATE event SET export_released_at = $2 WHERE slug = $1`, [
|
c.query(`UPDATE event SET export_released_at = $2 WHERE slug = $1`, [
|
||||||
|
|||||||
136
e2e/specs/03-feed/social-rate-limit.spec.ts
Normal file
136
e2e/specs/03-feed/social-rate-limit.spec.ts
Normal file
@@ -0,0 +1,136 @@
|
|||||||
|
/**
|
||||||
|
* Regression guard — likes, comments and comment deletions are rate limited.
|
||||||
|
*
|
||||||
|
* These were the only mutating endpoints in the app with no limit at all. Every other write path
|
||||||
|
* -- upload, join, recover, export, admin login -- carried one; `social.rs` carried none, so the
|
||||||
|
* coverage was asymmetric rather than deliberately open.
|
||||||
|
*
|
||||||
|
* Severity is genuinely low for an invited-guest event, and the amplification worry is contained:
|
||||||
|
* a like does fan an SSE broadcast to every connected client, but the export regeneration a
|
||||||
|
* comment deletion triggers is debounced (REGEN_DEBOUNCE 20s) and superseded workers are inert. So
|
||||||
|
* this closes the gap for symmetry, and the ceiling is set well above anything a real guest
|
||||||
|
* produces -- it bounds a script, not an enthusiastic double-tapper.
|
||||||
|
*
|
||||||
|
* The bucket is shared across all three actions on purpose: separate buckets would let a caller
|
||||||
|
* triple the aggregate write rate just by alternating between them. That is what the second test
|
||||||
|
* pins, and it is the part most likely to be lost in a refactor.
|
||||||
|
*
|
||||||
|
* Keyed per USER, not per IP — at a venue every guest is behind one NAT, so an IP key would hand
|
||||||
|
* the whole party one bucket. Third test.
|
||||||
|
*/
|
||||||
|
import { test, expect } from '../../fixtures/test';
|
||||||
|
import { seedUpload } from '../../helpers/seed';
|
||||||
|
import { BASE } from '../../helpers/env';
|
||||||
|
|
||||||
|
const like = (jwt: string, uploadId: string) =>
|
||||||
|
fetch(`${BASE}/api/v1/upload/${uploadId}/like`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { Authorization: `Bearer ${jwt}` },
|
||||||
|
});
|
||||||
|
|
||||||
|
const comment = (jwt: string, uploadId: string, body: string) =>
|
||||||
|
fetch(`${BASE}/api/v1/upload/${uploadId}/comments`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { Authorization: `Bearer ${jwt}`, 'Content-Type': 'application/json' },
|
||||||
|
body: JSON.stringify({ body }),
|
||||||
|
});
|
||||||
|
|
||||||
|
test.describe('Social — rate limit', () => {
|
||||||
|
test('a burst of likes past the ceiling returns 429 with Retry-After', async ({
|
||||||
|
api,
|
||||||
|
adminToken,
|
||||||
|
guest,
|
||||||
|
}) => {
|
||||||
|
await api.patchConfig(adminToken, {
|
||||||
|
rate_limits_enabled: 'true',
|
||||||
|
social_rate_enabled: 'true',
|
||||||
|
social_rate_per_min: '3',
|
||||||
|
});
|
||||||
|
|
||||||
|
const g = await guest('Tapper');
|
||||||
|
const uploadId = await seedUpload(g.jwt);
|
||||||
|
|
||||||
|
// Sequential, not parallel: a toggle flips state, so ordering matters for the assertion.
|
||||||
|
const statuses: number[] = [];
|
||||||
|
for (let i = 0; i < 5; i++) statuses.push((await like(g.jwt, uploadId)).status);
|
||||||
|
|
||||||
|
expect(statuses.slice(0, 3), 'the first three are within the ceiling').toEqual([200, 200, 200]);
|
||||||
|
expect(statuses.slice(3), 'everything past it is refused').toEqual([429, 429]);
|
||||||
|
|
||||||
|
const limited = await like(g.jwt, uploadId);
|
||||||
|
expect(limited.status).toBe(429);
|
||||||
|
expect(
|
||||||
|
limited.headers.get('retry-after'),
|
||||||
|
'a 429 without Retry-After tells the client nothing about when to come back'
|
||||||
|
).toBeTruthy();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('likes and comments share one bucket', async ({ api, adminToken, guest }) => {
|
||||||
|
// THE assertion. Per-action buckets would let a caller triple the aggregate write rate by
|
||||||
|
// alternating, which defeats the point of having a ceiling at all.
|
||||||
|
await api.patchConfig(adminToken, {
|
||||||
|
rate_limits_enabled: 'true',
|
||||||
|
social_rate_enabled: 'true',
|
||||||
|
social_rate_per_min: '2',
|
||||||
|
});
|
||||||
|
|
||||||
|
const g = await guest('Mixer');
|
||||||
|
const uploadId = await seedUpload(g.jwt);
|
||||||
|
|
||||||
|
expect((await like(g.jwt, uploadId)).status).toBe(200);
|
||||||
|
expect((await comment(g.jwt, uploadId, 'schön!')).status).toBe(201);
|
||||||
|
// Two writes spent, whichever endpoints they went to.
|
||||||
|
expect(
|
||||||
|
(await comment(g.jwt, uploadId, 'noch eins')).status,
|
||||||
|
'a comment must consume the same budget a like does'
|
||||||
|
).toBe(429);
|
||||||
|
expect((await like(g.jwt, uploadId)).status).toBe(429);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('one guest hitting the ceiling does not block another', async ({
|
||||||
|
api,
|
||||||
|
adminToken,
|
||||||
|
guest,
|
||||||
|
}) => {
|
||||||
|
// Keyed per user, not per IP. Every request in this suite comes from one address, which is
|
||||||
|
// exactly the venue-NAT shape that made the /join and /feed limits turn guests away.
|
||||||
|
await api.patchConfig(adminToken, {
|
||||||
|
rate_limits_enabled: 'true',
|
||||||
|
social_rate_enabled: 'true',
|
||||||
|
social_rate_per_min: '2',
|
||||||
|
});
|
||||||
|
|
||||||
|
const noisy = await guest('Noisy');
|
||||||
|
const quiet = await guest('Quiet');
|
||||||
|
const uploadId = await seedUpload(noisy.jwt);
|
||||||
|
|
||||||
|
for (let i = 0; i < 3; i++) await like(noisy.jwt, uploadId);
|
||||||
|
expect((await like(noisy.jwt, uploadId)).status).toBe(429);
|
||||||
|
|
||||||
|
expect(
|
||||||
|
(await like(quiet.jwt, uploadId)).status,
|
||||||
|
'a second guest behind the same IP must have their own budget'
|
||||||
|
).toBe(200);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('flipping social_rate_enabled off bypasses the limit', async ({
|
||||||
|
api,
|
||||||
|
adminToken,
|
||||||
|
guest,
|
||||||
|
}) => {
|
||||||
|
// The toggle has to actually be honoured, or the admin switch is decorative — the failure
|
||||||
|
// mode two other per-area toggles already shipped with.
|
||||||
|
await api.patchConfig(adminToken, {
|
||||||
|
rate_limits_enabled: 'true',
|
||||||
|
social_rate_enabled: 'false',
|
||||||
|
social_rate_per_min: '2',
|
||||||
|
});
|
||||||
|
|
||||||
|
const g = await guest('Unlimited');
|
||||||
|
const uploadId = await seedUpload(g.jwt);
|
||||||
|
|
||||||
|
const statuses: number[] = [];
|
||||||
|
for (let i = 0; i < 6; i++) statuses.push((await like(g.jwt, uploadId)).status);
|
||||||
|
expect(statuses.every((s) => s === 200)).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -40,10 +40,19 @@ test.describe('Video — the lightbox plays it', () => {
|
|||||||
page,
|
page,
|
||||||
guest,
|
guest,
|
||||||
signIn,
|
signIn,
|
||||||
|
db,
|
||||||
}) => {
|
}) => {
|
||||||
const g = await guest('VideoWatcher');
|
const g = await guest('VideoWatcher');
|
||||||
const id = await seedVideo(g.jwt);
|
const id = await seedVideo(g.jwt);
|
||||||
|
|
||||||
|
// The poster assertion below needs the ffmpeg thumbnail to EXIST — the lightbox binds
|
||||||
|
// `poster={upload.thumbnail_url ?? undefined}`, so the attribute is simply absent until
|
||||||
|
// compression finishes. Without this wait the test races the worker and fails against a
|
||||||
|
// cold stack (first run after `stack:down -v`, cold ffmpeg), which is exactly when a suite
|
||||||
|
// is least likely to be believed. The `src` assertion is unconditional; only the poster
|
||||||
|
// needs the wait.
|
||||||
|
await expect.poll(() => db.compressionStatus(id), { timeout: 60_000 }).toBe('done');
|
||||||
|
|
||||||
await signIn(page, g);
|
await signIn(page, g);
|
||||||
await page.goto('/feed');
|
await page.goto('/feed');
|
||||||
|
|
||||||
|
|||||||
97
e2e/specs/04-host/low-disk-warning.spec.ts
Normal file
97
e2e/specs/04-host/low-disk-warning.spec.ts
Normal file
@@ -0,0 +1,97 @@
|
|||||||
|
/**
|
||||||
|
* Regression guard — the host is warned about storage BEFORE it becomes unrecoverable.
|
||||||
|
*
|
||||||
|
* Storage visibility used to exist in exactly one place: a passive "Speicherauslastung" widget on
|
||||||
|
* the ADMIN dashboard. A host who isn't the admin had no view of it at all, and nothing anywhere
|
||||||
|
* warned anyone. README listed a low-disk alert under "Planned (v1.x)".
|
||||||
|
*
|
||||||
|
* Two things make that a safety net rather than a nice-to-have:
|
||||||
|
*
|
||||||
|
* - `postgres_data`, `media_data` and `exports_data` are all Docker named volumes on ONE
|
||||||
|
* filesystem. A full disk doesn't degrade a subsystem; Postgres stops being able to write and
|
||||||
|
* the whole event goes down.
|
||||||
|
* - The keepsake needs room for TWO gallery-sized archives (both write their media
|
||||||
|
* `Compression::Stored`; `Memories.zip` streams the original for every video and every image
|
||||||
|
* at or under 5 MB). The export preflight can refuse cleanly, but only AFTER the release —
|
||||||
|
* when the event is over, the gallery is full, and every remedy is harder.
|
||||||
|
*
|
||||||
|
* So the threshold is deliberately NOT a fixed number alone. It fires on an absolute floor (10 GB,
|
||||||
|
* the figure the README always carried) OR on "you could not build the keepsake right now", which
|
||||||
|
* is the trigger a host can still act on.
|
||||||
|
*
|
||||||
|
* These drive it through `original_size_bytes` rather than a genuinely full disk: the estimate is
|
||||||
|
* pure SQL over that column, so overstating one row moves the accounting the warning reads without
|
||||||
|
* touching a byte on disk.
|
||||||
|
*/
|
||||||
|
import { test, expect } from '../../fixtures/test';
|
||||||
|
import { seedUpload } from '../../helpers/seed';
|
||||||
|
import { BASE } from '../../helpers/env';
|
||||||
|
|
||||||
|
/** Comfortably larger than any disk this suite could run on. */
|
||||||
|
const ABSURD_BYTES = 500_000_000_000_000;
|
||||||
|
|
||||||
|
test.describe('Host — low-disk warning', () => {
|
||||||
|
test('a gallery too big to export warns the host, with the numbers', async ({
|
||||||
|
page,
|
||||||
|
host,
|
||||||
|
guest,
|
||||||
|
signIn,
|
||||||
|
db,
|
||||||
|
}) => {
|
||||||
|
const g = await guest('BigShooter');
|
||||||
|
const uploadId = await seedUpload(g.jwt);
|
||||||
|
await db.setUploadSizeBytes(uploadId, ABSURD_BYTES);
|
||||||
|
|
||||||
|
await signIn(page, host);
|
||||||
|
await page.goto('/host');
|
||||||
|
|
||||||
|
const warning = page.getByTestId('low-disk-warning');
|
||||||
|
await expect(warning, 'the host must be warned before releasing').toBeVisible({
|
||||||
|
timeout: 15_000,
|
||||||
|
});
|
||||||
|
// The actionable half: not just "low", but "the keepsake cannot be built".
|
||||||
|
await expect(warning).toContainText(/nicht.*erstellt werden/i);
|
||||||
|
// And the consequence that makes it urgent — the event, not just the download.
|
||||||
|
await expect(warning).toContainText(/gesamte Event/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the API reports the requirement and the verdict together', async ({ host, guest, db }) => {
|
||||||
|
const g = await guest('BigShooter2');
|
||||||
|
const uploadId = await seedUpload(g.jwt);
|
||||||
|
await db.setUploadSizeBytes(uploadId, ABSURD_BYTES);
|
||||||
|
|
||||||
|
const res = await fetch(`${BASE}/api/v1/host/event`, {
|
||||||
|
headers: { Authorization: `Bearer ${host.jwt}` },
|
||||||
|
});
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
const body = (await res.json()) as {
|
||||||
|
disk_low: boolean;
|
||||||
|
disk_free_bytes: number | null;
|
||||||
|
keepsake_required_bytes: number;
|
||||||
|
};
|
||||||
|
|
||||||
|
expect(body.disk_low).toBe(true);
|
||||||
|
expect(
|
||||||
|
body.keepsake_required_bytes,
|
||||||
|
'both halves are armed by a release, so the requirement covers two archives'
|
||||||
|
).toBeGreaterThan(ABSURD_BYTES);
|
||||||
|
expect(body.disk_free_bytes).not.toBeNull();
|
||||||
|
expect(body.keepsake_required_bytes).toBeGreaterThan(body.disk_free_bytes!);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('an ordinary gallery shows no warning at all', async ({ page, host, guest, signIn }) => {
|
||||||
|
// The mirror that keeps the above honest. A warning that is always on is a warning nobody
|
||||||
|
// reads — and it would sit at the very top of the dashboard, above the PIN-reset queue.
|
||||||
|
const g = await guest('NormalShooter');
|
||||||
|
await seedUpload(g.jwt);
|
||||||
|
|
||||||
|
await signIn(page, host);
|
||||||
|
await page.goto('/host');
|
||||||
|
|
||||||
|
// Wait for the dashboard to actually be loaded before asserting on an absence.
|
||||||
|
await expect(page.getByRole('heading', { name: 'Host-Dashboard' })).toBeVisible({
|
||||||
|
timeout: 15_000,
|
||||||
|
});
|
||||||
|
await expect(page.getByTestId('low-disk-warning')).toHaveCount(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -84,9 +84,16 @@
|
|||||||
{ key: 'feed_rate_enabled', label: 'Feed-Limit aktiv', kind: 'bool' },
|
{ key: 'feed_rate_enabled', label: 'Feed-Limit aktiv', kind: 'bool' },
|
||||||
{ key: 'export_rate_enabled', label: 'Export-Limit aktiv', kind: 'bool' },
|
{ key: 'export_rate_enabled', label: 'Export-Limit aktiv', kind: 'bool' },
|
||||||
{ key: 'join_rate_enabled', label: 'Join-Limit aktiv', kind: 'bool' },
|
{ key: 'join_rate_enabled', label: 'Join-Limit aktiv', kind: 'bool' },
|
||||||
|
{ key: 'social_rate_enabled', label: 'Interaktions-Limit aktiv', kind: 'bool' },
|
||||||
{ key: 'upload_rate_per_hour', label: 'Upload-Limit pro Stunde', kind: 'number' },
|
{ key: 'upload_rate_per_hour', label: 'Upload-Limit pro Stunde', kind: 'number' },
|
||||||
{ key: 'feed_rate_per_min', label: 'Feed-Anfragen pro Minute', kind: 'number' },
|
{ key: 'feed_rate_per_min', label: 'Feed-Anfragen pro Minute', kind: 'number' },
|
||||||
{ key: 'export_rate_per_day', label: 'Export-Downloads pro Tag', kind: 'number' }
|
{ key: 'export_rate_per_day', label: 'Export-Downloads pro Tag', kind: 'number' },
|
||||||
|
{
|
||||||
|
key: 'social_rate_per_min',
|
||||||
|
label: 'Interaktionen pro Minute',
|
||||||
|
kind: 'number',
|
||||||
|
hint: 'Likes, Kommentare und Kommentar-Löschungen zusammen, pro Gast. Bewusst hoch angesetzt — soll ein Skript bremsen, keinen begeisterten Gast.'
|
||||||
|
}
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -105,7 +112,20 @@
|
|||||||
kind: 'bool',
|
kind: 'bool',
|
||||||
hint: 'Reserviert für künftige Anzahl-Limits.'
|
hint: 'Reserviert für künftige Anzahl-Limits.'
|
||||||
},
|
},
|
||||||
{ key: 'quota_tolerance', label: 'Toleranz (0–1)', kind: 'number' },
|
{
|
||||||
|
key: 'quota_tolerance',
|
||||||
|
label: 'Speicher-Anteil für Gäste (0–1)',
|
||||||
|
kind: 'number',
|
||||||
|
// "Toleranz (0–1)" with no hint invited exactly the wrong reading — that a higher
|
||||||
|
// number means "warn me later". It is the multiplier in
|
||||||
|
// `floor(freier Speicher × Anteil / aktive Uploader)`, so raising it authorises
|
||||||
|
// guests to fill MORE of the disk, not less.
|
||||||
|
hint:
|
||||||
|
'Anteil des freien Speichers, den alle Gäste zusammen belegen dürfen: ' +
|
||||||
|
'Limit = freier Speicher × Anteil ÷ aktive Uploader. Kein Warnschwellenwert — ' +
|
||||||
|
'ein höherer Wert gibt MEHR Speicher frei. Das Keepsake braucht zusätzlich ' +
|
||||||
|
'etwa das Doppelte der Mediengröße; 0,75 ist der getestete Standard.'
|
||||||
|
},
|
||||||
{ key: 'estimated_guest_count', label: 'Geschätzte Gästezahl', kind: 'number' }
|
{ key: 'estimated_guest_count', label: 'Geschätzte Gästezahl', kind: 'number' }
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -28,6 +28,9 @@
|
|||||||
is_active: boolean;
|
is_active: boolean;
|
||||||
uploads_locked: boolean;
|
uploads_locked: boolean;
|
||||||
export_released: boolean;
|
export_released: boolean;
|
||||||
|
disk_free_bytes: number | null;
|
||||||
|
keepsake_required_bytes: number;
|
||||||
|
disk_low: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
interface PinResetRequest {
|
interface PinResetRequest {
|
||||||
@@ -395,7 +398,11 @@
|
|||||||
function formatBytes(bytes: number): string {
|
function formatBytes(bytes: number): string {
|
||||||
if (bytes < 1024) return `${bytes} B`;
|
if (bytes < 1024) return `${bytes} B`;
|
||||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
|
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
|
||||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
|
// GB matters here now that this also renders free disk and keepsake size — the previous
|
||||||
|
// version topped out at MB, so 30 GB free read as "30720.0 MB" (and a guest with 2 GB of
|
||||||
|
// uploads was already being rendered the same way in the user list).
|
||||||
|
if (bytes < 1024 * 1024 * 1024) return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
|
||||||
|
return `${(bytes / (1024 * 1024 * 1024)).toFixed(1)} GB`;
|
||||||
}
|
}
|
||||||
</script>
|
</script>
|
||||||
|
|
||||||
@@ -507,6 +514,38 @@
|
|||||||
{error}
|
{error}
|
||||||
</div>
|
</div>
|
||||||
{:else if event}
|
{:else if event}
|
||||||
|
<!-- ── Speicherwarnung ─────────────────────────────────────────────
|
||||||
|
Above everything else on purpose. All three volumes (postgres_data, media_data,
|
||||||
|
exports_data) sit on one filesystem, so running out doesn't degrade a subsystem —
|
||||||
|
it stops Postgres writing and takes the event down. And the keepsake needs room
|
||||||
|
for TWO gallery-sized archives, which is only actionable BEFORE the release: the
|
||||||
|
export preflight can say "this didn't fit", but by then the event is over and the
|
||||||
|
remedies are all much harder.
|
||||||
|
|
||||||
|
Only the admin dashboard had any storage visibility at all, and a host is often
|
||||||
|
not the admin. `disk_low` fails closed to "not low" on an unreadable mount, so
|
||||||
|
this cannot cry wolf. -->
|
||||||
|
{#if event.disk_low && event.disk_free_bytes !== null}
|
||||||
|
<div
|
||||||
|
class="rounded-xl border border-red-300 bg-red-50 p-4 dark:border-red-800 dark:bg-red-950/30"
|
||||||
|
data-testid="low-disk-warning"
|
||||||
|
>
|
||||||
|
<h2 class="font-semibold text-red-900 dark:text-red-200">Speicherplatz wird knapp</h2>
|
||||||
|
<p class="mt-1 text-sm text-red-800 dark:text-red-300">
|
||||||
|
Noch <strong>{formatBytes(event.disk_free_bytes)}</strong> frei.
|
||||||
|
{#if event.keepsake_required_bytes > event.disk_free_bytes}
|
||||||
|
Für das Keepsake werden derzeit ca.
|
||||||
|
<strong>{formatBytes(event.keepsake_required_bytes)}</strong> benötigt — es kann
|
||||||
|
momentan <strong>nicht</strong> erstellt werden.
|
||||||
|
{/if}
|
||||||
|
</p>
|
||||||
|
<p class="mt-1.5 text-xs text-red-700 dark:text-red-400">
|
||||||
|
Schaffe Speicher frei oder vergrößere den Datenträger. Wenn der Datenträger vollläuft,
|
||||||
|
fällt das gesamte Event aus — nicht nur der Download.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
{/if}
|
||||||
|
|
||||||
<!-- ── PIN-Reset-Anfragen ──────────────────────────────────────── -->
|
<!-- ── PIN-Reset-Anfragen ──────────────────────────────────────── -->
|
||||||
{#if pinResetRequests.length > 0}
|
{#if pinResetRequests.length > 0}
|
||||||
<div
|
<div
|
||||||
|
|||||||
Reference in New Issue
Block a user