fabi/EventSnap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(compression): reclaim failed originals instead of leaking them

Browse Source 
Round 1 stopped the compression worker deleting an upload's original on failure —
a transient ENOSPC or a codec panic must never destroy the only copy of a photo a
guest cannot retake. But it left `Upload::soft_delete`'s quota refund in place, so
the bytes stayed on disk while the uploader was charged nothing for them.

That is worse than it first looks. The row is soft-deleted, so the file is
invisible and unowned; a guest hitting a reproducible codec failure can accumulate
orphans indefinitely at zero personal cost. And `active_uploaders` counts only
users with non-deleted uploads, so dropping out of that count RAISES everyone's
per-user ceiling — the leak loosens the very quota meant to contain it.

Keep the refund: the uploader didn't cause the failure and shouldn't silently lose
quota to it. Bound the leak instead, with an hourly sweep alongside the existing
session cleanup in `spawn_periodic_tasks`, reclaiming failed originals older than
14 days — comfortably longer than any single event, so an operator investigating a
failed upload still has the file.

The selection predicate is the entire safety argument, so it is deliberately
narrow: `compression_status = 'failed'` AND soft-deleted AND past the window AND
`original_path <> ''`. That is exactly the state the give-up path leaves behind,
and it cannot reach a live upload, an owner-deleted one, or a failure still inside
its recovery window. `original_path` is cleared after a successful reclaim, which
makes the sweep idempotent — otherwise a row whose file is already gone is
re-selected on every tick forever. The row itself is kept as the audit trail.

Tests reproduce the selection verbatim (same pattern as upload_concurrency) and
assert it against five near-misses that must survive, both sides of the retention
boundary, and the idempotence property.

Also fixes two comments in export.rs still claiming "the compression worker
hard-deletes an original when its transcode fails" — no longer true, and the
defensive handling they justify is now justified by this sweep and by ordinary
deletes instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
fabi
2026-07-28 20:40:15 +02:00
parent 9c8cc7c069
commit c14ccd2df1
4 changed files with 305 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
backend/src/main.rs
View File

@@ -70,6 +70,7 @@ async fn main() -> Result<()> {
pool,
state.rate_limiter.clone(),
state.sse_tickets.clone(),
config.media_path.clone(),
);
// Ensure media directories exist

15
backend/src/services/export.rs
View File

@@ -720,10 +720,11 @@ async fn run_html_export_inner(
let src = media_path.join(&row.original_path);
// Stat ONCE, up front, and skip this upload if the source is gone. The old code probed with
// `exists()` here and then did `metadata(&src).await?` further down — a TOCTOU whose `?`
// aborted the ENTIRE keepsake if the file vanished in between. It genuinely can: the
// compression worker hard-deletes an original when its transcode fails, and it can still be
// running when the gallery is released. A missing source must degrade one entry, never the
// whole archive (which, once released, the host cannot rebuild without reopening uploads).
// aborted the ENTIRE keepsake if the file vanished in between. It can still happen: the
// compression worker no longer deletes originals on failure, but the hourly sweep reclaims
// them once past the retention window, and an owner or host delete can land mid-export. A
// missing source must degrade one entry, never the whole archive (which, once released, the
// host cannot rebuild without reopening uploads).
let src_meta = match tokio::fs::metadata(&src).await {
Ok(m) => m,
Err(e) => {
@@ -953,9 +954,9 @@ async fn run_html_export_inner(
for (name, source) in &media_manifest {
let path = source.path();
// Open-first: a source that disappeared between the manifest being built and now (the
// compression worker deletes originals on transcode failure) must skip this entry, not
// fail the whole viewer. Opening collapses the check and the use into one operation.
// Open-first: a source that disappeared between the manifest being built and now (a
// delete, or the hourly sweep reclaiming a long-failed original) must skip this entry,
// not fail the whole viewer. Opening collapses the check and the use into one operation.
let src_file = match tokio::fs::File::open(path).await {
Ok(f) => f,
Err(e) => {

92
backend/src/services/maintenance.rs
View File

@@ -9,10 +9,12 @@
//! users staring at a spinner. Resetting them on startup recovers gracefully.
//!
//! 2. **Periodic tasks** — pruning that should happen "every hour" rather than per
//! request: expired sessions (otherwise the table grows unboundedly), and the
//! request: expired sessions (otherwise the table grows unboundedly), the
//! rate-limiter's in-memory windows (so keys for IPs that left long ago don't
//! accumulate).
//! accumulate), and the originals of uploads whose compression permanently failed
//! (which are deliberately retained for a recovery window, then reclaimed).
use std::path::PathBuf;
use std::time::Duration;
use sqlx::PgPool;
@@ -20,6 +22,21 @@ use sqlx::PgPool;
use crate::services::rate_limiter::RateLimiter;
use crate::services::sse_tickets::SseTicketStore;
/// How long a permanently-failed upload's original is kept on disk before it is
/// reclaimed.
///
/// The compression worker stops deleting originals on failure — a transient error must
/// never destroy the guest's only copy of a photo they can't retake. But the row is
/// soft-deleted and the uploader's quota IS refunded, so without a sweep those bytes are
/// invisible, unowned, and free: a reproducible codec failure lets one guest accumulate
/// orphans at no personal cost, and because `active_uploaders` counts only users with
/// non-deleted uploads, dropping out of that count actually RAISES everyone's per-user
/// ceiling while the disk gets fuller.
///
/// Two weeks is comfortably longer than any single event, so an operator investigating a
/// failed upload still has the file, while the leak stays bounded.
const FAILED_ORIGINAL_RETENTION_DAYS: i64 = 14;
/// Reset rows left in flight by a previous crashed instance. Run once on startup,
/// before the HTTP server starts taking requests, so users never observe the
/// half-state.
@@ -87,7 +104,12 @@ pub async fn startup_recovery(pool: &PgPool) {
/// - drops expired SSE tickets (30s TTL but the map keeps the slot until pruned)
///
/// Cadence is 1h — fine for both jobs at our scale.
pub fn spawn_periodic_tasks(pool: PgPool, rate_limiter: RateLimiter, sse_tickets: SseTicketStore) {
pub fn spawn_periodic_tasks(
pool: PgPool,
rate_limiter: RateLimiter,
sse_tickets: SseTicketStore,
media_path: PathBuf,
) {
tokio::spawn(async move {
let mut tick = tokio::time::interval(Duration::from_secs(3600));
// Fire the first tick immediately, then hourly.
@@ -95,12 +117,76 @@ pub fn spawn_periodic_tasks(pool: PgPool, rate_limiter: RateLimiter, sse_tickets
loop {
tick.tick().await;
cleanup_sessions(&pool).await;
cleanup_failed_originals(&pool, &media_path).await;
rate_limiter.prune();
sse_tickets.prune();
}
});
}
/// Reclaim the originals of uploads whose compression permanently failed, once they are
/// past [`FAILED_ORIGINAL_RETENTION_DAYS`].
///
/// Deliberately narrow. It only touches rows that are BOTH `compression_status = 'failed'`
/// AND soft-deleted — i.e. the exact state the compression worker's give-up path leaves
/// behind — so it can never reach a live upload or one whose preview works. `original_path`
/// is cleared in the same pass, which makes the sweep idempotent and stops a later run
/// re-reporting a file that is already gone. The row itself is kept: it is the audit trail
/// for the failure, and it costs a few hundred bytes.
async fn cleanup_failed_originals(pool: &PgPool, media_path: &std::path::Path) {
let rows = sqlx::query_as::<_, (uuid::Uuid, String)>(
"SELECT id, original_path FROM upload
WHERE compression_status = 'failed'
AND deleted_at IS NOT NULL
AND deleted_at < NOW() - ($1 || ' days')::interval
AND original_path <> ''",
)
.bind(FAILED_ORIGINAL_RETENTION_DAYS.to_string())
.fetch_all(pool)
.await;
let rows = match rows {
Ok(r) => r,
Err(e) => {
tracing::warn!(error = ?e, "failed-original sweep query failed");
return;
}
};
if rows.is_empty() {
return;
}
let mut reclaimed = 0usize;
for (id, original_path) in rows {
let absolute = media_path.join(&original_path);
match tokio::fs::remove_file(&absolute).await {
Ok(()) => reclaimed += 1,
// Already gone (manual cleanup, restored backup) — still clear the column so
// the row stops being re-selected every hour.
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {}
Err(e) => {
tracing::warn!(error = ?e, %id, path = %absolute.display(),
"could not reclaim failed original; leaving the row for the next sweep");
continue;
}
}
if let Err(e) = sqlx::query("UPDATE upload SET original_path = '' WHERE id = $1")
.bind(id)
.execute(pool)
.await
{
tracing::warn!(error = ?e, %id, "reclaimed the file but could not clear original_path");
}
}
if reclaimed > 0 {
tracing::info!(
"reclaimed {reclaimed} original(s) from uploads that failed compression more than \
{FAILED_ORIGINAL_RETENTION_DAYS} days ago"
);
}
}
async fn cleanup_sessions(pool: &PgPool) {
match sqlx::query("DELETE FROM session WHERE expires_at < NOW() - INTERVAL '1 day'")
.execute(pool)