fix(upload): reclaim the bytes of uploads that never finish
Reclaim was a dozen explicit remove_file calls on the handler's return paths. That covers every way the handler can FINISH and none of the ways it can simply STOP. When a client disconnects mid-body — a phone leaving wifi, iOS killing a backgrounded PWA, the user hitting back — axum drops the handler future at an .await inside field.chunk() and no return path runs at all. The partial file then survives forever: it has no upload row, so cleanup_deleted_media (row-driven) can never see it, and no sweeper covered the originals directory. The triggers are routine rather than adversarial, and the client keeps the blob and auto- retries on every `online` event, so one large video over bad wifi leaves several copies. Those bytes are also invisible to the quota while still consuming the free disk that compute_storage_quota divides among guests — so orphans silently shrink every guest's ceiling while the admin widget under-reports. All three volumes share one filesystem; the end state is Postgres unable to write WAL. TempFileGuard is an RAII guard, because dropping the future is exactly what runs Drop — it is the only construct that survives cancellation. Armed before the file can exist, disarmed only after tx.commit() succeeds. The twelve explicit cleanups are deleted so one owner holds the rule. The subtler half is the rename. It happens BEFORE the commit, so between them the file exists under its final name with no row pointing at it — an orphan that looks legitimate. The guard is RETARGETED there rather than disarmed, and the retarget sits on the same poll as the rename with no .await between, which is what makes that window uncancellable. sweep_orphan_originals is the backstop for the process that was killed, where no Drop can run at all. Hourly, alongside the existing sweeps: .tmp files past the window go unconditionally (a .tmp never has a row by construction), other files are batched 500 at a time through a single NOT EXISTS query. Two things that look like oversights and are not, both commented in place: - the 6h window is what makes the sweep safe against the rename-before-commit ordering, since a committing upload is briefly indistinguishable from an orphan. It must not be shortened to speed up a test. - the NOT EXISTS deliberately does NOT filter deleted_at IS NULL. A soft-deleted row still points at its file during its retention window, and reclaiming that is cleanup_deleted_media's job; filtering here would race the two sweeps and destroy the files the recovery window exists to preserve. Verified against the real schema: given a live original, a soft-deleted one and a true orphan, the query returns only the orphan. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
@@ -120,6 +120,19 @@ pub async fn startup_recovery(pool: &PgPool) {
|
||||
}
|
||||
}
|
||||
|
||||
/// How long a file in `originals/` may exist without a database row before it is treated as
|
||||
/// abandoned.
|
||||
///
|
||||
/// This window is the ONLY thing making the sweep safe, because the upload handler renames the
|
||||
/// temp file into its final path BEFORE committing the row: for a short moment a perfectly
|
||||
/// healthy upload legitimately looks exactly like an orphan. Six hours is far beyond any live
|
||||
/// request (a 576 MiB body over a bad venue uplink is minutes, and the request itself is bounded
|
||||
/// by the reverse proxy) while still reclaiming the leak inside a single event.
|
||||
///
|
||||
/// DO NOT SHORTEN THIS to make a test faster — a value below the longest possible in-flight
|
||||
/// upload deletes photos out from under the request that is committing them.
|
||||
const ORPHAN_UPLOAD_RETENTION_HOURS: u64 = 6;
|
||||
|
||||
/// Spawns a background task that periodically:
|
||||
/// - deletes session rows whose `expires_at` is more than a day in the past
|
||||
/// - prunes the in-memory rate-limiter HashMap of empty windows
|
||||
@@ -140,6 +153,7 @@ pub fn spawn_periodic_tasks(
|
||||
tick.tick().await;
|
||||
cleanup_sessions(&pool).await;
|
||||
cleanup_deleted_media(&pool, &media_path).await;
|
||||
sweep_orphan_originals(&pool, &media_path).await;
|
||||
rate_limiter.prune();
|
||||
sse_tickets.prune();
|
||||
}
|
||||
@@ -262,3 +276,123 @@ async fn cleanup_sessions(pool: &PgPool) {
|
||||
Err(e) => tracing::warn!("session cleanup failed: {e:#}"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Reclaim files in `originals/` that no upload row references.
|
||||
///
|
||||
/// The backstop behind [`TempFileGuard`](crate::handlers::upload). The guard covers the
|
||||
/// process that is running; this covers the process that was killed — a SIGKILL, an OOM, or a
|
||||
/// power cut leaves whatever bytes had been written with no `Drop` to reclaim them, and those
|
||||
/// files are then permanently invisible: they have no row, so `cleanup_deleted_media` (which is
|
||||
/// row-driven) can never see them, and they are not counted against any quota while still
|
||||
/// consuming the free disk that `compute_storage_quota` divides among guests. On a single box
|
||||
/// where all three volumes share a filesystem, that ends with Postgres unable to write WAL.
|
||||
///
|
||||
/// Two classes:
|
||||
/// - `*.tmp` — an upload that never got as far as being renamed. Always safe past the window.
|
||||
/// - everything else — a final-named original whose commit never happened.
|
||||
async fn sweep_orphan_originals(pool: &PgPool, media_path: &std::path::Path) {
|
||||
let originals = media_path.join("originals");
|
||||
let cutoff = Duration::from_secs(ORPHAN_UPLOAD_RETENTION_HOURS * 3600);
|
||||
|
||||
// originals/{event_slug}/{uuid}.{ext} — one level of per-event directories.
|
||||
let mut event_dirs = match tokio::fs::read_dir(&originals).await {
|
||||
Ok(rd) => rd,
|
||||
// Nothing uploaded yet; the directory is created lazily by the upload handler.
|
||||
Err(_) => return,
|
||||
};
|
||||
|
||||
let mut candidates: Vec<(String, std::path::PathBuf)> = Vec::new();
|
||||
let mut temps_removed = 0u32;
|
||||
|
||||
while let Ok(Some(event_dir)) = event_dirs.next_entry().await {
|
||||
if !event_dir
|
||||
.file_type()
|
||||
.await
|
||||
.map(|t| t.is_dir())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
let slug = event_dir.file_name().to_string_lossy().to_string();
|
||||
let Ok(mut files) = tokio::fs::read_dir(event_dir.path()).await else {
|
||||
continue;
|
||||
};
|
||||
while let Ok(Some(entry)) = files.next_entry().await {
|
||||
let Ok(meta) = entry.metadata().await else {
|
||||
continue;
|
||||
};
|
||||
if !meta.is_file() {
|
||||
continue;
|
||||
}
|
||||
// Too young to judge: an upload committing RIGHT NOW is indistinguishable from an
|
||||
// orphan, because the rename precedes the commit.
|
||||
let recent = meta
|
||||
.modified()
|
||||
.ok()
|
||||
.and_then(|m| m.elapsed().ok())
|
||||
.is_none_or(|age| age < cutoff);
|
||||
if recent {
|
||||
continue;
|
||||
}
|
||||
|
||||
let name = entry.file_name().to_string_lossy().to_string();
|
||||
if name.ends_with(".tmp") {
|
||||
// A `.tmp` never has a row by construction — no DB check needed.
|
||||
if tokio::fs::remove_file(entry.path()).await.is_ok() {
|
||||
temps_removed += 1;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
candidates.push((format!("originals/{slug}/{name}"), entry.path()));
|
||||
}
|
||||
}
|
||||
|
||||
if temps_removed > 0 {
|
||||
tracing::warn!(
|
||||
"reclaimed {temps_removed} abandoned upload temp file(s) older than \
|
||||
{ORPHAN_UPLOAD_RETENTION_HOURS}h"
|
||||
);
|
||||
}
|
||||
if candidates.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
// One query per batch, not one per file: a backlog of thousands of orphans must not turn
|
||||
// into thousands of round trips on an hourly timer.
|
||||
let mut orphans_removed = 0u32;
|
||||
for chunk in candidates.chunks(500) {
|
||||
let paths: Vec<String> = chunk.iter().map(|(rel, _)| rel.clone()).collect();
|
||||
// NO `deleted_at IS NULL` FILTER HERE. A soft-deleted row still points at its file
|
||||
// during its retention window, and reclaiming that file is `cleanup_deleted_media`'s
|
||||
// job — filtering here would race the two sweeps and destroy the exact files the
|
||||
// recovery window exists to preserve.
|
||||
let unreferenced: Result<Vec<(String,)>, _> = sqlx::query_as(
|
||||
"SELECT p FROM unnest($1::text[]) AS p
|
||||
WHERE NOT EXISTS (SELECT 1 FROM upload u WHERE u.original_path = p)",
|
||||
)
|
||||
.bind(&paths)
|
||||
.fetch_all(pool)
|
||||
.await;
|
||||
let unreferenced = match unreferenced {
|
||||
Ok(rows) => rows,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = ?e, "orphan-original sweep query failed");
|
||||
return;
|
||||
}
|
||||
};
|
||||
for (rel,) in unreferenced {
|
||||
if let Some((_, abs)) = chunk.iter().find(|(r, _)| *r == rel)
|
||||
&& tokio::fs::remove_file(abs).await.is_ok()
|
||||
{
|
||||
orphans_removed += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if orphans_removed > 0 {
|
||||
tracing::warn!(
|
||||
"reclaimed {orphans_removed} original(s) with no upload row, older than \
|
||||
{ORPHAN_UPLOAD_RETENTION_HOURS}h"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user