fix(compression): bound derivative retries so one bad upload can't loop forever
The OOM in the previous commit was survivable; what made it an outage was that it repeated. The upload row is committed before compression starts, derivatives_rev defaults to 0, and set_derivatives_rev only runs on success — so a row whose processing killed the container survived at rev 0, and backfill_stale_derivatives (called unconditionally at every boot) re-selected it and re-ran the identical workload. With restart: unless-stopped that is an infinite kill loop, and every cycle also drops every SSE stream and truncates every in-flight upload. Verified end to end against the real schema in a scratch database: with the new guard the backfill selects the row on boots 1-3 and zero rows from boot 4 on, and a later success resets the counter. migration 021 adds derivative_attempts and derivative_last_error. The counter is incremented WRITE-AHEAD, before the work is attempted. This is the whole design: the failure being bounded is a cgroup SIGKILL, so no Err is returned, no error handler runs and no Drop fires. A counter bumped in a failure path increments zero times per crash and the loop would be unchanged. set_derivatives_rev clears it, so success is the only reset and both the live path and the backfill get it without a new call site to forget. Also in the backfill: - one task walking the rows sequentially instead of one task per row. A large backlog used to spawn thousands of tasks, each holding a pool handle and queueing on the same two permits, competing with live uploads for a whole boot. - LIMIT 200 per boot, and original_path <> '' replacing an IS NOT NULL that was dead (the column is NOT NULL; cleanup_deleted_media blanks it instead). - a once-per-boot error log naming how many uploads have given up. Without it the give-up is invisible — the loop stops, which is the point, but the photos keep a stale derivative forever with nothing to notice. Adds backfill_video_posters for the mirror-image gap: a video interrupted by a restart has its compression_status flipped processing -> failed by startup_recovery and is never re-enqueued, so thumbnail_path stays NULL for the rest of the event while the clip itself plays fine. It shares the same attempt budget, which means a genuinely posterless sub-second clip (Live Photo, mis-tap) stops being re-ffmpeg'd after three boots. That is intended, not a bug to fix later — Ok(false) is a normal permanent outcome there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
3
backend/migrations/021_derivative_attempts.down.sql
Normal file
3
backend/migrations/021_derivative_attempts.down.sql
Normal file
@@ -0,0 +1,3 @@
|
||||
DROP INDEX IF EXISTS idx_upload_derivative_backfill;
|
||||
ALTER TABLE upload DROP COLUMN IF EXISTS derivative_last_error;
|
||||
ALTER TABLE upload DROP COLUMN IF EXISTS derivative_attempts;
|
||||
23
backend/migrations/021_derivative_attempts.up.sql
Normal file
23
backend/migrations/021_derivative_attempts.up.sql
Normal file
@@ -0,0 +1,23 @@
|
||||
-- Bound how many times a permanently-failing upload can be re-processed.
|
||||
--
|
||||
-- Without this, one poisoned row is an outage. The upload row is committed BEFORE compression
|
||||
-- starts, `derivatives_rev` defaults to 0, and `set_derivatives_rev` only runs on success — so
|
||||
-- a row whose processing kills the container survives at rev 0, the unconditional startup
|
||||
-- backfill re-selects it on the next boot, and `restart: unless-stopped` turns that into an
|
||||
-- infinite kill loop. Every restart also drops every SSE stream and truncates every in-flight
|
||||
-- upload. That was reachable via a single large PNG (see services/compression.rs), but the
|
||||
-- shape is general: any input that can kill or hang the worker repeats forever.
|
||||
--
|
||||
-- The counter is incremented WRITE-AHEAD, before the work is attempted, because the failure
|
||||
-- mode being defended against is a SIGKILL — no error is returned, no handler runs, no Drop
|
||||
-- fires. A counter bumped in an error path increments zero times per crash and changes nothing.
|
||||
ALTER TABLE upload ADD COLUMN IF NOT EXISTS derivative_attempts SMALLINT NOT NULL DEFAULT 0;
|
||||
|
||||
-- Last failure text, so a row that has given up can be diagnosed without reproducing it.
|
||||
-- Nothing reads this in code; it exists for the operator.
|
||||
ALTER TABLE upload ADD COLUMN IF NOT EXISTS derivative_last_error TEXT;
|
||||
|
||||
-- Serves the backfill selection, which now filters on both columns.
|
||||
CREATE INDEX IF NOT EXISTS idx_upload_derivative_backfill
|
||||
ON upload (derivatives_rev, derivative_attempts)
|
||||
WHERE deleted_at IS NULL;
|
||||
Reference in New Issue
Block a user