5969ec74ea
The OOM in the previous commit was survivable; what made it an outage was that it repeated. The upload row is committed before compression starts, derivatives_rev defaults to 0, and set_derivatives_rev only runs on success — so a row whose processing killed the container survived at rev 0, and backfill_stale_derivatives (called unconditionally at every boot) re-selected it and re-ran the identical workload. With restart: unless-stopped that is an infinite kill loop, and every cycle also drops every SSE stream and truncates every in-flight upload. Verified end to end against the real schema in a scratch database: with the new guard the backfill selects the row on boots 1-3 and zero rows from boot 4 on, and a later success resets the counter. migration 021 adds derivative_attempts and derivative_last_error. The counter is incremented WRITE-AHEAD, before the work is attempted. This is the whole design: the failure being bounded is a cgroup SIGKILL, so no Err is returned, no error handler runs and no Drop fires. A counter bumped in a failure path increments zero times per crash and the loop would be unchanged. set_derivatives_rev clears it, so success is the only reset and both the live path and the backfill get it without a new call site to forget. Also in the backfill: - one task walking the rows sequentially instead of one task per row. A large backlog used to spawn thousands of tasks, each holding a pool handle and queueing on the same two permits, competing with live uploads for a whole boot. - LIMIT 200 per boot, and original_path <> '' replacing an IS NOT NULL that was dead (the column is NOT NULL; cleanup_deleted_media blanks it instead). - a once-per-boot error log naming how many uploads have given up. Without it the give-up is invisible — the loop stops, which is the point, but the photos keep a stale derivative forever with nothing to notice. Adds backfill_video_posters for the mirror-image gap: a video interrupted by a restart has its compression_status flipped processing -> failed by startup_recovery and is never re-enqueued, so thumbnail_path stays NULL for the rest of the event while the clip itself plays fine. It shares the same attempt budget, which means a genuinely posterless sub-second clip (Live Photo, mis-tap) stops being re-ffmpeg'd after three boots. That is intended, not a bug to fix later — Ok(false) is a normal permanent outcome there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
24 lines
1.5 KiB
SQL
24 lines
1.5 KiB
SQL
-- Bound how many times a permanently-failing upload can be re-processed.
|
|
--
|
|
-- Without this, one poisoned row is an outage. The upload row is committed BEFORE compression
|
|
-- starts, `derivatives_rev` defaults to 0, and `set_derivatives_rev` only runs on success — so
|
|
-- a row whose processing kills the container survives at rev 0, the unconditional startup
|
|
-- backfill re-selects it on the next boot, and `restart: unless-stopped` turns that into an
|
|
-- infinite kill loop. Every restart also drops every SSE stream and truncates every in-flight
|
|
-- upload. That was reachable via a single large PNG (see services/compression.rs), but the
|
|
-- shape is general: any input that can kill or hang the worker repeats forever.
|
|
--
|
|
-- The counter is incremented WRITE-AHEAD, before the work is attempted, because the failure
|
|
-- mode being defended against is a SIGKILL — no error is returned, no handler runs, no Drop
|
|
-- fires. A counter bumped in an error path increments zero times per crash and changes nothing.
|
|
ALTER TABLE upload ADD COLUMN IF NOT EXISTS derivative_attempts SMALLINT NOT NULL DEFAULT 0;
|
|
|
|
-- Last failure text, so a row that has given up can be diagnosed without reproducing it.
|
|
-- Nothing reads this in code; it exists for the operator.
|
|
ALTER TABLE upload ADD COLUMN IF NOT EXISTS derivative_last_error TEXT;
|
|
|
|
-- Serves the backfill selection, which now filters on both columns.
|
|
CREATE INDEX IF NOT EXISTS idx_upload_derivative_backfill
|
|
ON upload (derivatives_rev, derivative_attempts)
|
|
WHERE deleted_at IS NULL;
|