5969ec74ea
The OOM in the previous commit was survivable; what made it an outage was that it repeated. The upload row is committed before compression starts, derivatives_rev defaults to 0, and set_derivatives_rev only runs on success — so a row whose processing killed the container survived at rev 0, and backfill_stale_derivatives (called unconditionally at every boot) re-selected it and re-ran the identical workload. With restart: unless-stopped that is an infinite kill loop, and every cycle also drops every SSE stream and truncates every in-flight upload. Verified end to end against the real schema in a scratch database: with the new guard the backfill selects the row on boots 1-3 and zero rows from boot 4 on, and a later success resets the counter. migration 021 adds derivative_attempts and derivative_last_error. The counter is incremented WRITE-AHEAD, before the work is attempted. This is the whole design: the failure being bounded is a cgroup SIGKILL, so no Err is returned, no error handler runs and no Drop fires. A counter bumped in a failure path increments zero times per crash and the loop would be unchanged. set_derivatives_rev clears it, so success is the only reset and both the live path and the backfill get it without a new call site to forget. Also in the backfill: - one task walking the rows sequentially instead of one task per row. A large backlog used to spawn thousands of tasks, each holding a pool handle and queueing on the same two permits, competing with live uploads for a whole boot. - LIMIT 200 per boot, and original_path <> '' replacing an IS NOT NULL that was dead (the column is NOT NULL; cleanup_deleted_media blanks it instead). - a once-per-boot error log naming how many uploads have given up. Without it the give-up is invisible — the loop stops, which is the point, but the photos keep a stale derivative forever with nothing to notice. Adds backfill_video_posters for the mirror-image gap: a video interrupted by a restart has its compression_status flipped processing -> failed by startup_recovery and is never re-enqueued, so thumbnail_path stays NULL for the rest of the event while the clip itself plays fine. It shares the same attempt budget, which means a genuinely posterless sub-second clip (Live Photo, mis-tap) stops being re-ffmpeg'd after three boots. That is intended, not a bug to fix later — Ok(false) is a normal permanent outcome there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Migrations
SQLx-managed Postgres migrations. Each NNN_topic.up.sql has a matching
NNN_topic.down.sql. Run by sqlx::migrate!() at app start.
Rules
- Never edit a shipped migration. If a column needs to change or a fix needs to land, write a new migration. Production has already applied the old one and SQLx tracks each by checksum — editing in place will fail to apply on existing databases.
- Always pair
.up.sqlwith a.down.sql. Reverts may not be perfect (data loss is sometimes unavoidable) but the file must exist and do the best it can. - Prefer additive changes. New columns, new tables, new keys in
config. Drop / rename only when there is no alternative. - No business logic in migrations. Schema + seeds only. Anything that needs Rust code goes in a one-off binary, not a migration file.
- One concern per migration. Easier to revert. Easier to read in
git log.
Numbering
Zero-padded three digits, monotonically increasing. The next free number lives at the bottom of the directory listing — pick that.
Seed-only migrations
When you only need to add config keys (feature flags, defaults), use
INSERT … ON CONFLICT DO NOTHING so existing operator overrides survive. See
009_feature_toggles.up.sql for the canonical shape.