bde72e47f4
reap_terminal was one unbatched DELETE with no updated_at index — a full scan and
long-held lock over a large backlog (audit M5). Migration 0040 adds a partial
index on (updated_at) WHERE state IN ('done','dead'); reap_terminal now deletes in
bounded 5k batches. Existing tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
10 lines
515 B
SQL
10 lines
515 B
SQL
-- Index the retention reaper's scan. reap_terminal deletes
|
|
-- WHERE state IN ('done','dead') AND updated_at < now() - interval
|
|
-- which had no supporting index, so each daily sweep sequential-scanned the whole
|
|
-- crawler_jobs table to find expired terminal rows (audit M5). A partial index on
|
|
-- updated_at over just the terminal states makes the batched reaper's per-batch
|
|
-- SELECT index-backed.
|
|
CREATE INDEX crawler_jobs_terminal_reap_idx
|
|
ON crawler_jobs (updated_at)
|
|
WHERE state IN ('done', 'dead');
|