feat: crawler job queue ops and dedup index (0.27.0)

Adds enqueue / lease / ack_done / ack_failed / release / reap_done on
crawler::jobs, backed by the existing crawler_jobs table. lease() uses
a single FOR UPDATE SKIP LOCKED CTE that also re-claims stale running
rows (crashed-worker recovery), and ack_failed applies an exponential
backoff capped at 1h before retrying.

Migration 0014 adds a partial unique index on
(payload->>'chapter_id') restricted to (pending|running)
sync_chapter_content jobs, so producers can just
INSERT ... ON CONFLICT DO NOTHING without racing each other. The slot
frees again the moment the job leaves the in-flight states, so a
future force-refetch can re-enqueue.

Library-only — no daemon, no API hook. Those land in the next two
phases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

This commit is contained in:

MechaCat02

2026-05-25 19:59:09 +02:00

parent 89b84252a5

commit 93c7fd63fc

6 changed files with 676 additions and 6 deletions

									
										2

backend/Cargo.toml
									
												View File
												
				@@ -1,6 +1,6 @@

				[package]

				name = "mangalord"

				version = "0.26.1"

				version = "0.27.0"

				edition = "2021"

				default-run = "mangalord"

feat: crawler job queue ops and dedup index (0.27.0)

2 backend/Cargo.toml Unescape Escape View File

2

backend/Cargo.toml

View File