Compare commits

...

178 Commits

Author SHA1 Message Date
MechaCat02
33cc41bacd fix: make the genre dedup migration collision-proof (0038)
All checks were successful
deploy / test-backend (push) Successful in 38m34s
deploy / test-frontend (push) Successful in 10m58s
deploy / build-and-push (push) Successful in 11m46s
deploy / deploy (push) Successful in 24s
Review caught a boot-crash risk: 0038's repoint-UPDATE could set two
non-canonical variant links of one manga to the same canonical id in a single
statement (NOT EXISTS sees the pre-statement snapshot) -> manga_genres PK
violation -> migration rollback -> startup failure. Replace with INSERT ... SELECT
DISTINCT ... ON CONFLICT DO NOTHING (collision-proof) then delete variants. Amended
in place (0038 is unpushed). Tested with the exact collision scenario.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 19:12:52 +02:00
MechaCat02
08a9819d76 test: await async render in AddToCollectionModal collection-name assertion
FE-2's CancellableLoader hop renders the list one microtask after the test's
fixed tick budget; switch getByText -> findByText (async retry). Fixes a
regression merged in ac5641b (FE-2 landed without running this component test).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:21:47 +02:00
MechaCat02
70a6598924 chore: harden docker-compose deployment (bind, memory limits, no-new-privileges)
- frontend publishes on ${FRONTEND_PUBLISH_ADDR:-127.0.0.1} not 0.0.0.0 (M4).
- mem_limit on backend(4g)/postgres(1g)/frontend(512m), .env-overridable (M3).
- security_opt: no-new-privileges on app + postgres services.
New knobs documented + allowlisted in the config meta-test; compose validates.
Deferred (need host verification): image pinning, backend healthcheck, cap_drop
on postgres/tor, Postgres backup sidecar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:10:29 +02:00
MechaCat02
e6aefaa804 test: pin AppError -> HTTP status mapping (incl 501 and 429 Retry-After)
Table-driven test over every AppError variant's into_response().status(), closing
the unexercised 501 NotImplemented path and asserting a deterministic 429
Retry-After header (audit GAP B). Test-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:03:28 +02:00
MechaCat02
6354a7a8a3 fix: tighten SSRF interceptor residuals
- verdict() now blocks file:/ftp:/gopher: subresources instead of allowing all
  non-http(s) schemes (keeps data:/blob:/internal for page loads).
- bin/crawler.rs uses should_attach_safe_resolver() so an http(s) proxy keeps the
  DNS-rebinding guard, matching the daemon.
- Regression test pins the url crate's decimal/hex/octal/short IPv4 normalization
  to loopback (ensure_public_target rejects them). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:00:55 +02:00
MechaCat02
60d5a2efea docs: clarify that MAX_PAGES_PER_CHAPTER=0 relies on the body-size backstop
Make explicit (config.rs + .env.example) that disabling the per-chapter page cap
leaves MAX_REQUEST_BYTES as the sole bound (audit footgun). No behavior change;
the thumbnail-source read is already bounded by MAX_FILE_BYTES + decode limits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:56:19 +02:00
MechaCat02
90f2398e56 fix: send HSTS on HTTPS document responses
Add Strict-Transport-Security (max-age=63072000; includeSubDomains) in
applySecurityHeaders, gated on x-forwarded-proto === 'https' so a plain-HTTP
dev/misconfig deploy never pins the browser to HTTPS (audit LOW). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:52:54 +02:00
MechaCat02
e960aae163 fix: enable browser SSRF interception by default (H1)
Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:51:05 +02:00
MechaCat02
adce950049 fix: exponential idle backoff for the analysis worker
The analysis worker's flat 1s idle sleep meant an empty queue fired a lease query
per worker per second forever (audit). Add idle_backoff (1s..30s, reset on lease),
mirroring the crawler. Unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:49:01 +02:00
MechaCat02
bde72e47f4 fix: batch and index the crawler_jobs retention reaper
reap_terminal was one unbatched DELETE with no updated_at index — a full scan and
long-held lock over a large backlog (audit M5). Migration 0040 adds a partial
index on (updated_at) WHERE state IN ('done','dead'); reap_terminal now deletes in
bounded 5k batches. Existing tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:46:01 +02:00
MechaCat02
cb757e7b69 fix: index the running/leased_until arm of the job-lease query
The lease predicate's `state='running' AND leased_until < now()` arm had no
index, so every lease poll seq-scanned the whole crawler_jobs table (audit H2).
migration 0039 adds a partial index on (leased_until) WHERE state='running'.
Behavior-neutral; 29 lease tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:43:33 +02:00
MechaCat02
042e7e9047 fix: fall back to the original when a thumbnail fails to decode
A blob with valid magic but a corrupt body passed upload then 500'd on a ?w=
thumbnail decode (audit). Fall back to serve_original (streams without decoding)
instead. Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:41:28 +02:00
MechaCat02
cc8ae2566f fix: reject a malformed reenqueue body instead of running the full scope
Option<Json<T>> collapses a malformed body to None, so a bad reenqueue body
silently ran the default full "All" scope (audit). Parse raw bytes: empty =
default All, malformed = 422. Tested both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:37:16 +02:00
MechaCat02
4f085f228a fix: commit cover DB pointer before mutating the old blob
put_cover/delete_cover mutated storage before the DB pointer, so a transient DB
failure could leave the row pointing at a deleted blob. Reorder to
put->set-DB->delete-old and clear-DB->delete-blob, so a failure only orphans a
blob, never dangles the pointer. Happy path covered by the 16 existing cover tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:33:57 +02:00
MechaCat02
ac5641bcd3 fix: guard AddToCollectionModal against out-of-order load results
Retargeting the modal mid-load could resolve responses out of order, overwriting
the newer target's membership with the older one's (audit FE-2). Route load()
through the tested CancellableLoader so a superseded result is dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:31:26 +02:00
MechaCat02
cd5358dc9b fix: stop reader key navigation leaking behind open overlays
Arrows/j/k/Home/End bubbled to the reader's window handler while a
jump/settings/action/collections/tags/context overlay was open, paging the
chapter behind it (FE-1); native select/contenteditable weren't exempt (FE-4).
New unit-tested isTypingTarget() + an overlay-open early return fix both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:28:57 +02:00
MechaCat02
c31830468c fix: enforce case-insensitive genre uniqueness
genres had only case-sensitive name UNIQUE while authors/tags use
UNIQUE (lower(name)); sync_genres could race two case variants into duplicate
rows (audit DB-3). Migration 0038 pre-dedups then adds
UNIQUE (lower(name)); sync_genres upserts ON CONFLICT (lower(name)). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:25:46 +02:00
MechaCat02
75f4481bbd fix: invalidate stale page analysis when a re-crawl overwrites a page
Deterministic per-page storage keys mean a re-crawl overwrites the image but the
pages row survives the upsert, so analysis keyed to page_id (page_analysis/
search_doc, OCR, auto-tags, content-warnings) stayed stale forever and
force=false re-enqueue never re-analyzed it — poisoning search + the derived
manga_content_warnings (audit DB-1). persist_pages now detects conflict-updated
pages via xmax and clears their four derived tables in the same tx (the
content-warnings DELETE fires the mcw_* triggers), returning them to unanalyzed
so they re-derive. Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:21:29 +02:00
MechaCat02
958d5bd713 fix: scheme-validate crawler source_url before rendering as a link
Harvested source_url (external content) rendered as an admin <a href> without
scheme validation — a javascript:/data: link would execute on click (audit M2).
safeExternalHref() gates to http(s); CrawlerHistoryTable/DeadJobsTable fall back
to plain text otherwise. Tested in safeHref.test.ts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:16:08 +02:00
MechaCat02
f2c9cfe162 test: cover the trusted_proxy X-Forwarded-For rate-limit gate
Two integration tests pinning both sides of the gate: trusted_proxy=true gives
each forwarded IP its own bucket; trusted_proxy=false ignores spoofed XFF and
shares the global bucket. Closes audit gap M7. Test-only, no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:13:37 +02:00
MechaCat02
b5f7467c47 refactor: remove unused constant_time_eq and correct the token doc
All checks were successful
deploy / test-backend (push) Successful in 35m33s
deploy / test-frontend (push) Successful in 10m46s
deploy / build-and-push (push) Successful in 11m31s
deploy / deploy (push) Successful in 13s
constant_time_eq (and the `subtle` dependency) was never wired into any
comparison — the module doc wrongly claimed token comparison used it, but lookup
is an indexed DB equality on the 256-bit SHA-256 hash, so there's no in-process
secret compare to time-attack. Delete the dead fn, its test, and the unused
`subtle` dep; rewrite the doc. No behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:22:39 +02:00
MechaCat02
9148e23da8 fix: reap expired sessions with a periodic background sweep
The sessions table only grew — find_active ignored expired rows but nothing
deleted them, so lapsed sessions accumulated indefinitely.

- repo::session::delete_expired: indexed DELETE (sessions_expires_idx) returning
  the reaped count.
- app::build spawns a detached hourly reaper (SESSION_GC_INTERVAL) that calls it,
  independent of crawler/analysis config.

Test: delete_expired_removes_only_lapsed_sessions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:19:41 +02:00
MechaCat02
f39307232c fix: force download for untyped (octet-stream) served blobs
The /files handler served the application/octet-stream fallback with no
Content-Disposition, letting a browser render it inline — a stored-XSS vector if
the blob is crafted HTML/JS. Add Content-Disposition: attachment for that type
(belt to the existing nosniff); known image types stay inline.

Test (new tests/api_files.rs): octet-stream → attachment; png → inline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:15:23 +02:00
MechaCat02
1b7b8a3038 fix: cap the multipart metadata part instead of buffering it unbounded
The `metadata` JSON part in the manga-create and chapter-upload handlers was
read with `Field::bytes()`, buffering the whole part before any check — a client
could send a multi-hundred-MB metadata blob (up to the 200 MiB request limit) as
one allocation. Image parts were already streamed with a cap; metadata was the
last unbounded read.

Cap it at a shared upload::MAX_METADATA_BYTES (64 KiB) via the read_capped
streaming loop (413 on overflow), and remove the now-unused read_field_bytes
helper so the unbounded path can't return.

Tests: manga + chapter oversized-metadata parts are rejected with 413.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:10:05 +02:00
MechaCat02
253d46c7e5 fix: return NotFound when a storage key resolves to a directory
`get` (fs::read → EISDIR) and `get_stream` (File::open succeeds on a Unix
directory, then streams garbage that fails mid-read) both mishandled a key
resolving to a directory. Mirror `size`'s existing guard: `get` maps
IsADirectory to NotFound; `get_stream` checks metadata.is_file() after open.

Test: get_on_directory_is_not_found.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:59:51 +02:00
MechaCat02
c570e0cc37 fix: escape LIKE wildcards in user-search ILIKE queries
A `%` or `_` in a search term silently acted as a LIKE wildcard rather than
matching literally (`50%` matched everything, `a_b` matched `axb`). Not
injection — terms are bound — but a search-correctness bug across every
user-facing ILIKE site.

- repo::escape_like: shared pub(crate) escaper (promoted from page_tag), unit-tested.
- Trigram-entangled sites (manga, tag, author) append a separate escaped param
  used only by the ILIKE branch; the trigram/similarity branches keep the raw term.
- Pattern-built sites (admin manga list, admin users, analysis coverage + history,
  crawler search incl. JSONB payload title) escape inside format!("%{}%", ..) and
  pair each ILIKE with ESCAPE '\'.

Tests: escape_like unit tests + integration tests on public manga search, author
autocomplete, and admin user search proving `_` matches literally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:57:06 +02:00
MechaCat02
5784483a57 fix: add CSP and defense-in-depth security response headers
Document responses carried no CSP, X-Frame-Options, Referrer-Policy, or
Permissions-Policy — leaving clickjacking and zero script-injection defense
in depth (security audit T4).

- svelte.config.js: enable kit.csp hash mode. SvelteKit hashes its own inline
  hydration scripts; the inline theme <script> in app.html isn't part of
  %sveltekit.head%, so its sha256 is pinned by hand (csp-config.js) and added
  to script-src alongside object-src 'none', base-uri 'self', frame-ancestors
  'none'. Styles stay unconstrained (Svelte emits dynamic inline style= attrs).
- hooks.server.ts: applySecurityHeaders() sets X-Frame-Options: DENY,
  Referrer-Policy: strict-origin-when-cross-origin, X-Content-Type-Options:
  nosniff, and a Permissions-Policy locking down unused features, only when the
  response hasn't already set them.
- src/csp-theme-hash.test.ts: drift guard against editing the theme script
  without updating THEME_SCRIPT_HASH.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:27:27 +02:00
MechaCat02
a44511983d fix: page through all chapters when the API omits total
All checks were successful
deploy / test-backend (push) Successful in 44m20s
deploy / test-frontend (push) Successful in 11m3s
deploy / build-and-push (push) Successful in 13m53s
deploy / deploy (push) Successful in 20s
listAllChapters used `page.total ?? all.length` as the loop bound. The
/chapters endpoint always returns `total: null`, so a full 200-row first
page broke the loop immediately — the reader dead-ended past chapter 200.
Stop on a short page instead; only trust `total` when actually reported.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 16:24:49 +02:00
MechaCat02
f8e53809d5 chore: sync backend version to 0.128.3 (lockstep with frontend)
Several frontend-only commits bumped both manifests but staged only the
frontend, leaving backend/Cargo.toml behind. Bring it back in lockstep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:26:12 +02:00
MechaCat02
0afd202164 fix: let AbortError propagate through the API client
The network-error wrapping added in the error-boundary work also caught
deliberate AbortController cancellations and turned them into a network_error
ApiError, breaking callers that distinguish "cancelled" from "failed" (the
cancellable admin fetches). Rethrow DOMException AbortError unchanged; only
genuine network failures become ApiError(status 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:25:01 +02:00
MechaCat02
ba3e5b481b fix: lock background scroll while a Modal or Sheet is open
Overlays didn't lock body scroll, so the page behind a modal/sheet scrolled
under it. Add a ref-counted scroll lock (so stacked overlays don't prematurely
release it) and hold it via an effect in Modal and Sheet while open, restoring
on close or unmount. Focus-trapping was already handled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:23:07 +02:00
MechaCat02
3b783c1d9b fix: upload retry reuses the created manga instead of duplicating it
After a partial upload failure (manga created, a chapter failed), re-submitting
re-ran createManga and spawned a duplicate manga. Track the created manga id:
a retry skips creation, re-sends only the not-yet-done chapters, disables the
now-saved title, relabels the button "Retry failed chapters", and shows a note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:20:03 +02:00
MechaCat02
a47b6895c2 feat: warn before leaving a form with unsaved changes
Drag-dropping page images then clicking a nav link silently discarded the
whole form. Add a reusable guardUnsavedChanges helper (SvelteKit beforeNavigate
confirm + native beforeunload) and wire it into the manga upload, manga edit,
and chapter upload forms. The dirty check returns false while submitting so the
post-save redirect isn't prompted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:16:40 +02:00
MechaCat02
3ca05dcb58 fix: paginate the bookmarks and collections lists with Load more
Both lists fetched a single capped page (100 / 200) with no pager, so entries
past the cap were silently unreachable. Add a reusable LoadMore component that
seeds from the streamed first page (preserving the auth gate + inline error
handling) and fetches further offset-based pages on demand, and wire it into
the bookmarks and collections pages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:12:03 +02:00
MechaCat02
c308eb3eac fix: reader loads the full chapter list for prev/next and the dropdown
The reader fetched only the first 200 chapters, so a deep chapter (#250) fell
outside the window and its prev/next chevrons resolved to null — a dead end —
and the chapter dropdown was truncated. Add listAllChapters, which pages
through the API's 200-row cap, and use it in the reader loader.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:02:44 +02:00
MechaCat02
69a9309c54 feat: honor OCR text and tag scope in search Chapters/Mangas views
The Chapters/Mangas aggregation views accept an OCR ?text= filter and the
page search intersects tags, but the client dropped both: any text query
collapsed every view into a flat page search that discarded the selected tag
and grouping. Add text to AggregateOptions, thread it into
listTaggedChapters/listTaggedMangas, and rework the search loader so a
tag-scoped view runs the aggregation (optionally text-filtered) and the page
search passes the selected tag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:58:48 +02:00
MechaCat02
1ed1a134ea feat: bot API token management page
The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:55:24 +02:00
MechaCat02
32d0a7e13b feat: root error boundary and network-error normalisation
A load() that rethrew (e.g. the profile overview) fell through to SvelteKit's
bare default error page, and a connection-refused fetch rejected as a raw
TypeError that blanked the view. Wrap fetch network failures in the client as
ApiError(status 0, network_error) so callers handle them uniformly, and add a
root +error.svelte with a friendly message plus Try again / Go home actions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:47:41 +02:00
MechaCat02
d779fa2b97 fix: confirm admin role toggle and keep the checkbox controlled
The admin checkbox fired an immediate, irreversible privilege change with no
confirmation, and on cancel/failure the optimistic DOM flip stuck while the
server state was unchanged. Add a confirm() gate (like Delete) and make the
checkbox controlled by u.is_admin: onchange reverts the DOM immediately and it
only flips once the server confirms — so a cancelled or failed toggle never
misrepresents the real role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:44:47 +02:00
MechaCat02
755417730f feat: serve width-bounded thumbnail variants for image grids
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:41:19 +02:00
MechaCat02
5b1ce581f3 perf: precompute mangas.sort_author for the author sort
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:29:26 +02:00
MechaCat02
c11df3182c perf: denormalize content-warning filter to manga_content_warnings
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:24:31 +02:00
MechaCat02
d749b56d58 docs: correct re-upload page-preservation semantics
CLAUDE.md claimed re-uploading a chapter unconditionally drops saved-page
references. In fact the crawler re-fetch (content::persist_pages) upserts by
(chapter_id, page_number) with ON CONFLICT DO UPDATE, preserving pages.id, so
collections/page-tags survive a re-fetch. Saves drop only when a pages row is
actually deleted (chapter deletion, or a user delete-then-recreate). Migration
0023's text is left untouched (sqlx checksums applied migrations).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:19:03 +02:00
MechaCat02
c24e296f07 fix: interleave uploaded chapters by number in the chapter list
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:17:18 +02:00
MechaCat02
9508fb8e86 fix: make bookmark add idempotent instead of 409
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:08:27 +02:00
MechaCat02
ca55712622 refactor: remove dead partial-render guard in resync_manga
The first guard block computed the partial-render condition but had an empty
if-body, so it never skipped anything — the real skip is the chapter-sync
guard further down. Delete the dead block and move its observability warning
into the live guard's else branch, eliminating the data-loss trap without
changing behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:04:44 +02:00
MechaCat02
4fe435cc76 fix: recover the auth rate limiter from a poisoned mutex
try_acquire used .expect on lock(), so a single panic while holding either
limiter mutex would poison it and make every subsequent auth request re-panic
— one blip became a persistent auth outage. Recover the guard with
unwrap_or_else(|e| e.into_inner()) so the limiter keeps serving.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:02:54 +02:00
MechaCat02
f5cb460aec fix: abandon dispatch when heartbeat loses the lease
Heartbeat renew failures were only logged, so under DB flakiness a job could
be leased twice concurrently — the original worker kept crawling past a lapsed
lease while another re-leased it. Track consecutive renew failures and, after
MAX_HEARTBEAT_RENEW_FAILURES, signal the worker (via a CancellationToken) to
abandon the in-flight dispatch instead of working a job it no longer owns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:01:04 +02:00
MechaCat02
f83d49b83e fix: exponential backoff for idle crawler workers
An idle worker re-issued a row-locking lease query every 1s per worker with
no backoff. Replace the flat sleep with idle_backoff: 1s, 2s, 4s, … capped at
30s, reset to 0 the moment a job is leased, so a quiet daemon stops hammering
the crawler_jobs table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:58:39 +02:00
MechaCat02
cef41ce76a fix: don't burn a job attempt when the browser is unavailable
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.

Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:55:50 +02:00
MechaCat02
7570524e5b fix: stream cover uploads with a per-chunk size cap
The cover multipart path called Field::bytes(), buffering the entire field
into memory before parse_image checked max_file_bytes — an oversized cover
was fully allocated before rejection. Add read_capped/push_capped, which
reject with 413 as soon as the running total exceeds the cap (the offending
chunk is never copied in), and route both cover handlers and stage_image_part
through it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:51:10 +02:00
MechaCat02
5b46eab73a fix: keep SSRF DNS resolver on http(s)-proxy crawler paths
Commit 134ab54 dropped the safe DNS resolver for any configured proxy, but
that guard is only pointless for SOCKS proxies (where the proxy resolves the
target). On an http(s):// proxy reqwest still resolves the target locally, so
a hostname resolving to a private IP would be reachable. Narrow the exemption
to SOCKS-only via should_attach_safe_resolver.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:46:39 +02:00
MechaCat02
91d6a50dba fix: guard all browser subresources against SSRF and fail closed
The CDP Fetch interceptor only paused main-frame Document requests, so a
scraped page's <img>/fetch()/XHR subresources to internal targets
(169.254.169.254, postgres:5432, RFC1918) reached those hosts unguarded.
Drop the ResourceType::Document constraint so every request runs through
is_blocked. Also make open_page fail-closed: if the guard can't install,
close the blank page and return the error instead of navigating unguarded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:44:37 +02:00
134ab54b34 fix: don't apply the SSRF DNS resolver to proxied/internal clients
All checks were successful
deploy / test-backend (push) Successful in 39m39s
deploy / test-frontend (push) Successful in 10m52s
deploy / build-and-push (push) Successful in 12m5s
deploy / deploy (push) Successful in 13s
ff4ca96 wired SafeResolver (drops any host resolving to a private IP) into
all four crawler/analysis reqwest clients. That broke every proxied fetch:
with a socks5h:// proxy the target is resolved by Tor, so reqwest only ever
resolves the proxy's OWN host — `tor`, on a private Docker IP (172.x) — which
the resolver then rejects ("SOCKS error: failed to create underlying
connection"). 100% of crawler image downloads failed. The vision clients
broke the same way against `mangalord-vision` (latent; analysis was off).

Fix: attach the resolver only on the crawler's direct (unproxied) path, where
it genuinely guards DNS rebinding. Behind a proxy it adds no protection (Tor
can't reach internal ranges and the target is invisible to reqwest) so it's
omitted. Drop it from the vision clients entirely — that endpoint is an
operator-configured internal service that must resolve to a private IP.

Security posture is >= the pre-ff4ca96 baseline (which had no resolver at all).
is_private_ip / retain_public_addrs and their tests are unchanged.

Bump 0.124.13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 22:20:25 +02:00
MechaCat02
f879ce1866 fix: per-IP auth rate limiting instead of one global bucket
All checks were successful
deploy / test-frontend (push) Successful in 10m44s
deploy / build-and-push (push) Successful in 11m35s
deploy / deploy (push) Successful in 12s
deploy / test-backend (push) Successful in 31m21s
A single global token bucket let one attacker at the sustained rate 429
every user's login/register/change-password. Key buckets by client IP: the
SvelteKit proxy now stamps the real peer address onto X-Forwarded-For
(overriding any client-supplied value, anti-spoof), and axum reads it via a
ClientIp extractor — but only when AUTH_TRUSTED_PROXY is set, else it falls
back to the shared bucket (today's behavior). The per-IP map is bounded
(10k IPs, idle buckets pruned) so a spoofed-IP spray can't grow it.

AUTH_TRUSTED_PROXY defaults false (safe for a directly-exposed backend);
compose sets it true since the proxy is the single trusted hop.

Bump to 0.124.12.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 07:07:06 +02:00
MechaCat02
bf425cf8e6 fix: opt-in CDP re-validation of headless-browser navigations (SSRF)
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.

Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.

Bump to 0.124.11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 06:29:17 +02:00
MechaCat02
ff4ca964f5 fix: reject private IPs after DNS resolution (SSRF/DNS-rebinding)
The SSRF guard only checked the host string, so an attacker-owned domain
resolving to 169.254.169.254 / 10.x (DNS rebinding, TOCTOU) bypassed it.
Add a reqwest dns::Resolve (SafeResolver) that drops resolved addresses in
private ranges, wired into all four crawler/analysis clients — it fires per
connection so it also covers redirect hops. Also close the is_private_ip
IPv6-embedding gaps (IPv4-compatible ::/96, NAT64 64:ff9b::/96, 6to4
2002::/16 all now unwrap to the embedded IPv4).

Bump to 0.124.10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:52:30 +02:00
MechaCat02
61669aac3f fix: don't mark private-mode blobs as publicly cacheable
The immutable Cache-Control added for the reader preload work was
`public` unconditionally. Under PRIVATE_MODE, /files is auth-gated, so a
shared cache/CDN would store the blob and serve it to unauthenticated
clients, defeating the gate. Emit `private, ...` when private_mode is on,
`public, ...` otherwise.

Found in the security audit; regression from 0.124.3. Bump to 0.124.9.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:39:20 +02:00
MechaCat02
795eb76f9a fix: document and wire the DB pool env vars into compose
The new DB_MAX_CONNECTIONS / DB_ACQUIRE_TIMEOUT_SECS knobs were read by the
backend but never added to .env.example or docker-compose.yml, so an operator
setting them in .env would see no effect (caught by the config meta-tests).
Document both in .env.example and wire them into backend.environment.

Bump to 0.124.8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:18:49 +02:00
MechaCat02
ebf0b8289b fix: compute the /mangas total only on the first page
GET /mangas recomputed count(*) over FILTER_WHERE (up to five correlated
subqueries) on every page, so pagination cost scaled with catalog size. The
total doesn't change as a caller walks pages, so compute it once at offset 0
and return null thereafter (the envelope already serialises total as
Option<i64>; the frontend types it number|null and the library route ignores
it).

Bump to 0.124.7.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:14:15 +02:00
MechaCat02
ce9a727c73 fix: enforce the analysis image cap while reading, not after
The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.

Bump to 0.124.6.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:10:29 +02:00
MechaCat02
46134c8760 fix: close Chromium tabs on crawler fetch error paths
new_page() opened a tab that was only closed on the happy path; a `?` on
wait_for_nav / content() leaked it (chromiumoxide doesn't close on drop),
so errored fetches accumulated tabs in the shared browser. Add a generic,
unit-tested `close_after(close, body)` helper and wrap the three fetch
sites (chapter content, session probe, source navigate) so the tab is
closed on every exit path — including navigate's content()-read branch,
which previously leaked.

Bump to 0.124.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:04:50 +02:00
MechaCat02
b7d8faadf7 fix: size the DB pool and fail fast on saturation
The single Postgres pool (shared by HTTP handlers and the crawler/analysis
daemons) hardcoded max_connections=10 with no acquire_timeout, so under
saturation callers hung on the driver's silent 30s default. Add a DbConfig
(DB_MAX_CONNECTIONS default 20, DB_ACQUIRE_TIMEOUT_SECS default 10, clamped
to >=1) and apply both in PgPoolOptions.

Bump to 0.124.4.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:56:14 +02:00
MechaCat02
b0500e8e48 fix: cache immutable blobs served from /files
Page and cover blobs are content-addressed by unguessable, immutable keys,
but /files sent no Cache-Control, so the browser refetched on every reader
open — defeating the whole-chapter and next-chapter preloading (and re-
proxying every byte through the node server in prod). Send
`Cache-Control: public, max-age=31536000, immutable`.

Bump to 0.124.3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:30:03 +02:00
MechaCat02
379224bee4 fix: lazy-load continuous-reader pages beyond an eager window
Continuous mode rendered every page (up to 2000) with loading="eager",
fetching and decoding the whole chapter up front — enough to OOM a mobile
tab on a long chapter. Eager-load only a leading window ([0..=eagerThrough],
covering the deep-link target + a lead, min 6 pages) and lazy-load the rest;
the browser fetches them as the reader scrolls near. Unloaded lazy pages
reserve height (page width x ~1.4) so they stay distinct scroll targets for
the progress IntersectionObserver, cleared on load.

Keeps the DOM dense so the scroll-to-?page=N effect, progress observer, and
next-chapter preload trigger are all unchanged. Rewrote reader-preload-window
spec to assert the eager/lazy attribute partition + deep-link scroll landing.

Bump to 0.124.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:28:41 +02:00
MechaCat02
1c955458d6 fix: run Argon2 password hashing on the blocking pool
Argon2id hashing (~15-50ms, 19 MiB) ran synchronously inside async
handlers, stalling every task sharing those runtime worker threads under
concurrent auth load. Add spawn_blocking wrappers hash_password_async /
verify_password_async and switch all async call sites; sync primitives
stay for the login timing-equaliser and unit tests. Bump to 0.124.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:19:14 +02:00
MechaCat02
987d1ba235 feat: preload the whole current chapter in continuous mode
Eager-load every page of the current chapter rather than a bounded
rolling window, so pages are fetched up front (the browser paces them
over its per-origin connection limit, top-to-bottom = reading order)
instead of filling in as they're scrolled onto. This drops the
scrollLeadIdx/eagerThrough machinery in favour of a plain
`loading="eager"`.

The first few pages of the *next* chapter are already warmed when the
reader nears the end of the current one (the next-chapter preloader), so
flipping over stays instant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:47:48 +02:00
MechaCat02
c69ff502f7 feat: preload the current chapter ahead of the reader
Continuous mode eager-loaded only pages 0..=initialIndex; everything
below stayed `loading="lazy"`, so pages fetched and reflowed in as they
scrolled into view. Now a rolling window eager-loads a few pages ahead of
the reading position (PRELOAD_AHEAD) so they arrive — fetched and sized —
while still below the fold, then swap in without pop-in.

The window rolls forward off `scrollLeadIdx`, the top-of-viewport page
measured from live rects and guarded by `scrollY`. It deliberately does
not use the read-progress high-water mark: before images load they are
0-height and pile into the viewport, which would latch that mark to the
last page and eager-load the whole chapter on open. Pages past the window
stay lazy so opening a long chapter doesn't fetch every page at once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:39:50 +02:00
MechaCat02
51ea254dde fix: surface streamed load failures inline
All checks were successful
deploy / test-backend (push) Successful in 30m44s
deploy / test-frontend (push) Successful in 10m46s
deploy / build-and-push (push) Successful in 10m59s
deploy / deploy (push) Successful in 13s
The streamed loaders on bookmarks, collections, library, search, and the
collection detail page swallowed only ApiError into their in-band `error`
field and re-threw anything else. A raw network failure (a non-ApiError
TypeError from fetch) therefore escaped the `{#await}` to SvelteKit's
generic error page. On the collection detail page a content-load failure
showed a misleading "this collection is empty", and on search a failed
results query showed a false "no matches" because the streamed `error`
was never rendered.

Fold non-ApiError failures into the same in-band `error` so every
transient blip renders as an inline message, add a `contentError` branch
to the collection detail grid, and render the search results `error` in
both result views.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:11:37 +02:00
MechaCat02
fee68dd9ac feat: loading skeleton on the manga detail page
The detail loader now streams its six-call bundle instead of blocking
navigation on it. The page component is a thin {#await} wrapper: it shows a
MangaDetailSkeleton (cover + meta + chapter rows) while the bundle loads,
renders the extracted DetailView once resolved, and shows an inline
not-found on a 404 (previously the framework error page). Extracting
DetailView also makes it remount per navigation, so its tag/bookmark/reaction
state re-seeds cleanly on manga-to-manga moves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:30:49 +02:00
MechaCat02
3c8e264f48 feat: loading skeleton on the library page
The library loader streams its one-shot bundle (also the auth gate), so the
active sub-tab shows a skeleton while it loads — a grid skeleton for the
Collections tab, a row skeleton for Bookmarks/History/Page-tags — instead of
only the global nav bar.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:21:20 +02:00
MechaCat02
16cdec051a feat: loading skeletons on the bookmarks and collections pages
Both loaders now stream the whole result (the single fetch is also the auth
gate) so the page shows a skeleton while it loads instead of the global nav
bar only: a ListRowSkeleton on bookmarks and a MangaGridSkeleton on the
collections grid, resolving to the sign-in / error / empty / list branches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:18:43 +02:00
MechaCat02
5b51bcf056 feat: results skeleton on the search page
The search loader now awaits only the chip cloud (its auth gate) and streams
the view-specific results, so switching tag / view / sort / text shows a
SearchResultsSkeleton in place of the frozen previous list until the query
resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:14:43 +02:00
MechaCat02
f1e66141f4 fix: add tags optimistically on the detail page
submitTag now shows the chip immediately with the typed name and reconciles
it with the server's (normalized) ref on success, rolling back on error —
matching the already-optimistic tag removal instead of blocking on the
round-trip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 22:05:23 +02:00
MechaCat02
cee1e73f98 feat: grid skeletons on the author and collection detail pages
Both loaders now stream their manga grid instead of blocking on it: the cheap
header renders immediately (and still handles 404/401 in the loader) while a
MangaGridSkeleton stands in until the grid resolves, then swaps in without a
layout shift. Collection detail seeds its optimistic-removal state from the
stream via a guarded effect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 22:01:53 +02:00
MechaCat02
ec73c6e001 feat: reserve home shelf space while it loads (fixes CLS)
The Continue-reading / Recommended shelves were fetched after the catalogue
and rendered only once populated, so they popped in late and shoved the grid
down on every authenticated home visit. Now the shelf fetch fires
concurrently with the catalogue and a reserved ShelfSkeleton holds the space
for logged-in users until it resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:53:03 +02:00
MechaCat02
bd6ae86a85 feat: preload the reader from the detail CTA
The Continue / Read-first-chapter CTA is the highest-intent link to the
heaviest route in the app. An effect now programmatically preloadData()s the
target chapter as soon as the detail resolves, so the reader's load and first
page images are warm before the tap. Complements the reader's own
next-chapter warming.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:47:53 +02:00
MechaCat02
3622dcc02f fix: reserve detail cover box and async-decode list/shelf covers
- The manga detail desktop overview cover had no reserved aspect-ratio, so
  the meta column reflowed when it decoded. Give .cover aspect-ratio: 2/3 +
  object-fit: cover (mirrors MangaCard).
- Add decoding="async" to the detail hero/overview covers and the list/shelf
  covers (bookmarks, history, continue-reading, collections, tagged rows) so
  decode stays off the main thread — previously only MangaCard had it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:43:21 +02:00
MechaCat02
9cb2f152d3 fix: bookmark toggle is optimistic and no longer fails silently
The detail-page bookmark toggle awaited the round-trip before updating and
had no catch — a failed create/delete threw unhandled and the user saw
nothing. It now flips optimistically (placeholder reconciled with the server
row on success), rolls back on error, and surfaces a toast, matching the
sibling like/dislike buttons.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:40:33 +02:00
MechaCat02
7c3c9cf699 feat: app-wide toast notifications
Adds a client-side toast store (info/success/error, auto-dismiss with a
sticky option) and a Toaster component mounted in the root layout, wired to
the reserved --z-toast token. Gives mutations a consistent way to surface
success/failure instead of inline-only text or silent catches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:36:34 +02:00
MechaCat02
0d9505ce9f fix: address skeleton/nav-progress review findings
Correctness:
- MangaGridSkeleton: covers rendered 0px tall because height="0" is a
  definite height that defeats aspect-ratio. Skeleton gains an `aspectRatio`
  prop (height stays auto) so the 2/3 cover box is reserved and the grid no
  longer shifts when real covers load.
- Move `.manga-card` layout into global tokens.css (it was scoped to
  MangaCard, so the skeleton's cards inherited none of it). Both consumers
  now share one definition.
- NavProgress: rebuilt as an idle/loading/done state machine. It now
  completes to full and fades on settle (was snapping away instantly),
  restarts the trickle when one navigation supersedes another (was parking
  frozen near 90%), animates transform/opacity only with will-change gated to
  the active phase, and hides itself in reader fullscreen.
- Skeleton: text-variant height resolved in one place (no inline style
  shadowing a stylesheet rule); shimmer gradient/animation gated behind
  prefers-reduced-motion: no-preference so reduced motion shows a flat block.

A11y:
- Home loading region announces via visually-hidden text instead of an
  aria-label a live region won't reliably read.

Cleanup:
- Drop the fragile :first-child selector and index-keying ceremony in
  MangaGridSkeleton; simplify Skeleton's style construction.

Tests: adds regression guards for the 0px cover collapse, the aspect-ratio
box, text-height defaulting, and the nav-progress complete/restart phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 20:02:27 +02:00
MechaCat02
7bdbe3ce5b fix: define missing --font-md and --success-soft-bg tokens
Both were referenced by components (RecommendationShelf, the manga detail
page) but never declared, so var() fell through to inherited/fallback
values. Adds --font-md (1rem, on the xs/sm/md/lg scale) and
--success-soft-bg in both the light and dark theme blocks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:44:43 +02:00
MechaCat02
3364ea52c9 feat: async decode for manga cover images
Adds decoding="async" to the MangaCard cover so image decode happens off the
main thread, keeping grid scroll/paint smooth as covers stream in. The 2/3
aspect-ratio already reserves space, so no layout shift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:41:28 +02:00
MechaCat02
def97d3087 feat: skeleton loading state on the home catalog
Replaces the plain "Loading…" text with a content-shaped MangaGridSkeleton
so the catalog grid's layout is reserved while it loads and swaps in without
a shift. Keeps the data-testid="loading" wrapper (now role="status") for
accessibility and selector stability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:40:28 +02:00
MechaCat02
09f12c8959 feat: global navigation progress bar
Most data routes run their load with ssr = false, so SvelteKit holds the
previous page on screen while the next one's data resolves — with no
feedback. NavProgress, mounted once in the root layout and driven by the
$navigating store, shows a thin top bar that trickles while a navigation is
pending and fades out on completion. Degrades to a static visible bar under
prefers-reduced-motion. New --z-nav-progress token sits above the fixed
header but below modals/toasts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:38:14 +02:00
MechaCat02
34c1122e4e feat: add MangaGridSkeleton loading placeholder
Composes Skeleton into a cover-grid placeholder that reuses the shared
.manga-grid / .manga-card layout, so it matches the real catalog grid at
every breakpoint and content swaps in without a layout shift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:31:58 +02:00
MechaCat02
200ab1c0a0 feat: add Skeleton loading-placeholder primitive
A reusable, token-driven shimmer block with configurable width/height,
radius, and variant. Decorative (aria-hidden) — loading state is announced
by the container. Freezes to a solid muted block under the global
prefers-reduced-motion override.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:28:00 +02:00
MechaCat02
ef8d226ba6 fix(detail): resync reaction toggle across manga navigation
All checks were successful
deploy / test-backend (push) Successful in 36m1s
deploy / test-frontend (push) Successful in 10m41s
deploy / build-and-push (push) Successful in 11m41s
deploy / deploy (push) Successful in 13s
The detail page component is reused across /manga/A -> /manga/B
navigations (clicking a similar or recommendation card), so
ReactionButtons' locally-held `current` state kept the previous manga's
reaction until interacted with — a visibly wrong thumb on the new page.
Re-seed `current` from the loader-supplied prop via an effect keyed on
mangaId. Also fetch the homepage's two independent /me/* shelves
concurrently (Promise.allSettled) instead of in series.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 18:17:18 +02:00
MechaCat02
19db66a845 feat(home): add a "Recommended for you" shelf
Surface the content-based /me/recommendations feed as a horizontal shelf on
the homepage, below the Continue-reading shelf. Fetched after the catalogue
(same non-blocking pattern), shown only when signed in and the feed is
non-empty; reuses MangaCard in a fixed-width scroller. Component unit test +
e2e (signed-in shows cards, anonymous hides).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:46:33 +02:00
MechaCat02
d6a109df2d feat(recommendations): content-based "Recommended for you" endpoint
Add GET /me/recommendations: rank mangas by weighted tag overlap with the
user's taste — explicit like +1.0, bookmark +0.5, dislike −1.0 (reaction
overrides bookmark). Per-tag affinities are summed, candidates scored by
their tags' affinity normalized by tag count (anti-tag-stuffing, à la
list_similar), with already-reacted/bookmarked/read mangas excluded and
net-negative candidates dropped (dislike down-ranks, not browse-hides).
Reuses manga_cols/cards_from_rows. Integration tests cover like-driven recs,
dislike down-rank, bookmark half-weight ordering, seen-exclusion, empty
cold-start, and auth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:42:05 +02:00
MechaCat02
bb833c7e71 feat(detail): like/dislike buttons on the manga page
Add a tri-state like/dislike toggle beside the bookmark action (signed-in
only): click a reaction to set it, the other to switch, or the active one to
clear — optimistic with rollback, mirroring the bookmark toggle. Seeds from
the new GET /me/reactions/:id in the detail loader (non-critical: any failure
degrades to an unset toggle). Unit tests cover the state machine; e2e drives
set → switch → clear against the real endpoints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:20:24 +02:00
MechaCat02
258a536254 feat(reactions): per-user like/dislike on mangas
Add a private per-user reaction resource: a manga_reactions table (one row
per user+manga, like/dislike, cascades) and endpoints PUT/DELETE
/mangas/:id/reaction plus GET /me/reactions/:manga_id, wired on the existing
per-user seam (mirrors bookmarks/read_progress). Unknown manga → 404 via the
FK-violation mapping; an invalid reaction value → 422. Reactions are never
exposed publicly — this is a taste signal for the upcoming content-based
recommendations. Integration tests cover set/toggle/clear/read, per-user
isolation, and the 404/422/401 paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 16:54:11 +02:00
MechaCat02
9c4a93c058 fix(reader): harden swipe/long-press coexistence and next-chapter prefetch
All checks were successful
deploy / test-backend (push) Successful in 32m21s
deploy / test-frontend (push) Successful in 10m33s
deploy / build-and-push (push) Successful in 11m40s
deploy / deploy (push) Successful in 18s
- A long-press now cancels any pending swipe, so a hold-then-drag gesture
  opens the action sheet without also turning the page (regression-tested).
- The next-chapter prefetch no longer retries on every page flip when the
  endpoint fails (guard clears only when the next chapter changes), and it
  clears a previous chapter's warmed images on navigation so stale hidden
  <img>s don't linger.
- Tap zones set touch-action: pan-y so vertical panning stays native while
  horizontal swipes reach our handler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:57:36 +02:00
MechaCat02
bfaa166e0a fix(home): keep finished series off the Continue-reading shelf
A "Continue reading" shelf listing series the reader already finished (read
to the last page, nothing new) is odd. Expose the last-read chapter's
page_count on the read-progress list, and filter the shelf to drop entries
that are caught up (on the last page of the latest chapter with no new
chapters). Mid-chapter and has-new-chapters entries stay; the full history
view is unaffected. Finished detection is a pure, unit-tested helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:52:58 +02:00
MechaCat02
3824eaafb2 fix(history): count distinct new chapter numbers, authoritative on detail
new_chapters_count over-counted when scanlations share a chapter number
((manga_id, number) is non-unique, migration 0013): COUNT(*) tallied rows,
so duplicate-numbered chapters inflated the badge. Switch both the list and
per-manga read-progress queries to COUNT(DISTINCT number), and drop the dead
COALESCE (an empty set counts as 0, and the NULL-number case is handled by
the predicate). Fix the misleading comment.

Expose new_chapters_count on GET /me/read-progress/:manga_id and drive the
detail page's "N new" badge from it, instead of recomputing over the
paginated chapter list — which under-counted series past 50 chapters and
disagreed with the homepage shelf. Read markers stay client-side over the
loaded chapters and are documented as by-number (all we persist).

Adds duplicate-scanlation tests at both endpoints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:48:36 +02:00
MechaCat02
f1349100d7 feat(reader): prefetch the next chapter's first pages near the end
When the reader reaches within two pages of the end of a chapter (tracked in
both modes off the furthest page reached), warm the next chapter's first few
page images via hidden imgs so flipping over renders instantly. Fetched once
per next-chapter id, guarded against mid-flight chapter changes, and retried
on failure. e2e asserts the prefetch is absent at the chapter start and
present (with the next chapter's URLs) near the end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:28:06 +02:00
MechaCat02
3f1a5e9c41 feat(reader): swipe left/right to turn pages in single mode
Wire horizontal swipe into the mobile tap-zone overlay (single mode only,
so continuous mode keeps native vertical scroll). A leftward swipe past the
threshold turns to the next page, rightward to the previous; a mostly-
vertical drag is ignored so panning a tall page never turns it. The gesture
suppresses the synthesized zone tap so it doesn't double-fire. Swipe
classification is a pure, unit-tested helper; e2e drives real touch pointer
events for horizontal (both directions) and vertical cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:23:34 +02:00
MechaCat02
c212deb7b0 feat(reader): add a keyboard-shortcut help overlay
Press `?` in the reader to open a Sheet documenting the existing bindings
(arrows/J/K paging, Home/End, chapter nav, Esc); Esc or the close button
dismisses it. While open, the reader swallows navigation keys so paging
can't happen behind the overlay. e2e covers open-on-? and close-on-Esc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:18:47 +02:00
MechaCat02
f5692ea109 feat(ui): show a tooltip with the full text on truncated titles
Add an `overflowTooltip` Svelte action that sets a native `title` only when
an element is actually clipped (ellipsis or line-clamp), re-checking on
resize. Native title is keyboard/touch reachable and zero-dep, unlike a
hover-only popover. Applied to the clipped titles on MangaCard, BookmarkList,
HistoryList, and the Continue-reading shelf.

Unit tests cover the overflow decision and the set/clear/update behaviour;
an e2e verifies a real-browser truncated card title gets the tooltip while a
short one does not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:08:28 +02:00
MechaCat02
ad1689b818 feat(detail): link genre and tag chips to the filtered catalogue
Genre chips now deep-link to /?genres=<id> and user-tag chips to
/?tags=<id>, reusing the homepage catalogue's existing filter params —
turning the detail page's metadata into one-tap discovery. Tag chips keep
their remove affordance (the Chip's href branch preventDefaults the remove
button). e2e asserts both chip hrefs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:04:38 +02:00
MechaCat02
01d18e7ba2 feat(detail): mark read chapters and count new ones on the manga page
On the manga detail page, dim chapters at or before the reader's last-read
chapter (with an accent check + screen-reader "Read." label) and show a
"N new since last read" badge counting chapters that have landed since.
All client-side from the already-loaded chapter list and per-manga read
progress — no extra requests. The count reflects the loaded (paginated)
chapter list. Logic lives in a pure, unit-tested chapterProgress helper;
e2e covers the mid-series and never-opened cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:01:58 +02:00
MechaCat02
2267a83f6c feat(home): add a Continue-reading shelf with new-chapter badges
Surface a horizontal "Continue reading" shelf at the top of the homepage,
fed by the user's read-progress history. Each card jumps back to the exact
page they left off (deep-linking `?page=N` past the first) and flags how
many chapters have landed since — the personal new_chapters_count from the
read-progress list. The shelf is fetched after the catalogue load so the
public browse path stays unauthenticated; guests get an empty list (401
swallowed) and the shelf stays hidden.

Add new_chapters_count to the frontend ReadProgressSummary type to match
the backend. Component unit tests cover href/deep-link, badge visibility,
and the no-chapter fallback; e2e covers signed-in (shelf + badge) and
anonymous (hidden).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:57:17 +02:00
MechaCat02
f318c3bf51 feat(history): report personal new-chapter count on read-progress list
Add `new_chapters_count` to each `GET /me/read-progress` item: how many
chapters sit past the reader's last-read chapter, computed by a correlated
subquery on chapter number. It is 0 when the reader is caught up, and 0
when the last-read chapter is unknown (manga-level progress or a deleted
chapter) so we never over-claim. (manga_id, number) is non-unique, so this
counts positions past the reader rather than distinct files.

Backs the personal "new since last read" indicator on the upcoming
Continue-reading shelf. Tests cover the mid-series, caught-up, and
no-read-chapter cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:50:49 +02:00
MechaCat02
cf8971faae fix(reader): render pages at a consistent width instead of thin stripes
Reader page sizing was height-driven — `max-height: 90vh` in single mode
and unbounded natural width in continuous mode. A long vertical page hit
the height cap first and, preserving aspect ratio, collapsed into a thin
vertical stripe; narrow and wide scans rendered at different widths.

Switch both modes to width-driven sizing via a capped `--reader-page-width`
reading column (min(100%, 700px)): every page renders at the same width
regardless of intrinsic dimensions, and tall pages extend downward and
scroll instead of shrinking. Top-align the single-mode grid and make the
prev/next chevrons sticky so they stay reachable on a long page.

Add a Playwright spec asserting equal rendered width across differently-
proportioned pages and that a tall page exceeds the viewport height. The
page-context-menu fixture served a 1x1 image that now renders as a 700x700
square taller than the viewport, making Playwright scroll it into view
before right-clicking — a synthetic scroll that trips the menu's by-design
close-on-scroll. Give that fixture a realistic landscape aspect so the page
fits the viewport, matching real user behaviour (no scroll on right-click).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:45:59 +02:00
MechaCat02
38146b4d03 chore(frontend): sync package-lock version to 0.94.0
All checks were successful
deploy / build-and-push (push) Successful in 11m29s
deploy / test-backend (push) Successful in 30m25s
deploy / test-frontend (push) Successful in 10m29s
deploy / deploy (push) Successful in 13s
The lockfile's root version field trailed the manifests (0.90.0 vs
0.94.0) — harmless to `npm ci` (it checks the dependency tree, not the
project version) but tidy to keep aligned. Also folds in the peer-dep
metadata npm regenerated on the last install. No behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 19:45:05 +02:00
MechaCat02
8acc0e6cc2 feat(frontend): use the Mangalord monogram as the browser tab icon
Some checks failed
deploy / test-frontend (push) Has been cancelled
deploy / build-and-push (push) Has been cancelled
deploy / deploy (push) Has been cancelled
deploy / test-backend (push) Has been cancelled
The favicon link pointed at a non-existent favicon.ico. Ship the
monogram SVGs from static/ and wire two theme-aware <link rel="icon">
tags so the tab icon follows the browser color scheme: the dark-M
"light" monogram on light chrome, the white-M "dark" monogram on dark.

Bump 0.93.10 -> 0.94.0 (feat).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 19:34:45 +02:00
MechaCat02
4e154434a1 fix(upload): stream chapter pages to storage instead of buffering the whole chapter
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.

Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.

Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.

Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:33:47 +02:00
MechaCat02
141cd52f7e fix(frontend): percent-encode fileUrl key segments
fileUrl interpolated the raw storage key into the path, so a key segment
containing a reserved character (`?`, `#`, `%`, space) could be
reinterpreted as a query/fragment delimiter. Encode each `/`-separated
segment with encodeURIComponent, keeping slashes as literal path
separators. Keys are backend-generated today, so this is defence-in-depth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:18:34 +02:00
MechaCat02
83a9ab40cd fix(crawler): reap dead jobs alongside done ones
The retention reaper deleted only `state = 'done'` rows, so terminal
`dead` jobs (retries exhausted) accumulated in crawler_jobs forever —
slow unbounded table growth. Both states are end-of-life and never
leased again.

Rename `reap_done` → `reap_terminal` and broaden the filter to
`state IN ('done','dead')`; `pending`/`running` stay untouched. Test
extended to assert old done + old dead both reap while fresh terminal and
old-but-active rows survive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:16:56 +02:00
MechaCat02
a141d65db1 fix(analysis): cap decoded pixels on the vision backend too
The vision prep pass decoded pages with bare `image::load_from_memory`,
which applies no allocation bound. `max_pixels` only downscales after a
full decode, so a tiny WebP/JPEG header declaring huge dimensions could
OOM the blocking worker — the same decompression-bomb the OCR backend
already guards against. Manga pages are commonly WebP/JPEG, where the
format's own self-limits are weaker than PNG's.

Route the decode through `decode_within`, applying an `image::Limits`
alloc cap sized from the shared `ocr_max_decode_pixels`
(ANALYSIS_OCR_MAX_DECODE_PIXELS). Add real-WebP bomb coverage asserting
both the helper and the end-to-end Undecodable fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:13:27 +02:00
MechaCat02
3a9e7ca2da fix(analysis): hold the OCR permit for the full blocking inference
The OCR dispatcher acquired a borrowed semaphore permit, then ran the
CPU-bound inference in `spawn_blocking`. On cancellation (graceful
shutdown) the future dropped the permit while the detached blocking task
kept running, briefly oversubscribing the blocking pool past
ANALYSIS_WORKERS.

Extract `run_ocr_blocking`, which acquires an *owned* permit and moves it
into the blocking task so the concurrency slot's lifetime matches the
actual work. Covered by a cancellation-invariant test asserting the
permit stays held after the caller future is aborted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:08:50 +02:00
MechaCat02
5dc93bfb84 fix(crawler): cap the number of page images per chapter
The per-image byte cap bounds each download but not the count, so a
hostile or compromised reader page listing thousands of <img> tags could
drive an unbounded disk fill. Add `CRAWLER_MAX_IMAGES_PER_CHAPTER`
(CrawlerConfig::max_images_per_chapter, default 2000, 0 = disabled) and
reject an over-cap chapter with a failed ack (exponential backoff) rather
than downloading it.

Threaded through sync_chapter_content and its three call sites (daemon
dispatcher, admin resync, CLI); enforced via `image_count_over_cap` right
after parse. The cap is an env-only safety knob, preserved across settings
reloads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:05:28 +02:00
MechaCat02
592747f1e0 fix(crawler): budget image tail against actual sniff-prefix length
The per-image download cap subtracted the constant SNIFF_PREFIX_BYTES (64)
from the budget instead of the bytes actually drained into the prefix.
Because the prefix loop appends whole chunks, a single ~16 KiB first chunk
fills the 64-byte sniff window in one drain, so the tail could then add
another near-full cap — storing up to ~2× max_image_bytes per page.

Capture prefix.len() before the prefix is moved into the stream and budget
the tail as `max_image_bytes - prefix_len` via a small `remaining_after_prefix`
helper (unit-tested for the overshoot and saturation cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:57:29 +02:00
MechaCat02
a0d63ac9fd fix(crawler): guard headless nav against internal SSRF targets on list/detail path
`navigate()` (list/detail/pagination) and the session probe called
`Browser::new_page()` without the SSRF check that already guards the
chapter-content path, so a hostile or compromised scraped source could
serve `<a href="http://169.254.169.254/…">` / `http://postgres:5432/` in a
listing and use the in-container Chromium as a read oracle.

Add `guard_navigate_url` (reusing `ensure_public_target`) at the top of
`navigate`, before rate-limiting or opening a page, and apply the same
check to `fetch_probe_html`. Covers base URL, pagination, and detail links.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:54:45 +02:00
MechaCat02
35c02066fe refactor(clippy): fix the never-loop error and clear all lint warnings
All checks were successful
deploy / test-backend (push) Successful in 30m17s
deploy / test-frontend (push) Successful in 10m33s
deploy / build-and-push (push) Successful in 11m11s
deploy / deploy (push) Successful in 13s
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:

- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
  literal `Err`).
- test helpers: build configs via struct-update syntax instead of
  `Default::default()` + field reassignment; add a type alias for a
  complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.

Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 20:45:37 +02:00
MechaCat02
ded77fe4ba test(e2e): kill cold-start flakiness with an /api/v1 fallback + one retry
A full parallel run intermittently failed the first few tests: unmocked
/api/v1 calls fell through to the dev proxy, whose backend isn't running
under E2E, so each incurred a slow ECONNREFUSED round-trip + Vite error
logging that piled up across 8 workers at cold start.

- Add e2e/fixtures.ts: a shared `test` whose auto-use fixture installs an
  `**/api/v1/**` fallback (fast 503, logged) for any endpoint a spec
  didn't mock. Registered before the test body, so per-test routes still
  win; only genuine gaps land here. Turns silent mock-gap hangs into
  instant, attributable failures — and cuts the full run ~96s → ~25s.
- Point all 22 specs at ./fixtures instead of @playwright/test.
- retries: 1 in the config as a safety net for the residual Vite
  cold-compile timing (re-runs on the now-warm server).

Two consecutive full runs: 110 passed, 0 flaky, no retries consumed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 20:06:20 +02:00
MechaCat02
4d02b56b77 test(e2e): realign mocks + assertions with the current API and OCR-era UI
The route-mocked E2E specs had drifted from the app they exercise, so
each rendered an empty/500 page and timed out on missing elements:

- Manga detail + reader loads now also fetch read-progress/{id} and
  /similar; several specs never mocked them, so the load's Promise.all
  rejected. Add the mocks and fill the manga fixtures out to a real
  MangaDetail (status/alt_titles/authors/genres/tags/... ) so the
  components stop throwing mid-render.
- back-nav + library: widen `read-progress*`/`page-tags` globs — `*`
  doesn't cross `/`, so `/me/read-progress/{id}` fell through to the
  proxy and 500'd; mock the library page-tags pair too.
- manga-list search: wait for the initial load to settle before
  searching so the search fetch can't race the mount-time load.
- reader-chapter-select: assert the shared `chapterLabel` output.
- admin-analysis: the active OCR backend extracts text only, so the
  detail modal shows OCR (no tags/NSFW) and metrics shows tiles +
  trend charts (no by-model) — assert what the OCR-era UI renders.
- profile: assert the wrong-password 401 stays inline (guards the
  fix in the preceding commit) and mock the guest bookmark/collection
  fetches the /profile load maps to the sign-in prompt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 20:11:23 +02:00
MechaCat02
0865659ab3 fix(auth): keep session on a wrong-current-password 401
The change-password endpoint returns 401 for a wrong *current* password,
the same status the global on401 hook uses to detect an expired session.
The hook fired first and cleared session.user, so the account form's
"still signed in?" guard always saw null and bounced the user to /login
instead of surfacing the error inline.

Add a per-call `suppressOn401` option to `request`; changePassword opts
in, so a 401 there stays inline and the session is left intact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 20:11:07 +02:00
MechaCat02
5596e1920d fix(compose): wire ANALYSIS_OCR_MAX_DECODE_PIXELS into the backend service
The OCR decompression-bomb guard added the env var to .env.example and
config.rs but not docker-compose.yml, so the container never received it
and the compose-env-coverage test failed once the branches were integrated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:25:59 +02:00
MechaCat02
3cba9ecf95 feat(auth): support optional expiry for bot API tokens
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:22:57 +02:00
MechaCat02
ed18d95bb0 feat(crawler): guard Chromium chapter navigation against internal targets
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:22:10 +02:00
MechaCat02
2ed42f7b9e feat(crawler): re-validate redirect hops to close SSRF via 3xx
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:21:15 +02:00
MechaCat02
cbbc626768 docs: correct ?text= OCR-search drift (no longer 501)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:19:38 +02:00
MechaCat02
a3e53f303b test(chapters): lock open-contribution upload contract + document intent
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:19:04 +02:00
MechaCat02
2af421893f fix(crawler): don't hold browser mutex across Chromium close()
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:49 +02:00
MechaCat02
48f0439273 fix(crawler): evict idle hosts from the per-host rate-limiter map
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:35 +02:00
MechaCat02
de7aefef69 fix(auth): guard the login + change-password verify paths against oversized passwords
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:19 +02:00
MechaCat02
3ba30f4ba9 fix(login): close ?next= control-char open-redirect + cover register
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:01 +02:00
MechaCat02
4306d1c96a fix(frontend): strip hop-by-hop headers from proxied responses too
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:17:32 +02:00
MechaCat02
886caaecfa fix(settings): bound manga_limit + the timeout knobs the finding named
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:17:14 +02:00
MechaCat02
de510cc19a fix(admin): make force-reanalyze audit transactional; document storage exception
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:16:08 +02:00
MechaCat02
5b76e0cc37 fix(analysis): bound concurrent OCR inferences with a shared semaphore
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:14:19 +02:00
MechaCat02
66ae4b221b fix(analysis): cap OCR decoded pixels to stop decompression-bomb OOM
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:12:40 +02:00
MechaCat02
83c2899373 feat(analysis): add ocrs OCR backend and OCR-driven page text search
All checks were successful
deploy / test-backend (push) Successful in 28m40s
deploy / test-frontend (push) Successful in 10m36s
deploy / build-and-push (push) Successful in 11m8s
deploy / deploy (push) Successful in 12s
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 19:52:43 +02:00
MechaCat02
4fb98e4a1e refactor(history): reuse IconButton; drop redundant emptyText overrides
All checks were successful
deploy / test-backend (push) Successful in 30m16s
deploy / test-frontend (push) Successful in 10m36s
deploy / build-and-push (push) Successful in 10m24s
deploy / deploy (push) Successful in 13s
The HistoryList clear button hand-rolled the same `.icon-btn danger`
styles that were extracted into the shared IconButton component one
commit earlier — the exact duplication that refactor targeted. Swap it
for <IconButton variant="danger">; data-testid/aria pass through the
component's {...rest} spread so behaviour is unchanged.

Both callers also passed an emptyText equal to the prop's default; drop
the overrides so the default lives in one place.

No behaviour change, so no version bump.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 21:51:05 +02:00
MechaCat02
5130c9933e feat(profile): add a desktop Page-tags section
Page tags were a first-class section of the mobile Library tab but had no
home in the desktop /profile tabs, so desktop users couldn't browse their
tagged pages without going through /search. Add a /profile/page-tags route
that reuses PageTagsList (the same component the mobile Library renders) and
a "Page tags" tab between Collections and History.

The loader mirrors the library wrapper's page-tags fetch; 401 → sign-in
prompt, matching the other profile sub-routes.

e2e: the desktop Page-tags tab navigates to /profile/page-tags and renders
the list.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
e9331747d0 feat(ui): present page-action editors as sheets on mobile
The Add-tags and Add-to-collection editors opened as centered Modals on
every form factor, so a mobile user went action-Sheet → centered Modal — an
inconsistent hop, and AddTagsSheet's docstring promised a sheet it never
got. Introduce AdaptiveDialog, which renders a bottom Sheet under 640px and
a centered Modal above it behind one shared contract (open / title / onClose
/ children / testid). Route both editors through it:

- AddToCollectionModal now wraps AdaptiveDialog (both its call sites — manga
  detail and the reader — adapt automatically).
- The reader hosts AddTagsSheet in an AdaptiveDialog instead of a Modal, so
  its "slots into a Sheet (mobile) or Modal (desktop)" contract is now true.

Modal/Sheet already share an API, so the switch is transparent: the dialog
keeps the same role, title, testid, and close affordance. Desktop is
unchanged.

AdaptiveDialog has its own unit test (Sheet on mobile, Modal on desktop);
an e2e proves the mobile long-press → Add-tag flow now opens a Sheet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
af6a07bd1f feat(nav): give admins a mobile entry point to the admin area
The admin area was reachable only from the desktop header's is_admin link;
on mobile there was no tab or row, so admins had to type the URL and no tab
highlighted. Add a conditional Admin row to the mobile Account hub and
`/admin` to the Account tab's match prefixes, so the tab stays highlighted
while an admin is in the admin area.

e2e: an admin session shows an Admin row linking to /admin; a non-admin
never sees it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
5e8323d7fa feat(reader): surface brightness on desktop, not just the mobile sheet
The brightness slider lived only in the mobile reader settings sheet, so
desktop users couldn't dim — and could be left stuck dimmed by a value a
phone had persisted (brightness is shared via localStorage and the
--reader-dim effect is form-factor agnostic). Add an inline brightness
slider to the desktop reader nav, marked `desktop-control` alongside the
existing mode toggle and gap select. It binds the same `brightness` state,
so a value set on either form factor stays in sync.

Also mocks `/me/read-progress` in the reader-mode e2e setup — without it the
GET/PUT fell through to the (absent) backend and stalled the reader load,
which is why those specs couldn't run headless. The spec is now hermetic and
covers the new desktop control (it dims/undims, and a phone-persisted value
is recoverable).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
2715275065 feat(history): share one HistoryList across desktop and mobile
The desktop /profile/history page and the mobile Library "History" tab
each hand-rendered reading history with separate markup, which had drifted:
the Library tab was a read-only 2-column list with no date and no
removed-chapter / whole-manga fallbacks, while /profile/history was a
3-column list with a per-row clear button. Extract a shared HistoryList
component so the two can't diverge again.

- New HistoryList.svelte owns row rendering, the optimistic-removal UX
  (instant remove, rollback + inline error on failure), and the empty
  state. Clear is opt-in via an `onClear` prop so a caller can render
  read-only, but both surfaces now pass it.
- The mobile Library History tab gains the clear action, the "Read {date}"
  line, and the (chapter removed) / whole-manga fallbacks it was missing.
- Continue label is built in script (was inline) so the " — page N" suffix
  keeps its spaces — Svelte trimmed them at the {#if} edge, rendering
  "Chapter 3— page 7". Both surfaces now read "Continue Chapter N".

Net -204 lines across the two pages. Uploads parity on the mobile tab is
left as a follow-up (it needs the library loader to fetch uploads).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
198a1d46b0 refactor(ui): fold the 36px header/search buttons into IconButton
Completes the IconButton extraction by adding `size` and `radius` props and
converting the two remaining outliers that were left out of the first pass:
the desktop header logout button (+layout) and the home search submit button
(+page), both 36px with the medium radius.

IconButton now drives every icon button in the app (except profile/history's,
which the shared-HistoryList change removes separately). Defaults stay 32px /
small radius, so the four already-converted sites are untouched. Rendered
size, radius, and variant match the old markup — no behaviour change, no
version bump.

This also sets up applying the --tap-min touch floor in one place once the
touch-target change lands.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
3a36796768 refactor(ui): extract shared IconButton from duplicated .icon-btn copies
Five files hand-rolled a near-identical 32px `.icon-btn` (same size, hover,
and primary/danger variants). Extract a single IconButton.svelte component
so the treatment lives in one place. Converts the four sites with the
standard 32px form: collections detail, manga edit, upload, and the
chapter-pages editor.

The component takes a `variant` (plain/primary/danger) and spreads any
button attributes (onclick, disabled, aria-label, title, data-testid)
straight through; `type="button"` defaults but a caller can override. The
rendered button keeps the same class, styles, and DOM position, so layout
and behaviour are unchanged — no version bump.

Three icon-button sites are intentionally left out:
- The header (+layout) and home search button are 36px / different radius —
  size outliers that need a size/radius prop before folding in.
- profile/history's copy is being removed in the shared-HistoryList change;
  touching it here would just conflict.

A component (not a global class) avoids colliding with those remaining
local `.icon-btn` definitions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:27:58 +02:00
MechaCat02
bd7c3fc28c refactor(catalog): derive desktop sort options from SORT_FIELD_LABELS
The desktop sort <select> hard-coded its four <option>s while the mobile
sort sheet rendered them from SORT_FIELD_LABELS. The two happened to match,
but a label rename in mangaSort.ts would have silently diverged the desktop
control. Render the desktop options from the same map so they can't.

No behaviour change (the rendered options are identical), so no version
bump. Adds an e2e guard asserting the desktop select's option values and
labels equal SORT_FIELD_LABELS in order — verified red against the old
hard-coded markup (with a label renamed) and green after.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:25:27 +02:00
MechaCat02
4456deb146 fix(collections): make per-card remove buttons usable on touch
The remove buttons on a collection's cards were revealed only on
hover / focus-within (opacity 0 → 1), so on touch they were invisible and
undiscoverable. Show them persistently at the 640px breakpoint and enlarge
the hit area from 24px to 32px (kept below the full 44px floor because the
button floats over a 4-up cover thumbnail).

Test pins the CSS contract (jsdom can't evaluate @media): the mobile block
makes `.remove` opaque.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:25:27 +02:00
MechaCat02
94591dfda4 fix(manga): show force-resync result on mobile, not just desktop
`.resync-msg` (the admin Force-resync result) was display:none under 640px,
so mobile admins — who trigger resync from the overflow sheet — got no
success or error feedback. The element is already a sibling of `.action-row`
(not a child), so it survives the row being hidden; the only thing hiding it
was an explicit mobile rule. Drop that rule.

e2e: a mobile admin triggers Force resync from the overflow sheet and now
sees the "Metadata updated" result (written test-first against the hidden
element, then unhidden).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:25:27 +02:00
MechaCat02
194adf1339 fix(ui): meet 44px touch-target floor on shared mobile primitives
The desktop/mobile consistency review found primary mobile controls
rendering below the ~44px comfortable touch floor. Address the genuinely
shared primitives in one pass:

- Add a --tap-min: 44px token (documented anchor for the floor).
- tokens.css: text inputs, selects, and submit buttons grow to --tap-min
  under the 640px breakpoint (fixes the 36px auth/login form controls).
  Scoped to form controls + type=submit so dense icon buttons aren't
  inflated.
- SegmentedControl: .seg options reach the floor on mobile (it doubles as
  the catalog sort toggle and the Library tab switcher).
- Chip: expand .chip-remove's tappable area to --tap-min via a centered
  pseudo-element so the 16px glyph stays compact and tag rows don't grow.

Pure-CSS responsive change — jsdom can't evaluate @media or layout, so the
test pins the stylesheet contract (each shared primitive bumps to --tap-min
inside the mobile breakpoint), guarding against the bump being dropped.

The per-route .icon-btn (copy-pasted across seven files) is left for a
separate refactor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 21:25:27 +02:00
MechaCat02
dee53fa212 feat(mangas): per-field sort options + direction toggle on the catalog (0.88.0)
Sort the manga catalog by created / updated / title / author with an
independent asc/desc direction control. An omitted `order` defaults per field
(dates newest-first, text A→Z), matching the UI, so a bare `?sort=<field>`
means the same thing in the browser and over the API; `sort=recent` remains a
back-compat alias for `created`.

- Backend: SortField/SortOrder parsed with validation (structured 422 on bad
  input), per-field default_order, NULLS LAST only on the nullable author key,
  and migration 0033 indexing mangas(updated_at DESC, id) to back the default
  sort and its id tie-break.
- Frontend: catalog sort field + direction (SegmentedControl) on desktop and a
  mobile bottom sheet; pure helpers in $lib/mangaSort; keyboard-accessible
  direction control; visible Direction labels and a sheet "Done" button.
- Tests: backend integration coverage (defaults, alias, invalid input,
  ascending-id tie-break), frontend unit + e2e.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-25 07:15:51 +02:00
MechaCat02
93b7e451bf fix(jobs): per-lease generation token closes the ack-from-dead-lease race (0.87.26)
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".

Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.

Split the release surface in two:
  * release(lease_id, lease_generation) — owner-aware path for
    workers (graceful shutdown, session-expired), matches the
    caller's specific lease.
  * release_unowned(lease_id) — id-only path for force-analyze and
    ops tools that drop a running lease without holding a Lease
    struct; unconditionally bumps generation so any pending ack
    from the in-flight original becomes a no-op.

Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.

Five new tests in tests/crawler_jobs.rs pin every facet:
  * lease_assigns_strictly_increasing_generation_per_release
  * ack_done_from_dead_lease_after_release_unowned_is_a_no_op
  * ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
  * renew_from_dead_lease_is_a_no_op
  * release_unowned_bumps_generation_even_when_lease_is_held

Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-24 20:05:55 +02:00
MechaCat02
ae708a72d0 fix(config): derive BACKEND_CONSUMED from .env.example; scan src/ for undocumented env reads (0.87.25)
0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.

Make .env.example the single source of truth:

  * docker_compose_wires_every_documented_backend_env_var now reads
    .env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
    (compose-side composed vars, frontend-only vars, sidecar vars),
    and requires every remaining key to be wired into compose with a
    matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
  * every_backend_src_env_read_is_documented_or_allowlisted scans
    backend/src/ for env::var / env_bool / env_i64 / ... reads and
    requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
    with a per-entry rationale. Catches the silent-no-op bug from the
    other side.

Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.

Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-24 19:28:00 +02:00
MechaCat02
579e6ade0f fix(cancellable): suppress non-Abort errors from superseded predecessors (0.87.24)
CancellableLoader.run() previously swallowed AbortError but propagated
every other rejection. A predecessor whose fetch survived its abort
(timing race, or fn that ignores signal) and then 5xx'd would bubble
the error into the caller's catch, clearing the successor's spinner
and writing its `error` field — exactly the flicker the loader was
supposed to prevent.

Strengthen the contract: a call that no longer owns this.current
returns null regardless of how it died. Symmetric with the existing
late-resolve suppression at the success path. The loadCoverage catch
in admin/analysis/+page.svelte now only fires when this call genuinely
failed AND still owns the loader — so clearing the spinner there is
sound.

Three new unit tests pin all three contract corners: a superseded
predecessor's non-Abort rejection resolves to null; a standalone call
with no successor still throws normally; the AbortError swallow on a
still-owning call (synthetic AbortError from fn with no successor or
cancel()) remains protected, since the new supersede check would
otherwise have eaten every existing AbortError-path test and left
that branch as silently-removable dead code.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-24 07:11:01 +02:00
MechaCat02
4e14ed09ef test(frontend): extract CancellableLoader; pin debounce-race invariants in unit tests (0.87.23)
0.87.16 followup. Extract the four debounce-race invariants
(predecessor abort, late-result suppression, AbortError swallow,
ownership-aware finally) from `loadCoverage` into a reusable
`CancellableLoader` helper at $lib/util/cancellable, with 9 unit
tests covering each. Page rewrite uses the helper.

Also corrects: hooks.server.test.ts (added Node-fetch rejection
shape for already-aborted entry; removed redundant tick); admin.test.ts
(AbortError-propagation now uses pre-aborted signal + mockRejectedValueOnce
mirroring real Node fetch).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 21:03:12 +02:00
MechaCat02
2ee77bc867 test(vision): tighten spawn_blocking interval test against Burst-replay (0.87.22)
0.87.14 followup. The 0.87.14 test used MissedTickBehavior::Burst —
any post-prep yield replayed missed ticks and could fake the >= 2
threshold. Switch to Skip and assert on `ticks_post - ticks_pre >= 50`,
which sits well above the inlined-prep ceiling (~5) and far below
the spawn_blocking floor (~700+). Mutation-confirmed.

Also corrects: prepare_analysis doc count ("five fallback paths" → 4).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:52:42 +02:00
MechaCat02
a62a5f155b test(analysis): assert Cancelled event is published; correct cron-test rationale (0.87.21)
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.

Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:46:08 +02:00
MechaCat02
e3e86843d6 fix(force-analyze): release running lease so worker drops stale-payload local (0.87.20)
0.87.11 followup. The UPDATE matched pending+running but the worker
holds the pre-update `force=false` in an in-memory local — so on the
running branch the row flipped but the worker's skip-if-done still
acked done without re-analyzing. Now `jobs::release` any running row
the UPDATE matched, so the worker's ack_done no-ops (state guard) and
a fresh lease picks up the updated payload.

New test mints the running-state shape, asserts state=pending +
force=true + attempts=0 post-click. Mutation-confirmed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:42:11 +02:00
MechaCat02
0f9254b054 test(authz): bearer-authed admin cannot edit/cover NULL-uploader mangas (0.87.19)
0.87.10 followup from the rereview. The fix lacked a test for the
specific axis it changed (admin authority via cookie only). New test
mints a bearer for a promoted admin and asserts PATCH/PUT-cover/DELETE-
cover all 403 on NULL-uploader rows. Mutation-confirmed.

Also: `MultipartBuilder::finalize` now `pub` for bearer-multipart use;
`admin_csrf_guard` rustdoc updated to describe the post-0.87.10
cookie-precedence rule.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:36:50 +02:00
MechaCat02
d6a4fd668c test(vision-manager): extract refresh_running_after_mem_check; assert log distinguishes fix from bug (0.87.18)
Two 0.87.15 ship-blockers:

1. mem_check_and_stop test now ALSO greps for the "deferring to next
   tick" warning — distinguishes fix from bug. Mutation-confirmed.

2. Extract `refresh_running_after_mem_check` (sets global instead of
   echoing so loop-scope flags persist across calls) and drive it
   directly from the harness, closing the structurally-untestable
   post-mem refresh ERR guard.

Adjacent: log-on-flip gating on both ERR warnings; `log` to stderr;
dangling-TempDir in app::tests reclaim test fixed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:18:56 +02:00
MechaCat02
e3d16e49b7 fix(config): ANALYSIS_VISION_MODEL reader + compose RHS-interpolation test (0.87.17)
Two ship-blockers from the rereview:

1. Env var name mismatch: 0.87.12 renamed every operator surface to
   ANALYSIS_VISION_MODEL but the reader still called env::var on
   ANALYSIS_MODEL — every documented deploy ran analysis with an
   empty model. Rename the reader.

2. Compose regression test was substring-only and missed exactly the
   ANALYSIS_VISION_MODEL vs ANALYSIS_MODEL RHS mismatch from 0.87.12.
   Tightened to require each backend-consumed var either be in a
   small hardcoded allowlist (BIND_ADDRESS, STORAGE_DIR) or have its
   RHS interpolate `${KEY...}` with the same name. Mutation-confirmed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 20:05:17 +02:00
MechaCat02
54cbbfc440 test(frontend): pin abort-chain + signal pass-through contracts (0.87.16)
Three 0.87.9 follow-ups:

1. hooks.server.ts: two new tests covering the
   already-aborted-at-entry and mid-flight client-disconnect branches
   of the abort chain. Mutation-confirmed.

2. lib/api/admin.ts: two new tests pinning the
   getAnalysisMangaCoverage(init?.signal) pass-through and the
   AbortError-rejection propagation the dashboard's debounce-race
   handler relies on. Mutation-confirmed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 19:22:48 +02:00
MechaCat02
27fc1a52f6 fix(vision-manager): symmetric ERR handling + smoke tests + analysis reload reclaim test (0.87.15)
0.87.7 + 0.87.8 follow-ups:

1. mem_check_and_stop now defers on docker ERR instead of silently
   no-op'ing the SIGTERM under pressure.
2. Post-mem_check_and_stop loop refresh now also handles the ERR
   sentinel (continues the loop) rather than letting it reset up_for
   and defeat MAX_UPTIME.
3. New `vision-manager/test_manager.sh` — 7 smoke tests covering
   vision_running's three exit shapes and mem_check_and_stop's ERR
   symmetry, via stubbed docker/psql/curl on PATH. Production loop
   guarded by MANAGER_TEST_NO_MAIN.
4. New `app::tests::spawn_analysis_daemon_reclaims_orphaned_analyze_leases`
   pins the 0.87.7 wire-through: seeds an expired analyze_page lease,
   spawns + shuts down, asserts pending + attempts refunded.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 19:18:54 +02:00
MechaCat02
95de02f583 fix(analysis-vision): pin spawn_blocking dispatch + warn on undecodable fallback (0.87.14)
Two 0.87.5 follow-ups:

1. Pin spawn_blocking dispatch: new current_thread runtime test runs
   analyze() on a 6 MP image while a counter task ticks every 5 ms.
   Without spawn_blocking the counter is starved (0 ticks); with it
   the runtime stays responsive (≥2 ticks). Mutation-confirmed.

2. Warn on Undecodable fallback: five fallback paths in prepare_analysis
   now emit a tracing::warn! with dimensions so a JPEG encoder
   regression isn't indistinguishable from "the page was just garbage."

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 19:07:02 +02:00
MechaCat02
651fb27522 fix(analysis): emit Cancelled event + assert lock-release after cancel (0.87.13)
Two 0.87.4 follow-ups:

1. Cron tick: existing shutdown-on-cancel test now also asserts the
   advisory lock is re-acquirable after cancel-during-tick. Without
   this, a refactor moving `pg_advisory_unlock` under the cancel arm
   would ship green and break multi-replica safety.

2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
   on the cancelled page until a later page overwrote it. Add a
   `Cancelled` AnalysisEvent variant, publish on the cancel-release
   branch, wire the analysis and admin overview dashboards to clear
   on it. Per-page chip rolls back from `analyzing` to `queued`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 19:00:38 +02:00
MechaCat02
95b98eebf1 fix(compose): wire all documented backend env vars + regression test (0.87.12)
0.87.1 added 12+ env vars to `.env.example` but the compose backend
`environment:` block forwarded none of them — `.env` interpolation
doesn't pass vars to the container. Operators got
anonymous-serving sites, silent ADMIN_* no-ops, and missing
ANALYSIS_* configuration.

Wire 36 backend-consumed vars; also substitute the hardcoded
`BACKEND_URL` on the frontend; emit a startup warning on half-set
`ADMIN_USERNAME`/`ADMIN_PASSWORD`. Regression-test parses
docker-compose.yml and asserts the wiring.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 07:25:57 +02:00
MechaCat02
34d6d570eb fix(analyze-dedup): migration pre-dedup, force-collision upgrade, gate test (0.87.11)
Four 0.87.6 follow-ups from the adversarial review:

1. **Migration 0031 pre-dedup.** Demote all-but-lowest-id duplicate
   `analyze_page` rows in `(pending|running)` to `dead` before
   creating the unique index, with a curator-recoverable last_error
   marker. Without this, `sqlx::migrate!` would refuse to boot on
   any dirty production DB.

2. **`enqueue_for_page(force=true)` collision.** The partial unique
   index used to silently swallow force requests when a
   `force=false` job was already pending. Repo function now upgrades
   the pending row's `force` flag in place (or falls through to
   re-INSERT if the sibling drained mid-call), and reports an
   `EnqueueForPageOutcome` for accurate auditing.

3. **`record_duration` gate test.** New test seeds a `done` row with
   known duration, force-re-analyzes with failing dispatcher +
   max_attempts=3 (non-terminal), asserts duration_ms wasn't
   overwritten.

4. **`bookmark.rs` PK comment correction.** Use `b.id DESC` instead
   of the wrong `manga_id`; PK is actually `id`, and migration 0004
   allows both chapter-level and manga-level bookmarks on the same
   manga.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-23 07:22:20 +02:00
MechaCat02
93bb156fba fix(admin-security): close the bearer-cookie ride; require session for admin gate (0.87.10)
Two follow-ups to 0.87.2 + 0.87.3 from the adversarial review:

1. **CSRF cookie-ride.** Bearer-bypass branch checked Authorization
   header presence only — an attacker page could mint `Authorization:
   Bearer junk` and ride the still-attached session cookie. Bypass now
   requires bearer AND no cookie. Drive the cookie-name comparison off
   `SESSION_COOKIE_NAME` and split on `=` (not prefix).

2. **`require_can_edit` admin path open to bearer.** Now reads admin
   authority off an `Option<CurrentSessionUser>` extractor — None on
   bearer-only requests. Owner-match path still accepts bearer; admin
   path is cookie-gated.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 22:28:16 +02:00
MechaCat02
747bdeda46 fix(frontend-admin): SSE/AbortController hygiene + signal-chain through proxy (0.87.9)
Three frontend findings (medium):

1. **`hooks.server.ts` didn't chain client-disconnect into the upstream
   fetch.** Closing an admin tab left the backend's `broadcast::Receiver`
   + spawned tokio task running until next backend restart. Listen for
   `event.request.signal.abort` and forward to the timeout controller.

2. **`admin/analysis/+page.svelte` missing visibility/pagehide/pageshow
   handlers.** Mobile Safari throttles SSE in background tabs; bfcache
   leaves a dead connection on back-nav. Apply the close/open-stream
   pattern from `admin/crawler/+page.svelte`.

3. **`loadCoverage` had no AbortController.** Debounced search
   keystrokes raced. Per-loader controller plumbed through
   `getAnalysisMangaCoverage`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:52:33 +02:00
MechaCat02
c0281f7e9b fix(vision-manager): disambiguate docker errors, fail-symmetrically on psql, stop wedged container (0.87.8)
Three medium vision-manager findings:

1. **`vision_running()` conflated "container absent" with "inspect
   failed".** A transient docker-socket-proxy hiccup made the manager
   think vision had stopped, which reset `up_for`/`idle_for` and
   defeated MAX_UPTIME. Distinguishes: exit 0 → echo true/false;
   "No such container" → false; else → ERR (loop skips tick).

2. **`crawl_running()` fail-closed where `analysis_enabled()` fail-opens.**
   A psql `ERR` logged a misleading "RAM mutex" line every tick.
   Numeric-guard at the call site with a distinct log.

3. **`start_vision` left wedged containers running** on
   `START_HEALTH_TIMEOUT` expiry — pinned forever in not-ready state.
   Now `stop_vision` so next tick retries from a clean state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:47:33 +02:00
MechaCat02
6444ddee29 fix(analysis): reclaim orphaned leases from analysis daemon startup too (0.87.7)
`reclaim_orphaned` previously ran only inside `spawn_crawler_daemon` at
startup. A deploy with the crawler disabled and analysis enabled
(analysis-only mode) never refunded orphaned `analyze_page` leases —
recovery deferred to the lazy lease-expiry path. Wire it into
`spawn_analysis_daemon` too. Safe under multi-replica + idempotent
against the crawler-side call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:44:14 +02:00
MechaCat02
b9dd75684e fix(correctness): pagination tiebreakers, dedup race, duration gating (0.87.6)
Three medium correctness fixes:

1. **Pagination ORDER BY missing tiebreakers** — four admin list queries
   sorted by a timestamp or `chapters.number` with no stable secondary
   sort. Add `id` tiebreakers across `repo::admin_view`,
   `repo::admin_audit`, `repo::bookmark`.

2. **`enqueue_pages` NOT EXISTS read-then-insert race** — concurrent
   admin clicks could land duplicate analyze_page jobs. Migration 0031
   adds a partial unique index mirroring 0014; query relies on
   `ON CONFLICT DO NOTHING`.

3. **`record_duration` overwrote a prior done row's duration on a
   non-terminal failed retry of a force-re-analyze.** Gate the call
   on "did this attempt actually write a row?".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:34:39 +02:00
MechaCat02
80f4819fad perf(analysis): run image decode/resize/encode on the blocking pool (0.87.5)
`VisionClient::analyze` ran `image::load_from_memory`, `DynamicImage::resize_exact(Triangle)`,
and per-band JPEG encodes directly on the tokio task. Each call is
hundreds of ms of pure CPU per page. With multiple workers, the runtime
threads got starved — axum handlers, SSE streams, the crawler daemon's
async timers, and other tasks sharing the runtime all stalled while
analysis was active.

Extract a `prepare_analysis` helper that does ALL the CPU work
(decode + plan + width-reduce + slice + JPEG encode) and returns a
`PreparedAnalysis { Single | Sliced | Undecodable }` of already-encoded
byte payloads. `analyze` calls it inside `tokio::task::spawn_blocking`,
then the async HTTP loop only iterates over the prepared bytes — no
image-crate operations happen on the runtime any more.

Single page → one spawn_blocking → one HTTP call.
Tall page → one spawn_blocking → N OCR calls + 1 grounding call.

Memory peak rises slightly for the Sliced path (all bands held
encoded at the same time instead of one-at-a-time) — for a typical
manga page split into 4 slices at ~200 KB each, that's ~600 KB of
extra peak. Negligible vs. the runtime-starvation cost.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:23:34 +02:00
MechaCat02
660184a048 fix(daemons): race in-flight work against cancellation on shutdown (0.87.4)
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.

**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.

Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.

**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:18:18 +02:00
MechaCat02
ee9f5c1a4d fix(authz): require admin to edit/cover crawler-imported mangas (0.87.3)
`require_can_edit` matched `Some(owner) != caller → Forbidden` but let
`None` through. Every crawler row (`repo::crawler::upsert_manga`) has
NULL `uploaded_by`, so any signed-in user could PATCH /api/v1/mangas/<id>
and rewrite the catalog — `update`, `put_cover`, and `delete_cover`
were all in scope. With self-registration on by default, the path was:
register → mass-edit catalog + delete cover blobs from storage.

The original carve-out at the comment said "Once an admin role lands the
NULL case can flip to admin-only." The admin role landed in migration
0018; flipping it now. New rule:

  * `Some(owner)` and owner == caller → ok (unchanged)
  * `Some(_)` and caller is admin → ok (admin moderation)
  * `Some(_)` otherwise → 403 (unchanged)
  * `None` and caller is admin → ok (NEW — operator can curate)
  * `None` otherwise → 403 (was: ok — the bug)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:09:34 +02:00
MechaCat02
dd25f073cd fix(admin-security): SSRF defence + CSRF fail-closed + analysis .no_proxy() (0.87.2)
Four high/medium findings from the audit, plus their tests:

1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
   `AnalysisSettings::endpoint` validated only with `Url::parse`. A
   hostile or CSRF-able admin could repoint at `169.254.169.254`
   (cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
   — and the vision worker bearer-attaches an env-managed API key to
   every call. Extract `ensure_public_target` from `safety::is_safe_url`
   (allowlist-free public-host check: scheme + private-IP literal +
   localhost) and wire it into both fields. Docker DNS names
   (`mangalord-vision`) keep passing because they're hostnames, not
   IP literals. The analysis check is gated on `enabled=true` so the
   dev-default `localhost:8000` stays usable until the worker is
   actually turned on (toggling enabled=true later re-runs the gate).

2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
   silently skipped the entire CSRF check, so an operator forgetting
   the env var shipped an unguarded admin surface. Cookie-auth POSTs
   now fail-closed when the allowlist is empty AND when neither
   `Origin` nor `Referer` accompanies the request. Two carve-outs:
   `Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
   and requests with NO session cookie at all bypass to let the auth
   extractor return a clean 401 instead of a confusing 403.

3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
   already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
   would exfiltrate page-image bytes + the bearer key through an
   upstream proxy. Same for the readiness probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 21:04:50 +02:00
MechaCat02
b14ed02670 fix(infra): plug operational defaults — pg restart, proxy timeout, env template (0.87.1)
Three operational gaps tripped a recent review:

1. **Postgres had no `restart:` policy.** tor / docker-socket-proxy /
   vision-manager all set `unless-stopped`; pg was the lone exception.
   A transient OOM or panic left backend's `depends_on: service_healthy`
   blocking every future restart and took the stack offline. Add
   `unless-stopped` on postgres and on backend/frontend in the base
   compose so the prod overlay is the same shape.

2. **`BACKEND_PROXY_TIMEOUT_MS` was documented but silently a no-op.**
   `.env.example` carried it but `docker-compose.yml` never forwarded it
   into the frontend service env, so the value never reached
   `hooks.server.ts`. Wire it through.

3. **`.env.example` omitted required env vars.** Operators copying the
   template got no admin bootstrap, no boot-seed crawler URL, no
   analysis configuration. Add: `ADMIN_USERNAME`, `ADMIN_PASSWORD`,
   `PRIVATE_MODE`, `ALLOW_SELF_REGISTER`, `CRAWLER_START_URL`,
   `CRAWLER_CDN_HOST`, `ANALYSIS_ENABLED`, `ANALYSIS_VISION_URL`,
   `ANALYSIS_VISION_MODEL`, `ANALYSIS_WORKERS`,
   `ANALYSIS_JOB_TIMEOUT_SECS`, `ANALYSIS_API_KEY` — each with the
   comment block that explains seed-vs-runtime semantics.

Also adds `frontend/vite.config.ts.timestamp-*.mjs` to `.gitignore` — a
crashed dev server leaves these transient files behind otherwise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-22 20:23:56 +02:00
MechaCat02
9f7dfe4d4e fix(admin): keyboard-reachable audit payload toggle; drop dead step ternary
All checks were successful
deploy / test-backend (push) Successful in 27m23s
deploy / test-frontend (push) Successful in 10m20s
deploy / build-and-push (push) Successful in 10m42s
deploy / deploy (push) Successful in 13s
Review follow-ups: the audit row's expand was mouse-only (onclick on a bare
<tr>). Move the toggle to a real <button> in the actions cell with
aria-expanded + aria-label so keyboard/AT users can open a row's payload; the
row onclick stays as a mouse convenience (stopPropagation avoids a double
toggle). Also simplify a no-op `step={'%' ? '1' : '1'}` to step="1" on the
threshold inputs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:55:48 +02:00
MechaCat02
c6a6d1690d feat(admin): time-series trend charts on the metrics tabs (0.87.0)
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.

Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.

Closes the Phase-1 observability set (audit log · health checks · trends).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:48:56 +02:00
MechaCat02
314fc8738b feat(admin): operational health checks page with editable thresholds
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.

GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).

Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:38:18 +02:00
MechaCat02
31013cc893 feat(admin): audit-log viewer
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.

Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:26:09 +02:00
MechaCat02
dd300a150c feat(crawler): reconcile pass to enqueue mangas missing from the DB (0.85.0)
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.

- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
  to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
  chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
  The crawl worker now leases both sync_chapter_content and sync_manga
  (jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
  the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
  skipped when a pending/running/dead SyncManga job already exists, so a gone
  manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
  manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
  phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
  them for sync_manga rows; both searches match the payload title.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-19 11:15:11 +02:00
341 changed files with 26784 additions and 3732 deletions

View File

@@ -25,6 +25,12 @@ DATABASE_URL=postgres://mangalord:mangalord@postgres:5432/mangalord
BIND_ADDRESS=0.0.0.0:8080
STORAGE_DIR=/var/lib/mangalord/storage
RUST_LOG=info,mangalord=debug,chromiumoxide::conn=off,chromiumoxide::handler=off
# Postgres connection-pool sizing. One pool serves HTTP handlers and the
# crawler/analysis daemons. DB_MAX_CONNECTIONS caps open connections;
# DB_ACQUIRE_TIMEOUT_SECS is how long a request waits for a free connection
# before failing fast (rather than hanging on the driver's 30s default).
DB_MAX_CONNECTIONS=20
DB_ACQUIRE_TIMEOUT_SECS=10
# ----- Auth / cookies -----
# COOKIE_SECURE controls whether the `Secure` flag is set on the session
@@ -45,6 +51,13 @@ SESSION_TTL_DAYS=30
# rate-limiting reverse proxy that already enforces a budget).
AUTH_RATE_PER_SEC=5
AUTH_RATE_BURST=10
# Trust a proxy-supplied X-Forwarded-For as the client IP for per-IP auth
# rate limiting. Enable ONLY when the backend sits behind a trusted proxy
# that overrides the header (the compose deploy: SvelteKit forwards the real
# peer IP). When false, the header is ignored and a single shared bucket is
# used — a directly-exposed backend MUST keep this off or clients could spoof
# their IP to dodge the limit.
AUTH_TRUSTED_PROXY=false
# ----- CORS -----
# Comma-separated origins allowed to call the API with credentials.
@@ -62,13 +75,39 @@ CORS_ALLOWED_ORIGINS=
# neither Origin nor Referer (curl, server-to-server callers) are
# always allowed.
#
# Default is empty: CSRF check disabled (operator opt-out). For a
# browser-exposed deployment this should be set to the SvelteKit
# origin, e.g. https://app.example.com. For a same-origin
# docker-compose deploy where only one origin exists, set the same
# value the browser uses.
# Empty does NOT mean "off" for browsers: cookie-authenticated admin
# mutations FAIL CLOSED (403) when this is empty, since there's no
# allowlist to check the Origin against. Non-cookie callers (curl,
# bots with a Bearer token, no Origin/Referer) are still allowed.
# So set this to the SvelteKit origin for any browser-exposed deploy,
# e.g. https://app.example.com. For a same-origin docker-compose deploy
# set the same value the browser uses.
# Local dev (native `npm run dev`): the Vite origin is
# http://localhost:5173 — set ADMIN_ALLOWED_ORIGINS=http://localhost:5173
# or admin toggles (e.g. enabling the analysis worker) return 403.
ADMIN_ALLOWED_ORIGINS=
# ----- Admin bootstrap -----
# When BOTH are set and non-empty, the backend ensures a user with this
# username exists at startup, promotes it to admin, and (for a brand-new
# row only) sets the password. An existing user's password is NEVER
# overwritten by this — rotate via the dashboard or `/auth/me/password`.
# Leave BOTH empty in a fully provisioned deploy.
ADMIN_USERNAME=
ADMIN_PASSWORD=
# ----- Site-wide auth gates (env-ONLY) -----
# PRIVATE_MODE: when `true`, every endpoint except a small public allowlist
# (/health, /auth/config, /auth/login, /auth/logout) demands a valid
# session — anonymous reads return 401. Self-registration is also
# force-disabled regardless of ALLOW_SELF_REGISTER. Default `false`.
PRIVATE_MODE=false
# ALLOW_SELF_REGISTER: when `false`, /auth/register returns 403 and the
# frontend hides its register affordance. Admins can still mint accounts
# via /admin/users. Default `true` (open registration). Forced to `false`
# when PRIVATE_MODE=true.
ALLOW_SELF_REGISTER=true
# ----- Upload limits -----
# Per-request body cap. axum rejects oversized requests with 413 before
# our handlers run. Default 200 MiB.
@@ -77,6 +116,13 @@ MAX_REQUEST_BYTES=209715200
# oversized image is rejected even when the total request fits.
# Default 20 MiB.
MAX_FILE_BYTES=20971520
# Max page images accepted in one chapter upload. Bounds how many parts
# the handler will stage before rejecting the request with 413, so a
# client can't pin a worker with an unbounded page count. Default 2000.
# Setting 0 disables THIS cap — the total is then bounded only by
# MAX_REQUEST_BYTES (the whole-request body limit above), so leave 0 only
# if you intend that body limit to be the sole backstop.
MAX_PAGES_PER_CHAPTER=2000
# ----- Crawler download safety -----
# Hosts the crawler is allowed to fetch images/covers from, in addition
@@ -91,6 +137,11 @@ CRAWLER_DOWNLOAD_ALLOWLIST=
CRAWLER_ALLOW_ANY_HOST=false
# Hard cap on a single image body. Default 32 MiB.
CRAWLER_MAX_IMAGE_BYTES=33554432
# Hard cap on the number of page images in one crawled chapter. The
# per-image byte cap doesn't stop a hostile reader page listing thousands
# of <img> tags; an over-cap chapter is acked failed instead of downloaded.
# 0 disables the cap. Default 2000.
CRAWLER_MAX_IMAGES_PER_CHAPTER=2000
# Max manga detail fetches per metadata pass (both the in-process daemon
# and the `bin/crawler` CLI). 0 means no cap — let the source walker run
# to completion. Useful for capped test runs against a new source.
@@ -101,6 +152,39 @@ CRAWLER_LIMIT=0
# A job that exceeds the budget is acked failed (with exponential
# backoff) instead of wedging a worker. Default 600s.
CRAWLER_JOB_TIMEOUT_SECS=600
# Idle seconds the daemon waits for new jobs before parking (between
# scheduled passes). Lower for snappier dev loops; higher for fewer
# wakeups in steady state. Default 600.
CRAWLER_IDLE_TIMEOUT_S=600
# Concurrent chapter-content workers. Each pulls one chapter at a time
# from the queue. Higher = faster on multi-core hosts BUT also more
# pressure on the source (raise CRAWLER_RATE_MS in tandem). Default 1.
CRAWLER_CHAPTER_WORKERS=1
# Days of completed `crawler_jobs` rows kept before vacuum (per-row
# audit trail; failed rows survive to make retries traceable). Default 7.
CRAWLER_JOB_RETENTION_DAYS=7
# Days of `crawl_metrics` rows kept (per-run aggregate dashboard
# data). Longer history → bigger dashboard charts; shorter → less DB.
# Default 90.
CRAWL_METRICS_RETENTION_DAYS=90
# Politeness rate limit between requests to the source (milliseconds).
# Pause inserted after each fetch. Raising slows the crawl but is
# friendlier to small sources. Default 1000ms.
CRAWLER_RATE_MS=1000
# Politeness rate for the CDN host (CRAWLER_CDN_HOST). Defaults to
# CRAWLER_RATE_MS when unset — set lower if the CDN tolerates faster
# pulls, or higher if it rate-limits aggressively. Default 1000ms.
CRAWLER_CDN_RATE_MS=1000
# User-Agent string for crawler HTTP requests. Leave unset for a
# generic default. Some sources fingerprint UA — set a realistic
# browser UA if the source 403s the default.
CRAWLER_USER_AGENT=
# Source session cookie value (PHPSESSID). Required when the source
# gates listings behind a logged-in session. Treat as a secret.
CRAWLER_PHPSESSID=
# Cookie domain the crawler scopes its session to. Usually the source
# host (e.g. `mangadex.org`). Leave unset to let reqwest infer it.
CRAWLER_COOKIE_DOMAIN=
# Consecutive metadata-pass `fetch_manga` failures that abort the pass
# (circuit breaker for a source outage). The pass does NOT mark a clean
# exit, so the next tick does a recovery sweep. Default 10.
@@ -109,6 +193,19 @@ CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
# exhausted) that trigger an automatic coordinated browser restart.
# Default 3.
CRAWLER_BROWSER_RESTART_THRESHOLD=3
# CDP Fetch interception that re-validates every headless-browser
# navigation/redirect/subresource against the SSRF check (blocks a scraped page
# that drives the browser — via a redirect OR page JS/subresource — to an
# internal target such as the cloud metadata service or postgres). Default TRUE:
# with it off, only the top-level URL string is checked and page JS/subresources
# reach internal IPs (the reqwest SafeResolver does not cover Chromium's own
# stack). The CDP Fetch hook can't be exercised in CI (no Chromium); before
# relying on a fresh deploy, validate it doesn't wedge navigation:
# CRAWLER_CHROMIUM_BINARY=/usr/bin/chromium \
# cargo test -p mangalord --test crawler_browser_smoke -- --ignored \
# ssrf_interception_does_not_wedge_allowed_navigation
# Set to false only as a break-glass if the hook destabilizes a deployment.
CRAWLER_SSRF_INTERCEPT=true
# Path to a system Chromium binary. When set, the crawler skips the
# bundled-fetcher download. Required on platforms without a usable
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On
@@ -141,6 +238,11 @@ CRAWLER_PROXY=socks5h://tor:9050
CRAWLER_TOR_CONTROL_URL=tcp://tor:9051
# Max NEWNYM-and-retry cycles per recircuit-eligible failure. Default 3.
CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS=3
# Path to a tor cookie-auth file. Alternative to TOR_CONTROL_PASSWORD —
# the TorController prefers cookie when both are present. Useful when
# running against an externally managed tor daemon that uses
# `CookieAuthentication 1` instead of HashedControlPassword. Default unset.
CRAWLER_TOR_CONTROL_COOKIE_PATH=
# ----- TOR control-port password -----
# Shared between the bundled dockurr/tor service (which hashes it into
@@ -182,6 +284,25 @@ BACKEND_PROXY_TIMEOUT_MS=300000
# analysis ANALYSIS_API_KEY. The important site levers (PRIVATE_MODE,
# ALLOW_SELF_REGISTER) also remain env-only by design.
#
# Crawler boot seeds (effective only on first boot, then editable from the
# dashboard):
# CRAWLER_START_URL The catalog listing URL the crawler walks. Without
# it, the cron is disabled (no daemon-driven sweeps;
# force-resync from the dashboard still works).
# CRAWLER_CDN_HOST Distinct host the source serves images from. Used to
# set a slower rate limit (CRAWLER_CDN_RATE_MS) and
# seed the download allowlist.
CRAWLER_START_URL=
CRAWLER_CDN_HOST=
# CRAWLER_DAEMON / CRAWLER_DAILY_AT / CRAWLER_TZ — the daemon sweep
# schedule. CRAWLER_DAEMON=false keeps the in-process daemon off; clicks
# in the dashboard still work. DAILY_AT is `HH:MM` in the chosen TZ;
# TZ is an IANA name (e.g. `Europe/Berlin`). Default: daemon on, runs
# at 00:00 UTC.
CRAWLER_DAEMON=true
CRAWLER_DAILY_AT=00:00
CRAWLER_TZ=UTC
#
# New analysis knobs (all optional, all seed defaults):
# ANALYSIS_TEMPERATURE Sampling temperature. Default 0 (deterministic).
# ANALYSIS_SYSTEM_PROMPT Override the single-call vision system prompt.
@@ -190,6 +311,82 @@ BACKEND_PROXY_TIMEOUT_MS=300000
# Leave the prompt vars unset to use the built-in defaults (also editable,
# with a per-prompt "reset to default", in the dashboard).
# ----- Analysis (vision) — boot seeds + env-only secret -----
# The page-analysis worker (off by default) POSTs each page image at a
# Chat-Completions-compatible vision endpoint and persists OCR + tags.
# All fields except ANALYSIS_API_KEY are boot seeds for the app_settings
# row and switch to dashboard-editable once persisted.
#
# ANALYSIS_ENABLED Turn the worker on at first boot. Toggleable live in
# the dashboard. Default `false`.
ANALYSIS_ENABLED=false
# ANALYSIS_BACKEND Which engine the worker runs. env-ONLY (deploy-time).
# `ocr` (default) = the in-process ocrs engine: fast,
# CPU-only, text-only, ideal for a Pi.
# NOTE: the `vision` backend (local LLM at ANALYSIS_VISION_URL,
# full OCR + tags + scene + safety) is TEMPORARILY DISABLED —
# its code is kept intact but the worker forces OCR regardless,
# logging a warning if `vision` is requested. So setting
# `vision` here currently has no effect; the ANALYSIS_VISION_* /
# ANALYSIS_API_KEY knobs below are dormant until it's re-enabled.
ANALYSIS_BACKEND=ocr
# OCRS_DETECTION_MODEL / OCRS_RECOGNITION_MODEL Paths to the ocrs `.rten`
# text-detection / -recognition models (only read when ANALYSIS_BACKEND=ocr).
# The backend image bakes both into /models, so the defaults work unchanged;
# override only to point at custom-trained models.
OCRS_DETECTION_MODEL=/models/text-detection.rten
OCRS_RECOGNITION_MODEL=/models/text-recognition.rten
# ANALYSIS_VISION_URL /v1/chat/completions endpoint. Required when enabled.
# For the bundled vision container, use
# http://mangalord-vision:8000/v1/chat/completions.
ANALYSIS_VISION_URL=
# ANALYSIS_VISION_MODEL Model id the endpoint expects (`model:` field).
ANALYSIS_VISION_MODEL=
# ANALYSIS_WORKERS Concurrent vision dispatches. Default 1.
ANALYSIS_WORKERS=1
# ANALYSIS_JOB_TIMEOUT_SECS Hard upper bound per vision call before the
# job is acked failed (backoff). Default 600.
ANALYSIS_JOB_TIMEOUT_SECS=600
# ANALYSIS_API_KEY env-ONLY. Bearer-attached to every vision call; never
# persisted, never logged.
ANALYSIS_API_KEY=
# Analysis tuning — boot seeds (live-editable from the dashboard once
# persisted). Defaults are tuned for a 4 GiB-class local vision model;
# raise/lower as the model and budget allow.
# ANALYSIS_MAX_TOKENS Output-token cap per call. Default 4096.
# ANALYSIS_MAX_PIXELS Per-image pixel budget; we resize to fit.
# Default 1_000_000 (~1 MP).
# ANALYSIS_MIN_SLICE_HEIGHT Slice grid height for tall pages, px.
# Default 640.
# ANALYSIS_SLICE_OVERLAP Fractional overlap between adjacent slices
# so text on a cut survives. Default 0.12.
# ANALYSIS_TALL_ASPECT Slice only when `height/width` exceeds this;
# normal pages take a single call. Default 1.6.
# ANALYSIS_MAX_SLICES Hard cap on slices per page (coarsens
# beyond cap). Default 16.
# ANALYSIS_MAX_IMAGE_BYTES Per-image byte cap before downscale.
# Default 8388608 (8 MiB).
# ANALYSIS_OCR_MAX_DECODE_PIXELS Decompression-bomb guard for the OCR
# backend: hard cap on a page's *decoded* pixel
# count (encoded size is bounded separately).
# Default 100000000 (100 MP).
# ANALYSIS_RESPONSE_FORMAT `json_schema` | `json_object` | `none`.
# Default `json_schema`.
# ANALYSIS_FREQUENCY_PENALTY Discourages repetition loops. Default 0.3.
# ANALYSIS_TEMPERATURE Sampling temperature; 0 = deterministic.
# Default 0.
ANALYSIS_MAX_TOKENS=4096
ANALYSIS_MAX_PIXELS=1000000
ANALYSIS_MIN_SLICE_HEIGHT=640
ANALYSIS_SLICE_OVERLAP=0.12
ANALYSIS_TALL_ASPECT=1.6
ANALYSIS_MAX_SLICES=16
ANALYSIS_MAX_IMAGE_BYTES=8388608
ANALYSIS_OCR_MAX_DECODE_PIXELS=100000000
ANALYSIS_RESPONSE_FORMAT=json_schema
ANALYSIS_FREQUENCY_PENALTY=0.3
ANALYSIS_TEMPERATURE=0.0
# ----- Vision autoscaling (the `ai` compose profile) -----
# The mangalord-vision (llama.cpp) container pins ~4.4 GiB and has no
# idle-unload, so the `vision-manager` sidecar starts it on demand and
@@ -230,3 +427,14 @@ VISION_MANAGER_DATABASE_URL=
# VISION_MEM_LOW_WATERMARK_PCT used% >= this → inhibit starts (keep HIGH - LOW >= ~10 so the band beats jitter). Default 80.
# VISION_MEM_YIELD_COOLDOWN Seconds after a pressure-stop during which restart is refused regardless of backlog. Default 300.
# VISION_MEM_POLL_INTERVAL Mem sub-poll cadence, seconds (<= VISION_POLL_INTERVAL); catches spikes between backlog ticks. Default 5.
# ---- Deployment resource bounds & exposure (docker-compose.yml) ----
# Host interface the frontend port publishes on. Default 127.0.0.1 so SvelteKit
# (plain HTTP) is reachable only by a host-local reverse proxy. Set 0.0.0.0 only
# if your TLS terminator runs on a different host.
FRONTEND_PUBLISH_ADDR=127.0.0.1
# Per-container memory ceilings (docker mem_limit). Generous defaults; tune to
# your host. The backend runs OCR + headless Chromium and is the heavy one.
BACKEND_MEM_LIMIT=4g
FRONTEND_MEM_LIMIT=512m
POSTGRES_MEM_LIMIT=1g

4
.gitignore vendored
View File

@@ -8,6 +8,10 @@
/frontend/build
/frontend/test-results
/frontend/playwright-report
# Vite writes these transient files next to vite.config.ts while the dev
# server is running, then deletes them on clean shutdown — but a crashed
# server leaves them behind. Keep them out of `git status` either way.
/frontend/vite.config.ts.timestamp-*.mjs
# Local storage volume (manga files). `/data` is the root path the
# compose volume mounts at; `/backend/data` is where the dev backend

View File

@@ -136,8 +136,8 @@ docker compose -f docker-compose.dev.yml up -d
These are first-class slots in the architecture. When adding any of them, plug into the existing seam rather than building parallel infrastructure.
- **Tags / lists**: new tables joined to `mangas`. New `domain`, `repo`, and `api` modules; the existing manga endpoints do not need to change.
- **Per-page collections / tags**: `collections` is heterogeneous — `collection_mangas` holds whole mangas, `collection_pages` holds individual pages (FK to `pages.id`). Per-user page tags live in `page_tags`, which references the **shared** `tags` table by `tag_id` (migration 0024) — the same lookup table `manga_tags` uses, so manga tags and page tags share one global vocabulary. The HTTP contract still speaks tag *names*; `repo::page_tag` resolves name↔id via `repo::tag::upsert_by_name` and applies the stricter page-tag `normalize_tag` (lowercase, collapse whitespace, reject wildcards/control/invisible chars) at the API layer. Both `collection_pages` and `page_tags` cascade-delete with `pages` and `chapters`, so re-uploading a chapter drops saved-page references by design.
- **Tag-based content search (`/search`)**: the user-facing search surface lives at [frontend/src/routes/search/+page.svelte](frontend/src/routes/search/+page.svelte). Three result views (Pages / Chapters / Mangas) consume the matching `/v1/me/page-tags`, `/v1/me/page-tags/chapters`, and `/v1/me/page-tags/mangas` endpoints. Note the two distinct query-param spaces: `?q=` on `/v1/me/page-tags` is a tag-name prefix (for autocomplete in the "Add tag" sheet); `?text=` on the aggregation endpoints is **reserved** for the planned OCR text-search input. Both aggregation handlers accept `text=` on the wire but reject non-empty values with 501 `text_search_not_yet_supported` so adding OCR later doesn't break the API shape. Adding OCR is then: a background worker writes `page_ocr_text` rows, a JOIN on the existing aggregation queries adds the new filter, the `text=` param starts validating instead of rejecting.
- **Per-page collections / tags**: `collections` is heterogeneous — `collection_mangas` holds whole mangas, `collection_pages` holds individual pages (FK to `pages.id`). Per-user page tags live in `page_tags`, which references the **shared** `tags` table by `tag_id` (migration 0024) — the same lookup table `manga_tags` uses, so manga tags and page tags share one global vocabulary. The HTTP contract still speaks tag *names*; `repo::page_tag` resolves name↔id via `repo::tag::upsert_by_name` and applies the stricter page-tag `normalize_tag` (lowercase, collapse whitespace, reject wildcards/control/invisible chars) at the API layer. Both `collection_pages` and `page_tags` cascade-delete with `pages` and `chapters`, so a saved-page reference is dropped only when its `pages` row is genuinely deleted — i.e. when the chapter is deleted (cascade), or when a user deletes and re-creates a chapter (the user upload path in `api::chapters::finalize_chapter` inserts a *new* chapter row with fresh page ids). A crawler **re-fetch** does **not** drop saves: `content::persist_pages` upserts pages by `(chapter_id, page_number)` (`ON CONFLICT (…) DO UPDATE … RETURNING id`), preserving each `pages.id`, so collections and page tags keyed on that id survive the re-fetch. (Migration 0023's header comment predates this and describes the cascade as an unconditional "re-upload drops saves"; the checked-in migration text is intentionally left as-is because sqlx checksums applied migrations.)
- **Tag-based content search (`/search`)**: the user-facing search surface lives at [frontend/src/routes/search/+page.svelte](frontend/src/routes/search/+page.svelte). Three result views (Pages / Chapters / Mangas) consume the matching `/v1/me/page-tags`, `/v1/me/page-tags/chapters`, and `/v1/me/page-tags/mangas` endpoints. Note the two distinct query-param spaces: `?q=` on `/v1/me/page-tags` is a tag-name prefix (for autocomplete in the "Add tag" sheet); `?text=` on the aggregation endpoints performs **OCR full-text search** the active OCR backend writes `page_ocr_text` rows and a weighted `search_doc` tsvector, and the aggregation queries JOIN on a `plainto_tsquery` filter ranked by `ts_rank` (see [backend/src/repo/page_analysis.rs](backend/src/repo/page_analysis.rs)). (`text=` was previously reserved and returned 501 `text_search_not_yet_supported`; that placeholder is gone now that the OCR backend is active. The generic `AppError::NotImplemented` 501 mechanism remains for future feature reservations.)
- **Full-text / fuzzy search**: enable `pg_trgm` in a migration and add a GIN index on `mangas.title`; swap the `WHERE` in `repo::manga::list` to use `%` operator or `tsvector`. The API shape (`?search=...`) does not change.
- **OCR / autotagging**: a background worker (a separate binary or a tokio task spawned in `app::build`) that reads pages from `storage::Storage` and writes tag rows. Do not couple OCR to upload handlers — it runs asynchronously.
- **S3 storage**: add `storage::S3Storage` implementing `Storage`. Branch in `app::build` based on a config field (e.g., `STORAGE_BACKEND=s3`). Handlers do not change.

136
HYBRID-OCR-VISION.md Normal file
View File

@@ -0,0 +1,136 @@
# Hybrid OCR→Vision analysis pipeline — design brief for the dev agent
**Status: design only (not implemented).** Forward-looking model for a future
`ANALYSIS_BACKEND=hybrid`, on top of the shipped `ocr` (default) and `vision`
backends. Constraint that shapes it: `ocrs` and the vision LLM must **not** be
resident at the same time (Pi RAM ceiling). OCR has priority; vision is a
preemptible, only-when-OCR-is-drained background phase.
## The key realization: this is the crawl mutex, again
The repo already runs a **`vision-manager`** sidecar
([vision-manager/manager.sh](vision-manager/manager.sh),
[VISION-AUTOSCALE.md](VISION-AUTOSCALE.md)) that owns the `mangalord-vision`
(llama.cpp, ~4 GiB) container lifecycle via a scoped docker-socket-proxy — the
backend gets no Docker access. It already:
- **Starts** vision when `pending_analysis()` > 0 (counts `analyze_page` jobs).
- **Idle-stops** it after `STOP_DEBOUNCE` (anti-thrash) once the queue drains.
- **Defers starting** vision while `crawl_running()` > 0 — the **"RAM mutex"**:
the crawler's browser and the LLM can't both fit, so vision waits for crawl.
- **Memory-yield**: SIGTERMs a running vision at a HIGH host-RAM watermark,
blocks starting at a LOW watermark (see [VISION-MEMORY-YIELD.md](VISION-MEMORY-YIELD.md)).
The OCR↔vision relationship is the relationship that already exists between
crawl and vision. OCR is just a third RAM-competing, higher-priority workload
that vision must defer to. Extend the existing arbiter rather than build a new one.
## Job model: split one queue into two
Today there is one job kind, `analyze_page`. Split it:
- **`ocr_page`** — the in-process ocrs pass (cheap, fast, no container).
- **`ground_page`** — the vision grounding pass (needs the LLM container).
**Pipeline (hybrid mode):** upload → enqueue `ocr_page`. The OCR daemon runs
ocrs, writes `page_ocr_text` + `search_doc` (text searchable *immediately*), then
enqueues `ground_page` for that page. The grounding daemon later adds
tags/scene/safety. Each kind gets its own dedup index (mirror migration 0031).
Two daemons, two queues.
## Arbitration: OCR priority + memory exclusivity
Responsibilities split cleanly between the in-backend daemons and the manager:
**Backend — OCR daemon** (already built): leases `ocr_page`, always allowed to
run (ocrs is small/local). This is the priority phase.
**Backend — grounding daemon:** leases `ground_page`, but gates leasing on
**both**:
1. `VisionReadiness` (the existing `/health` gate — vision container up), AND
2. **`ocr_backlog == 0`** (a cheap `count(ocr_page WHERE state IN pending,running)`).
Gate #2 is the whole priority mechanism, in-process, no aborts: the instant OCR
work appears, the grounding daemon **finishes its current page** (the in-flight
lease completes normally) and then **stops leasing** new pages and parks — "it
finishes, then yields." It resumes only when OCR has fully drained.
**vision-manager:** two one-line changes to the existing logic:
1. `pending_analysis()` counts **`ground_page`** (not `ocr_page`) — vision only
starts when there is *grounding* work.
2. Add an **OCR start-block mutex** identical to `RESPECT_CRAWL_MUTEX`: defer
starting vision while `ocr_page` backlog > 0. (`RESPECT_OCR_MUTEX=1`.)
**Why this yields memory exclusivity:** while OCR backlog > 0, the grounding
daemon won't lease → vision goes idle → the manager's idle-debounce stops the
container (and never restarts it under the OCR mutex). Only ocrs (small) is
resident. When OCR drains, grounding resumes, `pending_analysis()` > 0 again, the
manager starts vision. Only the *big* consumer (the LLM container) is ever
mutually exclusive with active OCR — which is the actual RAM constraint.
**The one overlap window:** if OCR work arrives mid-grounding-page, ocrs (small,
in-process) briefly coexists with the one in-flight vision page before the
grounding daemon parks. ocrs's footprint makes this a non-issue in practice. If a
deployment needs *hard* exclusivity even there, the OCR daemon can additionally
wait for `vision_running == false` before its first dispatch. No deadlock: OCR's
wait is transient (one grounding page, bounded by `job_timeout`), while vision's
deferral on OCR backlog is the persistent, priority-respecting side.
**Anti-thrash:** the manager's `STOP_DEBOUNCE` already prevents a stray OCR page
from cold-cycling the 4 GiB model. Chapter uploads arrive as bursts, so OCR
drains a whole batch in one phase before vision resumes — the natural good case.
## Data-model split (the real refactor)
`persist_analysis` currently writes OCR + tags + scene + safety in one
transaction and derives `search_doc` from the OCR rows (+ scene weight C). Two
passes means splitting it, sharing the tsvector builder:
- **`persist_ocr(page, lines)`** — writes `page_ocr_text`, sets
`search_doc` from OCR (A/B/D buckets), marks the page text-searchable. Status
reflects OCR completion.
- **`persist_grounding(page, tags, scene, safety)`** — writes auto-tags,
warnings, scene; **recomputes** `search_doc` to fold in the scene (weight C) on
top of the existing OCR buckets. Records grounding completion (e.g. a
`grounded_at` column or tags-present sentinel).
This keeps text search live after the cheap OCR phase, with semantic search/tags
arriving after the (deferred) grounding phase.
## Reusable pieces
- OCR side: the shipped `OcrsEngine` + `OcrAnalyzeDispatcher`
([backend/src/analysis/ocr.rs](backend/src/analysis/ocr.rs)) — retarget to
`ocr_page`, and on success enqueue `ground_page`.
- Vision side: factor `VisionClient::ground(image, mime, ocr_text)` out of
`analyze()`'s Pass-B block
([backend/src/analysis/vision.rs](backend/src/analysis/vision.rs) ~240261) —
pure refactor, existing tests cover it. The grounding daemon is the existing
analysis daemon retargeted to `ground_page` with the extra `ocr_backlog==0`
lease gate.
- Manager: `pending_analysis()` kind swap + `RESPECT_OCR_MUTEX` block, mirroring
the crawl-mutex branch already at [vision-manager/manager.sh](vision-manager/manager.sh).
## Scope / sequencing
Larger than an inline hybrid dispatcher: job-kind split + dedup migrations,
`persist_*` split + search_doc-builder extraction, pipeline enqueue (OCR→ground),
grounding daemon lease gate, and the manager mutex. No new privileged surface
(reuses the docker-socket-proxy). Suggested order:
1. Split `persist_analysis``persist_ocr` / `persist_grounding` (+ tests).
2. Split job kinds + dedup migrations; retarget the OCR daemon to `ocr_page`.
3. `VisionClient::ground()` refactor + grounding daemon (`ground_page`, gated on
`ocr_backlog==0`).
4. OCR→ground enqueue on OCR success (hybrid mode only).
5. vision-manager: `ground_page` counting + `RESPECT_OCR_MUTEX`.
Pure-`ocr` (shipped default) and `vision` (legacy) backends are unaffected; this
is the `hybrid` backend's runtime model.
## Open decision
Hard exclusivity in the one overlap window (OCR daemon waits for `vision_running
== false` before its first dispatch) — needed only if ocrs's few-hundred-MB
footprint can't coexist with a single in-flight grounding page on the target
Pi's RAM. Default: don't gate (rely on the grounding daemon parking fast).

View File

@@ -81,7 +81,7 @@ Everything is namespaced under `/api/v1/`. `/api/*` outside the version prefix i
| Method | Path | Description |
| ------ | ------------------------------------------------------- | ------------------------------------------ |
| GET | `/api/v1/health` | Liveness probe. |
| GET | `/api/v1/mangas?search=&sort=recent|title&limit=&offset=` | List/search mangas. Trigram fuzzy search. |
| GET | `/api/v1/mangas?search=&sort=created\|updated\|title\|author&order=asc\|desc&limit=&offset=` | List/search mangas. Trigram fuzzy search. `sort` defaults to `updated`; `order` defaults per field (dates `desc`, text `asc`). `sort=recent` is a back-compat alias for `created`. |
| GET | `/api/v1/mangas/{id}` | Single manga. |
| GET | `/api/v1/mangas/{id}/chapters` | Paginated chapter list, ordered by number. |
| GET | `/api/v1/mangas/{id}/chapters/{n}` | Single chapter. |

4
backend/.gitignore vendored
View File

@@ -1,3 +1,7 @@
/target
/.sqlx
.env
# Local OCR models for native dev (downloaded, not source)
models/
*.rten

259
backend/Cargo.lock generated
View File

@@ -214,6 +214,12 @@ version = "1.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
[[package]]
name = "bitflags"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
[[package]]
name = "bitflags"
version = "2.11.1"
@@ -514,6 +520,25 @@ dependencies = [
"cfg-if",
]
[[package]]
name = "crossbeam-deque"
version = "0.8.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51"
dependencies = [
"crossbeam-epoch",
"crossbeam-utils",
]
[[package]]
name = "crossbeam-epoch"
version = "0.9.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-queue"
version = "0.3.12"
@@ -638,7 +663,7 @@ version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1e0e367e4e7da84520dedcac1901e4da967309406d1e51017ae1abfb97adbd38"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"objc2",
]
@@ -772,6 +797,16 @@ version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "flatbuffers"
version = "24.12.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4f1baf0dbf96932ec9a3038d57900329c015b0bfb7b63d904f3bc27e2b02a096"
dependencies = [
"bitflags 1.3.2",
"rustc_version",
]
[[package]]
name = "flate2"
version = "1.1.9"
@@ -1059,6 +1094,12 @@ version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
[[package]]
name = "hermit-abi"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
[[package]]
name = "hex"
version = "0.4.3"
@@ -1448,7 +1489,7 @@ version = "0.1.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e02f3bb43d335493c96bf3fd3a321600bf6bd07ed34bc64118e9293bdffea46c"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"libc",
"plain",
"redox_syscall 0.7.5",
@@ -1517,7 +1558,7 @@ checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4"
[[package]]
name = "mangalord"
version = "0.85.1"
version = "0.128.25"
dependencies = [
"anyhow",
"argon2",
@@ -1537,14 +1578,15 @@ dependencies = [
"infer",
"mime",
"nix 0.29.0",
"ocrs",
"rand 0.8.6",
"reqwest",
"rten",
"scraper",
"serde",
"serde_json",
"sha2",
"sqlx",
"subtle",
"sysinfo",
"tempfile",
"thiserror 1.0.69",
@@ -1669,7 +1711,7 @@ version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"cfg-if",
"cfg_aliases",
"libc",
@@ -1681,7 +1723,7 @@ version = "0.31.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf20d2fde8ff38632c426f1165ed7436270b44f199fc55284c38276f9db47c3d"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"cfg-if",
"cfg_aliases",
"libc",
@@ -1757,6 +1799,16 @@ dependencies = [
"libm",
]
[[package]]
name = "num_cpus"
version = "1.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
dependencies = [
"hermit-abi",
"libc",
]
[[package]]
name = "objc2"
version = "0.6.4"
@@ -1772,7 +1824,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "73ad74d880bb43877038da939b7427bba67e9dd42004a18b809ba7d87cee241c"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"objc2",
"objc2-foundation",
]
@@ -1793,7 +1845,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2a180dd8642fa45cdb7dd721cd4c11b1cadd4929ce112ebd8b9f5803cc79d536"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"dispatch2",
"objc2",
]
@@ -1804,7 +1856,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e022c9d066895efa1345f8e33e584b9f958da2fd4cd116792e15e07e4720a807"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"dispatch2",
"objc2",
"objc2-core-foundation",
@@ -1837,7 +1889,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0cde0dfb48d25d2b4862161a4d5fcc0e3c24367869ad306b0c9ec0073bfed92d"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"objc2",
"objc2-core-foundation",
"objc2-core-graphics",
@@ -1855,7 +1907,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3e0adef53c21f888deb4fa59fc59f7eb17404926ee8a6f59f5df0fd7f9f3272"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"block2",
"libc",
"objc2",
@@ -1868,7 +1920,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "180788110936d59bab6bd83b6060ffdfffb3b922ba1396b312ae795e1de9d81d"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"objc2",
"objc2-core-foundation",
]
@@ -1879,7 +1931,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "96c1358452b371bf9f104e21ec536d37a650eb10f7ee379fff67d2e08d537f1f"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"objc2",
"objc2-core-foundation",
"objc2-foundation",
@@ -1891,7 +1943,7 @@ version = "0.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d87d638e33c06f577498cbcc50491496a3ed4246998a7fbba7ccb98b1e7eab22"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"block2",
"objc2",
"objc2-cloud-kit",
@@ -1916,6 +1968,21 @@ dependencies = [
"objc2-foundation",
]
[[package]]
name = "ocrs"
version = "0.12.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a5379fdd3f11522b5a2ff53017a189463dabf5d0a9c915cb3eb97fabec4ea11c"
dependencies = [
"anyhow",
"rayon",
"rten",
"rten-imageproc",
"rten-tensor",
"thiserror 2.0.18",
"wasm-bindgen",
]
[[package]]
name = "once_cell"
version = "1.21.4"
@@ -2142,7 +2209,7 @@ version = "0.18.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"crc32fast",
"fdeflate",
"flate2",
@@ -2361,13 +2428,33 @@ dependencies = [
"getrandom 0.3.4",
]
[[package]]
name = "rayon"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d"
dependencies = [
"either",
"rayon-core",
]
[[package]]
name = "rayon-core"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91"
dependencies = [
"crossbeam-deque",
"crossbeam-utils",
]
[[package]]
name = "redox_syscall"
version = "0.5.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d"
dependencies = [
"bitflags",
"bitflags 2.11.1",
]
[[package]]
@@ -2376,7 +2463,7 @@ version = "0.7.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4666a1a60d8412eab19d94f6d13dcc9cea0a5ef4fdf6a5db306537413c661b1b"
dependencies = [
"bitflags",
"bitflags 2.11.1",
]
[[package]]
@@ -2496,19 +2583,135 @@ dependencies = [
"zeroize",
]
[[package]]
name = "rten"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "43c230fa4ade87c913f61dbd911b7eb0d49460ceff3f1e4fabc837fac191137c"
dependencies = [
"flatbuffers",
"num_cpus",
"rayon",
"rten-base",
"rten-gemm",
"rten-model-file",
"rten-onnx",
"rten-shape-inference",
"rten-simd",
"rten-tensor",
"rten-vecmath",
"rustc-hash",
"smallvec",
"typeid",
"wasm-bindgen",
]
[[package]]
name = "rten-base"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2738cf8bb4c27f828ac788d01ccf4e367e8e773cfec6851f81851b5211de6a79"
dependencies = [
"rayon",
]
[[package]]
name = "rten-gemm"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "330a81a0ca209fb5ce21bd17efa0bd287d5881c6cebfbff0b21c4294a1a14a9e"
dependencies = [
"rayon",
"rten-base",
"rten-simd",
"rten-tensor",
]
[[package]]
name = "rten-imageproc"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d5f148e7e941fb5727b9046a5fa1b45525543d5105f14b384fd9261df0ee49bc"
dependencies = [
"rten-tensor",
]
[[package]]
name = "rten-model-file"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed2f8d270f07ab1bbfff47250c6039f6caa5da59d6da7d74f66aa48559aa6fea"
dependencies = [
"flatbuffers",
"rten-base",
]
[[package]]
name = "rten-onnx"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "23086eef75bfb55278cb0b45cf9f5a877d466d914914aafebee4ffca9b24d20c"
[[package]]
name = "rten-shape-inference"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e8a913c7ca40e2bfbb2a0cd447cce56b33ab19435f56693271a2ef37cf58984"
dependencies = [
"rten-tensor",
"smallvec",
]
[[package]]
name = "rten-simd"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b19a0032dfcb70dd20960c1c51a37674b237586cbc1ce586f45b46605d108e82"
[[package]]
name = "rten-tensor"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05dc744a270aa32d154f1a3df8e48740ccc1be9dfbcf23295ada66d83aa98de6"
dependencies = [
"rayon",
"rten-base",
"smallvec",
"typeid",
]
[[package]]
name = "rten-vecmath"
version = "0.24.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9574ddebf5671bc08ceb76e2e1638fadc57fdeff318634eab2c29e9a803cff64"
dependencies = [
"rten-base",
"rten-simd",
]
[[package]]
name = "rustc-hash"
version = "2.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe"
[[package]]
name = "rustc_version"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92"
dependencies = [
"semver",
]
[[package]]
name = "rustix"
version = "0.38.44"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"errno",
"libc",
"linux-raw-sys 0.4.15",
@@ -2521,7 +2724,7 @@ version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"errno",
"libc",
"linux-raw-sys 0.12.1",
@@ -2603,7 +2806,7 @@ version = "0.25.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4eb30575f3638fc8f6815f448d50cb1a2e255b0897985c8c59f4d37b72a07b06"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"cssparser",
"derive_more",
"fxhash",
@@ -2911,7 +3114,7 @@ checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526"
dependencies = [
"atoi",
"base64",
"bitflags",
"bitflags 2.11.1",
"byteorder",
"bytes",
"chrono",
@@ -2955,7 +3158,7 @@ checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46"
dependencies = [
"atoi",
"base64",
"bitflags",
"bitflags 2.11.1",
"byteorder",
"chrono",
"crc",
@@ -3317,7 +3520,7 @@ version = "0.6.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "68d6fdd9f81c2819c9a8b0e0cd91660e7746a8e6ea2ba7c6b2b057985f6bcb51"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"bytes",
"futures-util",
"http",
@@ -3428,6 +3631,12 @@ dependencies = [
"utf-8",
]
[[package]]
name = "typeid"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc7d623258602320d5c55d1bc22793b57daff0ec7efc270ea7d55ce1d5f5471c"
[[package]]
name = "typenum"
version = "1.20.0"
@@ -3668,7 +3877,7 @@ version = "0.244.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe"
dependencies = [
"bitflags",
"bitflags 2.11.1",
"hashbrown 0.15.5",
"indexmap",
"semver",
@@ -4096,7 +4305,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2"
dependencies = [
"anyhow",
"bitflags",
"bitflags 2.11.1",
"indexmap",
"log",
"serde",

View File

@@ -1,6 +1,6 @@
[package]
name = "mangalord"
version = "0.85.1"
version = "0.128.25"
edition = "2021"
default-run = "mangalord"
@@ -35,7 +35,6 @@ dotenvy = "0.15"
argon2 = "0.5"
rand = "0.8"
sha2 = "0.10"
subtle = "2"
base64 = "0.22"
# Image decode + downscale for the analysis worker (keep the page image
# under the local vision model's token budget). Only the manga page formats.
@@ -52,6 +51,8 @@ sysinfo = { version = "0.32", default-features = false, features = ["system", "c
nix = { version = "0.29", features = ["fs"] }
scraper = "0.20"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "socks", "cookies", "stream", "json"] }
ocrs = "0.12"
rten = "0.24"
[dev-dependencies]
tempfile = "3"
@@ -60,6 +61,7 @@ http-body-util = "0.1"
mime = "0.3"
futures-util = "0.3"
tokio = { version = "1", features = ["test-util"] }
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp"] }
# Trim debug builds: keep line numbers in panics / backtraces but drop the
# full DWARF info (variable-level inspection in gdb/lldb). With a sqlx +

View File

@@ -58,6 +58,20 @@ WORKDIR /app
COPY --from=builder /app/target/release/mangalord /usr/local/bin/mangalord
COPY --from=builder /app/migrations /app/migrations
# OCR models for the default `ANALYSIS_BACKEND=ocr` (ocrs) engine. The two
# `.rten` files are pulled at build time into /models, where the runtime's
# `OCRS_DETECTION_MODEL` / `OCRS_RECOGNITION_MODEL` defaults point. They're a
# few MB each and pure data (no native code), so they bake cleanly into the
# image and need no manual setup on the Pi. Set INSTALL_OCR_MODELS=false to
# skip (e.g. for a vision-only deploy that never runs ocrs).
ARG INSTALL_OCR_MODELS=true
ARG OCRS_MODEL_BASE_URL=https://ocrs-models.s3-accelerate.amazonaws.com
RUN if [ "$INSTALL_OCR_MODELS" = "true" ]; then \
mkdir -p /models \
&& curl -fsSL "${OCRS_MODEL_BASE_URL}/text-detection.rten" -o /models/text-detection.rten \
&& curl -fsSL "${OCRS_MODEL_BASE_URL}/text-recognition.rten" -o /models/text-recognition.rten; \
fi
ENV STORAGE_DIR=/var/lib/mangalord/storage
# Pre-create the storage dir so the entrypoint doesn't need to
# mkdir-as-root and so the named volume mount inherits the right

View File

@@ -0,0 +1,8 @@
-- Index for the Analysis trend-chart series, which buckets terminal page
-- analyses by `analyzed_at` (date_trunc) over a recent window. Without it the
-- bucketed GROUP BY falls back to a full scan of page_analysis (one row per
-- page in the library). Partial to the rows the series query actually reads
-- (terminal status with a timestamp), keeping the index small.
CREATE INDEX IF NOT EXISTS page_analysis_analyzed_at_idx
ON page_analysis (analyzed_at)
WHERE status <> 'pending' AND analyzed_at IS NOT NULL;

View File

@@ -0,0 +1,50 @@
-- Dedup analyze_page jobs in flight (mirrors 0014 for sync_chapter_content).
--
-- Without this, `repo::page_analysis::enqueue_pages`' `NOT EXISTS(... pending
-- | running ...)` read-then-insert race let concurrent admin POSTs land
-- duplicate jobs for the same page — visible as duplicate "now analyzing"
-- SSE events and inflated `pending` counters in the admin dashboard, even
-- though the worker's `skip-if-done` net (analysis/daemon.rs:210-217)
-- prevented the actual duplicate vision call.
--
-- The partial unique index lets enqueue_pages drop the NOT EXISTS subquery
-- and rely on `ON CONFLICT DO NOTHING` instead. At most one (pending|running)
-- job per page_id can exist; the slot frees the moment the job transitions
-- to done/failed/dead, so a force re-analyze still re-enqueues cleanly.
-- **Pre-dedup step.** Production runs of the prior buggy producer can
-- have left duplicate rows tied on (kind='analyze_page', payload->>'page_id',
-- state IN ('pending','running')). Adding the UNIQUE INDEX onto a dirty
-- table would fail to create and crash `sqlx::migrate!` at boot.
--
-- Demote all-but-the-lowest-id duplicates to `dead` first. Why dead and
-- not done: the dropped sibling never actually ran, and `dead` is the
-- terminal state operators inspect via the dead-jobs requeue endpoint —
-- so a curator can recover any genuinely-needed page. The lowest-id row
-- is the one a worker most likely already leased / has heartbeat'd, so
-- keeping it minimises lease-state thrash.
UPDATE crawler_jobs
SET state = 'dead',
last_error = COALESCE(last_error, '') ||
CASE WHEN last_error IS NULL OR last_error = '' THEN ''
ELSE ' | ' END ||
'pre-0031 duplicate analyze_page; superseded by earlier sibling',
updated_at = now()
WHERE id IN (
SELECT id FROM (
SELECT id,
row_number() OVER (
PARTITION BY payload->>'page_id'
ORDER BY id
) AS rn
FROM crawler_jobs
WHERE payload->>'kind' = 'analyze_page'
AND state IN ('pending', 'running')
) ranked
WHERE rn > 1
);
CREATE UNIQUE INDEX crawler_jobs_analyze_page_dedup_idx
ON crawler_jobs ((payload->>'page_id'))
WHERE state IN ('pending', 'running')
AND payload->>'kind' = 'analyze_page';

View File

@@ -0,0 +1,23 @@
-- Per-lease generation token to close the lease-identity race that
-- 0.87.20 only partially addressed.
--
-- Before this, `ack_done` / `ack_failed` / `renew` matched on
-- `(id, state='running')`. A worker whose lease was released mid-
-- flight (force-analyze, reclaim) could still ack-done the row once a
-- successor leased it back to `state='running'` — clobbering the
-- successor's lease with the original's stale result. The bug class is
-- ack-from-a-dead-lease.
--
-- Adding `lease_generation` makes lease identity = `(id, generation)`:
-- * `lease`/`lease_kinds` bump generation by 1 at lease time
-- * `release` (force-analyze, graceful shutdown) bumps generation too
-- * `reclaim_orphaned` bumps generation for every reclaimed row
-- * `ack_done` / `ack_failed` / `renew` match on `(id, generation,
-- state='running')`, so a stale ack from a prior generation finds
-- no row and falls back to the existing warn-and-skip branch.
--
-- BIGINT so the counter can't realistically overflow even for a row
-- that's been crash-leased thousands of times.
ALTER TABLE crawler_jobs
ADD COLUMN lease_generation BIGINT NOT NULL DEFAULT 0;

View File

@@ -0,0 +1,16 @@
-- Back the default catalog ordering. Since 0.88.0 the catalog list defaults to
-- `ORDER BY updated_at DESC, id` (the no-filter default path), but the only
-- existing updated_at index on mangas is the PARTIAL `mangas_missing_cover_idx`
-- (WHERE cover_image_path IS NULL), which Postgres can't use for the unfiltered
-- ordering. `created_at` and `lower(title)` already have full indexes from
-- 0001; this gives the new default sort column the same treatment.
--
-- `id` is included as a trailing key so the index also covers the stable
-- `, id` tie-break the listing appends for deterministic pagination.
--
-- Not CONCURRENTLY: sqlx::migrate! wraps each migration in a transaction, and
-- CREATE INDEX CONCURRENTLY can't run inside one. The table is small at our
-- scale, so a brief lock on an online deploy is acceptable.
CREATE INDEX IF NOT EXISTS mangas_updated_at_idx
ON mangas (updated_at DESC, id);

View File

@@ -0,0 +1,9 @@
-- Optional expiry for bot API tokens. NULL = never expires (the prior
-- behaviour, preserved for all existing rows). When set, `find_active`
-- rejects the token past this instant, mirroring the sessions table's
-- `expires_at > now()` gate.
ALTER TABLE api_tokens ADD COLUMN expires_at TIMESTAMPTZ;
-- Partial index to keep the active-token lookup cheap once expiries exist.
CREATE INDEX api_tokens_expires_at_idx ON api_tokens (expires_at)
WHERE expires_at IS NOT NULL;

View File

@@ -0,0 +1,15 @@
-- Per-user like/dislike reactions on mangas — a private taste signal that
-- powers content-based recommendations. One row per (user, manga); the
-- `reaction` column toggles between 'like' and 'dislike', and clearing a
-- reaction deletes the row. Reactions are never exposed publicly (no counts).
CREATE TABLE manga_reactions (
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
manga_id uuid NOT NULL REFERENCES mangas(id) ON DELETE CASCADE,
reaction text NOT NULL CHECK (reaction IN ('like', 'dislike')),
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, manga_id)
);
-- Recommendations aggregate a user's reacted mangas by tag; the PK covers
-- per-user lookups, this covers the reverse (all reactions on a manga).
CREATE INDEX manga_reactions_manga_idx ON manga_reactions (manga_id);

View File

@@ -0,0 +1,103 @@
-- Denormalized manga -> content-warning set.
--
-- The list filter previously tested each candidate manga with a correlated
-- `page_content_warnings -> pages -> chapters` join, i.e. O(mangas * pages) on
-- every filtered list AND its count. This table holds the DISTINCT union of a
-- manga's page warnings so the filter is a single indexed lookup.
--
-- Kept in sync by triggers that recompute an affected manga's set from current
-- data (the set is tiny — at most the five moderation labels — so a
-- delete-and-reinsert per change is cheap and always correct, sidestepping the
-- cascade-ordering hazards of incremental maintenance).
CREATE TABLE manga_content_warnings (
manga_id uuid NOT NULL REFERENCES mangas(id) ON DELETE CASCADE,
warning text NOT NULL
CHECK (warning IN ('sexual', 'nudity', 'gore', 'violence', 'disturbing')),
PRIMARY KEY (manga_id, warning)
);
-- warning -> mangas, for the include/exclude list filters.
CREATE INDEX manga_content_warnings_warning_idx ON manga_content_warnings (warning);
-- Recompute a single manga's warning set from the live per-page rows.
CREATE OR REPLACE FUNCTION mcw_refresh_for_manga(mid uuid) RETURNS void AS $$
BEGIN
-- Skip when the manga is gone (e.g. mid-cascade of a manga delete) so we
-- never re-insert a row that would violate the FK / resurrect a deleted set.
IF NOT EXISTS (SELECT 1 FROM mangas WHERE id = mid) THEN
RETURN;
END IF;
DELETE FROM manga_content_warnings WHERE manga_id = mid;
INSERT INTO manga_content_warnings (manga_id, warning)
SELECT DISTINCT mid, pw.warning
FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mid;
END;
$$ LANGUAGE plpgsql;
-- page_content_warnings changed for a page: refresh that page's manga.
CREATE OR REPLACE FUNCTION mcw_on_pcw_change() RETURNS trigger AS $$
DECLARE
mid uuid;
pid uuid := COALESCE(NEW.page_id, OLD.page_id);
BEGIN
-- The page (and thus chapter) may already be gone when this fires as part
-- of a pages/chapters cascade; in that case the pages/chapters triggers do
-- the refresh instead, so a missing join here is harmless.
SELECT c.manga_id INTO mid
FROM pages p JOIN chapters c ON c.id = p.chapter_id
WHERE p.id = pid;
IF mid IS NOT NULL THEN
PERFORM mcw_refresh_for_manga(mid);
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_pcw_ins AFTER INSERT ON page_content_warnings
FOR EACH ROW EXECUTE FUNCTION mcw_on_pcw_change();
CREATE TRIGGER mcw_pcw_del AFTER DELETE ON page_content_warnings
FOR EACH ROW EXECUTE FUNCTION mcw_on_pcw_change();
-- A page was deleted (directly, or via a chapter cascade): its
-- page_content_warnings rows are already cascade-gone, so recompute from the
-- chapter's manga. If the chapter is gone too, the chapters trigger covers it.
CREATE OR REPLACE FUNCTION mcw_on_page_delete() RETURNS trigger AS $$
DECLARE
mid uuid;
BEGIN
SELECT c.manga_id INTO mid FROM chapters c WHERE c.id = OLD.chapter_id;
IF mid IS NOT NULL THEN
PERFORM mcw_refresh_for_manga(mid);
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_page_del AFTER DELETE ON pages
FOR EACH ROW EXECUTE FUNCTION mcw_on_page_delete();
-- A chapter was deleted (directly, or via a manga cascade): recompute from the
-- chapter's manga. `chapters.manga_id` is on the row, so it is always available
-- even after the child pages have cascaded; the refresh no-ops when the manga
-- itself is being deleted.
CREATE OR REPLACE FUNCTION mcw_on_chapter_delete() RETURNS trigger AS $$
BEGIN
PERFORM mcw_refresh_for_manga(OLD.manga_id);
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_chapter_del AFTER DELETE ON chapters
FOR EACH ROW EXECUTE FUNCTION mcw_on_chapter_delete();
-- Backfill from existing per-page rows.
INSERT INTO manga_content_warnings (manga_id, warning)
SELECT DISTINCT c.manga_id, pw.warning
FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
ON CONFLICT DO NOTHING;

View File

@@ -0,0 +1,51 @@
-- Precomputed author sort key for `?sort=author`.
--
-- The author sort used a correlated `min(lower(a.name))` subquery as the ORDER
-- BY key, evaluated per filter-matching row before LIMIT — it scaled worse than
-- the indexed date/title sorts. Materialize the same value on `mangas` so the
-- sort is a plain indexed column read.
--
-- `sort_author` = the alphabetically-first attached author's lowercased name,
-- or NULL when the manga has no authors (kept last via NULLS LAST in the query).
-- Author names are immutable (authors are upserted by unique lowercased name and
-- never renamed), so the value only changes when the manga_authors join changes
-- — maintained by the triggers below.
ALTER TABLE mangas ADD COLUMN sort_author text;
CREATE INDEX mangas_sort_author_idx ON mangas (sort_author, id);
-- Backfill from existing links.
UPDATE mangas m
SET sort_author = (
SELECT min(lower(a.name))
FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = m.id
);
CREATE OR REPLACE FUNCTION refresh_manga_sort_author(mid uuid) RETURNS void AS $$
BEGIN
UPDATE mangas
SET sort_author = (
SELECT min(lower(a.name))
FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = mid
)
WHERE id = mid;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION mangas_sort_author_on_ma_change() RETURNS trigger AS $$
BEGIN
-- On a manga cascade-delete the UPDATE simply no-ops (row already gone).
PERFORM refresh_manga_sort_author(COALESCE(NEW.manga_id, OLD.manga_id));
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER manga_authors_sort_author_ins AFTER INSERT ON manga_authors
FOR EACH ROW EXECUTE FUNCTION mangas_sort_author_on_ma_change();
CREATE TRIGGER manga_authors_sort_author_del AFTER DELETE ON manga_authors
FOR EACH ROW EXECUTE FUNCTION mangas_sort_author_on_ma_change();

View File

@@ -0,0 +1,44 @@
-- Enforce case-insensitive genre uniqueness, matching `authors` and `tags`
-- (both got `UNIQUE (lower(name))` in 0009). `genres` only had a case-SENSITIVE
-- `name UNIQUE`, but the crawler's `sync_genres` treats genres as
-- case-insensitive (`WHERE lower(name) = lower($1)`) and can INSERT a new row
-- from a source string. Under a race (or across ticks) two different-cased
-- strings — "Isekai" vs "isekai" — could both miss the pre-check and both
-- insert, since the exact-name unique doesn't collide. That leaves duplicate
-- genre rows for one logical genre.
-- Heal any pre-existing case-variant duplicates before adding the index, so
-- `sqlx::migrate!` can't crash on a dirty table (mirrors 0031's pre-dedup).
-- Canonical = lowest id per lower(name).
-- 1. Ensure every manga linked to any case-variant also has the canonical link.
-- INSERT ... ON CONFLICT DO NOTHING is collision-proof: SELECT DISTINCT
-- collapses multiple variants of one manga to a single canonical row, and the
-- ON CONFLICT absorbs a canonical link that already exists. (A prior
-- UPDATE-repoint could set two non-canonical rows of the SAME manga to the
-- same canonical id in a single statement and violate the manga_genres PK,
-- rolling the migration back and wedging startup.)
INSERT INTO manga_genres (manga_id, genre_id)
SELECT DISTINCT mg.manga_id, k.keep_id
FROM manga_genres mg
JOIN genres g ON mg.genre_id = g.id
JOIN (SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
ON lower(g.name) = k.lname
WHERE g.id <> k.keep_id
ON CONFLICT (manga_id, genre_id) DO NOTHING;
-- 2. Every non-canonical link now has a canonical sibling — drop the variants.
DELETE FROM manga_genres mg
USING genres g,
(SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
WHERE mg.genre_id = g.id AND lower(g.name) = k.lname AND g.id <> k.keep_id;
-- Remove the now-orphaned duplicate genre rows.
DELETE FROM genres g
USING (SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
WHERE lower(g.name) = k.lname AND g.id <> k.keep_id;
CREATE UNIQUE INDEX genres_name_lower_uniq ON genres (lower(name));

View File

@@ -0,0 +1,18 @@
-- Index the crashed-worker-recovery arm of the job lease predicate.
--
-- lease/lease_kinds match:
-- WHERE (state = 'pending' OR (state = 'running' AND leased_until < now()))
-- crawler_jobs_ready_idx (0016) is `ON (scheduled_at) WHERE state = 'pending'`,
-- so it covers only the pending arm. The `state = 'running' AND leased_until`
-- arm had no usable index, so Postgres could not BitmapOr the two arms and
-- degraded to a sequential scan of the ENTIRE crawler_jobs table — including all
-- done/dead rows not yet reaped — on every lease poll, by every worker,
-- continuously (audit H2, the headline performance finding).
--
-- A partial index on leased_until over just the running rows makes the recovery
-- arm index-backed. `state = 'running'` is an immutable predicate (now() stays
-- in the query, not the index). The running set is tiny (in-flight jobs only),
-- so the index is cheap to maintain.
CREATE INDEX crawler_jobs_running_lease_idx
ON crawler_jobs (leased_until)
WHERE state = 'running';

View File

@@ -0,0 +1,9 @@
-- Index the retention reaper's scan. reap_terminal deletes
-- WHERE state IN ('done','dead') AND updated_at < now() - interval
-- which had no supporting index, so each daily sweep sequential-scanned the whole
-- crawler_jobs table to find expired terminal rows (audit M5). A partial index on
-- updated_at over just the terminal states makes the batched reaper's per-batch
-- SELECT index-backed.
CREATE INDEX crawler_jobs_terminal_reap_idx
ON crawler_jobs (updated_at)
WHERE state IN ('done', 'dead');

View File

@@ -37,6 +37,22 @@ const LEASE_HEARTBEAT: Duration = Duration::from_secs(20);
/// long enough not to hammer `/health` while vision is down.
const READINESS_POLL: Duration = Duration::from_secs(2);
/// Longest an idle analysis worker waits between lease polls, bounding the
/// exponential [`idle_backoff`].
const IDLE_BACKOFF_CAP: Duration = Duration::from_secs(30);
/// Exponential idle backoff (1s, 2s, 4s … capped at [`IDLE_BACKOFF_CAP`]).
/// `consecutive_empty` is the number of empty lease polls so far (0 on the first
/// miss); reset to 0 the moment a job is leased. Replaces the old flat 1s sleep
/// so an idle worker isn't firing a `SELECT … FOR UPDATE SKIP LOCKED` lease
/// query every second (audit: analysis daemon idle poll). Mirrors the crawler
/// daemon's backoff.
fn idle_backoff(consecutive_empty: u32) -> Duration {
let cap = IDLE_BACKOFF_CAP.as_secs();
let secs = 1u64.checked_shl(consecutive_empty).unwrap_or(cap).min(cap);
Duration::from_secs(secs)
}
/// The unit of work: analyze one page. Implemented by
/// [`RealAnalyzeDispatcher`] in production and stubbed in tests.
#[async_trait]
@@ -135,6 +151,9 @@ impl WorkerContext {
// Last observed readiness, so we log only on transitions (not every
// poll). `None` until the first probe.
let mut was_ready: Option<bool> = None;
// Empty lease polls seen in a row, driving the idle backoff. Reset to 0
// the moment a job is leased.
let mut consecutive_empty: u32 = 0;
loop {
if self.cancel.is_cancelled() {
tracing::info!(worker = self.id, "analysis worker: shutdown");
@@ -178,11 +197,15 @@ impl WorkerContext {
}
};
let Some(lease) = leases.into_iter().next() else {
if self.sleep_or_cancel(Duration::from_secs(1)).await {
let wait = idle_backoff(consecutive_empty);
consecutive_empty = consecutive_empty.saturating_add(1);
if self.sleep_or_cancel(wait).await {
return;
}
continue;
};
// Leased work — drop back to a tight poll cadence.
consecutive_empty = 0;
self.process_lease(lease).await;
}
}
@@ -200,7 +223,7 @@ impl WorkerContext {
// Shouldn't happen — we lease only analyze_page — but ack done
// so a misrouted job doesn't loop forever.
tracing::warn!(worker = self.id, "analysis worker: non-analyze payload leased");
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
return;
};
@@ -210,7 +233,7 @@ impl WorkerContext {
if !force {
if let Ok(Some(row)) = repo::page_analysis::load(&self.pool, page_id).await {
if row.status == crate::domain::page_analysis::AnalysisStatus::Done {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
return;
}
}
@@ -241,10 +264,11 @@ impl WorkerContext {
let heartbeat = {
let hb_pool = self.pool.clone();
let hb_id = lease.id;
let hb_gen = lease.lease_generation;
tokio::spawn(async move {
loop {
tokio::time::sleep(LEASE_HEARTBEAT).await;
match jobs::renew(&hb_pool, hb_id, LEASE_DURATION).await {
match jobs::renew(&hb_pool, hb_id, hb_gen, LEASE_DURATION).await {
Ok(true) => {}
Ok(false) => break,
Err(e) => tracing::warn!(lease_id = %hb_id, ?e, "heartbeat renew failed"),
@@ -255,9 +279,46 @@ impl WorkerContext {
let started = std::time::Instant::now();
let dispatch = AssertUnwindSafe(self.dispatcher.dispatch(page_id)).catch_unwind();
let outcome = tokio::time::timeout(self.job_timeout, dispatch).await;
// Race the vision call against shutdown. Without this, shutdown
// (SIGTERM / `Supervisors::reload_analysis` toggling analysis off)
// blocks for up to `job_timeout` (default 600s) on whichever call
// happens to be in flight. On cancel we drop the dispatch future
// and `jobs::release` the lease — next tick picks it up without
// burning an attempt.
let cancel_result = tokio::select! {
biased;
_ = self.cancel.cancelled() => None,
o = tokio::time::timeout(self.job_timeout, dispatch) => Some(o),
};
let elapsed_ms = started.elapsed().as_millis() as i64;
heartbeat.abort();
let Some(outcome) = cancel_result else {
tracing::info!(
worker = self.id,
lease_id = %lease.id,
"analysis worker: cancelled mid-dispatch — releasing lease"
);
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
// Don't publish Failed (the page didn't actually fail) and
// don't write a duration row — neither outcome is true. DO
// publish Cancelled so the dashboard's "now analyzing" banner
// can clear the page we previously sent Started for. Without
// this the banner stuck on the cancelled page until a later
// page kicked off and overwrote it.
if let Some((manga_id, manga_title, chapter_id, chapter_number, page_number)) =
breadcrumb
{
self.events.publish(AnalysisEvent::Cancelled {
page_id,
manga_id,
manga_title,
chapter_id,
chapter_number,
page_number,
});
}
return;
};
// Flatten timeout / panic / dispatch error into one Result + message.
let result: Result<(), String> = match outcome {
@@ -292,9 +353,20 @@ impl WorkerContext {
self.events.publish(event);
}
// Stamp how long the vision dispatch took — but ONLY when a row
// was actually written this attempt. Otherwise a non-terminal
// failure on a force-re-analyze (page already has a `done` row)
// would overwrite the prior duration with the duration of a
// failed retry that wrote nothing new.
let wrote_row = match &result {
Ok(()) => true,
Err(_) if lease.attempts >= lease.max_attempts => true, // mark_failed wrote below
Err(_) => false,
};
match result {
Ok(()) => {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
}
Err(msg) => {
tracing::warn!(
@@ -309,6 +381,7 @@ impl WorkerContext {
&msg,
lease.attempts,
lease.max_attempts,
lease.lease_generation,
)
.await;
// Terminal failure (retries exhausted) → record a `failed`
@@ -320,10 +393,9 @@ impl WorkerContext {
}
}
// Stamp how long the vision dispatch took. Best-effort and ordered
// after the row is written (persist on success / mark_failed on
// terminal failure); a transient failure with no row updates nothing.
let _ = repo::page_analysis::record_duration(&self.pool, page_id, elapsed_ms).await;
if wrote_row {
let _ = repo::page_analysis::record_duration(&self.pool, page_id, elapsed_ms).await;
}
}
}
@@ -344,19 +416,20 @@ impl AnalyzeDispatcher for RealAnalyzeDispatcher {
// Page was deleted between enqueue and dispatch — nothing to do.
return Ok(());
};
let bytes = self
// Stream through the byte cap so an oversized blob is rejected as it's
// read rather than after it's fully buffered into memory (mirrors the
// OCR dispatcher).
let file = self
.storage
.get(&page.storage_key)
.get_stream(&page.storage_key)
.await
.map_err(|e| anyhow::anyhow!("read page image {}: {e}", page.storage_key))?;
if bytes.len() > self.max_image_bytes {
anyhow::bail!(
"page image {} is {} bytes, over the {} cap",
page.storage_key,
bytes.len(),
self.max_image_bytes
);
}
let bytes =
crate::crawler::safety::accumulate_capped(file.stream, self.max_image_bytes)
.await
.map_err(|e| {
anyhow::anyhow!("page image {} over the byte cap: {e}", page.storage_key)
})?;
let analysis = self.vision.analyze(&bytes, &page.content_type).await?;
repo::page_analysis::persist_analysis(&self.db, page_id, &analysis, &self.model).await?;
Ok(())
@@ -413,6 +486,32 @@ pub mod test_support {
}
}
/// A dispatcher that sleeps for `delay` before returning `Ok`. Lets a
/// test fire shutdown WHILE a dispatch is in flight, to exercise the
/// cancellation race in `process_lease`.
pub struct SlowDispatcher {
pub calls: AtomicUsize,
pub delay: Duration,
}
impl SlowDispatcher {
pub fn new(delay: Duration) -> Arc<Self> {
Arc::new(Self { calls: AtomicUsize::new(0), delay })
}
pub fn call_count(&self) -> usize {
self.calls.load(Ordering::Acquire)
}
}
#[async_trait]
impl AnalyzeDispatcher for SlowDispatcher {
async fn dispatch(&self, _page_id: Uuid) -> anyhow::Result<()> {
self.calls.fetch_add(1, Ordering::AcqRel);
tokio::time::sleep(self.delay).await;
Ok(())
}
}
#[async_trait]
impl AnalyzeDispatcher for CountingDispatcher {
async fn dispatch(&self, _page_id: Uuid) -> anyhow::Result<()> {
@@ -434,6 +533,17 @@ mod tests {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
#[test]
fn idle_backoff_grows_exponentially_and_caps() {
assert_eq!(idle_backoff(0), Duration::from_secs(1));
assert_eq!(idle_backoff(1), Duration::from_secs(2));
assert_eq!(idle_backoff(2), Duration::from_secs(4));
assert_eq!(idle_backoff(4), Duration::from_secs(16));
// Caps at IDLE_BACKOFF_CAP (30s) and never overflows on large counts.
assert_eq!(idle_backoff(5), IDLE_BACKOFF_CAP);
assert_eq!(idle_backoff(100), IDLE_BACKOFF_CAP);
}
/// Bind an ephemeral port that answers every request with `status_line`
/// (e.g. `"200 OK"`), and return its `/health` URL. The probe under test
/// only inspects the status code, so a zero-length body is enough.

View File

@@ -54,6 +54,21 @@ pub enum AnalysisEvent {
chapter_number: i32,
page_number: i32,
},
/// The worker cancelled an in-flight dispatch (daemon shutdown or
/// settings reload toggling analysis off). The lease was released,
/// not failed. The dashboard's "now analyzing" banner clears on
/// `Completed`/`Failed`/`Cancelled`; before this variant existed it
/// listened only to the first two, so a cancel-mid-dispatch left
/// the banner stuck on the cancelled page until a later one
/// kicked off and overwrote it.
Cancelled {
page_id: Uuid,
manga_id: Uuid,
manga_title: String,
chapter_id: Uuid,
chapter_number: i32,
page_number: i32,
},
}
/// Broadcaster shared on `AppState`. Cheap to clone via `Arc`. Publishing

View File

@@ -9,5 +9,6 @@
pub mod daemon;
pub mod events;
pub mod ocr;
pub mod prompt;
pub mod vision;

388
backend/src/analysis/ocr.rs Normal file
View File

@@ -0,0 +1,388 @@
//! The in-process OCR analysis backend (the `ocrs` engine).
//!
//! A lightweight alternative to [`crate::analysis::vision`]: instead of a slow
//! local LLM, each page is run through `ocrs` — a pure-Rust detect→recognize
//! OCR pipeline on the `rten` runtime. It extracts **text only** (no tags,
//! scene description or NSFW flags — those stay the vision backend's job), then
//! reuses [`repo::page_analysis::persist_analysis`] so the OCR lines land in
//! `page_ocr_text` and the weighted `search_doc` tsvector exactly as the vision
//! path produces them. That makes the existing text-search surfaces
//! (`/v1/me/page-search` and the tag aggregations) work with no further wiring.
//!
//! The engine is split behind the [`OcrEngine`] trait so the dispatcher is
//! unit-testable without shipping the (multi-MB) `.rten` model files: tests use
//! [`test_support::StubOcrEngine`], production uses [`OcrsEngine`].
use std::sync::Arc;
use async_trait::async_trait;
use sqlx::PgPool;
use uuid::Uuid;
use crate::analysis::daemon::AnalyzeDispatcher;
use crate::domain::page_analysis::{OcrResult, SafetyFlag, VisionAnalysis};
use crate::repo;
use crate::storage::Storage;
/// The `model` label stamped onto `page_analysis` rows written by this backend.
pub const OCR_MODEL_LABEL: &str = "ocrs";
/// Extracts text lines from a decoded-or-encoded page image. The production
/// impl ([`OcrsEngine`]) decodes the bytes itself; the trait takes the raw
/// stored image bytes so the dispatcher stays engine-agnostic.
pub trait OcrEngine: Send + Sync {
/// Run OCR over one page image (the bytes as stored, e.g. PNG/JPEG/WebP).
/// Returns the recognized text lines in reading order (top→bottom).
fn recognize(&self, image: &[u8]) -> anyhow::Result<Vec<String>>;
}
/// Turn the OCR engine's ordered text lines into the [`VisionAnalysis`] shape
/// that [`repo::page_analysis::persist_analysis`] consumes. OCR-only: the tag,
/// scene and safety fields are left empty/default. The line `kind` is left
/// blank — `persist_analysis` maps an empty kind to the neutral mid-weight
/// `OcrKind::Narration` bucket (classifying speech/sfx/… is the deferred
/// vision backend's job).
pub fn lines_to_analysis(lines: Vec<String>) -> VisionAnalysis {
let ocr_results = lines
.into_iter()
.map(|text| OcrResult { text, kind: String::new(), y: None })
.collect();
VisionAnalysis {
ocr_results,
tagging_results: Vec::new(),
scene_description: String::new(),
safety_flag: SafetyFlag::default(),
}
}
/// Production OCR engine: an `ocrs` detect+recognize pipeline with the two
/// `.rten` models loaded once at startup. Cheap to share across workers — the
/// recognize path borrows `&self`.
pub struct OcrsEngine {
engine: ocrs::OcrEngine,
/// Hard cap on decoded pixel count (decompression-bomb guard). See
/// [`decode_rgb8_within`].
max_decode_pixels: u64,
}
impl OcrsEngine {
/// Load the detection + recognition models from disk and build the engine.
/// Fails (at startup) if either model file is missing or unreadable, so a
/// misconfigured path is a loud boot error rather than a per-page failure.
///
/// `max_decode_pixels` bounds the decoded image size (see
/// [`decode_rgb8_within`]) — wired from `AnalysisConfig::ocr_max_decode_pixels`.
pub fn from_model_paths(
detection: &str,
recognition: &str,
max_decode_pixels: u64,
) -> anyhow::Result<Self> {
use anyhow::Context;
let detection_model = rten::Model::load_file(detection)
.with_context(|| format!("load ocrs detection model {detection}"))?;
let recognition_model = rten::Model::load_file(recognition)
.with_context(|| format!("load ocrs recognition model {recognition}"))?;
let engine = ocrs::OcrEngine::new(ocrs::OcrEngineParams {
detection_model: Some(detection_model),
recognition_model: Some(recognition_model),
..Default::default()
})
.context("construct ocrs engine")?;
Ok(Self { engine, max_decode_pixels })
}
}
/// Decode encoded image bytes to RGB8 while refusing decompression bombs.
///
/// `image::load_from_memory` allocates the full decoded buffer up front, so a
/// tiny file declaring enormous dimensions (e.g. a 50000×50000 PNG) inflates
/// to billions of bytes and OOM-kills the process. We cap the decoder's
/// allocation at `max_decode_pixels` worth of RGBA (4 bytes/px headroom over
/// the RGB8 result), which the `image` crate checks against the header
/// *before* allocating — so an over-size image fails fast instead of dying.
fn decode_rgb8_within(image: &[u8], max_decode_pixels: u64) -> anyhow::Result<image::RgbImage> {
use anyhow::Context;
use std::io::Cursor;
let mut reader = image::ImageReader::new(Cursor::new(image))
.with_guessed_format()
.context("guess page image format for OCR")?;
let mut limits = image::Limits::default();
// 4 bytes/px (RGBA) gives headroom over the eventual RGB8 buffer and any
// single intermediate the decoder allocates per pixel.
limits.max_alloc = Some(max_decode_pixels.saturating_mul(4));
reader.limits(limits);
Ok(reader
.decode()
.context("decode page image for OCR")?
.into_rgb8())
}
impl OcrEngine for OcrsEngine {
fn recognize(&self, image: &[u8]) -> anyhow::Result<Vec<String>> {
// Decode to RGB8 so `ImageSource` gets a known channel layout,
// bounding the decoded size against the decompression-bomb cap.
let rgb = decode_rgb8_within(image, self.max_decode_pixels)?;
let source = ocrs::ImageSource::from_bytes(rgb.as_raw(), rgb.dimensions())
.map_err(|e| anyhow::anyhow!("build OCR image source: {e}"))?;
let input = self.engine.prepare_input(source)?;
// detect words → group into lines → recognize each line. Mirrors
// `OcrEngine::get_text`, but keeps the lines as a Vec instead of
// joining them, so each becomes its own `page_ocr_text` row.
let words = self.engine.detect_words(&input)?;
let line_rects = self.engine.find_text_lines(&input, &words);
let lines = self
.engine
.recognize_text(&input, &line_rects)?
.into_iter()
.filter_map(|line| line.map(|l| l.to_string()))
.map(|s| s.trim().to_string())
.filter(|s| !s.is_empty())
.collect();
Ok(lines)
}
}
/// Bound on concurrent CPU-bound OCR inferences, given the number of analysis
/// workers and the host's available parallelism.
///
/// Each worker's `dispatch` fires one `spawn_blocking` OCR run, so without a
/// bound `ANALYSIS_WORKERS` blocking tasks can run at once. OCR is fully
/// CPU-bound, so running more than there are cores just thrashes the scheduler
/// (and balloons the blocking pool). Cap at the core count, but never below 1
/// and never above the worker count (more permits than workers is pointless).
pub fn ocr_concurrency_limit(workers: usize, cores: usize) -> usize {
workers.min(cores.max(1)).max(1)
}
/// Run a CPU-bound OCR closure on the blocking pool while holding an
/// **owned** permit for the entire duration of the work.
///
/// The permit is acquired with `acquire_owned` and moved *into* the
/// blocking task rather than being held by the caller's future. This
/// matters on cancellation: if the dispatcher future is dropped (graceful
/// shutdown), a borrowed permit would be released the instant the future
/// unwinds — but `spawn_blocking` work is not cancellable and keeps
/// running detached, so the concurrency bound (ANALYSIS_WORKERS) would be
/// briefly exceeded. Moving the permit into the task ties the slot's
/// lifetime to the actual CPU work.
async fn run_ocr_blocking<T, F>(
permits: Arc<tokio::sync::Semaphore>,
f: F,
) -> anyhow::Result<T>
where
F: FnOnce() -> T + Send + 'static,
T: Send + 'static,
{
let permit = permits
.acquire_owned()
.await
.map_err(|e| anyhow::anyhow!("OCR semaphore closed: {e}"))?;
tokio::task::spawn_blocking(move || {
let _permit = permit; // held until f() returns, even if the caller is cancelled
f()
})
.await
.map_err(|e| anyhow::anyhow!("OCR task join error: {e}"))
}
/// Production dispatcher for the OCR backend: load the page, read its image
/// from storage, run OCR on the blocking pool, and persist the lines. Mirrors
/// [`crate::analysis::daemon::RealAnalyzeDispatcher`] but with no network I/O.
pub struct OcrAnalyzeDispatcher {
pub db: PgPool,
pub storage: Arc<dyn Storage>,
pub engine: Arc<dyn OcrEngine>,
pub max_image_bytes: usize,
/// Caps concurrent CPU-bound OCR inferences across all workers (see
/// [`ocr_concurrency_limit`]). Shared via the `Arc<dyn AnalyzeDispatcher>`,
/// so one permit pool covers every worker.
pub ocr_permits: Arc<tokio::sync::Semaphore>,
}
#[async_trait]
impl AnalyzeDispatcher for OcrAnalyzeDispatcher {
async fn dispatch(&self, page_id: Uuid) -> anyhow::Result<()> {
let Some(page) = repo::page::find_by_id(&self.db, page_id).await? else {
// Page was deleted between enqueue and dispatch — nothing to do.
return Ok(());
};
// Stream the blob through the byte cap so the read itself bails once
// the running total exceeds `max_image_bytes` — a plain `get()` would
// buffer the whole (possibly huge) image into memory before the cap
// could reject it.
let file = self
.storage
.get_stream(&page.storage_key)
.await
.map_err(|e| anyhow::anyhow!("read page image {}: {e}", page.storage_key))?;
let bytes =
crate::crawler::safety::accumulate_capped(file.stream, self.max_image_bytes)
.await
.map_err(|e| {
anyhow::anyhow!("page image {} over the byte cap: {e}", page.storage_key)
})?;
// OCR inference is CPU-bound and synchronous — keep it off the async
// worker's runtime thread, and gate it behind the shared permit pool so
// ANALYSIS_WORKERS > cores can't oversubscribe the blocking pool.
let engine = Arc::clone(&self.engine);
let lines =
run_ocr_blocking(Arc::clone(&self.ocr_permits), move || engine.recognize(&bytes))
.await??;
let analysis = lines_to_analysis(lines);
repo::page_analysis::persist_analysis(&self.db, page_id, &analysis, OCR_MODEL_LABEL).await?;
Ok(())
}
}
/// Stubs for the OCR dispatcher's integration tests. Public because the tests
/// live in the `tests/` dir (a separate crate).
pub mod test_support {
use super::*;
/// An [`OcrEngine`] that returns a fixed set of lines regardless of input,
/// so the dispatcher's storage→persist path can be tested without models.
pub struct StubOcrEngine {
pub lines: Vec<String>,
}
impl StubOcrEngine {
pub fn new(lines: &[&str]) -> Arc<Self> {
Arc::new(Self { lines: lines.iter().map(|s| s.to_string()).collect() })
}
}
impl OcrEngine for StubOcrEngine {
fn recognize(&self, _image: &[u8]) -> anyhow::Result<Vec<String>> {
Ok(self.lines.clone())
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn run_ocr_blocking_holds_permit_until_work_completes_despite_cancellation() {
use std::sync::mpsc;
use tokio::sync::{Notify, Semaphore};
let permits = Arc::new(Semaphore::new(1));
let (release_tx, release_rx) = mpsc::channel::<()>();
let started = Arc::new(Notify::new());
// Model the dispatch path: acquire an owned permit and run blocking
// work that we hold open via a channel.
let sem = Arc::clone(&permits);
let started2 = Arc::clone(&started);
let caller = tokio::spawn(async move {
run_ocr_blocking(sem, move || {
// Signal that the blocking task now holds the permit, then
// block until the test releases us.
started2.notify_one();
release_rx.recv().ok();
})
.await
.unwrap();
});
// Wait until the blocking task is running and owns the permit.
started.notified().await;
assert_eq!(permits.available_permits(), 0, "permit taken by blocking work");
// Cancel the caller future (simulates graceful shutdown). The
// blocking task is not cancellable and keeps running detached; with
// an *owned* permit the slot must stay occupied. A borrowed permit
// would have been released here, regressing the concurrency bound.
caller.abort();
let _ = caller.await;
assert_eq!(
permits.available_permits(),
0,
"owned permit must remain held by the still-running blocking task"
);
// Let the blocking work finish; the permit is then returned.
release_tx.send(()).unwrap();
let permit = tokio::time::timeout(std::time::Duration::from_secs(5), permits.acquire())
.await
.expect("permit should be released once blocking work completes")
.unwrap();
drop(permit);
}
#[test]
fn lines_to_analysis_maps_lines_in_order_and_leaves_rest_empty() {
let v = lines_to_analysis(vec!["Hello".to_string(), "world!".to_string()]);
assert_eq!(v.ocr_results.len(), 2);
assert_eq!(v.ocr_results[0].text, "Hello");
assert_eq!(v.ocr_results[1].text, "world!");
// OCR-only: kind blank (→ Narration at persist), no tags/scene/safety.
assert!(v.ocr_results.iter().all(|r| r.kind.is_empty()));
assert!(v.tagging_results.is_empty());
assert_eq!(v.scene_description, "");
assert!(!v.safety_flag.is_nsfw);
assert!(v.safety_flag.content_type.is_empty());
}
#[test]
fn lines_to_analysis_handles_no_text() {
let v = lines_to_analysis(Vec::new());
assert!(v.ocr_results.is_empty());
}
/// A minimal valid PNG of `w`×`h` (single black pixel scaled via IHDR is
/// not valid; instead encode a real tiny image, then we patch the IHDR
/// dimensions for the bomb case). For the happy path we just encode a real
/// small image.
fn encode_png(w: u32, h: u32) -> Vec<u8> {
let img = image::RgbImage::new(w, h);
let mut buf = std::io::Cursor::new(Vec::new());
image::DynamicImage::ImageRgb8(img)
.write_to(&mut buf, image::ImageFormat::Png)
.unwrap();
buf.into_inner()
}
#[test]
fn decode_rgb8_within_accepts_normal_page() {
// A perfectly ordinary page decodes fine under a generous cap.
let png = encode_png(64, 96);
let rgb = decode_rgb8_within(&png, 100_000_000).unwrap();
assert_eq!(rgb.dimensions(), (64, 96));
}
#[test]
fn decode_rgb8_within_rejects_oversize_image() {
// The same image, but the cap is set below its pixel count: the
// allocation limit must trip rather than the decode succeeding. This
// is the decompression-bomb guard in miniature — a real bomb declares
// a huge size in a few bytes; here we shrink the budget instead.
let png = encode_png(2000, 2000); // 4 MP
let err = decode_rgb8_within(&png, 1_000).unwrap_err();
assert!(
err.to_string().contains("decode page image"),
"expected a decode error, got: {err}"
);
}
#[test]
fn ocr_concurrency_limit_caps_at_cores() {
// More workers than cores → clamp to cores (don't oversubscribe).
assert_eq!(ocr_concurrency_limit(8, 4), 4);
}
#[test]
fn ocr_concurrency_limit_caps_at_workers() {
// Fewer workers than cores → only `workers` ever run anyway.
assert_eq!(ocr_concurrency_limit(2, 16), 2);
}
#[test]
fn ocr_concurrency_limit_never_zero() {
// Degenerate inputs still yield at least one permit.
assert_eq!(ocr_concurrency_limit(0, 0), 1);
assert_eq!(ocr_concurrency_limit(1, 0), 1);
}
}

View File

@@ -55,6 +55,116 @@ struct SliceParams {
overlap: f64,
tall_threshold: f64,
max_slices: usize,
/// Hard cap on the **decoded** pixel count, enforced as an allocation
/// limit on the decoder itself. `max_pixels` only downscales *after* a
/// full decode, so without this a tiny WebP/JPEG/PNG declaring huge
/// dimensions would OOM the blocking worker (decompression bomb).
/// Shared with the OCR backend's cap (`ANALYSIS_OCR_MAX_DECODE_PIXELS`).
max_decode_pixels: u64,
}
/// Result of the (blocking-pool) prep pass: a self-contained set of byte
/// payloads the async HTTP loop can dispatch without further CPU work.
/// The enum mirrors [`Plan`] but carries actual JPEGs instead of geometry.
enum PreparedAnalysis {
/// `image::load_from_memory` failed — fall back to sending the raw bytes.
Undecodable,
/// One vision call against `jpeg` (the encoded whole page).
Single { jpeg: Vec<u8> },
/// Per-band OCR calls plus a final whole-image grounding call. Each
/// slice tuple is `(y0, y1, encoded_jpeg)`.
Sliced {
slices: Vec<(u32, u32, Vec<u8>)>,
whole: Vec<u8>,
},
}
/// CPU-only image work: decode → plan → optional width-reduce → slice →
/// JPEG encode. Returns `Ok(Undecodable)` (never `Err`) when the bytes
/// don't parse, so the caller can fall through to the raw-pass-through
/// behaviour.
///
/// **Diagnosability.** The four fallback paths below (decode failure,
/// `render_whole` on `Single`, `render_slice` mid-loop, `render_whole`
/// on `Sliced`) all return `Undecodable`, which the caller treats as
/// "send raw bytes". That's correct semantically — a server can
/// sometimes salvage a broken page — but it also makes a silent
/// JPEG-encoder regression indistinguishable from "the page was just
/// garbage." Emit a `warn` on each fallback so an operator can grep
/// for "vision prep fell through" and tell the two apart.
/// Decode an encoded page image with a hard allocation cap so a
/// decompression bomb — a tiny WebP/JPEG/PNG header declaring enormous
/// dimensions — can't OOM the blocking worker before we get a chance to
/// downscale. The `image` crate's default reader applies no such bound;
/// format-specific self-limits (strongest for PNG) don't cover WebP/JPEG,
/// which manga pages commonly use. `4 bytes/px` (RGBA) leaves headroom
/// over any single intermediate the decoder allocates per pixel. Mirrors
/// `ocr::decode_rgb8_within`.
fn decode_within(image: &[u8], max_decode_pixels: u64) -> Option<DynamicImage> {
use std::io::Cursor;
let mut reader = image::ImageReader::new(Cursor::new(image))
.with_guessed_format()
.ok()?;
let mut limits = image::Limits::default();
limits.max_alloc = Some(max_decode_pixels.saturating_mul(4));
reader.limits(limits);
reader.decode().ok()
}
fn prepare_analysis(image: &[u8], params: SliceParams) -> PreparedAnalysis {
let Some(img) = decode_within(image, params.max_decode_pixels) else {
tracing::warn!(
bytes = image.len(),
"vision prep fell through to Undecodable: decode failed or exceeded pixel cap"
);
return PreparedAnalysis::Undecodable;
};
match plan_slices(img.width(), img.height(), &params) {
Plan::Single { .. } => match render_whole(&img, params.max_pixels) {
Some(jpeg) => PreparedAnalysis::Single { jpeg },
None => {
tracing::warn!(
width = img.width(),
height = img.height(),
"vision prep fell through to Undecodable: render_whole on Single plan failed"
);
PreparedAnalysis::Undecodable
}
},
Plan::Sliced { width, height, bands } => {
let work = if width == img.width() && height == img.height() {
img.clone()
} else {
img.resize_exact(width, height, FilterType::Triangle)
};
let mut slices = Vec::with_capacity(bands.len());
for (y0, y1) in &bands {
let Some(jpeg) = render_slice(&work, *y0, *y1, params.max_pixels) else {
// A failed slice falls back to the undecodable path —
// the server then gets one combined call instead of N
// broken slice calls.
tracing::warn!(
y0 = *y0,
y1 = *y1,
width,
height,
"vision prep fell through to Undecodable: render_slice failed"
);
return PreparedAnalysis::Undecodable;
};
slices.push((*y0, *y1, jpeg));
}
let Some(whole) = render_whole(&img, params.max_pixels) else {
tracing::warn!(
width = img.width(),
height = img.height(),
"vision prep fell through to Undecodable: render_whole on Sliced plan failed"
);
return PreparedAnalysis::Undecodable;
};
PreparedAnalysis::Sliced { slices, whole }
}
}
}
impl VisionClient {
@@ -77,6 +187,7 @@ impl VisionClient {
overlap: cfg.slice_overlap,
tall_threshold: cfg.tall_aspect_threshold,
max_slices: cfg.max_slices,
max_decode_pixels: cfg.ocr_max_decode_pixels,
},
}
}
@@ -84,26 +195,39 @@ impl VisionClient {
/// Analyze one page image. `mime` is the stored content type (used only
/// for the fallback when the image can't be decoded locally).
pub async fn analyze(&self, image: &[u8], mime: &str) -> anyhow::Result<VisionAnalysis> {
// Undecodable locally → send the raw bytes in a single combined call
// and let the server cope (preserves the prior behavior).
let Some(img) = image::load_from_memory(image).ok() else {
let url = format!("data:{mime};base64,{}", b64(image));
let body = build_request_body(
&self.model,
self.max_tokens,
&url,
self.response_format,
self.frequency_penalty,
self.temperature,
&self.system_prompt,
);
return parse_chat_completion(&self.post_chat(body).await?);
// All the CPU-heavy work — JPEG/PNG decode, optional width reduce,
// per-band slice, JPEG re-encode — runs on the blocking pool so it
// doesn't starve the tokio runtime (axum handlers, SSE streams,
// other daemons share the same threads). A single hop pays the
// overhead once per page; the per-call work stays on the blocking
// worker until the HTTP loop below picks back up.
let prepared = {
let bytes = image.to_vec();
let params = self.slice;
tokio::task::spawn_blocking(move || prepare_analysis(&bytes, params))
.await
.map_err(|e| anyhow!("vision prep join: {e}"))?
};
match plan_slices(img.width(), img.height(), &self.slice) {
Plan::Single { .. } => {
let jpeg = render_whole(&img, self.slice.max_pixels)
.ok_or_else(|| anyhow!("failed to encode page image"))?;
match prepared {
PreparedAnalysis::Undecodable => {
// Send raw bytes in a single combined call and let the server
// cope (preserves prior behavior). Base64 encoding still runs
// on the runtime; for an undecodable page this is the
// happy path's exit and stays brief.
let url = format!("data:{mime};base64,{}", b64(image));
let body = build_request_body(
&self.model,
self.max_tokens,
&url,
self.response_format,
self.frequency_penalty,
self.temperature,
&self.system_prompt,
);
parse_chat_completion(&self.post_chat(body).await?)
}
PreparedAnalysis::Single { jpeg } => {
let body = build_request_body(
&self.model,
self.max_tokens,
@@ -115,25 +239,14 @@ impl VisionClient {
);
parse_chat_completion(&self.post_chat(body).await?)
}
Plan::Sliced { width, height, bands } => {
PreparedAnalysis::Sliced { slices, whole } => {
tracing::debug!(
bands = bands.len(),
width,
height,
bands = slices.len(),
"analysis: slicing tall page"
);
// Work image at the (possibly width-reduced) slice space.
let work = if width == img.width() && height == img.height() {
img.clone()
} else {
img.resize_exact(width, height, FilterType::Triangle)
};
// Pass A: OCR each band (keep its y-range for seam dedup).
let mut slices: Vec<SliceOcr> = Vec::with_capacity(bands.len());
for (y0, y1) in &bands {
let jpeg = render_slice(&work, *y0, *y1, self.slice.max_pixels)
.ok_or_else(|| anyhow!("failed to encode page slice"))?;
let mut ocrs: Vec<SliceOcr> = Vec::with_capacity(slices.len());
for (y0, y1, jpeg) in &slices {
let body = build_ocr_body(
&self.model,
self.max_tokens,
@@ -141,20 +254,18 @@ impl VisionClient {
self.frequency_penalty,
self.temperature,
&self.ocr_prompt,
&data_url(&jpeg),
&data_url(jpeg),
);
let parsed = parse_chat_completion(&self.post_chat(body).await?)?;
slices.push(SliceOcr {
ocrs.push(SliceOcr {
y0: *y0 as f64,
y1: *y1 as f64,
pieces: parsed.ocr_results,
});
}
let merged = merge_ocr(slices);
let merged = merge_ocr(ocrs);
// Pass B: ground tags/scene/safety on the whole image + OCR.
let whole = render_whole(&img, self.slice.max_pixels)
.ok_or_else(|| anyhow!("failed to encode page image"))?;
let ocr_text = merged
.iter()
.map(|o| format!("[{}] {}", o.kind, o.text))
@@ -694,9 +805,40 @@ mod tests {
overlap: 0.12,
tall_threshold: 1.6,
max_slices: 16,
max_decode_pixels: 100_000_000,
}
}
fn encode_webp(w: u32, h: u32) -> Vec<u8> {
use image::{DynamicImage, ImageFormat, RgbImage};
let img = DynamicImage::ImageRgb8(RgbImage::new(w, h));
let mut buf = std::io::Cursor::new(Vec::new());
img.write_to(&mut buf, ImageFormat::WebP).expect("encode webp");
buf.into_inner()
}
#[test]
fn decode_within_rejects_oversize_webp() {
// Non-PNG decompression-bomb coverage: a real 2000×2000 WebP (4 MP)
// must be refused when the decode cap is set below its pixel count.
// The allocation limit trips inside the decoder rather than the
// full frame being materialized. Manga pages are commonly WebP/JPEG,
// where the format's own self-limits are weaker than PNG's.
let webp = encode_webp(2000, 2000);
assert!(decode_within(&webp, 1_000).is_none(), "cap below pixels must reject");
// A generous cap decodes the same image fine.
assert!(decode_within(&webp, 100_000_000).is_some(), "within cap must decode");
}
#[test]
fn prepare_analysis_falls_back_to_undecodable_on_decode_bomb() {
// End-to-end: an over-cap image drives the prep pass to the
// Undecodable fallback (raw-bytes path) instead of decoding it.
let webp = encode_webp(2000, 2000);
let p = SliceParams { max_decode_pixels: 1_000, ..params() };
assert!(matches!(prepare_analysis(&webp, p), PreparedAnalysis::Undecodable));
}
fn ocr(text: &str, kind: &str) -> OcrResult {
OcrResult {
text: text.into(),
@@ -1043,4 +1185,177 @@ mod tests {
let dec = image::load_from_memory(&out).unwrap();
assert_eq!((dec.width(), dec.height()), (100, 100));
}
fn small_params() -> SliceParams {
SliceParams {
max_pixels: 1_000_000,
min_slice_height: 100,
overlap: 0.05,
tall_threshold: 1.8,
max_slices: 6,
max_decode_pixels: 100_000_000,
}
}
#[test]
fn prepare_analysis_single_pages_emit_one_jpeg() {
let jpeg = jpeg_of(200, 200);
match prepare_analysis(&jpeg, small_params()) {
PreparedAnalysis::Single { jpeg } => assert!(!jpeg.is_empty()),
other => panic!("expected Single, got {:?}", std::mem::discriminant(&other)),
}
}
#[test]
fn prepare_analysis_tall_pages_emit_slice_and_whole_jpegs() {
// Long enough that height > slice_h_budget * tall_threshold —
// with the test params (max_pixels=1M, width=200, threshold=1.8)
// the threshold is 200×5000×1.8 = 9000 px. A 200×10_000 page
// pushes us into Sliced.
let jpeg = jpeg_of(200, 10_000);
match prepare_analysis(&jpeg, small_params()) {
PreparedAnalysis::Sliced { slices, whole } => {
assert!(slices.len() >= 2, "expected at least 2 bands, got {}", slices.len());
assert!(slices.iter().all(|(_, _, j)| !j.is_empty()));
assert!(!whole.is_empty());
}
other => panic!("expected Sliced, got {:?}", std::mem::discriminant(&other)),
}
}
#[test]
fn prepare_analysis_garbage_bytes_yield_undecodable() {
match prepare_analysis(&[0u8, 1, 2, 3], small_params()) {
PreparedAnalysis::Undecodable => {}
other => panic!("expected Undecodable, got {:?}", std::mem::discriminant(&other)),
}
}
/// Pin the contract that the 0.87.5 fix is about: the heavy image
/// work in `analyze()` MUST go through `spawn_blocking` so the tokio
/// runtime stays responsive. If a refactor accidentally inlined
/// `prepare_analysis` back into the async function, this test would
/// catch it.
///
/// How we detect it: run `analyze()` on a `current_thread` runtime
/// (one worker thread). Schedule a concurrent counter task that
/// ticks every 5ms with `MissedTickBehavior::Skip`. The image is
/// large enough that `prepare_analysis` takes meaningfully longer
/// than the tick interval. If prep is on `spawn_blocking`, the
/// worker thread stays free to drive the counter and we observe
/// several ticks. If prep is inlined onto the runtime, the worker
/// is starved and the counter cannot advance during the prep
/// window.
///
/// **Why `MissedTickBehavior::Skip`** (rereview fix): `Burst` (the
/// default) replays every missed tick as soon as the runtime yields
/// — even a brief reqwest connect-refused yield after inlined prep
/// would burst ~20 ticks at once, faking the `>= 2` threshold and
/// hiding the regression. Skip drops the backlog; the only way to
/// observe `>= 2` ticks is for the runtime to actually have been
/// making progress during the 5ms intervals.
///
/// We also snapshot the tick count at three checkpoints (pre-prep,
/// during, post) so the assertion measures progress DURING the
/// prep window, not just the analyse-call total. This survives
/// the post-prep HTTP fail-fast that the prior version of this
/// test conflated with prep ticks.
#[tokio::test(flavor = "current_thread", start_paused = false)]
async fn analyze_dispatches_image_prep_off_runtime() {
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::Duration;
use tokio::time::MissedTickBehavior;
// VisionClient pointed at a closed local port. Reqwest fails the
// POST immediately (connection refused), but the prep work runs
// before the POST is even built.
let cfg = AnalysisConfig {
endpoint: "http://127.0.0.1:1/v1/chat/completions".into(),
model: "stub".into(),
request_timeout: Duration::from_secs(1),
// Tune slice geometry to match the test image's shape.
max_pixels: 1_000_000,
min_slice_height: 100,
tall_aspect_threshold: 1.8,
..Default::default()
};
let http = reqwest::Client::builder()
.timeout(cfg.request_timeout)
.no_proxy()
.build()
.unwrap();
let client = VisionClient::new(http, &cfg);
// ~6 MP image so decode + JPEG encode comfortably exceeds the
// 5 ms tick interval on every realistic CI runner.
let big = jpeg_of(2000, 3000);
// Counter that ticks at 5ms with Skip semantics so a single
// post-prep yield can't burst-replay enough ticks to fake the
// assertion. If the runtime is starved during prep, this task
// makes no progress in that window.
let ticks = Arc::new(AtomicUsize::new(0));
let ticks_c = Arc::clone(&ticks);
let ticker = tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_millis(5));
interval.set_missed_tick_behavior(MissedTickBehavior::Skip);
interval.tick().await; // discard immediate first tick
loop {
interval.tick().await;
ticks_c.fetch_add(1, Ordering::Relaxed);
}
});
// Snapshot tick count at three points: pre-call, mid-call
// (during prep — if `spawn_blocking` is used, this fires within
// the prep window), and post-call. The mid-call snapshot is
// what distinguishes "runtime was free during prep" from
// "runtime caught up after analyze() returned". On
// current_thread with prep on the runtime, mid-call would tick
// 0 times no matter how long the call takes.
let ticks_pre = ticks.load(Ordering::Relaxed);
let mid_check = {
let ticks_c = Arc::clone(&ticks);
tokio::spawn(async move {
// Sleep just long enough that prep is provably mid-call
// (decode + resize alone exceeds 10 ms on every real
// runner). Snapshot during, return.
tokio::time::sleep(Duration::from_millis(30)).await;
ticks_c.load(Ordering::Relaxed)
})
};
let _ = client.analyze(&big, "image/jpeg").await;
let ticks_during = mid_check.await.unwrap();
let ticks_post = ticks.load(Ordering::Relaxed);
ticker.abort();
let during_window = ticks_during.saturating_sub(ticks_pre);
let total = ticks_post.saturating_sub(ticks_pre);
// Discrimination: the `during_window` snapshot at +30ms is
// dominated by mid_check's own scheduling latency on
// current_thread (the snapshot can't run until the analyze
// call yields), so under either strategy it sits at a similar
// small number. The `total` (post pre) is the unambiguous
// signal — measured over the WHOLE analyze duration plus the
// post-call mid_check wait:
// * spawn_blocking → runtime stays free; counter ticks
// continuously at 5 ms cadence with Skip semantics;
// observed ~700+ on a typical runner.
// * inlined prep → runtime worker is blocked during the
// ~100 ms prep CPU work; Skip drops the entire backlog so
// only the post-prep yield ticks count; observed ~5.
// Threshold of 50 sits well above the inlined-prep ceiling and
// far below the spawn_blocking floor — robust to CPU jitter
// without sacrificing the regression signal.
let _ = ticks_during;
let _ = during_window;
assert!(
total >= 50,
"runtime ticked only {total} times over the analyze() call window \
(mid-call snapshot = {during_window}) — image prep is likely \
blocking the runtime instead of using spawn_blocking"
);
}
}

View File

@@ -44,6 +44,7 @@ pub fn routes() -> Router<AppState> {
.route("/admin/analysis/pages/:id", get(page_detail))
.route("/admin/analysis/history", get(list_history))
.route("/admin/analysis/metrics", get(metrics))
.route("/admin/analysis/metrics/series", get(metrics_series))
.route("/admin/analysis/status/stream", get(stream_status))
}
@@ -57,20 +58,20 @@ async fn stream_status(
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
let rx = state.analysis_events.subscribe();
let stream = futures_util::stream::unfold(rx, |mut rx| async move {
loop {
match rx.recv().await {
Ok(ev) => {
let event = Event::default()
.event("analysis")
.json_data(&ev)
.unwrap_or_else(|_| Event::default().comment("serialize error"));
return Some((Ok(event), rx));
}
Err(RecvError::Lagged(_)) => {
return Some((Ok(Event::default().event("lagged").data("")), rx));
}
Err(RecvError::Closed) => return None,
// One recv per unfold step; the stream driver re-enters for the next
// event, so no explicit loop is needed here.
match rx.recv().await {
Ok(ev) => {
let event = Event::default()
.event("analysis")
.json_data(&ev)
.unwrap_or_else(|_| Event::default().comment("serialize error"));
Some((Ok(event), rx))
}
Err(RecvError::Lagged(_)) => {
Some((Ok(Event::default().event("lagged").data("")), rx))
}
Err(RecvError::Closed) => None,
}
});
Sse::new(stream).keep_alive(KeepAlive::default())
@@ -215,6 +216,46 @@ pub(super) fn window_since(days: i64) -> Option<chrono::DateTime<chrono::Utc>> {
}
}
// --- shared trend-series plumbing (crawler + analysis charts) ---------------
/// Query params for the bucketed trend-series endpoints. `bucket` is
/// optional; when absent it's derived from `days`.
#[derive(Debug, Deserialize, Default)]
pub struct SeriesParams {
#[serde(default)]
pub days: i64,
#[serde(default)]
pub bucket: Option<String>,
}
/// Envelope for a trend series.
#[derive(Debug, Serialize)]
pub struct SeriesResponse<T> {
pub buckets: Vec<T>,
}
/// Resolve the `date_trunc` granularity: an explicit `bucket` (validated to
/// `hour`|`day`, else 400) wins; otherwise short windows bucket by hour and
/// longer ones by day so the point count stays chart-friendly.
pub(super) fn resolve_bucket(
days: i64,
explicit: Option<&str>,
) -> Result<crate::repo::crawl_metrics::Bucket, AppError> {
use crate::repo::crawl_metrics::Bucket;
match explicit {
Some("hour") => Ok(Bucket::Hour),
Some("day") => Ok(Bucket::Day),
Some(other) => Err(AppError::InvalidInput(format!(
"invalid bucket '{other}' (expected 'hour' or 'day')"
))),
None => Ok(if days > 0 && days <= 2 {
Bucket::Hour
} else {
Bucket::Day
}),
}
}
async fn metrics(
State(state): State<AppState>,
_admin: RequireAdmin,
@@ -225,6 +266,17 @@ async fn metrics(
Ok(Json(m))
}
async fn metrics_series(
State(state): State<AppState>,
_admin: RequireAdmin,
Query(params): Query<SeriesParams>,
) -> AppResult<Json<SeriesResponse<crate::domain::crawl_metrics::MetricsBucket>>> {
let bucket = resolve_bucket(params.days, params.bucket.as_deref())?;
let buckets =
repo::page_analysis::analysis_series(&state.db, bucket, window_since(params.days)).await?;
Ok(Json(SeriesResponse { buckets }))
}
#[derive(Debug, Deserialize)]
pub struct ReenqueueBody {
/// Skip pages that already have a `done` analysis row. Defaults to
@@ -276,10 +328,21 @@ fn ensure_enabled(state: &AppState) -> AppResult<()> {
async fn reenqueue(
State(state): State<AppState>,
admin: RequireAdmin,
body: Option<Json<ReenqueueBody>>,
raw: axum::body::Bytes,
) -> AppResult<Json<ReenqueueResponse>> {
ensure_enabled(&state)?;
let body = body.map(|b| b.0).unwrap_or_default();
// An ABSENT/empty body means the default "All" scope. A PRESENT body must be
// valid JSON — `Option<Json<T>>` would silently collapse a malformed body to
// None and run the full-library default the caller never intended, so parse
// the raw bytes ourselves and 422 on a real parse error.
let body: ReenqueueBody = if raw.iter().all(u8::is_ascii_whitespace) {
ReenqueueBody::default()
} else {
serde_json::from_slice(&raw).map_err(|e| AppError::ValidationFailed {
message: "reenqueue body is not valid JSON".into(),
details: json!({ "body": e.to_string() }),
})?
};
// Resolve the scope. manga_id and chapter_id are mutually exclusive;
// an unknown target is a 404 rather than a silent zero-enqueue.
@@ -316,21 +379,15 @@ async fn reenqueue(
(repo::page_analysis::ReenqueueScope::All, "analysis", None)
};
// Enqueue + audit in one transaction so a failed audit insert rolls the
// enqueue back too — the audit trail can't silently miss a re-enqueue that
// actually landed. Both are plain DB writes, so they share the tx cleanly
// (the live event + response are emitted only after commit).
let mut tx = state.db.begin().await?;
let enqueued =
repo::page_analysis::enqueue_pages(&state.db, scope, body.only_unanalyzed).await?;
// Push a live event so connected dashboards mark the in-scope pages as
// queued. Skip the no-op (nothing actually enqueued).
if enqueued > 0 {
state.analysis_events.publish(AnalysisEvent::Enqueued {
count: enqueued,
manga_id: body.manga_id,
chapter_id: body.chapter_id,
});
}
repo::page_analysis::enqueue_pages(&mut *tx, scope, body.only_unanalyzed).await?;
repo::admin_audit::insert(
&state.db,
&mut *tx,
admin.0.id,
"analysis_reenqueue",
target_type,
@@ -343,6 +400,18 @@ async fn reenqueue(
}),
)
.await?;
tx.commit().await?;
// Push a live event so connected dashboards mark the in-scope pages as
// queued. Skip the no-op (nothing actually enqueued). After commit so a
// rolled-back enqueue never emits a phantom event.
if enqueued > 0 {
state.analysis_events.publish(AnalysisEvent::Enqueued {
count: enqueued,
manga_id: body.manga_id,
chapter_id: body.chapter_id,
});
}
Ok(Json(ReenqueueResponse { enqueued }))
}
@@ -363,17 +432,35 @@ async fn analyze_page(
return Err(AppError::NotFound);
}
repo::page_analysis::enqueue_for_page(&state.db, page_id, true).await?;
// Surface the actual outcome so the admin (and audit log) sees
// whether the click actually moved anything. Before 0.87.11 the
// partial unique index could silently swallow a force request
// when a `force=false` job was already pending — the worker would
// then pick up the non-force row, hit skip-if-done, ack done, and
// the admin saw "queued for re-analysis" with no re-analysis.
//
// Enqueue + audit in one transaction (via the `_conn` form) so a failed
// audit insert rolls the enqueue/upgrade back too — the audit trail can't
// miss a force-reanalyze that landed.
let mut tx = state.db.begin().await?;
let outcome =
repo::page_analysis::enqueue_for_page_conn(&mut tx, page_id, true).await?;
let outcome_label = match outcome {
repo::page_analysis::EnqueueForPageOutcome::Inserted => "inserted",
repo::page_analysis::EnqueueForPageOutcome::UpgradedToForce => "upgraded_pending_to_force",
repo::page_analysis::EnqueueForPageOutcome::AlreadyEnqueued => "already_force_enqueued",
};
repo::admin_audit::insert(
&state.db,
&mut *tx,
admin.0.id,
"analysis_force_page",
"page",
Some(page_id),
json!({ "force": true }),
json!({ "force": true, "outcome": outcome_label }),
)
.await?;
tx.commit().await?;
Ok(Json(AnalyzePageResponse { enqueued: true }))
}

View File

@@ -0,0 +1,70 @@
//! GET /admin/audit — paginated, filterable admin-action audit log.
//!
//! A pure DB read over the `admin_audit` table joined to the actor's
//! username. Drives the "Audit" admin tab: who did what, when. Every
//! mutating admin action writes a row here; this is the only reader.
use axum::extract::{Query, State};
use axum::routing::get;
use axum::{Json, Router};
use serde::Deserialize;
use uuid::Uuid;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::error::AppResult;
use crate::repo;
use crate::repo::admin_audit::{AuditEntryRow, AuditFilter};
// Reuse the analysis window helper so `days` clamps identically everywhere.
use crate::api::admin::analysis::window_since;
pub fn routes() -> Router<AppState> {
Router::new().route("/admin/audit", get(list_audit))
}
fn default_limit() -> i64 {
50
}
#[derive(Debug, Deserialize, Default)]
struct AuditParams {
#[serde(default)]
action: Option<String>,
#[serde(default)]
target_kind: Option<String>,
#[serde(default)]
actor_user_id: Option<Uuid>,
#[serde(default)]
days: i64,
#[serde(default = "default_limit")]
limit: i64,
#[serde(default)]
offset: i64,
}
async fn list_audit(
State(state): State<AppState>,
_admin: RequireAdmin,
Query(params): Query<AuditParams>,
) -> AppResult<Json<crate::api::pagination::PagedResponse<AuditEntryRow>>> {
let limit = params.limit.clamp(1, 200);
let offset = params.offset.max(0);
let action = params.action.filter(|s| !s.trim().is_empty());
let target_kind = params.target_kind.filter(|s| !s.trim().is_empty());
let (items, total) = repo::admin_audit::list(
&state.db,
AuditFilter {
action: action.as_deref(),
target_kind: target_kind.as_deref(),
actor_user_id: params.actor_user_id,
since: window_since(params.days),
},
limit,
offset,
)
.await?;
Ok(Json(crate::api::pagination::PagedResponse::with_total(
items, limit, offset, total,
)))
}

View File

@@ -24,6 +24,7 @@ use super::require_crawler;
pub(super) fn routes() -> Router<AppState> {
Router::new()
.route("/admin/crawler/run", post(run_now))
.route("/admin/crawler/reconcile", post(reconcile_now))
.route("/admin/crawler/browser/restart", post(restart_browser))
.route("/admin/crawler/session", post(update_session))
.route(
@@ -71,6 +72,45 @@ async fn run_now(
Ok(Json(RunResponse { started: true }))
}
async fn reconcile_now(
State(state): State<AppState>,
admin: RequireAdmin,
) -> AppResult<Json<RunResponse>> {
let c = require_crawler(&state)?;
let rp = c.reconcile_pass.as_ref().ok_or_else(|| {
AppError::ServiceUnavailable("no source configured (CRAWLER_START_URL unset)".into())
})?;
// Share `manual_pass_lock` with the metadata pass: reconcile's list walk
// and a metadata pass both contend for the single browser lease, so they
// must not run concurrently. A click while either is in flight gets 409.
let pass_guard = c
.manual_pass_lock
.clone()
.try_lock_owned()
.map_err(|_| AppError::Conflict("a metadata or reconcile pass is already running".into()))?;
let rp = std::sync::Arc::clone(rp);
// Fire-and-forget: the full list walk runs for minutes; progress streams
// over SSE (the Reconciling phase). The guard moves into the task so the
// lock releases only when the walk + enqueue finish.
tokio::spawn(async move {
let _pass_guard = pass_guard;
match rp.run().await {
Ok(stats) => tracing::info!(?stats, "manual reconcile pass complete"),
Err(e) => tracing::warn!(error = ?e, "manual reconcile pass failed"),
}
});
repo::admin_audit::insert(
&state.db,
admin.0.id,
"crawler_reconcile",
"crawler",
None,
json!({}),
)
.await?;
Ok(Json(RunResponse { started: true }))
}
#[derive(Debug, Serialize)]
struct RestartResponse {
ok: bool,

View File

@@ -11,20 +11,31 @@ use serde::{Deserialize, Serialize};
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::domain::crawl_metrics::{OpRow, OpSummary};
use crate::domain::crawl_metrics::{MetricsBucket, OpRow, OpSummary};
use crate::error::AppResult;
use crate::repo;
use crate::repo::crawl_metrics::OpFilter;
use super::default_limit;
// Reuse the analysis handler's window helper so both endpoints clamp `days`
// identically.
use crate::api::admin::analysis::window_since;
// Reuse the analysis handler's window + bucket helpers so all the metrics
// endpoints clamp `days` and resolve `bucket` identically.
use crate::api::admin::analysis::{resolve_bucket, window_since, SeriesParams, SeriesResponse};
pub(super) fn routes() -> Router<AppState> {
Router::new()
.route("/admin/crawler/metrics", get(summary))
.route("/admin/crawler/metrics/ops", get(list_ops))
.route("/admin/crawler/metrics/series", get(series))
}
async fn series(
State(state): State<AppState>,
_admin: RequireAdmin,
Query(params): Query<SeriesParams>,
) -> AppResult<Json<SeriesResponse<MetricsBucket>>> {
let bucket = resolve_bucket(params.days, params.bucket.as_deref())?;
let buckets = repo::crawl_metrics::series(&state.db, bucket, window_since(params.days)).await?;
Ok(Json(SeriesResponse { buckets }))
}
#[derive(Debug, Deserialize, Default)]

View File

@@ -0,0 +1,444 @@
//! GET /admin/health — aggregated operational health checks
//! PUT /admin/health/thresholds — edit the alerting thresholds
//!
//! A single read that rolls up signals already collected elsewhere (system
//! sensors, queue depths, recent failure rates, cron freshness, the live
//! crawler session/browser flags) into a flat list of checks with an
//! overall status. Thresholds live in `app_settings['health_thresholds']`
//! (JSONB, merged over compiled defaults) and are admin-editable + audited.
use axum::extract::State;
use axum::routing::{get, put};
use axum::{Json, Router};
use chrono::{Duration, Utc};
use futures_util::TryFutureExt;
use serde::{Deserialize, Serialize};
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::crawler::browser_manager::RestartPhase;
use crate::error::{AppError, AppResult};
use crate::repo;
const KEY: &str = "health_thresholds";
pub fn routes() -> Router<AppState> {
Router::new()
.route("/admin/health", get(health))
.route("/admin/health/thresholds", put(update_thresholds))
}
// ---------------------------------------------------------------------------
// Thresholds (persisted, editable)
// ---------------------------------------------------------------------------
fn d_disk() -> f64 {
90.0
}
fn d_mem() -> f64 {
90.0
}
fn d_dead() -> i64 {
100
}
fn d_crawl_fail() -> f64 {
20.0
}
fn d_analysis_fail() -> f64 {
20.0
}
fn d_cron() -> i64 {
26
}
fn d_covers() -> i64 {
500
}
/// Alerting thresholds. Each field has a `serde(default)` so a stored row
/// that predates a newly-added field still deserializes (the missing field
/// takes its compiled default — forward-compatible).
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
pub struct HealthThresholds {
#[serde(default = "d_disk")]
pub disk_pct: f64,
#[serde(default = "d_mem")]
pub mem_pct: f64,
#[serde(default = "d_dead")]
pub dead_jobs_max: i64,
#[serde(default = "d_crawl_fail")]
pub crawl_fail_pct: f64,
#[serde(default = "d_analysis_fail")]
pub analysis_fail_pct: f64,
#[serde(default = "d_cron")]
pub cron_freshness_hours: i64,
#[serde(default = "d_covers")]
pub missing_covers_max: i64,
}
impl Default for HealthThresholds {
fn default() -> Self {
Self {
disk_pct: d_disk(),
mem_pct: d_mem(),
dead_jobs_max: d_dead(),
crawl_fail_pct: d_crawl_fail(),
analysis_fail_pct: d_analysis_fail(),
cron_freshness_hours: d_cron(),
missing_covers_max: d_covers(),
}
}
}
impl HealthThresholds {
/// Reject nonsensical values so a check can't be silently disabled by a
/// fat-fingered edit. Percentages 1..=100; counts/hours must be positive.
fn validate(&self) -> Result<(), String> {
let pct = |v: f64, name: &str| {
if (1.0..=100.0).contains(&v) {
Ok(())
} else {
Err(format!("{name} must be between 1 and 100"))
}
};
pct(self.disk_pct, "disk_pct")?;
pct(self.mem_pct, "mem_pct")?;
pct(self.crawl_fail_pct, "crawl_fail_pct")?;
pct(self.analysis_fail_pct, "analysis_fail_pct")?;
if self.dead_jobs_max < 0 {
return Err("dead_jobs_max must be >= 0".into());
}
if self.missing_covers_max < 0 {
return Err("missing_covers_max must be >= 0".into());
}
if self.cron_freshness_hours < 1 {
return Err("cron_freshness_hours must be >= 1".into());
}
Ok(())
}
}
async fn load_thresholds(state: &AppState) -> AppResult<HealthThresholds> {
Ok(match repo::app_settings::get(&state.db, KEY).await? {
// serde(default) on each field fills anything the stored object omits.
Some(v) => serde_json::from_value(v).unwrap_or_default(),
None => HealthThresholds::default(),
})
}
// ---------------------------------------------------------------------------
// Checks
// ---------------------------------------------------------------------------
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum CheckStatus {
Ok,
Warn,
Critical,
}
impl CheckStatus {
fn rank(self) -> u8 {
match self {
CheckStatus::Ok => 0,
CheckStatus::Warn => 1,
CheckStatus::Critical => 2,
}
}
}
/// `Ok` while `value` is strictly below `limit`, else `Warn`. Used for
/// "near a ceiling" gauges (disk/memory/fail-rate percentages, cron-age
/// hours) where hitting the threshold is itself the alert. Pure so the
/// rule is unit-testable without the DB/sensors.
pub(crate) fn over_limit(value: f64, limit: f64) -> CheckStatus {
if value < limit {
CheckStatus::Ok
} else {
CheckStatus::Warn
}
}
/// `Ok` until `value` strictly exceeds `limit`. Used for "max allowed"
/// backlog counts so a limit of `0` warns on the first item — and `0` items
/// against a `0` limit stays `Ok` (no false alarm at exactly the boundary).
pub(crate) fn exceeds(value: i64, limit: i64) -> CheckStatus {
if value > limit {
CheckStatus::Warn
} else {
CheckStatus::Ok
}
}
#[derive(Debug, Clone, Serialize)]
pub struct HealthCheck {
pub id: &'static str,
pub label: &'static str,
pub status: CheckStatus,
pub value: String,
pub threshold: Option<String>,
pub hint: Option<String>,
/// Admin route that remediates this check, when one applies.
pub action_href: Option<&'static str>,
}
#[derive(Debug, Clone, Serialize)]
pub struct HealthReport {
/// Worst status across all checks (drives the page/summary badge).
pub status: CheckStatus,
pub checks: Vec<HealthCheck>,
pub thresholds: HealthThresholds,
}
async fn health(
State(state): State<AppState>,
_admin: RequireAdmin,
) -> AppResult<Json<HealthReport>> {
let thresholds = load_thresholds(&state).await?;
let since_24h = Utc::now() - Duration::hours(24);
// DB-derived signals run concurrently — all hit existing indexes.
let ((pending, running, dead), missing_covers, crawl, analysis, last_tick) = tokio::try_join!(
repo::crawler::job_state_counts(&state.db).map_err(AppError::from),
repo::crawler::count_missing_covers(&state.db).map_err(AppError::from),
repo::crawl_metrics::summary(&state.db, Some(since_24h)).map_err(AppError::from),
repo::page_analysis::analysis_metrics(&state.db, Some(since_24h)),
repo::crawler::last_metadata_tick_at(&state.db).map_err(AppError::from),
)?;
let _ = (pending, running); // queue depth shown elsewhere; not a check yet
let mut checks: Vec<HealthCheck> = Vec::new();
// --- system sensors ---
if let Some(d) = state.storage.local_root().and_then(super::system::disk_stats_for) {
checks.push(HealthCheck {
id: "disk",
label: "Disk usage",
status: over_limit(d.percent_used, thresholds.disk_pct),
value: format!("{:.0}%", d.percent_used),
threshold: Some(format!("{:.0}%", thresholds.disk_pct)),
hint: Some("free space on the storage volume".into()),
action_href: None,
});
}
let mem = super::system::memory_percent_used();
checks.push(HealthCheck {
id: "memory",
label: "Memory usage",
status: over_limit(mem, thresholds.mem_pct),
value: format!("{mem:.0}%"),
threshold: Some(format!("{:.0}%", thresholds.mem_pct)),
hint: None,
action_href: None,
});
// --- queue / backlog ---
checks.push(HealthCheck {
id: "dead_jobs",
label: "Dead jobs",
status: exceeds(dead, thresholds.dead_jobs_max),
value: dead.to_string(),
threshold: Some(thresholds.dead_jobs_max.to_string()),
hint: Some("jobs that exhausted their retries".into()),
action_href: Some("/admin/crawler"),
});
checks.push(HealthCheck {
id: "missing_covers",
label: "Missing covers",
status: exceeds(missing_covers, thresholds.missing_covers_max),
value: missing_covers.to_string(),
threshold: Some(thresholds.missing_covers_max.to_string()),
hint: Some("mangas still awaiting a cover download".into()),
action_href: Some("/admin/crawler"),
});
// --- failure rates (last 24h) ---
let crawl_total: i64 = crawl.iter().map(|s| s.n).sum();
let crawl_failed: i64 = crawl.iter().map(|s| s.failed).sum();
let crawl_pct = pct(crawl_failed, crawl_total);
checks.push(HealthCheck {
id: "crawl_fail_rate",
label: "Crawl failure rate (24h)",
status: if crawl_total == 0 {
CheckStatus::Ok
} else {
over_limit(crawl_pct, thresholds.crawl_fail_pct)
},
value: format!("{crawl_pct:.0}% ({crawl_failed}/{crawl_total})"),
threshold: Some(format!("{:.0}%", thresholds.crawl_fail_pct)),
hint: None,
action_href: Some("/admin/crawler"),
});
let an_pct = pct(analysis.failed, analysis.n);
checks.push(HealthCheck {
id: "analysis_fail_rate",
label: "Analysis failure rate (24h)",
status: if analysis.n == 0 {
CheckStatus::Ok
} else {
over_limit(an_pct, thresholds.analysis_fail_pct)
},
value: format!("{an_pct:.0}% ({}/{})", analysis.failed, analysis.n),
threshold: Some(format!("{:.0}%", thresholds.analysis_fail_pct)),
hint: None,
action_href: Some("/admin/analysis"),
});
// --- crawler daemon liveness (only when the daemon is configured) ---
if let Some(c) = state.crawler().as_ref() {
// Cron freshness: stale tick = the daily pass hasn't run on time.
match last_tick {
Some(t) => {
let age_h = (Utc::now() - t).num_hours();
checks.push(HealthCheck {
id: "cron_freshness",
label: "Metadata cron freshness",
status: over_limit(age_h as f64, thresholds.cron_freshness_hours as f64),
value: format!("{age_h}h ago"),
threshold: Some(format!("{}h", thresholds.cron_freshness_hours)),
hint: Some("time since the last metadata pass tick".into()),
action_href: Some("/admin/crawler"),
});
}
None => checks.push(HealthCheck {
id: "cron_freshness",
label: "Metadata cron freshness",
status: CheckStatus::Ok,
value: "no tick yet".into(),
threshold: None,
hint: Some("the daemon has not completed a pass yet".into()),
action_href: Some("/admin/crawler"),
}),
}
let expired = c.session.is_expired();
checks.push(HealthCheck {
id: "crawler_session",
label: "Crawler session",
status: if expired { CheckStatus::Critical } else { CheckStatus::Ok },
value: if expired { "expired".into() } else { "ok".into() },
threshold: None,
hint: expired.then(|| "chapter workers are idle until the session is refreshed".into()),
action_href: Some("/admin/crawler"),
});
let healthy = matches!(c.browser_manager.phase(), RestartPhase::Healthy);
let phase = match c.browser_manager.phase() {
RestartPhase::Healthy => "healthy",
RestartPhase::Draining => "draining",
RestartPhase::Restarting => "restarting",
};
checks.push(HealthCheck {
id: "crawler_browser",
label: "Crawler browser",
status: if healthy { CheckStatus::Ok } else { CheckStatus::Warn },
value: phase.to_string(),
threshold: None,
hint: (!healthy).then(|| "Chromium is restarting or unavailable".into()),
action_href: Some("/admin/crawler"),
});
}
let status = checks
.iter()
.map(|c| c.status)
.max_by_key(|s| s.rank())
.unwrap_or(CheckStatus::Ok);
Ok(Json(HealthReport {
status,
checks,
thresholds,
}))
}
/// Percentage `numerator/denominator`, guarding divide-by-zero.
fn pct(numerator: i64, denominator: i64) -> f64 {
if denominator <= 0 {
0.0
} else {
(numerator as f64) * 100.0 / (denominator as f64)
}
}
async fn update_thresholds(
State(state): State<AppState>,
admin: RequireAdmin,
Json(body): Json<HealthThresholds>,
) -> AppResult<Json<HealthReport>> {
body.validate().map_err(AppError::InvalidInput)?;
let value = serde_json::to_value(body).map_err(|e| AppError::Other(e.into()))?;
let mut tx = state.db.begin().await?;
repo::app_settings::upsert(&mut *tx, KEY, &value).await?;
repo::admin_audit::insert(
&mut *tx,
admin.0.id,
"update_health_thresholds",
"settings",
None,
value,
)
.await?;
tx.commit().await?;
// Re-evaluate so the client immediately sees checks recomputed against
// the new thresholds.
health(State(state), admin).await
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn over_limit_is_ok_below_warn_at_or_above() {
assert_eq!(over_limit(10.0, 20.0), CheckStatus::Ok);
assert_eq!(over_limit(19.999, 20.0), CheckStatus::Ok);
assert_eq!(over_limit(20.0, 20.0), CheckStatus::Warn);
assert_eq!(over_limit(99.0, 20.0), CheckStatus::Warn);
}
#[test]
fn exceeds_is_strict_and_safe_at_zero() {
assert_eq!(exceeds(0, 0), CheckStatus::Ok); // 0 backlog, 0 limit → no false alarm
assert_eq!(exceeds(1, 0), CheckStatus::Warn);
assert_eq!(exceeds(100, 100), CheckStatus::Ok);
assert_eq!(exceeds(101, 100), CheckStatus::Warn);
}
#[test]
fn pct_guards_zero_denominator() {
assert_eq!(pct(0, 0), 0.0);
assert_eq!(pct(5, 0), 0.0);
assert_eq!(pct(1, 4), 25.0);
}
#[test]
fn worst_status_ranking() {
assert!(CheckStatus::Critical.rank() > CheckStatus::Warn.rank());
assert!(CheckStatus::Warn.rank() > CheckStatus::Ok.rank());
}
#[test]
fn thresholds_validate_rejects_out_of_range() {
let mut t = HealthThresholds::default();
assert!(t.validate().is_ok());
t.disk_pct = 0.0;
assert!(t.validate().is_err());
t = HealthThresholds::default();
t.dead_jobs_max = -1;
assert!(t.validate().is_err());
t = HealthThresholds::default();
t.cron_freshness_hours = 0;
assert!(t.validate().is_err());
}
#[test]
fn thresholds_merge_defaults_for_missing_fields() {
// A stored object that predates `missing_covers_max` still decodes.
let partial = serde_json::json!({ "disk_pct": 80.0 });
let t: HealthThresholds = serde_json::from_value(partial).unwrap();
assert_eq!(t.disk_pct, 80.0);
assert_eq!(t.missing_covers_max, d_covers());
assert_eq!(t.cron_freshness_hours, d_cron());
}
}

View File

@@ -5,7 +5,9 @@
//! `crate::auth::extractor::RequireAdmin`).
pub mod analysis;
pub mod audit;
pub mod crawler;
pub mod health;
pub mod mangas;
pub mod overview;
pub mod resync;
@@ -26,6 +28,8 @@ pub fn routes() -> Router<AppState> {
.merge(storage::routes())
.merge(system::routes())
.merge(overview::routes())
.merge(audit::routes())
.merge(health::routes())
.merge(crawler::routes())
.merge(analysis::routes())
.merge(settings::routes())

View File

@@ -205,6 +205,13 @@ async fn backfill(
resp.more_remaining = more_pages || more_covers;
}
// Audit is written after the action here *by necessity*, not oversight
// (unlike reenqueue/analyze_page, which wrap their single DB mutation +
// audit in one tx). The backfill is a budgeted scan that interleaves
// filesystem `storage.size()` reads with per-batch DB writes across many
// iterations; wrapping it in a transaction would hold one open across all
// that I/O (a long-running tx). The batch size-writes are also idempotent
// recomputations, so a lost audit row has negligible impact.
repo::admin_audit::insert(
&state.db,
admin.0.id,

View File

@@ -169,6 +169,21 @@ fn temperatures() -> Vec<TempStat> {
.collect()
}
/// Lightweight memory-utilization percentage for the health check. Unlike
/// [`memory_and_cpu`] this skips CPU sampling and its 250ms settle delay —
/// health is polled more often and only needs the memory figure.
pub(crate) fn memory_percent_used() -> f64 {
let mut sys =
System::new_with_specifics(RefreshKind::new().with_memory(MemoryRefreshKind::everything()));
sys.refresh_memory();
let total = sys.total_memory();
if total == 0 {
0.0
} else {
(sys.used_memory() as f64) * 100.0 / (total as f64)
}
}
pub(crate) fn disk_stats_for(root: &Path) -> Option<DiskStats> {
let s = nix::sys::statvfs::statvfs(root).ok()?;
// statvfs reports `f_frsize * f_blocks` for total bytes. `f_bavail`

View File

@@ -16,7 +16,7 @@ use crate::api::auth::{validate_password, validate_username};
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::auth::password::hash_password;
use crate::auth::password::hash_password_async;
use crate::domain::User;
use crate::error::{AppError, AppResult};
use crate::repo;
@@ -115,7 +115,7 @@ async fn create_user(
// reject (and vice versa).
validate_username(username)?;
validate_password(&input.password)?;
let pwhash = hash_password(&input.password)?;
let pwhash = hash_password_async(input.password.clone()).await?;
let user = repo::user::admin_create_user(
&state.db,
actor.id,

View File

@@ -17,8 +17,8 @@ use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::app::AppState;
use crate::auth::extractor::{CurrentUser, SESSION_COOKIE_NAME};
use crate::auth::password::{hash_password, verify_password};
use crate::auth::extractor::{ClientIp, CurrentUser, SESSION_COOKIE_NAME};
use crate::auth::password::{hash_password_async, verify_password_async};
use crate::auth::token::{generate_token, hash_token};
use crate::config::AuthConfig;
use crate::domain::user_preferences::{READER_GAPS, READER_MODES};
@@ -38,7 +38,7 @@ pub fn routes() -> Router<AppState> {
"/auth/me/preferences",
get(get_preferences).patch(update_preferences),
)
.route("/auth/tokens", post(create_token))
.route("/auth/tokens", get(list_tokens).post(create_token))
.route("/auth/tokens/:id", delete(delete_token))
}
@@ -75,8 +75,16 @@ pub struct AuthResponse {
#[derive(Debug, Deserialize)]
pub struct CreateTokenInput {
pub name: String,
/// Optional lifetime in days. Omit (or `null`) for a non-expiring
/// token (the historical behaviour). When set, must be 1..=3650.
#[serde(default)]
pub expires_in_days: Option<i64>,
}
/// Upper bound on a requested token lifetime (~10 years). A token that needs
/// to outlive this should be rotated, not minted once forever.
const MAX_TOKEN_EXPIRY_DAYS: i64 = 3650;
#[derive(Debug, Deserialize)]
pub struct ChangePassword {
pub current_password: String,
@@ -99,6 +107,7 @@ pub struct CreatedTokenResponse {
async fn register(
State(state): State<AppState>,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<Credentials>,
) -> AppResult<impl IntoResponse> {
@@ -106,7 +115,7 @@ async fn register(
// the toggle can't be probed for the toggle state via timing —
// disabled and enabled paths both consume a token, and disabled
// returns 403 instead of running argon2.
check_auth_rate_limit(&state, "register")?;
check_auth_rate_limit(&state, "register", client_ip)?;
// Private mode force-blocks self-registration regardless of
// ALLOW_SELF_REGISTER — operators of locked-down instances mint
// accounts via `POST /admin/users` instead.
@@ -117,7 +126,7 @@ async fn register(
validate_username(username)?;
validate_password(&input.password)?;
let pwhash = hash_password(&input.password)?;
let pwhash = hash_password_async(input.password.clone()).await?;
let user = repo::user::create(&state.db, username, &pwhash).await?;
let jar = start_session(&state, &user, jar).await?;
Ok((StatusCode::CREATED, jar, Json(AuthResponse { user })))
@@ -125,16 +134,21 @@ async fn register(
async fn login(
State(state): State<AppState>,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<Credentials>,
) -> AppResult<impl IntoResponse> {
check_auth_rate_limit(&state, "login")?;
check_auth_rate_limit(&state, "login", client_ip)?;
let username = input.username.trim();
if username.is_empty() || input.password.is_empty() {
return Err(AppError::InvalidInput(
"username and password are required".into(),
));
}
// Bound the password before argon2 runs — guards BOTH the real-verify and
// the dummy-hash timing-equaliser branch below, so a giant password can't
// make every login attempt a CPU-DoS.
reject_oversized_password(&input.password)?;
let user = repo::user::find_by_username(&state.db, username).await?;
let Some(user) = user else {
@@ -142,10 +156,11 @@ async fn login(
// response time matches the wrong-password branch — otherwise
// an attacker can enumerate usernames by timing the no-user
// 401 against the wrong-password 401.
let _ = verify_password(&input.password, dummy_password_hash());
let _ = verify_password_async(input.password.clone(), dummy_password_hash().to_string())
.await;
return Err(AppError::Unauthenticated);
};
if !verify_password(&input.password, &user.password_hash) {
if !verify_password_async(input.password.clone(), user.password_hash.clone()).await {
return Err(AppError::Unauthenticated);
}
@@ -201,16 +216,20 @@ async fn me(CurrentUser(user): CurrentUser) -> AppResult<Json<AuthResponse>> {
async fn change_password(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<ChangePassword>,
) -> AppResult<impl IntoResponse> {
check_auth_rate_limit(&state, "change_password")?;
if !verify_password(&input.current_password, &user.password_hash) {
check_auth_rate_limit(&state, "change_password", client_ip)?;
// Cap current_password before verify_password runs argon2 (same DoS
// vector as login). new_password is bounded by validate_password below.
reject_oversized_password(&input.current_password)?;
if !verify_password_async(input.current_password.clone(), user.password_hash.clone()).await {
return Err(AppError::Unauthenticated);
}
validate_password(&input.new_password)?;
let new_hash = hash_password(&input.new_password)?;
let new_hash = hash_password_async(input.new_password.clone()).await?;
let mut tx = state.db.begin().await?;
sqlx::query("UPDATE users SET password_hash = $1 WHERE id = $2")
@@ -280,6 +299,22 @@ async fn update_preferences(
Ok(Json(saved))
}
/// `GET /auth/tokens` — the caller's bot tokens (newest first). The raw bearer
/// is only ever shown once at creation, so this list carries just the metadata
/// (name, created/last-used, expiry); `token_hash` is `#[serde(skip)]`.
#[derive(Debug, Serialize)]
struct TokenListResponse {
items: Vec<ApiToken>,
}
async fn list_tokens(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
) -> AppResult<Json<TokenListResponse>> {
let items = repo::api_token::list_for_user(&state.db, user.id).await?;
Ok(Json(TokenListResponse { items }))
}
async fn create_token(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
@@ -305,8 +340,22 @@ async fn create_token(
details: serde_json::json!({ "name": "max 64 characters" }),
});
}
let expires_at = match input.expires_in_days {
None => None,
Some(days) if (1..=MAX_TOKEN_EXPIRY_DAYS).contains(&days) => {
Some(Utc::now() + Duration::days(days))
}
Some(_) => {
return Err(AppError::ValidationFailed {
message: "token expiry out of range".into(),
details: serde_json::json!({
"expires_in_days": format!("must be between 1 and {MAX_TOKEN_EXPIRY_DAYS}")
}),
});
}
};
let (raw, hash) = generate_token();
let token = repo::api_token::create(&state.db, user.id, name, &hash).await?;
let token = repo::api_token::create(&state.db, user.id, name, &hash, expires_at).await?;
Ok((
StatusCode::CREATED,
Json(CreatedTokenResponse { token, bearer: raw }),
@@ -387,9 +436,13 @@ fn build_expired_cookie(cfg: &AuthConfig) -> Cookie<'static> {
/// any one of them in a tight loop should trip the limit. `endpoint`
/// is included in the rate-limit-hit log line so operators can tell
/// which endpoint is being probed.
fn check_auth_rate_limit(state: &AppState, endpoint: &'static str) -> AppResult<()> {
fn check_auth_rate_limit(
state: &AppState,
endpoint: &'static str,
client_ip: Option<std::net::IpAddr>,
) -> AppResult<()> {
use crate::auth::rate_limit::AcquireResult;
match state.auth_limiter.try_acquire() {
match state.auth_limiter.try_acquire(client_ip) {
AcquireResult::Allowed => Ok(()),
AcquireResult::Denied { retry_after_secs } => {
tracing::warn!(
@@ -425,11 +478,67 @@ pub(crate) fn validate_username(u: &str) -> AppResult<()> {
Ok(())
}
/// Upper bound on password length (bytes). argon2 has no inherent length
/// limit, so without a cap an attacker could submit a multi-megabyte password
/// and make every hash a CPU-heavy DoS. 1024 bytes is far longer than any real
/// passphrase. Measured in bytes (like the min check) since that's what argon2
/// actually processes.
pub(crate) const MAX_PASSWORD_BYTES: usize = 1024;
/// Reject an over-cap password *before* any argon2 work runs. The
/// verification paths (login, change-password) don't go through
/// [`validate_password`] — they hash/verify the raw input — so without this
/// an attacker could submit a multi-megabyte password and turn every
/// login/verify into a CPU-DoS. Length-only and independent of whether the
/// account exists, so it leaks nothing (no username enumeration).
pub(crate) fn reject_oversized_password(p: &str) -> AppResult<()> {
if p.len() > MAX_PASSWORD_BYTES {
return Err(AppError::InvalidInput(format!(
"password must be at most {MAX_PASSWORD_BYTES} bytes"
)));
}
Ok(())
}
pub(crate) fn validate_password(p: &str) -> AppResult<()> {
if p.len() < 8 {
return Err(AppError::InvalidInput(
"password must be at least 8 characters".into(),
));
}
if p.len() > MAX_PASSWORD_BYTES {
return Err(AppError::InvalidInput(
format!("password must be at most {MAX_PASSWORD_BYTES} bytes"),
));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn validate_password_rejects_too_short() {
assert!(validate_password("short").is_err());
assert!(validate_password("1234567").is_err()); // 7 chars
}
#[test]
fn validate_password_accepts_in_range() {
assert!(validate_password("hunter2hunter2").is_ok());
// Exactly at the cap is allowed.
assert!(validate_password(&"a".repeat(MAX_PASSWORD_BYTES)).is_ok());
// Minimum length boundary.
assert!(validate_password("12345678").is_ok());
}
#[test]
fn validate_password_rejects_over_cap() {
// One byte past the cap is refused so a giant password can't turn each
// login/register into an argon2 CPU-DoS.
let too_long = "a".repeat(MAX_PASSWORD_BYTES + 1);
let err = validate_password(&too_long).unwrap_err();
assert!(matches!(err, AppError::InvalidInput(_)));
}
}

View File

@@ -72,7 +72,7 @@ async fn create(
}
}
let bookmark = repo::bookmark::create(
let (bookmark, created) = repo::bookmark::create(
&state.db,
user.id,
input.manga_id,
@@ -103,7 +103,13 @@ async fn create(
}
});
Ok((StatusCode::CREATED, Json(bookmark)))
// 201 for a fresh bookmark, 200 when it already existed (idempotent add).
let status = if created {
StatusCode::CREATED
} else {
StatusCode::OK
};
Ok((status, Json(bookmark)))
}
async fn delete_one(

View File

@@ -13,7 +13,7 @@ use serde::Deserialize;
use serde_json::json;
use uuid::Uuid;
use crate::api::mangas::{next_field, read_field_bytes};
use crate::api::mangas::next_field;
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::CurrentUser;
@@ -21,7 +21,8 @@ use crate::domain::chapter::NewChapter;
use crate::domain::{Chapter, Page};
use crate::error::{AppError, AppResult};
use crate::repo;
use crate::upload::{parse_image, UploadedImage};
use crate::storage::Storage;
use crate::upload::{stage_image_part, StagedImage};
pub fn routes() -> Router<AppState> {
Router::new()
@@ -69,6 +70,21 @@ async fn get_one(
Ok(Json(chapter))
}
/// Add a chapter to a manga.
///
/// **Authorization is intentionally open**: any authenticated principal —
/// a browser session *or* a bot API token — may add a chapter to *any*
/// manga, including crawler-imported rows. This is by design: chapters are
/// treated as community contributions, unlike the manga record itself
/// (title/cover/metadata), whose edits gate through `require_can_edit`
/// (see [`crate::api::mangas`]). The `CurrentUser` binding still requires a
/// valid identity, so contributions are attributable, just not owner-scoped.
///
/// This contract is locked by `tests/api_chapters.rs`
/// (`non_owner_can_upload_chapter`); changing it to owner-only is a
/// deliberate decision, not a drive-by tightening. Until a richer
/// contributor/moderation model lands this is acknowledged, intended
/// behaviour — see the auth note in CLAUDE.md.
async fn create(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
@@ -77,97 +93,178 @@ async fn create(
) -> AppResult<(StatusCode, Json<Chapter>)> {
repo::manga::get(&state.db, manga_id).await?;
// Each `page` part is streamed straight to a staging key as it arrives,
// so at most one page's bytes sit in memory — the whole chapter is never
// buffered (previously every page was held at once, bounded only by the
// 200 MiB body limit and amplified by concurrency). The staged blobs are
// promoted to their final chapter-scoped keys in `finalize_chapter` once
// the chapter id exists; any early exit cleans them up.
let upload_id = Uuid::new_v4();
let mut metadata: Option<NewChapter> = None;
let mut pages: Vec<UploadedImage> = Vec::new();
let mut staged: Vec<StagedImage> = Vec::new();
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = read_field_bytes(field).await?;
metadata =
Some(serde_json::from_slice(&bytes).map_err(|e| {
let stage_result: AppResult<NewChapter> = async {
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = crate::upload::read_capped(
field,
crate::upload::MAX_METADATA_BYTES,
"metadata",
)
.await?;
metadata = Some(serde_json::from_slice(&bytes).map_err(|e| {
AppError::ValidationFailed {
message: "metadata is not valid JSON".into(),
details: json!({ "metadata": e.to_string() }),
}
})?);
}
Some("page") => {
if state.upload.max_pages_per_chapter != 0
&& staged.len() >= state.upload.max_pages_per_chapter
{
return Err(AppError::PayloadTooLarge(format!(
"chapter exceeds the {}-page limit",
state.upload.max_pages_per_chapter
)));
}
let field_name = format!("page[{}]", staged.len());
let img = stage_image_part(
state.storage.as_ref(),
field,
upload_id,
staged.len(),
state.upload.max_file_bytes,
&field_name,
)
.await?;
staged.push(img);
}
_ => continue,
}
Some("page") => {
let bytes = read_field_bytes(field).await?.to_vec();
let field_name = format!("page[{}]", pages.len());
pages.push(parse_image(bytes, state.upload.max_file_bytes, &field_name)?);
}
_ => continue,
}
}
let metadata = metadata.ok_or_else(|| AppError::ValidationFailed {
message: "metadata part is required".into(),
details: json!({ "metadata": "required" }),
})?;
// Chapter number is 1-indexed everywhere (URLs, upload form,
// reader). Reject 0 / negative numbers up front so the row never
// makes it into the DB. Mirrors the page>=1 rule on bookmarks.
if metadata.number < 1 {
return Err(AppError::ValidationFailed {
message: "chapter number must be 1 or greater".into(),
details: json!({ "number": "must be >= 1" }),
});
}
if pages.is_empty() {
return Err(AppError::ValidationFailed {
message: "at least one page is required".into(),
details: json!({ "page": "at least one required" }),
});
let metadata = metadata.take().ok_or_else(|| AppError::ValidationFailed {
message: "metadata part is required".into(),
details: json!({ "metadata": "required" }),
})?;
// Chapter number is 1-indexed everywhere (URLs, upload form,
// reader). Reject 0 / negative numbers up front so the row never
// makes it into the DB. Mirrors the page>=1 rule on bookmarks.
if metadata.number < 1 {
return Err(AppError::ValidationFailed {
message: "chapter number must be 1 or greater".into(),
details: json!({ "number": "must be >= 1" }),
});
}
if staged.is_empty() {
return Err(AppError::ValidationFailed {
message: "at least one page is required".into(),
details: json!({ "page": "at least one required" }),
});
}
Ok(metadata)
}
.await;
// Transactional create. If any storage put or page-row insert
// fails mid-loop, the chapter row + any earlier page rows are
// rolled back so we don't leave a chapter with stale page_count=0
// and orphaned page rows. Bytes already written to storage on a
// rolled-back transaction become orphans on disk; a future reaper
// can sweep them. DB consistency wins over storage tidiness here.
let mut tx = state.db.begin().await?;
let mut chapter = repo::chapter::create(
let metadata = match stage_result {
Ok(m) => m,
Err(e) => {
// Reject before any DB write — remove every page we staged.
cleanup_staging(state.storage.as_ref(), &staged).await;
return Err(e);
}
};
finalize_chapter(&state, manga_id, user.id, &metadata, &staged).await
}
/// Promote the staged pages into a real chapter, all-or-nothing. Creates the
/// chapter row, renames each staged blob to its final chapter-scoped key,
/// and inserts the page rows in one transaction. On any failure the DB rolls
/// back and every blob (already-finalized and still-staged) is removed, so a
/// rejected upload leaves neither partial rows nor orphaned files.
async fn finalize_chapter(
state: &AppState,
manga_id: Uuid,
user_id: Uuid,
metadata: &NewChapter,
staged: &[StagedImage],
) -> AppResult<(StatusCode, Json<Chapter>)> {
let storage = state.storage.as_ref();
let mut tx = match state.db.begin().await {
Ok(tx) => tx,
Err(e) => {
cleanup_staging(storage, staged).await;
return Err(e.into());
}
};
let mut chapter = match repo::chapter::create(
&mut *tx,
manga_id,
metadata.number,
metadata.title.as_deref(),
Some(user.id),
Some(user_id),
)
.await?;
.await
{
Ok(c) => c,
Err(e) => {
cleanup_staging(storage, staged).await;
return Err(e);
}
};
let mut page_ids: Vec<Uuid> = Vec::with_capacity(pages.len());
for (idx, page) in pages.iter().enumerate() {
let mut page_ids: Vec<Uuid> = Vec::with_capacity(staged.len());
let mut finalized: Vec<String> = Vec::with_capacity(staged.len());
for (idx, page) in staged.iter().enumerate() {
let page_number = (idx + 1) as i32;
let nnnn = format!("{:04}", page_number);
let key = format!(
"mangas/{}/chapters/{}/pages/{}.{}",
manga_id, chapter.id, nnnn, page.ext
let final_key = format!(
"mangas/{}/chapters/{}/pages/{:04}.{}",
manga_id, chapter.id, page_number, page.ext
);
state.storage.put(&key, &page.bytes).await?;
let created = repo::page::create(
if let Err(e) = storage.rename(&page.staging_key, &final_key).await {
cleanup_keys(storage, &finalized).await;
cleanup_staging(storage, &staged[idx..]).await;
return Err(e.into());
}
finalized.push(final_key.clone());
match repo::page::create(
&mut *tx,
chapter.id,
page_number,
&key,
&final_key,
page.mime,
page.bytes.len() as i64,
page.size_bytes,
)
.await?;
page_ids.push(created.id);
.await
{
Ok(created) => page_ids.push(created.id),
Err(e) => {
cleanup_keys(storage, &finalized).await;
cleanup_staging(storage, &staged[idx + 1..]).await;
return Err(e);
}
}
}
let page_count = pages.len() as i32;
repo::chapter::set_page_count(&mut *tx, chapter.id, page_count).await?;
let page_count = staged.len() as i32;
if let Err(e) = repo::chapter::set_page_count(&mut *tx, chapter.id, page_count).await {
cleanup_keys(storage, &finalized).await;
return Err(e);
}
chapter.page_count = page_count;
// `repo::chapter::create` returned the row before any pages existed, so
// its `size_bytes` is a stale 0. Every uploaded page's size was just
// captured, so the true total is the sum of their byte lengths — set it
// on the response so the 201 body matches the persisted state.
chapter.size_bytes = Some(pages.iter().map(|p| p.bytes.len() as i64).sum());
// its `size_bytes` is a stale 0. Each staged page carried its byte
// length, so their sum is the chapter's true storage — set it on the
// response so the 201 body matches the persisted state.
chapter.size_bytes = Some(staged.iter().map(|p| p.size_bytes).sum());
tx.commit().await?;
if let Err(e) = tx.commit().await {
cleanup_keys(storage, &finalized).await;
return Err(e.into());
}
// Enqueue AI content-analysis for each new page. Done after commit so a
// rolled-back upload never leaves jobs pointing at nonexistent pages; a
@@ -175,9 +272,7 @@ async fn create(
// re-enqueue endpoint can backfill).
if state.analysis_enabled() {
for page_id in page_ids {
if let Err(e) =
repo::page_analysis::enqueue_for_page(&state.db, page_id, false).await
{
if let Err(e) = repo::page_analysis::enqueue_for_page(&state.db, page_id, false).await {
tracing::warn!(%page_id, error = %e, "failed to enqueue page analysis");
}
}
@@ -186,6 +281,21 @@ async fn create(
Ok((StatusCode::CREATED, Json(chapter)))
}
/// Best-effort removal of staged page blobs after a rejected upload.
async fn cleanup_staging(storage: &dyn Storage, staged: &[StagedImage]) {
for page in staged {
let _ = storage.delete(&page.staging_key).await;
}
}
/// Best-effort removal of already-finalized page blobs when the upload
/// fails after some renames have landed.
async fn cleanup_keys(storage: &dyn Storage, keys: &[String]) {
for key in keys {
let _ = storage.delete(key).await;
}
}
#[derive(Debug, serde::Serialize)]
struct PagesResponse {
pages: Vec<Page>,

View File

@@ -5,6 +5,13 @@
//! The handler uses `Storage::get_stream` so a multi-MB page is piped to
//! the client a chunk at a time instead of buffered server-side.
//!
//! **Thumbnails.** `?w=<px>` serves a width-bounded variant (grids ask for a
//! small width so they download ~KB instead of the 15 MB original). The width
//! snaps to a small allow-list so the number of cached derivatives stays
//! bounded; the resized image is cached in storage under a `thumbs/w{W}/` prefix
//! and regenerated on demand. Only JPEG/PNG sources are thumbnailed (encoders we
//! ship); other formats fall back to the original.
//!
//! **Auth model — capability URLs by design.** This endpoint is
//! deliberately unauthenticated: reads stay public per the project
//! brief, and per-page authorisation would require either a per-request
@@ -16,41 +23,250 @@
//! would gate this endpoint behind a `Storage::owner_of(key)` check;
//! the seam is intentional.
use std::io::Cursor;
use axum::body::Body;
use axum::extract::{Path, State};
use axum::extract::{Path, Query, State};
use axum::http::{header, HeaderName};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::Router;
use image::imageops::FilterType;
use image::{ImageFormat, ImageReader};
use serde::Deserialize;
use crate::app::AppState;
use crate::error::AppResult;
use crate::storage::StorageError;
use crate::error::{AppError, AppResult};
use crate::storage::{Storage, StorageError};
/// Widths a thumbnail may be rendered at. A requested width snaps up to the
/// smallest of these so the set of cached derivatives stays tiny.
const ALLOWED_THUMB_WIDTHS: &[u32] = &[160, 320, 480, 640, 960];
/// Storage key prefix for cached thumbnails.
const THUMB_PREFIX: &str = "thumbs";
/// Decode allocation cap (mirrors `analysis::ocr`): a tiny file declaring huge
/// dimensions is rejected before the decoder allocates, not after (OOM guard).
const MAX_THUMB_DECODE_PIXELS: u64 = 40_000_000;
pub fn routes() -> Router<AppState> {
Router::new().route("/files/*key", get(serve))
}
async fn serve(State(state): State<AppState>, Path(key): Path<String>) -> AppResult<Response> {
let file = match state.storage.get_stream(&key).await {
#[derive(Debug, Deserialize)]
struct ServeQuery {
/// Requested thumbnail width in pixels; absent = serve the original.
w: Option<String>,
}
async fn serve(
State(state): State<AppState>,
Path(key): Path<String>,
Query(q): Query<ServeQuery>,
) -> AppResult<Response> {
// Thumbnail request: only for source formats we can re-encode; anything
// else falls through to serving the original.
if let Some(width) = resolve_thumb_width(q.w.as_deref()) {
if let Some(fmt) = thumb_format_for(&key) {
return serve_thumbnail(&state, &key, width, fmt).await;
}
}
serve_original(&state, &key).await
}
async fn serve_original(state: &AppState, key: &str) -> AppResult<Response> {
let file = match state.storage.get_stream(key).await {
Ok(f) => f,
Err(StorageError::NotFound) => return Err(crate::error::AppError::NotFound),
Err(StorageError::NotFound) => return Err(AppError::NotFound),
Err(e) => return Err(e.into()),
};
let ct = content_type_for(&key);
// `nosniff` makes the contract explicit: the browser must trust the
// Content-Type we declared (and that the magic-byte sniff at upload
// time produced) instead of trying to detect HTML/JS in the body.
// Belt-and-braces vs. polyglot files that survive the upload sniff.
let headers = [
(header::CONTENT_TYPE, ct.to_string()),
(header::CONTENT_LENGTH, file.size_bytes.to_string()),
Ok(image_response(
state,
content_type_for(key),
file.size_bytes.to_string(),
Body::from_stream(file.stream),
))
}
async fn serve_thumbnail(
state: &AppState,
key: &str,
width: u32,
fmt: ImageFormat,
) -> AppResult<Response> {
let derived = thumb_key(key, width);
// Serve the cached variant if it exists.
match state.storage.get_stream(&derived).await {
Ok(f) => {
return Ok(image_response(
state,
content_type_for(key),
f.size_bytes.to_string(),
Body::from_stream(f.stream),
));
}
Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
// Generate from the original.
let original = match state.storage.get(key).await {
Ok(b) => b,
Err(StorageError::NotFound) => return Err(AppError::NotFound),
Err(e) => return Err(e.into()),
};
// Resizing is CPU-bound; keep it off the async worker threads.
let thumb = match tokio::task::spawn_blocking(move || make_thumbnail(&original, width, fmt))
.await
.map_err(|e| AppError::Other(anyhow::anyhow!("thumbnail task join: {e}")))?
{
Ok(t) => t,
Err(e) => {
// The blob's magic bytes passed upload's sniff but the body won't
// decode (corrupt/truncated, or an encoder feature we don't support).
// Don't 500 the reader — fall back to streaming the original, which
// serves without decoding.
tracing::warn!(key, error = %format!("{e:#}"), "thumbnail decode failed; serving original");
return serve_original(state, key).await;
}
};
// Best-effort cache; a write failure just means we regenerate next time.
let _ = state.storage.put(&derived, &thumb).await;
let len = thumb.len().to_string();
Ok(image_response(
state,
content_type_for(key),
len,
Body::from(thumb),
))
}
/// Shared response builder for both the original and thumbnail paths.
fn image_response(
state: &AppState,
content_type: &str,
content_length: String,
body: Body,
) -> Response {
let mut headers = vec![
(header::CONTENT_TYPE, content_type.to_string()),
(header::CONTENT_LENGTH, content_length),
// `nosniff` makes the contract explicit: the browser must trust the
// Content-Type we declared (and that the magic-byte sniff at upload
// time produced) instead of trying to detect HTML/JS in the body.
(
HeaderName::from_static("x-content-type-options"),
"nosniff".to_string(),
),
// Blobs are content-addressed by unguessable, immutable keys (a
// re-upload mints new UUIDs), so a fetched page/cover never changes.
// Cache it for a year and mark it `immutable` so browsers skip
// revalidation entirely.
//
// BUT under PRIVATE_MODE these blobs are auth-gated, so they must NOT be
// marked `public`: a shared cache / CDN would store the object and serve
// it to unauthenticated clients. Use `private` so only the requesting
// user's browser caches it.
(
header::CACHE_CONTROL,
if state.auth.private_mode {
"private, max-age=31536000, immutable".to_string()
} else {
"public, max-age=31536000, immutable".to_string()
},
),
];
Ok((headers, Body::from_stream(file.stream)).into_response())
// Known image types render inline (covers/pages display in the reader). The
// `application/octet-stream` fallback is a blob we couldn't type — it could
// be crafted HTML/JS, so force a download rather than let the browser render
// it inline. Belt to `nosniff`'s braces.
if content_type == "application/octet-stream" {
headers.push((
header::CONTENT_DISPOSITION,
"attachment".to_string(),
));
}
(axum::response::AppendHeaders(headers), body).into_response()
}
/// Parse and clamp a requested thumbnail width. Returns `None` for absent /
/// unparseable / zero widths (serve the original); otherwise snaps the request
/// up to the smallest allowed width (capped at the largest) so cached variants
/// stay bounded.
fn resolve_thumb_width(raw: Option<&str>) -> Option<u32> {
let requested: u32 = raw?.trim().parse().ok()?;
if requested == 0 {
return None;
}
Some(
ALLOWED_THUMB_WIDTHS
.iter()
.copied()
.find(|&w| w >= requested)
.unwrap_or_else(|| *ALLOWED_THUMB_WIDTHS.last().expect("non-empty")),
)
}
/// The re-encode format for a source key, or `None` when it isn't one we ship an
/// encoder for (gif/avif → serve the original instead of a broken thumbnail).
fn thumb_format_for(key: &str) -> Option<ImageFormat> {
match content_type_for(key) {
"image/jpeg" => Some(ImageFormat::Jpeg),
"image/png" => Some(ImageFormat::Png),
_ => None,
}
}
/// The storage key a width-`w` thumbnail of `key` is cached under.
fn thumb_key(key: &str, width: u32) -> String {
format!("{THUMB_PREFIX}/w{width}/{key}")
}
/// Every cached thumbnail key for an original `key`, across all allowed widths.
/// Used by the cover handlers to purge stale variants when a cover (whose key is
/// reused, unlike content-addressed pages) is replaced or deleted.
pub(crate) fn thumbnail_keys(key: &str) -> Vec<String> {
ALLOWED_THUMB_WIDTHS
.iter()
.map(|&w| thumb_key(key, w))
.collect()
}
/// Best-effort deletion of every cached thumbnail for `key`. Call after the
/// underlying blob at `key` changes or is removed.
pub(crate) async fn purge_thumbnails(storage: &dyn Storage, key: &str) {
for derived in thumbnail_keys(key) {
let _ = storage.delete(&derived).await;
}
}
/// Decode, downscale to `width` (aspect-preserving, never upscaling), and
/// re-encode in `fmt`. Pure + synchronous so it runs under `spawn_blocking` and
/// is unit-testable without a server.
fn make_thumbnail(bytes: &[u8], width: u32, fmt: ImageFormat) -> anyhow::Result<Vec<u8>> {
use anyhow::Context;
let mut reader = ImageReader::new(Cursor::new(bytes))
.with_guessed_format()
.context("guess image format for thumbnail")?;
let mut limits = image::Limits::default();
limits.max_alloc = Some(MAX_THUMB_DECODE_PIXELS.saturating_mul(4));
reader.limits(limits);
let img = reader.decode().context("decode image for thumbnail")?;
// Only downscale; a source narrower than the target is served as-is.
let out = if img.width() > width {
img.resize(width, u32::MAX, FilterType::Lanczos3)
} else {
img
};
let mut buf = Cursor::new(Vec::new());
out.write_to(&mut buf, fmt).context("encode thumbnail")?;
Ok(buf.into_inner())
}
fn content_type_for(key: &str) -> &'static str {
@@ -64,3 +280,68 @@ fn content_type_for(key: &str) -> &'static str {
_ => "application/octet-stream",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolve_thumb_width_snaps_and_validates() {
// Absent / unparseable / zero → serve the original.
assert_eq!(resolve_thumb_width(None), None);
assert_eq!(resolve_thumb_width(Some("")), None);
assert_eq!(resolve_thumb_width(Some("abc")), None);
assert_eq!(resolve_thumb_width(Some("0")), None);
// Snap up to the smallest allowed width.
assert_eq!(resolve_thumb_width(Some("1")), Some(160));
assert_eq!(resolve_thumb_width(Some("160")), Some(160));
assert_eq!(resolve_thumb_width(Some("161")), Some(320));
assert_eq!(resolve_thumb_width(Some("640")), Some(640));
// Above the max → capped at the largest allowed width.
assert_eq!(resolve_thumb_width(Some("5000")), Some(960));
}
#[test]
fn thumb_format_only_for_encodable_sources() {
assert_eq!(thumb_format_for("a/b/cover.jpg"), Some(ImageFormat::Jpeg));
assert_eq!(thumb_format_for("a/b/cover.png"), Some(ImageFormat::Png));
// Formats we can't re-encode fall back to the original.
assert_eq!(thumb_format_for("a/b/cover.webp"), None);
assert_eq!(thumb_format_for("a/b/cover.gif"), None);
assert_eq!(thumb_format_for("a/b/cover.avif"), None);
}
#[test]
fn thumb_key_is_prefixed_by_width() {
assert_eq!(
thumb_key("mangas/x/cover.png", 320),
"thumbs/w320/mangas/x/cover.png"
);
assert_eq!(thumbnail_keys("k.png").len(), ALLOWED_THUMB_WIDTHS.len());
}
#[test]
fn make_thumbnail_downscales_and_preserves_aspect() {
// 100x50 red PNG → thumbnail width 40 → 40x20, still decodable PNG.
let mut src = Cursor::new(Vec::new());
image::RgbImage::from_pixel(100, 50, image::Rgb([255, 0, 0]))
.write_to(&mut src, ImageFormat::Png)
.unwrap();
let out = make_thumbnail(src.get_ref(), 40, ImageFormat::Png).unwrap();
let decoded = image::load_from_memory(&out).unwrap();
assert_eq!(decoded.width(), 40);
assert_eq!(decoded.height(), 20, "aspect ratio preserved");
}
#[test]
fn make_thumbnail_does_not_upscale() {
// A 30px-wide source requested at 40 stays 30 wide (no upscaling).
let mut src = Cursor::new(Vec::new());
image::RgbImage::from_pixel(30, 30, image::Rgb([0, 255, 0]))
.write_to(&mut src, ImageFormat::Png)
.unwrap();
let out = make_thumbnail(src.get_ref(), 40, ImageFormat::Png).unwrap();
let decoded = image::load_from_memory(&out).unwrap();
assert_eq!(decoded.width(), 30);
}
}

View File

@@ -8,7 +8,7 @@ use uuid::Uuid;
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::CurrentUser;
use crate::auth::extractor::{CurrentSessionUser, CurrentUser};
use crate::domain::manga::{MangaCard, MangaDetail, MangaPatch, NewManga};
use crate::domain::patch::Patch;
use crate::domain::tag::TagRef;
@@ -22,6 +22,7 @@ pub fn routes() -> Router<AppState> {
.route("/mangas", get(list).post(create))
.route("/mangas/:id", get(get_one).patch(update))
.route("/mangas/:id/similar", get(list_similar))
.route("/me/recommendations", get(list_recommendations))
.route("/mangas/:id/cover", put(put_cover).delete(delete_cover))
.route("/mangas/:id/tags", post(attach_tag))
.route("/mangas/:id/tags/:tag_id", delete(detach_tag))
@@ -51,14 +52,58 @@ pub struct ListParams {
pub limit: i64,
#[serde(default)]
pub offset: i64,
/// Sort field: `created`, `updated` (default), `title`, or `author`.
/// Also accepts the legacy `recent` alias (== `created`). Anything
/// else → 422.
#[serde(default)]
pub sort: repo::manga::ListSort,
pub sort: Option<String>,
/// `asc` or `desc`. Omitted → the field's natural default
/// (dates `desc`, text `asc`). Anything else → 422.
#[serde(default)]
pub order: Option<String>,
}
fn default_limit() -> i64 {
50
}
/// Parse the `sort` wire value into a [`repo::manga::SortField`]. Validation
/// and the legacy `recent` alias live here (not in serde) so unknown values
/// return the structured 422 envelope rather than axum's plain-text
/// `QueryRejection`, matching the rest of this handler's filters.
fn parse_sort(raw: Option<&str>) -> AppResult<repo::manga::SortField> {
use repo::manga::SortField;
match raw.map(str::trim) {
None | Some("") | Some("updated") => Ok(SortField::Updated),
// `recent` was the pre-0.88 name for "newest created first"; keep it
// as an alias so existing bots/scripts don't break.
Some("created") | Some("recent") => Ok(SortField::Created),
Some("title") => Ok(SortField::Title),
Some("author") => Ok(SortField::Author),
Some(other) => Err(AppError::ValidationFailed {
message: format!(
"sort must be one of created, updated, title, author (got {other:?})"
),
details: json!({ "sort": "invalid" }),
}),
}
}
/// Parse the `order` wire value. `None` means the client omitted it, so the
/// caller applies the field's natural default. An explicit unknown value → 422.
fn parse_order(raw: Option<&str>) -> AppResult<Option<repo::manga::SortOrder>> {
use repo::manga::SortOrder;
match raw.map(str::trim) {
None | Some("") => Ok(None),
Some("asc") => Ok(Some(SortOrder::Asc)),
Some("desc") => Ok(Some(SortOrder::Desc)),
Some(other) => Err(AppError::ValidationFailed {
message: format!("order must be 'asc' or 'desc' (got {other:?})"),
details: json!({ "order": "invalid" }),
}),
}
}
fn parse_uuid_csv(field: &str, raw: Option<&str>) -> AppResult<Vec<Uuid>> {
let Some(raw) = raw else { return Ok(Vec::new()) };
let mut out = Vec::new();
@@ -94,6 +139,10 @@ async fn list(
Some(s.to_string())
}
};
let sort = parse_sort(params.sort.as_deref())?;
// Omitted `order` falls back to the field's natural direction, so a bare
// `?sort=title` reads A→Z just like the UI shows it.
let order = parse_order(params.order.as_deref())?.unwrap_or_else(|| sort.default_order());
let q = repo::manga::ListQuery {
search: params.search.filter(|s| !s.trim().is_empty()),
status,
@@ -104,10 +153,13 @@ async fn list(
cw_exclude: crate::api::page_tags::parse_warnings_csv(params.cw_exclude.as_deref())?,
limit,
offset,
sort: params.sort,
sort,
order,
};
let (items, total) = repo::manga::list_cards(&state.db, &q).await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
Ok(Json(PagedResponse::with_optional_total(
items, limit, offset, total,
)))
}
async fn get_one(
@@ -139,6 +191,28 @@ async fn list_similar(
Ok(Json(json!({ "items": items })))
}
const RECOMMENDATIONS_LIMIT: i64 = 12;
#[derive(Debug, Deserialize)]
pub struct RecommendationParams {
#[serde(default)]
pub limit: Option<i64>,
}
/// `GET /api/v1/me/recommendations` — content-based "Recommended for you",
/// ranked by tag overlap with the signed-in user's likes/bookmarks (minus
/// dislikes). Returns `{ "items": [...] }` (a fixed top-N, like `/similar`);
/// empty when the user has no taste signals yet.
async fn list_recommendations(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Query(params): Query<RecommendationParams>,
) -> AppResult<Json<serde_json::Value>> {
let limit = params.limit.unwrap_or(RECOMMENDATIONS_LIMIT).clamp(1, 50);
let items = repo::manga::list_recommendations(&state.db, user.id, limit).await?;
Ok(Json(json!({ "items": items })))
}
/// `POST /api/v1/mangas` is multipart/form-data. Parts:
///
/// - `metadata` (required): JSON body matching `NewManga` — title, optional
@@ -161,11 +235,17 @@ async fn create(
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = read_field_bytes(field).await?;
let bytes = crate::upload::read_capped(
field,
crate::upload::MAX_METADATA_BYTES,
"metadata",
)
.await?;
metadata = Some(parse_metadata_json(&bytes)?);
}
Some("cover") => {
let bytes = read_field_bytes(field).await?.to_vec();
let bytes =
crate::upload::read_capped(field, state.upload.max_file_bytes, "cover").await?;
cover = Some(parse_image(bytes, state.upload.max_file_bytes, "cover")?);
}
_ => continue,
@@ -206,8 +286,8 @@ async fn create(
)
.await?;
let author_refs = repo::author::set_for_manga(&mut *tx, manga.id, &authors).await?;
repo::genre::set_for_manga(&mut *tx, manga.id, &metadata.genre_ids).await?;
let author_refs = repo::author::set_for_manga(&mut tx, manga.id, &authors).await?;
repo::genre::set_for_manga(&mut tx, manga.id, &metadata.genre_ids).await?;
if let Some(img) = cover {
let key = format!("mangas/{}/cover.{}", manga.id, img.ext);
@@ -229,13 +309,14 @@ async fn create(
async fn update(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
session: Option<CurrentSessionUser>,
Path(id): Path<Uuid>,
Json(patch): Json<MangaPatch>,
) -> AppResult<Json<MangaDetail>> {
if !repo::manga::exists(&state.db, id).await? {
return Err(AppError::NotFound);
}
require_can_edit(&state, id, user.id).await?;
require_can_edit(&state, id, user.id, admin_via_session(&session)).await?;
if let Some(ref status) = patch.status {
let trimmed = status.trim();
@@ -271,7 +352,7 @@ async fn update(
let mut tx = state.db.begin().await?;
let _updated = repo::manga::update_basics(
&mut *tx,
&mut tx,
id,
patch.title.as_deref().map(str::trim),
patch.status.as_deref().map(str::trim),
@@ -281,10 +362,10 @@ async fn update(
)
.await?;
if let Some(ref names) = authors_owned {
repo::author::set_for_manga(&mut *tx, id, names).await?;
repo::author::set_for_manga(&mut tx, id, names).await?;
}
if let Some(ref ids) = patch.genre_ids {
repo::genre::set_for_manga(&mut *tx, id, ids).await?;
repo::genre::set_for_manga(&mut tx, id, ids).await?;
}
tx.commit().await?;
@@ -300,18 +381,20 @@ async fn update(
async fn put_cover(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
session: Option<CurrentSessionUser>,
Path(id): Path<Uuid>,
mut multipart: Multipart,
) -> AppResult<Json<MangaDetail>> {
if !repo::manga::exists(&state.db, id).await? {
return Err(AppError::NotFound);
}
require_can_edit(&state, id, user.id).await?;
require_can_edit(&state, id, user.id, admin_via_session(&session)).await?;
let mut cover: Option<UploadedImage> = None;
while let Some(field) = next_field(&mut multipart).await? {
if field.name() == Some("cover") {
let bytes = read_field_bytes(field).await?.to_vec();
let bytes =
crate::upload::read_capped(field, state.upload.max_file_bytes, "cover").await?;
cover = Some(parse_image(bytes, state.upload.max_file_bytes, "cover")?);
}
}
@@ -326,6 +409,17 @@ async fn put_cover(
let old_key = repo::manga::get(&state.db, id).await?.cover_image_path;
let new_key = format!("mangas/{}/cover.{}", id, img.ext);
state.storage.put(&new_key, &img.bytes).await?;
// Cover keys are reused (mangas/{id}/cover.{ext}), so a same-extension
// replacement overwrites the blob at an existing key — drop any thumbnails
// cached for it so we don't serve a stale variant of the old cover.
crate::api::files::purge_thumbnails(state.storage.as_ref(), &new_key).await;
// Commit the DB pointer to the new blob BEFORE removing the old one. If we
// deleted the old blob first and this write then failed, the row would point
// at a deleted blob (a cover that 404s) and the new blob would be orphaned.
// Ordering it first means a failure here leaves the row pointing at the
// still-present old blob; only a harmless orphan (new blob) can result.
repo::manga::set_cover_image_path(&state.db, id, &new_key, img.bytes.len() as i64).await?;
if let Some(prev) = old_key.as_deref() {
if prev != new_key {
@@ -336,10 +430,10 @@ async fn put_cover(
Ok(()) | Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
// Old key's cached thumbnails are now orphaned.
crate::api::files::purge_thumbnails(state.storage.as_ref(), prev).await;
}
}
repo::manga::set_cover_image_path(&state.db, id, &new_key, img.bytes.len() as i64).await?;
Ok(Json(repo::manga::get_detail(&state.db, id).await?))
}
@@ -349,18 +443,23 @@ async fn put_cover(
async fn delete_cover(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
session: Option<CurrentSessionUser>,
Path(id): Path<Uuid>,
) -> AppResult<Json<MangaDetail>> {
if !repo::manga::exists(&state.db, id).await? {
return Err(AppError::NotFound);
}
require_can_edit(&state, id, user.id).await?;
require_can_edit(&state, id, user.id, admin_via_session(&session)).await?;
if let Some(key) = repo::manga::get(&state.db, id).await?.cover_image_path {
// Clear the DB pointer BEFORE deleting the blob, so a storage-delete
// failure can't leave the row pointing at a removed blob. A failure
// after the clear only orphans the blob (harmless).
repo::manga::clear_cover_image_path(&state.db, id).await?;
match state.storage.delete(&key).await {
Ok(()) | Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
repo::manga::clear_cover_image_path(&state.db, id).await?;
crate::api::files::purge_thumbnails(state.storage.as_ref(), &key).await;
}
Ok(Json(repo::manga::get_detail(&state.db, id).await?))
}
@@ -467,11 +566,24 @@ fn validate_new_manga(input: &NewManga) -> AppResult<()> {
/// exist (the caller runs [`repo::manga::exists`] first so a missing id
/// surfaces as `NotFound`, not `Forbidden`).
///
/// Rule: a non-NULL `uploaded_by` must match the current user. Legacy
/// rows with `uploaded_by IS NULL` (pre-migration-0011) are still
/// editable by any signed-in user — there's nobody to gate on yet, and
/// the historical-data note in 0011 acknowledges the gap. Once an
/// admin role lands the NULL case can flip to admin-only.
/// Rule: a non-NULL `uploaded_by` must match the current user, OR the
/// caller is an admin **via a session cookie**. Rows with
/// `uploaded_by IS NULL` (crawler-imported + legacy pre-0011) are
/// admin-via-session-only.
///
/// Why "via session": [`crate::auth::extractor`] documents that admin
/// authority is unreachable via bearer tokens — a leaked bot token
/// must not yield admin powers. Keep that invariant by reading the
/// `is_admin_session` flag from the optional `CurrentSessionUser`
/// extractor (`None` ⇒ caller authenticated via bearer ⇒ admin path
/// is closed) rather than from `User.is_admin` (which is true whichever
/// way the user authenticated).
///
/// Why NULL is admin-only: every crawler row has NULL `uploaded_by`
/// (see `repo::crawler::upsert_manga`). The earlier "any signed-in
/// user can edit a NULL row" rule meant `register → PATCH
/// /api/v1/mangas/<id>` rewrote the catalog for free, and
/// `delete_cover` removed blobs from `storage`.
///
/// Returns `Forbidden` (not `NotFound`) on owner mismatch — mangas
/// are listable via `GET /mangas`, so existence isn't a secret and
@@ -479,14 +591,32 @@ fn validate_new_manga(input: &NewManga) -> AppResult<()> {
/// `repo::collection::require_owner`, which collapses both states to
/// `NotFound` because collections are private to a user and existence
/// itself is information worth hiding from non-owners.
async fn require_can_edit(state: &AppState, manga_id: Uuid, user_id: Uuid) -> AppResult<()> {
async fn require_can_edit(
state: &AppState,
manga_id: Uuid,
user_id: Uuid,
is_admin_session: bool,
) -> AppResult<()> {
match repo::manga::uploaded_by(&state.db, manga_id).await? {
Some(owner) if owner != user_id => Err(AppError::Forbidden),
// Some(owner) == user_id (good) or None (legacy row, no owner).
_ => Ok(()),
Some(owner) if owner == user_id => Ok(()),
Some(_) if is_admin_session => Ok(()),
Some(_) => Err(AppError::Forbidden),
None if is_admin_session => Ok(()),
None => Err(AppError::Forbidden),
}
}
/// True iff the caller authenticated via session cookie AND is an admin.
/// Used to gate the admin-only branches of [`require_can_edit`]. The
/// `Option<CurrentSessionUser>` extractor returns `None` on a bearer-only
/// request, so admin authority over bot tokens never composes here.
fn admin_via_session(session: &Option<CurrentSessionUser>) -> bool {
session
.as_ref()
.map(|CurrentSessionUser(u)| u.is_admin)
.unwrap_or(false)
}
async fn validate_genre_ids(state: &AppState, ids: &[Uuid]) -> AppResult<()> {
if ids.is_empty() {
return Ok(());
@@ -560,13 +690,7 @@ pub(crate) async fn next_field(
.map_err(map_multipart_error)
}
pub(crate) async fn read_field_bytes(
field: axum::extract::multipart::Field<'_>,
) -> AppResult<axum::body::Bytes> {
field.bytes().await.map_err(map_multipart_error)
}
fn map_multipart_error(e: axum::extract::multipart::MultipartError) -> AppError {
pub(crate) fn map_multipart_error(e: axum::extract::multipart::MultipartError) -> AppError {
let status = e.status();
if status == StatusCode::PAYLOAD_TOO_LARGE {
AppError::PayloadTooLarge("upload exceeds the request size limit".into())

View File

@@ -11,6 +11,7 @@ pub mod history;
pub mod mangas;
pub mod page_tags;
pub mod pagination;
pub mod reactions;
pub mod tags;
use axum::Router;
@@ -31,5 +32,6 @@ pub fn routes() -> Router<AppState> {
.merge(collections::routes())
.merge(page_tags::routes())
.merge(history::routes())
.merge(reactions::routes())
.merge(admin::routes())
}

View File

@@ -392,10 +392,9 @@ pub struct AggregateParams {
pub limit: i64,
#[serde(default)]
pub offset: i64,
/// Reserved for the planned OCR text-search input. Accepted on
/// the wire so adding OCR later won't break the API shape, but
/// rejected with 501 `text_search_not_yet_supported` if non-empty
/// until the backend supports it.
/// OCR text filter. When non-empty, only pages whose analysis
/// `search_doc` matches the query (`plainto_tsquery`) are aggregated.
/// Blank/absent ⇒ tag-only aggregation.
#[serde(default)]
pub text: Option<String>,
}
@@ -411,32 +410,17 @@ fn parse_order(raw: Option<&str>) -> AppResult<Order> {
}
}
fn ensure_text_unsupported(text: Option<&str>) -> AppResult<()> {
// Future OCR search will plug in here. Until then, return a
// distinct code (`text_search_not_yet_supported`) so clients can
// detect "feature pending" vs. a generic 4xx — the code is the
// wire contract, not the message.
if text.is_some_and(|s| !s.trim().is_empty()) {
return Err(AppError::NotImplemented {
code: "text_search_not_yet_supported",
message: "text search is reserved for the planned OCR input but not yet supported",
});
}
Ok(())
}
async fn list_chapters_for_tag(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Query(params): Query<AggregateParams>,
) -> AppResult<Json<PagedResponse<TaggedChapterAggregate>>> {
ensure_text_unsupported(params.text.as_deref())?;
let tag = normalize_tag(&params.tag)?;
let order = parse_order(params.order.as_deref())?;
let limit = params.limit.clamp(1, 200);
let offset = params.offset.max(0);
let (items, total) = repo::page_tag::aggregate_chapters_for_tag(
&state.db, user.id, &tag, order, limit, offset,
&state.db, user.id, &tag, order, limit, offset, params.text.as_deref(),
)
.await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
@@ -447,13 +431,12 @@ async fn list_mangas_for_tag(
CurrentUser(user): CurrentUser,
Query(params): Query<AggregateParams>,
) -> AppResult<Json<PagedResponse<TaggedMangaAggregate>>> {
ensure_text_unsupported(params.text.as_deref())?;
let tag = normalize_tag(&params.tag)?;
let order = parse_order(params.order.as_deref())?;
let limit = params.limit.clamp(1, 200);
let offset = params.offset.max(0);
let (items, total) = repo::page_tag::aggregate_mangas_for_tag(
&state.db, user.id, &tag, order, limit, offset,
&state.db, user.id, &tag, order, limit, offset, params.text.as_deref(),
)
.await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))

View File

@@ -34,4 +34,18 @@ impl<T> PagedResponse<T> {
page: PageInfo { limit, offset, total: Some(total) },
}
}
/// For handlers that compute `total` only on some pages (e.g. the first)
/// and leave it `None` elsewhere.
pub fn with_optional_total(
items: Vec<T>,
limit: i64,
offset: i64,
total: Option<i64>,
) -> Self {
Self {
items,
page: PageInfo { limit, offset, total },
}
}
}

View File

@@ -0,0 +1,69 @@
//! Manga reactions (like/dislike) — a private, per-user taste signal.
//! Writes require auth; the read is scoped under `/me/` so the URL can't be
//! used to peek at another user's reactions.
use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::routing::{get, put};
use axum::{Json, Router};
use serde::Deserialize;
use serde_json::json;
use uuid::Uuid;
use crate::app::AppState;
use crate::auth::extractor::CurrentUser;
use crate::domain::reaction::{MangaReaction, Reaction};
use crate::error::{AppError, AppResult};
use crate::repo;
pub fn routes() -> Router<AppState> {
Router::new()
.route(
"/mangas/:id/reaction",
put(set_reaction).delete(clear_reaction),
)
.route("/me/reactions/:manga_id", get(get_reaction))
}
#[derive(Debug, Deserialize)]
pub struct SetReactionBody {
pub reaction: String,
}
async fn set_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
Json(body): Json<SetReactionBody>,
) -> AppResult<Json<MangaReaction>> {
// Validate against the closed vocabulary here so a bad value is a clean
// 422 rather than relying on the DB CHECK to surface as a 500.
let reaction = Reaction::parse(&body.reaction).ok_or_else(|| AppError::ValidationFailed {
message: "reaction must be 'like' or 'dislike'".into(),
details: json!({ "reaction": "must be 'like' or 'dislike'" }),
})?;
// Unknown manga → 404 via the FK-violation mapping in repo::reaction.
repo::reaction::upsert(&state.db, user.id, manga_id, reaction).await?;
Ok(Json(MangaReaction {
manga_id,
reaction: Some(reaction),
}))
}
async fn clear_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
) -> AppResult<StatusCode> {
repo::reaction::clear(&state.db, user.id, manga_id).await?;
Ok(StatusCode::NO_CONTENT)
}
async fn get_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
) -> AppResult<Json<MangaReaction>> {
let reaction = repo::reaction::get(&state.db, user.id, manga_id).await?;
Ok(Json(MangaReaction { manga_id, reaction }))
}

View File

@@ -170,6 +170,9 @@ pub struct CrawlerControl {
/// Used by the "run metadata pass now" endpoint; `None` when no
/// `CRAWLER_START_URL` is configured (cron disabled).
pub metadata_pass: Option<Arc<dyn MetadataPass>>,
/// Used by the "reconcile missing" endpoint; `None` when no
/// `CRAWLER_START_URL` is configured.
pub reconcile_pass: Option<Arc<dyn crate::crawler::daemon::ReconcilePass>>,
/// Drain budget for a manually-triggered coordinated browser restart.
pub drain_deadline: std::time::Duration,
/// Held for the duration of a `/admin/crawler/run` pass so a second
@@ -263,7 +266,8 @@ impl DaemonReloader for Supervisors {
Arc::clone(&self.storage),
&cfg,
Arc::clone(&self.analysis_events),
)?;
)
.await?;
*guard = Some(handle);
tracing::info!(model = %cfg.model, "analysis daemon (re)started from settings");
} else {
@@ -273,9 +277,15 @@ impl DaemonReloader for Supervisors {
}
}
/// How often the background reaper sweeps expired sessions. Hourly is ample:
/// the sweep is a single indexed DELETE and expired rows are already invisible
/// to auth, so this is purely storage hygiene.
const SESSION_GC_INTERVAL: std::time::Duration = std::time::Duration::from_secs(3600);
pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
let db = PgPoolOptions::new()
.max_connections(10)
.max_connections(config.db.max_connections)
.acquire_timeout(config.db.acquire_timeout)
.connect(&config.database_url)
.await?;
sqlx::migrate!("./migrations").run(&db).await?;
@@ -332,6 +342,27 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
tracing::info!("analysis worker disabled");
}
// Periodic reaper for lapsed sessions. `find_active` already ignores
// expired rows, so this only reclaims storage — without it the table grows
// unbounded as sessions lapse. Detached and best-effort: a failed sweep is
// logged and retried next tick. Runs regardless of crawler/analysis config
// since sessions exist in every deployment.
{
let db = db.clone();
tokio::spawn(async move {
let mut ticker = tokio::time::interval(SESSION_GC_INTERVAL);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
ticker.tick().await;
match repo::session::delete_expired(&db).await {
Ok(0) => {}
Ok(n) => tracing::info!(reaped = n, "session gc: removed expired sessions"),
Err(e) => tracing::warn!(?e, "session gc sweep failed; retrying next tick"),
}
}
});
}
let auth_limiter = Arc::new(AuthRateLimiter::new(config.auth.rate_limit));
let state = AppState {
db,
@@ -402,42 +433,121 @@ async fn load_effective_analysis(
/// Spawn the AI content-analysis worker daemon with the given config. Returns
/// its handle. Independent of the crawler daemon (works for uploads with the
/// crawler off). Uses a plain reqwest client — no cookie jar / proxy.
fn spawn_analysis_daemon(
async fn spawn_analysis_daemon(
db: PgPool,
storage: Arc<dyn Storage>,
cfg: &AnalysisConfig,
events: Arc<crate::analysis::events::AnalysisEvents>,
) -> anyhow::Result<crate::analysis::daemon::AnalysisDaemonHandle> {
let http = reqwest::Client::builder()
.timeout(cfg.request_timeout)
.build()
.context("build analysis http client")?;
let vision = crate::analysis::vision::VisionClient::new(http, cfg);
let dispatcher = Arc::new(crate::analysis::daemon::RealAnalyzeDispatcher {
db: db.clone(),
storage,
vision,
model: cfg.model.clone(),
max_image_bytes: cfg.max_image_bytes,
});
// When a readiness URL is configured, gate leasing on it so an
// autoscaler that idle-stops the vision container never lets a job burn
// its retries. A dedicated short-timeout client keeps the probe snappy
// and independent of the (long) per-request analysis timeout.
let readiness: Option<Arc<dyn crate::analysis::daemon::VisionReadiness>> =
match &cfg.vision_health_url {
Some(url) if !url.is_empty() => {
let probe = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
.build()
.context("build vision readiness http client")?;
Some(Arc::new(crate::analysis::daemon::HttpVisionReadiness {
http: probe,
health_url: url.clone(),
}))
}
_ => None,
};
// Reclaim jobs orphaned by a previous crash/kill. `crawler::jobs::reclaim_orphaned`
// is keyed only on `state='running' AND leased_until < now()`, so it
// covers any kind that uses the table — including this daemon's
// `analyze_page` jobs. Previously this ran only inside
// `spawn_crawler_daemon`, which meant a deploy with the crawler off
// (analysis-only) never refunded orphaned analysis leases until the
// lease expiry path lazily picked them up. Safe under multi-replica
// (only expired leases are touched) and idempotent (the crawler-side
// call happens-before this if both are running).
match crate::crawler::jobs::reclaim_orphaned(&db).await {
Ok(0) => {}
Ok(n) => tracing::info!(
reclaimed = n,
"analysis: reclaimed orphaned in-flight jobs at startup"
),
Err(e) => tracing::warn!(?e, "analysis: reclaim_orphaned at startup failed"),
}
// Pick the engine. The OCR backend runs in-process (no network, no
// readiness gate); the vision backend talks to a local LLM server.
let (dispatcher, readiness): (
Arc<dyn crate::analysis::daemon::AnalyzeDispatcher>,
Option<Arc<dyn crate::analysis::daemon::VisionReadiness>>,
) = match cfg.effective_backend() {
crate::config::AnalysisBackend::Ocr => {
// Load the `.rten` models once; a bad path is a loud boot error.
let engine = crate::analysis::ocr::OcrsEngine::from_model_paths(
&cfg.ocr_detection_model,
&cfg.ocr_recognition_model,
cfg.ocr_max_decode_pixels,
)
.context("build ocrs engine")?;
// Cap concurrent CPU-bound OCR runs across all workers so a high
// ANALYSIS_WORKERS can't oversubscribe the blocking pool.
let cores = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1);
let permits =
crate::analysis::ocr::ocr_concurrency_limit(cfg.workers, cores);
let dispatcher = Arc::new(crate::analysis::ocr::OcrAnalyzeDispatcher {
db: db.clone(),
storage,
engine: Arc::new(engine),
max_image_bytes: cfg.max_image_bytes,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(permits)),
});
// In-process engine is always ready — no gate.
(dispatcher, None)
}
crate::config::AnalysisBackend::Vision => {
let http = reqwest::Client::builder()
.timeout(cfg.request_timeout)
// Refuse to honour ambient HTTP_PROXY / HTTPS_PROXY container
// env. The vision call carries an env-managed bearer token +
// page image bytes; a stray upstream proxy would exfiltrate
// both. Mirrors the crawler client's `.no_proxy()` (see
// `spawn_crawler_daemon`).
.no_proxy()
// Re-validate redirect hops so a hostile/compromised vision
// endpoint can't 302 the bearer token + page bytes into the
// deployment's internal network. No allowlist here (the
// endpoint is a single admin-configured URL), so the policy
// enforces scheme + private-IP only.
.redirect(crate::crawler::safety::public_redirect_policy())
// NOTE: deliberately no `.dns_resolver(safe_dns_resolver())`.
// The vision endpoint is a single operator-configured internal
// service (e.g. `mangalord-vision`) that legitimately resolves
// to a private Docker IP; a private-IP-rejecting resolver drops
// every call. Redirect hops stay guarded by the policy above,
// and the URL is operator-set, not attacker-controlled input.
.build()
.context("build analysis http client")?;
let vision = crate::analysis::vision::VisionClient::new(http, cfg);
let dispatcher = Arc::new(crate::analysis::daemon::RealAnalyzeDispatcher {
db: db.clone(),
storage,
vision,
model: cfg.model.clone(),
max_image_bytes: cfg.max_image_bytes,
});
// When a readiness URL is configured, gate leasing on it so an
// autoscaler that idle-stops the vision container never lets a job
// burn its retries. A dedicated short-timeout client keeps the
// probe snappy and independent of the (long) per-request timeout.
let readiness: Option<Arc<dyn crate::analysis::daemon::VisionReadiness>> =
match &cfg.vision_health_url {
Some(url) if !url.is_empty() => {
let probe = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
// Same reasoning as the main analysis client: do
// not honour ambient HTTP_PROXY env. The readiness
// probe is unauthenticated but a hostile upstream
// still gets a useful side-channel on backend
// uptime + vision health.
.no_proxy()
.redirect(crate::crawler::safety::public_redirect_policy())
// No private-IP resolver: same operator-configured
// internal vision host as the analysis client above.
.build()
.context("build vision readiness http client")?;
Some(Arc::new(crate::analysis::daemon::HttpVisionReadiness {
http: probe,
health_url: url.clone(),
}))
}
_ => None,
};
(dispatcher, readiness)
}
};
let handle = crate::analysis::daemon::spawn(
db,
CancellationToken::new(),
@@ -449,7 +559,21 @@ fn spawn_analysis_daemon(
readiness,
},
);
tracing::info!(workers = cfg.workers, model = %cfg.model, "analysis worker daemon started");
// Log the *effective* backend (what the worker actually dispatches
// through), not the raw `cfg.backend` — with vision dormant they diverge
// when an operator requests `vision`, and a misleading line here was a
// real observability footgun. The model label tracks the effective engine.
let effective_backend = cfg.effective_backend();
let effective_model: &str = match effective_backend {
crate::config::AnalysisBackend::Ocr => crate::analysis::ocr::OCR_MODEL_LABEL,
crate::config::AnalysisBackend::Vision => &cfg.model,
};
tracing::info!(
workers = cfg.workers,
backend = ?effective_backend,
model = %effective_model,
"analysis worker daemon started"
);
Ok(handle)
}
@@ -469,6 +593,10 @@ async fn spawn_crawler_daemon(
cfg: &CrawlerConfig,
analysis_enabled: Arc<AtomicBool>,
) -> anyhow::Result<SpawnedDaemon> {
// Publish the opt-in browser SSRF-interception toggle so every headless
// navigation (via `intercept::open_page`) honours it. Off by default.
crate::crawler::intercept::set_enabled(cfg.ssrf_intercept);
// Reqwest client with a shared cookie jar so CDN image fetches include
// PHPSESSID. The same `Arc<Jar>` is held by the SessionController, so a
// runtime session refresh rewrites it in place. Initial value: a
@@ -489,6 +617,13 @@ async fn spawn_crawler_daemon(
let mut http_builder = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.no_proxy()
// Re-validate every redirect hop against the download allowlist:
// reqwest's default policy follows up to 10 redirects, and
// `is_safe_url` only guards the initial URL, so an allowlisted CDN
// 302ing to a private IP would otherwise be followed (SSRF).
.redirect(crate::crawler::safety::safe_redirect_policy(
cfg.download_allowlist.clone(),
))
.cookie_provider(Arc::clone(&cookie_jar));
if let Some(ua) = &cfg.user_agent {
http_builder = http_builder.user_agent(ua);
@@ -497,6 +632,17 @@ async fn spawn_crawler_daemon(
http_builder = http_builder
.proxy(reqwest::Proxy::all(proxy).with_context(|| format!("parse proxy: {proxy}"))?);
}
// DNS-rebinding guard: reject hosts that resolve to a private/internal IP,
// complementing the string-level allowlist check which can't see
// post-resolution addresses. Attached on the direct path AND on http(s)
// proxies (reqwest resolves the target itself there). Skipped only for SOCKS
// proxies, where the proxy — not reqwest — resolves the target, so the only
// name this resolver would see is the proxy's OWN host (legitimately on a
// private Docker IP, e.g. `tor` → 172.x); attaching it there rejected every
// fetch for zero gain. See `should_attach_safe_resolver`.
if crate::crawler::safety::should_attach_safe_resolver(cfg.proxy.as_deref()) {
http_builder = http_builder.dns_resolver(crate::crawler::safety::safe_dns_resolver());
}
let http = http_builder.build().context("build crawler reqwest")?;
let mut rate = HostRateLimiters::new(std::time::Duration::from_millis(cfg.rate_ms));
@@ -599,6 +745,19 @@ async fn spawn_crawler_daemon(
m
});
let reconcile_pass: Option<Arc<dyn crate::crawler::daemon::ReconcilePass>> =
cfg.start_url.as_ref().map(|url| {
let m: Arc<dyn crate::crawler::daemon::ReconcilePass> = Arc::new(RealReconcilePass {
browser_manager: Arc::clone(&browser_manager),
db: db.clone(),
rate: Arc::clone(&rate),
start_url: url.clone(),
status: status.clone(),
tor: tor.as_ref().map(Arc::clone),
});
m
});
let dispatcher: Arc<dyn ChapterDispatcher> = Arc::new(RealChapterDispatcher {
browser_manager: Arc::clone(&browser_manager),
db: db.clone(),
@@ -607,6 +766,7 @@ async fn spawn_crawler_daemon(
rate: Arc::clone(&rate),
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
max_images_per_chapter: cfg.max_images_per_chapter,
analysis_enabled,
transient_failures: Arc::new(AtomicU32::new(0)),
restart_threshold: cfg.browser_restart_threshold,
@@ -623,6 +783,7 @@ async fn spawn_crawler_daemon(
rate: Arc::clone(&rate),
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
max_images_per_chapter: cfg.max_images_per_chapter,
tor: tor.as_ref().map(Arc::clone),
});
@@ -679,6 +840,7 @@ async fn spawn_crawler_daemon(
session: session_controller,
status,
metadata_pass,
reconcile_pass,
drain_deadline: cfg.job_timeout,
manual_pass_lock: Arc::new(tokio::sync::Mutex::new(())),
});
@@ -770,6 +932,36 @@ impl MetadataPass for RealMetadataPass {
}
}
struct RealReconcilePass {
browser_manager: Arc<BrowserManager>,
db: PgPool,
rate: Arc<HostRateLimiters>,
start_url: String,
status: crate::crawler::status::StatusHandle,
tor: Option<Arc<crate::crawler::tor::TorController>>,
}
#[async_trait]
impl crate::crawler::daemon::ReconcilePass for RealReconcilePass {
async fn run(&self) -> anyhow::Result<crate::crawler::reconcile::ReconcileStats> {
let result = crate::crawler::reconcile::reconcile_missing(
&self.browser_manager,
&self.db,
&self.rate,
&self.start_url,
Some(&self.status),
self.tor.as_deref(),
)
.await;
if let Err(e) = &result {
if crate::crawler::nav::anyhow_looks_browser_dead(e) {
self.browser_manager.invalidate().await;
}
}
result
}
}
struct RealChapterDispatcher {
browser_manager: Arc<BrowserManager>,
db: PgPool,
@@ -778,6 +970,8 @@ struct RealChapterDispatcher {
rate: Arc<HostRateLimiters>,
download_allowlist: DownloadAllowlist,
max_image_bytes: usize,
/// Per-chapter image-count cap (see `CrawlerConfig::max_images_per_chapter`).
max_images_per_chapter: usize,
/// Enqueue `analyze_page` jobs for freshly-crawled pages. Shared gate
/// (read live) so toggling analysis at runtime takes effect without a
/// crawler respawn. Mirrors the analysis enable setting.
@@ -822,7 +1016,17 @@ impl ChapterDispatcher for RealChapterDispatcher {
pages_done: 0,
pages_total: None,
});
let lease = self.browser_manager.acquire().await?;
let lease = match self.browser_manager.acquire().await {
Ok(l) => l,
Err(e) => {
// Browser down / mid-restart: defer the job WITHOUT
// burning an attempt (the daemon releases it back to
// pending) rather than counting an infrastructure
// outage as a per-job failure.
tracing::warn!(error = ?e, "dispatch: browser unavailable — deferring job");
return Ok(SyncOutcome::BrowserUnavailable);
}
};
let result = content::sync_chapter_content(
&lease,
&self.db,
@@ -835,6 +1039,7 @@ impl ChapterDispatcher for RealChapterDispatcher {
false,
&self.download_allowlist,
self.max_image_bytes,
self.max_images_per_chapter,
self.tor.as_deref(),
Some(&self.status),
self.analysis_enabled.load(Ordering::Relaxed),
@@ -873,9 +1078,88 @@ impl ChapterDispatcher for RealChapterDispatcher {
}
}
}
// Other payload kinds aren't dispatched by this daemon yet —
// SyncManga / SyncChapterList are handled inline by the cron's
// metadata pass.
// Reconcile-enqueued manga-detail sync: fetch the detail page,
// upsert metadata, sync chapters — the identical per-ref work the
// cron metadata pass runs inline, via the shared
// `pipeline::process_manga_ref`.
JobPayload::SyncManga {
source_id: _,
source_manga_key,
url,
title,
} => {
let source = crate::crawler::source::target::TargetSource::new(url.clone());
let r = crate::crawler::source::SourceMangaRef {
source_manga_key,
title,
url,
};
// Scope the lease so it (and the borrowing FetchContext) drop
// before any browser-restart handling in the match below.
let result = {
let lease = match self.browser_manager.acquire().await {
Ok(l) => l,
Err(e) => {
// See the SyncChapterContent arm: defer without
// burning an attempt when the browser is unavailable.
tracing::warn!(error = ?e, "dispatch: browser unavailable — deferring job");
return Ok(SyncOutcome::BrowserUnavailable);
}
};
let ctx = crate::crawler::source::FetchContext {
browser: &lease,
rate: &self.rate,
tor: self.tor.as_deref(),
};
pipeline::process_manga_ref(
&ctx,
&source,
&self.db,
self.storage.as_ref(),
&self.http,
&self.rate,
&r,
false, // chapters ON — we want chapter rows synced
&self.download_allowlist,
self.max_image_bytes,
Some(&self.status),
)
.await
};
match result {
Ok(p) => {
self.transient_failures.store(0, Ordering::Release);
tracing::info!(
manga_id = %p.manga_id,
key = %r.source_manga_key,
"SyncManga: manga synced"
);
Ok(SyncOutcome::Fetched { pages: 0 })
}
Err(pipeline::RefError::Fetch(e)) | Err(pipeline::RefError::Skip(e)) => {
let streak = self.transient_failures.fetch_add(1, Ordering::AcqRel) + 1;
if crate::crawler::nav::anyhow_looks_browser_dead(&e) {
self.browser_manager.invalidate().await;
self.transient_failures.store(0, Ordering::Release);
} else if self.restart_threshold > 0 && streak >= self.restart_threshold {
tracing::warn!(
streak,
threshold = self.restart_threshold,
"auto browser restart: consecutive transient sync_manga failures"
);
let _ = self
.browser_manager
.coordinated_restart(self.drain_deadline)
.await;
self.transient_failures.store(0, Ordering::Release);
}
Err(e)
}
}
}
// Other payload kinds aren't dispatched by this daemon —
// SyncChapterList is handled inline by the cron's metadata pass;
// analyze_page is owned by the analysis daemon.
_ => Ok(SyncOutcome::Skipped),
}
}
@@ -910,10 +1194,36 @@ const ADMIN_PATH_PREFIX: &str = "/api/v1/admin/";
/// from a malicious page — this middleware rejects such requests by
/// comparing the request's `Origin` (with `Referer` as fallback) against
/// the configured allowlist. Safe methods (`GET`/`HEAD`/`OPTIONS`) are
/// always allowed. Requests with neither `Origin` nor `Referer` are
/// allowed (non-browser callers like curl can't be a CSRF vector). When
/// the allowlist is empty the check is skipped entirely (operator
/// opt-out — documented in `.env.example`).
/// always allowed.
///
/// **Bearer-token-only requests** (`Authorization: Bearer …` with NO
/// session cookie) are bot API callers — they can't be a CSRF vector
/// because the browser never attaches the Authorization header
/// automatically. Skip the check for them.
///
/// **Bearer + cookie (the "cookie-ride")** is treated as cookie-auth.
/// An attacker page can mint `Authorization: Bearer junk` on a
/// credentialed cross-site POST; the cookie carries the actual
/// authority. Closing this hole means cookie precedence: as soon as a
/// session cookie is present, the CSRF gate fires regardless of the
/// Authorization header. (0.87.10 closed a 0.87.2 regression here.)
///
/// **Cookie-auth requests** must:
/// * be from an allowed origin (`Origin` then `Referer`), AND
/// * have at least one of those headers present (a browser always
/// sends one on a cross-site POST; a missing pair on a cookie-auth
/// request is the exact niche an extension / no-referrer-policy
/// CSRF would try to exploit).
///
/// When the allowlist is empty we fail-closed for cookie-auth requests
/// — this used to silently let everything through, so an operator who
/// forgot to set `ADMIN_ALLOWED_ORIGINS` shipped an unguarded admin
/// surface. Operators on a pure-bot-token deploy aren't impacted (their
/// requests carry only `Authorization: Bearer …` with no session cookie
/// and skip the gate via the bearer-only branch above).
///
/// **No auth at all** (no cookie, no bearer): bypass the CSRF gate so
/// the auth extractor returns a clean 401 instead of a confusing 403.
async fn admin_csrf_guard(
State(state): State<AppState>,
req: Request,
@@ -928,16 +1238,74 @@ async fn admin_csrf_guard(
) {
return Ok(next.run(req).await);
}
if state.admin_allowed_origins.is_empty() {
let headers = req.headers();
// Detect both auth modes BEFORE choosing a bypass. The previous
// version short-circuited on "is_bearer" regardless of cookie state,
// which let an attacker page do a credentialed cross-site POST with
// a forged `Authorization: Bearer junk` header: the header existed,
// CSRF bypassed, then the auth extractor authenticated the victim
// via the session cookie. Cookie precedence here re-anchors the
// gate to the actual authority being ridden.
// Drive the parse off the auth-module constant so a rename of
// `SESSION_COOKIE_NAME` propagates here instead of silently
// reopening the cookie-ride. Split on `=` (not a prefix match) so
// an attacker can't shadow with `mangalord_session_x=` etc.
let has_session_cookie = headers
.get(axum::http::header::COOKIE)
.and_then(|v| v.to_str().ok())
.is_some_and(|raw| {
raw.split(';').any(|p| {
p.trim_start().split('=').next()
== Some(crate::auth::extractor::SESSION_COOKIE_NAME)
})
});
// Bearer-token-only requests are bot callers and bypass the gate —
// a browser can't set Authorization on a cross-site POST. But ONLY
// when no session cookie is also attached; with both present we
// must treat the request as cookie-auth to defeat the cookie-ride.
let is_bearer = headers
.get(axum::http::header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.is_some_and(|v| v.trim_start().to_ascii_lowercase().starts_with("bearer "));
if is_bearer && !has_session_cookie {
return Ok(next.run(req).await);
}
// No auth context at all → let the auth extractor return a clean 401
// ("log in first"). A no-auth request can't be a CSRF vector — there's
// no authority to ride. Without this, an anonymous curl POST to an
// admin endpoint surfaces 403 with our CSRF message instead of the
// expected 401, which is confusing for both operators and tests.
if !has_session_cookie {
return Ok(next.run(req).await);
}
let headers = req.headers();
let origin = headers.get("origin").and_then(|v| v.to_str().ok());
let referer = headers.get("referer").and_then(|v| v.to_str().ok());
// No Origin AND no Referer → server-to-server / curl / extension.
// Browsers always send one or the other on a cross-site POST.
let Some(candidate) = origin.or(referer) else {
return Ok(next.run(req).await);
let candidate = origin.or(referer);
if state.admin_allowed_origins.is_empty() {
// Fail-closed: cookie-auth admin mutations without an allowlist
// configuration are refused. Set `ADMIN_ALLOWED_ORIGINS` for the
// browser deployment, or call with `Authorization: Bearer …`.
tracing::warn!(
path = %req.uri().path(),
"admin CSRF: ADMIN_ALLOWED_ORIGINS is empty — cookie-auth admin mutations are refused (set the env var for browser-exposed deploys)"
);
return Err(AppError::Forbidden);
}
// Cookie-auth path with an allowlist configured: a browser always
// sends Origin or Referer on a cross-site POST. A missing pair is
// either a no-referrer-policy edge case or a tool deliberately
// hiding origin — refuse rather than risk it.
let Some(candidate) = candidate else {
tracing::warn!(
path = %req.uri().path(),
"admin CSRF: cookie-auth request with neither Origin nor Referer — refusing"
);
return Err(AppError::Forbidden);
};
if origin_in_allowlist(candidate, &state.admin_allowed_origins) {
return Ok(next.run(req).await);
@@ -971,8 +1339,8 @@ fn parse_origin(raw: &str) -> Option<String> {
let host = url.host_str()?;
let scheme = url.scheme();
let port_str = match (url.port(), scheme) {
(Some(p), "http") if p == 80 => String::new(),
(Some(p), "https") if p == 443 => String::new(),
(Some(80), "http") => String::new(),
(Some(443), "https") => String::new(),
(Some(p), _) => format!(":{p}"),
(None, _) => String::new(),
};
@@ -1149,4 +1517,108 @@ mod tests {
assert!(resp.headers().get("access-control-allow-origin").is_none());
assert!(resp.headers().get("access-control-allow-credentials").is_none());
}
/// `spawn_analysis_daemon` runs `crawler::jobs::reclaim_orphaned` at
/// startup so an analysis-only deploy refunds expired
/// `analyze_page` leases that a crashed previous run left running.
/// Without this, the call could be silently removed and the
/// reclaim test in `tests/crawler_jobs.rs` would still pass
/// (it covers the helper, not the wiring).
///
/// Seed an expired-lease `analyze_page` row in `crawler_jobs`, call
/// `spawn_analysis_daemon`, then assert the row went back to
/// `pending` with the attempt refunded.
#[sqlx::test(migrations = "./migrations")]
async fn spawn_analysis_daemon_reclaims_orphaned_analyze_leases(pool: PgPool) {
use crate::storage::LocalStorage;
use std::time::Duration;
use uuid::Uuid;
// Seed a chapter + page so the worker has something it COULD lease.
// We don't need the worker to actually run — we're testing the
// reclaim that happens before workers start.
let manga_id: Uuid =
sqlx::query_scalar("INSERT INTO mangas (title) VALUES ('M') RETURNING id")
.fetch_one(&pool)
.await
.unwrap();
let chapter_id: Uuid = sqlx::query_scalar(
"INSERT INTO chapters (manga_id, number) VALUES ($1, 1) RETURNING id",
)
.bind(manga_id)
.fetch_one(&pool)
.await
.unwrap();
let page_id: Uuid = sqlx::query_scalar(
"INSERT INTO pages (chapter_id, page_number, storage_key, content_type) \
VALUES ($1, 1, 'k/1.png', 'image/png') RETURNING id",
)
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
// Plant an expired-lease analyze_page row that mimics a previous
// worker that crashed mid-dispatch (attempts=1, leased_until in
// the past, state=running).
let payload = serde_json::json!({
"kind": "analyze_page",
"page_id": page_id,
"force": false
});
sqlx::query(
"INSERT INTO crawler_jobs (payload, state, attempts, leased_until) \
VALUES ($1, 'running', 1, now() - interval '1 hour')",
)
.bind(payload)
.execute(&pool)
.await
.unwrap();
// Bind to a local so the TempDir lives for the rest of the test.
// `tempfile::tempdir().unwrap().path()` would drop the TempDir
// at end-of-expression and `LocalStorage` would hold a path to
// a deleted directory.
let storage_dir = tempfile::tempdir().unwrap();
let storage: Arc<dyn Storage> =
Arc::new(LocalStorage::new(storage_dir.path()));
// The worker always runs OCR now (vision is dormant — see
// `effective_backend`), and the `.rten` models aren't shipped to unit
// CI. Point the engine at a path that can't exist so the *engine build*
// fails deterministically. Reclaim runs at the very top of
// `spawn_analysis_daemon`, before — and independently of — engine
// readiness, so the row must still be reclaimed even though spawn
// returns Err. That's exactly the regression this test guards (an
// analysis-only deploy must reclaim orphaned leases at startup).
let cfg = crate::config::AnalysisConfig {
ocr_detection_model: "/nonexistent/text-detection.rten".to_string(),
ocr_recognition_model: "/nonexistent/text-recognition.rten".to_string(),
workers: 1,
job_timeout: Duration::from_secs(1),
..Default::default()
};
let events = Arc::new(crate::analysis::events::AnalysisEvents::new());
let spawned = spawn_analysis_daemon(pool.clone(), storage, &cfg, events).await;
assert!(
spawned.is_err(),
"engine build must fail with a missing model path"
);
// The reclaim must have moved the row back to pending with the
// attempt refunded (attempts goes from 1 → 0). Reaching it on the
// initial running state would mean reclaim never ran.
let (state, attempts): (String, i32) = sqlx::query_as(
"SELECT state, attempts FROM crawler_jobs WHERE payload->>'page_id' = $1",
)
.bind(page_id.to_string())
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
state, "pending",
"expired-lease analyze_page row must be reclaimed to pending"
);
assert_eq!(attempts, 0, "reclaim_orphaned refunds the attempt");
}
}

View File

@@ -70,6 +70,45 @@ impl FromRequestParts<AppState> for CurrentUser {
}
}
/// Client IP for per-IP auth rate limiting. Resolves to the first hop of
/// `X-Forwarded-For` **only** when [`crate::config::AuthConfig::trusted_proxy`]
/// is set (the backend is behind a proxy that overrides the header — the
/// compose deploy). Otherwise `None`, so the limiter uses its shared bucket.
/// Never fails: a missing or malformed header simply yields `None`.
pub struct ClientIp(pub Option<std::net::IpAddr>);
/// Parse the client IP from an `X-Forwarded-For` value: the left-most hop is
/// the original client (later hops are intermediary proxies). Factored out so
/// the parsing is unit-testable without constructing a request.
pub(crate) fn first_forwarded_ip(header: &str) -> Option<std::net::IpAddr> {
header
.split(',')
.next()
.map(str::trim)
.filter(|s| !s.is_empty())
.and_then(|s| s.parse::<std::net::IpAddr>().ok())
}
#[async_trait]
impl FromRequestParts<AppState> for ClientIp {
type Rejection = std::convert::Infallible;
async fn from_request_parts(
parts: &mut Parts,
state: &AppState,
) -> Result<Self, Self::Rejection> {
if !state.auth.trusted_proxy {
return Ok(ClientIp(None));
}
let ip = parts
.headers
.get("x-forwarded-for")
.and_then(|v| v.to_str().ok())
.and_then(first_forwarded_ip);
Ok(ClientIp(ip))
}
}
/// Cookie-only authentication. Bot/API tokens are explicitly NOT accepted
/// here — this is the substrate for [`RequireAdmin`] and exists precisely
/// to keep admin authority out of bearer-token reach.
@@ -120,3 +159,34 @@ impl FromRequestParts<AppState> for RequireAdmin {
Ok(RequireAdmin(user))
}
}
#[cfg(test)]
mod tests {
use super::first_forwarded_ip;
use std::net::IpAddr;
#[test]
fn parses_left_most_client_hop() {
// The client is the first entry; later entries are intermediary proxies.
assert_eq!(
first_forwarded_ip("203.0.113.5, 10.0.0.1, 10.0.0.2"),
Some("203.0.113.5".parse::<IpAddr>().unwrap())
);
assert_eq!(
first_forwarded_ip(" 198.51.100.9 "),
Some("198.51.100.9".parse::<IpAddr>().unwrap())
);
assert_eq!(
first_forwarded_ip("2001:db8::1, 10.0.0.1"),
Some("2001:db8::1".parse::<IpAddr>().unwrap())
);
}
#[test]
fn rejects_empty_or_garbage() {
assert_eq!(first_forwarded_ip(""), None);
assert_eq!(first_forwarded_ip(" "), None);
assert_eq!(first_forwarded_ip("not-an-ip"), None);
assert_eq!(first_forwarded_ip(", 10.0.0.1"), None);
}
}

View File

@@ -28,6 +28,25 @@ pub fn verify_password(plain: &str, phc: &str) -> bool {
.is_ok()
}
/// Async wrapper around [`hash_password`] that offloads the CPU- and
/// memory-heavy Argon2 work to the blocking pool. Async handlers must call
/// this rather than the sync primitive: at ~15-50 ms per hash, running it
/// inline stalls every other task sharing the runtime worker thread.
pub async fn hash_password_async(plain: String) -> AppResult<String> {
tokio::task::spawn_blocking(move || hash_password(&plain))
.await
.map_err(|e| AppError::Other(anyhow::anyhow!("password hash task join: {e}")))?
}
/// Async wrapper around [`verify_password`]. A join failure (blocking pool
/// gone at shutdown, panic) resolves to `false` — the caller only ever uses
/// the result as a boolean gate, and denying auth is the safe default.
pub async fn verify_password_async(plain: String, phc: String) -> bool {
tokio::task::spawn_blocking(move || verify_password(&plain, &phc))
.await
.unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -56,4 +75,24 @@ mod tests {
let b = hash_password("same").unwrap();
assert_ne!(a, b, "two hashes of the same password must differ (salt)");
}
#[tokio::test]
async fn async_hash_then_verify_roundtrip() {
let phc = hash_password_async("correct horse battery staple".to_string())
.await
.unwrap();
assert!(phc.starts_with("$argon2id$"));
assert!(verify_password_async("correct horse battery staple".to_string(), phc).await);
}
#[tokio::test]
async fn async_verify_rejects_wrong_password() {
let phc = hash_password_async("hunter2".to_string()).await.unwrap();
assert!(!verify_password_async("hunter3".to_string(), phc).await);
}
#[tokio::test]
async fn async_verify_rejects_malformed_hash() {
assert!(!verify_password_async("anything".to_string(), "not a real phc".to_string()).await);
}
}

View File

@@ -15,9 +15,17 @@
//! tests run in isolated buckets and won't bleed across `#[sqlx::test]`
//! cases that share a process.
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::Mutex;
use std::time::Instant;
/// Upper bound on distinct client IPs tracked at once, so a spray from many
/// spoofed/rotating source IPs can't grow the map without limit. When full,
/// idle (refilled-to-burst) buckets are pruned first; if none are idle a new
/// IP falls back to the shared global bucket for that request.
const MAX_TRACKED_IPS: usize = 10_000;
/// Tunable limits. `per_sec == 0` disables the limiter — used by the
/// test harness and by anyone who wants to opt out via env config.
#[derive(Clone, Copy, Debug)]
@@ -50,6 +58,42 @@ struct Bucket {
last_refill: Instant,
}
impl Bucket {
fn new(burst: u32) -> Self {
Self {
tokens: f64::from(burst),
last_refill: Instant::now(),
}
}
/// Refill by elapsed time then try to consume one token.
fn try_take(&mut self, cfg: &RateLimitConfig, now: Instant) -> AcquireResult {
let elapsed = now.duration_since(self.last_refill).as_secs_f64();
self.tokens =
(self.tokens + elapsed * f64::from(cfg.per_sec)).min(f64::from(cfg.burst));
self.last_refill = now;
if self.tokens >= 1.0 {
self.tokens -= 1.0;
AcquireResult::Allowed
} else {
let deficit = 1.0 - self.tokens;
let wait_secs = (deficit / f64::from(cfg.per_sec)).ceil() as u64;
AcquireResult::Denied {
retry_after_secs: wait_secs.max(1),
}
}
}
/// Whether this bucket has fully refilled (i.e. the client has been idle).
/// Such buckets carry no state worth keeping — dropping and lazily
/// recreating one yields an identical full bucket — so they're the safe
/// eviction target under memory pressure.
fn is_idle(&self, cfg: &RateLimitConfig, now: Instant) -> bool {
let elapsed = now.duration_since(self.last_refill).as_secs_f64();
(self.tokens + elapsed * f64::from(cfg.per_sec)) >= f64::from(cfg.burst)
}
}
/// Outcome of [`AuthRateLimiter::try_acquire`]. When `Denied`, the
/// caller can use `retry_after_secs` for a `Retry-After: N` header
/// (RFC 6585 §4) so well-behaved clients back off correctly rather
@@ -60,51 +104,64 @@ pub enum AcquireResult {
Denied { retry_after_secs: u64 },
}
/// Single-bucket token-bucket limiter. `try_acquire` is cheap (one
/// mutex acquire, no allocations) so the auth path doesn't pay a real
/// cost for the check.
/// Token-bucket limiter keyed by client IP, with a shared global bucket for
/// requests whose IP is unknown (no trusted proxy).
///
/// The old design was a single global bucket. That let one attacker at the
/// sustained rate deny auth to *every* user (the bucket stayed drained). With
/// the SvelteKit proxy now forwarding the peer IP (`X-Forwarded-For`, honoured
/// only when `AUTH_TRUSTED_PROXY` is set), each source IP gets its own bucket,
/// so an attacker only throttles themselves. When no trustworthy IP is
/// available the caller passes `None` and the shared `global` bucket applies —
/// exactly the previous behaviour.
pub struct AuthRateLimiter {
cfg: RateLimitConfig,
bucket: Mutex<Bucket>,
global: Mutex<Bucket>,
per_ip: Mutex<HashMap<IpAddr, Bucket>>,
}
impl AuthRateLimiter {
pub fn new(cfg: RateLimitConfig) -> Self {
Self {
cfg,
bucket: Mutex::new(Bucket {
tokens: cfg.burst as f64,
last_refill: Instant::now(),
}),
global: Mutex::new(Bucket::new(cfg.burst)),
per_ip: Mutex::new(HashMap::new()),
}
}
/// Consume one token if available. Returns `Denied` with a
/// rounded-up seconds-until-refill so the caller can emit a
/// `Retry-After` header.
pub fn try_acquire(&self) -> AcquireResult {
/// Consume one token from the bucket for `key` (per-IP when `Some`, the
/// shared global bucket when `None`). Returns `Denied` with a rounded-up
/// seconds-until-refill so the caller can emit a `Retry-After` header.
pub fn try_acquire(&self, key: Option<IpAddr>) -> AcquireResult {
if self.cfg.per_sec == 0 {
return AcquireResult::Allowed;
}
let now = Instant::now();
let mut bucket = self.bucket.lock().expect("rate limiter mutex poisoned");
let elapsed = now.duration_since(bucket.last_refill).as_secs_f64();
bucket.tokens =
(bucket.tokens + elapsed * f64::from(self.cfg.per_sec)).min(f64::from(self.cfg.burst));
bucket.last_refill = now;
if bucket.tokens >= 1.0 {
bucket.tokens -= 1.0;
AcquireResult::Allowed
} else {
// ceil((1 - tokens) / per_sec), minimum 1 — a `Retry-After: 0`
// would tell clients to retry immediately, which is what we're
// actively trying to discourage.
let deficit = 1.0 - bucket.tokens;
let wait_secs = (deficit / f64::from(self.cfg.per_sec)).ceil() as u64;
AcquireResult::Denied {
retry_after_secs: wait_secs.max(1),
let Some(ip) = key else {
return self
.global
.lock()
.unwrap_or_else(|e| e.into_inner())
.try_take(&self.cfg, now);
};
let mut map = self.per_ip.lock().unwrap_or_else(|e| e.into_inner());
if map.len() >= MAX_TRACKED_IPS && !map.contains_key(&ip) {
map.retain(|_, b| !b.is_idle(&self.cfg, now));
if map.len() >= MAX_TRACKED_IPS {
// Still saturated with active attackers — degrade to the shared
// bucket rather than grow unbounded. Worst case is the old
// global-bucket behaviour under an extreme distributed flood.
drop(map);
return self
.global
.lock()
.unwrap_or_else(|e| e.into_inner())
.try_take(&self.cfg, now);
}
}
map.entry(ip)
.or_insert_with(|| Bucket::new(self.cfg.burst))
.try_take(&self.cfg, now)
}
}
@@ -119,7 +176,7 @@ mod tests {
burst: 0,
});
for _ in 0..1000 {
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
}
}
@@ -130,10 +187,10 @@ mod tests {
per_sec: 1,
burst: 3,
});
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
match rl.try_acquire() {
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
match rl.try_acquire(None) {
AcquireResult::Denied { retry_after_secs } => {
// Bucket is at ~0 tokens, refill rate 1/sec → ~1s wait.
assert!(
@@ -152,11 +209,11 @@ mod tests {
per_sec: 10,
burst: 1,
});
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(), AcquireResult::Denied { .. }));
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(None), AcquireResult::Denied { .. }));
std::thread::sleep(std::time::Duration::from_millis(150));
assert_eq!(
rl.try_acquire(),
rl.try_acquire(None),
AcquireResult::Allowed,
"token should have refilled"
);
@@ -170,10 +227,95 @@ mod tests {
per_sec: 1,
burst: 1,
});
slow.try_acquire();
match slow.try_acquire() {
slow.try_acquire(None);
match slow.try_acquire(None) {
AcquireResult::Denied { retry_after_secs } => assert_eq!(retry_after_secs, 1),
_ => panic!("expected Denied"),
}
}
fn ip(s: &str) -> Option<IpAddr> {
Some(s.parse().unwrap())
}
#[test]
fn per_ip_buckets_are_independent() {
// The whole point of the fix: one IP draining its bucket must not deny
// a different IP.
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 2,
});
let a = ip("203.0.113.7");
let b = ip("198.51.100.9");
assert_eq!(rl.try_acquire(a), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(a), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(a), AcquireResult::Denied { .. }));
// b is untouched.
assert_eq!(rl.try_acquire(b), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(b), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(b), AcquireResult::Denied { .. }));
}
#[test]
fn none_key_shares_the_global_bucket_independent_of_per_ip() {
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 2,
});
// Drain the global (None) bucket.
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(None), AcquireResult::Denied { .. }));
// A real IP is on its own bucket, unaffected by the drained global one.
assert_eq!(rl.try_acquire(ip("203.0.113.1")), AcquireResult::Allowed);
}
#[test]
fn tracked_ip_map_is_bounded() {
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 1,
});
// Distinct IPs beyond the cap must not grow the map without bound —
// excess requests fall back to the shared bucket instead.
for i in 0..(MAX_TRACKED_IPS as u64 + 50) {
let octet_a = (i >> 8) as u8;
let octet_b = (i & 0xff) as u8;
let addr = format!("10.20.{octet_a}.{octet_b}");
let _ = rl.try_acquire(Some(addr.parse().unwrap()));
}
assert!(rl.per_ip.lock().unwrap().len() <= MAX_TRACKED_IPS);
}
#[test]
fn survives_a_poisoned_mutex() {
// If any thread ever panics while holding a limiter mutex, the lock
// becomes poisoned. With the old `.expect(...)` every later auth request
// would re-panic — one blip turned into a permanent auth outage. Recover
// the guard via `into_inner()` instead so the limiter keeps serving.
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 5,
burst: 5,
});
// Poison per_ip by panicking while holding its guard.
let poisoned = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let _g = rl.per_ip.lock().unwrap();
panic!("poison the per-IP mutex");
}));
assert!(poisoned.is_err(), "the panic must unwind");
assert!(rl.per_ip.is_poisoned(), "the mutex must now be poisoned");
// The per-IP path (Some(ip)) must still work despite the poison.
assert_eq!(rl.try_acquire(ip("198.51.100.9")), AcquireResult::Allowed);
// And poison the global bucket too — the None-key path must recover.
let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let _g = rl.global.lock().unwrap();
panic!("poison the global mutex");
}));
assert!(rl.global.is_poisoned());
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
}
}

View File

@@ -3,15 +3,16 @@
//! `generate_token` draws 32 bytes from the OS CSPRNG, encodes them as
//! URL-safe base64 (no padding), and returns the raw string alongside its
//! SHA-256 hash. Storage holds only the hash; the raw value lives in the
//! cookie or `Authorization` header. Comparison goes through
//! `constant_time_eq` to keep timing side channels off the table.
//! cookie or `Authorization` header. Token lookup is an indexed equality on
//! that 256-bit hash in the database (`WHERE token_hash = $1`), so there's no
//! in-process secret comparison to time-attack: a guess has to match a full
//! SHA-256 digest, and the DB index reveals nothing about how close it came.
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine as _;
use rand::rngs::OsRng;
use rand::RngCore;
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq;
pub const TOKEN_BYTES: usize = 32;
pub const HASH_BYTES: usize = 32;
@@ -30,10 +31,6 @@ pub fn hash_token(raw: &str) -> [u8; HASH_BYTES] {
hasher.finalize().into()
}
pub fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
a.ct_eq(b).into()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -58,11 +55,4 @@ mod tests {
assert_eq!(hash_token("abc"), hash_token("abc"));
assert_ne!(hash_token("abc"), hash_token("abd"));
}
#[test]
fn constant_time_eq_compares_correctly() {
assert!(constant_time_eq(b"abc", b"abc"));
assert!(!constant_time_eq(b"abc", b"abd"));
assert!(!constant_time_eq(b"abc", b"abcd"));
}
}

View File

@@ -112,9 +112,20 @@ async fn main() -> anyhow::Result<()> {
cookie_jar.add_cookie_str(&cookie_str, &seed_url);
tracing::info!(domain, "seeded PHPSESSID into reqwest cookie jar");
}
// SSRF defence: only download from the catalog host + CDN host (plus
// optional CRAWLER_DOWNLOAD_ALLOWLIST extras). Built here so the same
// allowlist guards both the redirect policy and the per-image check.
let allowlist = Arc::new(build_download_allowlist(&start_url, cdn_host.as_deref()));
let mut http_builder = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.no_proxy()
// Re-validate every redirect hop against the allowlist — reqwest's
// default follows up to 10 redirects and `is_safe_url` only guards
// the initial URL, so a 302 to a private IP would otherwise pivot
// inside the deployment (SSRF).
.redirect(mangalord::crawler::safety::safe_redirect_policy(
(*allowlist).clone(),
))
.cookie_provider(cookie_jar);
if let Some(ua) = &user_agent {
http_builder = http_builder.user_agent(ua);
@@ -123,8 +134,25 @@ async fn main() -> anyhow::Result<()> {
http_builder = http_builder
.proxy(reqwest::Proxy::all(proxy).with_context(|| format!("parse proxy URL: {proxy}"))?);
}
// DNS-rebinding guard: attached on the direct path AND on http(s) proxies
// (reqwest resolves the target itself there), skipped only for SOCKS proxies
// where the proxy resolves the target. Use the shared predicate so this CLI
// and the daemon (app.rs) stay in lockstep — previously the CLI attached the
// resolver only on the fully-direct path, so an http(s) proxy silently lost
// the rebinding guard.
if mangalord::crawler::safety::should_attach_safe_resolver(proxy_url.as_deref()) {
http_builder =
http_builder.dns_resolver(mangalord::crawler::safety::safe_dns_resolver());
}
let http = http_builder.build().context("build http client")?;
// Opt-in browser SSRF interception (default off), mirroring the daemon.
mangalord::crawler::intercept::set_enabled(
std::env::var("CRAWLER_SSRF_INTERCEPT")
.map(|v| matches!(v.as_str(), "1" | "true" | "TRUE" | "yes"))
.unwrap_or(false),
);
let mut options = LaunchOptions::from_env();
if let Some(proxy) = &proxy_url {
let chromium_proxy = mangalord::crawler::url_utils::chromium_proxy_arg(proxy);
@@ -216,6 +244,7 @@ async fn main() -> anyhow::Result<()> {
rate_ms,
cdn_host.as_deref(),
cdn_rate_ms,
Arc::clone(&allowlist),
limit,
skip_chapters,
skip_chapter_content || !session_ready,
@@ -246,6 +275,7 @@ async fn run(
rate_ms: u64,
cdn_host: Option<&str>,
cdn_rate_ms: u64,
allowlist: Arc<mangalord::crawler::safety::DownloadAllowlist>,
limit: usize,
skip_chapters: bool,
skip_chapter_content: bool,
@@ -259,38 +289,18 @@ async fn run(
}
let rate = Arc::new(rate);
// SSRF defence: only download from the catalog host + CDN host
// (plus optional CRAWLER_DOWNLOAD_ALLOWLIST extras), and cap
// single-image downloads at CRAWLER_MAX_IMAGE_BYTES bytes.
// CRAWLER_ALLOW_ANY_HOST=true short-circuits the host check for
// sharded-CDN sources; private-IP and scheme guards still apply.
let allowlist = if env_bool("CRAWLER_ALLOW_ANY_HOST", false) {
mangalord::crawler::safety::DownloadAllowlist::allow_any()
} else {
let mut allow = mangalord::crawler::safety::DownloadAllowlist::new();
if let Ok(parsed) = reqwest::Url::parse(start_url) {
if let Some(h) = parsed.host_str() {
allow = allow.allow(h);
}
}
if let Some(host) = cdn_host {
allow = allow.allow(host);
}
if let Ok(extras) = std::env::var("CRAWLER_DOWNLOAD_ALLOWLIST") {
for piece in extras.split(',') {
let trimmed = piece.trim();
if !trimmed.is_empty() {
allow = allow.allow(trimmed);
}
}
}
allow
};
// Per-image download cap (the allowlist is built in `main` and passed in
// so the HTTP client's redirect policy and this check share one source).
let max_image_bytes: usize = std::env::var("CRAWLER_MAX_IMAGE_BYTES")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(mangalord::crawler::safety::DEFAULT_MAX_IMAGE_BYTES);
let allowlist = Arc::new(allowlist);
// Per-chapter image *count* cap — bounds total disk against a hostile
// reader page listing thousands of <img> tags. `0` disables it.
let max_images_per_chapter: usize = std::env::var("CRAWLER_MAX_IMAGES_PER_CHAPTER")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(2000);
let stats = pipeline::run_metadata_pass(
manager.as_ref(),
@@ -325,6 +335,7 @@ async fn run(
force_refetch_chapters,
Arc::clone(&allowlist),
max_image_bytes,
max_images_per_chapter,
tor.clone(),
)
.await?;
@@ -351,6 +362,7 @@ async fn sync_bookmarked_chapter_content(
force_refetch: bool,
allowlist: Arc<mangalord::crawler::safety::DownloadAllowlist>,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<Arc<mangalord::crawler::tor::TorController>>,
) -> anyhow::Result<()> {
let pending: Vec<(Uuid, Uuid, String)> = sqlx::query_as(
@@ -416,6 +428,7 @@ async fn sync_bookmarked_chapter_content(
force_refetch,
allowlist.as_ref(),
max_image_bytes,
max_images_per_chapter,
tor.as_deref(),
// CLI one-shot — no live status surface.
None,
@@ -431,6 +444,13 @@ async fn sync_bookmarked_chapter_content(
s.fetched += 1;
}
Ok(SyncOutcome::Skipped) => s.skipped += 1,
// Unreachable in the one-shot CLI (it holds its own lease
// and never dispatches through the queue), but count it as a
// failure for exhaustiveness.
Ok(SyncOutcome::BrowserUnavailable) => {
tracing::warn!(%chapter_id, "crawler browser unavailable");
s.failed += 1;
}
Ok(SyncOutcome::SessionExpired) => {
tracing::error!(
%chapter_id,
@@ -497,3 +517,37 @@ fn env_bool(name: &str, default: bool) -> bool {
}
}
/// Build the crawler download allowlist from env + the catalog/CDN hosts.
/// Shared by the HTTP client's redirect policy and the per-image safety
/// check so both agree on which hosts are reachable.
///
/// `CRAWLER_ALLOW_ANY_HOST=true` short-circuits the host check for
/// sharded-CDN sources; private-IP and scheme guards still apply.
fn build_download_allowlist(
start_url: &str,
cdn_host: Option<&str>,
) -> mangalord::crawler::safety::DownloadAllowlist {
use mangalord::crawler::safety::DownloadAllowlist;
if env_bool("CRAWLER_ALLOW_ANY_HOST", false) {
return DownloadAllowlist::allow_any();
}
let mut allow = DownloadAllowlist::new();
if let Ok(parsed) = reqwest::Url::parse(start_url) {
if let Some(h) = parsed.host_str() {
allow = allow.allow(h);
}
}
if let Some(host) = cdn_host {
allow = allow.allow(host);
}
if let Ok(extras) = std::env::var("CRAWLER_DOWNLOAD_ALLOWLIST") {
for piece in extras.split(',') {
let trimmed = piece.trim();
if !trimmed.is_empty() {
allow = allow.allow(trimmed);
}
}
}
allow
}

View File

@@ -27,6 +27,14 @@ pub struct AuthConfig {
/// so a private instance is locked down with a single switch.
/// Defaults to `false` (current public behaviour).
pub private_mode: bool,
/// Whether to trust a proxy-supplied `X-Forwarded-For` header as the
/// client IP for per-IP auth rate limiting. Enable ONLY when the backend
/// sits behind a trusted reverse proxy that overrides the header (the
/// compose deploy: SvelteKit's hooks.server.ts sets it from the real peer
/// address). When `false` (default), the header is ignored and the auth
/// limiter uses a single shared bucket — a directly-exposed backend must
/// keep this off or clients could spoof their IP to dodge the limit.
pub trusted_proxy: bool,
}
impl Default for AuthConfig {
@@ -42,6 +50,7 @@ impl Default for AuthConfig {
rate_limit: crate::auth::rate_limit::RateLimitConfig::default(),
allow_self_register: true,
private_mode: false,
trusted_proxy: false,
}
}
}
@@ -55,6 +64,13 @@ pub struct UploadConfig {
/// reject a single oversized cover/page without failing the whole
/// request just because the total happens to fit.
pub max_file_bytes: usize,
/// Max page images accepted in one chapter upload. Bounds how many
/// parts the handler will stage before giving up, so a client can't
/// pin a worker streaming an unbounded page count. `0` disables THIS
/// cap — the total is then bounded only by `max_request_bytes` (the
/// whole-request body limit), which stays the backstop either way.
/// Defaults to 2000. `MAX_PAGES_PER_CHAPTER`.
pub max_pages_per_chapter: usize,
}
impl Default for UploadConfig {
@@ -62,6 +78,46 @@ impl Default for UploadConfig {
Self {
max_request_bytes: 200 * 1024 * 1024, // 200 MiB
max_file_bytes: 20 * 1024 * 1024, // 20 MiB
max_pages_per_chapter: 2000,
}
}
}
/// Postgres connection-pool sizing. One pool backs every HTTP handler plus
/// the crawler/analysis daemons, so it must be large enough not to starve
/// interactive reads and fail fast (rather than hang on the driver's silent
/// 30 s default) when genuinely saturated.
#[derive(Clone, Debug)]
pub struct DbConfig {
/// `DB_MAX_CONNECTIONS`. Upper bound on open connections.
pub max_connections: u32,
/// `DB_ACQUIRE_TIMEOUT_SECS`. How long a caller waits for a free
/// connection before erroring — short so overload surfaces as a fast
/// 500 instead of a 30 s hang.
pub acquire_timeout: Duration,
}
impl Default for DbConfig {
fn default() -> Self {
Self {
max_connections: 20,
acquire_timeout: Duration::from_secs(10),
}
}
}
impl DbConfig {
pub fn from_env() -> Self {
let default = Self::default();
Self {
// `.max(1)`: a zero-size pool can never hand out a connection and
// would deadlock every query — clamp to at least one.
max_connections: env_u64("DB_MAX_CONNECTIONS", default.max_connections.into())
.max(1) as u32,
acquire_timeout: Duration::from_secs(env_u64(
"DB_ACQUIRE_TIMEOUT_SECS",
default.acquire_timeout.as_secs(),
)),
}
}
}
@@ -113,6 +169,32 @@ impl ResponseFormat {
}
}
/// Which engine the analysis worker dispatches each page through. A
/// deploy-time choice (the engine is either installed or not), so it lives in
/// env only and is not part of the admin-editable `AnalysisSettings`.
///
/// * `Ocr` — in-process [`crate::analysis::ocr`] (the `ocrs` engine): fast,
/// CPU-only, English text. Writes OCR text only (no tags/scene/safety).
/// * `Vision` — the local OpenAI-compatible LLM in [`crate::analysis::vision`]:
/// full OCR + tags + scene + safety, but heavy.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum AnalysisBackend {
Ocr,
Vision,
}
impl AnalysisBackend {
/// Lenient env parse. Defaults to `Ocr` (the Pi-friendly path) for any
/// unset or unrecognized value; `vision` opts back into the LLM engine.
fn from_str(s: &str) -> AnalysisBackend {
match s.trim().to_lowercase().as_str() {
"vision" | "llm" => AnalysisBackend::Vision,
// Default (incl. "ocr", "ocrs", and anything unrecognized).
_ => AnalysisBackend::Ocr,
}
}
}
/// AI content-analysis worker configuration: the enable gate, the local
/// OpenAI-compatible vision endpoint, and the worker / request knobs.
#[derive(Clone, Debug)]
@@ -120,6 +202,16 @@ pub struct AnalysisConfig {
/// Master switch (`ANALYSIS_ENABLED`). When `false`, no analysis jobs
/// are enqueued and no worker runs. Defaults to `false`.
pub enabled: bool,
/// Which engine the worker dispatches through (`ANALYSIS_BACKEND`):
/// `ocr` (default, the in-process `ocrs` engine) or `vision` (the local
/// LLM). Deploy-time, env-only — see [`AnalysisBackend`].
pub backend: AnalysisBackend,
/// Path to the ocrs text-*detection* `.rten` model
/// (`OCRS_DETECTION_MODEL`). Only read when `backend == Ocr`.
pub ocr_detection_model: String,
/// Path to the ocrs text-*recognition* `.rten` model
/// (`OCRS_RECOGNITION_MODEL`). Only read when `backend == Ocr`.
pub ocr_recognition_model: String,
/// Number of concurrent analysis workers (`ANALYSIS_WORKERS`).
pub workers: usize,
/// OpenAI-compatible chat/completions URL (`ANALYSIS_VISION_URL`).
@@ -131,7 +223,7 @@ pub struct AnalysisConfig {
/// land a `failed` row. `None` (the default) disables the gate, matching
/// the prior behavior for always-on endpoints. Env-only, like `api_key`.
pub vision_health_url: Option<String>,
/// Model id to request (`ANALYSIS_MODEL`).
/// Model id to request (`ANALYSIS_VISION_MODEL`).
pub model: String,
/// Optional bearer token (`ANALYSIS_API_KEY`); local servers usually
/// don't need one.
@@ -166,6 +258,13 @@ pub struct AnalysisConfig {
/// Hard cap on a page image's stored size; larger pages are skipped
/// (`ANALYSIS_MAX_IMAGE_BYTES`).
pub max_image_bytes: usize,
/// Hard cap on a page image's **decoded** pixel count for the OCR backend
/// (`ANALYSIS_OCR_MAX_DECODE_PIXELS`). `max_image_bytes` only bounds the
/// *encoded* size; without a decode bound a tiny image declaring
/// 50000×50000 inflates to billions of bytes and OOM-kills the worker
/// (decompression bomb). Generous by default (100 MP) so legitimately
/// tall, un-sliced manga pages still decode.
pub ocr_max_decode_pixels: u64,
/// Output-constraint mode (`ANALYSIS_RESPONSE_FORMAT`):
/// `json_schema` (default) | `json_object` | `none`.
pub response_format: ResponseFormat,
@@ -195,6 +294,9 @@ impl Default for AnalysisConfig {
fn default() -> Self {
Self {
enabled: false,
backend: AnalysisBackend::Ocr,
ocr_detection_model: "/models/text-detection.rten".to_string(),
ocr_recognition_model: "/models/text-recognition.rten".to_string(),
workers: 1,
endpoint: "http://localhost:8000/v1/chat/completions".to_string(),
vision_health_url: None,
@@ -212,6 +314,7 @@ impl Default for AnalysisConfig {
tall_aspect_threshold: 1.6,
max_slices: 16,
max_image_bytes: 8 * 1024 * 1024,
ocr_max_decode_pixels: 100_000_000,
response_format: ResponseFormat::JsonSchema,
frequency_penalty: 0.3,
temperature: 0.0,
@@ -223,16 +326,51 @@ impl Default for AnalysisConfig {
}
impl AnalysisConfig {
/// The backend the worker actually dispatches through.
///
/// Vision is **temporarily disabled**: the engine code (`analysis::vision`,
/// `RealAnalyzeDispatcher`, the readiness probe) is kept intact but never
/// selected. Until it's re-enabled, this returns [`AnalysisBackend::Ocr`]
/// regardless of the parsed `backend`, logging a warning if `vision` was
/// requested so an env override isn't silently ignored. Re-enabling vision
/// is then a one-line change here (return `self.backend`).
pub fn effective_backend(&self) -> AnalysisBackend {
if self.backend == AnalysisBackend::Vision {
tracing::warn!(
"ANALYSIS_BACKEND=vision requested but the vision backend is temporarily \
disabled; running OCR instead"
);
}
AnalysisBackend::Ocr
}
pub fn from_env() -> Self {
let d = AnalysisConfig::default();
Self {
enabled: env_bool("ANALYSIS_ENABLED", d.enabled),
backend: std::env::var("ANALYSIS_BACKEND")
.map(|s| AnalysisBackend::from_str(&s))
.unwrap_or(d.backend),
ocr_detection_model: std::env::var("OCRS_DETECTION_MODEL")
.ok()
.filter(|s| !s.is_empty())
.unwrap_or(d.ocr_detection_model),
ocr_recognition_model: std::env::var("OCRS_RECOGNITION_MODEL")
.ok()
.filter(|s| !s.is_empty())
.unwrap_or(d.ocr_recognition_model),
workers: env_usize("ANALYSIS_WORKERS", d.workers).max(1),
endpoint: std::env::var("ANALYSIS_VISION_URL").unwrap_or(d.endpoint),
vision_health_url: std::env::var("ANALYSIS_VISION_HEALTH_URL")
.ok()
.filter(|s| !s.is_empty()),
model: std::env::var("ANALYSIS_MODEL").unwrap_or(d.model),
// Renamed from `ANALYSIS_MODEL` in 0.87.17 to match the
// `ANALYSIS_VISION_*` naming the docs, compose, and 0.87.12's
// regression test had already standardised on. Pre-0.87.17
// deploys reading the old `ANALYSIS_MODEL` will fall through
// to the default — explicitly noted so an operator hitting
// an empty `model` after upgrade can grep this file.
model: std::env::var("ANALYSIS_VISION_MODEL").unwrap_or(d.model),
api_key: std::env::var("ANALYSIS_API_KEY")
.ok()
.filter(|s| !s.is_empty()),
@@ -253,6 +391,10 @@ impl AnalysisConfig {
.max(1.0),
max_slices: env_usize("ANALYSIS_MAX_SLICES", d.max_slices).max(1),
max_image_bytes: env_usize("ANALYSIS_MAX_IMAGE_BYTES", d.max_image_bytes),
ocr_max_decode_pixels: env_u64(
"ANALYSIS_OCR_MAX_DECODE_PIXELS",
d.ocr_max_decode_pixels,
),
response_format: std::env::var("ANALYSIS_RESPONSE_FORMAT")
.map(|s| ResponseFormat::from_str(&s))
.unwrap_or(d.response_format),
@@ -279,6 +421,7 @@ pub struct Config {
pub database_url: String,
pub bind_address: String,
pub storage_dir: PathBuf,
pub db: DbConfig,
pub auth: AuthConfig,
pub upload: UploadConfig,
pub cors_allowed_origins: Vec<String>,
@@ -355,6 +498,13 @@ pub struct CrawlerConfig {
pub download_allowlist: DownloadAllowlist,
/// Hard upper bound on a single image download. Defaults to 32 MiB.
pub max_image_bytes: usize,
/// Hard upper bound on the number of page images in one chapter. A
/// hostile reader page could otherwise list thousands of `<img>`
/// tags; `max_image_bytes` caps each one but not the count, so the
/// product is an unbounded disk-fill. A chapter exceeding this is
/// acked failed rather than downloaded. `0` disables the cap.
/// Defaults to 2000. `CRAWLER_MAX_IMAGES_PER_CHAPTER`.
pub max_images_per_chapter: usize,
/// Max manga detail fetches per metadata pass. `0` means no cap
/// (full sweep up to the source's own bound). Sourced from
/// `CRAWLER_LIMIT`, mirroring the CLI binary.
@@ -372,6 +522,16 @@ pub struct CrawlerConfig {
/// exhausted) that trigger an automatic coordinated browser restart.
/// Defaults to 3. `CRAWLER_BROWSER_RESTART_THRESHOLD`.
pub browser_restart_threshold: u32,
/// CDP `Fetch` interception that re-validates every headless-browser
/// navigation/redirect/subresource against the SSRF check. Default `true`:
/// with it off, only the top-level URL string is validated, so a scraped
/// page's JS/subresources (which use Chromium's own network stack, not the
/// reqwest `SafeResolver`) can reach internal targets like the cloud
/// metadata service or postgres. The Fetch hook can't be exercised in CI (no
/// Chromium) — the `#[ignore]`d `ssrf_interception_does_not_wedge_allowed_navigation`
/// smoke test validates it against a real binary. `CRAWLER_SSRF_INTERCEPT`;
/// set `false` only as a break-glass if the hook destabilizes a deployment.
pub ssrf_intercept: bool,
}
impl Default for CrawlerConfig {
@@ -399,10 +559,12 @@ impl Default for CrawlerConfig {
browser: LaunchOptions::headless(),
download_allowlist: DownloadAllowlist::new(),
max_image_bytes: DEFAULT_MAX_IMAGE_BYTES,
max_images_per_chapter: 2000,
manga_limit: 0,
job_timeout: Duration::from_secs(600),
metadata_max_consecutive_failures: 10,
browser_restart_threshold: 3,
ssrf_intercept: true,
}
}
}
@@ -417,6 +579,7 @@ impl Config {
storage_dir: std::env::var("STORAGE_DIR")
.unwrap_or_else(|_| "./data/storage".to_string())
.into(),
db: DbConfig::from_env(),
auth: AuthConfig {
cookie_secure: env_bool("COOKIE_SECURE", true),
cookie_domain: std::env::var("COOKIE_DOMAIN")
@@ -435,10 +598,12 @@ impl Config {
},
allow_self_register: env_bool("ALLOW_SELF_REGISTER", true),
private_mode: env_bool("PRIVATE_MODE", false),
trusted_proxy: env_bool("AUTH_TRUSTED_PROXY", false),
},
upload: UploadConfig {
max_request_bytes: env_usize("MAX_REQUEST_BYTES", 200 * 1024 * 1024),
max_file_bytes: env_usize("MAX_FILE_BYTES", 20 * 1024 * 1024),
max_pages_per_chapter: env_usize("MAX_PAGES_PER_CHAPTER", 2000),
},
cors_allowed_origins: std::env::var("CORS_ALLOWED_ORIGINS")
.ok()
@@ -468,11 +633,30 @@ impl Config {
/// Returns `Some((username, password))` only when BOTH `ADMIN_USERNAME`
/// and `ADMIN_PASSWORD` are set and non-empty. Half-set configuration is
/// treated as "no bootstrap" rather than a hard error, so an operator
/// can comment out one env var without crashing the server.
/// can comment out one env var without crashing the server. Emits a
/// warning in the half-set case — silent no-op startled operators when
/// the documented compose env vars finally landed in 0.87.12 (a typo
/// in ADMIN_USERNAME used to silently disable bootstrap with no log
/// trace explaining why no admin appeared).
fn admin_bootstrap_from_env() -> Option<(String, String)> {
let username = std::env::var("ADMIN_USERNAME").ok().filter(|s| !s.is_empty())?;
let password = std::env::var("ADMIN_PASSWORD").ok().filter(|s| !s.is_empty())?;
Some((username, password))
let username = std::env::var("ADMIN_USERNAME").ok().filter(|s| !s.is_empty());
let password = std::env::var("ADMIN_PASSWORD").ok().filter(|s| !s.is_empty());
match (username, password) {
(Some(u), Some(p)) => Some((u, p)),
(Some(_), None) => {
tracing::warn!(
"ADMIN_USERNAME is set but ADMIN_PASSWORD is empty — admin bootstrap skipped (set both, or unset both to silence this warning)"
);
None
}
(None, Some(_)) => {
tracing::warn!(
"ADMIN_PASSWORD is set but ADMIN_USERNAME is empty — admin bootstrap skipped (set both, or unset both to silence this warning)"
);
None
}
(None, None) => None,
}
}
impl CrawlerConfig {
@@ -537,6 +721,7 @@ impl CrawlerConfig {
browser: LaunchOptions::from_env(),
download_allowlist,
max_image_bytes: env_usize("CRAWLER_MAX_IMAGE_BYTES", DEFAULT_MAX_IMAGE_BYTES),
max_images_per_chapter: env_usize("CRAWLER_MAX_IMAGES_PER_CHAPTER", 2000),
manga_limit: env_usize("CRAWLER_LIMIT", 0),
job_timeout: Duration::from_secs(env_u64("CRAWLER_JOB_TIMEOUT_SECS", 600).max(1)),
metadata_max_consecutive_failures: env_u64(
@@ -545,6 +730,7 @@ impl CrawlerConfig {
) as u32,
browser_restart_threshold: env_u64("CRAWLER_BROWSER_RESTART_THRESHOLD", 3).max(1)
as u32,
ssrf_intercept: env_bool("CRAWLER_SSRF_INTERCEPT", true),
})
}
}
@@ -692,6 +878,37 @@ mod tests {
assert_eq!(cfg.browser_restart_threshold, 7);
}
#[test]
fn db_pool_defaults_when_unset() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::remove_var("DB_MAX_CONNECTIONS");
std::env::remove_var("DB_ACQUIRE_TIMEOUT_SECS");
let cfg = DbConfig::from_env();
assert_eq!(cfg.max_connections, 20);
assert_eq!(cfg.acquire_timeout, Duration::from_secs(10));
}
#[test]
fn db_pool_parses_from_env() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("DB_MAX_CONNECTIONS", "50");
std::env::set_var("DB_ACQUIRE_TIMEOUT_SECS", "3");
let cfg = DbConfig::from_env();
std::env::remove_var("DB_MAX_CONNECTIONS");
std::env::remove_var("DB_ACQUIRE_TIMEOUT_SECS");
assert_eq!(cfg.max_connections, 50);
assert_eq!(cfg.acquire_timeout, Duration::from_secs(3));
}
#[test]
fn db_pool_max_connections_clamps_to_at_least_one() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("DB_MAX_CONNECTIONS", "0");
let cfg = DbConfig::from_env();
std::env::remove_var("DB_MAX_CONNECTIONS");
assert_eq!(cfg.max_connections, 1);
}
#[test]
fn analysis_config_defaults_when_unset() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
@@ -699,7 +916,7 @@ mod tests {
"ANALYSIS_ENABLED",
"ANALYSIS_WORKERS",
"ANALYSIS_VISION_URL",
"ANALYSIS_MODEL",
"ANALYSIS_VISION_MODEL",
"ANALYSIS_API_KEY",
"ANALYSIS_MAX_TOKENS",
"ANALYSIS_MAX_PIXELS",
@@ -709,11 +926,18 @@ mod tests {
"ANALYSIS_MAX_SLICES",
"ANALYSIS_RESPONSE_FORMAT",
"ANALYSIS_FREQUENCY_PENALTY",
"ANALYSIS_BACKEND",
"OCRS_DETECTION_MODEL",
"OCRS_RECOGNITION_MODEL",
] {
std::env::remove_var(k);
}
let cfg = AnalysisConfig::from_env();
assert!(!cfg.enabled);
// OCR is the default engine (the Pi-friendly path).
assert_eq!(cfg.backend, AnalysisBackend::Ocr);
assert_eq!(cfg.ocr_detection_model, "/models/text-detection.rten");
assert_eq!(cfg.ocr_recognition_model, "/models/text-recognition.rten");
assert_eq!(cfg.workers, 1);
assert_eq!(cfg.max_pixels, 1_000_000);
assert_eq!(cfg.min_slice_height, 640);
@@ -740,13 +964,58 @@ mod tests {
std::env::remove_var("ANALYSIS_RESPONSE_FORMAT");
}
#[test]
fn analysis_backend_parses_and_defaults_to_ocr() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
for (raw, want) in [
("ocr", AnalysisBackend::Ocr),
("ocrs", AnalysisBackend::Ocr),
("vision", AnalysisBackend::Vision),
("llm", AnalysisBackend::Vision),
("anything-else", AnalysisBackend::Ocr),
] {
std::env::set_var("ANALYSIS_BACKEND", raw);
assert_eq!(AnalysisConfig::from_env().backend, want, "raw={raw}");
}
// Unset → OCR.
std::env::remove_var("ANALYSIS_BACKEND");
assert_eq!(AnalysisConfig::from_env().backend, AnalysisBackend::Ocr);
}
#[test]
fn effective_backend_is_ocr_even_when_vision_requested() {
// Vision is temporarily disabled: the worker always runs OCR no matter
// what `ANALYSIS_BACKEND` parsed to. `backend` still reflects the raw
// request (so the override is visible/loggable), but `effective_backend`
// is the value the daemon actually dispatches through.
let mut cfg = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
assert_eq!(cfg.effective_backend(), AnalysisBackend::Ocr);
cfg.backend = AnalysisBackend::Ocr;
assert_eq!(cfg.effective_backend(), AnalysisBackend::Ocr);
}
#[test]
fn ocr_model_paths_parse_from_env() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("OCRS_DETECTION_MODEL", "/opt/det.rten");
std::env::set_var("OCRS_RECOGNITION_MODEL", "/opt/rec.rten");
let cfg = AnalysisConfig::from_env();
std::env::remove_var("OCRS_DETECTION_MODEL");
std::env::remove_var("OCRS_RECOGNITION_MODEL");
assert_eq!(cfg.ocr_detection_model, "/opt/det.rten");
assert_eq!(cfg.ocr_recognition_model, "/opt/rec.rten");
}
#[test]
fn analysis_config_parses_from_env() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("ANALYSIS_ENABLED", "true");
std::env::set_var("ANALYSIS_WORKERS", "4");
std::env::set_var("ANALYSIS_VISION_URL", "http://vis/v1/chat");
std::env::set_var("ANALYSIS_MODEL", "qwen2-vl");
std::env::set_var("ANALYSIS_VISION_MODEL", "qwen2-vl");
std::env::set_var("ANALYSIS_MAX_PIXELS", "768000");
std::env::set_var("ANALYSIS_MAX_SLICES", "8");
std::env::set_var("ANALYSIS_SLICE_OVERLAP", "0.2");
@@ -755,7 +1024,7 @@ mod tests {
"ANALYSIS_ENABLED",
"ANALYSIS_WORKERS",
"ANALYSIS_VISION_URL",
"ANALYSIS_MODEL",
"ANALYSIS_VISION_MODEL",
"ANALYSIS_MAX_PIXELS",
"ANALYSIS_MAX_SLICES",
"ANALYSIS_SLICE_OVERLAP",
@@ -803,5 +1072,393 @@ mod tests {
let cfg = Config::from_env().expect("from_env");
assert!(!cfg.auth.private_mode);
}
/// Vars that live in `.env.example` for operator context but that
/// are NOT consumed by the backend service (so don't need to be in
/// backend.environment). Each entry needs a rationale — if a future
/// var is added to `.env.example` without being wired into compose,
/// the test fails until either the wiring is added or the var is
/// listed here with a justification.
///
/// - POSTGRES_DB/USER/PASSWORD — composed into DATABASE_URL on the
/// compose RHS; the backend container never sees them by name.
/// - TOR_CONTROL_PASSWORD — wired through to BOTH tor and backend
/// under different names (PASSWORD / CRAWLER_TOR_CONTROL_PASSWORD).
/// - BACKEND_URL / BACKEND_PROXY_TIMEOUT_MS — consumed by the
/// SvelteKit frontend container, not the backend.
/// - VISION_MANAGER_DATABASE_URL — consumed by the vision-manager
/// sidecar, not the backend.
const NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE: &[&str] = &[
"POSTGRES_DB",
"POSTGRES_USER",
"POSTGRES_PASSWORD",
"TOR_CONTROL_PASSWORD",
"BACKEND_URL",
"BACKEND_PROXY_TIMEOUT_MS",
"VISION_MANAGER_DATABASE_URL",
// Compose-level deployment knobs (port publish interface + per-container
// memory ceilings) — consumed by docker-compose.yml itself, never read
// by the backend process, so they don't belong in its environment block.
"FRONTEND_PUBLISH_ADDR",
"BACKEND_MEM_LIMIT",
"FRONTEND_MEM_LIMIT",
"POSTGRES_MEM_LIMIT",
];
/// Keys whose compose RHS is intentionally NOT a `${KEY...}`
/// interpolation. Each entry needs a justification:
///
/// - BIND_ADDRESS / STORAGE_DIR — container-internal binding /
/// mount target; overriding from the host would un-net the
/// container or break the volume.
/// - DATABASE_URL — composed in compose from POSTGRES_USER /
/// PASSWORD / DB; the backend container never sees the raw
/// `${DATABASE_URL...}` placeholder.
/// - CRAWLER_TOR_CONTROL_PASSWORD — wired through TOR_CONTROL_PASSWORD
/// (single source of truth for the tor service's own PASSWORD).
const COMPOSE_RHS_EXEMPT: &[&str] = &[
"BIND_ADDRESS",
"STORAGE_DIR",
"DATABASE_URL",
"CRAWLER_TOR_CONTROL_PASSWORD",
];
/// Vars read by backend src code that are intentionally NOT
/// operator-facing — kept out of `.env.example` because they are
/// either system vars, dev-only debug knobs, or model-tuning hooks
/// only meaningful when forking the codebase. Each entry needs a
/// rationale (paired here as `(name, why)` for grep-ability):
const INTERNAL_NOT_CONFIGURABLE: &[(&str, &str)] = &[
("HOME", "system var, fallback for $HOME/.cache/mangalord/chromium when CRAWLER_CHROMIUM_DIR is unset (see crawler::browser::default_chromium_dir)"),
("CRAWLER_TOR_CONTROL_PASSWORD", "internal name; compose maps TOR_CONTROL_PASSWORD (.env) into this var so the operator sets one secret in one place"),
("ANALYSIS_SYSTEM_PROMPT", "multi-paragraph default; env-override impractical, fork to change"),
("ANALYSIS_OCR_PROMPT", "multi-paragraph default; env-override impractical, fork to change"),
("ANALYSIS_GROUNDING_PROMPT", "multi-paragraph default; env-override impractical, fork to change"),
("CRAWLER_BROWSER_MODE", "local-dev only: chooses between bundled fetcher and system chromium"),
("CRAWLER_BROWSER_ARGS", "local-dev only: extra chromium launch args"),
("CRAWLER_CHROMIUM_DIR", "local-dev only: override for chromium download cache dir"),
("CRAWLER_KEEP_BROWSER_OPEN", "local-dev only: keep headed chromium alive between runs"),
("CRAWLER_FORCE_REFETCH_CHAPTERS", "debug-only: re-fetches even already-downloaded chapters"),
("CRAWLER_SKIP_CHAPTER_CONTENT", "debug-only: skips page downloads, keeps metadata"),
("CRAWLER_SKIP_CHAPTERS", "debug-only: skips chapter pass entirely"),
];
/// Parse the env-var keys listed at column-0 of `.env.example`.
/// Skips comment-only lines and blank lines.
fn parse_env_example_keys(env_example: &str) -> std::collections::HashSet<String> {
let mut keys = std::collections::HashSet::new();
for line in env_example.lines() {
let trimmed = line.trim_start();
if trimmed.is_empty() || trimmed.starts_with('#') {
continue;
}
// Only column-0 lines count (avoid grabbing keys mentioned in
// mid-line prose inside a comment continuation).
if line.starts_with(|c: char| c.is_ascii_uppercase() || c == '_') {
if let Some((k, _)) = line.split_once('=') {
let k = k.trim();
if !k.is_empty()
&& k.chars().all(|c| c.is_ascii_uppercase() || c.is_ascii_digit() || c == '_')
{
keys.insert(k.to_string());
}
}
}
}
keys
}
/// Regression test for the 0.87.1 compose env-wire-through gap, with
/// the 0.87.17 RHS-interpolation strengthening and the 0.87.25
/// list-derivation strengthening:
///
/// The 0.87.12 version substring-matched the env-var KEY in compose
/// — blind to whether the RHS substituted from `.env`. The 0.87.17
/// version added an RHS-interpolation check but still hardcoded the
/// `BACKEND_CONSUMED` list, which drifted (~22 code-read vars not
/// in compose). The 0.87.25 version derives the list from
/// `.env.example` so the operator-facing surface is the single
/// source of truth: anything documented there must reach the
/// container, and a new line in the example forces a wiring update.
///
/// Every key in `.env.example` that isn't in
/// `NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE` must:
/// * have a `KEY:` line under the backend service `environment:`, AND
/// * either be in `COMPOSE_RHS_EXEMPT` (intentional non-interp), OR
/// have its RHS substitute from `${KEY` so `.env` actually wins.
#[test]
fn docker_compose_wires_every_documented_backend_env_var() {
// Cargo run dir for `cargo test` is `backend/`; .env.example and
// compose file live one up.
let env_example = std::fs::read_to_string("../.env.example")
.expect("read ../.env.example");
let example_keys = parse_env_example_keys(&env_example);
let backend_consumed: Vec<&str> = example_keys
.iter()
.map(String::as_str)
.filter(|k| !NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE.contains(k))
.collect();
assert!(
!backend_consumed.is_empty(),
"parse_env_example_keys returned no keys — check .env.example formatting"
);
let compose = std::fs::read_to_string("../docker-compose.yml")
.expect("read ../docker-compose.yml");
// Extract the backend service block: everything from ` backend:`
// up to the next top-level service key (` <name>:` at 2-space
// indent, no further indent). Line-based scan instead of the
// 0.87.12 `\n frontend:\n` literal so adding/reordering services
// doesn't silently break the slicer.
let backend_block = extract_compose_service_block(&compose, "backend")
.expect("backend service block in docker-compose.yml");
let mut missing: Vec<String> = Vec::new();
let mut not_interpolated: Vec<(String, String)> = Vec::new();
for key in &backend_consumed {
// The env entries sit at exactly 6 spaces under `environment:`.
let needle = format!(" {key}:");
let Some(line_start) = backend_block.find(&needle) else {
missing.push((*key).to_string());
continue;
};
if COMPOSE_RHS_EXEMPT.contains(key) {
continue;
}
// RHS: everything from the colon to the end of line.
let from_colon = &backend_block[line_start + needle.len()..];
let rhs_end = from_colon.find('\n').unwrap_or(from_colon.len());
let rhs = from_colon[..rhs_end].trim();
// Expand-from-env requires `${KEY...}` where KEY matches this
// var's name. Accept both `${KEY:-default}` and `${KEY-default}`
// (and the bare `${KEY}` form for completeness).
let expected_prefixes =
[format!("${{{key}:-"), format!("${{{key}-"), format!("${{{key}}}")];
if !expected_prefixes.iter().any(|p| rhs.starts_with(p.as_str())) {
not_interpolated.push(((*key).to_string(), rhs.to_string()));
}
}
missing.sort();
not_interpolated.sort();
assert!(
missing.is_empty(),
"docker-compose.yml backend service is missing env wiring for: {missing:?}. \
Add `KEY: ${{KEY:-default}}` lines under backend.environment so the var \
reaches the container. (Source of truth is .env.example.)"
);
assert!(
not_interpolated.is_empty(),
"docker-compose.yml backend service has env entries whose RHS does NOT \
substitute from the matching `${{KEY...}}` placeholder, so an operator \
setting `KEY=…` in `.env` will be silently ignored. Either fix the RHS \
to `${{KEY:-default}}` or add the var to `COMPOSE_RHS_EXEMPT` with a \
rationale. Offenders: {not_interpolated:?}"
);
}
/// New in 0.87.25: every env::var / env_xxx call in backend/src/
/// must read a key that is EITHER documented in `.env.example`
/// (operator-tunable) OR listed in `INTERNAL_NOT_CONFIGURABLE` with
/// a rationale.
///
/// This catches the silent-no-op class of bug from the other side:
/// a new code path that reads e.g. `ANALYSIS_TEMPERATURE` would
/// previously compile and run fine, but an operator setting it in
/// `.env` would see no effect because `.env.example` and compose
/// never learned about it. Forcing the rationale (operator-facing
/// or internal-only?) at test time keeps the surface honest.
#[test]
fn every_backend_src_env_read_is_documented_or_allowlisted() {
let env_example = std::fs::read_to_string("../.env.example")
.expect("read ../.env.example");
let example_keys = parse_env_example_keys(&env_example);
let internal_keys: std::collections::HashSet<&str> = INTERNAL_NOT_CONFIGURABLE
.iter()
.map(|(k, _)| *k)
.collect();
// Walk backend/src/ for *.rs files.
let mut files: Vec<std::path::PathBuf> = Vec::new();
walk_rust_files(std::path::Path::new("src"), &mut files)
.expect("walk backend/src/");
let mut undocumented: std::collections::BTreeMap<String, Vec<String>> =
std::collections::BTreeMap::new();
for file in &files {
let src = std::fs::read_to_string(file).expect("read source");
for key in extract_env_reads(&src) {
if example_keys.contains(&key) || internal_keys.contains(key.as_str()) {
continue;
}
undocumented
.entry(key)
.or_default()
.push(file.display().to_string());
}
}
assert!(
undocumented.is_empty(),
"backend/src/ reads env vars that are neither in .env.example \
(operator-tunable) nor in INTERNAL_NOT_CONFIGURABLE (internal-only \
with rationale). For each entry below, either:\n\
(a) add it to .env.example AND wire it into docker-compose.yml's \
backend.environment as `KEY: ${{KEY:-default}}` so operators \
can override it, OR\n\
(b) add it to INTERNAL_NOT_CONFIGURABLE in this file with a \
one-line rationale (system var, dev-only knob, etc.).\n\
Offenders: {undocumented:#?}"
);
}
/// Recursively collect `*.rs` paths under `dir`. Test-only helper
/// for the env-read coverage scan above. Skips `target/` and any
/// hidden directory just in case.
fn walk_rust_files(dir: &std::path::Path, out: &mut Vec<std::path::PathBuf>) -> std::io::Result<()> {
for entry in std::fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
let name = entry.file_name();
let name_str = name.to_string_lossy();
if name_str.starts_with('.') || name_str == "target" {
continue;
}
if path.is_dir() {
walk_rust_files(&path, out)?;
} else if path.extension().and_then(|s| s.to_str()) == Some("rs") {
out.push(path);
}
}
Ok(())
}
/// Find every env-read key in a Rust source string. Matches the
/// project's env helpers (`env_bool`, `env_i64`, …, `env_prompt`)
/// and the std calls (`std::env::var`, `std::env::var_os`). Returns
/// the literal keys passed as the first arg.
///
/// Treats `KEY` as `[A-Z0-9_]+` to avoid false positives like
/// `env::var(some_string)` (not literal). Skips any call with a
/// non-literal first arg.
fn extract_env_reads(src: &str) -> Vec<String> {
const PREFIXES: &[&str] = &[
"env::var(\"",
"env::var_os(\"",
"env_var(\"",
"env_var_os(\"",
"env_bool(\"",
"env_i64(\"",
"env_f64(\"",
"env_usize(\"",
"env_u64(\"",
"env_u32(\"",
"env_prompt(\"",
];
let mut out = Vec::new();
for prefix in PREFIXES {
for (i, _) in src.match_indices(prefix) {
let rest = &src[i + prefix.len()..];
let Some(end) = rest.find('"') else { continue };
let key = &rest[..end];
if !key.is_empty()
&& key
.chars()
.all(|c| c.is_ascii_uppercase() || c.is_ascii_digit() || c == '_')
{
out.push(key.to_string());
}
}
}
out
}
/// Sanity test for the env-read extractor: hand-coded snippet
/// must yield the expected keys.
///
/// The snippet is built at runtime from format! parts so this
/// source file doesn't literally contain any prefix-then-key
/// pattern that the other coverage scan would otherwise pick up
/// and require allowlisting (the scan is text-based and oblivious
/// to comments).
#[test]
fn extract_env_reads_finds_helper_and_std_calls() {
let src = format!(
"let a = std::env::{v}(\"FOO_A\").unwrap();\n\
let b = std::env::{vo}(\"FOO_B\");\n\
let c = {eb}(\"FOO_C\", false);\n\
let d = {ei}(\"FOO_D\", 0);\n\
let e = {ep}(\"FOO_E\", String::new());\n\
// Non-literal call must be skipped:\n\
let _ = std::env::{v}(some_name).unwrap();\n\
// Mixed-case / lowercase keys are skipped (env vars are upper):\n\
let _ = std::env::{v}(\"Lowercase\");\n",
v = "var",
vo = "var_os",
eb = "env_bool",
ei = "env_i64",
ep = "env_prompt",
);
let mut keys = extract_env_reads(&src);
keys.sort();
keys.dedup();
assert_eq!(keys, vec!["FOO_A", "FOO_B", "FOO_C", "FOO_D", "FOO_E"]);
}
/// Extract a single service's block from a docker-compose YAML.
/// Reads from the line `^ <name>:$` (2-space top-level service
/// indent) up to the next top-level service header (`^ [A-Za-z…]:$`)
/// or end of file. Returns `None` if the service isn't found.
///
/// Line-based scan instead of `splitn("\n next:\n")` so a service
/// added between the target and the literal anchor doesn't silently
/// truncate the block. Tolerates blank lines and comments inside the
/// block (only the precise 2-space service-header shape ends it).
fn extract_compose_service_block<'a>(compose: &'a str, name: &str) -> Option<&'a str> {
let header = format!("\n {name}:\n");
let start = compose.find(&header).map(|i| i + 1)?; // include the header line
let body = &compose[start + header.len() - 1..]; // after the trailing \n
// Find the next 2-space top-level service-key line. The "next"
// candidate must be: a newline, then exactly two spaces, then a
// word char (services have alphanumeric names), then anything up
// to `:` followed by EOL. We scan line-by-line.
let mut offset: usize = 0;
for line in body.split('\n') {
// Detect a top-level service header at 2-space indent. Skip
// the first line iteration since `start` already points at
// our own header.
if offset > 0
&& line.starts_with(" ")
&& !line.starts_with(" ")
&& line.trim_end().ends_with(':')
&& line.trim_start().chars().next().is_some_and(|c| c.is_ascii_alphabetic())
{
return Some(&body[..offset.saturating_sub(1)]);
}
offset += line.len() + 1; // +1 for the newline
}
Some(body)
}
/// Sanity-check the slicer against a hand-built compose so a future
/// edit to the helper doesn't silently misframe the wiring test.
#[test]
fn extract_compose_service_block_finds_named_block() {
let yaml = "\
services:
postgres:
image: pg
backend:
environment:
KEY: ${KEY:-default}
expose:
- \"8080\"
frontend:
image: f
";
let backend = extract_compose_service_block(yaml, "backend").expect("found");
assert!(backend.contains("KEY: ${KEY:-default}"));
// Must NOT bleed into the next service.
assert!(!backend.contains("frontend"));
assert!(!backend.contains("image: f"));
}
}

View File

@@ -234,12 +234,20 @@ impl BrowserManager {
await_drain(&self.active, drain_deadline).await;
self.set_phase(RestartPhase::Restarting);
let relaunch = {
// Take the dead handle out under the lock, release the lock, THEN run
// the (slow) Chromium teardown — so a worker that raced past the drain
// can't block on `acquire()` behind one dead browser's close(). Mirrors
// the idle reaper's take-drop-close ordering. Re-acquire to relaunch.
let dead = {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
let _ = handle.close().await;
}
guard.handle.take()
};
if let Some(handle) = dead {
let _ = handle.close().await;
}
let relaunch = {
let mut guard = self.inner.lock().await;
self.launch_into(&mut guard).await
};
@@ -257,9 +265,14 @@ impl BrowserManager {
/// Used on daemon shutdown. After this returns the next acquire will
/// re-launch from scratch.
pub async fn shutdown(&self) {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
// Take-then-drop-then-close: don't hold the lock across Chromium
// teardown (see `invalidate` / the idle reaper).
let handle = {
let mut guard = self.inner.lock().await;
guard.shared = None;
guard.handle.take()
};
if let Some(handle) = handle {
let _ = handle.close().await;
}
}
@@ -278,9 +291,16 @@ impl BrowserManager {
/// Idempotent: calling on an already-invalidated manager is a
/// no-op.
pub async fn invalidate(&self) {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
// Take the handle out under the lock, then release the lock BEFORE the
// slow Chromium close() — otherwise every other worker's `acquire()`
// serializes behind one dead browser's teardown. Matches the idle
// reaper's ordering at the bottom of this file.
let handle = {
let mut guard = self.inner.lock().await;
guard.shared = None;
guard.handle.take()
};
if let Some(handle) = handle {
let _ = handle.close().await;
tracing::warn!("BrowserManager: handle invalidated — next acquire will relaunch");
}

View File

@@ -18,7 +18,7 @@ use uuid::Uuid;
use crate::crawler::detect::PageError;
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::safety::{fetch_stream, looks_like_image, DownloadAllowlist};
use crate::crawler::safety::{ensure_public_target, fetch_stream, looks_like_image, DownloadAllowlist};
use crate::crawler::session::{self, ChapterProbe};
use crate::storage::{Storage, StorageError};
@@ -71,6 +71,13 @@ pub enum SyncOutcome {
/// Session probe failed mid-sync (avatar selector missing on the
/// chapter page). Caller should abort the whole crawler run.
SessionExpired,
/// The headless browser could not be acquired (down or mid-restart).
/// Produced only by the queue dispatcher (which calls `acquire()`); it is
/// an *infrastructure* outage, not a job failure, so the daemon returns the
/// job to `pending` WITHOUT burning a retry attempt. `sync_chapter_content`
/// itself never returns this — callers that already hold a lease can treat
/// it as unreachable.
BrowserUnavailable,
}
/// Per-chapter max fetch attempts when TOR is configured. `N = 3` means
@@ -95,6 +102,19 @@ enum ChapterFetchOutcome {
PersistentTransient,
}
/// Refuse to navigate Chromium at a chapter URL that points inside the
/// deployment. The image-download path goes through `is_safe_url`, but
/// `new_page` is a second network surface: a `chapter_sources` row whose
/// host resolves to (or was crafted to be) a private IP would otherwise let
/// the headless browser probe `postgres:5432`, the cloud metadata service,
/// etc. Reuses the allowlist-free `ensure_public_target` (scheme +
/// private-IP literal check) since there is no per-host allowlist for the
/// scraped catalog itself.
fn guard_nav_url(source_url: &str) -> anyhow::Result<()> {
ensure_public_target(source_url)
.map_err(|e| anyhow::anyhow!("refuse to navigate unsafe chapter URL {source_url}: {e}"))
}
/// Single rate-limited Chromium navigation to the chapter URL,
/// returning the page HTML. Extracted from `sync_chapter_content` so
/// the recircuit loop can call it once per attempt.
@@ -103,27 +123,36 @@ async fn fetch_chapter_html_once(
rate: &HostRateLimiters,
source_url: &str,
) -> anyhow::Result<String> {
guard_nav_url(source_url)?;
rate.wait_for(source_url).await?;
let page = browser
.new_page(source_url)
let page = crate::crawler::intercept::open_page(browser, source_url)
.await
.with_context(|| format!("open chapter page {source_url}"))?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for chapter nav")?;
// Best-effort wait for the reader marker — same partial-render
// race that bit the chapter-list parser can hit here. Timeout is
// not an error; the chapter probe + parser sentinels still catch
// real failures.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"a#pic_container",
crate::crawler::nav::SELECTOR_TIMEOUT,
// Close the tab on every exit path — a `?` on wait_for_nav / content()
// would otherwise leak it (chromiumoxide doesn't close on drop).
let closer = page.clone();
crate::crawler::nav::close_after(
async move {
closer.close().await.ok();
},
async {
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for chapter nav")?;
// Best-effort wait for the reader marker — same partial-render
// race that bit the chapter-list parser can hit here. Timeout is
// not an error; the chapter probe + parser sentinels still catch
// real failures.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"a#pic_container",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
page.content().await.context("read chapter html")
},
)
.await;
let html = page.content().await.context("read chapter html")?;
page.close().await.ok();
Ok(html)
.await
}
/// Pure-over-IO loop: fetch + classify, up to `max_attempts` total
@@ -214,6 +243,7 @@ pub async fn sync_chapter_content(
force_refetch: bool,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<&crate::crawler::tor::TorController>,
progress: Option<&crate::crawler::status::StatusHandle>,
enqueue_analysis: bool,
@@ -221,7 +251,8 @@ pub async fn sync_chapter_content(
let started = std::time::Instant::now();
let result = sync_chapter_content_inner(
browser, db, storage, http, rate, chapter_id, manga_id, source_url,
force_refetch, allowlist, max_image_bytes, tor, progress, enqueue_analysis,
force_refetch, allowlist, max_image_bytes, max_images_per_chapter, tor, progress,
enqueue_analysis,
)
.await;
let duration_ms = started.elapsed().as_millis() as i64;
@@ -271,6 +302,7 @@ async fn sync_chapter_content_inner(
force_refetch: bool,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<&crate::crawler::tor::TorController>,
// Optional live-status sink for the realtime page counter. The daemon
// dispatcher passes the shared handle (the chapter has already been
@@ -331,6 +363,16 @@ async fn sync_chapter_content_inner(
if images.is_empty() {
anyhow::bail!("no page images parsed from {source_url}");
}
// Bound total disk per chapter: the per-image byte cap doesn't stop a
// hostile reader page from listing thousands of <img> tags. Ack failed
// (the caller records it and backs off) rather than downloading them.
if let Some(over) = image_count_over_cap(images.len(), max_images_per_chapter) {
anyhow::bail!(
"chapter at {source_url} lists {} page images, over the {} cap",
over.count,
over.cap
);
}
// Resolve image URLs against the chapter URL (they may be relative).
let base = reqwest::Url::parse(source_url).context("parse chapter URL")?;
@@ -407,6 +449,28 @@ pub(crate) struct StoredPage {
/// "first 16 bytes" diagnostic in the error path is useful.
const SNIFF_PREFIX_BYTES: usize = 64;
/// Bytes still admissible for the streaming tail after the sniff prefix
/// has been drained. The prefix already counts against the per-image cap,
/// so the tail budget is `max_image_bytes - prefix_len` using the
/// **actual** drained length — never a constant — so `prefix_len + tail`
/// can never exceed the cap.
fn remaining_after_prefix(max_image_bytes: usize, prefix_len: usize) -> usize {
max_image_bytes.saturating_sub(prefix_len)
}
/// A rejected over-cap image count, carrying the numbers for the error.
struct ImageCountOverCap {
count: usize,
cap: usize,
}
/// `Some(..)` when `count` exceeds the per-chapter image cap. A `cap` of
/// `0` disables the check (unbounded), matching the config's "0 = no cap"
/// contract.
fn image_count_over_cap(count: usize, cap: usize) -> Option<ImageCountOverCap> {
(cap != 0 && count > cap).then_some(ImageCountOverCap { count, cap })
}
/// Download a single page image, validate it's really an image, and
/// stream it to storage. Returns the storage key + content type. Does
/// not touch the DB — persistence is batched into one short transaction
@@ -480,11 +544,18 @@ async fn download_and_store_page(
// straight to storage. The cap is enforced via a running total in
// the stream adapter so a server that omits Content-Length still
// can't exhaust memory.
// Budget the streaming tail against the bytes *actually* drained into
// the prefix, not the constant `SNIFF_PREFIX_BYTES`. The prefix loop
// appends whole chunks, so a single 16 KiB first chunk fills the
// 64-byte sniff window in one drain — charging only 64 bytes would
// then let the tail add another full `max_image_bytes`, storing up to
// ~2× the cap. Capture the length before `prefix` is moved into the
// stream below.
let prefix_len = prefix.len();
let prefix_stream = futures_util::stream::once(async move {
Ok::<bytes::Bytes, StorageError>(prefix)
});
let prefix_len = SNIFF_PREFIX_BYTES.min(max_image_bytes);
let mut remaining = max_image_bytes.saturating_sub(prefix_len);
let mut remaining = remaining_after_prefix(max_image_bytes, prefix_len);
let url_for_err = url.clone();
let rest_stream = body.map(move |frame| match frame {
Ok(chunk) => {
@@ -524,15 +595,22 @@ pub(crate) async fn persist_pages(
let mut tx = db.begin().await.context("open chapter sync tx")?;
let mut page_ids: Vec<Uuid> = Vec::with_capacity(stored.len());
let mut page_numbers: Vec<i32> = Vec::with_capacity(stored.len());
// Pages whose row already existed and was overwritten by this re-crawl.
// Their image changed but pages.id (and any analysis keyed to it) survived,
// so the derived analysis is now stale and must be invalidated below.
let mut updated_page_ids: Vec<Uuid> = Vec::new();
for page in stored {
let (id,): (Uuid,) = sqlx::query_as(
// `xmax <> 0` on the RETURNING row distinguishes a conflict UPDATE
// (existing row overwritten) from a fresh INSERT — the standard upsert
// idiom. Cast through text/bigint since there is no direct xid<>int op.
let (id, was_update): (Uuid, bool) = sqlx::query_as(
"INSERT INTO pages (chapter_id, page_number, storage_key, content_type, size_bytes)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (chapter_id, page_number) DO UPDATE
SET storage_key = EXCLUDED.storage_key,
content_type = EXCLUDED.content_type,
size_bytes = EXCLUDED.size_bytes
RETURNING id",
RETURNING id, (xmax::text::bigint <> 0)",
)
.bind(chapter_id)
.bind(page.page_number)
@@ -544,6 +622,32 @@ pub(crate) async fn persist_pages(
.with_context(|| format!("insert page row {}", page.page_number))?;
page_ids.push(id);
page_numbers.push(page.page_number);
if was_update {
updated_page_ids.push(id);
}
}
// Invalidate stale analysis for re-crawled (overwritten) pages. Storage keys
// are deterministic per page number, so the row survives the upsert and its
// page_analysis / OCR / auto-tags / content-warnings would otherwise reflect
// the OLD image — poisoning search results and the derived
// manga_content_warnings. Clearing them (the DELETE on page_content_warnings
// fires the mcw_* triggers so manga_content_warnings recomputes) drops the
// pages back to "unanalyzed" so the enqueue below — and the admin
// only-unanalyzed backfill — re-derive them. Newly inserted pages have no
// prior analysis, so they're untouched.
if !updated_page_ids.is_empty() {
for table in [
"page_ocr_text",
"page_auto_tags",
"page_content_warnings",
"page_analysis",
] {
sqlx::query(&format!("DELETE FROM {table} WHERE page_id = ANY($1::uuid[])"))
.bind(&updated_page_ids)
.execute(&mut *tx)
.await
.with_context(|| format!("invalidate {table} for re-crawled pages"))?;
}
}
// Drop any rows left over from a prior, larger crawl of this chapter
// (re-crawl that now yields fewer pages). Without this the stale rows —
@@ -609,6 +713,64 @@ mod tests {
use super::*;
use crate::storage::LocalStorage;
#[test]
fn guard_nav_url_rejects_private_and_loopback_targets() {
// A chapter_sources row that resolves to / was crafted as an
// internal target must be refused before Chromium navigates.
for url in [
"http://127.0.0.1:5432/",
"http://169.254.169.254/latest/meta-data/",
"http://10.0.0.1/chapter/1",
"http://localhost:8080/",
"file:///etc/passwd",
] {
assert!(guard_nav_url(url).is_err(), "must reject {url}");
}
}
#[test]
fn guard_nav_url_allows_public_chapter_urls() {
assert!(guard_nav_url("https://reader.example.com/chapter/42").is_ok());
assert!(guard_nav_url("http://manga-host.test/c/1/p/2").is_ok());
}
#[test]
fn tail_budget_uses_actual_prefix_length_not_constant() {
// A single 16 KiB first chunk fills the 64-byte sniff window in one
// drain, so the streaming tail budget must be `cap - 16KiB`, not
// `cap - 64`. The constant-64 math previously admitted a further
// ~full cap on top of the already-drained prefix (~2× overshoot).
let cap = 20 * 1024;
let prefix_len = 16 * 1024; // one real chunk, well over SNIFF_PREFIX_BYTES
let remaining = remaining_after_prefix(cap, prefix_len);
assert_eq!(remaining, cap - prefix_len);
// Invariant: prefix + admitted tail never exceeds the cap.
assert!(prefix_len + remaining <= cap);
// And it's strictly tighter than the old constant-64 budget.
assert!(remaining < cap - SNIFF_PREFIX_BYTES);
}
#[test]
fn image_count_cap_rejects_only_over_cap_and_respects_disable() {
// Under / at the cap: accepted.
assert!(image_count_over_cap(0, 2000).is_none());
assert!(image_count_over_cap(2000, 2000).is_none());
// Over the cap: rejected, carrying the numbers for the error.
let over = image_count_over_cap(2001, 2000).expect("over cap");
assert_eq!(over.count, 2001);
assert_eq!(over.cap, 2000);
// `0` disables the cap entirely (unbounded), matching config contract.
assert!(image_count_over_cap(1_000_000, 0).is_none());
}
#[test]
fn tail_budget_saturates_when_prefix_hits_cap() {
// Body shorter than the sniff window: prefix drained == cap, tail
// budget is zero (no negative underflow).
assert_eq!(remaining_after_prefix(50, 50), 0);
assert_eq!(remaining_after_prefix(50, 64), 0);
}
#[tokio::test]
async fn cleanup_orphans_deletes_written_keys() {
let dir = tempfile::tempdir().unwrap();
@@ -1071,4 +1233,91 @@ mod tests {
assert_eq!(fetch_n, 1);
assert!(format!("{err:#}").contains("nav timeout"));
}
#[sqlx::test(migrations = "./migrations")]
async fn persist_pages_invalidates_stale_analysis_on_recrawl_update(pool: PgPool) {
// Page storage keys are deterministic per page number, so a re-crawl
// overwrites the image but keeps pages.id. Analysis keyed to page_id
// would otherwise stay stale (old OCR/search_doc/warnings). persist_pages
// must clear the derived analysis for updated pages so it re-derives.
let manga_id = Uuid::new_v4();
let chapter_id = Uuid::new_v4();
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, 'T')")
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
.bind(chapter_id)
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
let v1 = vec![StoredPage {
page_number: 1,
storage_key: "k/0001.jpg".into(),
content_type: "image/jpeg".into(),
size_bytes: 10,
}];
persist_pages(&pool, chapter_id, &v1, false).await.unwrap();
let page_id: Uuid = sqlx::query_scalar(
"SELECT id FROM pages WHERE chapter_id = $1 AND page_number = 1",
)
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
// Simulate a completed prior analysis pass for that page.
sqlx::query(
"INSERT INTO page_analysis (page_id, status, is_nsfw, analyzed_at) \
VALUES ($1, 'done', true, now())",
)
.bind(page_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO page_ocr_text (page_id, kind, text) VALUES ($1, 'speech', 'stale')")
.bind(page_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO page_content_warnings (page_id, warning) VALUES ($1, 'gore')")
.bind(page_id)
.execute(&pool)
.await
.unwrap();
// Re-crawl page 1 with a new image (same deterministic key, new size).
let v2 = vec![StoredPage {
page_number: 1,
storage_key: "k/0001.jpg".into(),
content_type: "image/jpeg".into(),
size_bytes: 999,
}];
persist_pages(&pool, chapter_id, &v2, false).await.unwrap();
// The page row survived (deterministic key).
let same_id: Uuid = sqlx::query_scalar(
"SELECT id FROM pages WHERE chapter_id = $1 AND page_number = 1",
)
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(same_id, page_id, "re-crawl keeps the page id");
// ...but its stale analysis is cleared so it gets re-derived.
for table in ["page_analysis", "page_ocr_text", "page_content_warnings"] {
let n: i64 = sqlx::query_scalar(&format!(
"SELECT COUNT(*) FROM {table} WHERE page_id = $1"
))
.bind(page_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(n, 0, "{table} must be invalidated when the re-crawl updates the page");
}
}
}

View File

@@ -46,7 +46,7 @@ use tokio::task::JoinSet;
use tokio_util::sync::CancellationToken;
use crate::crawler::content::SyncOutcome;
use crate::crawler::jobs::{self, JobPayload, Lease, KIND_SYNC_CHAPTER_CONTENT};
use crate::crawler::jobs::{self, JobPayload, Lease, KIND_SYNC_CHAPTER_CONTENT, KIND_SYNC_MANGA};
use crate::crawler::pipeline;
use crate::crawler::status::{Phase, StatusHandle};
@@ -66,6 +66,43 @@ const LEASE_DURATION: Duration = Duration::from_secs(60);
/// the lease window leaves two missed-beat's slack before expiry.
const LEASE_HEARTBEAT: Duration = Duration::from_secs(20);
/// Consecutive failed lease renews the heartbeat tolerates before it gives up
/// and signals the worker to abandon the in-flight dispatch. At
/// [`LEASE_HEARTBEAT`] spacing this is ~1 lease window of DB flakiness — past
/// that the lease has very likely lapsed and another worker may re-lease the
/// job, so continuing to crawl it is wasted (and duplicated) work.
const MAX_HEARTBEAT_RENEW_FAILURES: u32 = 3;
/// Whether the heartbeat should abandon the job after `consecutive_failures`
/// failed renews. Split out so the escalation threshold is unit-testable.
fn should_abort_after_renew_failures(consecutive_failures: u32) -> bool {
consecutive_failures >= MAX_HEARTBEAT_RENEW_FAILURES
}
/// How long a worker waits after a `BrowserUnavailable` outcome before looping
/// back to lease again. The job was released (not failed) so it stays pending;
/// this backoff keeps the worker from hot-looping lease→acquire→release while
/// the browser is down or mid-restart.
const BROWSER_UNAVAILABLE_BACKOFF: Duration = Duration::from_secs(5);
/// Longest an idle worker waits between lease polls. Bounds the exponential
/// [`idle_backoff`] so a worker still notices freshly-enqueued work reasonably
/// soon after a quiet spell.
const IDLE_BACKOFF_CAP: Duration = Duration::from_secs(30);
/// Backoff for a worker that keeps finding no work: 1s, 2s, 4s, … capped at
/// [`IDLE_BACKOFF_CAP`]. `consecutive_empty` is the number of empty polls seen
/// so far (0 on the first miss); reset to 0 the moment a job is leased. Replaces
/// the old flat 1s sleep so an idle daemon isn't firing a row-locking `SELECT …
/// FOR UPDATE SKIP LOCKED` lease query every second per worker.
fn idle_backoff(consecutive_empty: u32) -> Duration {
let cap = IDLE_BACKOFF_CAP.as_secs();
// 1 << n grows the interval; saturate to the cap once the shift overflows
// or the value exceeds the cap.
let secs = 1u64.checked_shl(consecutive_empty).unwrap_or(cap).min(cap);
Duration::from_secs(secs)
}
#[async_trait]
pub trait MetadataPass: Send + Sync {
async fn run(&self) -> anyhow::Result<pipeline::MetadataStats>;
@@ -76,6 +113,15 @@ pub trait ChapterDispatcher: Send + Sync {
async fn dispatch(&self, payload: JobPayload) -> anyhow::Result<SyncOutcome>;
}
/// A full list-only walk that enqueues mangas missing from the DB as
/// `SyncManga` jobs. Triggered on demand from the admin `/reconcile`
/// endpoint (not on the cron). Mirrors [`MetadataPass`] so the endpoint can
/// hold a `dyn` and tests can stub it.
#[async_trait]
pub trait ReconcilePass: Send + Sync {
async fn run(&self) -> anyhow::Result<crate::crawler::reconcile::ReconcileStats>;
}
/// Configuration for [`spawn`]. Use `None` for `metadata_pass` to disable
/// the cron entirely (worker-pool-only mode — useful when only the
/// bookmark-triggered enqueue path is wanted).
@@ -271,7 +317,8 @@ impl CronContext {
// no future tick would ever run — workers would keep going but
// no new metadata work would be scheduled until daemon restart.
// The advisory unlock below runs unconditionally so a panicked
// tick doesn't leave the lock held for another replica.
// (or cancelled) tick doesn't leave the lock held for another
// replica.
let metadata = &self.metadata;
let pool = &self.pool;
let retention_days = self.retention_days;
@@ -291,9 +338,9 @@ impl CronContext {
}
Err(e) => tracing::error!(?e, "cron: enqueue_bookmarked_pending failed"),
}
match jobs::reap_done(pool, retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: done-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: done-job reaper failed"),
match jobs::reap_terminal(pool, retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: terminal-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: terminal-job reaper failed"),
}
match crate::repo::crawl_metrics::reap(pool, metrics_retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: crawl-metrics reaper finished"),
@@ -303,8 +350,23 @@ impl CronContext {
tracing::warn!(?e, "cron: persist last_metadata_tick_at failed");
}
};
if let Err(_panic) = AssertUnwindSafe(body).catch_unwind().await {
tracing::error!("cron: tick body panicked — continuing");
// Race the tick body against shutdown. Without this, a long
// metadata pass on a fresh catalog (minutes) wedges
// `DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under
// the supervisor lock), and SIGTERM responsiveness for the
// entire pass. On cancel, dropping `body` cascades to dropping
// `metadata.run()`, which is what gives Chromium/lease cleanup
// their chance via the Browse Manager's idle reaper.
tokio::select! {
biased;
_ = self.cancel.cancelled() => {
tracing::info!("cron: cancelled mid-tick — abandoning to release advisory lock");
}
r = AssertUnwindSafe(body).catch_unwind() => {
if r.is_err() {
tracing::error!("cron: tick body panicked — continuing");
}
}
}
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
@@ -331,6 +393,9 @@ struct WorkerContext {
impl WorkerContext {
async fn run(self) {
// Consecutive empty lease polls, driving the idle backoff. Reset to 0
// the moment any job is leased.
let mut idle_streak: u32 = 0;
loop {
if self.cancel.is_cancelled() {
tracing::info!(worker = self.id, "worker: shutdown");
@@ -342,9 +407,9 @@ impl WorkerContext {
_ = self.cancel.cancelled() => return,
}
}
let leases = match jobs::lease(
let leases = match jobs::lease_kinds(
&self.pool,
Some(KIND_SYNC_CHAPTER_CONTENT),
&[KIND_SYNC_CHAPTER_CONTENT, KIND_SYNC_MANGA],
1,
LEASE_DURATION,
)
@@ -360,11 +425,14 @@ impl WorkerContext {
}
};
let Some(lease) = leases.into_iter().next() else {
let backoff = idle_backoff(idle_streak);
idle_streak = idle_streak.saturating_add(1);
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(1)) => continue,
_ = tokio::time::sleep(backoff) => continue,
_ = self.cancel.cancelled() => return,
}
};
idle_streak = 0;
self.process_lease(lease).await;
}
}
@@ -379,7 +447,7 @@ impl WorkerContext {
.ok()
.flatten();
if matches!(page_count, Some(n) if n > 0) {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
return;
}
}
@@ -388,17 +456,34 @@ impl WorkerContext {
// dispatch runs, so a slow-but-healthy job is never re-leased and
// never inflates `attempts` toward `max_attempts`. Stops itself
// once the job is no longer ours (renew returns false).
// Signalled by the heartbeat if it gives up after too many consecutive
// renew failures, so the worker can abandon a dispatch whose lease has
// very likely lapsed (rather than crawl a job another worker may now own).
let hb_lost = CancellationToken::new();
let heartbeat = {
let hb_pool = self.pool.clone();
let hb_id = lease.id;
let hb_gen = lease.lease_generation;
let hb_lost = hb_lost.clone();
tokio::spawn(async move {
let mut failures: u32 = 0;
loop {
tokio::time::sleep(LEASE_HEARTBEAT).await;
match jobs::renew(&hb_pool, hb_id, LEASE_DURATION).await {
Ok(true) => {}
match jobs::renew(&hb_pool, hb_id, hb_gen, LEASE_DURATION).await {
Ok(true) => failures = 0,
Ok(false) => break,
Err(e) => {
tracing::warn!(lease_id = %hb_id, ?e, "heartbeat renew failed");
failures += 1;
tracing::warn!(lease_id = %hb_id, failures, ?e, "heartbeat renew failed");
if should_abort_after_renew_failures(failures) {
tracing::error!(
lease_id = %hb_id,
failures,
"heartbeat lost the lease after repeated renew failures — signalling abandon"
);
hb_lost.cancel();
break;
}
}
}
}
@@ -428,7 +513,7 @@ impl WorkerContext {
// branch here, so a mid-dispatch SIGTERM left the row `running`
// until lease expiry and cost one attempt.
heartbeat.abort();
let _ = jobs::release(&self.pool, lease.id).await;
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
tracing::info!(
worker = self.id,
lease_id = %lease.id,
@@ -436,6 +521,20 @@ impl WorkerContext {
);
return;
}
_ = hb_lost.cancelled() => {
// The heartbeat gave up renewing: the lease has very likely
// expired and may already be re-leased elsewhere. Abandon the
// dispatch rather than keep crawling a job we no longer own. Do
// NOT ack/release — a generation-guarded write would no-op, and
// another worker may now hold this lease.
heartbeat.abort();
tracing::error!(
worker = self.id,
lease_id = %lease.id,
"worker: abandoning dispatch — lease lost (heartbeat renew failures)"
);
return;
}
o = tokio::time::timeout(self.job_timeout, dispatch) => o,
};
heartbeat.abort();
@@ -455,6 +554,7 @@ impl WorkerContext {
"dispatch timed out",
lease.attempts,
lease.max_attempts,
lease.lease_generation,
)
.await;
return;
@@ -462,7 +562,7 @@ impl WorkerContext {
};
match outcome {
Ok(Ok(SyncOutcome::Fetched { .. } | SyncOutcome::Skipped)) => {
let _ = jobs::ack_done(&self.pool, lease.id).await;
let _ = jobs::ack_done(&self.pool, lease.id, lease.lease_generation).await;
}
Ok(Ok(SyncOutcome::SessionExpired)) => {
tracing::error!(
@@ -473,7 +573,25 @@ impl WorkerContext {
self.session_expired.store(true, Ordering::Release);
// Push the session-expired flip to live status subscribers.
self.status.poke();
let _ = jobs::release(&self.pool, lease.id).await;
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
}
Ok(Ok(SyncOutcome::BrowserUnavailable)) => {
// Infrastructure outage, not a job failure: the browser was
// down or mid-restart when the dispatcher tried to acquire it.
// Return the job to `pending` WITHOUT burning an attempt (like
// the cancel/session paths) so an outage doesn't chew the whole
// backlog to `dead`, then back off to avoid hot-looping while
// the browser recovers.
tracing::warn!(
worker = self.id,
lease_id = %lease.id,
"worker: browser unavailable — released lease without burning an attempt"
);
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
tokio::select! {
_ = tokio::time::sleep(BROWSER_UNAVAILABLE_BACKOFF) => {}
_ = self.cancel.cancelled() => {}
}
}
Ok(Err(e)) => {
tracing::warn!(
@@ -488,6 +606,7 @@ impl WorkerContext {
&format!("{e:#}"),
lease.attempts,
lease.max_attempts,
lease.lease_generation,
)
.await;
}
@@ -503,6 +622,7 @@ impl WorkerContext {
"worker panicked",
lease.attempts,
lease.max_attempts,
lease.lease_generation,
)
.await;
}
@@ -653,6 +773,35 @@ pub mod test_support {
}
}
/// `MetadataPass` that sleeps for `delay` before returning. Lets a
/// test trigger shutdown WHILE a tick body is in flight, exercising
/// the cancellation race in `CronContext::run_tick`.
pub struct SlowMetadataPass {
pub delay: std::time::Duration,
pub started: AtomicUsize,
}
impl SlowMetadataPass {
pub fn new(delay: std::time::Duration) -> Arc<Self> {
Arc::new(Self {
delay,
started: AtomicUsize::new(0),
})
}
pub fn start_count(&self) -> usize {
self.started.load(Ordering::Acquire)
}
}
#[async_trait]
impl MetadataPass for SlowMetadataPass {
async fn run(&self) -> anyhow::Result<pipeline::MetadataStats> {
self.started.fetch_add(1, Ordering::AcqRel);
tokio::time::sleep(self.delay).await;
Ok(pipeline::MetadataStats::default())
}
}
pub type DispatchFn = Arc<
dyn Fn(JobPayload) -> futures_util::future::BoxFuture<'static, anyhow::Result<SyncOutcome>>
+ Send
@@ -692,6 +841,37 @@ mod tests {
Utc.with_ymd_and_hms(y, mo, d, h, mi, 0).unwrap()
}
#[test]
fn heartbeat_aborts_only_after_threshold_consecutive_failures() {
// A blip or two is tolerated; sustained failures escalate to abandon.
assert!(!should_abort_after_renew_failures(0));
assert!(!should_abort_after_renew_failures(1));
assert!(!should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES - 1));
assert!(should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES));
assert!(should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES + 1));
}
#[test]
fn idle_backoff_grows_then_caps() {
// First miss is a short 1s poll; the interval doubles each empty poll…
assert_eq!(idle_backoff(0), Duration::from_secs(1));
assert_eq!(idle_backoff(1), Duration::from_secs(2));
assert_eq!(idle_backoff(2), Duration::from_secs(4));
assert_eq!(idle_backoff(3), Duration::from_secs(8));
assert_eq!(idle_backoff(4), Duration::from_secs(16));
// …and saturates at the cap rather than growing unbounded.
assert_eq!(idle_backoff(5), IDLE_BACKOFF_CAP);
assert_eq!(idle_backoff(100), IDLE_BACKOFF_CAP);
// Never exceeds the cap and is monotonic non-decreasing.
let mut prev = Duration::ZERO;
for n in 0..40 {
let b = idle_backoff(n);
assert!(b >= prev, "backoff must be non-decreasing at n={n}");
assert!(b <= IDLE_BACKOFF_CAP, "backoff must never exceed the cap");
prev = b;
}
}
#[test]
fn next_fire_in_utc_at_midnight_advances_one_day() {
let now = dt_utc(2026, 5, 25, 12, 0); // noon UTC

View File

@@ -0,0 +1,301 @@
//! Optional CDP-level SSRF guard for headless-browser navigations.
//!
//! The reqwest clients get a DNS-filtering resolver (see
//! [`crate::crawler::safety::SafeResolver`]) so an image/API host that
//! resolves to an internal IP is refused at connect time. Chromium, however,
//! does its **own** DNS and connection handling, so that resolver can't see
//! browser navigations. Without a second guard, a scraped chapter page that
//! `302`-redirects the browser to `http://127.0.0.1:5432/` (or an
//! attacker-owned hostname that resolves to `169.254.169.254`) would be
//! loaded and parsed as if it were catalog content.
//!
//! This module installs CDP `Fetch` interception on a page so **every** request
//! it issues — the main-frame Document, its redirects, *and every subresource*
//! (`<img>`, `fetch()`/XHR, media, …) — is re-validated through the same
//! [`ensure_public_target`] + resolved-IP check the reqwest paths use, failing
//! any that target an internal address. Intercepting only the Document would
//! leave a scraped page free to pull `<img src="http://169.254.169.254/…">` or
//! `fetch('http://postgres:5432')` straight past the guard, so no resource type
//! is exempt. (WebSocket handshakes are not surfaced by CDP `Fetch`, so `ws://`
//! internal targets remain out of this hook's reach — the reqwest-layer
//! resolver does not see them either; documented as a known gap.)
//!
//! **Opt-in / default-off.** Enabling `Fetch` means every intercepted request
//! *must* be resolved by a live handler or the navigation hangs, so this is a
//! fragile hook in the crawler's critical path. It ships behind
//! `CRAWLER_SSRF_INTERCEPT` (default `false`) and the wiring has **not** been
//! exercised against a real Chromium in CI — validate with a manual crawl
//! before enabling in production. When disabled, [`open_page`] is byte-for-byte
//! the previous `browser.new_page(url)` behavior. When enabled, the guard is
//! **fail-closed**: if interception can't be installed, [`open_page`] closes the
//! blank page and returns the error rather than navigating unguarded.
use std::net::IpAddr;
use std::sync::atomic::{AtomicBool, Ordering};
use chromiumoxide::browser::Browser;
use chromiumoxide::cdp::browser_protocol::fetch::{
ContinueRequestParams, EnableParams, EventRequestPaused, FailRequestParams, RequestPattern,
RequestStage,
};
use chromiumoxide::cdp::browser_protocol::network::ErrorReason;
use chromiumoxide::error::Result as CdpResult;
use chromiumoxide::Page;
use futures_util::StreamExt;
use reqwest::Url;
use crate::crawler::safety::{ensure_public_target, is_private_ip};
/// Process-wide toggle, set once at startup from `CRAWLER_SSRF_INTERCEPT`.
/// A single boot-time flag (rather than threading a param through every
/// crawler navigation signature) keeps the off-path a no-op.
static ENABLED: AtomicBool = AtomicBool::new(false);
pub fn set_enabled(on: bool) {
ENABLED.store(on, Ordering::Relaxed);
}
pub fn is_enabled() -> bool {
ENABLED.load(Ordering::Relaxed)
}
/// What to do with a navigation URL, decided without DNS where possible so the
/// security-critical branching is unit-testable.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum Verdict {
/// Safe to continue (non-network scheme, or a public IP literal).
Allow,
/// Refuse — bad scheme handled elsewhere, localhost, or a private IP literal.
Block,
/// A hostname that must be resolved to decide (DNS-rebinding check).
ResolveHost { host: String, port: u16 },
}
/// Pure decision over a URL string. `data:` / `blob:` / `about:` and other
/// non-http(s) schemes are allowed (not a network-SSRF vector); http(s) with a
/// private/loopback/localhost literal is blocked; an http(s) hostname needs
/// resolution.
pub(crate) fn verdict(url: &str) -> Verdict {
let Ok(parsed) = Url::parse(url) else {
// Unparseable — let Chromium reject it; not our call to make.
return Verdict::Allow;
};
match parsed.scheme() {
"http" | "https" => {}
// Non-http(s) subresource schemes: block the ones that can reach the
// local filesystem or pivot to another protocol; continue the rest
// (data:/blob:/about: and Chrome-internal schemes) which legitimate page
// loads depend on — blocking those would wedge navigation.
"file" | "ftp" | "gopher" => return Verdict::Block,
_ => return Verdict::Allow,
}
// Literal private IP / localhost / missing host → block (string check).
if ensure_public_target(url).is_err() {
return Verdict::Block;
}
match parsed.host_str() {
Some(host) if !is_ip_literal(host) => Verdict::ResolveHost {
host: host.to_string(),
port: parsed.port_or_known_default().unwrap_or(80),
},
// Public IP literal (ensure_public_target already passed it).
_ => Verdict::Allow,
}
}
/// Whether `host` (as reqwest's `host_str()` yields it) is an IP literal.
/// IPv6 literals arrive bracketed (`[::1]`), which don't parse as `IpAddr`
/// directly — strip the brackets first.
fn is_ip_literal(host: &str) -> bool {
let unbracketed = host
.strip_prefix('[')
.and_then(|s| s.strip_suffix(']'))
.unwrap_or(host);
unbracketed.parse::<IpAddr>().is_ok()
}
/// Async decision: resolves hostnames and blocks if any resolved address is
/// private (DNS rebinding). A resolution failure is *not* treated as blocked —
/// Chromium won't be able to connect either, so there's nothing to exfiltrate.
pub(crate) async fn is_blocked(url: &str) -> bool {
match verdict(url) {
Verdict::Allow => false,
Verdict::Block => true,
Verdict::ResolveHost { host, port } => match tokio::net::lookup_host((host.as_str(), port))
.await
{
Ok(addrs) => addrs.map(|a| a.ip()).any(|ip| is_private_ip(&ip)),
Err(_) => false,
},
}
}
/// Open a page for navigation. With interception off, this is exactly
/// `browser.new_page(url)`. With it on, the page is created blank (no network),
/// the navigation guard is installed, and only then does it navigate — so the
/// initial request and any redirects pass through the guard.
pub async fn open_page(browser: &Browser, url: &str) -> CdpResult<Page> {
if !is_enabled() {
return browser.new_page(url).await;
}
let page = browser.new_page("about:blank").await?;
if let Err(e) = install_navigation_guard(&page).await {
// Fail closed: an unguarded page could be redirected (or pull a
// subresource) to an internal target, so refuse to navigate. Close the
// blank page and surface the error so the caller aborts this fetch.
tracing::warn!(url = %url, error = %e, "SSRF navigation guard failed to install; aborting navigation (fail-closed)");
let _ = page.close().await;
return Err(e);
}
page.goto(url).await?;
Ok(page)
}
/// The `Fetch` interception patterns to register. A single pattern with **no**
/// `resource_type` constraint matches every request the page makes — Document,
/// Image, Fetch/XHR, media, everything — so subresources to internal targets are
/// re-validated too, not only the main-frame navigation. Restricting this to
/// `ResourceType::Document` (the previous behavior) left every `<img>`/`fetch()`
/// subresource unguarded, which is the SSRF hole this closes. Pinned to the
/// request stage so each request is paused once, before it leaves the browser.
fn interception_patterns() -> Vec<RequestPattern> {
vec![RequestPattern::builder()
.request_stage(RequestStage::Request)
.build()]
}
/// Enable `Fetch` for all requests on `page` and spawn a task that
/// continues/fails each paused request per [`is_blocked`].
async fn install_navigation_guard(page: &Page) -> CdpResult<()> {
page.execute(EnableParams {
patterns: Some(interception_patterns()),
handle_auth_requests: None,
})
.await?;
let mut paused = page.event_listener::<EventRequestPaused>().await?;
let handler_page = page.clone();
tokio::spawn(async move {
while let Some(ev) = paused.next().await {
let request_id = ev.request_id.clone();
let outcome = if is_blocked(&ev.request.url).await {
tracing::warn!(
url = %ev.request.url,
"SSRF guard blocked browser navigation to an internal target"
);
handler_page
.execute(FailRequestParams::new(request_id, ErrorReason::BlockedByClient))
.await
.map(|_| ())
} else {
handler_page
.execute(ContinueRequestParams::new(request_id))
.await
.map(|_| ())
};
if let Err(e) = outcome {
// The page/session is gone (page closed) — the stream will end
// too; stop handling so the task exits rather than spins.
tracing::debug!(error = %e, "fetch interceptor: continue/fail failed, ending handler");
break;
}
}
});
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn verdict_allows_non_network_schemes() {
for url in ["about:blank", "data:text/html,hi", "blob:abc", "chrome://version"] {
assert_eq!(verdict(url), Verdict::Allow, "{url} should be allowed");
}
}
#[test]
fn verdict_blocks_dangerous_non_http_schemes() {
// file:/ftp:/gopher: can reach the local filesystem or pivot protocols;
// block them explicitly rather than falling through to Allow.
for url in ["file:///etc/passwd", "ftp://internal/x", "gopher://internal:70/1"] {
assert_eq!(verdict(url), Verdict::Block, "{url} should be blocked");
}
}
#[test]
fn verdict_blocks_private_ip_literals_and_localhost() {
for url in [
"http://127.0.0.1/",
"http://10.0.0.1/",
"http://192.168.1.1:5432/",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://localhost/",
"https://[fd00::1]/",
] {
assert_eq!(verdict(url), Verdict::Block, "{url} should be blocked");
}
}
#[test]
fn verdict_allows_public_ip_literals() {
for url in ["http://93.184.216.34/", "https://[2606:4700:4700::1111]/"] {
assert_eq!(verdict(url), Verdict::Allow, "{url} should be allowed");
}
}
#[test]
fn verdict_defers_hostnames_to_resolution() {
assert_eq!(
verdict("https://cdn.example.com/img.jpg"),
Verdict::ResolveHost {
host: "cdn.example.com".to_string(),
port: 443
}
);
assert_eq!(
verdict("http://catalog.test:8080/list"),
Verdict::ResolveHost {
host: "catalog.test".to_string(),
port: 8080
}
);
}
#[tokio::test]
async fn is_blocked_true_for_private_literal_no_dns() {
assert!(is_blocked("http://169.254.169.254/").await);
assert!(!is_blocked("http://93.184.216.34/").await);
// Non-network scheme is always allowed through.
assert!(!is_blocked("about:blank").await);
}
#[test]
fn interception_covers_all_resource_types() {
// Regression guard for the SSRF subresource hole: the CDP Fetch pattern
// must NOT be constrained to Document, or `<img>`/`fetch()`/XHR
// subresources to internal targets slip past `is_blocked`. A pattern
// with no `resource_type` set intercepts every request type.
let patterns = interception_patterns();
assert_eq!(patterns.len(), 1);
assert!(
patterns[0].resource_type.is_none(),
"interception must cover all resource types (subresources included), not just Document"
);
assert_eq!(patterns[0].request_stage, Some(RequestStage::Request));
}
#[test]
fn enabled_toggle_roundtrips() {
// Global; restore afterwards so other tests see the default.
let prev = is_enabled();
set_enabled(true);
assert!(is_enabled());
set_enabled(false);
assert!(!is_enabled());
set_enabled(prev);
}
}

View File

@@ -15,11 +15,18 @@ use uuid::Uuid;
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum JobPayload {
/// Fetch one manga's detail page, upsert metadata, enqueue
/// `SyncChapterList`.
/// Fetch one manga's detail page, upsert metadata, sync its chapter
/// list. `url` and `title` are carried from the list ref so the worker
/// can reconstruct a `SourceMangaRef` without re-walking, and so the
/// dead-jobs/history UI can render a label even before any manga row
/// exists. `#[serde(default)]` keeps older/hand-inserted rows decodable.
SyncManga {
source_id: String,
source_manga_key: String,
#[serde(default)]
url: String,
#[serde(default)]
title: String,
},
/// Diff the chapter list, enqueue `SyncChapterContent` for new
/// chapters, soft-drop vanished ones.
@@ -61,6 +68,11 @@ pub enum JobState {
/// without re-spelling the literal.
pub const KIND_SYNC_CHAPTER_CONTENT: &str = "sync_chapter_content";
/// Kind discriminator for manga-detail sync jobs (used by the reconcile
/// pass to enqueue missing mangas). The crawl worker leases this alongside
/// `KIND_SYNC_CHAPTER_CONTENT`; both serialize on the single browser.
pub const KIND_SYNC_MANGA: &str = "sync_manga";
/// Kind discriminator for AI page-analysis jobs. The analysis daemon
/// leases with this filter so it never contends with crawl jobs.
pub const KIND_ANALYZE_PAGE: &str = "analyze_page";
@@ -77,6 +89,14 @@ pub struct Lease {
pub payload: JobPayload,
pub attempts: i32,
pub max_attempts: i32,
/// Strictly-increasing token bumped by every `lease`, `release` /
/// `release_unowned`, and `reclaim_orphaned`. `ack_done` /
/// `ack_failed` / `renew` match on `(id, lease_generation)` so a
/// late ack from a dead-but-still-dispatching lease cannot clobber
/// a successor that has re-leased the same row. See migration
/// 0032 for the column and the `ack_done_from_dead_lease_*` tests
/// in `crawler_jobs.rs` for the race the token closes.
pub lease_generation: i64,
}
/// Deterministic exponential backoff base for `ack_failed` retries.
@@ -112,14 +132,17 @@ fn backoff_for(attempts: i32) -> Duration {
/// `Skipped`. The slot frees again once the previous job leaves the
/// in-flight states (done/failed/dead), so a re-enqueue after a force
/// refetch succeeds.
pub async fn enqueue(pool: &PgPool, payload: &JobPayload) -> sqlx::Result<EnqueueResult> {
pub async fn enqueue<'e, E>(executor: E, payload: &JobPayload) -> sqlx::Result<EnqueueResult>
where
E: sqlx::PgExecutor<'e>,
{
let json = serde_json::to_value(payload).expect("JobPayload is always serializable");
let id: Option<Uuid> = sqlx::query_scalar(
"INSERT INTO crawler_jobs (payload) VALUES ($1) \
ON CONFLICT DO NOTHING RETURNING id",
)
.bind(json)
.fetch_optional(pool)
.fetch_optional(executor)
.await?;
Ok(match id {
Some(id) => EnqueueResult::Inserted(id),
@@ -147,7 +170,7 @@ pub async fn lease(
lease_duration: Duration,
) -> sqlx::Result<Vec<Lease>> {
let lease_ms: i64 = lease_duration.as_millis().min(i64::MAX as u128) as i64;
let rows: Vec<(Uuid, serde_json::Value, i32, i32)> = sqlx::query_as(
let rows: Vec<(Uuid, serde_json::Value, i32, i32, i64)> = sqlx::query_as(
r#"
WITH leased AS (
SELECT id FROM crawler_jobs
@@ -161,11 +184,12 @@ pub async fn lease(
UPDATE crawler_jobs j
SET state = 'running',
attempts = j.attempts + 1,
lease_generation = j.lease_generation + 1,
leased_until = now() + ($3::bigint || ' milliseconds')::interval,
updated_at = now()
FROM leased l
WHERE j.id = l.id
RETURNING j.id, j.payload, j.attempts, j.max_attempts
RETURNING j.id, j.payload, j.attempts, j.max_attempts, j.lease_generation
"#,
)
.bind(kind_filter)
@@ -174,8 +198,56 @@ pub async fn lease(
.fetch_all(pool)
.await?;
decode_leases(rows)
}
/// Like [`lease`] but matches any of several `payload->>'kind'` values. The
/// crawl worker uses this to drain both `sync_chapter_content` and
/// `sync_manga` from one loop (both serialize on the single browser); the
/// analysis daemon keeps using single-kind [`lease`] so it never contends.
pub async fn lease_kinds(
pool: &PgPool,
kinds: &[&str],
max: i64,
lease_duration: Duration,
) -> sqlx::Result<Vec<Lease>> {
let lease_ms: i64 = lease_duration.as_millis().min(i64::MAX as u128) as i64;
let kinds_vec: Vec<String> = kinds.iter().map(|s| s.to_string()).collect();
let rows: Vec<(Uuid, serde_json::Value, i32, i32, i64)> = sqlx::query_as(
r#"
WITH leased AS (
SELECT id FROM crawler_jobs
WHERE (state = 'pending' OR (state = 'running' AND leased_until < now()))
AND scheduled_at <= now()
AND payload->>'kind' = ANY($1)
ORDER BY scheduled_at, created_at
LIMIT $2
FOR UPDATE SKIP LOCKED
)
UPDATE crawler_jobs j
SET state = 'running',
attempts = j.attempts + 1,
lease_generation = j.lease_generation + 1,
leased_until = now() + ($3::bigint || ' milliseconds')::interval,
updated_at = now()
FROM leased l
WHERE j.id = l.id
RETURNING j.id, j.payload, j.attempts, j.max_attempts, j.lease_generation
"#,
)
.bind(&kinds_vec)
.bind(max)
.bind(lease_ms)
.fetch_all(pool)
.await?;
decode_leases(rows)
}
fn decode_leases(
rows: Vec<(Uuid, serde_json::Value, i32, i32, i64)>,
) -> sqlx::Result<Vec<Lease>> {
let mut leases = Vec::with_capacity(rows.len());
for (id, payload_json, attempts, max_attempts) in rows {
for (id, payload_json, attempts, max_attempts, lease_generation) in rows {
let payload: JobPayload = serde_json::from_value(payload_json).map_err(|e| {
sqlx::Error::Decode(format!("invalid JobPayload JSON for job {id}: {e}").into())
})?;
@@ -184,6 +256,7 @@ pub async fn lease(
payload,
attempts,
max_attempts,
lease_generation,
});
}
Ok(leases)
@@ -202,42 +275,52 @@ pub async fn lease(
pub async fn renew(
pool: &PgPool,
lease_id: Uuid,
lease_generation: i64,
lease_duration: Duration,
) -> sqlx::Result<bool> {
let lease_ms: i64 = lease_duration.as_millis().min(i64::MAX as u128) as i64;
let res = sqlx::query(
"UPDATE crawler_jobs \
SET leased_until = now() + ($2::bigint || ' milliseconds')::interval, \
SET leased_until = now() + ($3::bigint || ' milliseconds')::interval, \
updated_at = now() \
WHERE id = $1 AND state = 'running'",
WHERE id = $1 AND lease_generation = $2 AND state = 'running'",
)
.bind(lease_id)
.bind(lease_generation)
.bind(lease_ms)
.execute(pool)
.await?;
Ok(res.rows_affected() > 0)
}
/// Mark a leased job as successfully completed. The `state = 'running'`
/// predicate guards against a late ack from a worker whose lease expired
/// and was already re-leased by another worker: without it, the late ack
/// would clobber the new lease's `state` and `leased_until`. `rows_affected
/// == 0` means we lost the lease — surfaced as a warn rather than an
/// error because the new lease holder is doing real work; the late ack
/// just has to step aside.
pub async fn ack_done(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
/// Mark a leased job as successfully completed. The
/// `(lease_generation, state = 'running')` predicate guards against a
/// late ack from a *specific* dead lease — the case where the worker's
/// lease was released or expired and a successor re-leased the same id.
/// Before migration 0032 the guard was state-only, so a late ack from
/// the original could still match (state was running again, owned by the
/// successor) and clobber. Scoping to the original's `lease_generation`
/// makes the match fail cleanly. `rows_affected == 0` is surfaced as a
/// warn — the successor is doing real work; the late ack steps aside.
pub async fn ack_done(
pool: &PgPool,
lease_id: Uuid,
lease_generation: i64,
) -> sqlx::Result<()> {
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'done', leased_until = NULL, updated_at = now() \
WHERE id = $1 AND state = 'running'",
WHERE id = $1 AND lease_generation = $2 AND state = 'running'",
)
.bind(lease_id)
.bind(lease_generation)
.execute(pool)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"ack_done: lease no longer running — likely re-leased by another worker; skipping update"
lease_generation,
"ack_done: lease no longer current — generation mismatch (force-analyzed, reclaimed, or re-leased); skipping update"
);
}
Ok(())
@@ -254,15 +337,17 @@ pub async fn ack_failed(
error: &str,
attempts: i32,
max_attempts: i32,
lease_generation: i64,
) -> sqlx::Result<()> {
let res = if attempts >= max_attempts {
sqlx::query(
"UPDATE crawler_jobs \
SET state = 'dead', last_error = $2, leased_until = NULL, updated_at = now() \
WHERE id = $1 AND state = 'running'",
WHERE id = $1 AND lease_generation = $3 AND state = 'running'",
)
.bind(lease_id)
.bind(error)
.bind(lease_generation)
.execute(pool)
.await?
} else {
@@ -272,43 +357,93 @@ pub async fn ack_failed(
SET state = 'pending', last_error = $2, leased_until = NULL, \
scheduled_at = now() + ($3::bigint || ' milliseconds')::interval, \
updated_at = now() \
WHERE id = $1 AND state = 'running'",
WHERE id = $1 AND lease_generation = $4 AND state = 'running'",
)
.bind(lease_id)
.bind(error)
.bind(backoff_ms)
.bind(lease_generation)
.execute(pool)
.await?
};
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"ack_failed: lease no longer running — likely re-leased by another worker; skipping update"
lease_generation,
"ack_failed: lease no longer current — generation mismatch (force-analyzed, reclaimed, or re-leased); skipping update"
);
}
Ok(())
}
/// Return a leased job to `pending` without burning a retry attempt.
/// Used on graceful shutdown and on session-expired aborts where the
/// failure isn't the job's fault. See [`ack_done`] for the
/// `state = 'running'` guard rationale — important here because
/// `attempts - 1` would otherwise spuriously decrement the new lease's
/// attempt count.
pub async fn release(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
/// Used by workers on graceful shutdown and on session-expired aborts
/// where the failure isn't the job's fault. The
/// `(lease_generation, state = 'running')` guard scopes the release to
/// the *specific* lease the caller was holding — otherwise a late
/// release could clobber a successor's lease (drop the row back to
/// pending mid-dispatch). Also bumps `lease_generation` so any
/// subsequent ack from this same lease finds nothing.
///
/// For force-analyze and other external paths that release a
/// running lease without holding a [`Lease`] struct, use
/// [`release_unowned`] instead.
pub async fn release(
pool: &PgPool,
lease_id: Uuid,
lease_generation: i64,
) -> sqlx::Result<()> {
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'pending', leased_until = NULL, \
attempts = GREATEST(0, attempts - 1), updated_at = now() \
WHERE id = $1 AND state = 'running'",
attempts = GREATEST(0, attempts - 1), \
lease_generation = lease_generation + 1, \
updated_at = now() \
WHERE id = $1 AND lease_generation = $2 AND state = 'running'",
)
.bind(lease_id)
.bind(lease_generation)
.execute(pool)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"release: lease no longer running — likely re-leased by another worker; skipping update"
lease_generation,
"release: lease no longer current — generation mismatch (force-analyzed, reclaimed, or re-leased); skipping update"
);
}
Ok(())
}
/// Release a `running` lease without owning it: caller has only a
/// job id, not a [`Lease`] struct. Used by force-analyze (admin
/// click that drops an in-flight lease so a refreshed payload is
/// picked up) and ops tools.
///
/// Unconditionally matches the row by id and bumps
/// `lease_generation` so the dropped lease's pending ack from
/// the original worker becomes a no-op. Returns the attempt to
/// `pending` and refunds the retry attempt (the operator click
/// isn't a job-level failure).
pub async fn release_unowned<'e, E>(executor: E, lease_id: Uuid) -> sqlx::Result<()>
where
E: sqlx::PgExecutor<'e>,
{
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'pending', leased_until = NULL, \
attempts = GREATEST(0, attempts - 1), \
lease_generation = lease_generation + 1, \
updated_at = now() \
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.execute(executor)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"release_unowned: row not running — likely already completed or never leased; skipping"
);
}
Ok(())
@@ -340,7 +475,9 @@ pub async fn reclaim_orphaned(pool: &PgPool) -> sqlx::Result<u64> {
let result = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'pending', leased_until = NULL, \
attempts = GREATEST(0, attempts - 1), updated_at = now() \
attempts = GREATEST(0, attempts - 1), \
lease_generation = lease_generation + 1, \
updated_at = now() \
WHERE state = 'running' AND leased_until < now()",
)
.execute(pool)
@@ -348,22 +485,43 @@ pub async fn reclaim_orphaned(pool: &PgPool) -> sqlx::Result<u64> {
Ok(result.rows_affected())
}
/// Delete `done` jobs whose `updated_at` is older than `retention_days`
/// days. `0` disables the reaper without touching the table. Returns the
/// number of rows removed.
pub async fn reap_done(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
/// Delete **terminal** jobs (`done` or `dead`) whose `updated_at` is older
/// than `retention_days` days. Both states are end-of-life — `done`
/// succeeded, `dead` exhausted its retries — and neither is ever leased
/// again, so reaping only `done` left `dead` rows to accumulate forever.
/// `pending` / `running` are active and never touched. `0` disables the
/// reaper without touching the table. Returns the number of rows removed.
pub async fn reap_terminal(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
if retention_days == 0 {
return Ok(0);
}
let result = sqlx::query(
"DELETE FROM crawler_jobs \
WHERE state = 'done' \
AND updated_at < now() - ($1::bigint || ' days')::interval",
)
.bind(retention_days as i64)
.execute(pool)
.await?;
Ok(result.rows_affected())
// Delete in bounded batches rather than one statement. A single unbatched
// DELETE over an unbounded backlog takes a long-held lock and one huge
// transaction; batching keeps each delete short (index-backed by
// crawler_jobs_terminal_reap_idx, 0040) so the reaper never pins the table
// (audit M5). Loop until a batch comes back short.
const BATCH: i64 = 5_000;
let mut total: u64 = 0;
loop {
let result = sqlx::query(
"DELETE FROM crawler_jobs \
WHERE ctid IN ( \
SELECT ctid FROM crawler_jobs \
WHERE state IN ('done', 'dead') \
AND updated_at < now() - ($1::bigint || ' days')::interval \
LIMIT $2 )",
)
.bind(retention_days as i64)
.bind(BATCH)
.execute(pool)
.await?;
let n = result.rows_affected();
total += n;
if n < BATCH as u64 {
break;
}
}
Ok(total)
}
#[cfg(test)]
@@ -393,6 +551,60 @@ mod tests {
}
}
#[test]
fn sync_manga_payload_round_trips_with_url_and_title() {
let payload = JobPayload::SyncManga {
source_id: "target".into(),
source_manga_key: "foo".into(),
url: "https://target.example/manga/foo".into(),
title: "Foo Title".into(),
};
let json = serde_json::to_value(&payload).unwrap();
assert_eq!(json["kind"], KIND_SYNC_MANGA);
assert_eq!(json["source_id"], "target");
assert_eq!(json["source_manga_key"], "foo");
assert_eq!(json["url"], "https://target.example/manga/foo");
assert_eq!(json["title"], "Foo Title");
match serde_json::from_value::<JobPayload>(json).unwrap() {
JobPayload::SyncManga {
source_id,
source_manga_key,
url,
title,
} => {
assert_eq!(source_id, "target");
assert_eq!(source_manga_key, "foo");
assert_eq!(url, "https://target.example/manga/foo");
assert_eq!(title, "Foo Title");
}
other => panic!("expected SyncManga, got {other:?}"),
}
}
#[test]
fn sync_manga_payload_defaults_missing_url_and_title() {
// A row that predates the url/title fields must still decode.
let legacy = serde_json::json!({
"kind": "sync_manga",
"source_id": "target",
"source_manga_key": "foo",
});
match serde_json::from_value::<JobPayload>(legacy).unwrap() {
JobPayload::SyncManga {
url,
title,
source_manga_key,
..
} => {
assert_eq!(source_manga_key, "foo");
assert_eq!(url, "");
assert_eq!(title, "");
}
other => panic!("expected SyncManga, got {other:?}"),
}
}
#[test]
fn backoff_base_grows_exponentially_and_caps_at_one_hour() {
// attempts == 1 → 60s, doubling each step.

View File

@@ -19,10 +19,12 @@ pub mod content;
pub mod daemon;
pub mod detect;
pub mod diff;
pub mod intercept;
pub mod jobs;
pub mod nav;
pub mod pipeline;
pub mod rate_limit;
pub mod reconcile;
pub mod resync;
pub mod safety;
pub mod session;

View File

@@ -85,6 +85,28 @@ pub async fn wait_for_selector(
/// under a second.
pub const SELECTOR_TIMEOUT: Duration = Duration::from_secs(10);
/// Run `body` to completion, then run `close` — on *every* exit path,
/// including an early `?` / `return` inside `body`. chromiumoxide's `Page`
/// does **not** close its CDP target on drop, so a fetch helper that opens a
/// page with `browser.new_page(...)` and then bails via `?` on a nav /
/// content-read error would leak a browser tab for the process's lifetime.
/// Wrapping the fallible work in `close_after` — with `close` built from a
/// clone of the page — guarantees the tab is closed regardless of how `body`
/// returns.
///
/// Both arguments are pre-built futures, so this stays generic over the
/// body's return type (the anyhow and `PageError` fetch paths both use it)
/// and references no browser types — which also lets it be unit-tested
/// without standing up a real `Page`.
pub(crate) async fn close_after<R>(
close: impl std::future::Future<Output = ()>,
body: impl std::future::Future<Output = R>,
) -> R {
let result = body.await;
close.await;
result
}
impl NavError {
/// Does this navigation error indicate the underlying Chromium
/// process has died or its CDP connection has dropped? Used by the
@@ -150,6 +172,41 @@ mod tests {
assert!(result.is_err(), "expected Elapsed on a hung future");
}
#[tokio::test]
async fn close_after_runs_close_and_returns_value_on_ok() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let closed = Arc::new(AtomicBool::new(false));
let c = closed.clone();
let out: anyhow::Result<i32> = close_after(
async move { c.store(true, Ordering::SeqCst) },
async { Ok(7) },
)
.await;
assert_eq!(out.unwrap(), 7);
assert!(closed.load(Ordering::SeqCst), "close must run on the happy path");
}
#[tokio::test]
async fn close_after_runs_close_even_when_body_errs() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let closed = Arc::new(AtomicBool::new(false));
let c = closed.clone();
// This is the leak the fix targets: the body bails before it could
// close the page itself, and `close_after` must still close it.
let out: anyhow::Result<()> =
close_after(async move { c.store(true, Ordering::SeqCst) }, async {
anyhow::bail!("nav failed")
})
.await;
assert!(out.is_err());
assert!(
closed.load(Ordering::SeqCst),
"the page must be closed even when the body returns Err"
);
}
#[test]
fn nav_error_timeout_message_includes_duration() {
let e = NavError::Timeout(Duration::from_secs(30));
@@ -164,8 +221,7 @@ mod tests {
#[test]
fn anyhow_with_nav_timeout_in_chain_is_flagged() {
let inner: Result<(), NavError> = Err(NavError::Timeout(NAV_TIMEOUT));
let outer = inner.unwrap_err();
let outer = NavError::Timeout(NAV_TIMEOUT);
let wrapped: anyhow::Error =
anyhow::Error::new(outer).context("wait for chapter nav");
assert!(anyhow_looks_browser_dead(&wrapped));

View File

@@ -76,6 +76,213 @@ pub(crate) fn should_abort_pass(consecutive: u32, threshold: u32) -> bool {
threshold > 0 && consecutive >= threshold
}
/// Success outcome of [`process_manga_ref`].
pub(crate) struct RefProcessed {
pub manga_id: Uuid,
pub status: UpsertStatus,
pub chapters_new: Option<usize>,
pub cover_fetched: bool,
}
/// Why [`process_manga_ref`] produced no upsert.
pub(crate) enum RefError {
/// `fetch_manga` itself failed. Counts toward the consecutive-failure
/// breaker in the metadata pass; the `SyncManga` worker treats it as a
/// retryable job failure.
Fetch(anyhow::Error),
/// The detail fetched but the ref was skipped — partial-render guard or
/// an upsert error. The fetch *succeeded*, so the breaker resets, but no
/// manga was upserted.
Skip(anyhow::Error),
}
/// Run fetch → partial-render guard → upsert → cover → chapter-sync for a
/// single discovered ref. Extracted from [`run_metadata_pass`] so the
/// `SyncManga` worker drives the identical per-manga work and the two paths
/// can't drift. Records the detail/cover crawl metrics internally; the
/// caller owns the discovered/seen/stop/breaker bookkeeping.
#[allow(clippy::too_many_arguments)]
pub(crate) async fn process_manga_ref(
ctx: &FetchContext<'_>,
source: &TargetSource,
db: &PgPool,
storage: &dyn Storage,
http: &reqwest::Client,
rate: &HostRateLimiters,
r: &SourceMangaRef,
skip_chapters: bool,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
status: Option<&crate::crawler::status::StatusHandle>,
) -> Result<RefProcessed, RefError> {
let source_id = source.id();
let detail_started = std::time::Instant::now();
let manga = match source.fetch_manga(ctx, r).await {
Ok(m) => m,
Err(e) => {
// Detail-fetch timing (failed). manga_id is unknown — the manga
// was never upserted.
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_DETAIL,
None,
None,
"failed",
detail_started.elapsed().as_millis() as i64,
None,
Some(&format!("{e:#}")),
)
.await;
return Err(RefError::Fetch(e));
}
};
// Detail fetch succeeded; stash its duration to record once the
// manga_id is known (after upsert).
let detail_ms = detail_started.elapsed().as_millis() as i64;
// Partial-render guard: an empty chapter list paired with a prior count
// > 0 is overwhelmingly a chromium snapshot taken between the wrapper
// render and its rows render. Treat as a transient skip (no upsert) so
// a later attempt retries. Skipped in `skip_chapters` mode because the
// parser returns an empty Vec there by design.
if !skip_chapters && manga.chapters.is_empty() {
match repo::crawler::live_chapter_count_for_source_manga(db, source_id, &r.source_manga_key)
.await
{
Ok(prior) if prior > 0 => {
return Err(RefError::Skip(anyhow::anyhow!(
"fetch_manga returned empty chapters but prior count {prior} > 0; \
treating as partial-render transient"
)));
}
Ok(_) => {}
Err(e) => {
// DB lookup failed — fail safe: skip rather than risk a
// soft-drop on a manga whose prior count we couldn't confirm.
return Err(RefError::Skip(
anyhow::Error::new(e).context("live_chapter_count_for_source_manga"),
));
}
}
}
let upsert = match repo::crawler::upsert_manga_from_source(db, source_id, &r.url, &manga).await {
Ok(u) => u,
Err(e) => {
return Err(RefError::Skip(
anyhow::Error::new(e).context("upsert_manga_from_source"),
));
}
};
// Detail-fetch timing (ok), now that we have the manga_id.
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_DETAIL,
Some(upsert.manga_id),
None,
"ok",
detail_ms,
None,
None,
)
.await;
tracing::info!(
key = %manga.source_manga_key,
manga_id = %upsert.manga_id,
status = ?upsert.status,
title = %manga.title,
"manga upserted"
);
// Cover image: download when missing in storage or when metadata
// signaled an update (cover URL is part of metadata_hash, so Updated
// implies the URL may have moved). Failures are non-fatal.
let mut cover_fetched = false;
let needs_cover =
upsert.cover_image_path.is_none() || matches!(upsert.status, UpsertStatus::Updated);
if needs_cover {
if let Some(cover_url) = manga.cover_url.as_deref() {
// RAII: the guard clears `current_cover` on every exit path.
let _cover_guard = status.map(|s| {
s.begin_cover(crate::crawler::status::CoverTarget {
manga_id: upsert.manga_id,
manga_title: manga.title.clone(),
})
});
let cover_started = std::time::Instant::now();
let cover_result = download_and_store_cover(
db,
storage,
http,
rate,
&r.url,
upsert.manga_id,
cover_url,
allowlist,
max_image_bytes,
)
.await;
let cover_ms = cover_started.elapsed().as_millis() as i64;
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_COVER,
Some(upsert.manga_id),
None,
if cover_result.is_ok() { "ok" } else { "failed" },
cover_ms,
None,
cover_result.as_ref().err().map(|e| format!("{e:#}")).as_deref(),
)
.await;
match cover_result {
Ok(()) => cover_fetched = true,
Err(e) => tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"cover download failed"
),
}
}
}
// Chapter sync. `None` (skip_chapters mode, or a logged-and-swallowed
// sync error) refuses to stop on this manga because we can't confirm
// "no new chapters."
let chapters_new: Option<usize> = if skip_chapters {
None
} else {
match repo::crawler::sync_manga_chapters(db, source_id, upsert.manga_id, &manga.chapters)
.await
{
Ok(diff) => {
tracing::info!(
manga_id = %upsert.manga_id,
new = diff.new,
refreshed = diff.refreshed,
dropped = diff.dropped,
"chapters synced"
);
Some(diff.new)
}
Err(e) => {
tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"chapter sync failed"
);
None
}
}
};
Ok(RefProcessed {
manga_id: upsert.manga_id,
status: upsert.status,
chapters_new,
cover_fetched,
})
}
/// Runs the discover → fetch → upsert → cover → chapter-list-diff pipeline
/// for the target source. Pure metadata; chapter content is enqueued as
/// separate `SyncChapterContent` jobs by the caller after this returns.
@@ -244,32 +451,48 @@ pub async fn run_metadata_pass(
key = %r.source_manga_key,
"fetching metadata"
);
let detail_started = std::time::Instant::now();
let manga = match source.fetch_manga(&ctx, &r).await {
Ok(m) => {
match process_manga_ref(
&ctx,
&source,
db,
storage,
http,
rate,
&r,
skip_chapters,
allowlist,
max_image_bytes,
status,
)
.await
{
Ok(p) => {
consecutive_failures = 0;
m
stats.upserted += 1;
if p.cover_fetched {
stats.covers_fetched += 1;
}
// Record success in the dedup set. Cover and chapter-sync
// failures inside process_manga_ref are non-fatal and
// don't roll this back — metadata is the durable source
// of truth for the dedup.
seen.insert(r.source_manga_key.clone());
if should_stop(was_clean, p.status, p.chapters_new) {
hit_stop_condition = true;
tracing::info!(
key = %r.source_manga_key,
"stop condition met (Unchanged metadata + 0 new chapters); halting walk"
);
break 'outer;
}
}
Err(e) => {
Err(RefError::Fetch(e)) => {
tracing::warn!(
key = %r.source_manga_key,
url = %r.url,
error = ?e,
"fetch_manga failed"
);
// Detail-fetch timing (failed). manga_id is unknown — the
// manga was never upserted.
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_DETAIL,
None,
None,
"failed",
detail_started.elapsed().as_millis() as i64,
None,
Some(&format!("{e:#}")),
)
.await;
stats.mangas_failed += 1;
consecutive_failures += 1;
if should_abort_pass(consecutive_failures, max_consecutive_failures) {
@@ -282,191 +505,19 @@ pub async fn run_metadata_pass(
);
break 'outer;
}
continue;
}
};
// Detail fetch succeeded; stash its duration to record once the
// manga_id is known (after upsert).
let detail_ms = detail_started.elapsed().as_millis() as i64;
// Partial-render guard: an empty chapter list paired with a
// prior count > 0 is overwhelmingly a chromium snapshot
// taken between the #chapter_table wrapper render and its
// rows render. The wait_for_selector wait in `navigate`
// narrows this window but cannot close it for slow renders
// beyond the selector budget. Treat as a transient failure
// here — skip upsert, skip seen.insert — so the next batch
// (or the next tick) retries. Skipped in `skip_chapters`
// mode because the parser is configured to return an empty
// Vec by design there.
if !skip_chapters && manga.chapters.is_empty() {
match repo::crawler::live_chapter_count_for_source_manga(
db, source_id, &r.source_manga_key,
)
.await
{
Ok(prior) if prior > 0 => {
tracing::warn!(
key = %r.source_manga_key,
url = %r.url,
prior_chapter_count = prior,
"fetch_manga returned empty chapters but prior count > 0; treating as partial-render transient and skipping"
);
stats.mangas_failed += 1;
continue;
}
Ok(_) => {}
Err(e) => {
// DB lookup failed — fail safe: skip rather
// than risk a soft-drop on a manga whose prior
// count we couldn't confirm.
tracing::warn!(
key = %r.source_manga_key,
error = ?e,
"live_chapter_count_for_source_manga failed; skipping cautiously"
);
stats.mangas_failed += 1;
continue;
}
}
}
let upsert = match repo::crawler::upsert_manga_from_source(
db, source_id, &r.url, &manga,
)
.await
{
Ok(u) => u,
Err(e) => {
tracing::error!(
Err(RefError::Skip(e)) => {
// Fetch succeeded (so the breaker resets) but the ref was
// skipped — partial render or upsert error. Left out of
// `seen` so a reappearance in a later batch retries.
tracing::warn!(
key = %r.source_manga_key,
error = ?e,
"upsert_manga_from_source failed"
"manga ref skipped (partial-render or upsert error)"
);
consecutive_failures = 0;
stats.mangas_failed += 1;
continue;
}
};
stats.upserted += 1;
// Detail-fetch timing (ok), now that we have the manga_id.
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_DETAIL,
Some(upsert.manga_id),
None,
"ok",
detail_ms,
None,
None,
)
.await;
// Record success in the dedup set. Cover and chapter-sync
// failures below are non-fatal and don't roll this back —
// metadata is the durable source of truth for the dedup.
seen.insert(r.source_manga_key.clone());
tracing::info!(
key = %manga.source_manga_key,
manga_id = %upsert.manga_id,
status = ?upsert.status,
title = %manga.title,
"manga upserted"
);
// Cover image: download when missing in storage or when metadata
// signaled an update (cover URL is part of metadata_hash, so
// Updated implies the URL may have moved). Failures are non-fatal.
let needs_cover = upsert.cover_image_path.is_none()
|| matches!(upsert.status, repo::crawler::UpsertStatus::Updated);
if needs_cover {
if let Some(cover_url) = manga.cover_url.as_deref() {
// RAII: the guard clears `current_cover` on every
// exit path (success, panic, future early-return).
// Mirrors the chapter-side ChapterGuard.
let _cover_guard = status.map(|s| {
s.begin_cover(crate::crawler::status::CoverTarget {
manga_id: upsert.manga_id,
manga_title: manga.title.clone(),
})
});
let cover_started = std::time::Instant::now();
let cover_result = download_and_store_cover(
db,
storage,
http,
rate,
&r.url,
upsert.manga_id,
cover_url,
allowlist,
max_image_bytes,
)
.await;
let cover_ms = cover_started.elapsed().as_millis() as i64;
let _ = repo::crawl_metrics::record(
db,
repo::crawl_metrics::OP_MANGA_COVER,
Some(upsert.manga_id),
None,
if cover_result.is_ok() { "ok" } else { "failed" },
cover_ms,
None,
cover_result.as_ref().err().map(|e| format!("{e:#}")).as_deref(),
)
.await;
match cover_result {
Ok(()) => stats.covers_fetched += 1,
Err(e) => tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"cover download failed"
),
}
}
}
// Chapter sync. `chapters_new` feeds the stop check below:
// `None` (skip_chapters mode, or a logged-and-swallowed sync
// error) refuses to stop on this manga because we can't
// confirm "no new chapters."
let chapters_new: Option<usize> = if skip_chapters {
None
} else {
match repo::crawler::sync_manga_chapters(
db,
source_id,
upsert.manga_id,
&manga.chapters,
)
.await
{
Ok(diff) => {
tracing::info!(
manga_id = %upsert.manga_id,
new = diff.new,
refreshed = diff.refreshed,
dropped = diff.dropped,
"chapters synced"
);
Some(diff.new)
}
Err(e) => {
tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"chapter sync failed"
);
None
}
}
};
if should_stop(was_clean, upsert.status, chapters_new) {
hit_stop_condition = true;
tracing::info!(
key = %manga.source_manga_key,
"stop condition met (Unchanged metadata + 0 new chapters); halting walk"
);
break 'outer;
}
}
}

View File

@@ -44,6 +44,25 @@ impl RateLimiter {
}
}
/// Once the per-host map grows past this many entries, the next `wait_for`
/// sweeps out hosts idle longer than [`IDLE_EVICT_AFTER`]. A long crawl that
/// touches thousands of distinct CDN shards would otherwise retain one bucket
/// per host for the daemon's whole lifetime. Far above any single source's
/// real host count, so the sweep is rare.
const MAX_TRACKED_HOSTS: usize = 1024;
/// A host bucket untouched for at least this long is evicted on the next
/// over-cap sweep. Generous: a host still being crawled is touched every
/// `interval`, so only genuinely-finished hosts age out.
const IDLE_EVICT_AFTER: Duration = Duration::from_secs(3600);
/// A host's bucket plus when it was last used, so idle entries can be evicted.
#[derive(Debug)]
struct HostEntry {
limiter: Arc<Mutex<RateLimiter>>,
last_used: Instant,
}
/// Per-host rate limiter map. The outer `Mutex<HashMap>` is held only
/// during the entry-or-insert + Arc clone; the per-host `Mutex<RateLimiter>`
/// is held during the actual `wait().await`. So N workers calling
@@ -54,7 +73,7 @@ impl RateLimiter {
pub struct HostRateLimiters {
default_interval: Duration,
overrides: HashMap<String, Duration>,
map: Mutex<HashMap<String, Arc<Mutex<RateLimiter>>>>,
map: Mutex<HashMap<String, HostEntry>>,
}
impl HostRateLimiters {
@@ -82,20 +101,37 @@ impl HostRateLimiters {
.ok_or_else(|| anyhow::anyhow!("no host in url: {url}"))?;
let limiter = {
let mut map = self.map.lock().await;
map.entry(host.clone())
.or_insert_with(|| {
let interval = self
.overrides
.get(&host)
.copied()
.unwrap_or(self.default_interval);
Arc::new(Mutex::new(RateLimiter::new(interval)))
})
.clone()
let now = Instant::now();
// Bound the map: when it grows past the soft cap, drop hosts that
// have been idle past the TTL. Only fires over-cap, so the common
// path is a plain lookup. A host still being crawled is touched
// every `interval` and so never ages out.
if map.len() >= MAX_TRACKED_HOSTS {
map.retain(|_, e| now.duration_since(e.last_used) < IDLE_EVICT_AFTER);
}
let entry = map.entry(host.clone()).or_insert_with(|| {
let interval = self
.overrides
.get(&host)
.copied()
.unwrap_or(self.default_interval);
HostEntry {
limiter: Arc::new(Mutex::new(RateLimiter::new(interval))),
last_used: now,
}
});
entry.last_used = now;
entry.limiter.clone()
};
limiter.lock().await.wait().await;
Ok(())
}
/// Number of host buckets currently tracked. Test/observability hook.
#[cfg(test)]
pub(crate) async fn tracked_hosts(&self) -> usize {
self.map.lock().await.len()
}
}
// `host_of` was duplicated across session/rate_limit/pipeline; the
@@ -165,6 +201,46 @@ mod tests {
);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_evict_idle_hosts_over_cap() {
// Fill the map to the soft cap with distinct hosts (first call to a
// fresh host never sleeps, so this is fast even under real time).
let rl = HostRateLimiters::new(Duration::from_millis(1));
for i in 0..MAX_TRACKED_HOSTS {
rl.wait_for(&format!("https://host{i}.example/x")).await.unwrap();
}
assert_eq!(rl.tracked_hosts().await, MAX_TRACKED_HOSTS);
// Let every tracked host age past the idle TTL, then touch one new
// host: the over-cap sweep should evict all the idle ones, leaving
// only the freshly-inserted entry.
tokio::time::sleep(IDLE_EVICT_AFTER + Duration::from_secs(1)).await;
rl.wait_for("https://newcomer.example/y").await.unwrap();
assert_eq!(
rl.tracked_hosts().await,
1,
"idle hosts should be swept once the map is over the cap"
);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_keep_recently_used_hosts() {
// A host touched within the TTL must survive a sweep so active crawls
// aren't reset. Fill to the cap, then re-touch one host right before
// adding a newcomer that triggers the sweep.
let rl = HostRateLimiters::new(Duration::from_millis(1));
for i in 0..MAX_TRACKED_HOSTS {
rl.wait_for(&format!("https://host{i}.example/x")).await.unwrap();
}
tokio::time::sleep(IDLE_EVICT_AFTER + Duration::from_secs(1)).await;
// Re-touch host0 so it's recent again.
rl.wait_for("https://host0.example/x").await.unwrap();
// Newcomer triggers the sweep (map is at cap+1 conceptually).
rl.wait_for("https://newcomer.example/y").await.unwrap();
// host0 (recent) + newcomer survive; the rest aged out.
assert_eq!(rl.tracked_hosts().await, 2);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_honor_overrides() {
let rl = HostRateLimiters::new(Duration::from_millis(1000))

View File

@@ -0,0 +1,260 @@
//! Reconcile pass — find mangas the metadata pass missed.
//!
//! The interleaved metadata pass ([`crate::crawler::pipeline`]) walks the
//! source list newest-first and visits each manga inline, which makes it
//! vulnerable to list drift: a manga can slip a pagination slot while the
//! walk spends minutes on detail work, and never get upserted.
//!
//! Reconcile fixes that with a cheap, full, unconditional list-only walk
//! (refs only — no `fetch_manga`, no early stop), set-diffs the walked keys
//! against `manga_sources`, and enqueues the strictly-missing ones as
//! `SyncManga` jobs for the crawl worker to visit. A set-diff is immune to
//! drift: a manga only has to appear *somewhere* in the list, not survive a
//! specific slot during a slow walk.
//!
//! Identity matches the DB exactly because both sides derive the key the
//! same way: the walker's refs carry `source_manga_key =
//! derive_key_from_url(list_href)`, which is precisely what
//! `manga_sources.source_manga_key` stores.
use std::collections::HashSet;
use anyhow::Context;
use sqlx::PgPool;
use crate::crawler::browser_manager::BrowserManager;
use crate::crawler::jobs::{self, EnqueueResult, JobPayload};
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::source::target::TargetSource;
use crate::crawler::source::{FetchContext, Source, SourceMangaRef};
use crate::crawler::status::Phase;
use crate::repo;
use crate::repo::crawler::ensure_source;
/// Counters surfaced when a reconcile pass finishes.
#[derive(Debug, Default, Clone, Copy)]
pub struct ReconcileStats {
/// Distinct refs collected from the full list walk.
pub walked: usize,
/// Walked keys with no `manga_sources` row (strict `NOT EXISTS`).
pub missing: usize,
/// Missing mangas newly enqueued as `SyncManga` jobs this pass.
pub enqueued: usize,
/// Missing mangas skipped because a blocking (pending/running/dead)
/// `SyncManga` job already exists, or the enqueue lost an insert race.
pub skipped: usize,
}
/// Pure set-diff: which walked refs should be enqueued. A ref is selected
/// when its key is neither already in the DB (`existing`) nor already has a
/// blocking job (`blocked`). Repeated keys in `walked` (source index drift
/// can surface the same manga twice) are de-duplicated — the first ref wins.
pub(crate) fn select_missing<'a>(
walked: &'a [SourceMangaRef],
existing: &HashSet<String>,
blocked: &HashSet<String>,
) -> Vec<&'a SourceMangaRef> {
let mut seen = HashSet::new();
let mut out = Vec::new();
for r in walked {
if existing.contains(&r.source_manga_key) || blocked.contains(&r.source_manga_key) {
continue;
}
if seen.insert(r.source_manga_key.clone()) {
out.push(r);
}
}
out
}
/// Run a full list-only walk, diff against the DB, and enqueue missing
/// mangas as `SyncManga` jobs. Holds the (exclusive) browser lease only for
/// the cheap walk; the enqueued jobs are drained later by the crawl worker.
#[allow(clippy::too_many_arguments)]
pub async fn reconcile_missing(
browser_manager: &BrowserManager,
db: &PgPool,
rate: &HostRateLimiters,
start_url: &str,
status: Option<&crate::crawler::status::StatusHandle>,
tor: Option<&crate::crawler::tor::TorController>,
) -> anyhow::Result<ReconcileStats> {
let lease = browser_manager
.acquire()
.await
.context("acquire browser lease for reconcile pass")?;
let browser_ref: &chromiumoxide::Browser = &lease;
if let Some(s) = status {
s.set_phase(Phase::Reconciling {
walked: 0,
enqueued: 0,
})
.await;
}
// Chapter parsing is irrelevant here (we never call fetch_manga); the
// metadata-only constructor is intent-revealing and harmless.
let source = TargetSource::new(start_url.to_string()).without_chapter_parsing();
let ctx = FetchContext {
browser: browser_ref,
rate,
tor,
};
let source_id = source.id();
ensure_source(
db,
source_id,
"Target Site",
&crate::crawler::url_utils::origin_of(start_url).unwrap_or_else(|| start_url.to_string()),
)
.await
.context("ensure_source")?;
tracing::info!("starting reconcile pass (full list-only walk)");
let mut walker = source.discover(&ctx).await.context("discover failed")?;
// Full unconditional walk: collect every ref, no early stop. Yield the
// lease if a coordinated browser restart is pending so the drain isn't
// stalled — the missing set we've gathered so far is still enqueued.
let mut walked: Vec<SourceMangaRef> = Vec::new();
'outer: loop {
if browser_manager.is_restart_pending() {
tracing::info!("reconcile pass: browser restart pending — yielding partial walk");
break;
}
match walker.next_batch(&ctx).await? {
Some(batch) => {
for r in batch {
walked.push(r);
}
if let Some(s) = status {
s.set_phase(Phase::Reconciling {
walked: walked.len(),
enqueued: 0,
})
.await;
}
if browser_manager.is_restart_pending() {
tracing::info!(
"reconcile pass: browser restart pending mid-batch — yielding partial walk"
);
break 'outer;
}
}
None => break,
}
}
// Browser work is done — release before the (browser-free) diff+enqueue.
drop(lease);
let existing = repo::crawler::existing_source_keys(db, source_id)
.await
.context("existing_source_keys")?;
let blocked = repo::crawler::sync_manga_keys_with_blocking_job(db, source_id)
.await
.context("sync_manga_keys_with_blocking_job")?;
let missing = select_missing(&walked, &existing, &blocked);
let mut stats = ReconcileStats {
walked: walked.len(),
missing: missing.len(),
..Default::default()
};
for r in missing {
let payload = JobPayload::SyncManga {
source_id: source_id.to_string(),
source_manga_key: r.source_manga_key.clone(),
url: r.url.clone(),
title: r.title.clone(),
};
match jobs::enqueue(db, &payload).await {
Ok(EnqueueResult::Inserted(_)) => stats.enqueued += 1,
Ok(EnqueueResult::Skipped) => stats.skipped += 1,
Err(e) => {
tracing::warn!(key = %r.source_manga_key, error = ?e, "enqueue SyncManga failed");
stats.skipped += 1;
}
}
if let Some(s) = status {
s.set_phase(Phase::Reconciling {
walked: stats.walked,
enqueued: stats.enqueued,
})
.await;
}
}
tracing::info!(
walked = stats.walked,
missing = stats.missing,
enqueued = stats.enqueued,
skipped = stats.skipped,
"reconcile pass complete"
);
Ok(stats)
}
#[cfg(test)]
mod tests {
use super::*;
fn r(key: &str) -> SourceMangaRef {
SourceMangaRef {
source_manga_key: key.to_string(),
title: format!("Title {key}"),
url: format!("https://target.example/manga/{key}"),
}
}
fn set(keys: &[&str]) -> HashSet<String> {
keys.iter().map(|s| s.to_string()).collect()
}
#[test]
fn select_missing_returns_keys_not_in_manga_sources() {
let walked = vec![r("a"), r("b"), r("c")];
let existing = set(&["b"]);
let blocked = set(&[]);
let got: Vec<&str> = select_missing(&walked, &existing, &blocked)
.iter()
.map(|r| r.source_manga_key.as_str())
.collect();
assert_eq!(got, vec!["a", "c"]);
}
#[test]
fn select_missing_treats_dropped_rows_as_present() {
// The caller passes dropped keys in `existing` (existing_source_keys
// does not filter dropped_at), so a dropped manga is NOT re-enqueued.
let walked = vec![r("a"), r("dropped")];
let existing = set(&["dropped"]);
let got: Vec<&str> = select_missing(&walked, &existing, &set(&[]))
.iter()
.map(|r| r.source_manga_key.as_str())
.collect();
assert_eq!(got, vec!["a"]);
}
#[test]
fn select_missing_skips_keys_with_blocking_job() {
let walked = vec![r("a"), r("queued"), r("dead")];
let blocked = set(&["queued", "dead"]);
let got: Vec<&str> = select_missing(&walked, &set(&[]), &blocked)
.iter()
.map(|r| r.source_manga_key.as_str())
.collect();
assert_eq!(got, vec!["a"]);
}
#[test]
fn select_missing_dedups_repeated_walked_keys() {
// Index drift can surface the same manga twice in one walk.
let walked = vec![r("a"), r("a"), r("b")];
let got: Vec<&str> = select_missing(&walked, &set(&[]), &set(&[]))
.iter()
.map(|r| r.source_manga_key.as_str())
.collect();
assert_eq!(got, vec!["a", "b"]);
}
}

View File

@@ -81,6 +81,8 @@ pub struct RealResyncService {
pub rate: Arc<HostRateLimiters>,
pub download_allowlist: DownloadAllowlist,
pub max_image_bytes: usize,
/// Per-chapter image-count cap (see `CrawlerConfig::max_images_per_chapter`).
pub max_images_per_chapter: usize,
pub tor: Option<Arc<TorController>>,
}
@@ -133,28 +135,7 @@ impl ResyncService for RealResyncService {
.await
.with_context(|| format!("fetch_manga during resync of {manga_id}"))?;
// Partial-render guard: same logic as run_metadata_pass.
let source_id = source.id();
if !manga.chapters.is_empty() || {
let prior = repo::crawler::live_chapter_count_for_source_manga(
&self.db,
source_id,
&source_manga_key,
)
.await
.unwrap_or(0);
prior == 0
} {
// Either the new fetch surfaced chapters, or there were
// none before either — chapter sync is safe to run.
} else {
tracing::warn!(
%manga_id,
source_url = %source_url,
"resync_manga: fetch returned empty chapters but prior count > 0; skipping chapter sync to avoid soft-drop"
);
}
let upsert = repo::crawler::upsert_manga_from_source(
&self.db,
source_id,
@@ -199,6 +180,11 @@ impl ResyncService for RealResyncService {
)
.await
.unwrap_or(0);
// Partial-render guard (same logic as run_metadata_pass): only sync
// chapters when the fetch surfaced some, or the manga never had any.
// A fetch that returned empty while a prior count exists is almost
// certainly a partial render — skip the sync so we don't soft-drop the
// real chapters.
if !manga.chapters.is_empty() || prior_chapter_count == 0 {
match repo::crawler::sync_manga_chapters(
&self.db,
@@ -221,6 +207,12 @@ impl ResyncService for RealResyncService {
"resync_manga: chapter sync failed"
),
}
} else {
tracing::warn!(
%manga_id,
source_url = %source_url,
"resync_manga: fetch returned empty chapters but prior count > 0; skipping chapter sync to avoid soft-drop"
);
}
drop(lease);
@@ -256,6 +248,7 @@ impl ResyncService for RealResyncService {
true,
&self.download_allowlist,
self.max_image_bytes,
self.max_images_per_chapter,
self.tor.as_deref(),
// Admin resync isn't a daemon worker slot — no live status.
None,
@@ -277,6 +270,12 @@ impl ResyncService for RealResyncService {
SyncOutcome::SessionExpired => {
anyhow::bail!("source session expired — operator must refresh PHPSESSID")
}
// Unreachable here: resync already holds its own browser lease and
// `sync_chapter_content` never acquires one, so it can't report the
// browser unavailable. Handled defensively for exhaustiveness.
SyncOutcome::BrowserUnavailable => {
anyhow::bail!("crawler browser unavailable")
}
}
}
}

View File

@@ -31,11 +31,13 @@
//! URL string, and the byte accumulator is keyed off a generic stream.
//! Easy to unit-test without a live network or browser.
use std::net::IpAddr;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use anyhow::{bail, Context};
use bytes::BytesMut;
use futures_util::StreamExt;
use reqwest::dns::{Addrs, Name, Resolve, Resolving};
use reqwest::Url;
/// Default per-image download cap. A page image is generally <2 MiB;
@@ -124,6 +126,34 @@ impl DownloadAllowlist {
/// An empty allowlist rejects everything (the conservative default —
/// callers must explicitly allow the catalog and CDN hosts).
pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSafetyError> {
let url = ensure_public_target_inner(raw_url)?;
let lower_host = url
.host_str()
.expect("host validated by ensure_public_target_inner")
.to_ascii_lowercase();
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
}
/// Validate that an admin-supplied URL points at a publicly-routable target:
/// scheme is http or https, host is present, host isn't `localhost`, and (if
/// the host is an IP literal) it isn't loopback/private/link-local/CGNAT/etc.
///
/// Used to validate `endpoint`-style admin settings (crawler `start_url`,
/// analysis `endpoint`) where there is no per-deployment allowlist to consult,
/// but where a hostile or careless admin value (e.g. `http://169.254.169.254/`,
/// `http://127.0.0.1:5432/`) would let the worker pivot inside the deployment.
///
/// Note: DNS hostnames (`mangalord-vision`, `vision.internal.corp`) pass — the
/// check is on *literal* IP private-range strings only, so the documented
/// docker-internal vision endpoint keeps working.
pub fn ensure_public_target(raw_url: &str) -> Result<(), UrlSafetyError> {
ensure_public_target_inner(raw_url).map(|_| ())
}
fn ensure_public_target_inner(raw_url: &str) -> Result<Url, UrlSafetyError> {
let url = Url::parse(raw_url).map_err(|_| UrlSafetyError::Unparseable)?;
let scheme = url.scheme();
if scheme != "http" && scheme != "https" {
@@ -148,13 +178,10 @@ pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSa
return Err(UrlSafetyError::PrivateIp(ip));
}
}
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
Ok(url)
}
fn is_private_ip(ip: &IpAddr) -> bool {
pub(crate) fn is_private_ip(ip: &IpAddr) -> bool {
match ip {
IpAddr::V4(v4) => {
v4.is_loopback()
@@ -168,11 +195,14 @@ fn is_private_ip(ip: &IpAddr) -> bool {
|| v4.octets()[0] == 0
}
IpAddr::V6(v6) => {
// IPv4-mapped IPv6 (::ffff:0:0/96): unwrap to the embedded
// IPv4 and recurse so `::ffff:127.0.0.1` is caught by the
// IPv4 loopback check rather than passing through.
// `Ipv6Addr::is_loopback()` only matches `::1` exactly.
if let Some(v4) = v6.to_ipv4_mapped() {
// Any IPv6 form that *embeds* an IPv4 address (mapped
// `::ffff:0:0/96`, compatible `::/96`, NAT64 `64:ff9b::/96`,
// 6to4 `2002::/16`) is unwrapped and re-checked as its IPv4 —
// otherwise `::127.0.0.1` / `2002:7f00:1::` / `64:ff9b::7f00:1`
// would smuggle an internal IPv4 past the check (the audit's
// IPv6-embedding gap). `Ipv6Addr::is_loopback()` only matches
// `::1` exactly, so these embeddings need explicit handling.
if let Some(v4) = embedded_ipv4(v6) {
return is_private_ip(&IpAddr::V4(v4));
}
v6.is_loopback()
@@ -185,6 +215,114 @@ fn is_private_ip(ip: &IpAddr) -> bool {
}
}
/// Extract the IPv4 address embedded in an IPv6 literal, for every
/// transitional encoding that can carry one: IPv4-mapped (`::ffff:0:0/96`),
/// IPv4-compatible (`::/96`, deprecated but still routable via some stacks),
/// NAT64 (`64:ff9b::/96`), and 6to4 (`2002::/16`). Returns `None` for a
/// native IPv6 address. Callers recurse into [`is_private_ip`] on the result
/// so a private IPv4 can't hide inside an IPv6 literal.
fn embedded_ipv4(v6: &Ipv6Addr) -> Option<Ipv4Addr> {
let seg = v6.segments();
let low32 = |a: u16, b: u16| Ipv4Addr::new((a >> 8) as u8, (a & 0xff) as u8, (b >> 8) as u8, (b & 0xff) as u8);
// ::ffff:0:0/96 (mapped) and ::/96 (compatible) — top 96 bits zero,
// except mapped which has 0xffff at seg[5]. to_ipv4() covers both.
if seg[0..5] == [0, 0, 0, 0, 0] && (seg[5] == 0 || seg[5] == 0xffff) {
return Some(low32(seg[6], seg[7]));
}
// NAT64 64:ff9b::/96
if seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0] {
return Some(low32(seg[6], seg[7]));
}
// 6to4 2002::/16 — embedded IPv4 is bits 16..48 (seg[1], seg[2]).
if seg[0] == 0x2002 {
return Some(low32(seg[1], seg[2]));
}
None
}
/// A `reqwest::dns::Resolve` that performs normal system resolution, then
/// drops any resolved address in a private / loopback / link-local / metadata
/// range. Installed on every crawler + analysis reqwest client so a hostname
/// that resolves to an internal IP (DNS rebinding: an attacker-owned domain
/// with an `A` record for `169.254.169.254` or `10.x`) can never be connected
/// to — closing the TOCTOU gap that the string-only `ensure_public_target`
/// check leaves open. Fires per connection, so it also guards redirect hops.
#[derive(Debug, Default)]
pub struct SafeResolver;
/// Partition resolved addresses into the public ones, rejecting when nothing
/// survives. Split out from the async `resolve` so the security-critical
/// filter is unit-testable without real DNS.
fn retain_public_addrs(
host: &str,
addrs: impl Iterator<Item = SocketAddr>,
) -> Result<Vec<SocketAddr>, BlockedResolution> {
let public: Vec<SocketAddr> = addrs.filter(|a| !is_private_ip(&a.ip())).collect();
if public.is_empty() {
return Err(BlockedResolution(host.to_string()));
}
Ok(public)
}
#[derive(Debug, thiserror::Error)]
#[error("host {0} resolved only to private/blocked addresses")]
struct BlockedResolution(String);
impl Resolve for SafeResolver {
fn resolve(&self, name: Name) -> Resolving {
Box::pin(async move {
let host = name.as_str().to_string();
// Port 0: reqwest overrides it with the URL's port after
// resolution (same convention as reqwest's default GaiResolver).
let resolved = tokio::net::lookup_host((host.as_str(), 0))
.await
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
let public = retain_public_addrs(&host, resolved)
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
Ok(Box::new(public.into_iter()) as Addrs)
})
}
}
/// Shared [`SafeResolver`] for wiring into `ClientBuilder::dns_resolver`.
pub fn safe_dns_resolver() -> Arc<SafeResolver> {
Arc::new(SafeResolver)
}
/// Whether the DNS-rebinding [`SafeResolver`] should be attached to a crawler
/// reqwest client given the configured proxy (if any).
///
/// The resolver only guards targets that *reqwest itself* resolves:
/// - **No proxy** (direct): reqwest resolves the target — attach.
/// - **`http(s)://` proxy**: reqwest still resolves the target host locally and
/// issues a `CONNECT`, so a hostname resolving to a private IP must be
/// refused — attach.
/// - **`socks5://` / `socks5h://` / `socks4://` (Tor)**: the *proxy* resolves
/// the target; reqwest only ever resolves the proxy's own host, which
/// legitimately lives on a private Docker IP (e.g. `tor` → 172.x). Attaching
/// the resolver there rejects every fetch for zero security gain (a SOCKS
/// proxy can't route into internal ranges anyway) — do **not** attach.
///
/// This narrows commit 134ab54, which dropped the resolver for *any* proxy, back
/// to SOCKS-only so the http(s)-proxy path keeps its DNS-rebinding guard.
pub fn should_attach_safe_resolver(proxy: Option<&str>) -> bool {
match proxy {
None => true,
Some(p) => !is_socks_proxy(p),
}
}
/// True when `proxy`'s scheme is a SOCKS variant (`socks4`, `socks5`,
/// `socks5h`). A scheme-less value (reqwest treats it as HTTP) is not SOCKS.
fn is_socks_proxy(proxy: &str) -> bool {
proxy
.split_once("://")
.map(|(scheme, _)| scheme)
.unwrap_or("")
.to_ascii_lowercase()
.starts_with("socks")
}
#[derive(Debug, thiserror::Error, PartialEq, Eq)]
pub enum UrlSafetyError {
#[error("URL is not parseable")]
@@ -201,6 +339,73 @@ pub enum UrlSafetyError {
HostNotAllowed(String),
}
/// Maximum number of redirects the crawler will follow before giving up.
/// Matches reqwest's historical default; every hop is re-validated by
/// [`check_redirect_hop`], so the cap is a belt-and-braces stop against a
/// redirect loop rather than the primary SSRF defence.
pub const MAX_REDIRECTS: usize = 10;
/// Why a redirect hop was refused.
#[derive(Debug, thiserror::Error)]
pub enum RedirectError {
#[error("redirect chain exceeded {0} hops")]
TooManyHops(usize),
#[error("redirect target rejected: {0}")]
Unsafe(#[from] UrlSafetyError),
}
/// Decide whether a single redirect hop is safe to follow.
///
/// `is_safe_url` only inspects the *initial* URL a caller hands to reqwest;
/// without re-validation an allowlisted CDN that answers `302 ->
/// http://169.254.169.254/...` or `-> http://127.0.0.1:5432/` would be
/// followed transparently (reqwest's default policy follows up to 10
/// redirects). This re-runs the full allowlist + private-IP + scheme check on
/// each hop and enforces [`MAX_REDIRECTS`]. `completed_hops` is the number of
/// URLs already visited in the chain (reqwest's `attempt.previous().len()`).
pub fn check_redirect_hop(
next_url: &str,
completed_hops: usize,
allow: &DownloadAllowlist,
) -> Result<(), RedirectError> {
if completed_hops >= MAX_REDIRECTS {
return Err(RedirectError::TooManyHops(completed_hops));
}
is_safe_url(next_url, allow)?;
Ok(())
}
/// Build a reqwest redirect policy that re-validates every hop against the
/// download allowlist (see [`check_redirect_hop`]). Use for the crawler image
/// clients, which fetch attacker-influenced URLs.
pub fn safe_redirect_policy(allow: DownloadAllowlist) -> reqwest::redirect::Policy {
reqwest::redirect::Policy::custom(move |attempt| {
let hops = attempt.previous().len();
match check_redirect_hop(attempt.url().as_str(), hops, &allow) {
Ok(()) => attempt.follow(),
Err(e) => attempt.error(e),
}
})
}
/// Build a reqwest redirect policy that re-validates every hop with
/// [`ensure_public_target`] (scheme + private-IP, no allowlist). Use for
/// single-endpoint clients (the analysis vision endpoint / its probe) where
/// there is no per-deployment allowlist but a redirect into the deployment's
/// internal network must still be refused.
pub fn public_redirect_policy() -> reqwest::redirect::Policy {
reqwest::redirect::Policy::custom(move |attempt| {
let hops = attempt.previous().len();
if hops >= MAX_REDIRECTS {
return attempt.error(RedirectError::TooManyHops(hops));
}
match ensure_public_target(attempt.url().as_str()) {
Ok(()) => attempt.follow(),
Err(e) => attempt.error(RedirectError::Unsafe(e)),
}
})
}
/// Drain a byte stream into a single buffer, bailing out as soon as
/// the running total exceeds `max_bytes`. Generic over the stream so
/// it's testable without a live HTTP response.
@@ -302,6 +507,26 @@ mod tests {
DownloadAllowlist::new().allow(host)
}
#[test]
fn rejects_alternate_ipv4_encodings_of_loopback() {
// The `url` crate normalizes special-scheme IPv4 hosts per WHATWG, so
// decimal/hex/octal/short forms all collapse to 127.0.0.1 and must be
// rejected. This normalization is load-bearing (it's what stops these
// encodings smuggling past the literal-IP check) but was untested — pin
// it so a future URL-parser swap can't silently reopen the hole.
for url in [
"http://2130706433/", // decimal 127.0.0.1
"http://0x7f000001/", // hex
"http://0177.0.0.1/", // octal first octet
"http://127.1/", // short form
] {
assert!(
ensure_public_target(url).is_err(),
"{url} normalizes to loopback and must be rejected"
);
}
}
#[test]
fn allow_any_admits_arbitrary_public_host() {
// Operators who can't pre-enumerate a numbered-CDN fleet
@@ -383,6 +608,71 @@ mod tests {
));
}
// --- ensure_public_target (allowlist-free admin URL validation) ---
#[test]
fn public_target_allows_dns_hostnames() {
// The whole point of this helper is to validate admin-supplied
// endpoint URLs (analysis vision endpoint, crawler start_url)
// WITHOUT consulting an allowlist. Docker-internal DNS names
// (which resolve at runtime to private IPs) MUST pass.
assert!(ensure_public_target("http://mangalord-vision:8000/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://api.openai.com/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://example.com/").is_ok());
}
#[test]
fn public_target_blocks_ip_literal_attacks() {
// Literal IPs in private/loopback/link-local ranges — the routes
// an attacker would actually use (AWS IMDS, postgres on
// 127.0.0.1, RFC1918 inside a corp network).
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:5432/",
"http://10.0.0.1/",
"http://192.168.1.1/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://0.0.0.0/",
] {
assert!(
matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::PrivateIp(_)
),
"must reject {url}"
);
}
}
#[test]
fn public_target_blocks_localhost_hostname() {
assert!(matches!(
ensure_public_target("http://localhost:5432/").unwrap_err(),
UrlSafetyError::Loopback
));
}
#[test]
fn public_target_blocks_non_http_schemes() {
for url in ["file:///etc/passwd", "gopher://x.example/", "ftp://x/"] {
assert!(matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::BadScheme(_)
));
}
}
#[test]
fn public_target_rejects_unparseable() {
assert!(matches!(
ensure_public_target("not a url").unwrap_err(),
UrlSafetyError::Unparseable
));
}
// --- back to is_safe_url ---
#[test]
fn safe_url_blocks_rfc1918() {
let allow = allow_just("10.0.0.1");
@@ -452,6 +742,79 @@ mod tests {
assert!(matches!(err, UrlSafetyError::PrivateIp(_)));
}
#[test]
fn is_private_ip_unwraps_embedded_ipv4_encodings() {
// Every IPv6 encoding that can smuggle an internal IPv4 must be
// caught. The audit flagged compatible ::/96, NAT64, and 6to4 as
// gaps past the original mapped-only handling.
for s in [
"::ffff:127.0.0.1", // IPv4-mapped loopback
"::127.0.0.1", // IPv4-compatible loopback (was a gap)
"::ffff:10.1.2.3", // mapped RFC1918
"::10.1.2.3", // compatible RFC1918 (was a gap)
"64:ff9b::7f00:1", // NAT64 of 127.0.0.1 (was a gap)
"64:ff9b::a01:203", // NAT64 of 10.1.2.3
"2002:7f00:1::", // 6to4 of 127.0.0.1 (was a gap)
"2002:a01:203::", // 6to4 of 10.1.2.3
"2002:a9fe:a9fe::", // 6to4 of 169.254.169.254 (metadata)
] {
let ip: IpAddr = s.parse().unwrap();
assert!(is_private_ip(&ip), "{s} must be flagged private");
}
}
#[test]
fn is_private_ip_allows_public_embedded_and_native_ipv6() {
// A public IPv4 embedded in IPv6, and a native public IPv6, must
// NOT be flagged — the unwrap only blocks when the embedded v4 is
// itself private.
for s in [
"::ffff:8.8.8.8", // mapped public
"2002:808:808::", // 6to4 of 8.8.8.8 (public)
"2606:4700:4700::1111", // native public (Cloudflare)
] {
let ip: IpAddr = s.parse().unwrap();
assert!(!is_private_ip(&ip), "{s} must be allowed");
}
}
#[test]
fn retain_public_addrs_drops_private_and_errors_when_all_private() {
use std::net::{Ipv4Addr, SocketAddr};
let pub_addr = SocketAddr::from((Ipv4Addr::new(93, 184, 216, 34), 0));
let loopback = SocketAddr::from((Ipv4Addr::new(127, 0, 0, 1), 0));
let metadata = SocketAddr::from((Ipv4Addr::new(169, 254, 169, 254), 0));
// Mixed result keeps only the public address.
let kept =
retain_public_addrs("mixed.example", [pub_addr, loopback, metadata].into_iter())
.expect("public address survives");
assert_eq!(kept, vec![pub_addr]);
// All-private (DNS rebinding to internal) is rejected outright.
let err =
retain_public_addrs("rebind.attacker", [loopback, metadata].into_iter()).unwrap_err();
assert!(err.to_string().contains("rebind.attacker"));
}
#[test]
fn safe_resolver_attaches_except_for_socks_proxies() {
// Direct + http(s) proxies: reqwest resolves the target, so the
// DNS-rebinding guard must be attached.
assert!(should_attach_safe_resolver(None));
assert!(should_attach_safe_resolver(Some("http://proxy.internal:8080")));
assert!(should_attach_safe_resolver(Some("https://proxy.internal:8080")));
assert!(should_attach_safe_resolver(Some("HTTP://Proxy:8080")));
// A scheme-less proxy is treated as HTTP by reqwest — keep the guard.
assert!(should_attach_safe_resolver(Some("proxy.internal:8080")));
// SOCKS variants (incl. Tor): the proxy resolves, so attaching the
// resolver would reject every fetch for zero gain.
assert!(!should_attach_safe_resolver(Some("socks5://tor:9050")));
assert!(!should_attach_safe_resolver(Some("socks5h://tor:9050")));
assert!(!should_attach_safe_resolver(Some("socks4://x:1080")));
assert!(!should_attach_safe_resolver(Some("SOCKS5://Tor:9050")));
}
#[test]
fn safe_url_blocks_non_http_schemes() {
let allow = allow_just("anywhere");
@@ -488,6 +851,54 @@ mod tests {
assert!(is_safe_url("https://CDN.EXAMPLE.com/x.jpg", &allow).is_ok());
}
// --- redirect-hop re-validation (SSRF via 3xx) ---
#[test]
fn redirect_hop_allows_listed_public_target() {
let allow = allow_just("cdn.example.com");
assert!(check_redirect_hop("https://cdn.example.com/next.jpg", 1, &allow).is_ok());
}
#[test]
fn redirect_hop_blocks_private_ip_target() {
// The core SSRF case: an allowlisted CDN 302s to the cloud metadata
// service / an intra-compose port. Must be refused mid-chain.
let allow = allow_just("cdn.example.com");
for url in ["http://169.254.169.254/", "http://127.0.0.1:5432/", "http://10.0.0.1/"] {
let err = check_redirect_hop(url, 1, &allow).unwrap_err();
assert!(
matches!(err, RedirectError::Unsafe(UrlSafetyError::PrivateIp(_))),
"expected PrivateIp for {url}, got {err:?}"
);
}
}
#[test]
fn redirect_hop_blocks_off_allowlist_public_host() {
// Per the strict policy: a redirect to an unlisted *public* host is
// also refused (allow_any covers the numbered-CDN case instead).
let allow = allow_just("cdn.example.com");
let err = check_redirect_hop("https://evil.example.org/x", 1, &allow).unwrap_err();
assert!(matches!(err, RedirectError::Unsafe(UrlSafetyError::HostNotAllowed(_))));
}
#[test]
fn redirect_hop_blocks_bad_scheme_target() {
let allow = DownloadAllowlist::allow_any();
let err = check_redirect_hop("file:///etc/passwd", 1, &allow).unwrap_err();
assert!(matches!(err, RedirectError::Unsafe(UrlSafetyError::BadScheme(_))));
}
#[test]
fn redirect_hop_caps_chain_length() {
let allow = allow_just("cdn.example.com");
// A safe target is still refused once the hop cap is reached, so a
// redirect loop can't spin forever.
let err = check_redirect_hop("https://cdn.example.com/x", MAX_REDIRECTS, &allow)
.unwrap_err();
assert!(matches!(err, RedirectError::TooManyHops(n) if n == MAX_REDIRECTS));
}
#[tokio::test]
async fn accumulate_capped_returns_full_body_under_cap() {
let chunks: Vec<Result<bytes::Bytes, std::io::Error>> = vec![

View File

@@ -285,25 +285,39 @@ where
}
async fn fetch_probe_html(browser: &Browser, probe_url: &str) -> anyhow::Result<String> {
let page = browser
.new_page(probe_url)
// Guard the probe navigation for parity with the list/detail and
// chapter-content paths — the probe URL is operator-controlled, but
// keeping every `new_page` behind the same SSRF check avoids a gap if
// the URL ever becomes attacker-influenced.
crate::crawler::safety::ensure_public_target(probe_url)
.with_context(|| format!("refuse to navigate unsafe probe URL {probe_url}"))?;
let page = crate::crawler::intercept::open_page(browser, probe_url)
.await
.with_context(|| format!("open probe page {probe_url}"))?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for nav on probe")?;
// Best-effort wait for the layout marker. Timeout is fine — the
// probe classifier handles a missing `#logo` as Transient anyway,
// and the verify loop retries on Transient.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"#logo",
crate::crawler::nav::SELECTOR_TIMEOUT,
// Close the tab on every exit path — a `?` on wait_for_nav / content()
// would otherwise leak it (chromiumoxide doesn't close on drop).
let closer = page.clone();
crate::crawler::nav::close_after(
async move {
closer.close().await.ok();
},
async {
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for nav on probe")?;
// Best-effort wait for the layout marker. Timeout is fine — the
// probe classifier handles a missing `#logo` as Transient anyway,
// and the verify loop retries on Transient.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"#logo",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
page.content().await.context("read probe html")
},
)
.await;
let html = page.content().await.context("read probe html")?;
page.close().await.ok();
Ok(html)
.await
}
#[cfg(test)]

View File

@@ -20,7 +20,8 @@ use super::{
use crate::crawler::detect::{
has_logo_sentinel, is_broken_page_body, retry_on_transient_with_hook, PageError,
};
use crate::crawler::nav::{wait_for_nav, wait_for_selector, NavError, SELECTOR_TIMEOUT};
use crate::crawler::nav::{close_after, wait_for_nav, wait_for_selector, NavError, SELECTOR_TIMEOUT};
use crate::crawler::safety::ensure_public_target;
/// `sources.id` value for this Source impl. Exposed as a const so the
/// daemon can look up per-source state (e.g. the recovery flag) before
@@ -220,6 +221,19 @@ const LIST_PAGE_MARKER: &str = "#left_side .pic_list .updatesli";
const DETAIL_PAGE_CHAPTERS_MARKER: &str = "#chapter_table td h4 a.chico";
const DETAIL_PAGE_LAYOUT_MARKER: &str = "#logo";
/// Refuse to point the headless browser at a private/internal target.
/// The list/detail URLs driven through [`navigate`] originate from
/// scraped hrefs (base URL, pagination, and detail links harvested from
/// listings), so a hostile or compromised source could otherwise steer
/// Chromium at `http://169.254.169.254/`, `http://postgres:5432/`, etc.
/// and read the response body as an SSRF oracle. Mirrors the
/// chapter-content guard in [`crate::crawler::content`].
fn guard_navigate_url(url: &str) -> Result<(), PageError> {
ensure_public_target(url).map_err(|e| {
PageError::Other(anyhow::anyhow!("refuse to navigate unsafe URL {url}: {e}"))
})
}
/// Single point of rate-limited navigation. Every Source request goes
/// through here, so the per-host limiter map is the only knob that
/// controls per-origin RPS. Also the choke point for transient-page
@@ -237,32 +251,39 @@ async fn navigate(
url: &str,
marker: &str,
) -> Result<String, PageError> {
guard_navigate_url(url)?;
ctx.rate.wait_for(url).await?;
let page = ctx
.browser
.new_page(url)
let page = crate::crawler::intercept::open_page(ctx.browser, url)
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
match wait_for_nav(&page).await {
Ok(()) => {}
Err(NavError::Timeout(_)) => {
page.close().await.ok();
return Err(PageError::transient("nav timeout"));
}
Err(NavError::Cdp(e)) => {
page.close().await.ok();
return Err(PageError::Other(anyhow::Error::from(e)));
}
}
// Best-effort wait for the page-type marker. We deliberately
// discard a timeout here — see fn-level doc.
let _ = wait_for_selector(&page, marker, SELECTOR_TIMEOUT).await;
let html = page
.content()
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
page.close().await.ok();
classify_navigate_html(html)
// Close the tab on every exit path — the previous code closed on the two
// nav-error branches but leaked it on a content() read error.
let closer = page.clone();
close_after(
async move {
closer.close().await.ok();
},
async {
match wait_for_nav(&page).await {
Ok(()) => {}
Err(NavError::Timeout(_)) => {
return Err(PageError::transient("nav timeout"));
}
Err(NavError::Cdp(e)) => {
return Err(PageError::Other(anyhow::Error::from(e)));
}
}
// Best-effort wait for the page-type marker. We deliberately
// discard a timeout here — see fn-level doc.
let _ = wait_for_selector(&page, marker, SELECTOR_TIMEOUT).await;
let html = page
.content()
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
classify_navigate_html(html)
},
)
.await
}
/// Classify a fetched body. The broken-page template is universal across
@@ -1107,4 +1128,37 @@ mod tests {
.expect("metadata-only parse must not require chapter table");
assert!(manga.chapters.is_empty());
}
#[test]
fn navigate_guard_rejects_private_and_internal_targets() {
// The SSRF guard `navigate` runs before opening any headless page.
// A scraped listing/detail href pointing at cloud metadata, an
// internal service, or the loopback interface must be refused.
for bad in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:5432/",
"http://postgres:5432/", // resolves to a private range name, but…
"http://[::1]/",
"http://10.0.0.5/",
"file:///etc/passwd",
] {
// `postgres` is a bare hostname, not an IP literal, so the
// literal-IP guard alone passes it — assert only the cases the
// guard is designed to catch (IP literals + bad schemes).
if bad.contains("postgres") {
assert!(guard_navigate_url(bad).is_ok(), "bare hostname passes literal check");
continue;
}
assert!(
guard_navigate_url(bad).is_err(),
"expected {bad} to be refused before navigation"
);
}
}
#[test]
fn navigate_guard_allows_public_targets() {
assert!(guard_navigate_url("https://target.example/manga/foo").is_ok());
assert!(guard_navigate_url("https://8.8.8.8/").is_ok());
}
}

View File

@@ -41,6 +41,10 @@ pub enum Phase {
/// Backfilling covers that failed on first attempt. `index`/`total`
/// track progress through this tick's batch.
CoverBackfill { index: usize, total: usize },
/// Reconcile pass: walking the full source list (refs only, no detail
/// visit) to find mangas missing from the DB. `walked` is refs seen so
/// far this walk; `enqueued` is missing mangas queued as `SyncManga`.
Reconciling { walked: usize, enqueued: usize },
}
/// A chapter being downloaded right now, with a live page count. Keyed in

View File

@@ -12,4 +12,6 @@ pub struct ApiToken {
pub token_hash: Vec<u8>,
pub created_at: DateTime<Utc>,
pub last_used_at: Option<DateTime<Utc>>,
/// When the token stops authenticating. `None` = never expires.
pub expires_at: Option<DateTime<Utc>>,
}

View File

@@ -11,6 +11,20 @@ use serde::Serialize;
use sqlx::FromRow;
use uuid::Uuid;
/// One time bucket of a metrics series — count, ok/failed split, and mean
/// duration over the bucket. Shared by the crawl-ops and analysis trend
/// charts (both are `{t, n, ok, failed, avg_ms}` over a `date_trunc` window).
/// Empty intervals are simply absent; the client fills the continuous axis.
#[derive(Debug, Clone, Serialize, FromRow)]
pub struct MetricsBucket {
/// Bucket start (`date_trunc(unit, finished_at|analyzed_at)`).
pub t: DateTime<Utc>,
pub n: i64,
pub ok: i64,
pub failed: i64,
pub avg_ms: Option<f64>,
}
/// Per-operation-type average roll-up over a time window.
#[derive(Debug, Clone, Serialize, FromRow)]
pub struct OpSummary {

View File

@@ -11,6 +11,7 @@ pub mod page;
pub mod page_analysis;
pub mod page_tag;
pub mod patch;
pub mod reaction;
pub mod read_progress;
pub mod session;
pub mod storage_stats;

View File

@@ -0,0 +1,40 @@
use serde::{Deserialize, Serialize};
use uuid::Uuid;
/// A user's private taste signal on a manga. Stored as text in
/// `manga_reactions.reaction` (CHECK-constrained), so we map to/from a
/// `&str` at the repo layer rather than deriving a Postgres enum type.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Reaction {
Like,
Dislike,
}
impl Reaction {
pub fn as_str(self) -> &'static str {
match self {
Reaction::Like => "like",
Reaction::Dislike => "dislike",
}
}
/// Parse a stored/inbound value. Returns `None` for anything outside the
/// closed vocabulary (the DB CHECK guarantees stored rows are valid; this
/// also guards the inbound API body).
pub fn parse(s: &str) -> Option<Reaction> {
match s {
"like" => Some(Reaction::Like),
"dislike" => Some(Reaction::Dislike),
_ => None,
}
}
}
/// Response shape for `GET /me/reactions/:manga_id` — the current user's
/// reaction on one manga, or `null` when they haven't reacted.
#[derive(Debug, Clone, Serialize)]
pub struct MangaReaction {
pub manga_id: Uuid,
pub reaction: Option<Reaction>,
}

View File

@@ -24,8 +24,19 @@ pub struct ReadProgressSummary {
/// `None` when the chapter was deleted after this row was written
/// (FK ON DELETE SET NULL on `chapter_id`).
pub chapter_number: Option<i32>,
/// Page count of the last-read chapter (`None` when unknown / deleted).
/// Lets a client distinguish a finished series (read to the last page,
/// nothing new) from one still in progress — e.g. to keep finished
/// series off a "Continue reading" shelf.
pub chapter_page_count: Option<i32>,
pub page: i32,
pub updated_at: DateTime<Utc>,
/// How many chapters sit past the reader's last-read chapter (by
/// chapter number) — a personal "new since you last read" count.
/// `0` when the reader is caught up or when `chapter_number` is
/// unknown (manga-level progress or a deleted chapter), so we never
/// claim chapters are new when we can't place the reader.
pub new_chapters_count: i64,
}
/// Returned by `GET /me/read-progress/:manga_id`. Same shape as
@@ -40,6 +51,11 @@ pub struct ReadProgressForManga {
pub chapter_number: Option<i32>,
pub page: i32,
pub updated_at: DateTime<Utc>,
/// Distinct chapter numbers past the last-read chapter — the detail
/// page's authoritative "new since last read" count (computed over all
/// chapters, not the client's paginated list). `0` when caught up or
/// the position is unknown. See [`ReadProgressSummary::new_chapters_count`].
pub new_chapters_count: i64,
}
#[derive(Debug, Clone, Deserialize)]

View File

@@ -39,10 +39,10 @@ pub enum AppError {
details: serde_json::Value,
},
/// 501 — the wire shape is accepted but the feature isn't built yet.
/// Carries a `&'static str` snake_case code so clients can detect
/// the specific pending feature (`text_search_not_yet_supported`,
/// etc.) without parsing the message. Used today by the `?text=`
/// reservation on the page-tag aggregation endpoints.
/// Carries a `&'static str` snake_case code so clients can detect the
/// specific pending feature without parsing the message. Generic; kept
/// for future reservations (the `?text=` page-tag reservation that first
/// used it now performs real OCR search).
#[error("not implemented: {code}")]
NotImplemented {
code: &'static str,
@@ -207,4 +207,68 @@ mod tests {
"text_search_not_yet_supported"
);
}
#[test]
fn status_mapping_is_stable() {
// Pin the code -> HTTP status mapping so a refactor of into_response
// can't silently change a status (e.g. 501 -> 500). The 501 NotImplemented
// path and a deterministic 429 Retry-After were previously unexercised
// (audit GAP B).
let cases: Vec<(AppError, StatusCode)> = vec![
(AppError::NotFound, StatusCode::NOT_FOUND),
(AppError::InvalidInput("x".into()), StatusCode::BAD_REQUEST),
(AppError::Unauthenticated, StatusCode::UNAUTHORIZED),
(AppError::Forbidden, StatusCode::FORBIDDEN),
(AppError::Conflict("x".into()), StatusCode::CONFLICT),
(AppError::PayloadTooLarge("x".into()), StatusCode::PAYLOAD_TOO_LARGE),
(
AppError::UnsupportedMediaType("x".into()),
StatusCode::UNSUPPORTED_MEDIA_TYPE,
),
(
AppError::ServiceUnavailable("x".into()),
StatusCode::SERVICE_UNAVAILABLE,
),
(
AppError::ValidationFailed {
message: "x".into(),
details: json!({}),
},
StatusCode::UNPROCESSABLE_ENTITY,
),
(
AppError::NotImplemented {
code: "c",
message: "m",
},
StatusCode::NOT_IMPLEMENTED,
),
(
AppError::TooManyRequests {
retry_after_secs: None,
},
StatusCode::TOO_MANY_REQUESTS,
),
(
AppError::Other(anyhow::anyhow!("boom")),
StatusCode::INTERNAL_SERVER_ERROR,
),
];
for (err, want) in cases {
let code = err.code();
assert_eq!(err.into_response().status(), want, "status for code {code}");
}
// A 429 with retry_after carries the Retry-After header deterministically
// (without needing a live rate limiter).
let resp = AppError::TooManyRequests {
retry_after_secs: Some(7),
}
.into_response();
assert_eq!(resp.status(), StatusCode::TOO_MANY_REQUESTS);
assert_eq!(
resp.headers().get(axum::http::header::RETRY_AFTER).unwrap(),
"7"
);
}
}

View File

@@ -1,14 +1,102 @@
//! Admin-action audit log writes.
//! Admin-action audit log writes + the admin-facing read query.
//!
//! Insert is always called from inside the same transaction as the
//! action it audits — the executor parameter is `PgExecutor` so the
//! caller passes `&mut *tx` directly.
//! caller passes `&mut *tx` directly. [`list`] backs the audit-log viewer.
use sqlx::PgExecutor;
use chrono::{DateTime, Utc};
use serde::Serialize;
use sqlx::{FromRow, PgExecutor, PgPool};
use uuid::Uuid;
use crate::error::AppResult;
/// One audit row joined to the actor's current username (NULL when the
/// actor account was deleted — `actor_user_id` is `ON DELETE SET NULL`).
#[derive(Debug, Clone, Serialize, FromRow)]
pub struct AuditEntryRow {
pub id: Uuid,
pub actor_user_id: Option<Uuid>,
pub actor_username: Option<String>,
pub action: String,
pub target_kind: String,
pub target_id: Option<Uuid>,
pub payload: serde_json::Value,
pub at: DateTime<Utc>,
}
/// Optional filters for [`list`]. `None`/absent widens the scope; an
/// empty-string action/target_kind is treated as absent by the caller.
#[derive(Debug, Default, Clone)]
pub struct AuditFilter<'a> {
pub action: Option<&'a str>,
pub target_kind: Option<&'a str>,
pub actor_user_id: Option<Uuid>,
pub since: Option<DateTime<Utc>>,
}
/// Paginated, newest-first audit log. Returns the page slice plus the
/// filtered total. Ordering by `at DESC` uses `admin_audit_at_idx`.
pub async fn list(
pool: &PgPool,
filter: AuditFilter<'_>,
limit: i64,
offset: i64,
) -> AppResult<(Vec<AuditEntryRow>, i64)> {
let items = sqlx::query_as::<_, AuditEntryRow>(
r#"
SELECT
a.id,
a.actor_user_id,
u.username AS actor_username,
a.action,
a.target_kind,
a.target_id,
a.payload,
a.at
FROM admin_audit a
LEFT JOIN users u ON u.id = a.actor_user_id
WHERE ($1::text IS NULL OR a.action = $1)
AND ($2::text IS NULL OR a.target_kind = $2)
AND ($3::uuid IS NULL OR a.actor_user_id = $3)
AND ($4::timestamptz IS NULL OR a.at >= $4)
-- `id` tiebreaker for stable pagination: a single admin action
-- can write multiple audit rows in the same statement_timestamp
-- (e.g. bulk requeue) and ties on the timestamp would otherwise
-- skip or repeat rows across pages.
ORDER BY a.at DESC, a.id DESC
LIMIT $5 OFFSET $6
"#,
)
.bind(filter.action)
.bind(filter.target_kind)
.bind(filter.actor_user_id)
.bind(filter.since)
.bind(limit)
.bind(offset)
.fetch_all(pool)
.await?;
let (total,): (i64,) = sqlx::query_as(
r#"
SELECT COUNT(*)
FROM admin_audit a
WHERE ($1::text IS NULL OR a.action = $1)
AND ($2::text IS NULL OR a.target_kind = $2)
AND ($3::uuid IS NULL OR a.actor_user_id = $3)
AND ($4::timestamptz IS NULL OR a.at >= $4)
"#,
)
.bind(filter.action)
.bind(filter.target_kind)
.bind(filter.actor_user_id)
.bind(filter.since)
.fetch_one(pool)
.await?;
Ok((items, total))
}
pub async fn insert<'e, E: PgExecutor<'e>>(
executor: E,
actor_user_id: Uuid,

View File

@@ -214,7 +214,7 @@ pub async fn list_mangas_with_sync_state(
let search_pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
// sqlx::Type → text: bind the snake_case representation manually so
// the SQL can compare it as text without an explicit cast.
@@ -235,11 +235,15 @@ pub async fn list_mangas_with_sync_state(
(SELECT MAX(last_seen_at) FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL) AS latest_seen_at
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
WHERE ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
)
SELECT * FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)
ORDER BY updated_at DESC
-- `id` tiebreaker stabilises pagination across mangas that share an
-- `updated_at` (multi-row crawl-tick bulk updates land within the
-- same statement_timestamp() and tie on Postgres' default ts
-- resolution). Without it, page boundaries can skip or repeat rows.
ORDER BY updated_at DESC, id DESC
LIMIT $3 OFFSET $4
"#,
case = MANGA_SYNC_STATE_CASE
@@ -257,7 +261,7 @@ pub async fn list_mangas_with_sync_state(
WITH classified AS (
SELECT {case} AS sync_state
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
WHERE ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
)
SELECT COUNT(*) FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)
@@ -332,7 +336,12 @@ pub async fn list_chapters_with_sync_state(
WHERE cs.chapter_id = c.id AND cs.dropped_at IS NULL) AS latest_seen_at
FROM chapters c
WHERE c.manga_id = $1
ORDER BY c.number ASC
-- `id` tiebreaker: migration 0013 dropped the (manga_id, number)
-- UNIQUE constraint to allow variants ("Ch.14: PH" / "Ch.14:
-- Official") and non-numeric entries (all parse to 0). Without a
-- tiebreaker, pagination over those tied rows can skip or repeat
-- depending on Postgres' chosen sort order.
ORDER BY c.number ASC, c.id ASC
LIMIT $2 OFFSET $3
"#,
)

View File

@@ -2,6 +2,7 @@
//! token; the raw value is shown to the user once at creation and never
//! stored.
use chrono::{DateTime, Utc};
use sqlx::PgPool;
use uuid::Uuid;
@@ -13,28 +14,48 @@ pub async fn create(
user_id: Uuid,
name: &str,
token_hash: &[u8],
expires_at: Option<DateTime<Utc>>,
) -> AppResult<ApiToken> {
let row = sqlx::query_as::<_, ApiToken>(
r#"
INSERT INTO api_tokens (user_id, name, token_hash)
VALUES ($1, $2, $3)
RETURNING id, user_id, name, token_hash, created_at, last_used_at
INSERT INTO api_tokens (user_id, name, token_hash, expires_at)
VALUES ($1, $2, $3, $4)
RETURNING id, user_id, name, token_hash, created_at, last_used_at, expires_at
"#,
)
.bind(user_id)
.bind(name)
.bind(token_hash)
.bind(expires_at)
.fetch_one(pool)
.await?;
Ok(row)
}
/// The caller's tokens, newest first. `token_hash` is `#[serde(skip)]` on the
/// domain type, so returning the full row never leaks the secret.
pub async fn list_for_user(pool: &PgPool, user_id: Uuid) -> AppResult<Vec<ApiToken>> {
let rows = sqlx::query_as::<_, ApiToken>(
r#"
SELECT id, user_id, name, token_hash, created_at, last_used_at, expires_at
FROM api_tokens
WHERE user_id = $1
ORDER BY created_at DESC
"#,
)
.bind(user_id)
.fetch_all(pool)
.await?;
Ok(rows)
}
pub async fn find_active(pool: &PgPool, token_hash: &[u8]) -> AppResult<Option<ApiToken>> {
let row = sqlx::query_as::<_, ApiToken>(
r#"
SELECT id, user_id, name, token_hash, created_at, last_used_at
SELECT id, user_id, name, token_hash, created_at, last_used_at, expires_at
FROM api_tokens
WHERE token_hash = $1
AND (expires_at IS NULL OR expires_at > now())
"#,
)
.bind(token_hash)

View File

@@ -81,7 +81,7 @@ pub async fn list(
SELECT id, name, created_at
FROM authors
WHERE $1::text IS NULL
OR name ILIKE '%' || $1 || '%'
OR name ILIKE '%' || $4 || '%' ESCAPE '\'
OR name % $1
ORDER BY CASE WHEN $1::text IS NULL THEN 0 ELSE similarity(name, $1) END DESC,
lower(name) ASC
@@ -91,6 +91,9 @@ pub async fn list(
.bind(search)
.bind(limit)
.bind(offset)
// $4: LIKE-escaped term for the ILIKE branch so `%`/`_` match literally; the
// trigram/similarity branches keep the raw $1.
.bind(search.map(crate::repo::escape_like))
.fetch_all(pool)
.await?;
Ok(rows)

View File

@@ -6,13 +6,23 @@ use uuid::Uuid;
use crate::domain::{Bookmark, BookmarkSummary};
use crate::error::{AppError, AppResult};
/// Add a bookmark, idempotently. Returns the bookmark plus whether it was
/// newly created (`true`) or already existed (`false`), so the handler can
/// answer 201 vs 200. Re-adding an existing bookmark is a no-op success rather
/// than a 409 — matching the idempotent collection semantics, so the UI doesn't
/// show a false "Could not add bookmark" toast when the manga is already saved.
///
/// Uniqueness is per `(user_id, manga_id, chapter_id)` — enforced by the 0001
/// constraint for chapter-level rows and the 0004 partial index for manga-level
/// (NULL chapter) rows. `page` is not part of the key, so the existing row is
/// returned unchanged (its page is not overwritten).
pub async fn create(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
chapter_id: Option<Uuid>,
page: Option<i32>,
) -> AppResult<Bookmark> {
) -> AppResult<(Bookmark, bool)> {
let result = sqlx::query_as::<_, Bookmark>(
r#"
INSERT INTO bookmarks (user_id, manga_id, chapter_id, page)
@@ -28,10 +38,27 @@ pub async fn create(
.await;
match result {
Ok(b) => Ok(b),
Err(sqlx::Error::Database(ref db_err)) if db_err.is_unique_violation() => Err(
AppError::Conflict("bookmark already exists for this manga/chapter".into()),
),
Ok(b) => Ok((b, true)),
Err(sqlx::Error::Database(ref db_err)) if db_err.is_unique_violation() => {
// A bookmark for this (user, manga, chapter) already exists — fetch
// and return it. `IS NOT DISTINCT FROM` matches a NULL chapter_id
// (manga-level bookmark) as well as a concrete one, covering both
// uniqueness paths with a single lookup.
let existing = sqlx::query_as::<_, Bookmark>(
r#"
SELECT id, user_id, manga_id, chapter_id, page, created_at
FROM bookmarks
WHERE user_id = $1 AND manga_id = $2
AND chapter_id IS NOT DISTINCT FROM $3
"#,
)
.bind(user_id)
.bind(manga_id)
.bind(chapter_id)
.fetch_one(pool)
.await?;
Ok((existing, false))
}
Err(e) => Err(AppError::Database(e)),
}
}
@@ -63,7 +90,13 @@ pub async fn list_for_user(
INNER JOIN mangas m ON m.id = b.manga_id
LEFT JOIN chapters c ON c.id = b.chapter_id
WHERE b.user_id = $1
ORDER BY b.created_at DESC
-- `id` tiebreaker for deterministic pagination. The bookmarks PK
-- is `id` (see migration 0001); the (user_id, manga_id) pair
-- alone is not unique — migration 0004 lets a user keep both a
-- chapter-level bookmark and a manga-level bookmark on the same
-- manga, so two rows can tie on (created_at, manga_id). A 0.87.6
-- earlier revision mistakenly used `manga_id` here.
ORDER BY b.created_at DESC, b.id DESC
LIMIT $2 OFFSET $3
"#,
)

View File

@@ -12,14 +12,19 @@ pub async fn list_for_manga(
limit: i64,
offset: i64,
) -> AppResult<Vec<Chapter>> {
// Display order = source-site order reversed. The crawler stamps
// `source_index` = position in the source DOM (0 = first = newest
// on this site, see migration 0021), so DESC puts the oldest
// chapter first and keeps the site's variant grouping and the
// placement of non-numeric entries (e.g. "notice. : Officials")
// intact. NULLS LAST keeps user-uploaded chapters (no source row)
// and rows that pre-date the migration below crawled rows; the
// (number, created_at) tail then orders them deterministically.
// Display order. Crawled chapters carry `source_index` = position in the
// source DOM (0 = newest on this site, migration 0021); we display them
// reversed (oldest first) via the `-source_index` key, which keeps the
// site's variant grouping and non-numeric entries (e.g. "notice.") in the
// spot the site placed them — NOT clustered at number 0.
//
// User-uploaded chapters have no `source_index`. Instead of dumping them
// after every crawled chapter (the old `NULLS LAST` bug, which put an
// uploaded chapter 5 after crawled chapter 100), each is slotted by NUMBER:
// just before the crawled chapter with the smallest number greater than it,
// so it interleaves. An upload newer than every crawled chapter goes last;
// when there are no crawled chapters at all, uploads fall back to the
// `number, created_at` tail.
let rows = sqlx::query_as::<_, Chapter>(
r#"
SELECT id, manga_id, number, title, page_count, created_at,
@@ -31,7 +36,22 @@ pub async fn list_for_manga(
FROM pages p WHERE p.chapter_id = chapters.id)::bigint AS size_bytes
FROM chapters
WHERE manga_id = $1
ORDER BY source_index DESC NULLS LAST, number ASC, created_at ASC
ORDER BY
CASE
WHEN source_index IS NOT NULL THEN (-source_index)::float8
ELSE COALESCE(
(SELECT MIN(-c2.source_index)::float8 - 0.5
FROM chapters c2
WHERE c2.manga_id = chapters.manga_id
AND c2.source_index IS NOT NULL
AND c2.number > chapters.number),
(SELECT COALESCE(MAX(-c3.source_index), 0)::float8 + 0.5
FROM chapters c3
WHERE c3.manga_id = chapters.manga_id
AND c3.source_index IS NOT NULL)
)
END,
number ASC, created_at ASC
LIMIT $2 OFFSET $3
"#,
)

View File

@@ -8,7 +8,24 @@ use chrono::{DateTime, Utc};
use sqlx::PgPool;
use uuid::Uuid;
use crate::domain::crawl_metrics::{OpRow, OpSummary};
use crate::domain::crawl_metrics::{MetricsBucket, OpRow, OpSummary};
/// Time-bucket granularity for trend series. A closed enum (not a free
/// string) so the `date_trunc` unit can never be attacker-controlled.
#[derive(Debug, Clone, Copy)]
pub enum Bucket {
Hour,
Day,
}
impl Bucket {
pub fn as_str(self) -> &'static str {
match self {
Bucket::Hour => "hour",
Bucket::Day => "day",
}
}
}
/// Operation kinds, kept as consts so call sites and the CHECK constraint
/// don't drift from a free-typed literal.
@@ -75,6 +92,34 @@ pub async fn summary(
.await
}
/// Bucketed throughput/success/duration series over an optional window.
/// Plain `GROUP BY date_trunc(...)` — empty intervals are absent and the
/// client fills the continuous axis. Uses `crawl_metrics_time_idx`.
pub async fn series(
pool: &PgPool,
bucket: Bucket,
since: Option<DateTime<Utc>>,
) -> sqlx::Result<Vec<MetricsBucket>> {
sqlx::query_as::<_, MetricsBucket>(
r#"
SELECT
date_trunc($1, finished_at) AS t,
COUNT(*) AS n,
COUNT(*) FILTER (WHERE outcome = 'ok') AS ok,
COUNT(*) FILTER (WHERE outcome = 'failed') AS failed,
AVG(duration_ms)::float8 AS avg_ms
FROM crawl_metrics
WHERE ($2::timestamptz IS NULL OR finished_at >= $2)
GROUP BY t
ORDER BY t
"#,
)
.bind(bucket.as_str())
.bind(since)
.fetch_all(pool)
.await
}
/// Filters for [`list_ops`]. `None`/empty widens the scope.
#[derive(Debug, Default, Clone)]
pub struct OpFilter<'a> {
@@ -142,7 +187,7 @@ pub async fn list_ops(
}
/// Delete metric rows older than `retention_days`. `0` disables the reaper
/// (returns 0 without touching the table). Mirrors `jobs::reap_done`.
/// (returns 0 without touching the table). Mirrors `jobs::reap_terminal`.
pub async fn reap(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
if retention_days == 0 {
return Ok(0);

View File

@@ -247,10 +247,13 @@ async fn sync_genres(
let genre_id = match existing {
Some((id,)) => id,
None => {
// Conflict on lower(name) (0038) not name, so a racing insert of
// a differently-cased variant resolves to the existing row
// instead of raising a unique violation.
let (id,): (Uuid,) = sqlx::query_as(
r#"
INSERT INTO genres (name) VALUES ($1)
ON CONFLICT (name) DO UPDATE SET name = genres.name
ON CONFLICT (lower(name)) DO UPDATE SET name = genres.name
RETURNING id
"#,
)
@@ -619,6 +622,67 @@ pub async fn last_run_completed_cleanly(
.unwrap_or(true))
}
/// Timestamp of the most recent metadata-pass cron tick, or `None` if the
/// daemon has never ticked (fresh DB / cron disabled). Stored by the daemon
/// under `crawler_state['last_metadata_tick_at']` as `{"at": <rfc3339>}`.
/// Used by the health check that flags a stale/stuck cron.
pub async fn last_metadata_tick_at(
pool: &PgPool,
) -> sqlx::Result<Option<DateTime<Utc>>> {
let row: Option<serde_json::Value> =
sqlx::query_scalar("SELECT value FROM crawler_state WHERE key = 'last_metadata_tick_at'")
.fetch_optional(pool)
.await?;
Ok(row.and_then(|v| {
v.get("at")
.and_then(|s| s.as_str())
.and_then(|s| DateTime::parse_from_rfc3339(s).ok())
.map(|dt| dt.with_timezone(&Utc))
}))
}
// ---------------------------------------------------------------------------
// Reconcile: find source-list mangas missing from the DB.
// ---------------------------------------------------------------------------
/// All `source_manga_key`s we have a `manga_sources` row for under
/// `source_id`. Intentionally **not** filtered by `dropped_at`: a
/// soft-dropped row still counts as "present" so reconcile only enqueues
/// mangas it has truly never seen (strict `NOT EXISTS`).
pub async fn existing_source_keys(
pool: &PgPool,
source_id: &str,
) -> sqlx::Result<std::collections::HashSet<String>> {
let rows: Vec<String> =
sqlx::query_scalar("SELECT source_manga_key FROM manga_sources WHERE source_id = $1")
.bind(source_id)
.fetch_all(pool)
.await?;
Ok(rows.into_iter().collect())
}
/// `source_manga_key`s that already have a `SyncManga` job in a state that
/// should block re-enqueue: `pending`/`running` (in flight) **and** `dead`
/// (leave-dead — a gone manga is not retried by a later reconcile). `done`
/// and `failed` (mid-backoff) do not block.
pub async fn sync_manga_keys_with_blocking_job(
pool: &PgPool,
source_id: &str,
) -> sqlx::Result<std::collections::HashSet<String>> {
let rows: Vec<String> = sqlx::query_scalar(
"SELECT DISTINCT payload->>'source_manga_key' \
FROM crawler_jobs \
WHERE payload->>'kind' = 'sync_manga' \
AND payload->>'source_id' = $1 \
AND state IN ('pending', 'running', 'dead') \
AND payload->>'source_manga_key' IS NOT NULL",
)
.bind(source_id)
.fetch_all(pool)
.await?;
Ok(rows.into_iter().collect())
}
// ---------------------------------------------------------------------------
// Dead-letter jobs: admin observability + requeue.
// ---------------------------------------------------------------------------
@@ -635,6 +699,14 @@ pub struct DeadJob {
pub manga_id: Option<Uuid>,
pub manga_title: Option<String>,
pub chapter_number: Option<i32>,
/// Title carried in the job payload. For `sync_manga` jobs whose manga
/// was never upserted there is no `manga_id`/`manga_title`, so the UI
/// falls back to this.
pub payload_title: Option<String>,
/// Source detail URL carried in the payload (currently `sync_manga`).
pub source_url: Option<String>,
/// Source-native key carried in the payload (currently `sync_manga`).
pub source_key: Option<String>,
pub attempts: i32,
pub max_attempts: i32,
pub last_error: Option<String>,
@@ -651,7 +723,7 @@ pub async fn list_dead_jobs(
offset: i64,
) -> sqlx::Result<(Vec<DeadJob>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<DeadJob> = sqlx::query_as(
@@ -663,6 +735,9 @@ pub async fn list_dead_jobs(
c.manga_id AS manga_id,
m.title AS manga_title,
c.number AS chapter_number,
cj.payload->>'title' AS payload_title,
cj.payload->>'url' AS source_url,
cj.payload->>'source_manga_key' AS source_key,
cj.attempts,
cj.max_attempts,
cj.last_error,
@@ -671,7 +746,7 @@ pub async fn list_dead_jobs(
LEFT JOIN chapters c ON c.id = (cj.payload->>'chapter_id')::uuid
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state = 'dead'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\' OR cj.payload->>'title' ILIKE $1 ESCAPE '\')
ORDER BY cj.updated_at DESC
LIMIT $2 OFFSET $3
"#,
@@ -689,7 +764,7 @@ pub async fn list_dead_jobs(
LEFT JOIN chapters c ON c.id = (cj.payload->>'chapter_id')::uuid
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state = 'dead'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\' OR cj.payload->>'title' ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)
@@ -725,7 +800,7 @@ pub async fn list_active_jobs(
offset: i64,
) -> sqlx::Result<(Vec<ActiveJob>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<ActiveJob> = sqlx::query_as(
@@ -745,7 +820,7 @@ pub async fn list_active_jobs(
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state IN ('pending','running')
AND cj.payload->>'kind' = 'sync_chapter_content'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
ORDER BY (cj.state = 'running') DESC, cj.scheduled_at, cj.created_at
LIMIT $2 OFFSET $3
"#,
@@ -764,7 +839,7 @@ pub async fn list_active_jobs(
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state IN ('pending','running')
AND cj.payload->>'kind' = 'sync_chapter_content'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)
@@ -799,6 +874,11 @@ pub struct JobHistoryRow {
pub page_number: Option<i32>,
/// Source-side key, the only target a `sync_manga` job carries.
pub source_key: Option<String>,
/// Title carried in the payload — the display fallback for `sync_manga`
/// jobs whose manga was never upserted (no `manga_title`).
pub payload_title: Option<String>,
/// Source detail URL carried in the payload (currently `sync_manga`).
pub source_url: Option<String>,
pub attempts: i32,
pub max_attempts: i32,
pub last_error: Option<String>,
@@ -826,9 +906,8 @@ pub struct JobHistoryFilter<'a> {
/// manga/chapter/page context (best-effort) so the table can label rows.
/// Returns the page slice plus the filtered total for pagination.
///
/// History depth is bounded by the done-job reaper (`reap_done`): completed
/// jobs older than the retention window are gone. Terminal `dead` jobs
/// persist until requeued.
/// History depth is bounded by the terminal-job reaper (`reap_terminal`):
/// `done` and `dead` jobs older than the retention window are gone.
pub async fn list_job_history(
pool: &PgPool,
filter: JobHistoryFilter<'_>,
@@ -837,7 +916,7 @@ pub async fn list_job_history(
) -> sqlx::Result<(Vec<JobHistoryRow>, i64)> {
let search_pat = filter
.search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
// The same FROM/JOIN/WHERE drives both the page slice and the count, so
@@ -855,6 +934,8 @@ pub async fn list_job_history(
c.number AS chapter_number,
pg.page_number AS page_number,
cj.payload->>'source_manga_key' AS source_key,
cj.payload->>'title' AS payload_title,
cj.payload->>'url' AS source_url,
cj.attempts,
cj.max_attempts,
cj.last_error,
@@ -873,7 +954,7 @@ pub async fn list_job_history(
) cm ON true
WHERE ($1::text IS NULL OR cj.state = $1)
AND ($2::text IS NULL OR cj.payload->>'kind' = $2)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\' OR cj.payload->>'title' ILIKE $3 ESCAPE '\')
ORDER BY cj.updated_at DESC
LIMIT $4 OFFSET $5
"#,
@@ -895,7 +976,7 @@ pub async fn list_job_history(
LEFT JOIN mangas m ON m.id = COALESCE(c.manga_id, (cj.payload->>'manga_id')::uuid)
WHERE ($1::text IS NULL OR cj.state = $1)
AND ($2::text IS NULL OR cj.payload->>'kind' = $2)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\' OR cj.payload->>'title' ILIKE $3 ESCAPE '\')
"#,
)
.bind(filter.state)
@@ -940,7 +1021,7 @@ pub async fn list_missing_cover_mangas(
offset: i64,
) -> sqlx::Result<(Vec<MissingCoverRow>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<MissingCoverRow> = sqlx::query_as(
@@ -952,7 +1033,7 @@ pub async fn list_missing_cover_mangas(
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL
)
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
ORDER BY m.updated_at DESC
LIMIT $2 OFFSET $3
"#,
@@ -971,7 +1052,7 @@ pub async fn list_missing_cover_mangas(
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL
)
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)

View File

@@ -5,27 +5,55 @@
//! handlers depend only on `sqlx::PgPool`, not on a trait object. Swap to
//! a trait + impl if a second backend ever becomes necessary.
use serde::Deserialize;
use sqlx::{PgConnection, PgExecutor, PgPool};
use uuid::Uuid;
use crate::domain::manga::{Manga, MangaCard, MangaDetail};
use crate::error::{AppError, AppResult};
use crate::repo;
use crate::repo::{self, escape_like};
/// Status values mirror the CHECK constraint in 0009. Centralized so
/// the API layer can validate uploads against the same vocabulary.
pub const STATUSES: &[&str] = &["ongoing", "completed"];
pub const DEFAULT_STATUS: &str = "ongoing";
#[derive(Debug, Clone, Copy, Default, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ListSort {
/// Newest first (default).
/// Which column the listing is ordered by. The direction lives in a separate
/// [`SortOrder`] so any field can be sorted either way. Wire values are parsed
/// (with validation and the legacy `recent` alias) in `api::mangas`, not via
/// serde, so this stays a plain domain enum.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum SortField {
/// Creation time.
Created,
/// Last-modified time (default — "last updated first" pairs with `Desc`).
#[default]
Recent,
/// A→Z by title (case-insensitive).
Updated,
/// Title (case-insensitive).
Title,
/// Alphabetically-first attached author (case-insensitive); authorless
/// mangas sort last regardless of direction.
Author,
}
impl SortField {
/// The direction applied when the client omits `order`. Dates read
/// newest-first (`Desc`); text reads A→Z (`Asc`). This mirrors the
/// frontend's `defaultOrderFor` (see `frontend/src/lib/mangaSort.ts`) so a
/// bare `?sort=<field>` link means the same thing to the UI and the API.
pub fn default_order(self) -> SortOrder {
match self {
SortField::Created | SortField::Updated => SortOrder::Desc,
SortField::Title | SortField::Author => SortOrder::Asc,
}
}
}
/// Sort direction.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub enum SortOrder {
Asc,
#[default]
Desc,
}
#[derive(Debug, Clone, Default)]
@@ -42,7 +70,8 @@ pub struct ListQuery {
pub cw_exclude: Vec<String>,
pub limit: i64,
pub offset: i64,
pub sort: ListSort,
pub sort: SortField,
pub order: SortOrder,
}
/// Single source of truth for the `mangas` columns that hydrate a [`Manga`],
@@ -81,13 +110,13 @@ fn manga_cols(alias: &str) -> String {
/// true.
const FILTER_WHERE: &str = r#"
($1::text IS NULL
OR title ILIKE '%' || $1 || '%'
OR title ILIKE '%' || $8 || '%' ESCAPE '\'
OR title % $1
OR EXISTS (
SELECT 1 FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = mangas.id
AND (a.name ILIKE '%' || $1 || '%' OR a.name % $1)
AND (a.name ILIKE '%' || $8 || '%' ESCAPE '\' OR a.name % $1)
)
)
AND ($2::text IS NULL OR status = $2)
@@ -115,31 +144,48 @@ const FILTER_WHERE: &str = r#"
AND NOT EXISTS (
SELECT 1 FROM unnest($6::text[]) AS req(w)
WHERE NOT EXISTS (
SELECT 1 FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mangas.id AND pw.warning = req.w
SELECT 1 FROM manga_content_warnings mcw
WHERE mcw.manga_id = mangas.id AND mcw.warning = req.w
)
)
AND NOT EXISTS (
SELECT 1 FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mangas.id AND pw.warning = ANY($7::text[])
SELECT 1 FROM manga_content_warnings mcw
WHERE mcw.manga_id = mangas.id AND mcw.warning = ANY($7::text[])
)
"#;
/// Returns the page of mangas matching `query` plus the unfiltered total
/// count for the same filter. The trigram GIN indexes (see 0005_search.sql
/// and 0009_manga_metadata.sql) keep both queries cheap as the library
/// grows.
pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i64)> {
// `order_by` is interpolated from a hard-coded enum, never from request
// input, so this is not a SQL injection seam.
let order_by = match query.sort {
ListSort::Recent => "created_at DESC, id",
ListSort::Title => "lower(title) ASC, id",
/// count for the same filter. The date/title sort columns are index-backed
/// (`mangas_created_at_idx`, `mangas_updated_at_idx`, `mangas_title_lower_idx`)
/// and the trigram GIN indexes (0005_search.sql, 0009_manga_metadata.sql) keep
/// the search filter cheap as the library grows.
pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, Option<i64>)> {
// Both `col` and `dir` are interpolated from hard-coded enums, never from
// request input, so this is not a SQL injection seam. The trailing `id` is
// a stable tie-break that keeps pagination deterministic across rows with
// equal sort keys.
let col = match query.sort {
SortField::Created => "created_at",
SortField::Updated => "updated_at",
SortField::Title => "lower(title)",
// Sorts on the alphabetically-first attached author. Precomputed into
// `mangas.sort_author` (migration 0037, maintained by triggers on
// manga_authors) and index-backed by `mangas_sort_author_idx`, so this
// is a plain column read rather than a per-row correlated subquery.
SortField::Author => "sort_author",
};
let dir = match query.order {
SortOrder::Asc => "ASC",
SortOrder::Desc => "DESC",
};
// Only the author key can be NULL (authorless mangas); keep those last in
// both directions. The date/title columns are NOT NULL, so they need no
// NULLS clause — and omitting it lets a reverse index scan satisfy `DESC`.
let nulls = match query.sort {
SortField::Author => " NULLS LAST",
_ => "",
};
let order_by = format!("{col} {dir}{nulls}, id");
let search = query.search.as_deref();
let status = query.status.as_deref();
@@ -150,11 +196,15 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
FROM mangas
WHERE {FILTER_WHERE}
ORDER BY {order_by}
LIMIT $8 OFFSET $9
LIMIT $9 OFFSET $10
"#,
cols = manga_cols(""),
);
// $8 is the LIKE-escaped search term used by the ILIKE branches so `%`/`_`
// in the term match literally; the trigram `%` branches keep the raw $1.
let search_escaped = search.map(escape_like);
let rows = sqlx::query_as::<_, Manga>(&list_sql)
.bind(search)
.bind(status)
@@ -163,27 +213,40 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.bind(&search_escaped)
.bind(query.limit)
.bind(query.offset)
.fetch_all(pool)
.await?;
let count_sql = format!(
r#"
SELECT count(*) FROM mangas
WHERE {FILTER_WHERE}
"#
);
let (total,): (i64,) = sqlx::query_as(&count_sql)
.bind(search)
.bind(status)
.bind(&query.author_ids)
.bind(&query.genre_ids)
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.fetch_one(pool)
.await?;
// The count reuses FILTER_WHERE — up to five correlated NOT EXISTS/unnest
// subqueries (plus a page_content_warnings join under CW filters) over the
// whole filtered set. Recomputing it on every page made pagination scale
// with catalog size. It doesn't change as the caller walks pages, so
// compute it once on the first page (offset 0) and return None thereafter;
// the pagination envelope serialises that as `total: null`.
let total = if query.offset == 0 {
let count_sql = format!(
r#"
SELECT count(*) FROM mangas
WHERE {FILTER_WHERE}
"#
);
let (total,): (i64,) = sqlx::query_as(&count_sql)
.bind(search)
.bind(status)
.bind(&query.author_ids)
.bind(&query.genre_ids)
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.bind(&search_escaped)
.fetch_one(pool)
.await?;
Some(total)
} else {
None
};
Ok((rows, total))
}
@@ -194,7 +257,7 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
pub async fn list_cards(
pool: &PgPool,
query: &ListQuery,
) -> AppResult<(Vec<MangaCard>, i64)> {
) -> AppResult<(Vec<MangaCard>, Option<i64>)> {
let (rows, total) = list(pool, query).await?;
let cards = cards_from_rows(pool, rows).await?;
Ok((cards, total))
@@ -252,6 +315,73 @@ pub async fn list_similar(
cards_from_rows(pool, rows).await
}
/// Content-based "Recommended for you": rank mangas by weighted tag overlap
/// with the user's taste. Signals: explicit like = +1.0, bookmark = +0.5,
/// dislike = -1.0 (a reaction overrides a bookmark on the same manga). Per
/// tag we sum those weights into an affinity, then score each candidate by
/// the sum of its tags' affinities, normalized by the candidate's tag count
/// (same anti-tag-stuffing rationale as `list_similar`). Candidates the user
/// already reacted to, bookmarked, or read are excluded; net-negative
/// candidates (dominated by disliked-tag affinity) are dropped, so a dislike
/// down-ranks rather than the manga being hidden from normal browse. No
/// signals → empty. Reuses `cards_from_rows` for author/genre hydration.
pub async fn list_recommendations(
pool: &PgPool,
user_id: Uuid,
limit: i64,
) -> AppResult<Vec<MangaCard>> {
let sql = format!(
r#"
WITH signals AS (
SELECT s.manga_id,
CASE WHEN r.reaction = 'dislike' THEN -1.0
WHEN r.reaction = 'like' THEN 1.0
ELSE 0.5 END AS weight
FROM (
SELECT manga_id FROM manga_reactions WHERE user_id = $1
UNION
SELECT manga_id FROM bookmarks WHERE user_id = $1
) s
LEFT JOIN manga_reactions r
ON r.user_id = $1 AND r.manga_id = s.manga_id
),
tag_affinity AS (
SELECT mt.tag_id, SUM(sig.weight) AS affinity
FROM signals sig
JOIN manga_tags mt ON mt.manga_id = sig.manga_id
GROUP BY mt.tag_id
)
SELECT {cols}
FROM manga_tags cand
JOIN tag_affinity ta ON ta.tag_id = cand.tag_id
JOIN mangas m ON m.id = cand.manga_id
WHERE cand.manga_id NOT IN (SELECT manga_id FROM signals)
AND cand.manga_id NOT IN (
SELECT manga_id FROM read_progress WHERE user_id = $1
)
GROUP BY m.id
HAVING SUM(ta.affinity) > 0
ORDER BY
SUM(ta.affinity)
/ (SELECT count(*) FROM manga_tags WHERE manga_id = m.id) DESC,
SUM(ta.affinity) DESC,
m.updated_at DESC,
lower(m.title) ASC,
m.id
LIMIT $2
"#,
cols = manga_cols("m"),
);
let rows = sqlx::query_as::<_, Manga>(&sql)
.bind(user_id)
.bind(limit)
.fetch_all(pool)
.await?;
cards_from_rows(pool, rows).await
}
/// Hydrate a batch of `Manga` rows into `MangaCard`s by attaching their
/// authors and genres in two batched round-trips. The input order is
/// preserved (callers rely on this to keep list/ranking order), so we

View File

@@ -13,6 +13,7 @@ pub mod manga;
pub mod page;
pub mod page_analysis;
pub mod page_tag;
pub mod reaction;
pub mod read_progress;
pub mod session;
pub mod storage_stats;
@@ -20,3 +21,45 @@ pub mod tag;
pub mod upload_history;
pub mod user;
pub mod user_preferences;
/// Escape the LIKE/ILIKE metacharacters (`%`, `_`, and the escape char `\`
/// itself) in a user-supplied search term so they match literally rather than
/// as wildcards. Pair the resulting value with `ESCAPE '\'` in the SQL — a
/// single backslash, which under `standard_conforming_strings` (Postgres
/// default) is one backslash in a single-quoted literal.
///
/// This is a search-correctness fix, not an injection fix: every term is
/// already a bound parameter, so `%`/`_` can never break out of the string —
/// they were just silently acting as wildcards (`50%` matching everything,
/// `a_b` matching `axb`). Callers that build a substring pattern wrap the
/// escaped term themselves, e.g. `format!("%{}%", escape_like(term))`.
pub(crate) fn escape_like(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for ch in s.chars() {
if matches!(ch, '\\' | '%' | '_') {
out.push('\\');
}
out.push(ch);
}
out
}
#[cfg(test)]
mod tests {
use super::escape_like;
#[test]
fn escapes_wildcards_and_the_escape_char() {
assert_eq!(escape_like("50%"), r"50\%");
assert_eq!(escape_like("a_b"), r"a\_b");
assert_eq!(escape_like(r"back\slash"), r"back\\slash");
// A already-escaped-looking input is double-escaped so it stays literal.
assert_eq!(escape_like(r"\%"), r"\\\%");
}
#[test]
fn leaves_ordinary_text_untouched() {
assert_eq!(escape_like("naruto"), "naruto");
assert_eq!(escape_like(""), "");
}
}

View File

@@ -44,12 +44,126 @@ pub struct PageSearchQuery {
/// aborting the whole page's analysis on one bad model output.
const MAX_TAG_CHARS: usize = 64;
/// Outcome of [`enqueue_for_page`] — surfaces whether the admin's intent
/// actually landed as a worker-visible change.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum EnqueueForPageOutcome {
/// A fresh `analyze_page` job row was inserted.
Inserted,
/// A pending `force=false` row was upgraded in-place to `force=true`,
/// because the partial unique index `crawler_jobs_analyze_page_dedup_idx`
/// blocks a second pending insert per page. Returned only for force
/// requests.
UpgradedToForce,
/// A matching `(pending|running)` job already encodes the request;
/// nothing changed. For force requests this is the retry-race tail:
/// the INSERT was Skipped, the UPDATE matched zero rows (the sibling
/// drained between the two), and the second INSERT was Skipped too
/// (a fresh sibling landed). The next ack of either job will leave
/// the page in the admin's intended state without further action.
AlreadyEnqueued,
}
/// Enqueue an `analyze_page` job for `page_id`. `force` re-analyzes a page
/// that is already `done`. Enqueue is idempotent at the job level only in
/// that duplicate pending jobs are harmless — processing is idempotent.
pub async fn enqueue_for_page(pool: &PgPool, page_id: Uuid, force: bool) -> AppResult<()> {
jobs::enqueue(pool, &JobPayload::AnalyzePage { page_id, force }).await?;
Ok(())
/// that is already `done`.
///
/// **Force-vs-dedup correctness.** Migration 0031's partial unique index
/// is keyed on `page_id` (not on `(page_id, force)`), so a plain
/// `jobs::enqueue` of a `force=true` payload when a `force=false` job is
/// already pending silently goes to `Skipped` — the worker would then
/// pick up the non-force row, hit the skip-if-done net, and ack done
/// without re-analyzing. The admin's force request was lost.
///
/// This function fixes that by: (1) trying the insert; (2) on Skipped
/// with `force=true`, running an UPDATE that flips the existing pending
/// row's `force` flag to `true`; (3) reporting which path ran so the
/// caller can audit accurately.
/// Pool convenience wrapper for [`enqueue_for_page_conn`]. Use this from
/// callers that don't need to share a transaction (chapter upload, the
/// crawler). The admin force-reanalyze handler uses the `_conn` form so the
/// enqueue and its `admin_audit` row commit together.
pub async fn enqueue_for_page(
pool: &PgPool,
page_id: Uuid,
force: bool,
) -> AppResult<EnqueueForPageOutcome> {
let mut conn = pool.acquire().await?;
enqueue_for_page_conn(&mut conn, page_id, force).await
}
/// Enqueue (or force-upgrade) an `analyze_page` job on a caller-supplied
/// connection, so the admin handler can run it inside the same transaction as
/// its audit insert. The body is a sequence of single statements, each
/// reborrowing `&mut *conn`.
pub async fn enqueue_for_page_conn(
conn: &mut sqlx::PgConnection,
page_id: Uuid,
force: bool,
) -> AppResult<EnqueueForPageOutcome> {
use crate::crawler::jobs::EnqueueResult;
match jobs::enqueue(&mut *conn, &JobPayload::AnalyzePage { page_id, force }).await? {
EnqueueResult::Inserted(_) => Ok(EnqueueForPageOutcome::Inserted),
EnqueueResult::Skipped if !force => Ok(EnqueueForPageOutcome::AlreadyEnqueued),
EnqueueResult::Skipped => {
// Force-specific upgrade. The WHERE clause matches the same
// partial index predicate (`pending|running` + `analyze_page`
// + this page_id) and adds `force=false` so we never overwrite
// a row that's already what the admin wants. RETURNING tells
// us whether the upgrade actually moved anything AND its
// pre-update state so the running-state race fix below can
// release the in-flight lease.
let upgraded: Vec<(Uuid, String)> = sqlx::query_as(
"UPDATE crawler_jobs \
SET payload = jsonb_set(payload, '{force}', 'true'::jsonb), \
updated_at = now() \
WHERE state IN ('pending', 'running') \
AND payload->>'kind' = 'analyze_page' \
AND payload->>'page_id' = $1 \
AND (payload->>'force')::boolean = false \
RETURNING id, state",
)
.bind(page_id.to_string())
.fetch_all(&mut *conn)
.await?;
if upgraded.is_empty() {
// Race: between the skipped INSERT and the UPDATE the
// sibling row could have been completed (job state moves
// out of pending/running). At that point a re-INSERT
// would succeed. Retry once to close the race without
// unbounded loops.
return Ok(
match jobs::enqueue(&mut *conn, &JobPayload::AnalyzePage { page_id, force })
.await?
{
EnqueueResult::Inserted(_) => EnqueueForPageOutcome::Inserted,
EnqueueResult::Skipped => EnqueueForPageOutcome::AlreadyEnqueued,
},
);
}
// Race fix for the running-state branch (0.87.20 followup):
// a worker that has already leased the row holds the
// pre-upgrade `force=false` in its in-memory `lease.payload`
// (see `analysis::daemon::WorkerContext::process_lease`).
// Even though we've flipped the row's payload to
// `force=true`, the worker's skip-if-done check uses its
// own pre-upgrade value and will ack done without
// re-analyzing. `release_unowned` returns the row to
// `pending` AND bumps `lease_generation` so the original
// worker's later `ack_done(id, old_generation)` finds no
// matching row and is a no-op — the successor's lease (a
// fresh generation) is preserved. The state-only guard
// that previously protected this (and was the 0.87.20
// narrowing) wasn't enough on its own: a successor that
// re-leased the id would re-enter `state='running'`, and
// the original's ack would clobber it.
for (id, state) in &upgraded {
if state == "running" {
let _ = jobs::release_unowned(&mut *conn, *id).await;
}
}
Ok(EnqueueForPageOutcome::UpgradedToForce)
}
}
}
/// How wide a bulk re-enqueue reaches.
@@ -73,11 +187,14 @@ pub enum ReenqueueScope {
/// skip-if-done net would no-op them). Pages with a pending/running
/// `analyze_page` job are always skipped so repeated calls don't pile up
/// duplicates. Returns the number of jobs enqueued.
pub async fn enqueue_pages(
pool: &PgPool,
pub async fn enqueue_pages<'e, E>(
executor: E,
scope: ReenqueueScope,
only_unanalyzed: bool,
) -> AppResult<u64> {
) -> AppResult<u64>
where
E: sqlx::PgExecutor<'e>,
{
// Scope predicate; the bound uuid (when present) is always $2.
let scope_clause = match scope {
ReenqueueScope::All => "",
@@ -86,6 +203,14 @@ pub async fn enqueue_pages(
}
ReenqueueScope::Chapter(_) => "AND p.chapter_id = $2",
};
// The `NOT EXISTS(... pending|running ...)` pre-check used to live in
// this query, but it races with concurrent admin clicks (two requests
// both see "no in-flight job" and both INSERT). The partial unique
// index `crawler_jobs_analyze_page_dedup_idx` (migration 0031) covers
// (payload->>'page_id') WHERE state IN ('pending', 'running') AND
// kind = 'analyze_page', so `ON CONFLICT DO NOTHING` is now the
// atomic primitive — duplicates can't land and concurrent enqueue
// calls are both observably correct.
let sql = format!(
r#"
INSERT INTO crawler_jobs (payload)
@@ -95,11 +220,7 @@ pub async fn enqueue_pages(
SELECT 1 FROM page_analysis pa
WHERE pa.page_id = p.id AND pa.status = 'done'))
{scope_clause}
AND NOT EXISTS (
SELECT 1 FROM crawler_jobs j
WHERE j.payload->>'kind' = 'analyze_page'
AND j.payload->>'page_id' = p.id::text
AND j.state IN ('pending', 'running'))
ON CONFLICT DO NOTHING
"#
);
let query = sqlx::query(&sql).bind(only_unanalyzed);
@@ -107,7 +228,7 @@ pub async fn enqueue_pages(
ReenqueueScope::All => query,
ReenqueueScope::Manga(id) | ReenqueueScope::Chapter(id) => query.bind(id),
};
Ok(query.execute(pool).await?.rows_affected())
Ok(query.execute(executor).await?.rows_affected())
}
/// Per-manga analysis coverage for the admin overview. Only mangas that
@@ -120,6 +241,10 @@ pub async fn manga_coverage(
limit: i64,
offset: i64,
) -> AppResult<(Vec<MangaCoverage>, i64)> {
// LIKE-escape so `%`/`_` in the title filter match literally. No trigram
// branch here, so binding the escaped term directly (rather than appending a
// second param) is safe — $1 feeds only the ILIKE.
let search = search.map(crate::repo::escape_like);
let rows = sqlx::query_as::<_, MangaCoverage>(
r#"
SELECT m.id AS manga_id, m.title,
@@ -129,13 +254,13 @@ pub async fn manga_coverage(
JOIN chapters c ON c.manga_id = m.id
JOIN pages p ON p.chapter_id = c.id
LEFT JOIN page_analysis pa ON pa.page_id = p.id
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%')
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%' ESCAPE '\')
GROUP BY m.id, m.title
ORDER BY lower(m.title), m.id
LIMIT $2 OFFSET $3
"#,
)
.bind(search)
.bind(&search)
.bind(limit)
.bind(offset)
.fetch_all(pool)
@@ -148,12 +273,12 @@ pub async fn manga_coverage(
FROM mangas m
JOIN chapters c ON c.manga_id = m.id
JOIN pages p ON p.chapter_id = c.id
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%')
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%' ESCAPE '\')
GROUP BY m.id
) x
"#,
)
.bind(search)
.bind(&search)
.fetch_one(pool)
.await?;
Ok((rows, total))
@@ -256,7 +381,7 @@ pub async fn list_history(
) -> AppResult<(Vec<AnalysisHistoryRow>, i64)> {
let search_pat = filter
.search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items = sqlx::query_as::<_, AnalysisHistoryRow>(
@@ -281,7 +406,7 @@ pub async fn list_history(
WHERE pa.status <> 'pending'
AND ($1::text IS NULL OR pa.status = $1)
AND ($2::bool IS FALSE OR pa.is_nsfw)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\')
ORDER BY pa.analyzed_at DESC NULLS LAST, pa.page_id
LIMIT $4 OFFSET $5
"#,
@@ -304,7 +429,7 @@ pub async fn list_history(
WHERE pa.status <> 'pending'
AND ($1::text IS NULL OR pa.status = $1)
AND ($2::bool IS FALSE OR pa.is_nsfw)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\')
"#,
)
.bind(filter.status)
@@ -495,6 +620,39 @@ pub async fn analysis_metrics(
Ok(AnalysisMetrics { n, ok, failed, avg_ms, by_model })
}
/// Bucketed analysis throughput/success/duration series over an optional
/// window, for the Analysis-tab trend charts. `ok`/`failed` follow the
/// `done`/`failed` statuses; mean duration is over completed pages only.
/// Empty intervals are absent (client fills the axis). The
/// `page_analysis_analyzed_at_idx` (migration 0030) backs the scan.
pub async fn analysis_series(
pool: &PgPool,
bucket: crate::repo::crawl_metrics::Bucket,
since: Option<DateTime<Utc>>,
) -> AppResult<Vec<crate::domain::crawl_metrics::MetricsBucket>> {
let rows = sqlx::query_as::<_, crate::domain::crawl_metrics::MetricsBucket>(
r#"
SELECT
date_trunc($1, analyzed_at) AS t,
COUNT(*) AS n,
COUNT(*) FILTER (WHERE status = 'done') AS ok,
COUNT(*) FILTER (WHERE status = 'failed') AS failed,
AVG(duration_ms) FILTER (WHERE status = 'done')::float8 AS avg_ms
FROM page_analysis
WHERE status <> 'pending'
AND analyzed_at IS NOT NULL
AND ($2::timestamptz IS NULL OR analyzed_at >= $2)
GROUP BY t
ORDER BY t
"#,
)
.bind(bucket.as_str())
.bind(since)
.fetch_all(pool)
.await?;
Ok(rows)
}
/// Persist a completed analysis for a page, replacing any previous result.
///
/// The model's free-form `kind` / `content_type` strings are mapped onto

View File

@@ -120,27 +120,11 @@ pub async fn list_for_page(
Ok(rows.into_iter().map(|(t,)| t).collect())
}
/// Escape a string for use as a LIKE pattern fragment: `%`, `_`, and
/// `\` get a leading backslash so they're matched literally rather
/// than as wildcards / escapes. The matching queries below pair this
/// with `ESCAPE '\'` for explicitness — a single backslash, since the
/// SQL lives in a raw string and Postgres treats `\\` in a single-
/// quoted literal as one backslash under `standard_conforming_strings`.
///
/// The public API rejects `%`/`_`/`\` in `normalize_tag` before
/// they reach this repo, so this is defence-in-depth — a future
/// internal caller (worker, CLI) that bypasses the normalizer can't
/// turn a prefix filter into a wildcard search by accident.
fn escape_like_prefix(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for ch in s.chars() {
if matches!(ch, '\\' | '%' | '_') {
out.push('\\');
}
out.push(ch);
}
out
}
// LIKE-escaping the autocomplete prefix is defence-in-depth: the public API
// already rejects `%`/`_`/`\` in `normalize_tag` before they reach this repo, so
// a stray wildcard can only arrive from a future internal caller (worker, CLI)
// that bypasses the normalizer. Shared with the other search sites.
use crate::repo::escape_like as escape_like_prefix;
/// Paged list of `user_id`'s tagged pages, with breadcrumb. When
/// `tag_filter` is `Some(_)`, restrict to that exact tag (used by the
@@ -247,6 +231,11 @@ pub async fn distinct_tags_for_user(
/// storage keys (page-number ascending) so the row can render a
/// thumbnail strip without a follow-up fetch.
///
/// When `text` is non-blank, results are further restricted to pages whose
/// analysis `search_doc` matches the query (OCR text search), and both
/// `match_count` and the sample thumbnails reflect that filtered set —
/// i.e. `match_count` counts tagged pages whose OCR also matches `text`.
///
/// `order` is inlined via `format!()` — the enum value space is
/// closed (`ASC` / `DESC`) so this is not a SQL-injection vector.
pub async fn aggregate_chapters_for_tag(
@@ -256,7 +245,30 @@ pub async fn aggregate_chapters_for_tag(
order: Order,
limit: i64,
offset: i64,
text: Option<&str>,
) -> AppResult<(Vec<TaggedChapterAggregate>, i64)> {
// OCR text filter: when present, additionally require the page's analysis
// `search_doc` to match the query. Reuses the precomputed tsvector exactly
// like `repo::page_analysis::page_search`. `None`/blank ⇒ tag-only.
let text = text.map(str::trim).filter(|s| !s.is_empty());
let (text_join, text_where) = match text {
// `$5` in the main query, `$3` in the count query (see binds below).
Some(_) => (
"JOIN page_analysis pa ON pa.page_id = p.id",
"AND pa.search_doc @@ plainto_tsquery('simple', {n})",
),
None => ("", ""),
};
// Same filter inside the correlated sample-thumbnail subquery (its page is
// aliased `p`), so the thumbnails match what the text search matched. The
// subquery lives in the main query, so it reuses the `$5` text bind.
let (sample_join, sample_where) = match text {
Some(_) => (
"JOIN page_analysis pa2 ON pa2.page_id = p.id",
"AND pa2.search_doc @@ plainto_tsquery('simple', $5)",
),
None => ("", ""),
};
let sql = format!(
r#"
SELECT
@@ -274,9 +286,11 @@ pub async fn aggregate_chapters_for_tag(
FROM pages p
JOIN page_tags pt2 ON pt2.page_id = p.id
JOIN tags t2 ON t2.id = pt2.tag_id
{sample_join}
WHERE p.chapter_id = ch.id
AND pt2.user_id = $1
AND lower(t2.name) = $2
{sample_where}
ORDER BY p.page_number ASC
LIMIT 3
) p2
@@ -288,43 +302,60 @@ pub async fn aggregate_chapters_for_tag(
JOIN pages p ON p.id = pt.page_id
JOIN chapters ch ON ch.id = p.chapter_id
JOIN mangas m ON m.id = ch.manga_id
{text_join}
WHERE pt.user_id = $1
AND lower(t.name) = $2
{text_where}
GROUP BY ch.id, ch.manga_id, m.title, ch.number, ch.title
ORDER BY match_count {dir}, ch.id
LIMIT $3 OFFSET $4
"#,
dir = order.as_sql(),
text_join = text_join,
text_where = text_where.replace("{n}", "$5"),
sample_join = sample_join,
sample_where = sample_where,
);
let rows = sqlx::query_as::<_, TaggedChapterAggregate>(&sql)
let mut q = sqlx::query_as::<_, TaggedChapterAggregate>(&sql)
.bind(user_id)
.bind(tag)
.bind(limit)
.bind(offset)
.fetch_all(pool)
.await?;
.bind(offset);
if let Some(text) = text {
q = q.bind(text);
}
let rows = q.fetch_all(pool).await?;
let (total,): (i64,) = sqlx::query_as(
let count_sql = format!(
r#"
SELECT count(*) FROM (
SELECT 1
FROM page_tags pt
JOIN tags t ON t.id = pt.tag_id
JOIN pages p ON p.id = pt.page_id
{text_join}
WHERE pt.user_id = $1 AND lower(t.name) = $2
{text_where}
GROUP BY p.chapter_id
) c
"#,
)
.bind(user_id)
.bind(tag)
.fetch_one(pool)
.await?;
text_join = text_join,
text_where = text_where.replace("{n}", "$3"),
);
let mut cq = sqlx::query_as::<_, (i64,)>(&count_sql).bind(user_id).bind(tag);
if let Some(text) = text {
cq = cq.bind(text);
}
let (total,) = cq.fetch_one(pool).await?;
Ok((rows, total))
}
/// Paged list of mangas containing pages tagged `tag` for `user_id`,
/// ranked by `match_count` summed across all their chapters.
///
/// `text` behaves as in [`aggregate_chapters_for_tag`]: non-blank restricts to
/// pages whose OCR `search_doc` matches, and both `match_count` and the sample
/// thumbnails reflect that filtered set.
pub async fn aggregate_mangas_for_tag(
pool: &PgPool,
user_id: Uuid,
@@ -332,7 +363,26 @@ pub async fn aggregate_mangas_for_tag(
order: Order,
limit: i64,
offset: i64,
text: Option<&str>,
) -> AppResult<(Vec<TaggedMangaAggregate>, i64)> {
// OCR text filter — see `aggregate_chapters_for_tag` for the rationale.
let text = text.map(str::trim).filter(|s| !s.is_empty());
let (text_join, text_where) = match text {
Some(_) => (
"JOIN page_analysis pa ON pa.page_id = p.id",
"AND pa.search_doc @@ plainto_tsquery('simple', {n})",
),
None => ("", ""),
};
// Same filter inside the sample-thumbnail subquery (page aliased `p`),
// reusing the main query's `$5` text bind.
let (sample_join, sample_where) = match text {
Some(_) => (
"JOIN page_analysis pa2 ON pa2.page_id = p.id",
"AND pa2.search_doc @@ plainto_tsquery('simple', $5)",
),
None => ("", ""),
};
let sql = format!(
r#"
SELECT
@@ -349,9 +399,11 @@ pub async fn aggregate_mangas_for_tag(
JOIN chapters ch2 ON ch2.id = p.chapter_id
JOIN page_tags pt2 ON pt2.page_id = p.id
JOIN tags t2 ON t2.id = pt2.tag_id
{sample_join}
WHERE ch2.manga_id = m.id
AND pt2.user_id = $1
AND lower(t2.name) = $2
{sample_where}
ORDER BY p.page_number ASC
LIMIT 3
) p2
@@ -363,23 +415,31 @@ pub async fn aggregate_mangas_for_tag(
JOIN pages p ON p.id = pt.page_id
JOIN chapters ch ON ch.id = p.chapter_id
JOIN mangas m ON m.id = ch.manga_id
{text_join}
WHERE pt.user_id = $1
AND lower(t.name) = $2
{text_where}
GROUP BY m.id, m.title, m.cover_image_path
ORDER BY match_count {dir}, m.id
LIMIT $3 OFFSET $4
"#,
dir = order.as_sql(),
text_join = text_join,
text_where = text_where.replace("{n}", "$5"),
sample_join = sample_join,
sample_where = sample_where,
);
let rows = sqlx::query_as::<_, TaggedMangaAggregate>(&sql)
let mut q = sqlx::query_as::<_, TaggedMangaAggregate>(&sql)
.bind(user_id)
.bind(tag)
.bind(limit)
.bind(offset)
.fetch_all(pool)
.await?;
.bind(offset);
if let Some(text) = text {
q = q.bind(text);
}
let rows = q.fetch_all(pool).await?;
let (total,): (i64,) = sqlx::query_as(
let count_sql = format!(
r#"
SELECT count(*) FROM (
SELECT 1
@@ -387,15 +447,20 @@ pub async fn aggregate_mangas_for_tag(
JOIN tags t ON t.id = pt.tag_id
JOIN pages p ON p.id = pt.page_id
JOIN chapters ch ON ch.id = p.chapter_id
{text_join}
WHERE pt.user_id = $1 AND lower(t.name) = $2
{text_where}
GROUP BY ch.manga_id
) m
"#,
)
.bind(user_id)
.bind(tag)
.fetch_one(pool)
.await?;
text_join = text_join,
text_where = text_where.replace("{n}", "$3"),
);
let mut cq = sqlx::query_as::<_, (i64,)>(&count_sql).bind(user_id).bind(tag);
if let Some(text) = text {
cq = cq.bind(text);
}
let (total,) = cq.fetch_one(pool).await?;
Ok((rows, total))
}

View File

@@ -0,0 +1,66 @@
//! Per-user manga reaction (like/dislike) persistence.
use sqlx::PgPool;
use uuid::Uuid;
use crate::domain::reaction::Reaction;
use crate::error::{AppError, AppResult};
/// Insert-or-overwrite the user's reaction on this manga (like ↔ dislike).
/// A foreign-key violation (unknown manga) maps to `NotFound` so the API
/// returns 404 rather than 500 — mirrors `read_progress::upsert`.
pub async fn upsert(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
reaction: Reaction,
) -> AppResult<()> {
sqlx::query(
r#"
INSERT INTO manga_reactions (user_id, manga_id, reaction, created_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id, manga_id) DO UPDATE
SET reaction = EXCLUDED.reaction,
created_at = now()
"#,
)
.bind(user_id)
.bind(manga_id)
.bind(reaction.as_str())
.execute(pool)
.await
.map_err(|e| match e {
sqlx::Error::Database(ref db_err) if db_err.is_foreign_key_violation() => {
AppError::NotFound
}
other => AppError::Database(other),
})?;
Ok(())
}
/// Remove the user's reaction on this manga. Idempotent — clearing a
/// non-existent reaction is a no-op.
pub async fn clear(pool: &PgPool, user_id: Uuid, manga_id: Uuid) -> AppResult<()> {
sqlx::query("DELETE FROM manga_reactions WHERE user_id = $1 AND manga_id = $2")
.bind(user_id)
.bind(manga_id)
.execute(pool)
.await?;
Ok(())
}
/// The user's current reaction on this manga, or `None` if unset.
pub async fn get(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
) -> AppResult<Option<Reaction>> {
let row: Option<(String,)> = sqlx::query_as(
"SELECT reaction FROM manga_reactions WHERE user_id = $1 AND manga_id = $2",
)
.bind(user_id)
.bind(manga_id)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|(s,)| Reaction::parse(&s)))
}

View File

@@ -80,7 +80,17 @@ pub async fn get_for_manga(
rp.chapter_id,
c.number AS chapter_number,
rp.page,
rp.updated_at
rp.updated_at,
-- Distinct chapter numbers past the reader's last-read chapter
-- (see list_for_user for the non-unique-number rationale). 0
-- when the last-read chapter is unknown.
(
SELECT count(DISTINCT c2.number)
FROM chapters c2
WHERE c2.manga_id = rp.manga_id
AND c.number IS NOT NULL
AND c2.number > c.number
) AS new_chapters_count
FROM read_progress rp
LEFT JOIN chapters c ON c.id = rp.chapter_id
WHERE rp.user_id = $1 AND rp.manga_id = $2
@@ -131,8 +141,23 @@ pub async fn list_for_user(
m.cover_image_path AS manga_cover_image_path,
rp.chapter_id,
c.number AS chapter_number,
c.page_count AS chapter_page_count,
rp.page,
rp.updated_at
rp.updated_at,
-- Personal "new since last read": how many distinct chapter
-- numbers sit past the reader's last-read chapter.
-- COUNT(DISTINCT number) — not COUNT(*) — because
-- (manga_id, number) is non-unique (scanlations share a
-- number, migration 0013), so raw rows would over-count. When
-- `c.number` is NULL (manga-level progress or a deleted
-- chapter) the predicate matches no rows, yielding 0.
(
SELECT count(DISTINCT c2.number)
FROM chapters c2
WHERE c2.manga_id = rp.manga_id
AND c.number IS NOT NULL
AND c2.number > c.number
) AS new_chapters_count
FROM read_progress rp
JOIN mangas m ON m.id = rp.manga_id
LEFT JOIN chapters c ON c.id = rp.chapter_id

View File

@@ -65,3 +65,15 @@ pub async fn delete_by_token_hash(pool: &PgPool, token_hash: &[u8]) -> AppResult
.await?;
Ok(())
}
/// Delete every session whose `expires_at` has passed, returning the number
/// reaped. `find_active` already refuses expired sessions, so this only
/// reclaims storage — running it on any schedule (or not at all) is safe.
/// Backed by `sessions_expires_idx` (0002). Called by the periodic reaper in
/// `app::build`.
pub async fn delete_expired(pool: &PgPool) -> AppResult<u64> {
let result = sqlx::query("DELETE FROM sessions WHERE expires_at <= now()")
.execute(pool)
.await?;
Ok(result.rows_affected())
}

View File

@@ -124,7 +124,7 @@ pub async fn list(
SELECT id, name, created_at
FROM tags
WHERE $1::text IS NULL
OR name ILIKE '%' || $1 || '%'
OR name ILIKE '%' || $3 || '%' ESCAPE '\'
OR name % $1
ORDER BY CASE WHEN $1::text IS NULL THEN 0 ELSE similarity(name, $1) END DESC,
lower(name) ASC
@@ -133,6 +133,9 @@ pub async fn list(
)
.bind(search)
.bind(limit)
// $3: LIKE-escaped term for the ILIKE branch so `%`/`_` match literally; the
// trigram/similarity branches keep the raw $1.
.bind(search.map(crate::repo::escape_like))
.fetch_all(pool)
.await?;
Ok(rows)

View File

@@ -110,7 +110,7 @@ pub async fn list_for_user(
});
}
// Newest first; trim to limit after the merge.
entries.sort_by(|a, b| b.created_at().cmp(&a.created_at()));
entries.sort_by_key(|b| std::cmp::Reverse(b.created_at()));
entries.truncate(limit as usize);
let (manga_total, chapter_total): (i64, i64) = sqlx::query_as(

View File

@@ -106,14 +106,14 @@ pub async fn list_with_total(
let pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items = sqlx::query_as::<_, User>(
r#"
SELECT id, username, password_hash, created_at, is_admin
FROM users
WHERE ($1::text IS NULL OR username ILIKE $1)
WHERE ($1::text IS NULL OR username ILIKE $1 ESCAPE '\')
ORDER BY username
LIMIT $2 OFFSET $3
"#,
@@ -125,7 +125,7 @@ pub async fn list_with_total(
.await?;
let total: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM users WHERE ($1::text IS NULL OR username ILIKE $1)",
"SELECT COUNT(*) FROM users WHERE ($1::text IS NULL OR username ILIKE $1 ESCAPE '\\')",
)
.bind(&pat)
.fetch_one(pool)
@@ -157,8 +157,10 @@ pub async fn set_is_admin_unchecked(pool: &PgPool, id: Uuid, value: bool) -> App
/// - If a row already exists: flip `is_admin` to true if needed; **never**
/// touch the existing `password_hash`. Lets the operator rotate the
/// admin password through the UI without env-var conflict.
///
/// Wrapped in a transaction so a concurrent `register` for the same
/// username can't slip an INSERT between the SELECT and UPDATE/INSERT.
///
/// Set `is_admin` on a user with full safety checks: rejects self-demote,
/// rejects demoting the only remaining admin (under `ADMIN_INVARIANT_LOCK_KEY`
/// to close the parallel-demote race), and writes an `admin_audit` row
@@ -379,7 +381,7 @@ pub async fn bootstrap_admin(
.await?;
}
None => {
let hash = crate::auth::password::hash_password(password)?;
let hash = crate::auth::password::hash_password_async(password.to_string()).await?;
sqlx::query("INSERT INTO users (username, password_hash, is_admin) VALUES ($1, $2, true)")
.bind(username)
.bind(&hash)

View File

@@ -25,7 +25,7 @@ use serde::{Deserialize, Serialize};
use crate::analysis::prompt::{
GROUNDING_PROMPT_DEFAULT, OCR_PROMPT_DEFAULT, SYSTEM_PROMPT_DEFAULT,
};
use crate::config::{AnalysisConfig, CrawlerConfig, ResponseFormat};
use crate::config::{AnalysisBackend, AnalysisConfig, CrawlerConfig, ResponseFormat};
use crate::crawler::safety::DownloadAllowlist;
/// `app_settings.key` for the crawler group.
@@ -33,6 +33,28 @@ pub const KEY_CRAWLER: &str = "crawler";
/// `app_settings.key` for the analysis group.
pub const KEY_ANALYSIS: &str = "analysis";
// Upper bounds on numeric settings. These are sanity caps, not tuned limits —
// they keep a fat-fingered (or CSRF-injected) value from spawning thousands of
// workers, demanding gigabyte buffers, or sending out-of-range sampling
// params. Generous enough that no realistic deployment hits them.
const MAX_WORKERS: u64 = 64;
const MAX_CHAPTER_WORKERS: u64 = 64;
const MAX_ANALYSIS_MAX_TOKENS: u32 = 1_000_000;
const MAX_ANALYSIS_SLICES: u64 = 1024;
/// 1 GiB — far above any real page, well below "exhaust the host".
const MAX_ANALYSIS_IMAGE_BYTES: u64 = 1024 * 1024 * 1024;
/// OpenAI-compatible sampling ranges (the analysis endpoint speaks that API).
const MAX_TEMPERATURE: f64 = 2.0;
const FREQUENCY_PENALTY_MIN: f64 = -2.0;
const FREQUENCY_PENALTY_MAX: f64 = 2.0;
/// Max manga-detail fetches per metadata pass. `0` stays special-cased as
/// "unlimited"; this only bounds an explicit positive value.
const MAX_MANGA_LIMIT: u64 = 1_000_000;
/// Upper bound (seconds) shared by every timeout knob — 24h. A timeout
/// longer than a day is almost certainly a fat-fingered value (e.g. ms
/// mistaken for s) and would wedge a worker for far too long.
const MAX_TIMEOUT_SECS: u64 = 86_400;
/// One field-level validation failure, surfaced to the UI per input.
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct FieldError {
@@ -151,14 +173,31 @@ impl CrawlerSettings {
};
if self.chapter_workers < 1 {
errs.push("chapter_workers", "must be at least 1");
} else if self.chapter_workers > MAX_CHAPTER_WORKERS {
errs.push("chapter_workers", format!("must be at most {MAX_CHAPTER_WORKERS}"));
}
if let Some(url) = self.start_url.as_deref().map(str::trim).filter(|s| !s.is_empty()) {
if reqwest::Url::parse(url).is_err() {
errs.push("start_url", "must be a valid absolute URL");
// SSRF defence: Url::parse alone admits http://169.254.169.254
// / http://127.0.0.1:5432 etc., and the browser would happily
// navigate there with the admin's session. ensure_public_target
// refuses literal-IP private ranges + localhost while still
// accepting public hosts.
if let Err(e) = crate::crawler::safety::ensure_public_target(url) {
errs.push("start_url", url_safety_message(&e));
}
}
if self.job_timeout_secs < 1 {
errs.push("job_timeout_secs", "must be at least 1 second");
} else if self.job_timeout_secs > MAX_TIMEOUT_SECS {
errs.push("job_timeout_secs", format!("must be at most {MAX_TIMEOUT_SECS} seconds"));
}
if self.idle_timeout_secs > MAX_TIMEOUT_SECS {
errs.push("idle_timeout_secs", format!("must be at most {MAX_TIMEOUT_SECS} seconds"));
}
// 0 is intentionally "unlimited"; only an explicit positive value is
// capped.
if self.manga_limit > MAX_MANGA_LIMIT {
errs.push("manga_limit", format!("must be at most {MAX_MANGA_LIMIT}"));
}
if !errs.is_empty() {
@@ -215,6 +254,9 @@ impl CrawlerSettings {
.map(str::to_string),
download_allowlist,
max_image_bytes: self.max_image_bytes as usize,
// Env-only safety cap, not a runtime-editable setting — preserve
// it from the base so a settings reload keeps the boot value.
max_images_per_chapter: base.max_images_per_chapter,
manga_limit: self.manga_limit as usize,
job_timeout: Duration::from_secs(self.job_timeout_secs.max(1)),
metadata_max_consecutive_failures: self.metadata_max_consecutive_failures,
@@ -227,10 +269,27 @@ impl CrawlerSettings {
tor_control_cookie_path: base.tor_control_cookie_path.clone(),
tor_recircuit_max_attempts: base.tor_recircuit_max_attempts,
browser: base.browser.clone(),
ssrf_intercept: base.ssrf_intercept,
})
}
}
/// Render a [`crate::crawler::safety::UrlSafetyError`] as an operator-friendly
/// validation message. The variant detail is preserved so a deliberate
/// `10.0.0.1` value gets "private/loopback IP not allowed" rather than the
/// generic "invalid URL".
fn url_safety_message(e: &crate::crawler::safety::UrlSafetyError) -> &'static str {
use crate::crawler::safety::UrlSafetyError as E;
match e {
E::Unparseable => "must be a valid absolute URL",
E::BadScheme(_) => "must use http:// or https://",
E::NoHost => "must include a host",
E::Loopback => "must not point at localhost",
E::PrivateIp(_) => "must not point at a private/loopback/link-local IP (SSRF risk)",
E::HostNotAllowed(_) => "host is not on the allowlist",
}
}
/// Rebuild a [`DownloadAllowlist`] from the DTO, always seeding the start-url
/// and CDN hosts (mirrors `config::build_download_allowlist`).
fn build_allowlist(
@@ -333,27 +392,76 @@ impl AnalysisSettings {
if self.workers < 1 {
errs.push("workers", "must be at least 1");
} else if self.workers > MAX_WORKERS {
errs.push("workers", format!("must be at most {MAX_WORKERS}"));
}
if self.endpoint.trim().is_empty() || reqwest::Url::parse(self.endpoint.trim()).is_err() {
// The endpoint/model are vision-only knobs — the OCR backend never
// dials a URL or sends a model id. While vision is dormant
// (`base.backend == Ocr`), skip the live-worker SSRF gate and the
// model-required check so an OCR operator can enable the worker
// without a vision endpoint/model (the OCR settings UI doesn't even
// expose them). The basic malformed-URL sanity check below stays
// unconditional. Re-enabling vision restores the full gate via the
// `base.backend == Vision` predicate.
let vision_active = base.backend == AnalysisBackend::Vision;
let trimmed_endpoint = self.endpoint.trim();
if trimmed_endpoint.is_empty() {
errs.push("endpoint", "must be a valid absolute URL");
} else if let Err(e) = reqwest::Url::parse(trimmed_endpoint) {
// Always reject obviously-malformed URLs (matches prior behaviour).
let _ = e;
errs.push("endpoint", "must be a valid absolute URL");
} else if self.enabled && vision_active {
// SSRF + API-key exfiltration defence — only enforced when the
// worker is actually live. Worker attaches an env-managed bearer
// token to every call; without this check, an admin (or one
// CSRF-able request) could repoint the endpoint at
// http://169.254.169.254 (cloud metadata) or http://postgres:5432
// (an internal service that would receive the bearer header).
// Docker DNS names (`mangalord-vision`) still pass because
// they're hostnames, not IP literals.
//
// A disabled config is allowed to carry the dev-default
// `http://localhost:8000/...` placeholder; the worker won't dial
// it, and toggling `enabled=true` later re-runs this validator.
if let Err(e) = crate::crawler::safety::ensure_public_target(trimmed_endpoint) {
errs.push("endpoint", url_safety_message(&e));
}
}
if self.enabled && self.model.trim().is_empty() {
if self.enabled && vision_active && self.model.trim().is_empty() {
errs.push("model", "required when analysis is enabled");
}
if self.max_tokens < 1 {
errs.push("max_tokens", "must be at least 1");
} else if self.max_tokens > MAX_ANALYSIS_MAX_TOKENS {
errs.push("max_tokens", format!("must be at most {MAX_ANALYSIS_MAX_TOKENS}"));
}
if self.request_timeout_secs < 1 {
errs.push("request_timeout_secs", "must be at least 1 second");
} else if self.request_timeout_secs > MAX_TIMEOUT_SECS {
errs.push(
"request_timeout_secs",
format!("must be at most {MAX_TIMEOUT_SECS} seconds"),
);
}
if self.job_timeout_secs < 1 {
errs.push("job_timeout_secs", "must be at least 1 second");
} else if self.job_timeout_secs > MAX_TIMEOUT_SECS {
errs.push(
"job_timeout_secs",
format!("must be at most {MAX_TIMEOUT_SECS} seconds"),
);
}
if self.max_pixels < 1 {
errs.push("max_pixels", "must be at least 1");
}
if self.max_image_bytes < 1 {
errs.push("max_image_bytes", "must be greater than 0");
} else if self.max_image_bytes > MAX_ANALYSIS_IMAGE_BYTES {
errs.push(
"max_image_bytes",
format!("must be at most {MAX_ANALYSIS_IMAGE_BYTES} (1 GiB)"),
);
}
if !(0.0..=0.9).contains(&self.slice_overlap) {
errs.push("slice_overlap", "must be between 0.0 and 0.9");
@@ -366,9 +474,17 @@ impl AnalysisSettings {
}
if self.max_slices < 1 {
errs.push("max_slices", "must be at least 1");
} else if self.max_slices > MAX_ANALYSIS_SLICES {
errs.push("max_slices", format!("must be at most {MAX_ANALYSIS_SLICES}"));
}
if self.temperature < 0.0 {
errs.push("temperature", "must be 0 or greater");
if !(0.0..=MAX_TEMPERATURE).contains(&self.temperature) {
errs.push("temperature", format!("must be between 0 and {MAX_TEMPERATURE}"));
}
if !(FREQUENCY_PENALTY_MIN..=FREQUENCY_PENALTY_MAX).contains(&self.frequency_penalty) {
errs.push(
"frequency_penalty",
format!("must be between {FREQUENCY_PENALTY_MIN} and {FREQUENCY_PENALTY_MAX}"),
);
}
let response_format = match ResponseFormat::parse_strict(&self.response_format) {
Some(rf) => rf,
@@ -417,6 +533,13 @@ impl AnalysisSettings {
api_key: base.api_key.clone(),
// Env-only readiness probe URL preserved from the base.
vision_health_url: base.vision_health_url.clone(),
// Deploy-time engine selection + ocrs model paths: env-only, not
// admin-tunable, so carry them through from the base unchanged.
backend: base.backend,
ocr_detection_model: base.ocr_detection_model.clone(),
ocr_recognition_model: base.ocr_recognition_model.clone(),
// Env-only decompression-bomb decode cap, carried from the base.
ocr_max_decode_pixels: base.ocr_max_decode_pixels,
})
}
}
@@ -448,11 +571,13 @@ mod tests {
#[test]
fn crawler_round_trips_through_dto() {
let mut base = CrawlerConfig::default();
base.start_url = Some("https://example.com/".to_string());
base.tz = Tz::Europe__Berlin;
base.chapter_workers = 3;
base.cookie_domain = Some("example.com".to_string());
let base = CrawlerConfig {
start_url: Some("https://example.com/".to_string()),
tz: Tz::Europe__Berlin,
chapter_workers: 3,
cookie_domain: Some("example.com".to_string()),
..Default::default()
};
let dto = CrawlerSettings::from_config(&base);
let back = dto.to_config(&base).expect("valid");
assert_eq!(back.tz, Tz::Europe__Berlin);
@@ -478,10 +603,12 @@ mod tests {
#[test]
fn crawler_overlay_preserves_env_only_fields() {
let mut base = CrawlerConfig::default();
base.proxy = Some("socks5://127.0.0.1:9050".to_string());
base.tor_control_password = Some("secret".to_string());
base.phpsessid = Some("abc123".to_string());
let base = CrawlerConfig {
proxy: Some("socks5://127.0.0.1:9050".to_string()),
tor_control_password: Some("secret".to_string()),
phpsessid: Some("abc123".to_string()),
..Default::default()
};
// A DTO that knows nothing about the env-only fields.
let dto = CrawlerSettings {
rate_ms: 2000,
@@ -525,6 +652,37 @@ mod tests {
assert!(!out.download_allowlist.is_allow_any());
}
#[test]
fn crawler_start_url_rejects_ip_literal_attacks() {
let base = CrawlerConfig::default();
for url in [
"http://169.254.169.254/",
"http://127.0.0.1/catalog",
"http://localhost:5432/",
"http://10.0.0.1/",
] {
let dto = CrawlerSettings {
start_url: Some(url.to_string()),
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"start_url"), "must reject {url}");
}
}
#[test]
fn crawler_start_url_accepts_public_targets() {
let base = CrawlerConfig::default();
for url in ["https://catalog.example.com/", "http://catalog.example.com/"] {
let dto = CrawlerSettings {
start_url: Some(url.to_string()),
..CrawlerSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok(), "must accept {url}");
}
}
#[test]
fn crawler_allow_any_host_bypasses_list() {
let base = CrawlerConfig::default();
@@ -554,8 +712,10 @@ mod tests {
#[test]
fn analysis_captures_env_prompt_override() {
let mut base = AnalysisConfig::default();
base.system_prompt = "custom env prompt".to_string();
let base = AnalysisConfig {
system_prompt: "custom env prompt".to_string(),
..Default::default()
};
let dto = AnalysisSettings::from_config(&base);
assert_eq!(dto.system_prompt.as_deref(), Some("custom env prompt"));
}
@@ -579,8 +739,10 @@ mod tests {
#[test]
fn analysis_overlay_preserves_api_key() {
let mut base = AnalysisConfig::default();
base.api_key = Some("sk-secret".to_string());
let base = AnalysisConfig {
api_key: Some("sk-secret".to_string()),
..Default::default()
};
let dto = AnalysisSettings::from_config(&base);
assert_eq!(dto.to_config(&base).unwrap().api_key.as_deref(), Some("sk-secret"));
}
@@ -618,8 +780,177 @@ mod tests {
}
#[test]
fn analysis_requires_model_only_when_enabled() {
fn analysis_rejects_over_upper_bounds() {
// Sanity caps: an absurdly large worker count / buffer / token budget
// and out-of-range sampling params are all refused so a fat-fingered
// or CSRF-injected value can't exhaust the host or break the upstream
// API contract.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
workers: MAX_WORKERS + 1,
max_tokens: MAX_ANALYSIS_MAX_TOKENS + 1,
max_slices: MAX_ANALYSIS_SLICES + 1,
max_image_bytes: MAX_ANALYSIS_IMAGE_BYTES + 1,
temperature: MAX_TEMPERATURE + 0.5,
frequency_penalty: FREQUENCY_PENALTY_MAX + 0.5,
request_timeout_secs: MAX_TIMEOUT_SECS + 1,
job_timeout_secs: MAX_TIMEOUT_SECS + 1,
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
for f in [
"workers",
"max_tokens",
"max_slices",
"max_image_bytes",
"temperature",
"frequency_penalty",
"request_timeout_secs",
"job_timeout_secs",
] {
assert!(fields.contains(&f), "missing upper-bound error for {f}");
}
}
#[test]
fn crawler_rejects_over_upper_bounds() {
// manga_limit ceiling + timeout caps. manga_limit=0 stays "unlimited"
// and is asserted valid by the round-trip tests above.
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
manga_limit: MAX_MANGA_LIMIT + 1,
job_timeout_secs: MAX_TIMEOUT_SECS + 1,
idle_timeout_secs: MAX_TIMEOUT_SECS + 1,
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
for f in ["manga_limit", "job_timeout_secs", "idle_timeout_secs"] {
assert!(fields.contains(&f), "missing upper-bound error for {f}");
}
}
#[test]
fn crawler_allows_unlimited_manga_limit() {
// 0 means "no cap" and must stay valid.
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
manga_limit: 0,
..CrawlerSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn analysis_rejects_negative_frequency_penalty_below_range() {
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
frequency_penalty: FREQUENCY_PENALTY_MIN - 0.5,
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"frequency_penalty"));
}
#[test]
fn analysis_accepts_in_range_sampling_params() {
// A normal config with mid-range sampling values stays valid.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
temperature: 0.7,
frequency_penalty: 0.3,
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn crawler_rejects_over_chapter_worker_cap() {
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
chapter_workers: MAX_CHAPTER_WORKERS + 1,
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"chapter_workers"));
}
#[test]
fn analysis_endpoint_rejects_ip_literal_attacks_when_enabled() {
// The vision worker bearer-attaches an env secret to every call;
// a hostile/CSRF-able admin must NOT be able to point endpoint at
// cloud metadata, loopback services, or RFC1918 hosts — when the
// worker is enabled (toggling enabled=true later re-runs this gate).
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
for url in [
"http://169.254.169.254/v1/chat/completions",
"http://127.0.0.1:5432/",
"http://localhost:11434/v1/chat/completions",
"http://10.0.0.5/v1/chat/completions",
] {
let dto = AnalysisSettings {
enabled: true,
model: "test-model".to_string(),
endpoint: url.to_string(),
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"endpoint"), "must reject {url}");
}
}
#[test]
fn analysis_endpoint_accepts_docker_dns_and_public_hosts_when_enabled() {
// The documented default — docker DNS name resolving to a private IP
// at runtime — must still validate, because the bearer recipient
// identity is the operator-chosen hostname, not the underlying IP.
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
for url in [
"http://mangalord-vision:8000/v1/chat/completions",
"https://api.openai.com/v1/chat/completions",
] {
let dto = AnalysisSettings {
enabled: true,
model: "test-model".to_string(),
endpoint: url.to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok(), "must accept {url}");
}
}
#[test]
fn analysis_endpoint_skips_safety_check_when_disabled() {
// Disabled-but-saved bad URL is harmless (worker won't dial it);
// operators get to keep the dev-localhost placeholder until they
// toggle the worker on, when the validator re-runs.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
enabled: false,
endpoint: "http://localhost:8000/v1/chat/completions".to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn analysis_requires_model_only_when_enabled() {
// Vision base: the model id is required only when the vision worker
// is actually live (see the OCR carve-out below).
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
let disabled = AnalysisSettings {
enabled: false,
model: "".to_string(),
@@ -635,6 +966,28 @@ mod tests {
assert!(errs.errors.iter().any(|e| e.field == "model"));
}
#[test]
fn analysis_ocr_backend_enable_skips_vision_endpoint_and_model_validation() {
// With the OCR backend active the worker never dials the vision
// endpoint nor sends a model id, so enabling analysis must NOT
// validate those vision-only fields. The default base carries the
// dev-localhost endpoint (which the SSRF gate would otherwise reject)
// and an empty model — under OCR, enabling is still valid. Without
// this carve-out an OCR operator can't turn the worker on, since the
// OCR settings UI doesn't expose endpoint/model to fix.
let base = AnalysisConfig::default(); // backend = Ocr
let dto = AnalysisSettings {
enabled: true,
endpoint: "http://localhost:8000/v1/chat/completions".to_string(),
model: "".to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(
dto.to_config(&base).is_ok(),
"OCR-enabled config must not fail on vision endpoint/model"
);
}
#[test]
fn dtos_serialize_to_json_and_back() {
let c = CrawlerSettings::default();

View File

@@ -95,6 +95,11 @@ impl Storage for LocalStorage {
match fs::read(&path).await {
Ok(b) => Ok(b),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Err(StorageError::NotFound),
// A key resolving to a directory isn't a stored blob; `fs::read`
// fails with EISDIR. Treat it as absent, mirroring `size`.
Err(e) if e.kind() == std::io::ErrorKind::IsADirectory => {
Err(StorageError::NotFound)
}
Err(e) => Err(e.into()),
}
}
@@ -108,7 +113,14 @@ impl Storage for LocalStorage {
}
Err(e) => return Err(e.into()),
};
let size_bytes = file.metadata().await?.len();
let meta = file.metadata().await?;
// Opening a directory succeeds on Unix, but it isn't a stored blob:
// its inode "size" is meaningless and ReaderStream would fail mid-read.
// Treat it as absent, mirroring `size` and `get`.
if !meta.is_file() {
return Err(StorageError::NotFound);
}
let size_bytes = meta.len();
// 64 KiB chunks: small enough that a few-MB page emits many frames
// (so streaming is observable), large enough to keep syscalls cheap.
let stream = ReaderStream::with_capacity(file, 64 * 1024);
@@ -127,6 +139,23 @@ impl Storage for LocalStorage {
}
}
async fn rename(&self, from: &str, to: &str) -> Result<(), StorageError> {
let from_path = self.resolve(from)?;
let to_path = self.resolve(to)?;
// Create the destination parent so a rename into a not-yet-existing
// chapter directory succeeds. `from` and `to` share the storage
// root (same filesystem), so this is a cheap atomic metadata move,
// not a copy.
if let Some(parent) = to_path.parent() {
fs::create_dir_all(parent).await?;
}
match fs::rename(&from_path, &to_path).await {
Ok(()) => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Err(StorageError::NotFound),
Err(e) => Err(e.into()),
}
}
async fn exists(&self, key: &str) -> Result<bool, StorageError> {
let path: &Path = &self.resolve(key)?;
Ok(fs::try_exists(path).await?)
@@ -227,6 +256,21 @@ mod tests {
assert!(matches!(s.size("adir").await, Err(StorageError::NotFound)));
}
#[tokio::test]
async fn get_on_directory_is_not_found() {
// A key resolving to a directory isn't a stored blob. `fs::read` on a
// dir errors with EISDIR (not NotFound), and `File::open` on a dir
// succeeds on Unix then streams garbage — both must surface NotFound.
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
std::fs::create_dir(dir.path().join("adir")).unwrap();
assert!(matches!(s.get("adir").await, Err(StorageError::NotFound)));
assert!(matches!(
s.get_stream("adir").await.err(),
Some(StorageError::NotFound)
));
}
#[tokio::test]
async fn put_stream_writes_full_body_and_removes_temp_on_error() {
use bytes::Bytes;
@@ -273,6 +317,34 @@ mod tests {
assert_eq!(entries, vec!["ok.bin"]);
}
#[tokio::test]
async fn rename_moves_blob_and_creates_destination_dirs() {
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
s.put("staging/up/0000.png", b"page-bytes").await.unwrap();
// Destination dir doesn't exist yet — rename must create it.
s.rename("staging/up/0000.png", "mangas/m/chapters/c/pages/0001.png")
.await
.unwrap();
assert!(!s.exists("staging/up/0000.png").await.unwrap(), "source gone");
assert_eq!(
s.get("mangas/m/chapters/c/pages/0001.png").await.unwrap(),
b"page-bytes"
);
}
#[tokio::test]
async fn rename_missing_source_is_not_found() {
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
assert!(matches!(
s.rename("staging/nope.png", "dest/x.png").await,
Err(StorageError::NotFound)
));
}
#[tokio::test]
async fn get_stream_emits_multiple_chunks_for_large_files() {
use futures_util::StreamExt as _;

View File

@@ -82,6 +82,27 @@ pub trait Storage: Send + Sync {
async fn delete(&self, key: &str) -> Result<(), StorageError>;
async fn exists(&self, key: &str) -> Result<bool, StorageError>;
/// Move a blob from `from` to `to`, overwriting any existing blob at
/// `to`. Returns `NotFound` if `from` doesn't exist. The chapter
/// upload path uses this to promote a staged page to its final,
/// chapter-scoped key once the chapter row (and thus its id) exists —
/// so pages can be streamed to storage as their multipart parts
/// arrive, without buffering the whole chapter in memory.
///
/// The default implementation streams `from` to `to` and deletes the
/// source, so backends without a native move still satisfy the
/// contract. LocalStorage overrides it with a filesystem rename (an
/// atomic metadata op within a mount); a future S3Storage would
/// override with a server-side copy + delete.
async fn rename(&self, from: &str, to: &str) -> Result<(), StorageError> {
use futures_util::StreamExt as _;
let StreamingFile { stream, .. } = self.get_stream(from).await?;
let mapped: PutByteStream<'_> = Box::pin(stream.map(|r| r.map_err(StorageError::Io)));
self.put_stream(to, mapped).await?;
self.delete(from).await?;
Ok(())
}
/// Size in bytes of the blob at `key`, `NotFound` when it doesn't
/// exist. Cheap metadata lookup (local: `fs::metadata`; a future
/// `S3Storage`: HEAD object). Used by the cover-capture path and the

View File

@@ -6,7 +6,12 @@
//! whitelist with 415. Filename and extension never reach the storage
//! key — we derive both from the sniffed type.
use axum::extract::multipart::Field;
use uuid::Uuid;
use crate::api::mangas::map_multipart_error;
use crate::error::{AppError, AppResult};
use crate::storage::Storage;
#[derive(Debug, Clone)]
pub struct UploadedImage {
@@ -15,6 +20,100 @@ pub struct UploadedImage {
pub ext: &'static str,
}
/// A page image written to a temporary staging key during a chapter
/// upload, awaiting promotion to its final chapter-scoped key once the
/// chapter row (and thus its id) exists. Carries only the small metadata
/// the caller needs — never the image bytes.
#[derive(Debug, Clone)]
pub struct StagedImage {
pub staging_key: String,
pub mime: &'static str,
pub ext: &'static str,
pub size_bytes: i64,
}
/// The staging-key prefix. Blobs left here by a failed or abandoned upload
/// are orphans a future reaper can sweep; a successful upload renames every
/// staged page out of this prefix.
pub const STAGING_PREFIX: &str = "staging";
/// Upper bound on a multipart `metadata` JSON part, enforced as bytes arrive
/// (via [`read_capped`]). Manga/chapter metadata is title + a few short lists +
/// a description — kilobytes at most. Without this, `metadata` was the one
/// remaining part read with the unbounded `Field::bytes()`, letting a client
/// buffer up to the whole 200 MiB request body in memory as a single JSON blob
/// before any validation ran. 64 KiB is generous headroom over any legitimate
/// payload while keeping the worst case tiny.
pub const MAX_METADATA_BYTES: usize = 64 * 1024;
/// Read one multipart image part and write it straight to a staging key,
/// returning only its metadata. The per-file byte cap is enforced as bytes
/// arrive, so an oversized part is rejected (413) without being fully
/// buffered, and at most one page's bytes are held in memory at a time —
/// the whole chapter is never buffered, unlike the previous
/// read-all-then-persist path.
pub async fn stage_image_part(
storage: &dyn Storage,
field: Field<'_>,
upload_id: Uuid,
seq: usize,
max_size: usize,
field_name: &str,
) -> AppResult<StagedImage> {
let bytes = read_capped(field, max_size, field_name).await?;
// Reuse the shared sniff + whitelist check; it re-verifies the size cap
// (already enforced above) and derives mime/ext from magic bytes.
let img = parse_image(bytes, max_size, field_name)?;
let staging_key = format!("{STAGING_PREFIX}/{}/{:04}.{}", upload_id.simple(), seq, img.ext);
storage.put(&staging_key, &img.bytes).await?;
Ok(StagedImage {
staging_key,
mime: img.mime,
ext: img.ext,
size_bytes: img.bytes.len() as i64,
})
}
/// Read one multipart part fully into memory, enforcing the per-file byte cap
/// as chunks arrive so an oversized part is rejected (413) **without being fully
/// buffered**. Use for parts whose bytes the caller needs in hand — e.g. the
/// cover image, which is written to a manga-scoped key. Page parts should prefer
/// [`stage_image_part`], which streams straight to storage.
///
/// Replaces the `Field::bytes()` path, which buffered the entire field before
/// any size check ran, letting an attacker allocate an arbitrarily large body
/// before the cap kicked in.
pub async fn read_capped(
mut field: Field<'_>,
max_size: usize,
field_name: &str,
) -> AppResult<Vec<u8>> {
let mut bytes: Vec<u8> = Vec::new();
while let Some(chunk) = field.chunk().await.map_err(map_multipart_error)? {
push_capped(&mut bytes, &chunk, max_size, field_name)?;
}
Ok(bytes)
}
/// Append `chunk` to `buf`, rejecting with 413 as soon as the running total
/// would exceed `max_size` — so the oversized chunk is never copied in. Split
/// out from the read loop so the cap logic is unit-testable without a live
/// multipart field.
fn push_capped(
buf: &mut Vec<u8>,
chunk: &[u8],
max_size: usize,
field_name: &str,
) -> AppResult<()> {
if buf.len().saturating_add(chunk.len()) > max_size {
return Err(AppError::PayloadTooLarge(format!(
"{field_name} exceeds {max_size}-byte cap"
)));
}
buf.extend_from_slice(chunk);
Ok(())
}
pub fn parse_image(bytes: Vec<u8>, max_size: usize, field_name: &str) -> AppResult<UploadedImage> {
if bytes.len() > max_size {
return Err(AppError::PayloadTooLarge(format!(
@@ -114,4 +213,30 @@ mod tests {
assert!(matches!(err, AppError::PayloadTooLarge(_)));
assert_eq!(err.code(), "payload_too_large");
}
#[test]
fn push_capped_accumulates_under_cap() {
let mut buf = Vec::new();
push_capped(&mut buf, b"hello ", 100, "cover").unwrap();
push_capped(&mut buf, b"world", 100, "cover").unwrap();
assert_eq!(buf, b"hello world");
}
#[test]
fn push_capped_rejects_before_copying_oversized_chunk() {
// The whole point: the offending chunk must NOT be appended — an
// oversized part is rejected without buffering it.
let mut buf = vec![0u8; 90];
let err = push_capped(&mut buf, &[0u8; 20], 100, "cover").unwrap_err();
assert!(matches!(err, AppError::PayloadTooLarge(_)));
assert_eq!(err.code(), "payload_too_large");
assert_eq!(buf.len(), 90, "buffer must not grow past the cap");
}
#[test]
fn push_capped_allows_exactly_at_cap() {
let mut buf = vec![0u8; 90];
push_capped(&mut buf, &[0u8; 10], 100, "cover").unwrap();
assert_eq!(buf.len(), 100);
}
}

View File

@@ -0,0 +1,194 @@
//! Integration tests for the bucketed trend-series queries + endpoints:
//! `repo::crawl_metrics::series`, `repo::page_analysis::analysis_series`, and
//! `GET /v1/admin/{crawler,analysis}/metrics/series`.
mod common;
use axum::http::StatusCode;
use axum::Router;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
use common::{body_json, get_with_cookie, harness, register_user};
use mangalord::repo::crawl_metrics::{self, Bucket};
use mangalord::repo::page_analysis;
async fn seed_admin(pool: &PgPool, app: &Router) -> String {
let (username, cookie) = register_user(app).await;
let u = mangalord::repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
mangalord::repo::user::set_is_admin_unchecked(pool, u.id, true)
.await
.unwrap();
cookie
}
async fn insert_metric(pool: &PgPool, outcome: &str, duration_ms: i64, finished_at: &str) {
sqlx::query(
"INSERT INTO crawl_metrics (op, outcome, duration_ms, finished_at) \
VALUES ('manga_detail', $1, $2, $3::timestamptz)",
)
.bind(outcome)
.bind(duration_ms)
.bind(finished_at)
.execute(pool)
.await
.unwrap();
}
#[sqlx::test(migrations = "./migrations")]
async fn crawl_series_buckets_by_hour_and_day(pool: PgPool) {
// Two rows in hour 09, one in hour 10 (same day).
insert_metric(&pool, "ok", 100, "2026-06-18T09:10:00Z").await;
insert_metric(&pool, "failed", 300, "2026-06-18T09:50:00Z").await;
insert_metric(&pool, "ok", 200, "2026-06-18T10:05:00Z").await;
let hourly = crawl_metrics::series(&pool, Bucket::Hour, None).await.unwrap();
assert_eq!(hourly.len(), 2, "two hour buckets");
// Ordered ascending; first bucket = hour 09 with 2 rows (1 ok / 1 failed).
assert_eq!(hourly[0].n, 2);
assert_eq!(hourly[0].ok, 1);
assert_eq!(hourly[0].failed, 1);
assert_eq!(hourly[0].avg_ms, Some(200.0)); // (100+300)/2
assert_eq!(hourly[1].n, 1);
let daily = crawl_metrics::series(&pool, Bucket::Day, None).await.unwrap();
assert_eq!(daily.len(), 1, "all three collapse into one day bucket");
assert_eq!(daily[0].n, 3);
assert_eq!(daily[0].failed, 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn crawl_series_respects_since_window(pool: PgPool) {
insert_metric(&pool, "ok", 100, "2020-01-01T00:00:00Z").await; // ancient
insert_metric(&pool, "ok", 100, "2026-06-18T10:00:00Z").await; // recent
let since = chrono::DateTime::parse_from_rfc3339("2026-01-01T00:00:00Z")
.unwrap()
.with_timezone(&chrono::Utc);
let got = crawl_metrics::series(&pool, Bucket::Day, Some(since))
.await
.unwrap();
assert_eq!(got.len(), 1, "only the in-window row");
}
async fn seed_page(pool: &PgPool) -> Uuid {
let manga = Uuid::new_v4();
let chapter = Uuid::new_v4();
let page = Uuid::new_v4();
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, 'M')")
.bind(manga)
.execute(pool)
.await
.unwrap();
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
.bind(chapter)
.bind(manga)
.execute(pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO pages (id, chapter_id, page_number, storage_key, content_type) \
VALUES ($1, $2, 1, 'k', 'image/png')",
)
.bind(page)
.bind(chapter)
.execute(pool)
.await
.unwrap();
page
}
async fn insert_analysis(pool: &PgPool, page_id: Uuid, status: &str, dur: i64, analyzed_at: &str) {
sqlx::query(
"INSERT INTO page_analysis (page_id, status, duration_ms, analyzed_at) \
VALUES ($1, $2, $3, $4::timestamptz)",
)
.bind(page_id)
.bind(status)
.bind(dur)
.bind(analyzed_at)
.execute(pool)
.await
.unwrap();
}
#[sqlx::test(migrations = "./migrations")]
async fn analysis_series_splits_ok_failed_by_bucket(pool: PgPool) {
let p1 = seed_page(&pool).await;
let p2 = seed_page(&pool).await;
let p3 = seed_page(&pool).await;
insert_analysis(&pool, p1, "done", 500, "2026-06-18T09:00:00Z").await;
insert_analysis(&pool, p2, "failed", 999, "2026-06-18T09:30:00Z").await;
insert_analysis(&pool, p3, "done", 700, "2026-06-18T11:00:00Z").await;
let hourly = page_analysis::analysis_series(&pool, Bucket::Hour, None)
.await
.unwrap();
assert_eq!(hourly.len(), 2);
assert_eq!(hourly[0].n, 2);
assert_eq!(hourly[0].ok, 1);
assert_eq!(hourly[0].failed, 1);
// Mean duration is done-only → 500 (the failed 999 is excluded).
assert_eq!(hourly[0].avg_ms, Some(500.0));
}
#[sqlx::test(migrations = "./migrations")]
async fn crawler_series_endpoint_shape_and_gating(pool: PgPool) {
let h = harness(pool.clone());
// Non-admin → 403.
let (_u, plain) = register_user(&h.app).await;
let resp = h
.app
.clone()
.oneshot(get_with_cookie("/api/v1/admin/crawler/metrics/series", &plain))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
let cookie = seed_admin(&pool, &h.app).await;
insert_metric(&pool, "ok", 100, "2026-06-18T09:10:00Z").await;
let resp = h
.app
.clone()
.oneshot(get_with_cookie(
"/api/v1/admin/crawler/metrics/series?days=1",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
assert!(body["buckets"].is_array());
// Bad bucket → 400.
let resp = h
.app
.clone()
.oneshot(get_with_cookie(
"/api/v1/admin/crawler/metrics/series?bucket=year",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
}
#[sqlx::test(migrations = "./migrations")]
async fn analysis_series_endpoint_ok(pool: PgPool) {
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(get_with_cookie(
"/api/v1/admin/analysis/metrics/series?days=7",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
assert!(body["buckets"].is_array());
}

View File

@@ -0,0 +1,140 @@
//! Integration tests for the OCR analysis backend
//! (`analysis::ocr::OcrAnalyzeDispatcher`). A stub OCR engine stands in for
//! `ocrs` (whose `.rten` models aren't shipped to CI), so these pin the
//! storage→OCR→persist wiring: the dispatcher reads the page image, runs the
//! engine, and persists the lines via the shared `persist_analysis` path —
//! landing `page_ocr_text` rows and a populated `search_doc` exactly like the
//! vision backend. Each `#[sqlx::test]` gets a fresh migrated DB.
mod common;
use std::sync::Arc;
use mangalord::analysis::daemon::AnalyzeDispatcher;
use mangalord::analysis::ocr::test_support::StubOcrEngine;
use mangalord::analysis::ocr::OcrAnalyzeDispatcher;
use mangalord::domain::page_analysis::AnalysisStatus;
use mangalord::repo;
use mangalord::storage::{LocalStorage, Storage};
use sqlx::PgPool;
use tempfile::TempDir;
use uuid::Uuid;
/// Seed a manga → chapter → page chain whose page points at `storage_key`,
/// and return the page id.
async fn seed_page(pool: &PgPool, storage_key: &str) -> Uuid {
let manga_id: Uuid =
sqlx::query_scalar("INSERT INTO mangas (title) VALUES ('M') RETURNING id")
.fetch_one(pool)
.await
.unwrap();
let chapter_id: Uuid = sqlx::query_scalar(
"INSERT INTO chapters (manga_id, number) VALUES ($1, 1) RETURNING id",
)
.bind(manga_id)
.fetch_one(pool)
.await
.unwrap();
sqlx::query_scalar(
"INSERT INTO pages (chapter_id, page_number, storage_key, content_type) \
VALUES ($1, 1, $2, 'image/png') RETURNING id",
)
.bind(chapter_id)
.bind(storage_key)
.fetch_one(pool)
.await
.unwrap()
}
fn ocr_dispatcher(
pool: &PgPool,
storage: Arc<dyn Storage>,
lines: &[&str],
) -> OcrAnalyzeDispatcher {
OcrAnalyzeDispatcher {
db: pool.clone(),
storage,
engine: StubOcrEngine::new(lines),
max_image_bytes: 8 * 1024 * 1024,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(1)),
}
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_persists_ocr_lines_and_search_doc(pool: PgPool) {
let dir = TempDir::new().unwrap();
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(dir.path()));
let key = "mangas/x/p1.png";
storage.put(key, &common::fake_png_bytes()).await.unwrap();
let page_id = seed_page(&pool, key).await;
let dispatcher = ocr_dispatcher(&pool, Arc::clone(&storage), &["Hello there", "general"]);
dispatcher.dispatch(page_id).await.unwrap();
// Two OCR rows, in order, with the recognized text.
let rows: Vec<(String, i32)> = sqlx::query_as(
"SELECT text, ord FROM page_ocr_text WHERE page_id = $1 ORDER BY ord",
)
.bind(page_id)
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(rows.len(), 2);
assert_eq!(rows[0].0, "Hello there");
assert_eq!(rows[1].0, "general");
// The analysis row is `done`, stamped with the ocrs model label, and has a
// non-empty tsvector so text search works.
let row = repo::page_analysis::load(&pool, page_id).await.unwrap().unwrap();
assert_eq!(row.status, AnalysisStatus::Done);
assert_eq!(row.model.as_deref(), Some("ocrs"));
let has_doc: bool = sqlx::query_scalar(
"SELECT search_doc IS NOT NULL AND search_doc != ''::tsvector \
FROM page_analysis WHERE page_id = $1",
)
.bind(page_id)
.fetch_one(&pool)
.await
.unwrap();
assert!(has_doc, "search_doc must be populated from OCR text");
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_rejects_page_image_over_the_byte_cap(pool: PgPool) {
let dir = TempDir::new().unwrap();
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(dir.path()));
let key = "mangas/x/big.png";
// 4 KiB on disk, 1 KiB cap — the streamed read must bail on the cap
// rather than buffering the whole blob and OCR-ing it.
storage.put(key, &vec![0u8; 4096]).await.unwrap();
let page_id = seed_page(&pool, key).await;
let dispatcher = OcrAnalyzeDispatcher {
db: pool.clone(),
storage: Arc::clone(&storage),
engine: StubOcrEngine::new(&["should not run"]),
max_image_bytes: 1024,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(1)),
};
let err = dispatcher.dispatch(page_id).await.unwrap_err();
assert!(
err.chain().any(|c| c.to_string().contains("cap")),
"expected an over-cap error, got: {err:#}"
);
// A rejected page must not land an analysis row.
assert!(repo::page_analysis::load(&pool, page_id)
.await
.unwrap()
.is_none());
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_missing_page_is_noop(pool: PgPool) {
let dir = TempDir::new().unwrap();
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(dir.path()));
// A page id that was never inserted — the dispatcher must treat it as a
// deleted page and succeed without writing anything.
let dispatcher = ocr_dispatcher(&pool, storage, &["whatever"]);
dispatcher.dispatch(Uuid::new_v4()).await.unwrap();
}

View File

@@ -5,7 +5,11 @@
use std::sync::Arc;
use std::time::Duration;
use mangalord::analysis::daemon::{self, test_support::CountingDispatcher, AnalysisDaemonConfig};
use mangalord::analysis::daemon::{
self,
test_support::{CountingDispatcher, SlowDispatcher},
AnalysisDaemonConfig,
};
use mangalord::domain::page_analysis::{
AnalysisStatus, SafetyFlag, VisionAnalysis,
};
@@ -196,6 +200,80 @@ async fn worker_marks_failed_row_on_terminal_failure(pool: PgPool) {
assert!(row.error.is_some());
}
/// Non-terminal failure of a force-re-analyze must NOT overwrite the
/// prior `done` row's `duration_ms`. Before the 0.87.6 gate change,
/// `record_duration` ran unconditionally — a transient failure-retry
/// of a force-re-analyze would clobber the legitimately-measured
/// previous duration with the duration of an attempt that wrote
/// nothing. Test pins the gate.
#[sqlx::test(migrations = "./migrations")]
async fn worker_non_terminal_force_failure_does_not_overwrite_done_duration(pool: PgPool) {
let page_id = seed_page(&pool).await;
// 1) Pre-seed a done analysis row with a meaningful duration so we
// have something to be "overwritten".
page_analysis::persist_analysis(
&pool,
page_id,
&VisionAnalysis {
ocr_results: vec![],
tagging_results: vec![],
scene_description: String::new(),
safety_flag: SafetyFlag::default(),
},
"m",
)
.await
.unwrap();
sqlx::query("UPDATE page_analysis SET duration_ms = $1 WHERE page_id = $2")
.bind(12345_i64)
.bind(page_id)
.execute(&pool)
.await
.unwrap();
// 2) Force re-analyze (force=true) — but with a failing dispatcher
// AND max_attempts=3 so the first failure is NON-terminal.
page_analysis::enqueue_for_page(&pool, page_id, true)
.await
.unwrap();
sqlx::query("UPDATE crawler_jobs SET max_attempts = 3 WHERE payload->>'page_id' = $1")
.bind(page_id.to_string())
.execute(&pool)
.await
.unwrap();
let dispatcher = CountingDispatcher::failing();
let (handle, cancel) = spawn_with(&pool, dispatcher.clone());
// 3) Wait until the worker has dispatched once — the job goes back
// to `pending` because the retry is non-terminal (attempts < max).
let deadline = std::time::Instant::now() + Duration::from_secs(5);
while dispatcher.call_count() == 0 && std::time::Instant::now() < deadline {
tokio::time::sleep(Duration::from_millis(50)).await;
}
assert!(dispatcher.call_count() >= 1, "dispatcher must have run");
// Wait briefly for `record_duration` to either run or get gated.
// Then cancel before the next backoff fires (it'd burn 60s otherwise).
tokio::time::sleep(Duration::from_millis(200)).await;
cancel.cancel();
handle.shutdown().await;
// 4) The prior `done` row's `duration_ms` must NOT have been
// overwritten by the failed retry's duration.
let dur: Option<i64> =
sqlx::query_scalar("SELECT duration_ms FROM page_analysis WHERE page_id = $1")
.bind(page_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
dur,
Some(12345),
"non-terminal failure must not overwrite the prior done row's duration"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn worker_isolates_dispatcher_panics(pool: PgPool) {
let page_id = seed_page(&pool).await;
@@ -317,6 +395,8 @@ async fn worker_ignores_non_analyze_jobs(pool: PgPool) {
&JobPayload::SyncManga {
source_id: "s".into(),
source_manga_key: "k".into(),
url: "https://target.example/manga/k".into(),
title: "K".into(),
},
)
.await
@@ -344,3 +424,161 @@ async fn worker_ignores_non_analyze_jobs(pool: PgPool) {
.unwrap();
assert_eq!(crawl_state, "pending", "crawl job must be left for the crawler");
}
/// Shutdown must race in-flight dispatches against cancellation, not wait
/// for them to finish. The previous behaviour blocked `handle.shutdown()`
/// for up to `job_timeout` (default 600s) on one in-flight vision call,
/// which also stranded SIGTERM and any `Supervisors::reload_analysis`
/// caller under the supervisor lock.
#[sqlx::test(migrations = "./migrations")]
async fn worker_shutdown_does_not_block_on_in_flight_dispatch(pool: PgPool) {
let page_id = seed_page(&pool).await;
page_analysis::enqueue_for_page(&pool, page_id, false)
.await
.unwrap();
// 30s dispatch + 30s job_timeout. Shutdown must observe the cancel
// signal and return well before either deadline; if it waits for
// dispatch to finish the test would take 30s.
let dispatcher = SlowDispatcher::new(Duration::from_secs(30));
let cancel = CancellationToken::new();
let handle = daemon::spawn(
pool.clone(),
cancel.clone(),
AnalysisDaemonConfig {
dispatcher: dispatcher.clone(),
workers: 1,
job_timeout: Duration::from_secs(30),
events: std::sync::Arc::new(
mangalord::analysis::events::AnalysisEvents::new(),
),
readiness: None,
},
);
// Wait for the worker to lease the job and enter the dispatch.
wait_for_state(&pool, page_id, "running").await;
assert_eq!(
dispatcher.call_count(),
1,
"dispatcher should be in flight when we cancel"
);
// Now cancel. Shutdown must return within a few seconds — comfortably
// under both the slow-dispatch and the job_timeout deadlines.
let started = std::time::Instant::now();
tokio::time::timeout(Duration::from_secs(5), handle.shutdown())
.await
.expect("shutdown must observe cancel and return promptly");
assert!(
started.elapsed() < Duration::from_secs(5),
"shutdown took too long; cancel didn't propagate into dispatch"
);
// The lease was released, not failed — attempts must not have grown.
let attempts: i32 = sqlx::query_scalar(
"SELECT attempts FROM crawler_jobs \
WHERE payload->>'kind' = 'analyze_page' AND payload->>'page_id' = $1",
)
.bind(page_id.to_string())
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
attempts, 0,
"cancellation must not burn a job attempt (release, not ack_failed)"
);
}
/// Cancel-mid-dispatch must publish a `Cancelled` event so the admin
/// dashboard's "now analyzing" banner clears — it listens only for
/// `Completed`/`Failed`/`Cancelled` and would otherwise stick on the
/// in-flight page until a later one overwrites it. 0.87.13 added the
/// publish but no test asserted the frame; mechanical revert of the
/// publish block ships green under
/// `worker_shutdown_does_not_block_on_in_flight_dispatch` because that
/// test builds `AnalysisEvents::new()` but never subscribes.
#[sqlx::test(migrations = "./migrations")]
async fn worker_publishes_cancelled_event_on_cancel_mid_dispatch(pool: PgPool) {
let page_id = seed_page(&pool).await;
page_analysis::enqueue_for_page(&pool, page_id, false)
.await
.unwrap();
// Subscribe BEFORE spawning so we never miss a frame.
let events =
std::sync::Arc::new(mangalord::analysis::events::AnalysisEvents::new());
let mut rx = events.subscribe();
let dispatcher = SlowDispatcher::new(Duration::from_secs(30));
let cancel = CancellationToken::new();
let handle = daemon::spawn(
pool.clone(),
cancel.clone(),
AnalysisDaemonConfig {
dispatcher: dispatcher.clone(),
workers: 1,
job_timeout: Duration::from_secs(30),
events: events.clone(),
readiness: None,
},
);
// Wait for the worker to lease the job and enter the dispatch so
// it has already published `Started`.
wait_for_state(&pool, page_id, "running").await;
// Cancel.
cancel.cancel();
tokio::time::timeout(Duration::from_secs(5), handle.shutdown())
.await
.expect("shutdown must observe cancel and return promptly");
// Drain the broadcaster. We must observe `Started` (for our page_id)
// followed by `Cancelled` (for the same page_id). Drop any other
// frames (Enqueued from the initial enqueue path, frames for other
// pages a sibling test might leak via the shared schema).
let mut saw_started = false;
let mut saw_cancelled = false;
let _ = tokio::time::timeout(Duration::from_secs(2), async {
loop {
match rx.try_recv() {
Ok(ev) => {
let v = serde_json::to_value(&ev).unwrap();
let kind = v["kind"].as_str().unwrap_or("");
let ev_page = v["page_id"].as_str().map(str::to_string);
if ev_page.as_deref() != Some(&page_id.to_string()) {
continue;
}
if kind == "started" {
saw_started = true;
} else if kind == "cancelled" {
saw_cancelled = true;
// Sanity-check the labelled breadcrumb fields
// the dashboard banner needs to render.
assert!(v["manga_id"].as_str().is_some());
assert!(v["manga_title"].as_str().is_some());
assert!(v["chapter_id"].as_str().is_some());
assert!(v["chapter_number"].is_number());
assert!(v["page_number"].is_number());
break;
}
}
Err(tokio::sync::broadcast::error::TryRecvError::Empty) => {
tokio::time::sleep(Duration::from_millis(20)).await;
}
Err(e) => panic!("subscribe error: {e:?}"),
}
}
})
.await;
assert!(
saw_started,
"worker should publish a Started frame for the leased page before the cancel"
);
assert!(
saw_cancelled,
"worker must publish a Cancelled frame on cancel-mid-dispatch so the dashboard banner clears"
);
}

View File

@@ -74,6 +74,38 @@ async fn analyze_job_count(pool: &PgPool) -> i64 {
.unwrap()
}
#[sqlx::test(migrations = "./migrations")]
async fn force_analyze_enqueue_rolls_back_with_its_transaction(pool: PgPool) {
// The admin force-reanalyze handler runs enqueue_for_page_conn + the audit
// insert in one transaction. Prove the enqueue genuinely participates in
// that tx: when the tx is abandoned (the path taken if the audit insert
// fails and the `?` propagates), no job survives — so the audit trail can
// never miss a force-reanalyze that actually landed.
let page_id = seed_pages(&pool, 1).await[0];
let mut tx = pool.begin().await.unwrap();
let outcome =
mangalord::repo::page_analysis::enqueue_for_page_conn(&mut tx, page_id, true)
.await
.unwrap();
assert!(matches!(
outcome,
mangalord::repo::page_analysis::EnqueueForPageOutcome::Inserted
));
// Visible inside the open transaction…
let in_tx: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM crawler_jobs WHERE payload->>'kind' = 'analyze_page'",
)
.fetch_one(&mut *tx)
.await
.unwrap();
assert_eq!(in_tx, 1);
// …but gone once the tx rolls back instead of committing.
tx.rollback().await.unwrap();
assert_eq!(analyze_job_count(&pool).await, 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_upload_enqueues_one_analysis_job_per_page(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
@@ -157,6 +189,51 @@ async fn reenqueue_returns_503_when_analysis_disabled(pool: PgPool) {
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_rejects_a_malformed_body(pool: PgPool) {
// A present-but-malformed JSON body must be a 422 — NOT silently coerced to
// an absent body, which would run the default full "All" scope the caller
// never asked for and enqueue jobs across the whole library.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_raw_with_cookie(
"/api/v1/admin/analysis/reenqueue",
"{ not valid json",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNPROCESSABLE_ENTITY);
assert_eq!(analyze_job_count(&pool).await, 0, "a rejected body enqueues nothing");
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_treats_empty_body_as_default_all_scope(pool: PgPool) {
// An absent/empty body is still valid: it means the default "All" scope.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_raw_with_cookie(
"/api/v1/admin/analysis/reenqueue",
"",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["enqueued"], 2, "empty body backfills the whole library");
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_backfills_existing_pages(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
@@ -194,6 +271,38 @@ async fn reenqueue_backfills_existing_pages(pool: PgPool) {
assert_eq!(analyze_job_count(&pool).await, 3);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_writes_audit_row_atomically_with_jobs(pool: PgPool) {
// The enqueue and its admin_audit row are committed in one transaction,
// so a successful re-enqueue always leaves both the jobs AND exactly one
// matching audit row — the audit trail can't miss an enqueue that landed.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/analysis/reenqueue",
json!({ "only_unanalyzed": true }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(analyze_job_count(&pool).await, 2);
let audit: serde_json::Value = sqlx::query_scalar(
"SELECT payload FROM admin_audit WHERE action = 'analysis_reenqueue'",
)
.fetch_one(&pool)
.await
.expect("exactly one analysis_reenqueue audit row committed with the jobs");
assert_eq!(audit["enqueued"], 2);
assert_eq!(audit["only_unanalyzed"], true);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_scoped_to_manga(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
@@ -392,6 +501,167 @@ async fn force_analyze_page_enqueues_with_force_flag(pool: PgPool) {
assert_eq!(payload["force"].as_bool(), Some(true));
}
#[sqlx::test(migrations = "./migrations")]
async fn force_analyze_upgrades_pending_non_force_to_force(pool: PgPool) {
// The cover-up case the partial unique index used to silently
// create: when a `force=false` job is already pending and admin
// clicks force, the INSERT used to dedup-skip silently. The worker
// then picked up the non-force row, hit skip-if-done on a `done`
// page, and admin saw "queued" with no re-analysis. 0.87.11
// upgrades the pending row in-place instead.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let ids = seed_pages(&pool, 1).await;
let page_id = ids[0];
// 1) Pre-existing non-force pending row (simulates the crawler's
// post-upload enqueue path).
mangalord::repo::page_analysis::enqueue_for_page(&pool, page_id, false)
.await
.unwrap();
let payload_before: serde_json::Value = sqlx::query_scalar(
"SELECT payload FROM crawler_jobs WHERE payload->>'kind' = 'analyze_page'",
)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(payload_before["force"].as_bool(), Some(false));
// 2) Admin force-click.
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
&format!("/api/v1/admin/pages/{page_id}/analyze"),
json!({}),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
// 3) STILL one row (no dedup duplicate), but its `force` flag
// flipped to true — the admin intent landed where the worker
// can see it.
let rows: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM crawler_jobs WHERE payload->>'kind' = 'analyze_page'",
)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(rows, 1, "upgrade in place; no duplicate row");
let payload_after: serde_json::Value = sqlx::query_scalar(
"SELECT payload FROM crawler_jobs WHERE payload->>'kind' = 'analyze_page'",
)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(payload_after["force"].as_bool(), Some(true));
// 4) Audit row carries the outcome so a moderator can tell
// upgrade-paths from fresh-inserts.
let audit_payload: serde_json::Value = sqlx::query_scalar(
"SELECT payload FROM admin_audit WHERE action = 'analysis_force_page'",
)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
audit_payload["outcome"].as_str(),
Some("upgraded_pending_to_force")
);
}
/// The running-state branch of `force=true` collision. A worker has
/// already leased the row (state=running, payload.force=false in its
/// in-memory `lease.payload`). Admin force-clicks → my code must NOT
/// only flip the row's payload to force=true (the worker holds the
/// stale value) but also release the in-flight lease so the worker
/// drops the stale payload and a fresh lease re-leases with the
/// updated value.
///
/// Mutation-confirmed: removing the `jobs::release` loop in
/// `enqueue_for_page` (leaving only the UPDATE) makes this test fail
/// — the row's state stays `running` until the lease expires.
#[sqlx::test(migrations = "./migrations")]
async fn force_analyze_releases_running_lease_so_worker_re_picks_force(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let ids = seed_pages(&pool, 1).await;
let page_id = ids[0];
// 1) Pre-existing non-force pending row.
mangalord::repo::page_analysis::enqueue_for_page(&pool, page_id, false)
.await
.unwrap();
// 2) Simulate a worker leasing the job — state moves to `running`
// with leased_until in the future and attempts=1. The worker
// process (in real life) destructures its in-memory `lease.payload`
// here, capturing `force=false` as a Rust local.
let leases = mangalord::crawler::jobs::lease(
&pool,
Some(mangalord::crawler::jobs::KIND_ANALYZE_PAGE),
1,
std::time::Duration::from_secs(300),
)
.await
.unwrap();
assert_eq!(leases.len(), 1, "test setup: should have leased the seeded job");
let leased_id = leases[0].id;
// Sanity: row is now running with force=false in payload.
let (state, force): (String, bool) = sqlx::query_as(
"SELECT state, (payload->>'force')::boolean \
FROM crawler_jobs WHERE id = $1",
)
.bind(leased_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(state, "running");
assert!(!force);
// 3) Admin force-clicks. Without 0.87.20's release loop, the row's
// payload flips to force=true but state stays `running`; the
// worker (holding its stale `force=false` local) acks done and
// the page is never re-analyzed. With the release loop, the row
// goes back to `pending`.
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
&format!("/api/v1/admin/pages/{page_id}/analyze"),
json!({}),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
// 4) Row is back to pending, with the payload's force flag flipped
// to true. A subsequent worker lease would now see `force=true`
// and re-analyze. attempts is decremented by `jobs::release` so
// the page doesn't burn its retry budget on the admin click.
let (state, force, attempts): (String, bool, i32) = sqlx::query_as(
"SELECT state, (payload->>'force')::boolean, attempts \
FROM crawler_jobs WHERE id = $1",
)
.bind(leased_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
state, "pending",
"running lease must be released so the worker drops its stale-payload local"
);
assert!(force, "payload's force flag must be flipped to true");
assert_eq!(
attempts, 0,
"release refunds the attempt — the admin click doesn't burn the retry budget"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn force_analyze_unknown_page_is_404(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());

View File

@@ -0,0 +1,175 @@
//! Integration tests for the admin audit-log viewer: the `repo::admin_audit`
//! list query (filters, username join, NULL actor) and the
//! `GET /v1/admin/audit` endpoint (admin-gating, shape, filter params).
mod common;
use axum::http::StatusCode;
use axum::Router;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
use common::{body_json, get_with_cookie, harness, register_user};
use mangalord::repo::admin_audit::{self, AuditFilter};
async fn seed_admin(pool: &PgPool, app: &Router) -> (Uuid, String) {
let (username, cookie) = register_user(app).await;
let u = mangalord::repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
mangalord::repo::user::set_is_admin_unchecked(pool, u.id, true)
.await
.unwrap();
(u.id, cookie)
}
/// Insert an audit row with an explicit `at` so ordering/since are testable.
async fn insert_audit(
pool: &PgPool,
actor: Option<Uuid>,
action: &str,
target_kind: &str,
at: &str,
) {
sqlx::query(
"INSERT INTO admin_audit (actor_user_id, action, target_kind, target_id, payload, at) \
VALUES ($1, $2, $3, NULL, $4, $5::timestamptz)",
)
.bind(actor)
.bind(action)
.bind(target_kind)
.bind(json!({ "k": action }))
.bind(at)
.execute(pool)
.await
.unwrap();
}
#[sqlx::test(migrations = "./migrations")]
async fn list_joins_username_and_orders_newest_first(pool: PgPool) {
let h = harness(pool.clone());
let (admin_id, _cookie) = seed_admin(&pool, &h.app).await;
insert_audit(&pool, Some(admin_id), "crawler_run", "crawler", "2026-06-01T00:00:00Z").await;
insert_audit(&pool, None, "manga_resync", "manga", "2026-06-02T00:00:00Z").await;
let (items, total) = admin_audit::list(&pool, AuditFilter::default(), 50, 0)
.await
.unwrap();
assert_eq!(total, 2);
// Newest first.
assert_eq!(items[0].action, "manga_resync");
assert_eq!(items[1].action, "crawler_run");
// Username join: the admin actor resolves; the NULL actor stays None.
assert_eq!(items[1].actor_user_id, Some(admin_id));
assert!(items[1].actor_username.is_some());
assert_eq!(items[0].actor_user_id, None);
assert_eq!(items[0].actor_username, None);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_filters_by_action_target_kind_actor_and_since(pool: PgPool) {
let h = harness(pool.clone());
let (admin_id, _cookie) = seed_admin(&pool, &h.app).await;
insert_audit(&pool, Some(admin_id), "crawler_run", "crawler", "2026-05-01T00:00:00Z").await;
insert_audit(&pool, Some(admin_id), "manga_resync", "manga", "2026-06-10T00:00:00Z").await;
insert_audit(&pool, None, "crawler_run", "crawler", "2026-06-11T00:00:00Z").await;
// Filter by action.
let (items, total) = admin_audit::list(
&pool,
AuditFilter { action: Some("crawler_run"), ..Default::default() },
50,
0,
)
.await
.unwrap();
assert_eq!(total, 2);
assert!(items.iter().all(|r| r.action == "crawler_run"));
// Filter by target_kind.
let (_items, total) = admin_audit::list(
&pool,
AuditFilter { target_kind: Some("manga"), ..Default::default() },
50,
0,
)
.await
.unwrap();
assert_eq!(total, 1);
// Filter by actor.
let (_items, total) = admin_audit::list(
&pool,
AuditFilter { actor_user_id: Some(admin_id), ..Default::default() },
50,
0,
)
.await
.unwrap();
assert_eq!(total, 2);
// Filter by since (only the two June rows).
let since = chrono::DateTime::parse_from_rfc3339("2026-06-01T00:00:00Z")
.unwrap()
.with_timezone(&chrono::Utc);
let (_items, total) = admin_audit::list(
&pool,
AuditFilter { since: Some(since), ..Default::default() },
50,
0,
)
.await
.unwrap();
assert_eq!(total, 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn endpoint_requires_admin(pool: PgPool) {
let h = harness(pool.clone());
let (_u, cookie) = register_user(&h.app).await;
let resp = h
.app
.oneshot(get_with_cookie("/api/v1/admin/audit", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn endpoint_returns_paged_shape_and_honors_filter(pool: PgPool) {
let h = harness(pool.clone());
let (admin_id, cookie) = seed_admin(&pool, &h.app).await;
insert_audit(&pool, Some(admin_id), "crawler_run", "crawler", "2026-06-01T00:00:00Z").await;
insert_audit(&pool, Some(admin_id), "manga_resync", "manga", "2026-06-02T00:00:00Z").await;
// Unfiltered: both rows + paged envelope.
let resp = h
.app
.clone()
.oneshot(get_with_cookie("/api/v1/admin/audit", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
assert_eq!(body["page"]["total"], 2);
assert_eq!(body["items"].as_array().unwrap().len(), 2);
assert!(body["items"][0]["actor_username"].is_string());
// Filter by action.
let resp = h
.app
.clone()
.oneshot(get_with_cookie(
"/api/v1/admin/audit?action=manga_resync",
&cookie,
))
.await
.unwrap();
let body = body_json(resp).await;
assert_eq!(body["page"]["total"], 1);
assert_eq!(body["items"][0]["action"], "manga_resync");
assert_eq!(body["items"][0]["target_kind"], "manga");
}

View File

@@ -206,6 +206,7 @@ async fn control_endpoints_return_503_when_daemon_disabled(pool: PgPool) {
let cookie = seed_admin(&pool, &h.app).await;
for uri in [
"/api/v1/admin/crawler/run",
"/api/v1/admin/crawler/reconcile",
"/api/v1/admin/crawler/browser/restart",
"/api/v1/admin/crawler/session/clear-expired",
] {
@@ -356,9 +357,12 @@ async fn csrf_allows_mutation_with_allowed_origin(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_allows_mutation_without_origin_or_referer(pool: PgPool) {
// curl/server-to-server callers send neither — they can't be a CSRF
// vector since there's no third-party browser context.
async fn csrf_rejects_cookie_auth_without_origin_or_referer(pool: PgPool) {
// Cookie-auth + neither Origin nor Referer was previously waved through
// ("curl bypass"). It's now refused: an extension or no-referrer-policy
// page can produce exactly this shape and would otherwise be a CSRF
// vector. Real curl/script callers should authenticate with a bearer
// token instead — see `csrf_bearer_only_request_bypasses_gate`.
let h = harness_with_admin_origins(
pool.clone(),
vec!["http://localhost:3000".to_string()],
@@ -367,16 +371,90 @@ async fn csrf_allows_mutation_without_origin_or_referer(pool: PgPool) {
let resp = h
.app
.clone()
.oneshot(post_json_with_cookie(
.oneshot(post_json_with_cookie_origin(
"/api/v1/admin/crawler/session/clear-expired",
json!({}),
&cookie,
None,
None,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_bearer_only_request_bypasses_gate(pool: PgPool) {
// Bearer-only (no session cookie, no Origin) bypasses the CSRF gate
// because a browser can't set `Authorization` on a cross-site POST.
// RequireAdmin then rejects bot tokens for admin endpoints — we
// expect 401, not the CSRF 403. The path proves the bypass branch
// is reachable for legitimate bot callers.
let h = harness_with_admin_origins(
pool.clone(),
vec!["http://localhost:3000".to_string()],
);
let req = axum::http::Request::builder()
.method("POST")
.uri("/api/v1/admin/crawler/session/clear-expired")
.header(axum::http::header::CONTENT_TYPE, "application/json")
.header(axum::http::header::AUTHORIZATION, "Bearer fake-bot-token")
.body(axum::body::Body::from("{}"))
.unwrap();
let resp = h.app.clone().oneshot(req).await.unwrap();
// 401 from RequireAdmin (bad/non-admin token), NOT 403 from CSRF.
// If the CSRF gate had fired, we'd get FORBIDDEN.
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_bearer_plus_cookie_still_enforces_gate(pool: PgPool) {
// The cookie-ride vector: an attacker page mints `Authorization:
// Bearer junk` on a cross-site credentialed POST. The cookie is
// attached automatically by the browser; the bearer header alone
// used to bypass CSRF, then `CurrentUser` authenticated the victim
// via the cookie. Cookie precedence here means: as soon as a
// session cookie is present, the CSRF gate fires regardless of
// Authorization — the bearer can't whitelist the cookie ride.
let h = harness_with_admin_origins(
pool.clone(),
vec!["http://localhost:3000".to_string()],
);
let cookie = seed_admin(&pool, &h.app).await;
let req = axum::http::Request::builder()
.method("POST")
.uri("/api/v1/admin/crawler/session/clear-expired")
.header(axum::http::header::CONTENT_TYPE, "application/json")
.header(axum::http::header::COOKIE, cookie)
.header(axum::http::header::AUTHORIZATION, "Bearer junk")
.header(axum::http::header::ORIGIN, "https://evil.example.com")
.body(axum::body::Body::from("{}"))
.unwrap();
let resp = h.app.clone().oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_no_auth_falls_through_to_401(pool: PgPool) {
// No cookie, no bearer — there's no authority to ride, so the CSRF
// gate yields and lets `RequireAdmin` return the more meaningful 401.
// Pins the carve-out that keeps anonymous curl-against-admin from
// 403-via-CSRF (which would mislead operators about what's blocking).
let h = harness_with_admin_origins(
pool.clone(),
vec!["http://localhost:3000".to_string()],
);
let req = axum::http::Request::builder()
.method("POST")
.uri("/api/v1/admin/crawler/session/clear-expired")
.header(axum::http::header::CONTENT_TYPE, "application/json")
.body(axum::body::Body::from("{}"))
.unwrap();
let resp = h.app.clone().oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_falls_back_to_referer_when_origin_missing(pool: PgPool) {
let h = harness_with_admin_origins(
@@ -421,11 +499,13 @@ async fn csrf_skipped_on_safe_methods(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn csrf_disabled_when_allowlist_empty(pool: PgPool) {
// Default harness has admin_allowed_origins = empty → operator
// opt-out documented in .env.example. A cross-origin POST passes
// through to the handler.
let h = harness(pool.clone());
async fn csrf_rejects_cookie_auth_when_allowlist_empty(pool: PgPool) {
// Previously an empty allowlist silently skipped the CSRF check, so an
// operator who forgot to set ADMIN_ALLOWED_ORIGINS shipped an unguarded
// admin surface. We now fail-closed for cookie-auth admin mutations
// when the allowlist is empty — operators must either set the env var
// (browser deploy) or authenticate with `Authorization: Bearer …`.
let h = harness_with_admin_origins(pool.clone(), vec![]);
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
@@ -439,7 +519,7 @@ async fn csrf_disabled_when_allowlist_empty(pool: PgPool) {
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
// ---------------------------------------------------------------------------
@@ -779,6 +859,45 @@ async fn job_history_lists_filters_and_paginates(pool: PgPool) {
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn job_history_surfaces_sync_manga_payload_title(pool: PgPool) {
// A reconcile-enqueued sync_manga job that died has no manga row, so the
// history row must fall back to the payload title/url and be searchable.
sqlx::query("INSERT INTO crawler_jobs (id, payload, state, last_error) VALUES ($1, $2, 'dead', 'broken-page body signature')")
.bind(Uuid::new_v4())
.bind(json!({
"kind": "sync_manga",
"source_id": "target",
"source_manga_key": "gone-1",
"url": "http://x/manga/gone-1",
"title": "Ghost Manga",
}))
.execute(&pool)
.await
.unwrap();
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
.clone()
.oneshot(get_with_cookie(
"/api/v1/admin/crawler/history?search=ghost",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
assert_eq!(body["page"]["total"], 1);
let row = &body["items"][0];
assert_eq!(row["kind"], "sync_manga");
assert_eq!(row["manga_title"], serde_json::Value::Null);
assert_eq!(row["payload_title"], "Ghost Manga");
assert_eq!(row["source_url"], "http://x/manga/gone-1");
assert_eq!(row["source_key"], "gone-1");
}
// ---------------------------------------------------------------------------
// Operation metrics: durations, averages, recent-ops log.
// ---------------------------------------------------------------------------

View File

@@ -0,0 +1,197 @@
//! Integration tests for the admin health endpoints: `GET /v1/admin/health`
//! (shape, admin-gating, default checks present without a daemon) and
//! `PUT /v1/admin/health/thresholds` (validation, persistence, audit row).
mod common;
use axum::http::StatusCode;
use axum::Router;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use common::{body_json, get_with_cookie, harness, put_json_with_cookie, register_user};
async fn seed_admin(pool: &PgPool, app: &Router) -> String {
let (username, cookie) = register_user(app).await;
let u = mangalord::repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
mangalord::repo::user::set_is_admin_unchecked(pool, u.id, true)
.await
.unwrap();
cookie
}
/// PUT helper (the common module exposes POST helpers; build a PUT here).
fn valid_thresholds() -> serde_json::Value {
json!({
"disk_pct": 85.0,
"mem_pct": 80.0,
"dead_jobs_max": 50,
"crawl_fail_pct": 25.0,
"analysis_fail_pct": 30.0,
"cron_freshness_hours": 12,
"missing_covers_max": 200
})
}
#[sqlx::test(migrations = "./migrations")]
async fn requires_admin(pool: PgPool) {
let h = harness(pool.clone());
let (_u, cookie) = register_user(&h.app).await;
let resp = h
.app
.oneshot(get_with_cookie("/api/v1/admin/health", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn report_has_status_checks_and_default_thresholds(pool: PgPool) {
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(get_with_cookie("/api/v1/admin/health", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
// Overall status + a non-empty checks array (memory / dead_jobs /
// missing_covers / fail-rates are always present, daemon or not).
assert!(body["status"].is_string());
let checks = body["checks"].as_array().unwrap();
assert!(!checks.is_empty());
let ids: Vec<&str> = checks.iter().map(|c| c["id"].as_str().unwrap()).collect();
assert!(ids.contains(&"memory"));
assert!(ids.contains(&"dead_jobs"));
assert!(ids.contains(&"crawl_fail_rate"));
// The harness wires no crawler daemon, so session/browser checks are absent.
assert!(!ids.contains(&"crawler_session"));
// Default thresholds echoed back.
assert_eq!(body["thresholds"]["dead_jobs_max"], 100);
assert_eq!(body["thresholds"]["disk_pct"], 90.0);
}
#[sqlx::test(migrations = "./migrations")]
async fn dead_jobs_check_warns_when_over_threshold(pool: PgPool) {
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
// Lower the threshold to 0, then a single dead job trips the check.
let mut t = valid_thresholds();
t["dead_jobs_max"] = json!(0);
let resp = h
.app
.clone()
.oneshot(put_json_with_cookie(
"/api/v1/admin/health/thresholds",
t,
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
sqlx::query(
"INSERT INTO crawler_jobs (payload, state) VALUES ($1, 'dead')",
)
.bind(json!({ "kind": "sync_manga", "source_id": "target", "source_manga_key": "k", "url": "http://x", "title": "T" }))
.execute(&pool)
.await
.unwrap();
let resp = h
.app
.clone()
.oneshot(get_with_cookie("/api/v1/admin/health", &cookie))
.await
.unwrap();
let body = body_json(resp).await;
let dead = body["checks"]
.as_array()
.unwrap()
.iter()
.find(|c| c["id"] == "dead_jobs")
.unwrap();
assert_eq!(dead["status"], "warn");
assert_eq!(body["status"], "warn"); // overall rolls up the worst
}
#[sqlx::test(migrations = "./migrations")]
async fn update_thresholds_persists_and_audits(pool: PgPool) {
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
.clone()
.oneshot(put_json_with_cookie(
"/api/v1/admin/health/thresholds",
valid_thresholds(),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = body_json(resp).await;
// Response re-evaluates against the new thresholds.
assert_eq!(body["thresholds"]["dead_jobs_max"], 50);
// Persisted to app_settings.
let stored: serde_json::Value =
sqlx::query_scalar("SELECT value FROM app_settings WHERE key = 'health_thresholds'")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(stored["mem_pct"], 80.0);
// Audit row written.
let action: String =
sqlx::query_scalar("SELECT action FROM admin_audit ORDER BY at DESC LIMIT 1")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(action, "update_health_thresholds");
}
#[sqlx::test(migrations = "./migrations")]
async fn last_metadata_tick_at_reads_crawler_state(pool: PgPool) {
// Absent → None.
assert!(mangalord::repo::crawler::last_metadata_tick_at(&pool)
.await
.unwrap()
.is_none());
// Stored in the daemon's `{"at": rfc3339}` shape → parsed back.
sqlx::query(
"INSERT INTO crawler_state (key, value) VALUES ('last_metadata_tick_at', $1)",
)
.bind(json!({ "at": "2026-06-18T09:30:00Z" }))
.execute(&pool)
.await
.unwrap();
let got = mangalord::repo::crawler::last_metadata_tick_at(&pool)
.await
.unwrap()
.unwrap();
assert_eq!(got.to_rfc3339(), "2026-06-18T09:30:00+00:00");
}
#[sqlx::test(migrations = "./migrations")]
async fn update_thresholds_rejects_out_of_range(pool: PgPool) {
let h = harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let mut bad = valid_thresholds();
bad["disk_pct"] = json!(150.0); // > 100
let resp = h
.app
.oneshot(put_json_with_cookie(
"/api/v1/admin/health/thresholds",
bad,
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
}

Some files were not shown because too many files have changed in this diff Show More