Compare commits

..

134 Commits

Author SHA1 Message Date
MechaCat02
33cc41bacd fix: make the genre dedup migration collision-proof (0038)
All checks were successful
deploy / test-backend (push) Successful in 38m34s
deploy / test-frontend (push) Successful in 10m58s
deploy / build-and-push (push) Successful in 11m46s
deploy / deploy (push) Successful in 24s
Review caught a boot-crash risk: 0038's repoint-UPDATE could set two
non-canonical variant links of one manga to the same canonical id in a single
statement (NOT EXISTS sees the pre-statement snapshot) -> manga_genres PK
violation -> migration rollback -> startup failure. Replace with INSERT ... SELECT
DISTINCT ... ON CONFLICT DO NOTHING (collision-proof) then delete variants. Amended
in place (0038 is unpushed). Tested with the exact collision scenario.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 19:12:52 +02:00
MechaCat02
08a9819d76 test: await async render in AddToCollectionModal collection-name assertion
FE-2's CancellableLoader hop renders the list one microtask after the test's
fixed tick budget; switch getByText -> findByText (async retry). Fixes a
regression merged in ac5641b (FE-2 landed without running this component test).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:21:47 +02:00
MechaCat02
70a6598924 chore: harden docker-compose deployment (bind, memory limits, no-new-privileges)
- frontend publishes on ${FRONTEND_PUBLISH_ADDR:-127.0.0.1} not 0.0.0.0 (M4).
- mem_limit on backend(4g)/postgres(1g)/frontend(512m), .env-overridable (M3).
- security_opt: no-new-privileges on app + postgres services.
New knobs documented + allowlisted in the config meta-test; compose validates.
Deferred (need host verification): image pinning, backend healthcheck, cap_drop
on postgres/tor, Postgres backup sidecar.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:10:29 +02:00
MechaCat02
e6aefaa804 test: pin AppError -> HTTP status mapping (incl 501 and 429 Retry-After)
Table-driven test over every AppError variant's into_response().status(), closing
the unexercised 501 NotImplemented path and asserting a deterministic 429
Retry-After header (audit GAP B). Test-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:03:28 +02:00
MechaCat02
6354a7a8a3 fix: tighten SSRF interceptor residuals
- verdict() now blocks file:/ftp:/gopher: subresources instead of allowing all
  non-http(s) schemes (keeps data:/blob:/internal for page loads).
- bin/crawler.rs uses should_attach_safe_resolver() so an http(s) proxy keeps the
  DNS-rebinding guard, matching the daemon.
- Regression test pins the url crate's decimal/hex/octal/short IPv4 normalization
  to loopback (ensure_public_target rejects them). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 22:00:55 +02:00
MechaCat02
60d5a2efea docs: clarify that MAX_PAGES_PER_CHAPTER=0 relies on the body-size backstop
Make explicit (config.rs + .env.example) that disabling the per-chapter page cap
leaves MAX_REQUEST_BYTES as the sole bound (audit footgun). No behavior change;
the thumbnail-source read is already bounded by MAX_FILE_BYTES + decode limits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:56:19 +02:00
MechaCat02
90f2398e56 fix: send HSTS on HTTPS document responses
Add Strict-Transport-Security (max-age=63072000; includeSubDomains) in
applySecurityHeaders, gated on x-forwarded-proto === 'https' so a plain-HTTP
dev/misconfig deploy never pins the browser to HTTPS (audit LOW). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:52:54 +02:00
MechaCat02
e960aae163 fix: enable browser SSRF interception by default (H1)
Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:51:05 +02:00
MechaCat02
adce950049 fix: exponential idle backoff for the analysis worker
The analysis worker's flat 1s idle sleep meant an empty queue fired a lease query
per worker per second forever (audit). Add idle_backoff (1s..30s, reset on lease),
mirroring the crawler. Unit-tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:49:01 +02:00
MechaCat02
bde72e47f4 fix: batch and index the crawler_jobs retention reaper
reap_terminal was one unbatched DELETE with no updated_at index — a full scan and
long-held lock over a large backlog (audit M5). Migration 0040 adds a partial
index on (updated_at) WHERE state IN ('done','dead'); reap_terminal now deletes in
bounded 5k batches. Existing tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:46:01 +02:00
MechaCat02
cb757e7b69 fix: index the running/leased_until arm of the job-lease query
The lease predicate's `state='running' AND leased_until < now()` arm had no
index, so every lease poll seq-scanned the whole crawler_jobs table (audit H2).
migration 0039 adds a partial index on (leased_until) WHERE state='running'.
Behavior-neutral; 29 lease tests green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:43:33 +02:00
MechaCat02
042e7e9047 fix: fall back to the original when a thumbnail fails to decode
A blob with valid magic but a corrupt body passed upload then 500'd on a ?w=
thumbnail decode (audit). Fall back to serve_original (streams without decoding)
instead. Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:41:28 +02:00
MechaCat02
cc8ae2566f fix: reject a malformed reenqueue body instead of running the full scope
Option<Json<T>> collapses a malformed body to None, so a bad reenqueue body
silently ran the default full "All" scope (audit). Parse raw bytes: empty =
default All, malformed = 422. Tested both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:37:16 +02:00
MechaCat02
4f085f228a fix: commit cover DB pointer before mutating the old blob
put_cover/delete_cover mutated storage before the DB pointer, so a transient DB
failure could leave the row pointing at a deleted blob. Reorder to
put->set-DB->delete-old and clear-DB->delete-blob, so a failure only orphans a
blob, never dangles the pointer. Happy path covered by the 16 existing cover tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:33:57 +02:00
MechaCat02
ac5641bcd3 fix: guard AddToCollectionModal against out-of-order load results
Retargeting the modal mid-load could resolve responses out of order, overwriting
the newer target's membership with the older one's (audit FE-2). Route load()
through the tested CancellableLoader so a superseded result is dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:31:26 +02:00
MechaCat02
cd5358dc9b fix: stop reader key navigation leaking behind open overlays
Arrows/j/k/Home/End bubbled to the reader's window handler while a
jump/settings/action/collections/tags/context overlay was open, paging the
chapter behind it (FE-1); native select/contenteditable weren't exempt (FE-4).
New unit-tested isTypingTarget() + an overlay-open early return fix both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:28:57 +02:00
MechaCat02
c31830468c fix: enforce case-insensitive genre uniqueness
genres had only case-sensitive name UNIQUE while authors/tags use
UNIQUE (lower(name)); sync_genres could race two case variants into duplicate
rows (audit DB-3). Migration 0038 pre-dedups then adds
UNIQUE (lower(name)); sync_genres upserts ON CONFLICT (lower(name)). Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:25:46 +02:00
MechaCat02
75f4481bbd fix: invalidate stale page analysis when a re-crawl overwrites a page
Deterministic per-page storage keys mean a re-crawl overwrites the image but the
pages row survives the upsert, so analysis keyed to page_id (page_analysis/
search_doc, OCR, auto-tags, content-warnings) stayed stale forever and
force=false re-enqueue never re-analyzed it — poisoning search + the derived
manga_content_warnings (audit DB-1). persist_pages now detects conflict-updated
pages via xmax and clears their four derived tables in the same tx (the
content-warnings DELETE fires the mcw_* triggers), returning them to unanalyzed
so they re-derive. Tested.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:21:29 +02:00
MechaCat02
958d5bd713 fix: scheme-validate crawler source_url before rendering as a link
Harvested source_url (external content) rendered as an admin <a href> without
scheme validation — a javascript:/data: link would execute on click (audit M2).
safeExternalHref() gates to http(s); CrawlerHistoryTable/DeadJobsTable fall back
to plain text otherwise. Tested in safeHref.test.ts.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:16:08 +02:00
MechaCat02
f2c9cfe162 test: cover the trusted_proxy X-Forwarded-For rate-limit gate
Two integration tests pinning both sides of the gate: trusted_proxy=true gives
each forwarded IP its own bucket; trusted_proxy=false ignores spoofed XFF and
shares the global bucket. Closes audit gap M7. Test-only, no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:13:37 +02:00
MechaCat02
b5f7467c47 refactor: remove unused constant_time_eq and correct the token doc
All checks were successful
deploy / test-backend (push) Successful in 35m33s
deploy / test-frontend (push) Successful in 10m46s
deploy / build-and-push (push) Successful in 11m31s
deploy / deploy (push) Successful in 13s
constant_time_eq (and the `subtle` dependency) was never wired into any
comparison — the module doc wrongly claimed token comparison used it, but lookup
is an indexed DB equality on the 256-bit SHA-256 hash, so there's no in-process
secret compare to time-attack. Delete the dead fn, its test, and the unused
`subtle` dep; rewrite the doc. No behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:22:39 +02:00
MechaCat02
9148e23da8 fix: reap expired sessions with a periodic background sweep
The sessions table only grew — find_active ignored expired rows but nothing
deleted them, so lapsed sessions accumulated indefinitely.

- repo::session::delete_expired: indexed DELETE (sessions_expires_idx) returning
  the reaped count.
- app::build spawns a detached hourly reaper (SESSION_GC_INTERVAL) that calls it,
  independent of crawler/analysis config.

Test: delete_expired_removes_only_lapsed_sessions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:19:41 +02:00
MechaCat02
f39307232c fix: force download for untyped (octet-stream) served blobs
The /files handler served the application/octet-stream fallback with no
Content-Disposition, letting a browser render it inline — a stored-XSS vector if
the blob is crafted HTML/JS. Add Content-Disposition: attachment for that type
(belt to the existing nosniff); known image types stay inline.

Test (new tests/api_files.rs): octet-stream → attachment; png → inline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:15:23 +02:00
MechaCat02
1b7b8a3038 fix: cap the multipart metadata part instead of buffering it unbounded
The `metadata` JSON part in the manga-create and chapter-upload handlers was
read with `Field::bytes()`, buffering the whole part before any check — a client
could send a multi-hundred-MB metadata blob (up to the 200 MiB request limit) as
one allocation. Image parts were already streamed with a cap; metadata was the
last unbounded read.

Cap it at a shared upload::MAX_METADATA_BYTES (64 KiB) via the read_capped
streaming loop (413 on overflow), and remove the now-unused read_field_bytes
helper so the unbounded path can't return.

Tests: manga + chapter oversized-metadata parts are rejected with 413.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 20:10:05 +02:00
MechaCat02
253d46c7e5 fix: return NotFound when a storage key resolves to a directory
`get` (fs::read → EISDIR) and `get_stream` (File::open succeeds on a Unix
directory, then streams garbage that fails mid-read) both mishandled a key
resolving to a directory. Mirror `size`'s existing guard: `get` maps
IsADirectory to NotFound; `get_stream` checks metadata.is_file() after open.

Test: get_on_directory_is_not_found.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:59:51 +02:00
MechaCat02
c570e0cc37 fix: escape LIKE wildcards in user-search ILIKE queries
A `%` or `_` in a search term silently acted as a LIKE wildcard rather than
matching literally (`50%` matched everything, `a_b` matched `axb`). Not
injection — terms are bound — but a search-correctness bug across every
user-facing ILIKE site.

- repo::escape_like: shared pub(crate) escaper (promoted from page_tag), unit-tested.
- Trigram-entangled sites (manga, tag, author) append a separate escaped param
  used only by the ILIKE branch; the trigram/similarity branches keep the raw term.
- Pattern-built sites (admin manga list, admin users, analysis coverage + history,
  crawler search incl. JSONB payload title) escape inside format!("%{}%", ..) and
  pair each ILIKE with ESCAPE '\'.

Tests: escape_like unit tests + integration tests on public manga search, author
autocomplete, and admin user search proving `_` matches literally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:57:06 +02:00
MechaCat02
5784483a57 fix: add CSP and defense-in-depth security response headers
Document responses carried no CSP, X-Frame-Options, Referrer-Policy, or
Permissions-Policy — leaving clickjacking and zero script-injection defense
in depth (security audit T4).

- svelte.config.js: enable kit.csp hash mode. SvelteKit hashes its own inline
  hydration scripts; the inline theme <script> in app.html isn't part of
  %sveltekit.head%, so its sha256 is pinned by hand (csp-config.js) and added
  to script-src alongside object-src 'none', base-uri 'self', frame-ancestors
  'none'. Styles stay unconstrained (Svelte emits dynamic inline style= attrs).
- hooks.server.ts: applySecurityHeaders() sets X-Frame-Options: DENY,
  Referrer-Policy: strict-origin-when-cross-origin, X-Content-Type-Options:
  nosniff, and a Permissions-Policy locking down unused features, only when the
  response hasn't already set them.
- src/csp-theme-hash.test.ts: drift guard against editing the theme script
  without updating THEME_SCRIPT_HASH.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 19:27:27 +02:00
MechaCat02
a44511983d fix: page through all chapters when the API omits total
All checks were successful
deploy / test-backend (push) Successful in 44m20s
deploy / test-frontend (push) Successful in 11m3s
deploy / build-and-push (push) Successful in 13m53s
deploy / deploy (push) Successful in 20s
listAllChapters used `page.total ?? all.length` as the loop bound. The
/chapters endpoint always returns `total: null`, so a full 200-row first
page broke the loop immediately — the reader dead-ended past chapter 200.
Stop on a short page instead; only trust `total` when actually reported.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 16:24:49 +02:00
MechaCat02
f8e53809d5 chore: sync backend version to 0.128.3 (lockstep with frontend)
Several frontend-only commits bumped both manifests but staged only the
frontend, leaving backend/Cargo.toml behind. Bring it back in lockstep.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:26:12 +02:00
MechaCat02
0afd202164 fix: let AbortError propagate through the API client
The network-error wrapping added in the error-boundary work also caught
deliberate AbortController cancellations and turned them into a network_error
ApiError, breaking callers that distinguish "cancelled" from "failed" (the
cancellable admin fetches). Rethrow DOMException AbortError unchanged; only
genuine network failures become ApiError(status 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:25:01 +02:00
MechaCat02
ba3e5b481b fix: lock background scroll while a Modal or Sheet is open
Overlays didn't lock body scroll, so the page behind a modal/sheet scrolled
under it. Add a ref-counted scroll lock (so stacked overlays don't prematurely
release it) and hold it via an effect in Modal and Sheet while open, restoring
on close or unmount. Focus-trapping was already handled.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:23:07 +02:00
MechaCat02
3b783c1d9b fix: upload retry reuses the created manga instead of duplicating it
After a partial upload failure (manga created, a chapter failed), re-submitting
re-ran createManga and spawned a duplicate manga. Track the created manga id:
a retry skips creation, re-sends only the not-yet-done chapters, disables the
now-saved title, relabels the button "Retry failed chapters", and shows a note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:20:03 +02:00
MechaCat02
a47b6895c2 feat: warn before leaving a form with unsaved changes
Drag-dropping page images then clicking a nav link silently discarded the
whole form. Add a reusable guardUnsavedChanges helper (SvelteKit beforeNavigate
confirm + native beforeunload) and wire it into the manga upload, manga edit,
and chapter upload forms. The dirty check returns false while submitting so the
post-save redirect isn't prompted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:16:40 +02:00
MechaCat02
3ca05dcb58 fix: paginate the bookmarks and collections lists with Load more
Both lists fetched a single capped page (100 / 200) with no pager, so entries
past the cap were silently unreachable. Add a reusable LoadMore component that
seeds from the streamed first page (preserving the auth gate + inline error
handling) and fetches further offset-based pages on demand, and wire it into
the bookmarks and collections pages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:12:03 +02:00
MechaCat02
c308eb3eac fix: reader loads the full chapter list for prev/next and the dropdown
The reader fetched only the first 200 chapters, so a deep chapter (#250) fell
outside the window and its prev/next chevrons resolved to null — a dead end —
and the chapter dropdown was truncated. Add listAllChapters, which pages
through the API's 200-row cap, and use it in the reader loader.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 15:02:44 +02:00
MechaCat02
69a9309c54 feat: honor OCR text and tag scope in search Chapters/Mangas views
The Chapters/Mangas aggregation views accept an OCR ?text= filter and the
page search intersects tags, but the client dropped both: any text query
collapsed every view into a flat page search that discarded the selected tag
and grouping. Add text to AggregateOptions, thread it into
listTaggedChapters/listTaggedMangas, and rework the search loader so a
tag-scoped view runs the aggregation (optionally text-filtered) and the page
search passes the selected tag.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:58:48 +02:00
MechaCat02
1ed1a134ea feat: bot API token management page
The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:55:24 +02:00
MechaCat02
32d0a7e13b feat: root error boundary and network-error normalisation
A load() that rethrew (e.g. the profile overview) fell through to SvelteKit's
bare default error page, and a connection-refused fetch rejected as a raw
TypeError that blanked the view. Wrap fetch network failures in the client as
ApiError(status 0, network_error) so callers handle them uniformly, and add a
root +error.svelte with a friendly message plus Try again / Go home actions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:47:41 +02:00
MechaCat02
d779fa2b97 fix: confirm admin role toggle and keep the checkbox controlled
The admin checkbox fired an immediate, irreversible privilege change with no
confirmation, and on cancel/failure the optimistic DOM flip stuck while the
server state was unchanged. Add a confirm() gate (like Delete) and make the
checkbox controlled by u.is_admin: onchange reverts the DOM immediately and it
only flips once the server confirms — so a cancelled or failed toggle never
misrepresents the real role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:44:47 +02:00
MechaCat02
755417730f feat: serve width-bounded thumbnail variants for image grids
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:41:19 +02:00
MechaCat02
5b1ce581f3 perf: precompute mangas.sort_author for the author sort
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:29:26 +02:00
MechaCat02
c11df3182c perf: denormalize content-warning filter to manga_content_warnings
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:24:31 +02:00
MechaCat02
d749b56d58 docs: correct re-upload page-preservation semantics
CLAUDE.md claimed re-uploading a chapter unconditionally drops saved-page
references. In fact the crawler re-fetch (content::persist_pages) upserts by
(chapter_id, page_number) with ON CONFLICT DO UPDATE, preserving pages.id, so
collections/page-tags survive a re-fetch. Saves drop only when a pages row is
actually deleted (chapter deletion, or a user delete-then-recreate). Migration
0023's text is left untouched (sqlx checksums applied migrations).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:19:03 +02:00
MechaCat02
c24e296f07 fix: interleave uploaded chapters by number in the chapter list
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:17:18 +02:00
MechaCat02
9508fb8e86 fix: make bookmark add idempotent instead of 409
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:08:27 +02:00
MechaCat02
ca55712622 refactor: remove dead partial-render guard in resync_manga
The first guard block computed the partial-render condition but had an empty
if-body, so it never skipped anything — the real skip is the chapter-sync
guard further down. Delete the dead block and move its observability warning
into the live guard's else branch, eliminating the data-loss trap without
changing behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:04:44 +02:00
MechaCat02
4fe435cc76 fix: recover the auth rate limiter from a poisoned mutex
try_acquire used .expect on lock(), so a single panic while holding either
limiter mutex would poison it and make every subsequent auth request re-panic
— one blip became a persistent auth outage. Recover the guard with
unwrap_or_else(|e| e.into_inner()) so the limiter keeps serving.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:02:54 +02:00
MechaCat02
f5cb460aec fix: abandon dispatch when heartbeat loses the lease
Heartbeat renew failures were only logged, so under DB flakiness a job could
be leased twice concurrently — the original worker kept crawling past a lapsed
lease while another re-leased it. Track consecutive renew failures and, after
MAX_HEARTBEAT_RENEW_FAILURES, signal the worker (via a CancellationToken) to
abandon the in-flight dispatch instead of working a job it no longer owns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 14:01:04 +02:00
MechaCat02
f83d49b83e fix: exponential backoff for idle crawler workers
An idle worker re-issued a row-locking lease query every 1s per worker with
no backoff. Replace the flat sleep with idle_backoff: 1s, 2s, 4s, … capped at
30s, reset to 0 the moment a job is leased, so a quiet daemon stops hammering
the crawler_jobs table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:58:39 +02:00
MechaCat02
cef41ce76a fix: don't burn a job attempt when the browser is unavailable
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.

Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:55:50 +02:00
MechaCat02
7570524e5b fix: stream cover uploads with a per-chunk size cap
The cover multipart path called Field::bytes(), buffering the entire field
into memory before parse_image checked max_file_bytes — an oversized cover
was fully allocated before rejection. Add read_capped/push_capped, which
reject with 413 as soon as the running total exceeds the cap (the offending
chunk is never copied in), and route both cover handlers and stage_image_part
through it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:51:10 +02:00
MechaCat02
5b46eab73a fix: keep SSRF DNS resolver on http(s)-proxy crawler paths
Commit 134ab54 dropped the safe DNS resolver for any configured proxy, but
that guard is only pointless for SOCKS proxies (where the proxy resolves the
target). On an http(s):// proxy reqwest still resolves the target locally, so
a hostname resolving to a private IP would be reachable. Narrow the exemption
to SOCKS-only via should_attach_safe_resolver.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:46:39 +02:00
MechaCat02
91d6a50dba fix: guard all browser subresources against SSRF and fail closed
The CDP Fetch interceptor only paused main-frame Document requests, so a
scraped page's <img>/fetch()/XHR subresources to internal targets
(169.254.169.254, postgres:5432, RFC1918) reached those hosts unguarded.
Drop the ResourceType::Document constraint so every request runs through
is_blocked. Also make open_page fail-closed: if the guard can't install,
close the blank page and return the error instead of navigating unguarded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 13:44:37 +02:00
134ab54b34 fix: don't apply the SSRF DNS resolver to proxied/internal clients
All checks were successful
deploy / test-backend (push) Successful in 39m39s
deploy / test-frontend (push) Successful in 10m52s
deploy / build-and-push (push) Successful in 12m5s
deploy / deploy (push) Successful in 13s
ff4ca96 wired SafeResolver (drops any host resolving to a private IP) into
all four crawler/analysis reqwest clients. That broke every proxied fetch:
with a socks5h:// proxy the target is resolved by Tor, so reqwest only ever
resolves the proxy's OWN host — `tor`, on a private Docker IP (172.x) — which
the resolver then rejects ("SOCKS error: failed to create underlying
connection"). 100% of crawler image downloads failed. The vision clients
broke the same way against `mangalord-vision` (latent; analysis was off).

Fix: attach the resolver only on the crawler's direct (unproxied) path, where
it genuinely guards DNS rebinding. Behind a proxy it adds no protection (Tor
can't reach internal ranges and the target is invisible to reqwest) so it's
omitted. Drop it from the vision clients entirely — that endpoint is an
operator-configured internal service that must resolve to a private IP.

Security posture is >= the pre-ff4ca96 baseline (which had no resolver at all).
is_private_ip / retain_public_addrs and their tests are unchanged.

Bump 0.124.13.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-10 22:20:25 +02:00
MechaCat02
f879ce1866 fix: per-IP auth rate limiting instead of one global bucket
All checks were successful
deploy / test-frontend (push) Successful in 10m44s
deploy / build-and-push (push) Successful in 11m35s
deploy / deploy (push) Successful in 12s
deploy / test-backend (push) Successful in 31m21s
A single global token bucket let one attacker at the sustained rate 429
every user's login/register/change-password. Key buckets by client IP: the
SvelteKit proxy now stamps the real peer address onto X-Forwarded-For
(overriding any client-supplied value, anti-spoof), and axum reads it via a
ClientIp extractor — but only when AUTH_TRUSTED_PROXY is set, else it falls
back to the shared bucket (today's behavior). The per-IP map is bounded
(10k IPs, idle buckets pruned) so a spoofed-IP spray can't grow it.

AUTH_TRUSTED_PROXY defaults false (safe for a directly-exposed backend);
compose sets it true since the proxy is the single trusted hop.

Bump to 0.124.12.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 07:07:06 +02:00
MechaCat02
bf425cf8e6 fix: opt-in CDP re-validation of headless-browser navigations (SSRF)
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.

Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.

Bump to 0.124.11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 06:29:17 +02:00
MechaCat02
ff4ca964f5 fix: reject private IPs after DNS resolution (SSRF/DNS-rebinding)
The SSRF guard only checked the host string, so an attacker-owned domain
resolving to 169.254.169.254 / 10.x (DNS rebinding, TOCTOU) bypassed it.
Add a reqwest dns::Resolve (SafeResolver) that drops resolved addresses in
private ranges, wired into all four crawler/analysis clients — it fires per
connection so it also covers redirect hops. Also close the is_private_ip
IPv6-embedding gaps (IPv4-compatible ::/96, NAT64 64:ff9b::/96, 6to4
2002::/16 all now unwrap to the embedded IPv4).

Bump to 0.124.10.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:52:30 +02:00
MechaCat02
61669aac3f fix: don't mark private-mode blobs as publicly cacheable
The immutable Cache-Control added for the reader preload work was
`public` unconditionally. Under PRIVATE_MODE, /files is auth-gated, so a
shared cache/CDN would store the blob and serve it to unauthenticated
clients, defeating the gate. Emit `private, ...` when private_mode is on,
`public, ...` otherwise.

Found in the security audit; regression from 0.124.3. Bump to 0.124.9.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:39:20 +02:00
MechaCat02
795eb76f9a fix: document and wire the DB pool env vars into compose
The new DB_MAX_CONNECTIONS / DB_ACQUIRE_TIMEOUT_SECS knobs were read by the
backend but never added to .env.example or docker-compose.yml, so an operator
setting them in .env would see no effect (caught by the config meta-tests).
Document both in .env.example and wire them into backend.environment.

Bump to 0.124.8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:18:49 +02:00
MechaCat02
ebf0b8289b fix: compute the /mangas total only on the first page
GET /mangas recomputed count(*) over FILTER_WHERE (up to five correlated
subqueries) on every page, so pagination cost scaled with catalog size. The
total doesn't change as a caller walks pages, so compute it once at offset 0
and return null thereafter (the envelope already serialises total as
Option<i64>; the frontend types it number|null and the library route ignores
it).

Bump to 0.124.7.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:14:15 +02:00
MechaCat02
ce9a727c73 fix: enforce the analysis image cap while reading, not after
The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.

Bump to 0.124.6.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:10:29 +02:00
MechaCat02
46134c8760 fix: close Chromium tabs on crawler fetch error paths
new_page() opened a tab that was only closed on the happy path; a `?` on
wait_for_nav / content() leaked it (chromiumoxide doesn't close on drop),
so errored fetches accumulated tabs in the shared browser. Add a generic,
unit-tested `close_after(close, body)` helper and wrap the three fetch
sites (chapter content, session probe, source navigate) so the tab is
closed on every exit path — including navigate's content()-read branch,
which previously leaked.

Bump to 0.124.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 21:04:50 +02:00
MechaCat02
b7d8faadf7 fix: size the DB pool and fail fast on saturation
The single Postgres pool (shared by HTTP handlers and the crawler/analysis
daemons) hardcoded max_connections=10 with no acquire_timeout, so under
saturation callers hung on the driver's silent 30s default. Add a DbConfig
(DB_MAX_CONNECTIONS default 20, DB_ACQUIRE_TIMEOUT_SECS default 10, clamped
to >=1) and apply both in PgPoolOptions.

Bump to 0.124.4.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:56:14 +02:00
MechaCat02
b0500e8e48 fix: cache immutable blobs served from /files
Page and cover blobs are content-addressed by unguessable, immutable keys,
but /files sent no Cache-Control, so the browser refetched on every reader
open — defeating the whole-chapter and next-chapter preloading (and re-
proxying every byte through the node server in prod). Send
`Cache-Control: public, max-age=31536000, immutable`.

Bump to 0.124.3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:30:03 +02:00
MechaCat02
379224bee4 fix: lazy-load continuous-reader pages beyond an eager window
Continuous mode rendered every page (up to 2000) with loading="eager",
fetching and decoding the whole chapter up front — enough to OOM a mobile
tab on a long chapter. Eager-load only a leading window ([0..=eagerThrough],
covering the deep-link target + a lead, min 6 pages) and lazy-load the rest;
the browser fetches them as the reader scrolls near. Unloaded lazy pages
reserve height (page width x ~1.4) so they stay distinct scroll targets for
the progress IntersectionObserver, cleared on load.

Keeps the DOM dense so the scroll-to-?page=N effect, progress observer, and
next-chapter preload trigger are all unchanged. Rewrote reader-preload-window
spec to assert the eager/lazy attribute partition + deep-link scroll landing.

Bump to 0.124.2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:28:41 +02:00
MechaCat02
1c955458d6 fix: run Argon2 password hashing on the blocking pool
Argon2id hashing (~15-50ms, 19 MiB) ran synchronously inside async
handlers, stalling every task sharing those runtime worker threads under
concurrent auth load. Add spawn_blocking wrappers hash_password_async /
verify_password_async and switch all async call sites; sync primitives
stay for the login timing-equaliser and unit tests. Bump to 0.124.1.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 20:19:14 +02:00
MechaCat02
987d1ba235 feat: preload the whole current chapter in continuous mode
Eager-load every page of the current chapter rather than a bounded
rolling window, so pages are fetched up front (the browser paces them
over its per-origin connection limit, top-to-bottom = reading order)
instead of filling in as they're scrolled onto. This drops the
scrollLeadIdx/eagerThrough machinery in favour of a plain
`loading="eager"`.

The first few pages of the *next* chapter are already warmed when the
reader nears the end of the current one (the next-chapter preloader), so
flipping over stays instant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:47:48 +02:00
MechaCat02
c69ff502f7 feat: preload the current chapter ahead of the reader
Continuous mode eager-loaded only pages 0..=initialIndex; everything
below stayed `loading="lazy"`, so pages fetched and reflowed in as they
scrolled into view. Now a rolling window eager-loads a few pages ahead of
the reading position (PRELOAD_AHEAD) so they arrive — fetched and sized —
while still below the fold, then swap in without pop-in.

The window rolls forward off `scrollLeadIdx`, the top-of-viewport page
measured from live rects and guarded by `scrollY`. It deliberately does
not use the read-progress high-water mark: before images load they are
0-height and pile into the viewport, which would latch that mark to the
last page and eager-load the whole chapter on open. Pages past the window
stay lazy so opening a long chapter doesn't fetch every page at once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:39:50 +02:00
MechaCat02
51ea254dde fix: surface streamed load failures inline
All checks were successful
deploy / test-backend (push) Successful in 30m44s
deploy / test-frontend (push) Successful in 10m46s
deploy / build-and-push (push) Successful in 10m59s
deploy / deploy (push) Successful in 13s
The streamed loaders on bookmarks, collections, library, search, and the
collection detail page swallowed only ApiError into their in-band `error`
field and re-threw anything else. A raw network failure (a non-ApiError
TypeError from fetch) therefore escaped the `{#await}` to SvelteKit's
generic error page. On the collection detail page a content-load failure
showed a misleading "this collection is empty", and on search a failed
results query showed a false "no matches" because the streamed `error`
was never rendered.

Fold non-ApiError failures into the same in-band `error` so every
transient blip renders as an inline message, add a `contentError` branch
to the collection detail grid, and render the search results `error` in
both result views.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:11:37 +02:00
MechaCat02
fee68dd9ac feat: loading skeleton on the manga detail page
The detail loader now streams its six-call bundle instead of blocking
navigation on it. The page component is a thin {#await} wrapper: it shows a
MangaDetailSkeleton (cover + meta + chapter rows) while the bundle loads,
renders the extracted DetailView once resolved, and shows an inline
not-found on a 404 (previously the framework error page). Extracting
DetailView also makes it remount per navigation, so its tag/bookmark/reaction
state re-seeds cleanly on manga-to-manga moves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:30:49 +02:00
MechaCat02
3c8e264f48 feat: loading skeleton on the library page
The library loader streams its one-shot bundle (also the auth gate), so the
active sub-tab shows a skeleton while it loads — a grid skeleton for the
Collections tab, a row skeleton for Bookmarks/History/Page-tags — instead of
only the global nav bar.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:21:20 +02:00
MechaCat02
16cdec051a feat: loading skeletons on the bookmarks and collections pages
Both loaders now stream the whole result (the single fetch is also the auth
gate) so the page shows a skeleton while it loads instead of the global nav
bar only: a ListRowSkeleton on bookmarks and a MangaGridSkeleton on the
collections grid, resolving to the sign-in / error / empty / list branches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:18:43 +02:00
MechaCat02
5b51bcf056 feat: results skeleton on the search page
The search loader now awaits only the chip cloud (its auth gate) and streams
the view-specific results, so switching tag / view / sort / text shows a
SearchResultsSkeleton in place of the frozen previous list until the query
resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 07:14:43 +02:00
MechaCat02
f1e66141f4 fix: add tags optimistically on the detail page
submitTag now shows the chip immediately with the typed name and reconciles
it with the server's (normalized) ref on success, rolling back on error —
matching the already-optimistic tag removal instead of blocking on the
round-trip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 22:05:23 +02:00
MechaCat02
cee1e73f98 feat: grid skeletons on the author and collection detail pages
Both loaders now stream their manga grid instead of blocking on it: the cheap
header renders immediately (and still handles 404/401 in the loader) while a
MangaGridSkeleton stands in until the grid resolves, then swaps in without a
layout shift. Collection detail seeds its optimistic-removal state from the
stream via a guarded effect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 22:01:53 +02:00
MechaCat02
ec73c6e001 feat: reserve home shelf space while it loads (fixes CLS)
The Continue-reading / Recommended shelves were fetched after the catalogue
and rendered only once populated, so they popped in late and shoved the grid
down on every authenticated home visit. Now the shelf fetch fires
concurrently with the catalogue and a reserved ShelfSkeleton holds the space
for logged-in users until it resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:53:03 +02:00
MechaCat02
bd6ae86a85 feat: preload the reader from the detail CTA
The Continue / Read-first-chapter CTA is the highest-intent link to the
heaviest route in the app. An effect now programmatically preloadData()s the
target chapter as soon as the detail resolves, so the reader's load and first
page images are warm before the tap. Complements the reader's own
next-chapter warming.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:47:53 +02:00
MechaCat02
3622dcc02f fix: reserve detail cover box and async-decode list/shelf covers
- The manga detail desktop overview cover had no reserved aspect-ratio, so
  the meta column reflowed when it decoded. Give .cover aspect-ratio: 2/3 +
  object-fit: cover (mirrors MangaCard).
- Add decoding="async" to the detail hero/overview covers and the list/shelf
  covers (bookmarks, history, continue-reading, collections, tagged rows) so
  decode stays off the main thread — previously only MangaCard had it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:43:21 +02:00
MechaCat02
9cb2f152d3 fix: bookmark toggle is optimistic and no longer fails silently
The detail-page bookmark toggle awaited the round-trip before updating and
had no catch — a failed create/delete threw unhandled and the user saw
nothing. It now flips optimistically (placeholder reconciled with the server
row on success), rolls back on error, and surfaces a toast, matching the
sibling like/dislike buttons.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:40:33 +02:00
MechaCat02
7c3c9cf699 feat: app-wide toast notifications
Adds a client-side toast store (info/success/error, auto-dismiss with a
sticky option) and a Toaster component mounted in the root layout, wired to
the reserved --z-toast token. Gives mutations a consistent way to surface
success/failure instead of inline-only text or silent catches.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:36:34 +02:00
MechaCat02
0d9505ce9f fix: address skeleton/nav-progress review findings
Correctness:
- MangaGridSkeleton: covers rendered 0px tall because height="0" is a
  definite height that defeats aspect-ratio. Skeleton gains an `aspectRatio`
  prop (height stays auto) so the 2/3 cover box is reserved and the grid no
  longer shifts when real covers load.
- Move `.manga-card` layout into global tokens.css (it was scoped to
  MangaCard, so the skeleton's cards inherited none of it). Both consumers
  now share one definition.
- NavProgress: rebuilt as an idle/loading/done state machine. It now
  completes to full and fades on settle (was snapping away instantly),
  restarts the trickle when one navigation supersedes another (was parking
  frozen near 90%), animates transform/opacity only with will-change gated to
  the active phase, and hides itself in reader fullscreen.
- Skeleton: text-variant height resolved in one place (no inline style
  shadowing a stylesheet rule); shimmer gradient/animation gated behind
  prefers-reduced-motion: no-preference so reduced motion shows a flat block.

A11y:
- Home loading region announces via visually-hidden text instead of an
  aria-label a live region won't reliably read.

Cleanup:
- Drop the fragile :first-child selector and index-keying ceremony in
  MangaGridSkeleton; simplify Skeleton's style construction.

Tests: adds regression guards for the 0px cover collapse, the aspect-ratio
box, text-height defaulting, and the nav-progress complete/restart phases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 20:02:27 +02:00
MechaCat02
7bdbe3ce5b fix: define missing --font-md and --success-soft-bg tokens
Both were referenced by components (RecommendationShelf, the manga detail
page) but never declared, so var() fell through to inherited/fallback
values. Adds --font-md (1rem, on the xs/sm/md/lg scale) and
--success-soft-bg in both the light and dark theme blocks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:44:43 +02:00
MechaCat02
3364ea52c9 feat: async decode for manga cover images
Adds decoding="async" to the MangaCard cover so image decode happens off the
main thread, keeping grid scroll/paint smooth as covers stream in. The 2/3
aspect-ratio already reserves space, so no layout shift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:41:28 +02:00
MechaCat02
def97d3087 feat: skeleton loading state on the home catalog
Replaces the plain "Loading…" text with a content-shaped MangaGridSkeleton
so the catalog grid's layout is reserved while it loads and swaps in without
a shift. Keeps the data-testid="loading" wrapper (now role="status") for
accessibility and selector stability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:40:28 +02:00
MechaCat02
09f12c8959 feat: global navigation progress bar
Most data routes run their load with ssr = false, so SvelteKit holds the
previous page on screen while the next one's data resolves — with no
feedback. NavProgress, mounted once in the root layout and driven by the
$navigating store, shows a thin top bar that trickles while a navigation is
pending and fades out on completion. Degrades to a static visible bar under
prefers-reduced-motion. New --z-nav-progress token sits above the fixed
header but below modals/toasts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:38:14 +02:00
MechaCat02
34c1122e4e feat: add MangaGridSkeleton loading placeholder
Composes Skeleton into a cover-grid placeholder that reuses the shared
.manga-grid / .manga-card layout, so it matches the real catalog grid at
every breakpoint and content swaps in without a layout shift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:31:58 +02:00
MechaCat02
200ab1c0a0 feat: add Skeleton loading-placeholder primitive
A reusable, token-driven shimmer block with configurable width/height,
radius, and variant. Decorative (aria-hidden) — loading state is announced
by the container. Freezes to a solid muted block under the global
prefers-reduced-motion override.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 19:28:00 +02:00
MechaCat02
ef8d226ba6 fix(detail): resync reaction toggle across manga navigation
All checks were successful
deploy / test-backend (push) Successful in 36m1s
deploy / test-frontend (push) Successful in 10m41s
deploy / build-and-push (push) Successful in 11m41s
deploy / deploy (push) Successful in 13s
The detail page component is reused across /manga/A -> /manga/B
navigations (clicking a similar or recommendation card), so
ReactionButtons' locally-held `current` state kept the previous manga's
reaction until interacted with — a visibly wrong thumb on the new page.
Re-seed `current` from the loader-supplied prop via an effect keyed on
mangaId. Also fetch the homepage's two independent /me/* shelves
concurrently (Promise.allSettled) instead of in series.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-05 18:17:18 +02:00
MechaCat02
19db66a845 feat(home): add a "Recommended for you" shelf
Surface the content-based /me/recommendations feed as a horizontal shelf on
the homepage, below the Continue-reading shelf. Fetched after the catalogue
(same non-blocking pattern), shown only when signed in and the feed is
non-empty; reuses MangaCard in a fixed-width scroller. Component unit test +
e2e (signed-in shows cards, anonymous hides).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:46:33 +02:00
MechaCat02
d6a109df2d feat(recommendations): content-based "Recommended for you" endpoint
Add GET /me/recommendations: rank mangas by weighted tag overlap with the
user's taste — explicit like +1.0, bookmark +0.5, dislike −1.0 (reaction
overrides bookmark). Per-tag affinities are summed, candidates scored by
their tags' affinity normalized by tag count (anti-tag-stuffing, à la
list_similar), with already-reacted/bookmarked/read mangas excluded and
net-negative candidates dropped (dislike down-ranks, not browse-hides).
Reuses manga_cols/cards_from_rows. Integration tests cover like-driven recs,
dislike down-rank, bookmark half-weight ordering, seen-exclusion, empty
cold-start, and auth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:42:05 +02:00
MechaCat02
bb833c7e71 feat(detail): like/dislike buttons on the manga page
Add a tri-state like/dislike toggle beside the bookmark action (signed-in
only): click a reaction to set it, the other to switch, or the active one to
clear — optimistic with rollback, mirroring the bookmark toggle. Seeds from
the new GET /me/reactions/:id in the detail loader (non-critical: any failure
degrades to an unset toggle). Unit tests cover the state machine; e2e drives
set → switch → clear against the real endpoints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 17:20:24 +02:00
MechaCat02
258a536254 feat(reactions): per-user like/dislike on mangas
Add a private per-user reaction resource: a manga_reactions table (one row
per user+manga, like/dislike, cascades) and endpoints PUT/DELETE
/mangas/:id/reaction plus GET /me/reactions/:manga_id, wired on the existing
per-user seam (mirrors bookmarks/read_progress). Unknown manga → 404 via the
FK-violation mapping; an invalid reaction value → 422. Reactions are never
exposed publicly — this is a taste signal for the upcoming content-based
recommendations. Integration tests cover set/toggle/clear/read, per-user
isolation, and the 404/422/401 paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-05 16:54:11 +02:00
MechaCat02
9c4a93c058 fix(reader): harden swipe/long-press coexistence and next-chapter prefetch
All checks were successful
deploy / test-backend (push) Successful in 32m21s
deploy / test-frontend (push) Successful in 10m33s
deploy / build-and-push (push) Successful in 11m40s
deploy / deploy (push) Successful in 18s
- A long-press now cancels any pending swipe, so a hold-then-drag gesture
  opens the action sheet without also turning the page (regression-tested).
- The next-chapter prefetch no longer retries on every page flip when the
  endpoint fails (guard clears only when the next chapter changes), and it
  clears a previous chapter's warmed images on navigation so stale hidden
  <img>s don't linger.
- Tap zones set touch-action: pan-y so vertical panning stays native while
  horizontal swipes reach our handler.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:57:36 +02:00
MechaCat02
bfaa166e0a fix(home): keep finished series off the Continue-reading shelf
A "Continue reading" shelf listing series the reader already finished (read
to the last page, nothing new) is odd. Expose the last-read chapter's
page_count on the read-progress list, and filter the shelf to drop entries
that are caught up (on the last page of the latest chapter with no new
chapters). Mid-chapter and has-new-chapters entries stay; the full history
view is unaffected. Finished detection is a pure, unit-tested helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:52:58 +02:00
MechaCat02
3824eaafb2 fix(history): count distinct new chapter numbers, authoritative on detail
new_chapters_count over-counted when scanlations share a chapter number
((manga_id, number) is non-unique, migration 0013): COUNT(*) tallied rows,
so duplicate-numbered chapters inflated the badge. Switch both the list and
per-manga read-progress queries to COUNT(DISTINCT number), and drop the dead
COALESCE (an empty set counts as 0, and the NULL-number case is handled by
the predicate). Fix the misleading comment.

Expose new_chapters_count on GET /me/read-progress/:manga_id and drive the
detail page's "N new" badge from it, instead of recomputing over the
paginated chapter list — which under-counted series past 50 chapters and
disagreed with the homepage shelf. Read markers stay client-side over the
loaded chapters and are documented as by-number (all we persist).

Adds duplicate-scanlation tests at both endpoints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:48:36 +02:00
MechaCat02
f1349100d7 feat(reader): prefetch the next chapter's first pages near the end
When the reader reaches within two pages of the end of a chapter (tracked in
both modes off the furthest page reached), warm the next chapter's first few
page images via hidden imgs so flipping over renders instantly. Fetched once
per next-chapter id, guarded against mid-flight chapter changes, and retried
on failure. e2e asserts the prefetch is absent at the chapter start and
present (with the next chapter's URLs) near the end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:28:06 +02:00
MechaCat02
3f1a5e9c41 feat(reader): swipe left/right to turn pages in single mode
Wire horizontal swipe into the mobile tap-zone overlay (single mode only,
so continuous mode keeps native vertical scroll). A leftward swipe past the
threshold turns to the next page, rightward to the previous; a mostly-
vertical drag is ignored so panning a tall page never turns it. The gesture
suppresses the synthesized zone tap so it doesn't double-fire. Swipe
classification is a pure, unit-tested helper; e2e drives real touch pointer
events for horizontal (both directions) and vertical cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:23:34 +02:00
MechaCat02
c212deb7b0 feat(reader): add a keyboard-shortcut help overlay
Press `?` in the reader to open a Sheet documenting the existing bindings
(arrows/J/K paging, Home/End, chapter nav, Esc); Esc or the close button
dismisses it. While open, the reader swallows navigation keys so paging
can't happen behind the overlay. e2e covers open-on-? and close-on-Esc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:18:47 +02:00
MechaCat02
f5692ea109 feat(ui): show a tooltip with the full text on truncated titles
Add an `overflowTooltip` Svelte action that sets a native `title` only when
an element is actually clipped (ellipsis or line-clamp), re-checking on
resize. Native title is keyboard/touch reachable and zero-dep, unlike a
hover-only popover. Applied to the clipped titles on MangaCard, BookmarkList,
HistoryList, and the Continue-reading shelf.

Unit tests cover the overflow decision and the set/clear/update behaviour;
an e2e verifies a real-browser truncated card title gets the tooltip while a
short one does not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:08:28 +02:00
MechaCat02
ad1689b818 feat(detail): link genre and tag chips to the filtered catalogue
Genre chips now deep-link to /?genres=<id> and user-tag chips to
/?tags=<id>, reusing the homepage catalogue's existing filter params —
turning the detail page's metadata into one-tap discovery. Tag chips keep
their remove affordance (the Chip's href branch preventDefaults the remove
button). e2e asserts both chip hrefs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:04:38 +02:00
MechaCat02
01d18e7ba2 feat(detail): mark read chapters and count new ones on the manga page
On the manga detail page, dim chapters at or before the reader's last-read
chapter (with an accent check + screen-reader "Read." label) and show a
"N new since last read" badge counting chapters that have landed since.
All client-side from the already-loaded chapter list and per-manga read
progress — no extra requests. The count reflects the loaded (paginated)
chapter list. Logic lives in a pure, unit-tested chapterProgress helper;
e2e covers the mid-series and never-opened cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 21:01:58 +02:00
MechaCat02
2267a83f6c feat(home): add a Continue-reading shelf with new-chapter badges
Surface a horizontal "Continue reading" shelf at the top of the homepage,
fed by the user's read-progress history. Each card jumps back to the exact
page they left off (deep-linking `?page=N` past the first) and flags how
many chapters have landed since — the personal new_chapters_count from the
read-progress list. The shelf is fetched after the catalogue load so the
public browse path stays unauthenticated; guests get an empty list (401
swallowed) and the shelf stays hidden.

Add new_chapters_count to the frontend ReadProgressSummary type to match
the backend. Component unit tests cover href/deep-link, badge visibility,
and the no-chapter fallback; e2e covers signed-in (shelf + badge) and
anonymous (hidden).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:57:17 +02:00
MechaCat02
f318c3bf51 feat(history): report personal new-chapter count on read-progress list
Add `new_chapters_count` to each `GET /me/read-progress` item: how many
chapters sit past the reader's last-read chapter, computed by a correlated
subquery on chapter number. It is 0 when the reader is caught up, and 0
when the last-read chapter is unknown (manga-level progress or a deleted
chapter) so we never over-claim. (manga_id, number) is non-unique, so this
counts positions past the reader rather than distinct files.

Backs the personal "new since last read" indicator on the upcoming
Continue-reading shelf. Tests cover the mid-series, caught-up, and
no-read-chapter cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:50:49 +02:00
MechaCat02
cf8971faae fix(reader): render pages at a consistent width instead of thin stripes
Reader page sizing was height-driven — `max-height: 90vh` in single mode
and unbounded natural width in continuous mode. A long vertical page hit
the height cap first and, preserving aspect ratio, collapsed into a thin
vertical stripe; narrow and wide scans rendered at different widths.

Switch both modes to width-driven sizing via a capped `--reader-page-width`
reading column (min(100%, 700px)): every page renders at the same width
regardless of intrinsic dimensions, and tall pages extend downward and
scroll instead of shrinking. Top-align the single-mode grid and make the
prev/next chevrons sticky so they stay reachable on a long page.

Add a Playwright spec asserting equal rendered width across differently-
proportioned pages and that a tall page exceeds the viewport height. The
page-context-menu fixture served a 1x1 image that now renders as a 700x700
square taller than the viewport, making Playwright scroll it into view
before right-clicking — a synthetic scroll that trips the menu's by-design
close-on-scroll. Give that fixture a realistic landscape aspect so the page
fits the viewport, matching real user behaviour (no scroll on right-click).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 20:45:59 +02:00
MechaCat02
38146b4d03 chore(frontend): sync package-lock version to 0.94.0
All checks were successful
deploy / build-and-push (push) Successful in 11m29s
deploy / test-backend (push) Successful in 30m25s
deploy / test-frontend (push) Successful in 10m29s
deploy / deploy (push) Successful in 13s
The lockfile's root version field trailed the manifests (0.90.0 vs
0.94.0) — harmless to `npm ci` (it checks the dependency tree, not the
project version) but tidy to keep aligned. Also folds in the peer-dep
metadata npm regenerated on the last install. No behaviour change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 19:45:05 +02:00
MechaCat02
8acc0e6cc2 feat(frontend): use the Mangalord monogram as the browser tab icon
Some checks failed
deploy / test-frontend (push) Has been cancelled
deploy / build-and-push (push) Has been cancelled
deploy / deploy (push) Has been cancelled
deploy / test-backend (push) Has been cancelled
The favicon link pointed at a non-existent favicon.ico. Ship the
monogram SVGs from static/ and wire two theme-aware <link rel="icon">
tags so the tab icon follows the browser color scheme: the dark-M
"light" monogram on light chrome, the white-M "dark" monogram on dark.

Bump 0.93.10 -> 0.94.0 (feat).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 19:34:45 +02:00
MechaCat02
4e154434a1 fix(upload): stream chapter pages to storage instead of buffering the whole chapter
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.

Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.

Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.

Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:33:47 +02:00
MechaCat02
141cd52f7e fix(frontend): percent-encode fileUrl key segments
fileUrl interpolated the raw storage key into the path, so a key segment
containing a reserved character (`?`, `#`, `%`, space) could be
reinterpreted as a query/fragment delimiter. Encode each `/`-separated
segment with encodeURIComponent, keeping slashes as literal path
separators. Keys are backend-generated today, so this is defence-in-depth.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:18:34 +02:00
MechaCat02
83a9ab40cd fix(crawler): reap dead jobs alongside done ones
The retention reaper deleted only `state = 'done'` rows, so terminal
`dead` jobs (retries exhausted) accumulated in crawler_jobs forever —
slow unbounded table growth. Both states are end-of-life and never
leased again.

Rename `reap_done` → `reap_terminal` and broaden the filter to
`state IN ('done','dead')`; `pending`/`running` stay untouched. Test
extended to assert old done + old dead both reap while fresh terminal and
old-but-active rows survive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:16:56 +02:00
MechaCat02
a141d65db1 fix(analysis): cap decoded pixels on the vision backend too
The vision prep pass decoded pages with bare `image::load_from_memory`,
which applies no allocation bound. `max_pixels` only downscales after a
full decode, so a tiny WebP/JPEG header declaring huge dimensions could
OOM the blocking worker — the same decompression-bomb the OCR backend
already guards against. Manga pages are commonly WebP/JPEG, where the
format's own self-limits are weaker than PNG's.

Route the decode through `decode_within`, applying an `image::Limits`
alloc cap sized from the shared `ocr_max_decode_pixels`
(ANALYSIS_OCR_MAX_DECODE_PIXELS). Add real-WebP bomb coverage asserting
both the helper and the end-to-end Undecodable fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:13:27 +02:00
MechaCat02
3a9e7ca2da fix(analysis): hold the OCR permit for the full blocking inference
The OCR dispatcher acquired a borrowed semaphore permit, then ran the
CPU-bound inference in `spawn_blocking`. On cancellation (graceful
shutdown) the future dropped the permit while the detached blocking task
kept running, briefly oversubscribing the blocking pool past
ANALYSIS_WORKERS.

Extract `run_ocr_blocking`, which acquires an *owned* permit and moves it
into the blocking task so the concurrency slot's lifetime matches the
actual work. Covered by a cancellation-invariant test asserting the
permit stays held after the caller future is aborted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:08:50 +02:00
MechaCat02
5dc93bfb84 fix(crawler): cap the number of page images per chapter
The per-image byte cap bounds each download but not the count, so a
hostile or compromised reader page listing thousands of <img> tags could
drive an unbounded disk fill. Add `CRAWLER_MAX_IMAGES_PER_CHAPTER`
(CrawlerConfig::max_images_per_chapter, default 2000, 0 = disabled) and
reject an over-cap chapter with a failed ack (exponential backoff) rather
than downloading it.

Threaded through sync_chapter_content and its three call sites (daemon
dispatcher, admin resync, CLI); enforced via `image_count_over_cap` right
after parse. The cap is an env-only safety knob, preserved across settings
reloads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 21:05:28 +02:00
MechaCat02
592747f1e0 fix(crawler): budget image tail against actual sniff-prefix length
The per-image download cap subtracted the constant SNIFF_PREFIX_BYTES (64)
from the budget instead of the bytes actually drained into the prefix.
Because the prefix loop appends whole chunks, a single ~16 KiB first chunk
fills the 64-byte sniff window in one drain, so the tail could then add
another near-full cap — storing up to ~2× max_image_bytes per page.

Capture prefix.len() before the prefix is moved into the stream and budget
the tail as `max_image_bytes - prefix_len` via a small `remaining_after_prefix`
helper (unit-tested for the overshoot and saturation cases).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:57:29 +02:00
MechaCat02
a0d63ac9fd fix(crawler): guard headless nav against internal SSRF targets on list/detail path
`navigate()` (list/detail/pagination) and the session probe called
`Browser::new_page()` without the SSRF check that already guards the
chapter-content path, so a hostile or compromised scraped source could
serve `<a href="http://169.254.169.254/…">` / `http://postgres:5432/` in a
listing and use the in-container Chromium as a read oracle.

Add `guard_navigate_url` (reusing `ensure_public_target`) at the top of
`navigate`, before rate-limiting or opening a page, and apply the same
check to `fetch_probe_html`. Covers base URL, pagination, and detail links.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 20:54:45 +02:00
MechaCat02
35c02066fe refactor(clippy): fix the never-loop error and clear all lint warnings
All checks were successful
deploy / test-backend (push) Successful in 30m17s
deploy / test-frontend (push) Successful in 10m33s
deploy / build-and-push (push) Successful in 11m11s
deploy / deploy (push) Successful in 13s
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:

- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
  literal `Err`).
- test helpers: build configs via struct-update syntax instead of
  `Default::default()` + field reassignment; add a type alias for a
  complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.

Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 20:45:37 +02:00
MechaCat02
ded77fe4ba test(e2e): kill cold-start flakiness with an /api/v1 fallback + one retry
A full parallel run intermittently failed the first few tests: unmocked
/api/v1 calls fell through to the dev proxy, whose backend isn't running
under E2E, so each incurred a slow ECONNREFUSED round-trip + Vite error
logging that piled up across 8 workers at cold start.

- Add e2e/fixtures.ts: a shared `test` whose auto-use fixture installs an
  `**/api/v1/**` fallback (fast 503, logged) for any endpoint a spec
  didn't mock. Registered before the test body, so per-test routes still
  win; only genuine gaps land here. Turns silent mock-gap hangs into
  instant, attributable failures — and cuts the full run ~96s → ~25s.
- Point all 22 specs at ./fixtures instead of @playwright/test.
- retries: 1 in the config as a safety net for the residual Vite
  cold-compile timing (re-runs on the now-warm server).

Two consecutive full runs: 110 passed, 0 flaky, no retries consumed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-02 20:06:20 +02:00
MechaCat02
4d02b56b77 test(e2e): realign mocks + assertions with the current API and OCR-era UI
The route-mocked E2E specs had drifted from the app they exercise, so
each rendered an empty/500 page and timed out on missing elements:

- Manga detail + reader loads now also fetch read-progress/{id} and
  /similar; several specs never mocked them, so the load's Promise.all
  rejected. Add the mocks and fill the manga fixtures out to a real
  MangaDetail (status/alt_titles/authors/genres/tags/... ) so the
  components stop throwing mid-render.
- back-nav + library: widen `read-progress*`/`page-tags` globs — `*`
  doesn't cross `/`, so `/me/read-progress/{id}` fell through to the
  proxy and 500'd; mock the library page-tags pair too.
- manga-list search: wait for the initial load to settle before
  searching so the search fetch can't race the mount-time load.
- reader-chapter-select: assert the shared `chapterLabel` output.
- admin-analysis: the active OCR backend extracts text only, so the
  detail modal shows OCR (no tags/NSFW) and metrics shows tiles +
  trend charts (no by-model) — assert what the OCR-era UI renders.
- profile: assert the wrong-password 401 stays inline (guards the
  fix in the preceding commit) and mock the guest bookmark/collection
  fetches the /profile load maps to the sign-in prompt.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 20:11:23 +02:00
MechaCat02
0865659ab3 fix(auth): keep session on a wrong-current-password 401
The change-password endpoint returns 401 for a wrong *current* password,
the same status the global on401 hook uses to detect an expired session.
The hook fired first and cleared session.user, so the account form's
"still signed in?" guard always saw null and bounced the user to /login
instead of surfacing the error inline.

Add a per-call `suppressOn401` option to `request`; changePassword opts
in, so a 401 there stays inline and the session is left intact.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 20:11:07 +02:00
MechaCat02
5596e1920d fix(compose): wire ANALYSIS_OCR_MAX_DECODE_PIXELS into the backend service
The OCR decompression-bomb guard added the env var to .env.example and
config.rs but not docker-compose.yml, so the container never received it
and the compose-env-coverage test failed once the branches were integrated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:25:59 +02:00
MechaCat02
3cba9ecf95 feat(auth): support optional expiry for bot API tokens
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:22:57 +02:00
MechaCat02
ed18d95bb0 feat(crawler): guard Chromium chapter navigation against internal targets
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:22:10 +02:00
MechaCat02
2ed42f7b9e feat(crawler): re-validate redirect hops to close SSRF via 3xx
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:21:15 +02:00
MechaCat02
cbbc626768 docs: correct ?text= OCR-search drift (no longer 501)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:19:38 +02:00
MechaCat02
a3e53f303b test(chapters): lock open-contribution upload contract + document intent
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:19:04 +02:00
MechaCat02
2af421893f fix(crawler): don't hold browser mutex across Chromium close()
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:49 +02:00
MechaCat02
48f0439273 fix(crawler): evict idle hosts from the per-host rate-limiter map
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:35 +02:00
MechaCat02
de7aefef69 fix(auth): guard the login + change-password verify paths against oversized passwords
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:19 +02:00
MechaCat02
3ba30f4ba9 fix(login): close ?next= control-char open-redirect + cover register
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:18:01 +02:00
MechaCat02
4306d1c96a fix(frontend): strip hop-by-hop headers from proxied responses too
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:17:32 +02:00
MechaCat02
886caaecfa fix(settings): bound manga_limit + the timeout knobs the finding named
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:17:14 +02:00
MechaCat02
de510cc19a fix(admin): make force-reanalyze audit transactional; document storage exception
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:16:08 +02:00
MechaCat02
5b76e0cc37 fix(analysis): bound concurrent OCR inferences with a shared semaphore
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:14:19 +02:00
MechaCat02
66ae4b221b fix(analysis): cap OCR decoded pixels to stop decompression-bomb OOM
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 07:12:40 +02:00
MechaCat02
83c2899373 feat(analysis): add ocrs OCR backend and OCR-driven page text search
All checks were successful
deploy / test-backend (push) Successful in 28m40s
deploy / test-frontend (push) Successful in 10m36s
deploy / build-and-push (push) Successful in 11m8s
deploy / deploy (push) Successful in 12s
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 19:52:43 +02:00
269 changed files with 15282 additions and 2776 deletions

View File

@@ -25,6 +25,12 @@ DATABASE_URL=postgres://mangalord:mangalord@postgres:5432/mangalord
BIND_ADDRESS=0.0.0.0:8080
STORAGE_DIR=/var/lib/mangalord/storage
RUST_LOG=info,mangalord=debug,chromiumoxide::conn=off,chromiumoxide::handler=off
# Postgres connection-pool sizing. One pool serves HTTP handlers and the
# crawler/analysis daemons. DB_MAX_CONNECTIONS caps open connections;
# DB_ACQUIRE_TIMEOUT_SECS is how long a request waits for a free connection
# before failing fast (rather than hanging on the driver's 30s default).
DB_MAX_CONNECTIONS=20
DB_ACQUIRE_TIMEOUT_SECS=10
# ----- Auth / cookies -----
# COOKIE_SECURE controls whether the `Secure` flag is set on the session
@@ -45,6 +51,13 @@ SESSION_TTL_DAYS=30
# rate-limiting reverse proxy that already enforces a budget).
AUTH_RATE_PER_SEC=5
AUTH_RATE_BURST=10
# Trust a proxy-supplied X-Forwarded-For as the client IP for per-IP auth
# rate limiting. Enable ONLY when the backend sits behind a trusted proxy
# that overrides the header (the compose deploy: SvelteKit forwards the real
# peer IP). When false, the header is ignored and a single shared bucket is
# used — a directly-exposed backend MUST keep this off or clients could spoof
# their IP to dodge the limit.
AUTH_TRUSTED_PROXY=false
# ----- CORS -----
# Comma-separated origins allowed to call the API with credentials.
@@ -62,11 +75,16 @@ CORS_ALLOWED_ORIGINS=
# neither Origin nor Referer (curl, server-to-server callers) are
# always allowed.
#
# Default is empty: CSRF check disabled (operator opt-out). For a
# browser-exposed deployment this should be set to the SvelteKit
# origin, e.g. https://app.example.com. For a same-origin
# docker-compose deploy where only one origin exists, set the same
# value the browser uses.
# Empty does NOT mean "off" for browsers: cookie-authenticated admin
# mutations FAIL CLOSED (403) when this is empty, since there's no
# allowlist to check the Origin against. Non-cookie callers (curl,
# bots with a Bearer token, no Origin/Referer) are still allowed.
# So set this to the SvelteKit origin for any browser-exposed deploy,
# e.g. https://app.example.com. For a same-origin docker-compose deploy
# set the same value the browser uses.
# Local dev (native `npm run dev`): the Vite origin is
# http://localhost:5173 — set ADMIN_ALLOWED_ORIGINS=http://localhost:5173
# or admin toggles (e.g. enabling the analysis worker) return 403.
ADMIN_ALLOWED_ORIGINS=
# ----- Admin bootstrap -----
@@ -98,6 +116,13 @@ MAX_REQUEST_BYTES=209715200
# oversized image is rejected even when the total request fits.
# Default 20 MiB.
MAX_FILE_BYTES=20971520
# Max page images accepted in one chapter upload. Bounds how many parts
# the handler will stage before rejecting the request with 413, so a
# client can't pin a worker with an unbounded page count. Default 2000.
# Setting 0 disables THIS cap — the total is then bounded only by
# MAX_REQUEST_BYTES (the whole-request body limit above), so leave 0 only
# if you intend that body limit to be the sole backstop.
MAX_PAGES_PER_CHAPTER=2000
# ----- Crawler download safety -----
# Hosts the crawler is allowed to fetch images/covers from, in addition
@@ -112,6 +137,11 @@ CRAWLER_DOWNLOAD_ALLOWLIST=
CRAWLER_ALLOW_ANY_HOST=false
# Hard cap on a single image body. Default 32 MiB.
CRAWLER_MAX_IMAGE_BYTES=33554432
# Hard cap on the number of page images in one crawled chapter. The
# per-image byte cap doesn't stop a hostile reader page listing thousands
# of <img> tags; an over-cap chapter is acked failed instead of downloaded.
# 0 disables the cap. Default 2000.
CRAWLER_MAX_IMAGES_PER_CHAPTER=2000
# Max manga detail fetches per metadata pass (both the in-process daemon
# and the `bin/crawler` CLI). 0 means no cap — let the source walker run
# to completion. Useful for capped test runs against a new source.
@@ -163,6 +193,19 @@ CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
# exhausted) that trigger an automatic coordinated browser restart.
# Default 3.
CRAWLER_BROWSER_RESTART_THRESHOLD=3
# CDP Fetch interception that re-validates every headless-browser
# navigation/redirect/subresource against the SSRF check (blocks a scraped page
# that drives the browser — via a redirect OR page JS/subresource — to an
# internal target such as the cloud metadata service or postgres). Default TRUE:
# with it off, only the top-level URL string is checked and page JS/subresources
# reach internal IPs (the reqwest SafeResolver does not cover Chromium's own
# stack). The CDP Fetch hook can't be exercised in CI (no Chromium); before
# relying on a fresh deploy, validate it doesn't wedge navigation:
# CRAWLER_CHROMIUM_BINARY=/usr/bin/chromium \
# cargo test -p mangalord --test crawler_browser_smoke -- --ignored \
# ssrf_interception_does_not_wedge_allowed_navigation
# Set to false only as a break-glass if the hook destabilizes a deployment.
CRAWLER_SSRF_INTERCEPT=true
# Path to a system Chromium binary. When set, the crawler skips the
# bundled-fetcher download. Required on platforms without a usable
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On
@@ -279,10 +322,13 @@ CRAWLER_TZ=UTC
ANALYSIS_ENABLED=false
# ANALYSIS_BACKEND Which engine the worker runs. env-ONLY (deploy-time).
# `ocr` (default) = the in-process ocrs engine: fast,
# CPU-only, text-only, ideal for a Pi. `vision` = the local
# LLM at ANALYSIS_VISION_URL (full OCR + tags + scene +
# safety, but heavy). The ANALYSIS_VISION_* / ANALYSIS_API_KEY
# knobs below only apply to `vision`.
# CPU-only, text-only, ideal for a Pi.
# NOTE: the `vision` backend (local LLM at ANALYSIS_VISION_URL,
# full OCR + tags + scene + safety) is TEMPORARILY DISABLED —
# its code is kept intact but the worker forces OCR regardless,
# logging a warning if `vision` is requested. So setting
# `vision` here currently has no effect; the ANALYSIS_VISION_* /
# ANALYSIS_API_KEY knobs below are dormant until it's re-enabled.
ANALYSIS_BACKEND=ocr
# OCRS_DETECTION_MODEL / OCRS_RECOGNITION_MODEL Paths to the ocrs `.rten`
# text-detection / -recognition models (only read when ANALYSIS_BACKEND=ocr).
@@ -320,6 +366,10 @@ ANALYSIS_API_KEY=
# beyond cap). Default 16.
# ANALYSIS_MAX_IMAGE_BYTES Per-image byte cap before downscale.
# Default 8388608 (8 MiB).
# ANALYSIS_OCR_MAX_DECODE_PIXELS Decompression-bomb guard for the OCR
# backend: hard cap on a page's *decoded* pixel
# count (encoded size is bounded separately).
# Default 100000000 (100 MP).
# ANALYSIS_RESPONSE_FORMAT `json_schema` | `json_object` | `none`.
# Default `json_schema`.
# ANALYSIS_FREQUENCY_PENALTY Discourages repetition loops. Default 0.3.
@@ -332,6 +382,7 @@ ANALYSIS_SLICE_OVERLAP=0.12
ANALYSIS_TALL_ASPECT=1.6
ANALYSIS_MAX_SLICES=16
ANALYSIS_MAX_IMAGE_BYTES=8388608
ANALYSIS_OCR_MAX_DECODE_PIXELS=100000000
ANALYSIS_RESPONSE_FORMAT=json_schema
ANALYSIS_FREQUENCY_PENALTY=0.3
ANALYSIS_TEMPERATURE=0.0
@@ -376,3 +427,14 @@ VISION_MANAGER_DATABASE_URL=
# VISION_MEM_LOW_WATERMARK_PCT used% >= this → inhibit starts (keep HIGH - LOW >= ~10 so the band beats jitter). Default 80.
# VISION_MEM_YIELD_COOLDOWN Seconds after a pressure-stop during which restart is refused regardless of backlog. Default 300.
# VISION_MEM_POLL_INTERVAL Mem sub-poll cadence, seconds (<= VISION_POLL_INTERVAL); catches spikes between backlog ticks. Default 5.
# ---- Deployment resource bounds & exposure (docker-compose.yml) ----
# Host interface the frontend port publishes on. Default 127.0.0.1 so SvelteKit
# (plain HTTP) is reachable only by a host-local reverse proxy. Set 0.0.0.0 only
# if your TLS terminator runs on a different host.
FRONTEND_PUBLISH_ADDR=127.0.0.1
# Per-container memory ceilings (docker mem_limit). Generous defaults; tune to
# your host. The backend runs OCR + headless Chromium and is the heavy one.
BACKEND_MEM_LIMIT=4g
FRONTEND_MEM_LIMIT=512m
POSTGRES_MEM_LIMIT=1g

View File

@@ -136,8 +136,8 @@ docker compose -f docker-compose.dev.yml up -d
These are first-class slots in the architecture. When adding any of them, plug into the existing seam rather than building parallel infrastructure.
- **Tags / lists**: new tables joined to `mangas`. New `domain`, `repo`, and `api` modules; the existing manga endpoints do not need to change.
- **Per-page collections / tags**: `collections` is heterogeneous — `collection_mangas` holds whole mangas, `collection_pages` holds individual pages (FK to `pages.id`). Per-user page tags live in `page_tags`, which references the **shared** `tags` table by `tag_id` (migration 0024) — the same lookup table `manga_tags` uses, so manga tags and page tags share one global vocabulary. The HTTP contract still speaks tag *names*; `repo::page_tag` resolves name↔id via `repo::tag::upsert_by_name` and applies the stricter page-tag `normalize_tag` (lowercase, collapse whitespace, reject wildcards/control/invisible chars) at the API layer. Both `collection_pages` and `page_tags` cascade-delete with `pages` and `chapters`, so re-uploading a chapter drops saved-page references by design.
- **Tag-based content search (`/search`)**: the user-facing search surface lives at [frontend/src/routes/search/+page.svelte](frontend/src/routes/search/+page.svelte). Three result views (Pages / Chapters / Mangas) consume the matching `/v1/me/page-tags`, `/v1/me/page-tags/chapters`, and `/v1/me/page-tags/mangas` endpoints. Note the two distinct query-param spaces: `?q=` on `/v1/me/page-tags` is a tag-name prefix (for autocomplete in the "Add tag" sheet); `?text=` on the aggregation endpoints is **reserved** for the planned OCR text-search input. Both aggregation handlers accept `text=` on the wire but reject non-empty values with 501 `text_search_not_yet_supported` so adding OCR later doesn't break the API shape. Adding OCR is then: a background worker writes `page_ocr_text` rows, a JOIN on the existing aggregation queries adds the new filter, the `text=` param starts validating instead of rejecting.
- **Per-page collections / tags**: `collections` is heterogeneous — `collection_mangas` holds whole mangas, `collection_pages` holds individual pages (FK to `pages.id`). Per-user page tags live in `page_tags`, which references the **shared** `tags` table by `tag_id` (migration 0024) — the same lookup table `manga_tags` uses, so manga tags and page tags share one global vocabulary. The HTTP contract still speaks tag *names*; `repo::page_tag` resolves name↔id via `repo::tag::upsert_by_name` and applies the stricter page-tag `normalize_tag` (lowercase, collapse whitespace, reject wildcards/control/invisible chars) at the API layer. Both `collection_pages` and `page_tags` cascade-delete with `pages` and `chapters`, so a saved-page reference is dropped only when its `pages` row is genuinely deleted — i.e. when the chapter is deleted (cascade), or when a user deletes and re-creates a chapter (the user upload path in `api::chapters::finalize_chapter` inserts a *new* chapter row with fresh page ids). A crawler **re-fetch** does **not** drop saves: `content::persist_pages` upserts pages by `(chapter_id, page_number)` (`ON CONFLICT (…) DO UPDATE … RETURNING id`), preserving each `pages.id`, so collections and page tags keyed on that id survive the re-fetch. (Migration 0023's header comment predates this and describes the cascade as an unconditional "re-upload drops saves"; the checked-in migration text is intentionally left as-is because sqlx checksums applied migrations.)
- **Tag-based content search (`/search`)**: the user-facing search surface lives at [frontend/src/routes/search/+page.svelte](frontend/src/routes/search/+page.svelte). Three result views (Pages / Chapters / Mangas) consume the matching `/v1/me/page-tags`, `/v1/me/page-tags/chapters`, and `/v1/me/page-tags/mangas` endpoints. Note the two distinct query-param spaces: `?q=` on `/v1/me/page-tags` is a tag-name prefix (for autocomplete in the "Add tag" sheet); `?text=` on the aggregation endpoints performs **OCR full-text search** the active OCR backend writes `page_ocr_text` rows and a weighted `search_doc` tsvector, and the aggregation queries JOIN on a `plainto_tsquery` filter ranked by `ts_rank` (see [backend/src/repo/page_analysis.rs](backend/src/repo/page_analysis.rs)). (`text=` was previously reserved and returned 501 `text_search_not_yet_supported`; that placeholder is gone now that the OCR backend is active. The generic `AppError::NotImplemented` 501 mechanism remains for future feature reservations.)
- **Full-text / fuzzy search**: enable `pg_trgm` in a migration and add a GIN index on `mangas.title`; swap the `WHERE` in `repo::manga::list` to use `%` operator or `tsvector`. The API shape (`?search=...`) does not change.
- **OCR / autotagging**: a background worker (a separate binary or a tokio task spawned in `app::build`) that reads pages from `storage::Storage` and writes tag rows. Do not couple OCR to upload handlers — it runs asynchronously.
- **S3 storage**: add `storage::S3Storage` implementing `Storage`. Branch in `app::build` based on a config field (e.g., `STORAGE_BACKEND=s3`). Handlers do not change.

4
backend/.gitignore vendored
View File

@@ -1,3 +1,7 @@
/target
/.sqlx
.env
# Local OCR models for native dev (downloaded, not source)
models/
*.rten

3
backend/Cargo.lock generated
View File

@@ -1558,7 +1558,7 @@ checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4"
[[package]]
name = "mangalord"
version = "0.90.0"
version = "0.128.25"
dependencies = [
"anyhow",
"argon2",
@@ -1587,7 +1587,6 @@ dependencies = [
"serde_json",
"sha2",
"sqlx",
"subtle",
"sysinfo",
"tempfile",
"thiserror 1.0.69",

View File

@@ -1,6 +1,6 @@
[package]
name = "mangalord"
version = "0.90.0"
version = "0.128.25"
edition = "2021"
default-run = "mangalord"
@@ -35,7 +35,6 @@ dotenvy = "0.15"
argon2 = "0.5"
rand = "0.8"
sha2 = "0.10"
subtle = "2"
base64 = "0.22"
# Image decode + downscale for the analysis worker (keep the page image
# under the local vision model's token budget). Only the manga page formats.
@@ -62,6 +61,7 @@ http-body-util = "0.1"
mime = "0.3"
futures-util = "0.3"
tokio = { version = "1", features = ["test-util"] }
image = { version = "0.25", default-features = false, features = ["jpeg", "png", "webp"] }
# Trim debug builds: keep line numbers in panics / backtraces but drop the
# full DWARF info (variable-level inspection in gdb/lldb). With a sqlx +

View File

@@ -0,0 +1,9 @@
-- Optional expiry for bot API tokens. NULL = never expires (the prior
-- behaviour, preserved for all existing rows). When set, `find_active`
-- rejects the token past this instant, mirroring the sessions table's
-- `expires_at > now()` gate.
ALTER TABLE api_tokens ADD COLUMN expires_at TIMESTAMPTZ;
-- Partial index to keep the active-token lookup cheap once expiries exist.
CREATE INDEX api_tokens_expires_at_idx ON api_tokens (expires_at)
WHERE expires_at IS NOT NULL;

View File

@@ -0,0 +1,15 @@
-- Per-user like/dislike reactions on mangas — a private taste signal that
-- powers content-based recommendations. One row per (user, manga); the
-- `reaction` column toggles between 'like' and 'dislike', and clearing a
-- reaction deletes the row. Reactions are never exposed publicly (no counts).
CREATE TABLE manga_reactions (
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
manga_id uuid NOT NULL REFERENCES mangas(id) ON DELETE CASCADE,
reaction text NOT NULL CHECK (reaction IN ('like', 'dislike')),
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, manga_id)
);
-- Recommendations aggregate a user's reacted mangas by tag; the PK covers
-- per-user lookups, this covers the reverse (all reactions on a manga).
CREATE INDEX manga_reactions_manga_idx ON manga_reactions (manga_id);

View File

@@ -0,0 +1,103 @@
-- Denormalized manga -> content-warning set.
--
-- The list filter previously tested each candidate manga with a correlated
-- `page_content_warnings -> pages -> chapters` join, i.e. O(mangas * pages) on
-- every filtered list AND its count. This table holds the DISTINCT union of a
-- manga's page warnings so the filter is a single indexed lookup.
--
-- Kept in sync by triggers that recompute an affected manga's set from current
-- data (the set is tiny — at most the five moderation labels — so a
-- delete-and-reinsert per change is cheap and always correct, sidestepping the
-- cascade-ordering hazards of incremental maintenance).
CREATE TABLE manga_content_warnings (
manga_id uuid NOT NULL REFERENCES mangas(id) ON DELETE CASCADE,
warning text NOT NULL
CHECK (warning IN ('sexual', 'nudity', 'gore', 'violence', 'disturbing')),
PRIMARY KEY (manga_id, warning)
);
-- warning -> mangas, for the include/exclude list filters.
CREATE INDEX manga_content_warnings_warning_idx ON manga_content_warnings (warning);
-- Recompute a single manga's warning set from the live per-page rows.
CREATE OR REPLACE FUNCTION mcw_refresh_for_manga(mid uuid) RETURNS void AS $$
BEGIN
-- Skip when the manga is gone (e.g. mid-cascade of a manga delete) so we
-- never re-insert a row that would violate the FK / resurrect a deleted set.
IF NOT EXISTS (SELECT 1 FROM mangas WHERE id = mid) THEN
RETURN;
END IF;
DELETE FROM manga_content_warnings WHERE manga_id = mid;
INSERT INTO manga_content_warnings (manga_id, warning)
SELECT DISTINCT mid, pw.warning
FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mid;
END;
$$ LANGUAGE plpgsql;
-- page_content_warnings changed for a page: refresh that page's manga.
CREATE OR REPLACE FUNCTION mcw_on_pcw_change() RETURNS trigger AS $$
DECLARE
mid uuid;
pid uuid := COALESCE(NEW.page_id, OLD.page_id);
BEGIN
-- The page (and thus chapter) may already be gone when this fires as part
-- of a pages/chapters cascade; in that case the pages/chapters triggers do
-- the refresh instead, so a missing join here is harmless.
SELECT c.manga_id INTO mid
FROM pages p JOIN chapters c ON c.id = p.chapter_id
WHERE p.id = pid;
IF mid IS NOT NULL THEN
PERFORM mcw_refresh_for_manga(mid);
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_pcw_ins AFTER INSERT ON page_content_warnings
FOR EACH ROW EXECUTE FUNCTION mcw_on_pcw_change();
CREATE TRIGGER mcw_pcw_del AFTER DELETE ON page_content_warnings
FOR EACH ROW EXECUTE FUNCTION mcw_on_pcw_change();
-- A page was deleted (directly, or via a chapter cascade): its
-- page_content_warnings rows are already cascade-gone, so recompute from the
-- chapter's manga. If the chapter is gone too, the chapters trigger covers it.
CREATE OR REPLACE FUNCTION mcw_on_page_delete() RETURNS trigger AS $$
DECLARE
mid uuid;
BEGIN
SELECT c.manga_id INTO mid FROM chapters c WHERE c.id = OLD.chapter_id;
IF mid IS NOT NULL THEN
PERFORM mcw_refresh_for_manga(mid);
END IF;
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_page_del AFTER DELETE ON pages
FOR EACH ROW EXECUTE FUNCTION mcw_on_page_delete();
-- A chapter was deleted (directly, or via a manga cascade): recompute from the
-- chapter's manga. `chapters.manga_id` is on the row, so it is always available
-- even after the child pages have cascaded; the refresh no-ops when the manga
-- itself is being deleted.
CREATE OR REPLACE FUNCTION mcw_on_chapter_delete() RETURNS trigger AS $$
BEGIN
PERFORM mcw_refresh_for_manga(OLD.manga_id);
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER mcw_chapter_del AFTER DELETE ON chapters
FOR EACH ROW EXECUTE FUNCTION mcw_on_chapter_delete();
-- Backfill from existing per-page rows.
INSERT INTO manga_content_warnings (manga_id, warning)
SELECT DISTINCT c.manga_id, pw.warning
FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
ON CONFLICT DO NOTHING;

View File

@@ -0,0 +1,51 @@
-- Precomputed author sort key for `?sort=author`.
--
-- The author sort used a correlated `min(lower(a.name))` subquery as the ORDER
-- BY key, evaluated per filter-matching row before LIMIT — it scaled worse than
-- the indexed date/title sorts. Materialize the same value on `mangas` so the
-- sort is a plain indexed column read.
--
-- `sort_author` = the alphabetically-first attached author's lowercased name,
-- or NULL when the manga has no authors (kept last via NULLS LAST in the query).
-- Author names are immutable (authors are upserted by unique lowercased name and
-- never renamed), so the value only changes when the manga_authors join changes
-- — maintained by the triggers below.
ALTER TABLE mangas ADD COLUMN sort_author text;
CREATE INDEX mangas_sort_author_idx ON mangas (sort_author, id);
-- Backfill from existing links.
UPDATE mangas m
SET sort_author = (
SELECT min(lower(a.name))
FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = m.id
);
CREATE OR REPLACE FUNCTION refresh_manga_sort_author(mid uuid) RETURNS void AS $$
BEGIN
UPDATE mangas
SET sort_author = (
SELECT min(lower(a.name))
FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = mid
)
WHERE id = mid;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE FUNCTION mangas_sort_author_on_ma_change() RETURNS trigger AS $$
BEGIN
-- On a manga cascade-delete the UPDATE simply no-ops (row already gone).
PERFORM refresh_manga_sort_author(COALESCE(NEW.manga_id, OLD.manga_id));
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER manga_authors_sort_author_ins AFTER INSERT ON manga_authors
FOR EACH ROW EXECUTE FUNCTION mangas_sort_author_on_ma_change();
CREATE TRIGGER manga_authors_sort_author_del AFTER DELETE ON manga_authors
FOR EACH ROW EXECUTE FUNCTION mangas_sort_author_on_ma_change();

View File

@@ -0,0 +1,44 @@
-- Enforce case-insensitive genre uniqueness, matching `authors` and `tags`
-- (both got `UNIQUE (lower(name))` in 0009). `genres` only had a case-SENSITIVE
-- `name UNIQUE`, but the crawler's `sync_genres` treats genres as
-- case-insensitive (`WHERE lower(name) = lower($1)`) and can INSERT a new row
-- from a source string. Under a race (or across ticks) two different-cased
-- strings — "Isekai" vs "isekai" — could both miss the pre-check and both
-- insert, since the exact-name unique doesn't collide. That leaves duplicate
-- genre rows for one logical genre.
-- Heal any pre-existing case-variant duplicates before adding the index, so
-- `sqlx::migrate!` can't crash on a dirty table (mirrors 0031's pre-dedup).
-- Canonical = lowest id per lower(name).
-- 1. Ensure every manga linked to any case-variant also has the canonical link.
-- INSERT ... ON CONFLICT DO NOTHING is collision-proof: SELECT DISTINCT
-- collapses multiple variants of one manga to a single canonical row, and the
-- ON CONFLICT absorbs a canonical link that already exists. (A prior
-- UPDATE-repoint could set two non-canonical rows of the SAME manga to the
-- same canonical id in a single statement and violate the manga_genres PK,
-- rolling the migration back and wedging startup.)
INSERT INTO manga_genres (manga_id, genre_id)
SELECT DISTINCT mg.manga_id, k.keep_id
FROM manga_genres mg
JOIN genres g ON mg.genre_id = g.id
JOIN (SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
ON lower(g.name) = k.lname
WHERE g.id <> k.keep_id
ON CONFLICT (manga_id, genre_id) DO NOTHING;
-- 2. Every non-canonical link now has a canonical sibling — drop the variants.
DELETE FROM manga_genres mg
USING genres g,
(SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
WHERE mg.genre_id = g.id AND lower(g.name) = k.lname AND g.id <> k.keep_id;
-- Remove the now-orphaned duplicate genre rows.
DELETE FROM genres g
USING (SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id
FROM genres GROUP BY lower(name)) k
WHERE lower(g.name) = k.lname AND g.id <> k.keep_id;
CREATE UNIQUE INDEX genres_name_lower_uniq ON genres (lower(name));

View File

@@ -0,0 +1,18 @@
-- Index the crashed-worker-recovery arm of the job lease predicate.
--
-- lease/lease_kinds match:
-- WHERE (state = 'pending' OR (state = 'running' AND leased_until < now()))
-- crawler_jobs_ready_idx (0016) is `ON (scheduled_at) WHERE state = 'pending'`,
-- so it covers only the pending arm. The `state = 'running' AND leased_until`
-- arm had no usable index, so Postgres could not BitmapOr the two arms and
-- degraded to a sequential scan of the ENTIRE crawler_jobs table — including all
-- done/dead rows not yet reaped — on every lease poll, by every worker,
-- continuously (audit H2, the headline performance finding).
--
-- A partial index on leased_until over just the running rows makes the recovery
-- arm index-backed. `state = 'running'` is an immutable predicate (now() stays
-- in the query, not the index). The running set is tiny (in-flight jobs only),
-- so the index is cheap to maintain.
CREATE INDEX crawler_jobs_running_lease_idx
ON crawler_jobs (leased_until)
WHERE state = 'running';

View File

@@ -0,0 +1,9 @@
-- Index the retention reaper's scan. reap_terminal deletes
-- WHERE state IN ('done','dead') AND updated_at < now() - interval
-- which had no supporting index, so each daily sweep sequential-scanned the whole
-- crawler_jobs table to find expired terminal rows (audit M5). A partial index on
-- updated_at over just the terminal states makes the batched reaper's per-batch
-- SELECT index-backed.
CREATE INDEX crawler_jobs_terminal_reap_idx
ON crawler_jobs (updated_at)
WHERE state IN ('done', 'dead');

View File

@@ -37,6 +37,22 @@ const LEASE_HEARTBEAT: Duration = Duration::from_secs(20);
/// long enough not to hammer `/health` while vision is down.
const READINESS_POLL: Duration = Duration::from_secs(2);
/// Longest an idle analysis worker waits between lease polls, bounding the
/// exponential [`idle_backoff`].
const IDLE_BACKOFF_CAP: Duration = Duration::from_secs(30);
/// Exponential idle backoff (1s, 2s, 4s … capped at [`IDLE_BACKOFF_CAP`]).
/// `consecutive_empty` is the number of empty lease polls so far (0 on the first
/// miss); reset to 0 the moment a job is leased. Replaces the old flat 1s sleep
/// so an idle worker isn't firing a `SELECT … FOR UPDATE SKIP LOCKED` lease
/// query every second (audit: analysis daemon idle poll). Mirrors the crawler
/// daemon's backoff.
fn idle_backoff(consecutive_empty: u32) -> Duration {
let cap = IDLE_BACKOFF_CAP.as_secs();
let secs = 1u64.checked_shl(consecutive_empty).unwrap_or(cap).min(cap);
Duration::from_secs(secs)
}
/// The unit of work: analyze one page. Implemented by
/// [`RealAnalyzeDispatcher`] in production and stubbed in tests.
#[async_trait]
@@ -135,6 +151,9 @@ impl WorkerContext {
// Last observed readiness, so we log only on transitions (not every
// poll). `None` until the first probe.
let mut was_ready: Option<bool> = None;
// Empty lease polls seen in a row, driving the idle backoff. Reset to 0
// the moment a job is leased.
let mut consecutive_empty: u32 = 0;
loop {
if self.cancel.is_cancelled() {
tracing::info!(worker = self.id, "analysis worker: shutdown");
@@ -178,11 +197,15 @@ impl WorkerContext {
}
};
let Some(lease) = leases.into_iter().next() else {
if self.sleep_or_cancel(Duration::from_secs(1)).await {
let wait = idle_backoff(consecutive_empty);
consecutive_empty = consecutive_empty.saturating_add(1);
if self.sleep_or_cancel(wait).await {
return;
}
continue;
};
// Leased work — drop back to a tight poll cadence.
consecutive_empty = 0;
self.process_lease(lease).await;
}
}
@@ -393,19 +416,20 @@ impl AnalyzeDispatcher for RealAnalyzeDispatcher {
// Page was deleted between enqueue and dispatch — nothing to do.
return Ok(());
};
let bytes = self
// Stream through the byte cap so an oversized blob is rejected as it's
// read rather than after it's fully buffered into memory (mirrors the
// OCR dispatcher).
let file = self
.storage
.get(&page.storage_key)
.get_stream(&page.storage_key)
.await
.map_err(|e| anyhow::anyhow!("read page image {}: {e}", page.storage_key))?;
if bytes.len() > self.max_image_bytes {
anyhow::bail!(
"page image {} is {} bytes, over the {} cap",
page.storage_key,
bytes.len(),
self.max_image_bytes
);
}
let bytes =
crate::crawler::safety::accumulate_capped(file.stream, self.max_image_bytes)
.await
.map_err(|e| {
anyhow::anyhow!("page image {} over the byte cap: {e}", page.storage_key)
})?;
let analysis = self.vision.analyze(&bytes, &page.content_type).await?;
repo::page_analysis::persist_analysis(&self.db, page_id, &analysis, &self.model).await?;
Ok(())
@@ -509,6 +533,17 @@ mod tests {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpListener;
#[test]
fn idle_backoff_grows_exponentially_and_caps() {
assert_eq!(idle_backoff(0), Duration::from_secs(1));
assert_eq!(idle_backoff(1), Duration::from_secs(2));
assert_eq!(idle_backoff(2), Duration::from_secs(4));
assert_eq!(idle_backoff(4), Duration::from_secs(16));
// Caps at IDLE_BACKOFF_CAP (30s) and never overflows on large counts.
assert_eq!(idle_backoff(5), IDLE_BACKOFF_CAP);
assert_eq!(idle_backoff(100), IDLE_BACKOFF_CAP);
}
/// Bind an ephemeral port that answers every request with `status_line`
/// (e.g. `"200 OK"`), and return its `/health` URL. The probe under test
/// only inspects the status code, so a zero-length body is enough.

View File

@@ -60,13 +60,23 @@ pub fn lines_to_analysis(lines: Vec<String>) -> VisionAnalysis {
/// recognize path borrows `&self`.
pub struct OcrsEngine {
engine: ocrs::OcrEngine,
/// Hard cap on decoded pixel count (decompression-bomb guard). See
/// [`decode_rgb8_within`].
max_decode_pixels: u64,
}
impl OcrsEngine {
/// Load the detection + recognition models from disk and build the engine.
/// Fails (at startup) if either model file is missing or unreadable, so a
/// misconfigured path is a loud boot error rather than a per-page failure.
pub fn from_model_paths(detection: &str, recognition: &str) -> anyhow::Result<Self> {
///
/// `max_decode_pixels` bounds the decoded image size (see
/// [`decode_rgb8_within`]) — wired from `AnalysisConfig::ocr_max_decode_pixels`.
pub fn from_model_paths(
detection: &str,
recognition: &str,
max_decode_pixels: u64,
) -> anyhow::Result<Self> {
use anyhow::Context;
let detection_model = rten::Model::load_file(detection)
.with_context(|| format!("load ocrs detection model {detection}"))?;
@@ -78,17 +88,41 @@ impl OcrsEngine {
..Default::default()
})
.context("construct ocrs engine")?;
Ok(Self { engine })
Ok(Self { engine, max_decode_pixels })
}
}
/// Decode encoded image bytes to RGB8 while refusing decompression bombs.
///
/// `image::load_from_memory` allocates the full decoded buffer up front, so a
/// tiny file declaring enormous dimensions (e.g. a 50000×50000 PNG) inflates
/// to billions of bytes and OOM-kills the process. We cap the decoder's
/// allocation at `max_decode_pixels` worth of RGBA (4 bytes/px headroom over
/// the RGB8 result), which the `image` crate checks against the header
/// *before* allocating — so an over-size image fails fast instead of dying.
fn decode_rgb8_within(image: &[u8], max_decode_pixels: u64) -> anyhow::Result<image::RgbImage> {
use anyhow::Context;
use std::io::Cursor;
let mut reader = image::ImageReader::new(Cursor::new(image))
.with_guessed_format()
.context("guess page image format for OCR")?;
let mut limits = image::Limits::default();
// 4 bytes/px (RGBA) gives headroom over the eventual RGB8 buffer and any
// single intermediate the decoder allocates per pixel.
limits.max_alloc = Some(max_decode_pixels.saturating_mul(4));
reader.limits(limits);
Ok(reader
.decode()
.context("decode page image for OCR")?
.into_rgb8())
}
impl OcrEngine for OcrsEngine {
fn recognize(&self, image: &[u8]) -> anyhow::Result<Vec<String>> {
use anyhow::Context;
// Decode to RGB8 so `ImageSource` gets a known channel layout.
let rgb = image::load_from_memory(image)
.context("decode page image for OCR")?
.into_rgb8();
// Decode to RGB8 so `ImageSource` gets a known channel layout,
// bounding the decoded size against the decompression-bomb cap.
let rgb = decode_rgb8_within(image, self.max_decode_pixels)?;
let source = ocrs::ImageSource::from_bytes(rgb.as_raw(), rgb.dimensions())
.map_err(|e| anyhow::anyhow!("build OCR image source: {e}"))?;
let input = self.engine.prepare_input(source)?;
@@ -109,6 +143,49 @@ impl OcrEngine for OcrsEngine {
}
}
/// Bound on concurrent CPU-bound OCR inferences, given the number of analysis
/// workers and the host's available parallelism.
///
/// Each worker's `dispatch` fires one `spawn_blocking` OCR run, so without a
/// bound `ANALYSIS_WORKERS` blocking tasks can run at once. OCR is fully
/// CPU-bound, so running more than there are cores just thrashes the scheduler
/// (and balloons the blocking pool). Cap at the core count, but never below 1
/// and never above the worker count (more permits than workers is pointless).
pub fn ocr_concurrency_limit(workers: usize, cores: usize) -> usize {
workers.min(cores.max(1)).max(1)
}
/// Run a CPU-bound OCR closure on the blocking pool while holding an
/// **owned** permit for the entire duration of the work.
///
/// The permit is acquired with `acquire_owned` and moved *into* the
/// blocking task rather than being held by the caller's future. This
/// matters on cancellation: if the dispatcher future is dropped (graceful
/// shutdown), a borrowed permit would be released the instant the future
/// unwinds — but `spawn_blocking` work is not cancellable and keeps
/// running detached, so the concurrency bound (ANALYSIS_WORKERS) would be
/// briefly exceeded. Moving the permit into the task ties the slot's
/// lifetime to the actual CPU work.
async fn run_ocr_blocking<T, F>(
permits: Arc<tokio::sync::Semaphore>,
f: F,
) -> anyhow::Result<T>
where
F: FnOnce() -> T + Send + 'static,
T: Send + 'static,
{
let permit = permits
.acquire_owned()
.await
.map_err(|e| anyhow::anyhow!("OCR semaphore closed: {e}"))?;
tokio::task::spawn_blocking(move || {
let _permit = permit; // held until f() returns, even if the caller is cancelled
f()
})
.await
.map_err(|e| anyhow::anyhow!("OCR task join error: {e}"))
}
/// Production dispatcher for the OCR backend: load the page, read its image
/// from storage, run OCR on the blocking pool, and persist the lines. Mirrors
/// [`crate::analysis::daemon::RealAnalyzeDispatcher`] but with no network I/O.
@@ -117,6 +194,10 @@ pub struct OcrAnalyzeDispatcher {
pub storage: Arc<dyn Storage>,
pub engine: Arc<dyn OcrEngine>,
pub max_image_bytes: usize,
/// Caps concurrent CPU-bound OCR inferences across all workers (see
/// [`ocr_concurrency_limit`]). Shared via the `Arc<dyn AnalyzeDispatcher>`,
/// so one permit pool covers every worker.
pub ocr_permits: Arc<tokio::sync::Semaphore>,
}
#[async_trait]
@@ -126,25 +207,28 @@ impl AnalyzeDispatcher for OcrAnalyzeDispatcher {
// Page was deleted between enqueue and dispatch — nothing to do.
return Ok(());
};
let bytes = self
// Stream the blob through the byte cap so the read itself bails once
// the running total exceeds `max_image_bytes` — a plain `get()` would
// buffer the whole (possibly huge) image into memory before the cap
// could reject it.
let file = self
.storage
.get(&page.storage_key)
.get_stream(&page.storage_key)
.await
.map_err(|e| anyhow::anyhow!("read page image {}: {e}", page.storage_key))?;
if bytes.len() > self.max_image_bytes {
anyhow::bail!(
"page image {} is {} bytes, over the {} cap",
page.storage_key,
bytes.len(),
self.max_image_bytes
);
}
let bytes =
crate::crawler::safety::accumulate_capped(file.stream, self.max_image_bytes)
.await
.map_err(|e| {
anyhow::anyhow!("page image {} over the byte cap: {e}", page.storage_key)
})?;
// OCR inference is CPU-bound and synchronous — keep it off the async
// worker's runtime thread.
// worker's runtime thread, and gate it behind the shared permit pool so
// ANALYSIS_WORKERS > cores can't oversubscribe the blocking pool.
let engine = Arc::clone(&self.engine);
let lines = tokio::task::spawn_blocking(move || engine.recognize(&bytes))
.await
.map_err(|e| anyhow::anyhow!("OCR task join error: {e}"))??;
let lines =
run_ocr_blocking(Arc::clone(&self.ocr_permits), move || engine.recognize(&bytes))
.await??;
let analysis = lines_to_analysis(lines);
repo::page_analysis::persist_analysis(&self.db, page_id, &analysis, OCR_MODEL_LABEL).await?;
Ok(())
@@ -179,6 +263,55 @@ pub mod test_support {
mod tests {
use super::*;
#[tokio::test]
async fn run_ocr_blocking_holds_permit_until_work_completes_despite_cancellation() {
use std::sync::mpsc;
use tokio::sync::{Notify, Semaphore};
let permits = Arc::new(Semaphore::new(1));
let (release_tx, release_rx) = mpsc::channel::<()>();
let started = Arc::new(Notify::new());
// Model the dispatch path: acquire an owned permit and run blocking
// work that we hold open via a channel.
let sem = Arc::clone(&permits);
let started2 = Arc::clone(&started);
let caller = tokio::spawn(async move {
run_ocr_blocking(sem, move || {
// Signal that the blocking task now holds the permit, then
// block until the test releases us.
started2.notify_one();
release_rx.recv().ok();
})
.await
.unwrap();
});
// Wait until the blocking task is running and owns the permit.
started.notified().await;
assert_eq!(permits.available_permits(), 0, "permit taken by blocking work");
// Cancel the caller future (simulates graceful shutdown). The
// blocking task is not cancellable and keeps running detached; with
// an *owned* permit the slot must stay occupied. A borrowed permit
// would have been released here, regressing the concurrency bound.
caller.abort();
let _ = caller.await;
assert_eq!(
permits.available_permits(),
0,
"owned permit must remain held by the still-running blocking task"
);
// Let the blocking work finish; the permit is then returned.
release_tx.send(()).unwrap();
let permit = tokio::time::timeout(std::time::Duration::from_secs(5), permits.acquire())
.await
.expect("permit should be released once blocking work completes")
.unwrap();
drop(permit);
}
#[test]
fn lines_to_analysis_maps_lines_in_order_and_leaves_rest_empty() {
let v = lines_to_analysis(vec!["Hello".to_string(), "world!".to_string()]);
@@ -198,4 +331,58 @@ mod tests {
let v = lines_to_analysis(Vec::new());
assert!(v.ocr_results.is_empty());
}
/// A minimal valid PNG of `w`×`h` (single black pixel scaled via IHDR is
/// not valid; instead encode a real tiny image, then we patch the IHDR
/// dimensions for the bomb case). For the happy path we just encode a real
/// small image.
fn encode_png(w: u32, h: u32) -> Vec<u8> {
let img = image::RgbImage::new(w, h);
let mut buf = std::io::Cursor::new(Vec::new());
image::DynamicImage::ImageRgb8(img)
.write_to(&mut buf, image::ImageFormat::Png)
.unwrap();
buf.into_inner()
}
#[test]
fn decode_rgb8_within_accepts_normal_page() {
// A perfectly ordinary page decodes fine under a generous cap.
let png = encode_png(64, 96);
let rgb = decode_rgb8_within(&png, 100_000_000).unwrap();
assert_eq!(rgb.dimensions(), (64, 96));
}
#[test]
fn decode_rgb8_within_rejects_oversize_image() {
// The same image, but the cap is set below its pixel count: the
// allocation limit must trip rather than the decode succeeding. This
// is the decompression-bomb guard in miniature — a real bomb declares
// a huge size in a few bytes; here we shrink the budget instead.
let png = encode_png(2000, 2000); // 4 MP
let err = decode_rgb8_within(&png, 1_000).unwrap_err();
assert!(
err.to_string().contains("decode page image"),
"expected a decode error, got: {err}"
);
}
#[test]
fn ocr_concurrency_limit_caps_at_cores() {
// More workers than cores → clamp to cores (don't oversubscribe).
assert_eq!(ocr_concurrency_limit(8, 4), 4);
}
#[test]
fn ocr_concurrency_limit_caps_at_workers() {
// Fewer workers than cores → only `workers` ever run anyway.
assert_eq!(ocr_concurrency_limit(2, 16), 2);
}
#[test]
fn ocr_concurrency_limit_never_zero() {
// Degenerate inputs still yield at least one permit.
assert_eq!(ocr_concurrency_limit(0, 0), 1);
assert_eq!(ocr_concurrency_limit(1, 0), 1);
}
}

View File

@@ -55,6 +55,12 @@ struct SliceParams {
overlap: f64,
tall_threshold: f64,
max_slices: usize,
/// Hard cap on the **decoded** pixel count, enforced as an allocation
/// limit on the decoder itself. `max_pixels` only downscales *after* a
/// full decode, so without this a tiny WebP/JPEG/PNG declaring huge
/// dimensions would OOM the blocking worker (decompression bomb).
/// Shared with the OCR backend's cap (`ANALYSIS_OCR_MAX_DECODE_PIXELS`).
max_decode_pixels: u64,
}
/// Result of the (blocking-pool) prep pass: a self-contained set of byte
@@ -86,11 +92,30 @@ enum PreparedAnalysis {
/// JPEG-encoder regression indistinguishable from "the page was just
/// garbage." Emit a `warn` on each fallback so an operator can grep
/// for "vision prep fell through" and tell the two apart.
/// Decode an encoded page image with a hard allocation cap so a
/// decompression bomb — a tiny WebP/JPEG/PNG header declaring enormous
/// dimensions — can't OOM the blocking worker before we get a chance to
/// downscale. The `image` crate's default reader applies no such bound;
/// format-specific self-limits (strongest for PNG) don't cover WebP/JPEG,
/// which manga pages commonly use. `4 bytes/px` (RGBA) leaves headroom
/// over any single intermediate the decoder allocates per pixel. Mirrors
/// `ocr::decode_rgb8_within`.
fn decode_within(image: &[u8], max_decode_pixels: u64) -> Option<DynamicImage> {
use std::io::Cursor;
let mut reader = image::ImageReader::new(Cursor::new(image))
.with_guessed_format()
.ok()?;
let mut limits = image::Limits::default();
limits.max_alloc = Some(max_decode_pixels.saturating_mul(4));
reader.limits(limits);
reader.decode().ok()
}
fn prepare_analysis(image: &[u8], params: SliceParams) -> PreparedAnalysis {
let Some(img) = image::load_from_memory(image).ok() else {
let Some(img) = decode_within(image, params.max_decode_pixels) else {
tracing::warn!(
bytes = image.len(),
"vision prep fell through to Undecodable: image::load_from_memory failed"
"vision prep fell through to Undecodable: decode failed or exceeded pixel cap"
);
return PreparedAnalysis::Undecodable;
};
@@ -162,6 +187,7 @@ impl VisionClient {
overlap: cfg.slice_overlap,
tall_threshold: cfg.tall_aspect_threshold,
max_slices: cfg.max_slices,
max_decode_pixels: cfg.ocr_max_decode_pixels,
},
}
}
@@ -779,9 +805,40 @@ mod tests {
overlap: 0.12,
tall_threshold: 1.6,
max_slices: 16,
max_decode_pixels: 100_000_000,
}
}
fn encode_webp(w: u32, h: u32) -> Vec<u8> {
use image::{DynamicImage, ImageFormat, RgbImage};
let img = DynamicImage::ImageRgb8(RgbImage::new(w, h));
let mut buf = std::io::Cursor::new(Vec::new());
img.write_to(&mut buf, ImageFormat::WebP).expect("encode webp");
buf.into_inner()
}
#[test]
fn decode_within_rejects_oversize_webp() {
// Non-PNG decompression-bomb coverage: a real 2000×2000 WebP (4 MP)
// must be refused when the decode cap is set below its pixel count.
// The allocation limit trips inside the decoder rather than the
// full frame being materialized. Manga pages are commonly WebP/JPEG,
// where the format's own self-limits are weaker than PNG's.
let webp = encode_webp(2000, 2000);
assert!(decode_within(&webp, 1_000).is_none(), "cap below pixels must reject");
// A generous cap decodes the same image fine.
assert!(decode_within(&webp, 100_000_000).is_some(), "within cap must decode");
}
#[test]
fn prepare_analysis_falls_back_to_undecodable_on_decode_bomb() {
// End-to-end: an over-cap image drives the prep pass to the
// Undecodable fallback (raw-bytes path) instead of decoding it.
let webp = encode_webp(2000, 2000);
let p = SliceParams { max_decode_pixels: 1_000, ..params() };
assert!(matches!(prepare_analysis(&webp, p), PreparedAnalysis::Undecodable));
}
fn ocr(text: &str, kind: &str) -> OcrResult {
OcrResult {
text: text.into(),
@@ -1136,6 +1193,7 @@ mod tests {
overlap: 0.05,
tall_threshold: 1.8,
max_slices: 6,
max_decode_pixels: 100_000_000,
}
}
@@ -1212,14 +1270,16 @@ mod tests {
// VisionClient pointed at a closed local port. Reqwest fails the
// POST immediately (connection refused), but the prep work runs
// before the POST is even built.
let mut cfg = AnalysisConfig::default();
cfg.endpoint = "http://127.0.0.1:1/v1/chat/completions".into();
cfg.model = "stub".into();
cfg.request_timeout = Duration::from_secs(1);
// Tune slice geometry to match the test image's shape.
cfg.max_pixels = 1_000_000;
cfg.min_slice_height = 100;
cfg.tall_aspect_threshold = 1.8;
let cfg = AnalysisConfig {
endpoint: "http://127.0.0.1:1/v1/chat/completions".into(),
model: "stub".into(),
request_timeout: Duration::from_secs(1),
// Tune slice geometry to match the test image's shape.
max_pixels: 1_000_000,
min_slice_height: 100,
tall_aspect_threshold: 1.8,
..Default::default()
};
let http = reqwest::Client::builder()
.timeout(cfg.request_timeout)
.no_proxy()

View File

@@ -58,20 +58,20 @@ async fn stream_status(
) -> Sse<impl Stream<Item = Result<Event, Infallible>>> {
let rx = state.analysis_events.subscribe();
let stream = futures_util::stream::unfold(rx, |mut rx| async move {
loop {
match rx.recv().await {
Ok(ev) => {
let event = Event::default()
.event("analysis")
.json_data(&ev)
.unwrap_or_else(|_| Event::default().comment("serialize error"));
return Some((Ok(event), rx));
}
Err(RecvError::Lagged(_)) => {
return Some((Ok(Event::default().event("lagged").data("")), rx));
}
Err(RecvError::Closed) => return None,
// One recv per unfold step; the stream driver re-enters for the next
// event, so no explicit loop is needed here.
match rx.recv().await {
Ok(ev) => {
let event = Event::default()
.event("analysis")
.json_data(&ev)
.unwrap_or_else(|_| Event::default().comment("serialize error"));
Some((Ok(event), rx))
}
Err(RecvError::Lagged(_)) => {
Some((Ok(Event::default().event("lagged").data("")), rx))
}
Err(RecvError::Closed) => None,
}
});
Sse::new(stream).keep_alive(KeepAlive::default())
@@ -328,10 +328,21 @@ fn ensure_enabled(state: &AppState) -> AppResult<()> {
async fn reenqueue(
State(state): State<AppState>,
admin: RequireAdmin,
body: Option<Json<ReenqueueBody>>,
raw: axum::body::Bytes,
) -> AppResult<Json<ReenqueueResponse>> {
ensure_enabled(&state)?;
let body = body.map(|b| b.0).unwrap_or_default();
// An ABSENT/empty body means the default "All" scope. A PRESENT body must be
// valid JSON — `Option<Json<T>>` would silently collapse a malformed body to
// None and run the full-library default the caller never intended, so parse
// the raw bytes ourselves and 422 on a real parse error.
let body: ReenqueueBody = if raw.iter().all(u8::is_ascii_whitespace) {
ReenqueueBody::default()
} else {
serde_json::from_slice(&raw).map_err(|e| AppError::ValidationFailed {
message: "reenqueue body is not valid JSON".into(),
details: json!({ "body": e.to_string() }),
})?
};
// Resolve the scope. manga_id and chapter_id are mutually exclusive;
// an unknown target is a 404 rather than a silent zero-enqueue.
@@ -368,21 +379,15 @@ async fn reenqueue(
(repo::page_analysis::ReenqueueScope::All, "analysis", None)
};
// Enqueue + audit in one transaction so a failed audit insert rolls the
// enqueue back too — the audit trail can't silently miss a re-enqueue that
// actually landed. Both are plain DB writes, so they share the tx cleanly
// (the live event + response are emitted only after commit).
let mut tx = state.db.begin().await?;
let enqueued =
repo::page_analysis::enqueue_pages(&state.db, scope, body.only_unanalyzed).await?;
// Push a live event so connected dashboards mark the in-scope pages as
// queued. Skip the no-op (nothing actually enqueued).
if enqueued > 0 {
state.analysis_events.publish(AnalysisEvent::Enqueued {
count: enqueued,
manga_id: body.manga_id,
chapter_id: body.chapter_id,
});
}
repo::page_analysis::enqueue_pages(&mut *tx, scope, body.only_unanalyzed).await?;
repo::admin_audit::insert(
&state.db,
&mut *tx,
admin.0.id,
"analysis_reenqueue",
target_type,
@@ -395,6 +400,18 @@ async fn reenqueue(
}),
)
.await?;
tx.commit().await?;
// Push a live event so connected dashboards mark the in-scope pages as
// queued. Skip the no-op (nothing actually enqueued). After commit so a
// rolled-back enqueue never emits a phantom event.
if enqueued > 0 {
state.analysis_events.publish(AnalysisEvent::Enqueued {
count: enqueued,
manga_id: body.manga_id,
chapter_id: body.chapter_id,
});
}
Ok(Json(ReenqueueResponse { enqueued }))
}
@@ -421,8 +438,13 @@ async fn analyze_page(
// when a `force=false` job was already pending — the worker would
// then pick up the non-force row, hit skip-if-done, ack done, and
// the admin saw "queued for re-analysis" with no re-analysis.
//
// Enqueue + audit in one transaction (via the `_conn` form) so a failed
// audit insert rolls the enqueue/upgrade back too — the audit trail can't
// miss a force-reanalyze that landed.
let mut tx = state.db.begin().await?;
let outcome =
repo::page_analysis::enqueue_for_page(&state.db, page_id, true).await?;
repo::page_analysis::enqueue_for_page_conn(&mut tx, page_id, true).await?;
let outcome_label = match outcome {
repo::page_analysis::EnqueueForPageOutcome::Inserted => "inserted",
@@ -430,7 +452,7 @@ async fn analyze_page(
repo::page_analysis::EnqueueForPageOutcome::AlreadyEnqueued => "already_force_enqueued",
};
repo::admin_audit::insert(
&state.db,
&mut *tx,
admin.0.id,
"analysis_force_page",
"page",
@@ -438,6 +460,7 @@ async fn analyze_page(
json!({ "force": true, "outcome": outcome_label }),
)
.await?;
tx.commit().await?;
Ok(Json(AnalyzePageResponse { enqueued: true }))
}

View File

@@ -205,6 +205,13 @@ async fn backfill(
resp.more_remaining = more_pages || more_covers;
}
// Audit is written after the action here *by necessity*, not oversight
// (unlike reenqueue/analyze_page, which wrap their single DB mutation +
// audit in one tx). The backfill is a budgeted scan that interleaves
// filesystem `storage.size()` reads with per-batch DB writes across many
// iterations; wrapping it in a transaction would hold one open across all
// that I/O (a long-running tx). The batch size-writes are also idempotent
// recomputations, so a lost audit row has negligible impact.
repo::admin_audit::insert(
&state.db,
admin.0.id,

View File

@@ -16,7 +16,7 @@ use crate::api::auth::{validate_password, validate_username};
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::auth::password::hash_password;
use crate::auth::password::hash_password_async;
use crate::domain::User;
use crate::error::{AppError, AppResult};
use crate::repo;
@@ -115,7 +115,7 @@ async fn create_user(
// reject (and vice versa).
validate_username(username)?;
validate_password(&input.password)?;
let pwhash = hash_password(&input.password)?;
let pwhash = hash_password_async(input.password.clone()).await?;
let user = repo::user::admin_create_user(
&state.db,
actor.id,

View File

@@ -17,8 +17,8 @@ use serde::{Deserialize, Serialize};
use uuid::Uuid;
use crate::app::AppState;
use crate::auth::extractor::{CurrentUser, SESSION_COOKIE_NAME};
use crate::auth::password::{hash_password, verify_password};
use crate::auth::extractor::{ClientIp, CurrentUser, SESSION_COOKIE_NAME};
use crate::auth::password::{hash_password_async, verify_password_async};
use crate::auth::token::{generate_token, hash_token};
use crate::config::AuthConfig;
use crate::domain::user_preferences::{READER_GAPS, READER_MODES};
@@ -38,7 +38,7 @@ pub fn routes() -> Router<AppState> {
"/auth/me/preferences",
get(get_preferences).patch(update_preferences),
)
.route("/auth/tokens", post(create_token))
.route("/auth/tokens", get(list_tokens).post(create_token))
.route("/auth/tokens/:id", delete(delete_token))
}
@@ -75,8 +75,16 @@ pub struct AuthResponse {
#[derive(Debug, Deserialize)]
pub struct CreateTokenInput {
pub name: String,
/// Optional lifetime in days. Omit (or `null`) for a non-expiring
/// token (the historical behaviour). When set, must be 1..=3650.
#[serde(default)]
pub expires_in_days: Option<i64>,
}
/// Upper bound on a requested token lifetime (~10 years). A token that needs
/// to outlive this should be rotated, not minted once forever.
const MAX_TOKEN_EXPIRY_DAYS: i64 = 3650;
#[derive(Debug, Deserialize)]
pub struct ChangePassword {
pub current_password: String,
@@ -99,6 +107,7 @@ pub struct CreatedTokenResponse {
async fn register(
State(state): State<AppState>,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<Credentials>,
) -> AppResult<impl IntoResponse> {
@@ -106,7 +115,7 @@ async fn register(
// the toggle can't be probed for the toggle state via timing —
// disabled and enabled paths both consume a token, and disabled
// returns 403 instead of running argon2.
check_auth_rate_limit(&state, "register")?;
check_auth_rate_limit(&state, "register", client_ip)?;
// Private mode force-blocks self-registration regardless of
// ALLOW_SELF_REGISTER — operators of locked-down instances mint
// accounts via `POST /admin/users` instead.
@@ -117,7 +126,7 @@ async fn register(
validate_username(username)?;
validate_password(&input.password)?;
let pwhash = hash_password(&input.password)?;
let pwhash = hash_password_async(input.password.clone()).await?;
let user = repo::user::create(&state.db, username, &pwhash).await?;
let jar = start_session(&state, &user, jar).await?;
Ok((StatusCode::CREATED, jar, Json(AuthResponse { user })))
@@ -125,16 +134,21 @@ async fn register(
async fn login(
State(state): State<AppState>,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<Credentials>,
) -> AppResult<impl IntoResponse> {
check_auth_rate_limit(&state, "login")?;
check_auth_rate_limit(&state, "login", client_ip)?;
let username = input.username.trim();
if username.is_empty() || input.password.is_empty() {
return Err(AppError::InvalidInput(
"username and password are required".into(),
));
}
// Bound the password before argon2 runs — guards BOTH the real-verify and
// the dummy-hash timing-equaliser branch below, so a giant password can't
// make every login attempt a CPU-DoS.
reject_oversized_password(&input.password)?;
let user = repo::user::find_by_username(&state.db, username).await?;
let Some(user) = user else {
@@ -142,10 +156,11 @@ async fn login(
// response time matches the wrong-password branch — otherwise
// an attacker can enumerate usernames by timing the no-user
// 401 against the wrong-password 401.
let _ = verify_password(&input.password, dummy_password_hash());
let _ = verify_password_async(input.password.clone(), dummy_password_hash().to_string())
.await;
return Err(AppError::Unauthenticated);
};
if !verify_password(&input.password, &user.password_hash) {
if !verify_password_async(input.password.clone(), user.password_hash.clone()).await {
return Err(AppError::Unauthenticated);
}
@@ -201,16 +216,20 @@ async fn me(CurrentUser(user): CurrentUser) -> AppResult<Json<AuthResponse>> {
async fn change_password(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
ClientIp(client_ip): ClientIp,
jar: CookieJar,
Json(input): Json<ChangePassword>,
) -> AppResult<impl IntoResponse> {
check_auth_rate_limit(&state, "change_password")?;
if !verify_password(&input.current_password, &user.password_hash) {
check_auth_rate_limit(&state, "change_password", client_ip)?;
// Cap current_password before verify_password runs argon2 (same DoS
// vector as login). new_password is bounded by validate_password below.
reject_oversized_password(&input.current_password)?;
if !verify_password_async(input.current_password.clone(), user.password_hash.clone()).await {
return Err(AppError::Unauthenticated);
}
validate_password(&input.new_password)?;
let new_hash = hash_password(&input.new_password)?;
let new_hash = hash_password_async(input.new_password.clone()).await?;
let mut tx = state.db.begin().await?;
sqlx::query("UPDATE users SET password_hash = $1 WHERE id = $2")
@@ -280,6 +299,22 @@ async fn update_preferences(
Ok(Json(saved))
}
/// `GET /auth/tokens` — the caller's bot tokens (newest first). The raw bearer
/// is only ever shown once at creation, so this list carries just the metadata
/// (name, created/last-used, expiry); `token_hash` is `#[serde(skip)]`.
#[derive(Debug, Serialize)]
struct TokenListResponse {
items: Vec<ApiToken>,
}
async fn list_tokens(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
) -> AppResult<Json<TokenListResponse>> {
let items = repo::api_token::list_for_user(&state.db, user.id).await?;
Ok(Json(TokenListResponse { items }))
}
async fn create_token(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
@@ -305,8 +340,22 @@ async fn create_token(
details: serde_json::json!({ "name": "max 64 characters" }),
});
}
let expires_at = match input.expires_in_days {
None => None,
Some(days) if (1..=MAX_TOKEN_EXPIRY_DAYS).contains(&days) => {
Some(Utc::now() + Duration::days(days))
}
Some(_) => {
return Err(AppError::ValidationFailed {
message: "token expiry out of range".into(),
details: serde_json::json!({
"expires_in_days": format!("must be between 1 and {MAX_TOKEN_EXPIRY_DAYS}")
}),
});
}
};
let (raw, hash) = generate_token();
let token = repo::api_token::create(&state.db, user.id, name, &hash).await?;
let token = repo::api_token::create(&state.db, user.id, name, &hash, expires_at).await?;
Ok((
StatusCode::CREATED,
Json(CreatedTokenResponse { token, bearer: raw }),
@@ -387,9 +436,13 @@ fn build_expired_cookie(cfg: &AuthConfig) -> Cookie<'static> {
/// any one of them in a tight loop should trip the limit. `endpoint`
/// is included in the rate-limit-hit log line so operators can tell
/// which endpoint is being probed.
fn check_auth_rate_limit(state: &AppState, endpoint: &'static str) -> AppResult<()> {
fn check_auth_rate_limit(
state: &AppState,
endpoint: &'static str,
client_ip: Option<std::net::IpAddr>,
) -> AppResult<()> {
use crate::auth::rate_limit::AcquireResult;
match state.auth_limiter.try_acquire() {
match state.auth_limiter.try_acquire(client_ip) {
AcquireResult::Allowed => Ok(()),
AcquireResult::Denied { retry_after_secs } => {
tracing::warn!(
@@ -425,11 +478,67 @@ pub(crate) fn validate_username(u: &str) -> AppResult<()> {
Ok(())
}
/// Upper bound on password length (bytes). argon2 has no inherent length
/// limit, so without a cap an attacker could submit a multi-megabyte password
/// and make every hash a CPU-heavy DoS. 1024 bytes is far longer than any real
/// passphrase. Measured in bytes (like the min check) since that's what argon2
/// actually processes.
pub(crate) const MAX_PASSWORD_BYTES: usize = 1024;
/// Reject an over-cap password *before* any argon2 work runs. The
/// verification paths (login, change-password) don't go through
/// [`validate_password`] — they hash/verify the raw input — so without this
/// an attacker could submit a multi-megabyte password and turn every
/// login/verify into a CPU-DoS. Length-only and independent of whether the
/// account exists, so it leaks nothing (no username enumeration).
pub(crate) fn reject_oversized_password(p: &str) -> AppResult<()> {
if p.len() > MAX_PASSWORD_BYTES {
return Err(AppError::InvalidInput(format!(
"password must be at most {MAX_PASSWORD_BYTES} bytes"
)));
}
Ok(())
}
pub(crate) fn validate_password(p: &str) -> AppResult<()> {
if p.len() < 8 {
return Err(AppError::InvalidInput(
"password must be at least 8 characters".into(),
));
}
if p.len() > MAX_PASSWORD_BYTES {
return Err(AppError::InvalidInput(
format!("password must be at most {MAX_PASSWORD_BYTES} bytes"),
));
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn validate_password_rejects_too_short() {
assert!(validate_password("short").is_err());
assert!(validate_password("1234567").is_err()); // 7 chars
}
#[test]
fn validate_password_accepts_in_range() {
assert!(validate_password("hunter2hunter2").is_ok());
// Exactly at the cap is allowed.
assert!(validate_password(&"a".repeat(MAX_PASSWORD_BYTES)).is_ok());
// Minimum length boundary.
assert!(validate_password("12345678").is_ok());
}
#[test]
fn validate_password_rejects_over_cap() {
// One byte past the cap is refused so a giant password can't turn each
// login/register into an argon2 CPU-DoS.
let too_long = "a".repeat(MAX_PASSWORD_BYTES + 1);
let err = validate_password(&too_long).unwrap_err();
assert!(matches!(err, AppError::InvalidInput(_)));
}
}

View File

@@ -72,7 +72,7 @@ async fn create(
}
}
let bookmark = repo::bookmark::create(
let (bookmark, created) = repo::bookmark::create(
&state.db,
user.id,
input.manga_id,
@@ -103,7 +103,13 @@ async fn create(
}
});
Ok((StatusCode::CREATED, Json(bookmark)))
// 201 for a fresh bookmark, 200 when it already existed (idempotent add).
let status = if created {
StatusCode::CREATED
} else {
StatusCode::OK
};
Ok((status, Json(bookmark)))
}
async fn delete_one(

View File

@@ -13,7 +13,7 @@ use serde::Deserialize;
use serde_json::json;
use uuid::Uuid;
use crate::api::mangas::{next_field, read_field_bytes};
use crate::api::mangas::next_field;
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::CurrentUser;
@@ -21,7 +21,8 @@ use crate::domain::chapter::NewChapter;
use crate::domain::{Chapter, Page};
use crate::error::{AppError, AppResult};
use crate::repo;
use crate::upload::{parse_image, UploadedImage};
use crate::storage::Storage;
use crate::upload::{stage_image_part, StagedImage};
pub fn routes() -> Router<AppState> {
Router::new()
@@ -69,6 +70,21 @@ async fn get_one(
Ok(Json(chapter))
}
/// Add a chapter to a manga.
///
/// **Authorization is intentionally open**: any authenticated principal —
/// a browser session *or* a bot API token — may add a chapter to *any*
/// manga, including crawler-imported rows. This is by design: chapters are
/// treated as community contributions, unlike the manga record itself
/// (title/cover/metadata), whose edits gate through `require_can_edit`
/// (see [`crate::api::mangas`]). The `CurrentUser` binding still requires a
/// valid identity, so contributions are attributable, just not owner-scoped.
///
/// This contract is locked by `tests/api_chapters.rs`
/// (`non_owner_can_upload_chapter`); changing it to owner-only is a
/// deliberate decision, not a drive-by tightening. Until a richer
/// contributor/moderation model lands this is acknowledged, intended
/// behaviour — see the auth note in CLAUDE.md.
async fn create(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
@@ -77,97 +93,178 @@ async fn create(
) -> AppResult<(StatusCode, Json<Chapter>)> {
repo::manga::get(&state.db, manga_id).await?;
// Each `page` part is streamed straight to a staging key as it arrives,
// so at most one page's bytes sit in memory — the whole chapter is never
// buffered (previously every page was held at once, bounded only by the
// 200 MiB body limit and amplified by concurrency). The staged blobs are
// promoted to their final chapter-scoped keys in `finalize_chapter` once
// the chapter id exists; any early exit cleans them up.
let upload_id = Uuid::new_v4();
let mut metadata: Option<NewChapter> = None;
let mut pages: Vec<UploadedImage> = Vec::new();
let mut staged: Vec<StagedImage> = Vec::new();
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = read_field_bytes(field).await?;
metadata =
Some(serde_json::from_slice(&bytes).map_err(|e| {
let stage_result: AppResult<NewChapter> = async {
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = crate::upload::read_capped(
field,
crate::upload::MAX_METADATA_BYTES,
"metadata",
)
.await?;
metadata = Some(serde_json::from_slice(&bytes).map_err(|e| {
AppError::ValidationFailed {
message: "metadata is not valid JSON".into(),
details: json!({ "metadata": e.to_string() }),
}
})?);
}
Some("page") => {
if state.upload.max_pages_per_chapter != 0
&& staged.len() >= state.upload.max_pages_per_chapter
{
return Err(AppError::PayloadTooLarge(format!(
"chapter exceeds the {}-page limit",
state.upload.max_pages_per_chapter
)));
}
let field_name = format!("page[{}]", staged.len());
let img = stage_image_part(
state.storage.as_ref(),
field,
upload_id,
staged.len(),
state.upload.max_file_bytes,
&field_name,
)
.await?;
staged.push(img);
}
_ => continue,
}
Some("page") => {
let bytes = read_field_bytes(field).await?.to_vec();
let field_name = format!("page[{}]", pages.len());
pages.push(parse_image(bytes, state.upload.max_file_bytes, &field_name)?);
}
_ => continue,
}
}
let metadata = metadata.ok_or_else(|| AppError::ValidationFailed {
message: "metadata part is required".into(),
details: json!({ "metadata": "required" }),
})?;
// Chapter number is 1-indexed everywhere (URLs, upload form,
// reader). Reject 0 / negative numbers up front so the row never
// makes it into the DB. Mirrors the page>=1 rule on bookmarks.
if metadata.number < 1 {
return Err(AppError::ValidationFailed {
message: "chapter number must be 1 or greater".into(),
details: json!({ "number": "must be >= 1" }),
});
}
if pages.is_empty() {
return Err(AppError::ValidationFailed {
message: "at least one page is required".into(),
details: json!({ "page": "at least one required" }),
});
let metadata = metadata.take().ok_or_else(|| AppError::ValidationFailed {
message: "metadata part is required".into(),
details: json!({ "metadata": "required" }),
})?;
// Chapter number is 1-indexed everywhere (URLs, upload form,
// reader). Reject 0 / negative numbers up front so the row never
// makes it into the DB. Mirrors the page>=1 rule on bookmarks.
if metadata.number < 1 {
return Err(AppError::ValidationFailed {
message: "chapter number must be 1 or greater".into(),
details: json!({ "number": "must be >= 1" }),
});
}
if staged.is_empty() {
return Err(AppError::ValidationFailed {
message: "at least one page is required".into(),
details: json!({ "page": "at least one required" }),
});
}
Ok(metadata)
}
.await;
// Transactional create. If any storage put or page-row insert
// fails mid-loop, the chapter row + any earlier page rows are
// rolled back so we don't leave a chapter with stale page_count=0
// and orphaned page rows. Bytes already written to storage on a
// rolled-back transaction become orphans on disk; a future reaper
// can sweep them. DB consistency wins over storage tidiness here.
let mut tx = state.db.begin().await?;
let mut chapter = repo::chapter::create(
let metadata = match stage_result {
Ok(m) => m,
Err(e) => {
// Reject before any DB write — remove every page we staged.
cleanup_staging(state.storage.as_ref(), &staged).await;
return Err(e);
}
};
finalize_chapter(&state, manga_id, user.id, &metadata, &staged).await
}
/// Promote the staged pages into a real chapter, all-or-nothing. Creates the
/// chapter row, renames each staged blob to its final chapter-scoped key,
/// and inserts the page rows in one transaction. On any failure the DB rolls
/// back and every blob (already-finalized and still-staged) is removed, so a
/// rejected upload leaves neither partial rows nor orphaned files.
async fn finalize_chapter(
state: &AppState,
manga_id: Uuid,
user_id: Uuid,
metadata: &NewChapter,
staged: &[StagedImage],
) -> AppResult<(StatusCode, Json<Chapter>)> {
let storage = state.storage.as_ref();
let mut tx = match state.db.begin().await {
Ok(tx) => tx,
Err(e) => {
cleanup_staging(storage, staged).await;
return Err(e.into());
}
};
let mut chapter = match repo::chapter::create(
&mut *tx,
manga_id,
metadata.number,
metadata.title.as_deref(),
Some(user.id),
Some(user_id),
)
.await?;
.await
{
Ok(c) => c,
Err(e) => {
cleanup_staging(storage, staged).await;
return Err(e);
}
};
let mut page_ids: Vec<Uuid> = Vec::with_capacity(pages.len());
for (idx, page) in pages.iter().enumerate() {
let mut page_ids: Vec<Uuid> = Vec::with_capacity(staged.len());
let mut finalized: Vec<String> = Vec::with_capacity(staged.len());
for (idx, page) in staged.iter().enumerate() {
let page_number = (idx + 1) as i32;
let nnnn = format!("{:04}", page_number);
let key = format!(
"mangas/{}/chapters/{}/pages/{}.{}",
manga_id, chapter.id, nnnn, page.ext
let final_key = format!(
"mangas/{}/chapters/{}/pages/{:04}.{}",
manga_id, chapter.id, page_number, page.ext
);
state.storage.put(&key, &page.bytes).await?;
let created = repo::page::create(
if let Err(e) = storage.rename(&page.staging_key, &final_key).await {
cleanup_keys(storage, &finalized).await;
cleanup_staging(storage, &staged[idx..]).await;
return Err(e.into());
}
finalized.push(final_key.clone());
match repo::page::create(
&mut *tx,
chapter.id,
page_number,
&key,
&final_key,
page.mime,
page.bytes.len() as i64,
page.size_bytes,
)
.await?;
page_ids.push(created.id);
.await
{
Ok(created) => page_ids.push(created.id),
Err(e) => {
cleanup_keys(storage, &finalized).await;
cleanup_staging(storage, &staged[idx + 1..]).await;
return Err(e);
}
}
}
let page_count = pages.len() as i32;
repo::chapter::set_page_count(&mut *tx, chapter.id, page_count).await?;
let page_count = staged.len() as i32;
if let Err(e) = repo::chapter::set_page_count(&mut *tx, chapter.id, page_count).await {
cleanup_keys(storage, &finalized).await;
return Err(e);
}
chapter.page_count = page_count;
// `repo::chapter::create` returned the row before any pages existed, so
// its `size_bytes` is a stale 0. Every uploaded page's size was just
// captured, so the true total is the sum of their byte lengths — set it
// on the response so the 201 body matches the persisted state.
chapter.size_bytes = Some(pages.iter().map(|p| p.bytes.len() as i64).sum());
// its `size_bytes` is a stale 0. Each staged page carried its byte
// length, so their sum is the chapter's true storage — set it on the
// response so the 201 body matches the persisted state.
chapter.size_bytes = Some(staged.iter().map(|p| p.size_bytes).sum());
tx.commit().await?;
if let Err(e) = tx.commit().await {
cleanup_keys(storage, &finalized).await;
return Err(e.into());
}
// Enqueue AI content-analysis for each new page. Done after commit so a
// rolled-back upload never leaves jobs pointing at nonexistent pages; a
@@ -175,9 +272,7 @@ async fn create(
// re-enqueue endpoint can backfill).
if state.analysis_enabled() {
for page_id in page_ids {
if let Err(e) =
repo::page_analysis::enqueue_for_page(&state.db, page_id, false).await
{
if let Err(e) = repo::page_analysis::enqueue_for_page(&state.db, page_id, false).await {
tracing::warn!(%page_id, error = %e, "failed to enqueue page analysis");
}
}
@@ -186,6 +281,21 @@ async fn create(
Ok((StatusCode::CREATED, Json(chapter)))
}
/// Best-effort removal of staged page blobs after a rejected upload.
async fn cleanup_staging(storage: &dyn Storage, staged: &[StagedImage]) {
for page in staged {
let _ = storage.delete(&page.staging_key).await;
}
}
/// Best-effort removal of already-finalized page blobs when the upload
/// fails after some renames have landed.
async fn cleanup_keys(storage: &dyn Storage, keys: &[String]) {
for key in keys {
let _ = storage.delete(key).await;
}
}
#[derive(Debug, serde::Serialize)]
struct PagesResponse {
pages: Vec<Page>,

View File

@@ -5,6 +5,13 @@
//! The handler uses `Storage::get_stream` so a multi-MB page is piped to
//! the client a chunk at a time instead of buffered server-side.
//!
//! **Thumbnails.** `?w=<px>` serves a width-bounded variant (grids ask for a
//! small width so they download ~KB instead of the 15 MB original). The width
//! snaps to a small allow-list so the number of cached derivatives stays
//! bounded; the resized image is cached in storage under a `thumbs/w{W}/` prefix
//! and regenerated on demand. Only JPEG/PNG sources are thumbnailed (encoders we
//! ship); other formats fall back to the original.
//!
//! **Auth model — capability URLs by design.** This endpoint is
//! deliberately unauthenticated: reads stay public per the project
//! brief, and per-page authorisation would require either a per-request
@@ -16,41 +23,250 @@
//! would gate this endpoint behind a `Storage::owner_of(key)` check;
//! the seam is intentional.
use std::io::Cursor;
use axum::body::Body;
use axum::extract::{Path, State};
use axum::extract::{Path, Query, State};
use axum::http::{header, HeaderName};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::Router;
use image::imageops::FilterType;
use image::{ImageFormat, ImageReader};
use serde::Deserialize;
use crate::app::AppState;
use crate::error::AppResult;
use crate::storage::StorageError;
use crate::error::{AppError, AppResult};
use crate::storage::{Storage, StorageError};
/// Widths a thumbnail may be rendered at. A requested width snaps up to the
/// smallest of these so the set of cached derivatives stays tiny.
const ALLOWED_THUMB_WIDTHS: &[u32] = &[160, 320, 480, 640, 960];
/// Storage key prefix for cached thumbnails.
const THUMB_PREFIX: &str = "thumbs";
/// Decode allocation cap (mirrors `analysis::ocr`): a tiny file declaring huge
/// dimensions is rejected before the decoder allocates, not after (OOM guard).
const MAX_THUMB_DECODE_PIXELS: u64 = 40_000_000;
pub fn routes() -> Router<AppState> {
Router::new().route("/files/*key", get(serve))
}
async fn serve(State(state): State<AppState>, Path(key): Path<String>) -> AppResult<Response> {
let file = match state.storage.get_stream(&key).await {
#[derive(Debug, Deserialize)]
struct ServeQuery {
/// Requested thumbnail width in pixels; absent = serve the original.
w: Option<String>,
}
async fn serve(
State(state): State<AppState>,
Path(key): Path<String>,
Query(q): Query<ServeQuery>,
) -> AppResult<Response> {
// Thumbnail request: only for source formats we can re-encode; anything
// else falls through to serving the original.
if let Some(width) = resolve_thumb_width(q.w.as_deref()) {
if let Some(fmt) = thumb_format_for(&key) {
return serve_thumbnail(&state, &key, width, fmt).await;
}
}
serve_original(&state, &key).await
}
async fn serve_original(state: &AppState, key: &str) -> AppResult<Response> {
let file = match state.storage.get_stream(key).await {
Ok(f) => f,
Err(StorageError::NotFound) => return Err(crate::error::AppError::NotFound),
Err(StorageError::NotFound) => return Err(AppError::NotFound),
Err(e) => return Err(e.into()),
};
let ct = content_type_for(&key);
// `nosniff` makes the contract explicit: the browser must trust the
// Content-Type we declared (and that the magic-byte sniff at upload
// time produced) instead of trying to detect HTML/JS in the body.
// Belt-and-braces vs. polyglot files that survive the upload sniff.
let headers = [
(header::CONTENT_TYPE, ct.to_string()),
(header::CONTENT_LENGTH, file.size_bytes.to_string()),
Ok(image_response(
state,
content_type_for(key),
file.size_bytes.to_string(),
Body::from_stream(file.stream),
))
}
async fn serve_thumbnail(
state: &AppState,
key: &str,
width: u32,
fmt: ImageFormat,
) -> AppResult<Response> {
let derived = thumb_key(key, width);
// Serve the cached variant if it exists.
match state.storage.get_stream(&derived).await {
Ok(f) => {
return Ok(image_response(
state,
content_type_for(key),
f.size_bytes.to_string(),
Body::from_stream(f.stream),
));
}
Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
// Generate from the original.
let original = match state.storage.get(key).await {
Ok(b) => b,
Err(StorageError::NotFound) => return Err(AppError::NotFound),
Err(e) => return Err(e.into()),
};
// Resizing is CPU-bound; keep it off the async worker threads.
let thumb = match tokio::task::spawn_blocking(move || make_thumbnail(&original, width, fmt))
.await
.map_err(|e| AppError::Other(anyhow::anyhow!("thumbnail task join: {e}")))?
{
Ok(t) => t,
Err(e) => {
// The blob's magic bytes passed upload's sniff but the body won't
// decode (corrupt/truncated, or an encoder feature we don't support).
// Don't 500 the reader — fall back to streaming the original, which
// serves without decoding.
tracing::warn!(key, error = %format!("{e:#}"), "thumbnail decode failed; serving original");
return serve_original(state, key).await;
}
};
// Best-effort cache; a write failure just means we regenerate next time.
let _ = state.storage.put(&derived, &thumb).await;
let len = thumb.len().to_string();
Ok(image_response(
state,
content_type_for(key),
len,
Body::from(thumb),
))
}
/// Shared response builder for both the original and thumbnail paths.
fn image_response(
state: &AppState,
content_type: &str,
content_length: String,
body: Body,
) -> Response {
let mut headers = vec![
(header::CONTENT_TYPE, content_type.to_string()),
(header::CONTENT_LENGTH, content_length),
// `nosniff` makes the contract explicit: the browser must trust the
// Content-Type we declared (and that the magic-byte sniff at upload
// time produced) instead of trying to detect HTML/JS in the body.
(
HeaderName::from_static("x-content-type-options"),
"nosniff".to_string(),
),
// Blobs are content-addressed by unguessable, immutable keys (a
// re-upload mints new UUIDs), so a fetched page/cover never changes.
// Cache it for a year and mark it `immutable` so browsers skip
// revalidation entirely.
//
// BUT under PRIVATE_MODE these blobs are auth-gated, so they must NOT be
// marked `public`: a shared cache / CDN would store the object and serve
// it to unauthenticated clients. Use `private` so only the requesting
// user's browser caches it.
(
header::CACHE_CONTROL,
if state.auth.private_mode {
"private, max-age=31536000, immutable".to_string()
} else {
"public, max-age=31536000, immutable".to_string()
},
),
];
Ok((headers, Body::from_stream(file.stream)).into_response())
// Known image types render inline (covers/pages display in the reader). The
// `application/octet-stream` fallback is a blob we couldn't type — it could
// be crafted HTML/JS, so force a download rather than let the browser render
// it inline. Belt to `nosniff`'s braces.
if content_type == "application/octet-stream" {
headers.push((
header::CONTENT_DISPOSITION,
"attachment".to_string(),
));
}
(axum::response::AppendHeaders(headers), body).into_response()
}
/// Parse and clamp a requested thumbnail width. Returns `None` for absent /
/// unparseable / zero widths (serve the original); otherwise snaps the request
/// up to the smallest allowed width (capped at the largest) so cached variants
/// stay bounded.
fn resolve_thumb_width(raw: Option<&str>) -> Option<u32> {
let requested: u32 = raw?.trim().parse().ok()?;
if requested == 0 {
return None;
}
Some(
ALLOWED_THUMB_WIDTHS
.iter()
.copied()
.find(|&w| w >= requested)
.unwrap_or_else(|| *ALLOWED_THUMB_WIDTHS.last().expect("non-empty")),
)
}
/// The re-encode format for a source key, or `None` when it isn't one we ship an
/// encoder for (gif/avif → serve the original instead of a broken thumbnail).
fn thumb_format_for(key: &str) -> Option<ImageFormat> {
match content_type_for(key) {
"image/jpeg" => Some(ImageFormat::Jpeg),
"image/png" => Some(ImageFormat::Png),
_ => None,
}
}
/// The storage key a width-`w` thumbnail of `key` is cached under.
fn thumb_key(key: &str, width: u32) -> String {
format!("{THUMB_PREFIX}/w{width}/{key}")
}
/// Every cached thumbnail key for an original `key`, across all allowed widths.
/// Used by the cover handlers to purge stale variants when a cover (whose key is
/// reused, unlike content-addressed pages) is replaced or deleted.
pub(crate) fn thumbnail_keys(key: &str) -> Vec<String> {
ALLOWED_THUMB_WIDTHS
.iter()
.map(|&w| thumb_key(key, w))
.collect()
}
/// Best-effort deletion of every cached thumbnail for `key`. Call after the
/// underlying blob at `key` changes or is removed.
pub(crate) async fn purge_thumbnails(storage: &dyn Storage, key: &str) {
for derived in thumbnail_keys(key) {
let _ = storage.delete(&derived).await;
}
}
/// Decode, downscale to `width` (aspect-preserving, never upscaling), and
/// re-encode in `fmt`. Pure + synchronous so it runs under `spawn_blocking` and
/// is unit-testable without a server.
fn make_thumbnail(bytes: &[u8], width: u32, fmt: ImageFormat) -> anyhow::Result<Vec<u8>> {
use anyhow::Context;
let mut reader = ImageReader::new(Cursor::new(bytes))
.with_guessed_format()
.context("guess image format for thumbnail")?;
let mut limits = image::Limits::default();
limits.max_alloc = Some(MAX_THUMB_DECODE_PIXELS.saturating_mul(4));
reader.limits(limits);
let img = reader.decode().context("decode image for thumbnail")?;
// Only downscale; a source narrower than the target is served as-is.
let out = if img.width() > width {
img.resize(width, u32::MAX, FilterType::Lanczos3)
} else {
img
};
let mut buf = Cursor::new(Vec::new());
out.write_to(&mut buf, fmt).context("encode thumbnail")?;
Ok(buf.into_inner())
}
fn content_type_for(key: &str) -> &'static str {
@@ -64,3 +280,68 @@ fn content_type_for(key: &str) -> &'static str {
_ => "application/octet-stream",
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolve_thumb_width_snaps_and_validates() {
// Absent / unparseable / zero → serve the original.
assert_eq!(resolve_thumb_width(None), None);
assert_eq!(resolve_thumb_width(Some("")), None);
assert_eq!(resolve_thumb_width(Some("abc")), None);
assert_eq!(resolve_thumb_width(Some("0")), None);
// Snap up to the smallest allowed width.
assert_eq!(resolve_thumb_width(Some("1")), Some(160));
assert_eq!(resolve_thumb_width(Some("160")), Some(160));
assert_eq!(resolve_thumb_width(Some("161")), Some(320));
assert_eq!(resolve_thumb_width(Some("640")), Some(640));
// Above the max → capped at the largest allowed width.
assert_eq!(resolve_thumb_width(Some("5000")), Some(960));
}
#[test]
fn thumb_format_only_for_encodable_sources() {
assert_eq!(thumb_format_for("a/b/cover.jpg"), Some(ImageFormat::Jpeg));
assert_eq!(thumb_format_for("a/b/cover.png"), Some(ImageFormat::Png));
// Formats we can't re-encode fall back to the original.
assert_eq!(thumb_format_for("a/b/cover.webp"), None);
assert_eq!(thumb_format_for("a/b/cover.gif"), None);
assert_eq!(thumb_format_for("a/b/cover.avif"), None);
}
#[test]
fn thumb_key_is_prefixed_by_width() {
assert_eq!(
thumb_key("mangas/x/cover.png", 320),
"thumbs/w320/mangas/x/cover.png"
);
assert_eq!(thumbnail_keys("k.png").len(), ALLOWED_THUMB_WIDTHS.len());
}
#[test]
fn make_thumbnail_downscales_and_preserves_aspect() {
// 100x50 red PNG → thumbnail width 40 → 40x20, still decodable PNG.
let mut src = Cursor::new(Vec::new());
image::RgbImage::from_pixel(100, 50, image::Rgb([255, 0, 0]))
.write_to(&mut src, ImageFormat::Png)
.unwrap();
let out = make_thumbnail(src.get_ref(), 40, ImageFormat::Png).unwrap();
let decoded = image::load_from_memory(&out).unwrap();
assert_eq!(decoded.width(), 40);
assert_eq!(decoded.height(), 20, "aspect ratio preserved");
}
#[test]
fn make_thumbnail_does_not_upscale() {
// A 30px-wide source requested at 40 stays 30 wide (no upscaling).
let mut src = Cursor::new(Vec::new());
image::RgbImage::from_pixel(30, 30, image::Rgb([0, 255, 0]))
.write_to(&mut src, ImageFormat::Png)
.unwrap();
let out = make_thumbnail(src.get_ref(), 40, ImageFormat::Png).unwrap();
let decoded = image::load_from_memory(&out).unwrap();
assert_eq!(decoded.width(), 30);
}
}

View File

@@ -22,6 +22,7 @@ pub fn routes() -> Router<AppState> {
.route("/mangas", get(list).post(create))
.route("/mangas/:id", get(get_one).patch(update))
.route("/mangas/:id/similar", get(list_similar))
.route("/me/recommendations", get(list_recommendations))
.route("/mangas/:id/cover", put(put_cover).delete(delete_cover))
.route("/mangas/:id/tags", post(attach_tag))
.route("/mangas/:id/tags/:tag_id", delete(detach_tag))
@@ -156,7 +157,9 @@ async fn list(
order,
};
let (items, total) = repo::manga::list_cards(&state.db, &q).await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
Ok(Json(PagedResponse::with_optional_total(
items, limit, offset, total,
)))
}
async fn get_one(
@@ -188,6 +191,28 @@ async fn list_similar(
Ok(Json(json!({ "items": items })))
}
const RECOMMENDATIONS_LIMIT: i64 = 12;
#[derive(Debug, Deserialize)]
pub struct RecommendationParams {
#[serde(default)]
pub limit: Option<i64>,
}
/// `GET /api/v1/me/recommendations` — content-based "Recommended for you",
/// ranked by tag overlap with the signed-in user's likes/bookmarks (minus
/// dislikes). Returns `{ "items": [...] }` (a fixed top-N, like `/similar`);
/// empty when the user has no taste signals yet.
async fn list_recommendations(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Query(params): Query<RecommendationParams>,
) -> AppResult<Json<serde_json::Value>> {
let limit = params.limit.unwrap_or(RECOMMENDATIONS_LIMIT).clamp(1, 50);
let items = repo::manga::list_recommendations(&state.db, user.id, limit).await?;
Ok(Json(json!({ "items": items })))
}
/// `POST /api/v1/mangas` is multipart/form-data. Parts:
///
/// - `metadata` (required): JSON body matching `NewManga` — title, optional
@@ -210,11 +235,17 @@ async fn create(
while let Some(field) = next_field(&mut multipart).await? {
match field.name() {
Some("metadata") => {
let bytes = read_field_bytes(field).await?;
let bytes = crate::upload::read_capped(
field,
crate::upload::MAX_METADATA_BYTES,
"metadata",
)
.await?;
metadata = Some(parse_metadata_json(&bytes)?);
}
Some("cover") => {
let bytes = read_field_bytes(field).await?.to_vec();
let bytes =
crate::upload::read_capped(field, state.upload.max_file_bytes, "cover").await?;
cover = Some(parse_image(bytes, state.upload.max_file_bytes, "cover")?);
}
_ => continue,
@@ -255,8 +286,8 @@ async fn create(
)
.await?;
let author_refs = repo::author::set_for_manga(&mut *tx, manga.id, &authors).await?;
repo::genre::set_for_manga(&mut *tx, manga.id, &metadata.genre_ids).await?;
let author_refs = repo::author::set_for_manga(&mut tx, manga.id, &authors).await?;
repo::genre::set_for_manga(&mut tx, manga.id, &metadata.genre_ids).await?;
if let Some(img) = cover {
let key = format!("mangas/{}/cover.{}", manga.id, img.ext);
@@ -321,7 +352,7 @@ async fn update(
let mut tx = state.db.begin().await?;
let _updated = repo::manga::update_basics(
&mut *tx,
&mut tx,
id,
patch.title.as_deref().map(str::trim),
patch.status.as_deref().map(str::trim),
@@ -331,10 +362,10 @@ async fn update(
)
.await?;
if let Some(ref names) = authors_owned {
repo::author::set_for_manga(&mut *tx, id, names).await?;
repo::author::set_for_manga(&mut tx, id, names).await?;
}
if let Some(ref ids) = patch.genre_ids {
repo::genre::set_for_manga(&mut *tx, id, ids).await?;
repo::genre::set_for_manga(&mut tx, id, ids).await?;
}
tx.commit().await?;
@@ -362,7 +393,8 @@ async fn put_cover(
let mut cover: Option<UploadedImage> = None;
while let Some(field) = next_field(&mut multipart).await? {
if field.name() == Some("cover") {
let bytes = read_field_bytes(field).await?.to_vec();
let bytes =
crate::upload::read_capped(field, state.upload.max_file_bytes, "cover").await?;
cover = Some(parse_image(bytes, state.upload.max_file_bytes, "cover")?);
}
}
@@ -377,6 +409,17 @@ async fn put_cover(
let old_key = repo::manga::get(&state.db, id).await?.cover_image_path;
let new_key = format!("mangas/{}/cover.{}", id, img.ext);
state.storage.put(&new_key, &img.bytes).await?;
// Cover keys are reused (mangas/{id}/cover.{ext}), so a same-extension
// replacement overwrites the blob at an existing key — drop any thumbnails
// cached for it so we don't serve a stale variant of the old cover.
crate::api::files::purge_thumbnails(state.storage.as_ref(), &new_key).await;
// Commit the DB pointer to the new blob BEFORE removing the old one. If we
// deleted the old blob first and this write then failed, the row would point
// at a deleted blob (a cover that 404s) and the new blob would be orphaned.
// Ordering it first means a failure here leaves the row pointing at the
// still-present old blob; only a harmless orphan (new blob) can result.
repo::manga::set_cover_image_path(&state.db, id, &new_key, img.bytes.len() as i64).await?;
if let Some(prev) = old_key.as_deref() {
if prev != new_key {
@@ -387,10 +430,10 @@ async fn put_cover(
Ok(()) | Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
// Old key's cached thumbnails are now orphaned.
crate::api::files::purge_thumbnails(state.storage.as_ref(), prev).await;
}
}
repo::manga::set_cover_image_path(&state.db, id, &new_key, img.bytes.len() as i64).await?;
Ok(Json(repo::manga::get_detail(&state.db, id).await?))
}
@@ -408,11 +451,15 @@ async fn delete_cover(
}
require_can_edit(&state, id, user.id, admin_via_session(&session)).await?;
if let Some(key) = repo::manga::get(&state.db, id).await?.cover_image_path {
// Clear the DB pointer BEFORE deleting the blob, so a storage-delete
// failure can't leave the row pointing at a removed blob. A failure
// after the clear only orphans the blob (harmless).
repo::manga::clear_cover_image_path(&state.db, id).await?;
match state.storage.delete(&key).await {
Ok(()) | Err(StorageError::NotFound) => {}
Err(e) => return Err(e.into()),
}
repo::manga::clear_cover_image_path(&state.db, id).await?;
crate::api::files::purge_thumbnails(state.storage.as_ref(), &key).await;
}
Ok(Json(repo::manga::get_detail(&state.db, id).await?))
}
@@ -643,13 +690,7 @@ pub(crate) async fn next_field(
.map_err(map_multipart_error)
}
pub(crate) async fn read_field_bytes(
field: axum::extract::multipart::Field<'_>,
) -> AppResult<axum::body::Bytes> {
field.bytes().await.map_err(map_multipart_error)
}
fn map_multipart_error(e: axum::extract::multipart::MultipartError) -> AppError {
pub(crate) fn map_multipart_error(e: axum::extract::multipart::MultipartError) -> AppError {
let status = e.status();
if status == StatusCode::PAYLOAD_TOO_LARGE {
AppError::PayloadTooLarge("upload exceeds the request size limit".into())

View File

@@ -11,6 +11,7 @@ pub mod history;
pub mod mangas;
pub mod page_tags;
pub mod pagination;
pub mod reactions;
pub mod tags;
use axum::Router;
@@ -31,5 +32,6 @@ pub fn routes() -> Router<AppState> {
.merge(collections::routes())
.merge(page_tags::routes())
.merge(history::routes())
.merge(reactions::routes())
.merge(admin::routes())
}

View File

@@ -34,4 +34,18 @@ impl<T> PagedResponse<T> {
page: PageInfo { limit, offset, total: Some(total) },
}
}
/// For handlers that compute `total` only on some pages (e.g. the first)
/// and leave it `None` elsewhere.
pub fn with_optional_total(
items: Vec<T>,
limit: i64,
offset: i64,
total: Option<i64>,
) -> Self {
Self {
items,
page: PageInfo { limit, offset, total },
}
}
}

View File

@@ -0,0 +1,69 @@
//! Manga reactions (like/dislike) — a private, per-user taste signal.
//! Writes require auth; the read is scoped under `/me/` so the URL can't be
//! used to peek at another user's reactions.
use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::routing::{get, put};
use axum::{Json, Router};
use serde::Deserialize;
use serde_json::json;
use uuid::Uuid;
use crate::app::AppState;
use crate::auth::extractor::CurrentUser;
use crate::domain::reaction::{MangaReaction, Reaction};
use crate::error::{AppError, AppResult};
use crate::repo;
pub fn routes() -> Router<AppState> {
Router::new()
.route(
"/mangas/:id/reaction",
put(set_reaction).delete(clear_reaction),
)
.route("/me/reactions/:manga_id", get(get_reaction))
}
#[derive(Debug, Deserialize)]
pub struct SetReactionBody {
pub reaction: String,
}
async fn set_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
Json(body): Json<SetReactionBody>,
) -> AppResult<Json<MangaReaction>> {
// Validate against the closed vocabulary here so a bad value is a clean
// 422 rather than relying on the DB CHECK to surface as a 500.
let reaction = Reaction::parse(&body.reaction).ok_or_else(|| AppError::ValidationFailed {
message: "reaction must be 'like' or 'dislike'".into(),
details: json!({ "reaction": "must be 'like' or 'dislike'" }),
})?;
// Unknown manga → 404 via the FK-violation mapping in repo::reaction.
repo::reaction::upsert(&state.db, user.id, manga_id, reaction).await?;
Ok(Json(MangaReaction {
manga_id,
reaction: Some(reaction),
}))
}
async fn clear_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
) -> AppResult<StatusCode> {
repo::reaction::clear(&state.db, user.id, manga_id).await?;
Ok(StatusCode::NO_CONTENT)
}
async fn get_reaction(
State(state): State<AppState>,
CurrentUser(user): CurrentUser,
Path(manga_id): Path<Uuid>,
) -> AppResult<Json<MangaReaction>> {
let reaction = repo::reaction::get(&state.db, user.id, manga_id).await?;
Ok(Json(MangaReaction { manga_id, reaction }))
}

View File

@@ -277,9 +277,15 @@ impl DaemonReloader for Supervisors {
}
}
/// How often the background reaper sweeps expired sessions. Hourly is ample:
/// the sweep is a single indexed DELETE and expired rows are already invisible
/// to auth, so this is purely storage hygiene.
const SESSION_GC_INTERVAL: std::time::Duration = std::time::Duration::from_secs(3600);
pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
let db = PgPoolOptions::new()
.max_connections(10)
.max_connections(config.db.max_connections)
.acquire_timeout(config.db.acquire_timeout)
.connect(&config.database_url)
.await?;
sqlx::migrate!("./migrations").run(&db).await?;
@@ -336,6 +342,27 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
tracing::info!("analysis worker disabled");
}
// Periodic reaper for lapsed sessions. `find_active` already ignores
// expired rows, so this only reclaims storage — without it the table grows
// unbounded as sessions lapse. Detached and best-effort: a failed sweep is
// logged and retried next tick. Runs regardless of crawler/analysis config
// since sessions exist in every deployment.
{
let db = db.clone();
tokio::spawn(async move {
let mut ticker = tokio::time::interval(SESSION_GC_INTERVAL);
ticker.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
loop {
ticker.tick().await;
match repo::session::delete_expired(&db).await {
Ok(0) => {}
Ok(n) => tracing::info!(reaped = n, "session gc: removed expired sessions"),
Err(e) => tracing::warn!(?e, "session gc sweep failed; retrying next tick"),
}
}
});
}
let auth_limiter = Arc::new(AuthRateLimiter::new(config.auth.rate_limit));
let state = AppState {
db,
@@ -434,19 +461,28 @@ async fn spawn_analysis_daemon(
let (dispatcher, readiness): (
Arc<dyn crate::analysis::daemon::AnalyzeDispatcher>,
Option<Arc<dyn crate::analysis::daemon::VisionReadiness>>,
) = match cfg.backend {
) = match cfg.effective_backend() {
crate::config::AnalysisBackend::Ocr => {
// Load the `.rten` models once; a bad path is a loud boot error.
let engine = crate::analysis::ocr::OcrsEngine::from_model_paths(
&cfg.ocr_detection_model,
&cfg.ocr_recognition_model,
cfg.ocr_max_decode_pixels,
)
.context("build ocrs engine")?;
// Cap concurrent CPU-bound OCR runs across all workers so a high
// ANALYSIS_WORKERS can't oversubscribe the blocking pool.
let cores = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1);
let permits =
crate::analysis::ocr::ocr_concurrency_limit(cfg.workers, cores);
let dispatcher = Arc::new(crate::analysis::ocr::OcrAnalyzeDispatcher {
db: db.clone(),
storage,
engine: Arc::new(engine),
max_image_bytes: cfg.max_image_bytes,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(permits)),
});
// In-process engine is always ready — no gate.
(dispatcher, None)
@@ -460,6 +496,18 @@ async fn spawn_analysis_daemon(
// both. Mirrors the crawler client's `.no_proxy()` (see
// `spawn_crawler_daemon`).
.no_proxy()
// Re-validate redirect hops so a hostile/compromised vision
// endpoint can't 302 the bearer token + page bytes into the
// deployment's internal network. No allowlist here (the
// endpoint is a single admin-configured URL), so the policy
// enforces scheme + private-IP only.
.redirect(crate::crawler::safety::public_redirect_policy())
// NOTE: deliberately no `.dns_resolver(safe_dns_resolver())`.
// The vision endpoint is a single operator-configured internal
// service (e.g. `mangalord-vision`) that legitimately resolves
// to a private Docker IP; a private-IP-rejecting resolver drops
// every call. Redirect hops stay guarded by the policy above,
// and the URL is operator-set, not attacker-controlled input.
.build()
.context("build analysis http client")?;
let vision = crate::analysis::vision::VisionClient::new(http, cfg);
@@ -485,6 +533,9 @@ async fn spawn_analysis_daemon(
// still gets a useful side-channel on backend
// uptime + vision health.
.no_proxy()
.redirect(crate::crawler::safety::public_redirect_policy())
// No private-IP resolver: same operator-configured
// internal vision host as the analysis client above.
.build()
.context("build vision readiness http client")?;
Some(Arc::new(crate::analysis::daemon::HttpVisionReadiness {
@@ -508,10 +559,19 @@ async fn spawn_analysis_daemon(
readiness,
},
);
// Log the *effective* backend (what the worker actually dispatches
// through), not the raw `cfg.backend` — with vision dormant they diverge
// when an operator requests `vision`, and a misleading line here was a
// real observability footgun. The model label tracks the effective engine.
let effective_backend = cfg.effective_backend();
let effective_model: &str = match effective_backend {
crate::config::AnalysisBackend::Ocr => crate::analysis::ocr::OCR_MODEL_LABEL,
crate::config::AnalysisBackend::Vision => &cfg.model,
};
tracing::info!(
workers = cfg.workers,
backend = ?cfg.backend,
model = %cfg.model,
backend = ?effective_backend,
model = %effective_model,
"analysis worker daemon started"
);
Ok(handle)
@@ -533,6 +593,10 @@ async fn spawn_crawler_daemon(
cfg: &CrawlerConfig,
analysis_enabled: Arc<AtomicBool>,
) -> anyhow::Result<SpawnedDaemon> {
// Publish the opt-in browser SSRF-interception toggle so every headless
// navigation (via `intercept::open_page`) honours it. Off by default.
crate::crawler::intercept::set_enabled(cfg.ssrf_intercept);
// Reqwest client with a shared cookie jar so CDN image fetches include
// PHPSESSID. The same `Arc<Jar>` is held by the SessionController, so a
// runtime session refresh rewrites it in place. Initial value: a
@@ -553,6 +617,13 @@ async fn spawn_crawler_daemon(
let mut http_builder = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(30))
.no_proxy()
// Re-validate every redirect hop against the download allowlist:
// reqwest's default policy follows up to 10 redirects, and
// `is_safe_url` only guards the initial URL, so an allowlisted CDN
// 302ing to a private IP would otherwise be followed (SSRF).
.redirect(crate::crawler::safety::safe_redirect_policy(
cfg.download_allowlist.clone(),
))
.cookie_provider(Arc::clone(&cookie_jar));
if let Some(ua) = &cfg.user_agent {
http_builder = http_builder.user_agent(ua);
@@ -561,6 +632,17 @@ async fn spawn_crawler_daemon(
http_builder = http_builder
.proxy(reqwest::Proxy::all(proxy).with_context(|| format!("parse proxy: {proxy}"))?);
}
// DNS-rebinding guard: reject hosts that resolve to a private/internal IP,
// complementing the string-level allowlist check which can't see
// post-resolution addresses. Attached on the direct path AND on http(s)
// proxies (reqwest resolves the target itself there). Skipped only for SOCKS
// proxies, where the proxy — not reqwest — resolves the target, so the only
// name this resolver would see is the proxy's OWN host (legitimately on a
// private Docker IP, e.g. `tor` → 172.x); attaching it there rejected every
// fetch for zero gain. See `should_attach_safe_resolver`.
if crate::crawler::safety::should_attach_safe_resolver(cfg.proxy.as_deref()) {
http_builder = http_builder.dns_resolver(crate::crawler::safety::safe_dns_resolver());
}
let http = http_builder.build().context("build crawler reqwest")?;
let mut rate = HostRateLimiters::new(std::time::Duration::from_millis(cfg.rate_ms));
@@ -684,6 +766,7 @@ async fn spawn_crawler_daemon(
rate: Arc::clone(&rate),
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
max_images_per_chapter: cfg.max_images_per_chapter,
analysis_enabled,
transient_failures: Arc::new(AtomicU32::new(0)),
restart_threshold: cfg.browser_restart_threshold,
@@ -700,6 +783,7 @@ async fn spawn_crawler_daemon(
rate: Arc::clone(&rate),
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
max_images_per_chapter: cfg.max_images_per_chapter,
tor: tor.as_ref().map(Arc::clone),
});
@@ -886,6 +970,8 @@ struct RealChapterDispatcher {
rate: Arc<HostRateLimiters>,
download_allowlist: DownloadAllowlist,
max_image_bytes: usize,
/// Per-chapter image-count cap (see `CrawlerConfig::max_images_per_chapter`).
max_images_per_chapter: usize,
/// Enqueue `analyze_page` jobs for freshly-crawled pages. Shared gate
/// (read live) so toggling analysis at runtime takes effect without a
/// crawler respawn. Mirrors the analysis enable setting.
@@ -930,7 +1016,17 @@ impl ChapterDispatcher for RealChapterDispatcher {
pages_done: 0,
pages_total: None,
});
let lease = self.browser_manager.acquire().await?;
let lease = match self.browser_manager.acquire().await {
Ok(l) => l,
Err(e) => {
// Browser down / mid-restart: defer the job WITHOUT
// burning an attempt (the daemon releases it back to
// pending) rather than counting an infrastructure
// outage as a per-job failure.
tracing::warn!(error = ?e, "dispatch: browser unavailable — deferring job");
return Ok(SyncOutcome::BrowserUnavailable);
}
};
let result = content::sync_chapter_content(
&lease,
&self.db,
@@ -943,6 +1039,7 @@ impl ChapterDispatcher for RealChapterDispatcher {
false,
&self.download_allowlist,
self.max_image_bytes,
self.max_images_per_chapter,
self.tor.as_deref(),
Some(&self.status),
self.analysis_enabled.load(Ordering::Relaxed),
@@ -1000,7 +1097,15 @@ impl ChapterDispatcher for RealChapterDispatcher {
// Scope the lease so it (and the borrowing FetchContext) drop
// before any browser-restart handling in the match below.
let result = {
let lease = self.browser_manager.acquire().await?;
let lease = match self.browser_manager.acquire().await {
Ok(l) => l,
Err(e) => {
// See the SyncChapterContent arm: defer without
// burning an attempt when the browser is unavailable.
tracing::warn!(error = ?e, "dispatch: browser unavailable — deferring job");
return Ok(SyncOutcome::BrowserUnavailable);
}
};
let ctx = crate::crawler::source::FetchContext {
browser: &lease,
rate: &self.rate,
@@ -1234,8 +1339,8 @@ fn parse_origin(raw: &str) -> Option<String> {
let host = url.host_str()?;
let scheme = url.scheme();
let port_str = match (url.port(), scheme) {
(Some(p), "http") if p == 80 => String::new(),
(Some(p), "https") if p == 443 => String::new(),
(Some(80), "http") => String::new(),
(Some(443), "https") => String::new(),
(Some(p), _) => format!(":{p}"),
(None, _) => String::new(),
};
@@ -1473,29 +1578,32 @@ mod tests {
// Bind to a local so the TempDir lives for the rest of the test.
// `tempfile::tempdir().unwrap().path()` would drop the TempDir
// at end-of-expression and `LocalStorage` would hold a path to
// a deleted directory. (Today this is fine because the dispatch
// fails before storage is touched, but it makes the test fragile
// to any future code rearrangement.)
// a deleted directory.
let storage_dir = tempfile::tempdir().unwrap();
let storage: Arc<dyn Storage> =
Arc::new(LocalStorage::new(storage_dir.path()));
let mut cfg = crate::config::AnalysisConfig::default();
// Use the vision backend so the daemon doesn't try to load the ocrs
// `.rten` models (absent in unit CI). This test only exercises the
// pre-worker lease reclaim, which is engine-agnostic; no dispatch runs.
cfg.backend = crate::config::AnalysisBackend::Vision;
cfg.workers = 1;
cfg.job_timeout = Duration::from_secs(1);
// The worker always runs OCR now (vision is dormant — see
// `effective_backend`), and the `.rten` models aren't shipped to unit
// CI. Point the engine at a path that can't exist so the *engine build*
// fails deterministically. Reclaim runs at the very top of
// `spawn_analysis_daemon`, before — and independently of — engine
// readiness, so the row must still be reclaimed even though spawn
// returns Err. That's exactly the regression this test guards (an
// analysis-only deploy must reclaim orphaned leases at startup).
let cfg = crate::config::AnalysisConfig {
ocr_detection_model: "/nonexistent/text-detection.rten".to_string(),
ocr_recognition_model: "/nonexistent/text-recognition.rten".to_string(),
workers: 1,
job_timeout: Duration::from_secs(1),
..Default::default()
};
let events = Arc::new(crate::analysis::events::AnalysisEvents::new());
let handle = spawn_analysis_daemon(pool.clone(), storage, &cfg, events)
.await
.expect("spawn");
// Immediately shut down — we're only here to prove reclaim ran.
// The workers may briefly pick up the now-pending row; that's
// fine, but we cancel before any real dispatch (the LocalStorage
// key doesn't exist, so a dispatch would fail anyway).
handle.shutdown().await;
let spawned = spawn_analysis_daemon(pool.clone(), storage, &cfg, events).await;
assert!(
spawned.is_err(),
"engine build must fail with a missing model path"
);
// The reclaim must have moved the row back to pending with the
// attempt refunded (attempts goes from 1 → 0). Reaching it on the

View File

@@ -70,6 +70,45 @@ impl FromRequestParts<AppState> for CurrentUser {
}
}
/// Client IP for per-IP auth rate limiting. Resolves to the first hop of
/// `X-Forwarded-For` **only** when [`crate::config::AuthConfig::trusted_proxy`]
/// is set (the backend is behind a proxy that overrides the header — the
/// compose deploy). Otherwise `None`, so the limiter uses its shared bucket.
/// Never fails: a missing or malformed header simply yields `None`.
pub struct ClientIp(pub Option<std::net::IpAddr>);
/// Parse the client IP from an `X-Forwarded-For` value: the left-most hop is
/// the original client (later hops are intermediary proxies). Factored out so
/// the parsing is unit-testable without constructing a request.
pub(crate) fn first_forwarded_ip(header: &str) -> Option<std::net::IpAddr> {
header
.split(',')
.next()
.map(str::trim)
.filter(|s| !s.is_empty())
.and_then(|s| s.parse::<std::net::IpAddr>().ok())
}
#[async_trait]
impl FromRequestParts<AppState> for ClientIp {
type Rejection = std::convert::Infallible;
async fn from_request_parts(
parts: &mut Parts,
state: &AppState,
) -> Result<Self, Self::Rejection> {
if !state.auth.trusted_proxy {
return Ok(ClientIp(None));
}
let ip = parts
.headers
.get("x-forwarded-for")
.and_then(|v| v.to_str().ok())
.and_then(first_forwarded_ip);
Ok(ClientIp(ip))
}
}
/// Cookie-only authentication. Bot/API tokens are explicitly NOT accepted
/// here — this is the substrate for [`RequireAdmin`] and exists precisely
/// to keep admin authority out of bearer-token reach.
@@ -120,3 +159,34 @@ impl FromRequestParts<AppState> for RequireAdmin {
Ok(RequireAdmin(user))
}
}
#[cfg(test)]
mod tests {
use super::first_forwarded_ip;
use std::net::IpAddr;
#[test]
fn parses_left_most_client_hop() {
// The client is the first entry; later entries are intermediary proxies.
assert_eq!(
first_forwarded_ip("203.0.113.5, 10.0.0.1, 10.0.0.2"),
Some("203.0.113.5".parse::<IpAddr>().unwrap())
);
assert_eq!(
first_forwarded_ip(" 198.51.100.9 "),
Some("198.51.100.9".parse::<IpAddr>().unwrap())
);
assert_eq!(
first_forwarded_ip("2001:db8::1, 10.0.0.1"),
Some("2001:db8::1".parse::<IpAddr>().unwrap())
);
}
#[test]
fn rejects_empty_or_garbage() {
assert_eq!(first_forwarded_ip(""), None);
assert_eq!(first_forwarded_ip(" "), None);
assert_eq!(first_forwarded_ip("not-an-ip"), None);
assert_eq!(first_forwarded_ip(", 10.0.0.1"), None);
}
}

View File

@@ -28,6 +28,25 @@ pub fn verify_password(plain: &str, phc: &str) -> bool {
.is_ok()
}
/// Async wrapper around [`hash_password`] that offloads the CPU- and
/// memory-heavy Argon2 work to the blocking pool. Async handlers must call
/// this rather than the sync primitive: at ~15-50 ms per hash, running it
/// inline stalls every other task sharing the runtime worker thread.
pub async fn hash_password_async(plain: String) -> AppResult<String> {
tokio::task::spawn_blocking(move || hash_password(&plain))
.await
.map_err(|e| AppError::Other(anyhow::anyhow!("password hash task join: {e}")))?
}
/// Async wrapper around [`verify_password`]. A join failure (blocking pool
/// gone at shutdown, panic) resolves to `false` — the caller only ever uses
/// the result as a boolean gate, and denying auth is the safe default.
pub async fn verify_password_async(plain: String, phc: String) -> bool {
tokio::task::spawn_blocking(move || verify_password(&plain, &phc))
.await
.unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
@@ -56,4 +75,24 @@ mod tests {
let b = hash_password("same").unwrap();
assert_ne!(a, b, "two hashes of the same password must differ (salt)");
}
#[tokio::test]
async fn async_hash_then_verify_roundtrip() {
let phc = hash_password_async("correct horse battery staple".to_string())
.await
.unwrap();
assert!(phc.starts_with("$argon2id$"));
assert!(verify_password_async("correct horse battery staple".to_string(), phc).await);
}
#[tokio::test]
async fn async_verify_rejects_wrong_password() {
let phc = hash_password_async("hunter2".to_string()).await.unwrap();
assert!(!verify_password_async("hunter3".to_string(), phc).await);
}
#[tokio::test]
async fn async_verify_rejects_malformed_hash() {
assert!(!verify_password_async("anything".to_string(), "not a real phc".to_string()).await);
}
}

View File

@@ -15,9 +15,17 @@
//! tests run in isolated buckets and won't bleed across `#[sqlx::test]`
//! cases that share a process.
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::Mutex;
use std::time::Instant;
/// Upper bound on distinct client IPs tracked at once, so a spray from many
/// spoofed/rotating source IPs can't grow the map without limit. When full,
/// idle (refilled-to-burst) buckets are pruned first; if none are idle a new
/// IP falls back to the shared global bucket for that request.
const MAX_TRACKED_IPS: usize = 10_000;
/// Tunable limits. `per_sec == 0` disables the limiter — used by the
/// test harness and by anyone who wants to opt out via env config.
#[derive(Clone, Copy, Debug)]
@@ -50,6 +58,42 @@ struct Bucket {
last_refill: Instant,
}
impl Bucket {
fn new(burst: u32) -> Self {
Self {
tokens: f64::from(burst),
last_refill: Instant::now(),
}
}
/// Refill by elapsed time then try to consume one token.
fn try_take(&mut self, cfg: &RateLimitConfig, now: Instant) -> AcquireResult {
let elapsed = now.duration_since(self.last_refill).as_secs_f64();
self.tokens =
(self.tokens + elapsed * f64::from(cfg.per_sec)).min(f64::from(cfg.burst));
self.last_refill = now;
if self.tokens >= 1.0 {
self.tokens -= 1.0;
AcquireResult::Allowed
} else {
let deficit = 1.0 - self.tokens;
let wait_secs = (deficit / f64::from(cfg.per_sec)).ceil() as u64;
AcquireResult::Denied {
retry_after_secs: wait_secs.max(1),
}
}
}
/// Whether this bucket has fully refilled (i.e. the client has been idle).
/// Such buckets carry no state worth keeping — dropping and lazily
/// recreating one yields an identical full bucket — so they're the safe
/// eviction target under memory pressure.
fn is_idle(&self, cfg: &RateLimitConfig, now: Instant) -> bool {
let elapsed = now.duration_since(self.last_refill).as_secs_f64();
(self.tokens + elapsed * f64::from(cfg.per_sec)) >= f64::from(cfg.burst)
}
}
/// Outcome of [`AuthRateLimiter::try_acquire`]. When `Denied`, the
/// caller can use `retry_after_secs` for a `Retry-After: N` header
/// (RFC 6585 §4) so well-behaved clients back off correctly rather
@@ -60,51 +104,64 @@ pub enum AcquireResult {
Denied { retry_after_secs: u64 },
}
/// Single-bucket token-bucket limiter. `try_acquire` is cheap (one
/// mutex acquire, no allocations) so the auth path doesn't pay a real
/// cost for the check.
/// Token-bucket limiter keyed by client IP, with a shared global bucket for
/// requests whose IP is unknown (no trusted proxy).
///
/// The old design was a single global bucket. That let one attacker at the
/// sustained rate deny auth to *every* user (the bucket stayed drained). With
/// the SvelteKit proxy now forwarding the peer IP (`X-Forwarded-For`, honoured
/// only when `AUTH_TRUSTED_PROXY` is set), each source IP gets its own bucket,
/// so an attacker only throttles themselves. When no trustworthy IP is
/// available the caller passes `None` and the shared `global` bucket applies —
/// exactly the previous behaviour.
pub struct AuthRateLimiter {
cfg: RateLimitConfig,
bucket: Mutex<Bucket>,
global: Mutex<Bucket>,
per_ip: Mutex<HashMap<IpAddr, Bucket>>,
}
impl AuthRateLimiter {
pub fn new(cfg: RateLimitConfig) -> Self {
Self {
cfg,
bucket: Mutex::new(Bucket {
tokens: cfg.burst as f64,
last_refill: Instant::now(),
}),
global: Mutex::new(Bucket::new(cfg.burst)),
per_ip: Mutex::new(HashMap::new()),
}
}
/// Consume one token if available. Returns `Denied` with a
/// rounded-up seconds-until-refill so the caller can emit a
/// `Retry-After` header.
pub fn try_acquire(&self) -> AcquireResult {
/// Consume one token from the bucket for `key` (per-IP when `Some`, the
/// shared global bucket when `None`). Returns `Denied` with a rounded-up
/// seconds-until-refill so the caller can emit a `Retry-After` header.
pub fn try_acquire(&self, key: Option<IpAddr>) -> AcquireResult {
if self.cfg.per_sec == 0 {
return AcquireResult::Allowed;
}
let now = Instant::now();
let mut bucket = self.bucket.lock().expect("rate limiter mutex poisoned");
let elapsed = now.duration_since(bucket.last_refill).as_secs_f64();
bucket.tokens =
(bucket.tokens + elapsed * f64::from(self.cfg.per_sec)).min(f64::from(self.cfg.burst));
bucket.last_refill = now;
if bucket.tokens >= 1.0 {
bucket.tokens -= 1.0;
AcquireResult::Allowed
} else {
// ceil((1 - tokens) / per_sec), minimum 1 — a `Retry-After: 0`
// would tell clients to retry immediately, which is what we're
// actively trying to discourage.
let deficit = 1.0 - bucket.tokens;
let wait_secs = (deficit / f64::from(self.cfg.per_sec)).ceil() as u64;
AcquireResult::Denied {
retry_after_secs: wait_secs.max(1),
let Some(ip) = key else {
return self
.global
.lock()
.unwrap_or_else(|e| e.into_inner())
.try_take(&self.cfg, now);
};
let mut map = self.per_ip.lock().unwrap_or_else(|e| e.into_inner());
if map.len() >= MAX_TRACKED_IPS && !map.contains_key(&ip) {
map.retain(|_, b| !b.is_idle(&self.cfg, now));
if map.len() >= MAX_TRACKED_IPS {
// Still saturated with active attackers — degrade to the shared
// bucket rather than grow unbounded. Worst case is the old
// global-bucket behaviour under an extreme distributed flood.
drop(map);
return self
.global
.lock()
.unwrap_or_else(|e| e.into_inner())
.try_take(&self.cfg, now);
}
}
map.entry(ip)
.or_insert_with(|| Bucket::new(self.cfg.burst))
.try_take(&self.cfg, now)
}
}
@@ -119,7 +176,7 @@ mod tests {
burst: 0,
});
for _ in 0..1000 {
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
}
}
@@ -130,10 +187,10 @@ mod tests {
per_sec: 1,
burst: 3,
});
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
match rl.try_acquire() {
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
match rl.try_acquire(None) {
AcquireResult::Denied { retry_after_secs } => {
// Bucket is at ~0 tokens, refill rate 1/sec → ~1s wait.
assert!(
@@ -152,11 +209,11 @@ mod tests {
per_sec: 10,
burst: 1,
});
assert_eq!(rl.try_acquire(), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(), AcquireResult::Denied { .. }));
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(None), AcquireResult::Denied { .. }));
std::thread::sleep(std::time::Duration::from_millis(150));
assert_eq!(
rl.try_acquire(),
rl.try_acquire(None),
AcquireResult::Allowed,
"token should have refilled"
);
@@ -170,10 +227,95 @@ mod tests {
per_sec: 1,
burst: 1,
});
slow.try_acquire();
match slow.try_acquire() {
slow.try_acquire(None);
match slow.try_acquire(None) {
AcquireResult::Denied { retry_after_secs } => assert_eq!(retry_after_secs, 1),
_ => panic!("expected Denied"),
}
}
fn ip(s: &str) -> Option<IpAddr> {
Some(s.parse().unwrap())
}
#[test]
fn per_ip_buckets_are_independent() {
// The whole point of the fix: one IP draining its bucket must not deny
// a different IP.
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 2,
});
let a = ip("203.0.113.7");
let b = ip("198.51.100.9");
assert_eq!(rl.try_acquire(a), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(a), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(a), AcquireResult::Denied { .. }));
// b is untouched.
assert_eq!(rl.try_acquire(b), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(b), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(b), AcquireResult::Denied { .. }));
}
#[test]
fn none_key_shares_the_global_bucket_independent_of_per_ip() {
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 2,
});
// Drain the global (None) bucket.
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
assert!(matches!(rl.try_acquire(None), AcquireResult::Denied { .. }));
// A real IP is on its own bucket, unaffected by the drained global one.
assert_eq!(rl.try_acquire(ip("203.0.113.1")), AcquireResult::Allowed);
}
#[test]
fn tracked_ip_map_is_bounded() {
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 1,
burst: 1,
});
// Distinct IPs beyond the cap must not grow the map without bound —
// excess requests fall back to the shared bucket instead.
for i in 0..(MAX_TRACKED_IPS as u64 + 50) {
let octet_a = (i >> 8) as u8;
let octet_b = (i & 0xff) as u8;
let addr = format!("10.20.{octet_a}.{octet_b}");
let _ = rl.try_acquire(Some(addr.parse().unwrap()));
}
assert!(rl.per_ip.lock().unwrap().len() <= MAX_TRACKED_IPS);
}
#[test]
fn survives_a_poisoned_mutex() {
// If any thread ever panics while holding a limiter mutex, the lock
// becomes poisoned. With the old `.expect(...)` every later auth request
// would re-panic — one blip turned into a permanent auth outage. Recover
// the guard via `into_inner()` instead so the limiter keeps serving.
let rl = AuthRateLimiter::new(RateLimitConfig {
per_sec: 5,
burst: 5,
});
// Poison per_ip by panicking while holding its guard.
let poisoned = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let _g = rl.per_ip.lock().unwrap();
panic!("poison the per-IP mutex");
}));
assert!(poisoned.is_err(), "the panic must unwind");
assert!(rl.per_ip.is_poisoned(), "the mutex must now be poisoned");
// The per-IP path (Some(ip)) must still work despite the poison.
assert_eq!(rl.try_acquire(ip("198.51.100.9")), AcquireResult::Allowed);
// And poison the global bucket too — the None-key path must recover.
let _ = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let _g = rl.global.lock().unwrap();
panic!("poison the global mutex");
}));
assert!(rl.global.is_poisoned());
assert_eq!(rl.try_acquire(None), AcquireResult::Allowed);
}
}

View File

@@ -3,15 +3,16 @@
//! `generate_token` draws 32 bytes from the OS CSPRNG, encodes them as
//! URL-safe base64 (no padding), and returns the raw string alongside its
//! SHA-256 hash. Storage holds only the hash; the raw value lives in the
//! cookie or `Authorization` header. Comparison goes through
//! `constant_time_eq` to keep timing side channels off the table.
//! cookie or `Authorization` header. Token lookup is an indexed equality on
//! that 256-bit hash in the database (`WHERE token_hash = $1`), so there's no
//! in-process secret comparison to time-attack: a guess has to match a full
//! SHA-256 digest, and the DB index reveals nothing about how close it came.
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine as _;
use rand::rngs::OsRng;
use rand::RngCore;
use sha2::{Digest, Sha256};
use subtle::ConstantTimeEq;
pub const TOKEN_BYTES: usize = 32;
pub const HASH_BYTES: usize = 32;
@@ -30,10 +31,6 @@ pub fn hash_token(raw: &str) -> [u8; HASH_BYTES] {
hasher.finalize().into()
}
pub fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
a.ct_eq(b).into()
}
#[cfg(test)]
mod tests {
use super::*;
@@ -58,11 +55,4 @@ mod tests {
assert_eq!(hash_token("abc"), hash_token("abc"));
assert_ne!(hash_token("abc"), hash_token("abd"));
}
#[test]
fn constant_time_eq_compares_correctly() {
assert!(constant_time_eq(b"abc", b"abc"));
assert!(!constant_time_eq(b"abc", b"abd"));
assert!(!constant_time_eq(b"abc", b"abcd"));
}
}

View File

@@ -112,9 +112,20 @@ async fn main() -> anyhow::Result<()> {
cookie_jar.add_cookie_str(&cookie_str, &seed_url);
tracing::info!(domain, "seeded PHPSESSID into reqwest cookie jar");
}
// SSRF defence: only download from the catalog host + CDN host (plus
// optional CRAWLER_DOWNLOAD_ALLOWLIST extras). Built here so the same
// allowlist guards both the redirect policy and the per-image check.
let allowlist = Arc::new(build_download_allowlist(&start_url, cdn_host.as_deref()));
let mut http_builder = reqwest::Client::builder()
.timeout(Duration::from_secs(30))
.no_proxy()
// Re-validate every redirect hop against the allowlist — reqwest's
// default follows up to 10 redirects and `is_safe_url` only guards
// the initial URL, so a 302 to a private IP would otherwise pivot
// inside the deployment (SSRF).
.redirect(mangalord::crawler::safety::safe_redirect_policy(
(*allowlist).clone(),
))
.cookie_provider(cookie_jar);
if let Some(ua) = &user_agent {
http_builder = http_builder.user_agent(ua);
@@ -123,8 +134,25 @@ async fn main() -> anyhow::Result<()> {
http_builder = http_builder
.proxy(reqwest::Proxy::all(proxy).with_context(|| format!("parse proxy URL: {proxy}"))?);
}
// DNS-rebinding guard: attached on the direct path AND on http(s) proxies
// (reqwest resolves the target itself there), skipped only for SOCKS proxies
// where the proxy resolves the target. Use the shared predicate so this CLI
// and the daemon (app.rs) stay in lockstep — previously the CLI attached the
// resolver only on the fully-direct path, so an http(s) proxy silently lost
// the rebinding guard.
if mangalord::crawler::safety::should_attach_safe_resolver(proxy_url.as_deref()) {
http_builder =
http_builder.dns_resolver(mangalord::crawler::safety::safe_dns_resolver());
}
let http = http_builder.build().context("build http client")?;
// Opt-in browser SSRF interception (default off), mirroring the daemon.
mangalord::crawler::intercept::set_enabled(
std::env::var("CRAWLER_SSRF_INTERCEPT")
.map(|v| matches!(v.as_str(), "1" | "true" | "TRUE" | "yes"))
.unwrap_or(false),
);
let mut options = LaunchOptions::from_env();
if let Some(proxy) = &proxy_url {
let chromium_proxy = mangalord::crawler::url_utils::chromium_proxy_arg(proxy);
@@ -216,6 +244,7 @@ async fn main() -> anyhow::Result<()> {
rate_ms,
cdn_host.as_deref(),
cdn_rate_ms,
Arc::clone(&allowlist),
limit,
skip_chapters,
skip_chapter_content || !session_ready,
@@ -246,6 +275,7 @@ async fn run(
rate_ms: u64,
cdn_host: Option<&str>,
cdn_rate_ms: u64,
allowlist: Arc<mangalord::crawler::safety::DownloadAllowlist>,
limit: usize,
skip_chapters: bool,
skip_chapter_content: bool,
@@ -259,38 +289,18 @@ async fn run(
}
let rate = Arc::new(rate);
// SSRF defence: only download from the catalog host + CDN host
// (plus optional CRAWLER_DOWNLOAD_ALLOWLIST extras), and cap
// single-image downloads at CRAWLER_MAX_IMAGE_BYTES bytes.
// CRAWLER_ALLOW_ANY_HOST=true short-circuits the host check for
// sharded-CDN sources; private-IP and scheme guards still apply.
let allowlist = if env_bool("CRAWLER_ALLOW_ANY_HOST", false) {
mangalord::crawler::safety::DownloadAllowlist::allow_any()
} else {
let mut allow = mangalord::crawler::safety::DownloadAllowlist::new();
if let Ok(parsed) = reqwest::Url::parse(start_url) {
if let Some(h) = parsed.host_str() {
allow = allow.allow(h);
}
}
if let Some(host) = cdn_host {
allow = allow.allow(host);
}
if let Ok(extras) = std::env::var("CRAWLER_DOWNLOAD_ALLOWLIST") {
for piece in extras.split(',') {
let trimmed = piece.trim();
if !trimmed.is_empty() {
allow = allow.allow(trimmed);
}
}
}
allow
};
// Per-image download cap (the allowlist is built in `main` and passed in
// so the HTTP client's redirect policy and this check share one source).
let max_image_bytes: usize = std::env::var("CRAWLER_MAX_IMAGE_BYTES")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(mangalord::crawler::safety::DEFAULT_MAX_IMAGE_BYTES);
let allowlist = Arc::new(allowlist);
// Per-chapter image *count* cap — bounds total disk against a hostile
// reader page listing thousands of <img> tags. `0` disables it.
let max_images_per_chapter: usize = std::env::var("CRAWLER_MAX_IMAGES_PER_CHAPTER")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(2000);
let stats = pipeline::run_metadata_pass(
manager.as_ref(),
@@ -325,6 +335,7 @@ async fn run(
force_refetch_chapters,
Arc::clone(&allowlist),
max_image_bytes,
max_images_per_chapter,
tor.clone(),
)
.await?;
@@ -351,6 +362,7 @@ async fn sync_bookmarked_chapter_content(
force_refetch: bool,
allowlist: Arc<mangalord::crawler::safety::DownloadAllowlist>,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<Arc<mangalord::crawler::tor::TorController>>,
) -> anyhow::Result<()> {
let pending: Vec<(Uuid, Uuid, String)> = sqlx::query_as(
@@ -416,6 +428,7 @@ async fn sync_bookmarked_chapter_content(
force_refetch,
allowlist.as_ref(),
max_image_bytes,
max_images_per_chapter,
tor.as_deref(),
// CLI one-shot — no live status surface.
None,
@@ -431,6 +444,13 @@ async fn sync_bookmarked_chapter_content(
s.fetched += 1;
}
Ok(SyncOutcome::Skipped) => s.skipped += 1,
// Unreachable in the one-shot CLI (it holds its own lease
// and never dispatches through the queue), but count it as a
// failure for exhaustiveness.
Ok(SyncOutcome::BrowserUnavailable) => {
tracing::warn!(%chapter_id, "crawler browser unavailable");
s.failed += 1;
}
Ok(SyncOutcome::SessionExpired) => {
tracing::error!(
%chapter_id,
@@ -497,3 +517,37 @@ fn env_bool(name: &str, default: bool) -> bool {
}
}
/// Build the crawler download allowlist from env + the catalog/CDN hosts.
/// Shared by the HTTP client's redirect policy and the per-image safety
/// check so both agree on which hosts are reachable.
///
/// `CRAWLER_ALLOW_ANY_HOST=true` short-circuits the host check for
/// sharded-CDN sources; private-IP and scheme guards still apply.
fn build_download_allowlist(
start_url: &str,
cdn_host: Option<&str>,
) -> mangalord::crawler::safety::DownloadAllowlist {
use mangalord::crawler::safety::DownloadAllowlist;
if env_bool("CRAWLER_ALLOW_ANY_HOST", false) {
return DownloadAllowlist::allow_any();
}
let mut allow = DownloadAllowlist::new();
if let Ok(parsed) = reqwest::Url::parse(start_url) {
if let Some(h) = parsed.host_str() {
allow = allow.allow(h);
}
}
if let Some(host) = cdn_host {
allow = allow.allow(host);
}
if let Ok(extras) = std::env::var("CRAWLER_DOWNLOAD_ALLOWLIST") {
for piece in extras.split(',') {
let trimmed = piece.trim();
if !trimmed.is_empty() {
allow = allow.allow(trimmed);
}
}
}
allow
}

View File

@@ -27,6 +27,14 @@ pub struct AuthConfig {
/// so a private instance is locked down with a single switch.
/// Defaults to `false` (current public behaviour).
pub private_mode: bool,
/// Whether to trust a proxy-supplied `X-Forwarded-For` header as the
/// client IP for per-IP auth rate limiting. Enable ONLY when the backend
/// sits behind a trusted reverse proxy that overrides the header (the
/// compose deploy: SvelteKit's hooks.server.ts sets it from the real peer
/// address). When `false` (default), the header is ignored and the auth
/// limiter uses a single shared bucket — a directly-exposed backend must
/// keep this off or clients could spoof their IP to dodge the limit.
pub trusted_proxy: bool,
}
impl Default for AuthConfig {
@@ -42,6 +50,7 @@ impl Default for AuthConfig {
rate_limit: crate::auth::rate_limit::RateLimitConfig::default(),
allow_self_register: true,
private_mode: false,
trusted_proxy: false,
}
}
}
@@ -55,6 +64,13 @@ pub struct UploadConfig {
/// reject a single oversized cover/page without failing the whole
/// request just because the total happens to fit.
pub max_file_bytes: usize,
/// Max page images accepted in one chapter upload. Bounds how many
/// parts the handler will stage before giving up, so a client can't
/// pin a worker streaming an unbounded page count. `0` disables THIS
/// cap — the total is then bounded only by `max_request_bytes` (the
/// whole-request body limit), which stays the backstop either way.
/// Defaults to 2000. `MAX_PAGES_PER_CHAPTER`.
pub max_pages_per_chapter: usize,
}
impl Default for UploadConfig {
@@ -62,6 +78,46 @@ impl Default for UploadConfig {
Self {
max_request_bytes: 200 * 1024 * 1024, // 200 MiB
max_file_bytes: 20 * 1024 * 1024, // 20 MiB
max_pages_per_chapter: 2000,
}
}
}
/// Postgres connection-pool sizing. One pool backs every HTTP handler plus
/// the crawler/analysis daemons, so it must be large enough not to starve
/// interactive reads and fail fast (rather than hang on the driver's silent
/// 30 s default) when genuinely saturated.
#[derive(Clone, Debug)]
pub struct DbConfig {
/// `DB_MAX_CONNECTIONS`. Upper bound on open connections.
pub max_connections: u32,
/// `DB_ACQUIRE_TIMEOUT_SECS`. How long a caller waits for a free
/// connection before erroring — short so overload surfaces as a fast
/// 500 instead of a 30 s hang.
pub acquire_timeout: Duration,
}
impl Default for DbConfig {
fn default() -> Self {
Self {
max_connections: 20,
acquire_timeout: Duration::from_secs(10),
}
}
}
impl DbConfig {
pub fn from_env() -> Self {
let default = Self::default();
Self {
// `.max(1)`: a zero-size pool can never hand out a connection and
// would deadlock every query — clamp to at least one.
max_connections: env_u64("DB_MAX_CONNECTIONS", default.max_connections.into())
.max(1) as u32,
acquire_timeout: Duration::from_secs(env_u64(
"DB_ACQUIRE_TIMEOUT_SECS",
default.acquire_timeout.as_secs(),
)),
}
}
}
@@ -202,6 +258,13 @@ pub struct AnalysisConfig {
/// Hard cap on a page image's stored size; larger pages are skipped
/// (`ANALYSIS_MAX_IMAGE_BYTES`).
pub max_image_bytes: usize,
/// Hard cap on a page image's **decoded** pixel count for the OCR backend
/// (`ANALYSIS_OCR_MAX_DECODE_PIXELS`). `max_image_bytes` only bounds the
/// *encoded* size; without a decode bound a tiny image declaring
/// 50000×50000 inflates to billions of bytes and OOM-kills the worker
/// (decompression bomb). Generous by default (100 MP) so legitimately
/// tall, un-sliced manga pages still decode.
pub ocr_max_decode_pixels: u64,
/// Output-constraint mode (`ANALYSIS_RESPONSE_FORMAT`):
/// `json_schema` (default) | `json_object` | `none`.
pub response_format: ResponseFormat,
@@ -251,6 +314,7 @@ impl Default for AnalysisConfig {
tall_aspect_threshold: 1.6,
max_slices: 16,
max_image_bytes: 8 * 1024 * 1024,
ocr_max_decode_pixels: 100_000_000,
response_format: ResponseFormat::JsonSchema,
frequency_penalty: 0.3,
temperature: 0.0,
@@ -262,6 +326,24 @@ impl Default for AnalysisConfig {
}
impl AnalysisConfig {
/// The backend the worker actually dispatches through.
///
/// Vision is **temporarily disabled**: the engine code (`analysis::vision`,
/// `RealAnalyzeDispatcher`, the readiness probe) is kept intact but never
/// selected. Until it's re-enabled, this returns [`AnalysisBackend::Ocr`]
/// regardless of the parsed `backend`, logging a warning if `vision` was
/// requested so an env override isn't silently ignored. Re-enabling vision
/// is then a one-line change here (return `self.backend`).
pub fn effective_backend(&self) -> AnalysisBackend {
if self.backend == AnalysisBackend::Vision {
tracing::warn!(
"ANALYSIS_BACKEND=vision requested but the vision backend is temporarily \
disabled; running OCR instead"
);
}
AnalysisBackend::Ocr
}
pub fn from_env() -> Self {
let d = AnalysisConfig::default();
Self {
@@ -309,6 +391,10 @@ impl AnalysisConfig {
.max(1.0),
max_slices: env_usize("ANALYSIS_MAX_SLICES", d.max_slices).max(1),
max_image_bytes: env_usize("ANALYSIS_MAX_IMAGE_BYTES", d.max_image_bytes),
ocr_max_decode_pixels: env_u64(
"ANALYSIS_OCR_MAX_DECODE_PIXELS",
d.ocr_max_decode_pixels,
),
response_format: std::env::var("ANALYSIS_RESPONSE_FORMAT")
.map(|s| ResponseFormat::from_str(&s))
.unwrap_or(d.response_format),
@@ -335,6 +421,7 @@ pub struct Config {
pub database_url: String,
pub bind_address: String,
pub storage_dir: PathBuf,
pub db: DbConfig,
pub auth: AuthConfig,
pub upload: UploadConfig,
pub cors_allowed_origins: Vec<String>,
@@ -411,6 +498,13 @@ pub struct CrawlerConfig {
pub download_allowlist: DownloadAllowlist,
/// Hard upper bound on a single image download. Defaults to 32 MiB.
pub max_image_bytes: usize,
/// Hard upper bound on the number of page images in one chapter. A
/// hostile reader page could otherwise list thousands of `<img>`
/// tags; `max_image_bytes` caps each one but not the count, so the
/// product is an unbounded disk-fill. A chapter exceeding this is
/// acked failed rather than downloaded. `0` disables the cap.
/// Defaults to 2000. `CRAWLER_MAX_IMAGES_PER_CHAPTER`.
pub max_images_per_chapter: usize,
/// Max manga detail fetches per metadata pass. `0` means no cap
/// (full sweep up to the source's own bound). Sourced from
/// `CRAWLER_LIMIT`, mirroring the CLI binary.
@@ -428,6 +522,16 @@ pub struct CrawlerConfig {
/// exhausted) that trigger an automatic coordinated browser restart.
/// Defaults to 3. `CRAWLER_BROWSER_RESTART_THRESHOLD`.
pub browser_restart_threshold: u32,
/// CDP `Fetch` interception that re-validates every headless-browser
/// navigation/redirect/subresource against the SSRF check. Default `true`:
/// with it off, only the top-level URL string is validated, so a scraped
/// page's JS/subresources (which use Chromium's own network stack, not the
/// reqwest `SafeResolver`) can reach internal targets like the cloud
/// metadata service or postgres. The Fetch hook can't be exercised in CI (no
/// Chromium) — the `#[ignore]`d `ssrf_interception_does_not_wedge_allowed_navigation`
/// smoke test validates it against a real binary. `CRAWLER_SSRF_INTERCEPT`;
/// set `false` only as a break-glass if the hook destabilizes a deployment.
pub ssrf_intercept: bool,
}
impl Default for CrawlerConfig {
@@ -455,10 +559,12 @@ impl Default for CrawlerConfig {
browser: LaunchOptions::headless(),
download_allowlist: DownloadAllowlist::new(),
max_image_bytes: DEFAULT_MAX_IMAGE_BYTES,
max_images_per_chapter: 2000,
manga_limit: 0,
job_timeout: Duration::from_secs(600),
metadata_max_consecutive_failures: 10,
browser_restart_threshold: 3,
ssrf_intercept: true,
}
}
}
@@ -473,6 +579,7 @@ impl Config {
storage_dir: std::env::var("STORAGE_DIR")
.unwrap_or_else(|_| "./data/storage".to_string())
.into(),
db: DbConfig::from_env(),
auth: AuthConfig {
cookie_secure: env_bool("COOKIE_SECURE", true),
cookie_domain: std::env::var("COOKIE_DOMAIN")
@@ -491,10 +598,12 @@ impl Config {
},
allow_self_register: env_bool("ALLOW_SELF_REGISTER", true),
private_mode: env_bool("PRIVATE_MODE", false),
trusted_proxy: env_bool("AUTH_TRUSTED_PROXY", false),
},
upload: UploadConfig {
max_request_bytes: env_usize("MAX_REQUEST_BYTES", 200 * 1024 * 1024),
max_file_bytes: env_usize("MAX_FILE_BYTES", 20 * 1024 * 1024),
max_pages_per_chapter: env_usize("MAX_PAGES_PER_CHAPTER", 2000),
},
cors_allowed_origins: std::env::var("CORS_ALLOWED_ORIGINS")
.ok()
@@ -612,6 +721,7 @@ impl CrawlerConfig {
browser: LaunchOptions::from_env(),
download_allowlist,
max_image_bytes: env_usize("CRAWLER_MAX_IMAGE_BYTES", DEFAULT_MAX_IMAGE_BYTES),
max_images_per_chapter: env_usize("CRAWLER_MAX_IMAGES_PER_CHAPTER", 2000),
manga_limit: env_usize("CRAWLER_LIMIT", 0),
job_timeout: Duration::from_secs(env_u64("CRAWLER_JOB_TIMEOUT_SECS", 600).max(1)),
metadata_max_consecutive_failures: env_u64(
@@ -620,6 +730,7 @@ impl CrawlerConfig {
) as u32,
browser_restart_threshold: env_u64("CRAWLER_BROWSER_RESTART_THRESHOLD", 3).max(1)
as u32,
ssrf_intercept: env_bool("CRAWLER_SSRF_INTERCEPT", true),
})
}
}
@@ -767,6 +878,37 @@ mod tests {
assert_eq!(cfg.browser_restart_threshold, 7);
}
#[test]
fn db_pool_defaults_when_unset() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::remove_var("DB_MAX_CONNECTIONS");
std::env::remove_var("DB_ACQUIRE_TIMEOUT_SECS");
let cfg = DbConfig::from_env();
assert_eq!(cfg.max_connections, 20);
assert_eq!(cfg.acquire_timeout, Duration::from_secs(10));
}
#[test]
fn db_pool_parses_from_env() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("DB_MAX_CONNECTIONS", "50");
std::env::set_var("DB_ACQUIRE_TIMEOUT_SECS", "3");
let cfg = DbConfig::from_env();
std::env::remove_var("DB_MAX_CONNECTIONS");
std::env::remove_var("DB_ACQUIRE_TIMEOUT_SECS");
assert_eq!(cfg.max_connections, 50);
assert_eq!(cfg.acquire_timeout, Duration::from_secs(3));
}
#[test]
fn db_pool_max_connections_clamps_to_at_least_one() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
std::env::set_var("DB_MAX_CONNECTIONS", "0");
let cfg = DbConfig::from_env();
std::env::remove_var("DB_MAX_CONNECTIONS");
assert_eq!(cfg.max_connections, 1);
}
#[test]
fn analysis_config_defaults_when_unset() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
@@ -840,6 +982,21 @@ mod tests {
assert_eq!(AnalysisConfig::from_env().backend, AnalysisBackend::Ocr);
}
#[test]
fn effective_backend_is_ocr_even_when_vision_requested() {
// Vision is temporarily disabled: the worker always runs OCR no matter
// what `ANALYSIS_BACKEND` parsed to. `backend` still reflects the raw
// request (so the override is visible/loggable), but `effective_backend`
// is the value the daemon actually dispatches through.
let mut cfg = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
assert_eq!(cfg.effective_backend(), AnalysisBackend::Ocr);
cfg.backend = AnalysisBackend::Ocr;
assert_eq!(cfg.effective_backend(), AnalysisBackend::Ocr);
}
#[test]
fn ocr_model_paths_parse_from_env() {
let _g = ENV_GUARD.lock().unwrap_or_else(|p| p.into_inner());
@@ -939,6 +1096,13 @@ mod tests {
"BACKEND_URL",
"BACKEND_PROXY_TIMEOUT_MS",
"VISION_MANAGER_DATABASE_URL",
// Compose-level deployment knobs (port publish interface + per-container
// memory ceilings) — consumed by docker-compose.yml itself, never read
// by the backend process, so they don't belong in its environment block.
"FRONTEND_PUBLISH_ADDR",
"BACKEND_MEM_LIMIT",
"FRONTEND_MEM_LIMIT",
"POSTGRES_MEM_LIMIT",
];
/// Keys whose compose RHS is intentionally NOT a `${KEY...}`

View File

@@ -234,12 +234,20 @@ impl BrowserManager {
await_drain(&self.active, drain_deadline).await;
self.set_phase(RestartPhase::Restarting);
let relaunch = {
// Take the dead handle out under the lock, release the lock, THEN run
// the (slow) Chromium teardown — so a worker that raced past the drain
// can't block on `acquire()` behind one dead browser's close(). Mirrors
// the idle reaper's take-drop-close ordering. Re-acquire to relaunch.
let dead = {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
let _ = handle.close().await;
}
guard.handle.take()
};
if let Some(handle) = dead {
let _ = handle.close().await;
}
let relaunch = {
let mut guard = self.inner.lock().await;
self.launch_into(&mut guard).await
};
@@ -257,9 +265,14 @@ impl BrowserManager {
/// Used on daemon shutdown. After this returns the next acquire will
/// re-launch from scratch.
pub async fn shutdown(&self) {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
// Take-then-drop-then-close: don't hold the lock across Chromium
// teardown (see `invalidate` / the idle reaper).
let handle = {
let mut guard = self.inner.lock().await;
guard.shared = None;
guard.handle.take()
};
if let Some(handle) = handle {
let _ = handle.close().await;
}
}
@@ -278,9 +291,16 @@ impl BrowserManager {
/// Idempotent: calling on an already-invalidated manager is a
/// no-op.
pub async fn invalidate(&self) {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
// Take the handle out under the lock, then release the lock BEFORE the
// slow Chromium close() — otherwise every other worker's `acquire()`
// serializes behind one dead browser's teardown. Matches the idle
// reaper's ordering at the bottom of this file.
let handle = {
let mut guard = self.inner.lock().await;
guard.shared = None;
guard.handle.take()
};
if let Some(handle) = handle {
let _ = handle.close().await;
tracing::warn!("BrowserManager: handle invalidated — next acquire will relaunch");
}

View File

@@ -18,7 +18,7 @@ use uuid::Uuid;
use crate::crawler::detect::PageError;
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::safety::{fetch_stream, looks_like_image, DownloadAllowlist};
use crate::crawler::safety::{ensure_public_target, fetch_stream, looks_like_image, DownloadAllowlist};
use crate::crawler::session::{self, ChapterProbe};
use crate::storage::{Storage, StorageError};
@@ -71,6 +71,13 @@ pub enum SyncOutcome {
/// Session probe failed mid-sync (avatar selector missing on the
/// chapter page). Caller should abort the whole crawler run.
SessionExpired,
/// The headless browser could not be acquired (down or mid-restart).
/// Produced only by the queue dispatcher (which calls `acquire()`); it is
/// an *infrastructure* outage, not a job failure, so the daemon returns the
/// job to `pending` WITHOUT burning a retry attempt. `sync_chapter_content`
/// itself never returns this — callers that already hold a lease can treat
/// it as unreachable.
BrowserUnavailable,
}
/// Per-chapter max fetch attempts when TOR is configured. `N = 3` means
@@ -95,6 +102,19 @@ enum ChapterFetchOutcome {
PersistentTransient,
}
/// Refuse to navigate Chromium at a chapter URL that points inside the
/// deployment. The image-download path goes through `is_safe_url`, but
/// `new_page` is a second network surface: a `chapter_sources` row whose
/// host resolves to (or was crafted to be) a private IP would otherwise let
/// the headless browser probe `postgres:5432`, the cloud metadata service,
/// etc. Reuses the allowlist-free `ensure_public_target` (scheme +
/// private-IP literal check) since there is no per-host allowlist for the
/// scraped catalog itself.
fn guard_nav_url(source_url: &str) -> anyhow::Result<()> {
ensure_public_target(source_url)
.map_err(|e| anyhow::anyhow!("refuse to navigate unsafe chapter URL {source_url}: {e}"))
}
/// Single rate-limited Chromium navigation to the chapter URL,
/// returning the page HTML. Extracted from `sync_chapter_content` so
/// the recircuit loop can call it once per attempt.
@@ -103,27 +123,36 @@ async fn fetch_chapter_html_once(
rate: &HostRateLimiters,
source_url: &str,
) -> anyhow::Result<String> {
guard_nav_url(source_url)?;
rate.wait_for(source_url).await?;
let page = browser
.new_page(source_url)
let page = crate::crawler::intercept::open_page(browser, source_url)
.await
.with_context(|| format!("open chapter page {source_url}"))?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for chapter nav")?;
// Best-effort wait for the reader marker — same partial-render
// race that bit the chapter-list parser can hit here. Timeout is
// not an error; the chapter probe + parser sentinels still catch
// real failures.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"a#pic_container",
crate::crawler::nav::SELECTOR_TIMEOUT,
// Close the tab on every exit path — a `?` on wait_for_nav / content()
// would otherwise leak it (chromiumoxide doesn't close on drop).
let closer = page.clone();
crate::crawler::nav::close_after(
async move {
closer.close().await.ok();
},
async {
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for chapter nav")?;
// Best-effort wait for the reader marker — same partial-render
// race that bit the chapter-list parser can hit here. Timeout is
// not an error; the chapter probe + parser sentinels still catch
// real failures.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"a#pic_container",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
page.content().await.context("read chapter html")
},
)
.await;
let html = page.content().await.context("read chapter html")?;
page.close().await.ok();
Ok(html)
.await
}
/// Pure-over-IO loop: fetch + classify, up to `max_attempts` total
@@ -214,6 +243,7 @@ pub async fn sync_chapter_content(
force_refetch: bool,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<&crate::crawler::tor::TorController>,
progress: Option<&crate::crawler::status::StatusHandle>,
enqueue_analysis: bool,
@@ -221,7 +251,8 @@ pub async fn sync_chapter_content(
let started = std::time::Instant::now();
let result = sync_chapter_content_inner(
browser, db, storage, http, rate, chapter_id, manga_id, source_url,
force_refetch, allowlist, max_image_bytes, tor, progress, enqueue_analysis,
force_refetch, allowlist, max_image_bytes, max_images_per_chapter, tor, progress,
enqueue_analysis,
)
.await;
let duration_ms = started.elapsed().as_millis() as i64;
@@ -271,6 +302,7 @@ async fn sync_chapter_content_inner(
force_refetch: bool,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
max_images_per_chapter: usize,
tor: Option<&crate::crawler::tor::TorController>,
// Optional live-status sink for the realtime page counter. The daemon
// dispatcher passes the shared handle (the chapter has already been
@@ -331,6 +363,16 @@ async fn sync_chapter_content_inner(
if images.is_empty() {
anyhow::bail!("no page images parsed from {source_url}");
}
// Bound total disk per chapter: the per-image byte cap doesn't stop a
// hostile reader page from listing thousands of <img> tags. Ack failed
// (the caller records it and backs off) rather than downloading them.
if let Some(over) = image_count_over_cap(images.len(), max_images_per_chapter) {
anyhow::bail!(
"chapter at {source_url} lists {} page images, over the {} cap",
over.count,
over.cap
);
}
// Resolve image URLs against the chapter URL (they may be relative).
let base = reqwest::Url::parse(source_url).context("parse chapter URL")?;
@@ -407,6 +449,28 @@ pub(crate) struct StoredPage {
/// "first 16 bytes" diagnostic in the error path is useful.
const SNIFF_PREFIX_BYTES: usize = 64;
/// Bytes still admissible for the streaming tail after the sniff prefix
/// has been drained. The prefix already counts against the per-image cap,
/// so the tail budget is `max_image_bytes - prefix_len` using the
/// **actual** drained length — never a constant — so `prefix_len + tail`
/// can never exceed the cap.
fn remaining_after_prefix(max_image_bytes: usize, prefix_len: usize) -> usize {
max_image_bytes.saturating_sub(prefix_len)
}
/// A rejected over-cap image count, carrying the numbers for the error.
struct ImageCountOverCap {
count: usize,
cap: usize,
}
/// `Some(..)` when `count` exceeds the per-chapter image cap. A `cap` of
/// `0` disables the check (unbounded), matching the config's "0 = no cap"
/// contract.
fn image_count_over_cap(count: usize, cap: usize) -> Option<ImageCountOverCap> {
(cap != 0 && count > cap).then_some(ImageCountOverCap { count, cap })
}
/// Download a single page image, validate it's really an image, and
/// stream it to storage. Returns the storage key + content type. Does
/// not touch the DB — persistence is batched into one short transaction
@@ -480,11 +544,18 @@ async fn download_and_store_page(
// straight to storage. The cap is enforced via a running total in
// the stream adapter so a server that omits Content-Length still
// can't exhaust memory.
// Budget the streaming tail against the bytes *actually* drained into
// the prefix, not the constant `SNIFF_PREFIX_BYTES`. The prefix loop
// appends whole chunks, so a single 16 KiB first chunk fills the
// 64-byte sniff window in one drain — charging only 64 bytes would
// then let the tail add another full `max_image_bytes`, storing up to
// ~2× the cap. Capture the length before `prefix` is moved into the
// stream below.
let prefix_len = prefix.len();
let prefix_stream = futures_util::stream::once(async move {
Ok::<bytes::Bytes, StorageError>(prefix)
});
let prefix_len = SNIFF_PREFIX_BYTES.min(max_image_bytes);
let mut remaining = max_image_bytes.saturating_sub(prefix_len);
let mut remaining = remaining_after_prefix(max_image_bytes, prefix_len);
let url_for_err = url.clone();
let rest_stream = body.map(move |frame| match frame {
Ok(chunk) => {
@@ -524,15 +595,22 @@ pub(crate) async fn persist_pages(
let mut tx = db.begin().await.context("open chapter sync tx")?;
let mut page_ids: Vec<Uuid> = Vec::with_capacity(stored.len());
let mut page_numbers: Vec<i32> = Vec::with_capacity(stored.len());
// Pages whose row already existed and was overwritten by this re-crawl.
// Their image changed but pages.id (and any analysis keyed to it) survived,
// so the derived analysis is now stale and must be invalidated below.
let mut updated_page_ids: Vec<Uuid> = Vec::new();
for page in stored {
let (id,): (Uuid,) = sqlx::query_as(
// `xmax <> 0` on the RETURNING row distinguishes a conflict UPDATE
// (existing row overwritten) from a fresh INSERT — the standard upsert
// idiom. Cast through text/bigint since there is no direct xid<>int op.
let (id, was_update): (Uuid, bool) = sqlx::query_as(
"INSERT INTO pages (chapter_id, page_number, storage_key, content_type, size_bytes)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (chapter_id, page_number) DO UPDATE
SET storage_key = EXCLUDED.storage_key,
content_type = EXCLUDED.content_type,
size_bytes = EXCLUDED.size_bytes
RETURNING id",
RETURNING id, (xmax::text::bigint <> 0)",
)
.bind(chapter_id)
.bind(page.page_number)
@@ -544,6 +622,32 @@ pub(crate) async fn persist_pages(
.with_context(|| format!("insert page row {}", page.page_number))?;
page_ids.push(id);
page_numbers.push(page.page_number);
if was_update {
updated_page_ids.push(id);
}
}
// Invalidate stale analysis for re-crawled (overwritten) pages. Storage keys
// are deterministic per page number, so the row survives the upsert and its
// page_analysis / OCR / auto-tags / content-warnings would otherwise reflect
// the OLD image — poisoning search results and the derived
// manga_content_warnings. Clearing them (the DELETE on page_content_warnings
// fires the mcw_* triggers so manga_content_warnings recomputes) drops the
// pages back to "unanalyzed" so the enqueue below — and the admin
// only-unanalyzed backfill — re-derive them. Newly inserted pages have no
// prior analysis, so they're untouched.
if !updated_page_ids.is_empty() {
for table in [
"page_ocr_text",
"page_auto_tags",
"page_content_warnings",
"page_analysis",
] {
sqlx::query(&format!("DELETE FROM {table} WHERE page_id = ANY($1::uuid[])"))
.bind(&updated_page_ids)
.execute(&mut *tx)
.await
.with_context(|| format!("invalidate {table} for re-crawled pages"))?;
}
}
// Drop any rows left over from a prior, larger crawl of this chapter
// (re-crawl that now yields fewer pages). Without this the stale rows —
@@ -609,6 +713,64 @@ mod tests {
use super::*;
use crate::storage::LocalStorage;
#[test]
fn guard_nav_url_rejects_private_and_loopback_targets() {
// A chapter_sources row that resolves to / was crafted as an
// internal target must be refused before Chromium navigates.
for url in [
"http://127.0.0.1:5432/",
"http://169.254.169.254/latest/meta-data/",
"http://10.0.0.1/chapter/1",
"http://localhost:8080/",
"file:///etc/passwd",
] {
assert!(guard_nav_url(url).is_err(), "must reject {url}");
}
}
#[test]
fn guard_nav_url_allows_public_chapter_urls() {
assert!(guard_nav_url("https://reader.example.com/chapter/42").is_ok());
assert!(guard_nav_url("http://manga-host.test/c/1/p/2").is_ok());
}
#[test]
fn tail_budget_uses_actual_prefix_length_not_constant() {
// A single 16 KiB first chunk fills the 64-byte sniff window in one
// drain, so the streaming tail budget must be `cap - 16KiB`, not
// `cap - 64`. The constant-64 math previously admitted a further
// ~full cap on top of the already-drained prefix (~2× overshoot).
let cap = 20 * 1024;
let prefix_len = 16 * 1024; // one real chunk, well over SNIFF_PREFIX_BYTES
let remaining = remaining_after_prefix(cap, prefix_len);
assert_eq!(remaining, cap - prefix_len);
// Invariant: prefix + admitted tail never exceeds the cap.
assert!(prefix_len + remaining <= cap);
// And it's strictly tighter than the old constant-64 budget.
assert!(remaining < cap - SNIFF_PREFIX_BYTES);
}
#[test]
fn image_count_cap_rejects_only_over_cap_and_respects_disable() {
// Under / at the cap: accepted.
assert!(image_count_over_cap(0, 2000).is_none());
assert!(image_count_over_cap(2000, 2000).is_none());
// Over the cap: rejected, carrying the numbers for the error.
let over = image_count_over_cap(2001, 2000).expect("over cap");
assert_eq!(over.count, 2001);
assert_eq!(over.cap, 2000);
// `0` disables the cap entirely (unbounded), matching config contract.
assert!(image_count_over_cap(1_000_000, 0).is_none());
}
#[test]
fn tail_budget_saturates_when_prefix_hits_cap() {
// Body shorter than the sniff window: prefix drained == cap, tail
// budget is zero (no negative underflow).
assert_eq!(remaining_after_prefix(50, 50), 0);
assert_eq!(remaining_after_prefix(50, 64), 0);
}
#[tokio::test]
async fn cleanup_orphans_deletes_written_keys() {
let dir = tempfile::tempdir().unwrap();
@@ -1071,4 +1233,91 @@ mod tests {
assert_eq!(fetch_n, 1);
assert!(format!("{err:#}").contains("nav timeout"));
}
#[sqlx::test(migrations = "./migrations")]
async fn persist_pages_invalidates_stale_analysis_on_recrawl_update(pool: PgPool) {
// Page storage keys are deterministic per page number, so a re-crawl
// overwrites the image but keeps pages.id. Analysis keyed to page_id
// would otherwise stay stale (old OCR/search_doc/warnings). persist_pages
// must clear the derived analysis for updated pages so it re-derives.
let manga_id = Uuid::new_v4();
let chapter_id = Uuid::new_v4();
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, 'T')")
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO chapters (id, manga_id, number) VALUES ($1, $2, 1)")
.bind(chapter_id)
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
let v1 = vec![StoredPage {
page_number: 1,
storage_key: "k/0001.jpg".into(),
content_type: "image/jpeg".into(),
size_bytes: 10,
}];
persist_pages(&pool, chapter_id, &v1, false).await.unwrap();
let page_id: Uuid = sqlx::query_scalar(
"SELECT id FROM pages WHERE chapter_id = $1 AND page_number = 1",
)
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
// Simulate a completed prior analysis pass for that page.
sqlx::query(
"INSERT INTO page_analysis (page_id, status, is_nsfw, analyzed_at) \
VALUES ($1, 'done', true, now())",
)
.bind(page_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO page_ocr_text (page_id, kind, text) VALUES ($1, 'speech', 'stale')")
.bind(page_id)
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO page_content_warnings (page_id, warning) VALUES ($1, 'gore')")
.bind(page_id)
.execute(&pool)
.await
.unwrap();
// Re-crawl page 1 with a new image (same deterministic key, new size).
let v2 = vec![StoredPage {
page_number: 1,
storage_key: "k/0001.jpg".into(),
content_type: "image/jpeg".into(),
size_bytes: 999,
}];
persist_pages(&pool, chapter_id, &v2, false).await.unwrap();
// The page row survived (deterministic key).
let same_id: Uuid = sqlx::query_scalar(
"SELECT id FROM pages WHERE chapter_id = $1 AND page_number = 1",
)
.bind(chapter_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(same_id, page_id, "re-crawl keeps the page id");
// ...but its stale analysis is cleared so it gets re-derived.
for table in ["page_analysis", "page_ocr_text", "page_content_warnings"] {
let n: i64 = sqlx::query_scalar(&format!(
"SELECT COUNT(*) FROM {table} WHERE page_id = $1"
))
.bind(page_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(n, 0, "{table} must be invalidated when the re-crawl updates the page");
}
}
}

View File

@@ -66,6 +66,43 @@ const LEASE_DURATION: Duration = Duration::from_secs(60);
/// the lease window leaves two missed-beat's slack before expiry.
const LEASE_HEARTBEAT: Duration = Duration::from_secs(20);
/// Consecutive failed lease renews the heartbeat tolerates before it gives up
/// and signals the worker to abandon the in-flight dispatch. At
/// [`LEASE_HEARTBEAT`] spacing this is ~1 lease window of DB flakiness — past
/// that the lease has very likely lapsed and another worker may re-lease the
/// job, so continuing to crawl it is wasted (and duplicated) work.
const MAX_HEARTBEAT_RENEW_FAILURES: u32 = 3;
/// Whether the heartbeat should abandon the job after `consecutive_failures`
/// failed renews. Split out so the escalation threshold is unit-testable.
fn should_abort_after_renew_failures(consecutive_failures: u32) -> bool {
consecutive_failures >= MAX_HEARTBEAT_RENEW_FAILURES
}
/// How long a worker waits after a `BrowserUnavailable` outcome before looping
/// back to lease again. The job was released (not failed) so it stays pending;
/// this backoff keeps the worker from hot-looping lease→acquire→release while
/// the browser is down or mid-restart.
const BROWSER_UNAVAILABLE_BACKOFF: Duration = Duration::from_secs(5);
/// Longest an idle worker waits between lease polls. Bounds the exponential
/// [`idle_backoff`] so a worker still notices freshly-enqueued work reasonably
/// soon after a quiet spell.
const IDLE_BACKOFF_CAP: Duration = Duration::from_secs(30);
/// Backoff for a worker that keeps finding no work: 1s, 2s, 4s, … capped at
/// [`IDLE_BACKOFF_CAP`]. `consecutive_empty` is the number of empty polls seen
/// so far (0 on the first miss); reset to 0 the moment a job is leased. Replaces
/// the old flat 1s sleep so an idle daemon isn't firing a row-locking `SELECT …
/// FOR UPDATE SKIP LOCKED` lease query every second per worker.
fn idle_backoff(consecutive_empty: u32) -> Duration {
let cap = IDLE_BACKOFF_CAP.as_secs();
// 1 << n grows the interval; saturate to the cap once the shift overflows
// or the value exceeds the cap.
let secs = 1u64.checked_shl(consecutive_empty).unwrap_or(cap).min(cap);
Duration::from_secs(secs)
}
#[async_trait]
pub trait MetadataPass: Send + Sync {
async fn run(&self) -> anyhow::Result<pipeline::MetadataStats>;
@@ -301,9 +338,9 @@ impl CronContext {
}
Err(e) => tracing::error!(?e, "cron: enqueue_bookmarked_pending failed"),
}
match jobs::reap_done(pool, retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: done-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: done-job reaper failed"),
match jobs::reap_terminal(pool, retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: terminal-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: terminal-job reaper failed"),
}
match crate::repo::crawl_metrics::reap(pool, metrics_retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: crawl-metrics reaper finished"),
@@ -356,6 +393,9 @@ struct WorkerContext {
impl WorkerContext {
async fn run(self) {
// Consecutive empty lease polls, driving the idle backoff. Reset to 0
// the moment any job is leased.
let mut idle_streak: u32 = 0;
loop {
if self.cancel.is_cancelled() {
tracing::info!(worker = self.id, "worker: shutdown");
@@ -385,11 +425,14 @@ impl WorkerContext {
}
};
let Some(lease) = leases.into_iter().next() else {
let backoff = idle_backoff(idle_streak);
idle_streak = idle_streak.saturating_add(1);
tokio::select! {
_ = tokio::time::sleep(Duration::from_secs(1)) => continue,
_ = tokio::time::sleep(backoff) => continue,
_ = self.cancel.cancelled() => return,
}
};
idle_streak = 0;
self.process_lease(lease).await;
}
}
@@ -413,18 +456,34 @@ impl WorkerContext {
// dispatch runs, so a slow-but-healthy job is never re-leased and
// never inflates `attempts` toward `max_attempts`. Stops itself
// once the job is no longer ours (renew returns false).
// Signalled by the heartbeat if it gives up after too many consecutive
// renew failures, so the worker can abandon a dispatch whose lease has
// very likely lapsed (rather than crawl a job another worker may now own).
let hb_lost = CancellationToken::new();
let heartbeat = {
let hb_pool = self.pool.clone();
let hb_id = lease.id;
let hb_gen = lease.lease_generation;
let hb_lost = hb_lost.clone();
tokio::spawn(async move {
let mut failures: u32 = 0;
loop {
tokio::time::sleep(LEASE_HEARTBEAT).await;
match jobs::renew(&hb_pool, hb_id, hb_gen, LEASE_DURATION).await {
Ok(true) => {}
Ok(true) => failures = 0,
Ok(false) => break,
Err(e) => {
tracing::warn!(lease_id = %hb_id, ?e, "heartbeat renew failed");
failures += 1;
tracing::warn!(lease_id = %hb_id, failures, ?e, "heartbeat renew failed");
if should_abort_after_renew_failures(failures) {
tracing::error!(
lease_id = %hb_id,
failures,
"heartbeat lost the lease after repeated renew failures — signalling abandon"
);
hb_lost.cancel();
break;
}
}
}
}
@@ -462,6 +521,20 @@ impl WorkerContext {
);
return;
}
_ = hb_lost.cancelled() => {
// The heartbeat gave up renewing: the lease has very likely
// expired and may already be re-leased elsewhere. Abandon the
// dispatch rather than keep crawling a job we no longer own. Do
// NOT ack/release — a generation-guarded write would no-op, and
// another worker may now hold this lease.
heartbeat.abort();
tracing::error!(
worker = self.id,
lease_id = %lease.id,
"worker: abandoning dispatch — lease lost (heartbeat renew failures)"
);
return;
}
o = tokio::time::timeout(self.job_timeout, dispatch) => o,
};
heartbeat.abort();
@@ -502,6 +575,24 @@ impl WorkerContext {
self.status.poke();
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
}
Ok(Ok(SyncOutcome::BrowserUnavailable)) => {
// Infrastructure outage, not a job failure: the browser was
// down or mid-restart when the dispatcher tried to acquire it.
// Return the job to `pending` WITHOUT burning an attempt (like
// the cancel/session paths) so an outage doesn't chew the whole
// backlog to `dead`, then back off to avoid hot-looping while
// the browser recovers.
tracing::warn!(
worker = self.id,
lease_id = %lease.id,
"worker: browser unavailable — released lease without burning an attempt"
);
let _ = jobs::release(&self.pool, lease.id, lease.lease_generation).await;
tokio::select! {
_ = tokio::time::sleep(BROWSER_UNAVAILABLE_BACKOFF) => {}
_ = self.cancel.cancelled() => {}
}
}
Ok(Err(e)) => {
tracing::warn!(
worker = self.id,
@@ -750,6 +841,37 @@ mod tests {
Utc.with_ymd_and_hms(y, mo, d, h, mi, 0).unwrap()
}
#[test]
fn heartbeat_aborts_only_after_threshold_consecutive_failures() {
// A blip or two is tolerated; sustained failures escalate to abandon.
assert!(!should_abort_after_renew_failures(0));
assert!(!should_abort_after_renew_failures(1));
assert!(!should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES - 1));
assert!(should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES));
assert!(should_abort_after_renew_failures(MAX_HEARTBEAT_RENEW_FAILURES + 1));
}
#[test]
fn idle_backoff_grows_then_caps() {
// First miss is a short 1s poll; the interval doubles each empty poll…
assert_eq!(idle_backoff(0), Duration::from_secs(1));
assert_eq!(idle_backoff(1), Duration::from_secs(2));
assert_eq!(idle_backoff(2), Duration::from_secs(4));
assert_eq!(idle_backoff(3), Duration::from_secs(8));
assert_eq!(idle_backoff(4), Duration::from_secs(16));
// …and saturates at the cap rather than growing unbounded.
assert_eq!(idle_backoff(5), IDLE_BACKOFF_CAP);
assert_eq!(idle_backoff(100), IDLE_BACKOFF_CAP);
// Never exceeds the cap and is monotonic non-decreasing.
let mut prev = Duration::ZERO;
for n in 0..40 {
let b = idle_backoff(n);
assert!(b >= prev, "backoff must be non-decreasing at n={n}");
assert!(b <= IDLE_BACKOFF_CAP, "backoff must never exceed the cap");
prev = b;
}
}
#[test]
fn next_fire_in_utc_at_midnight_advances_one_day() {
let now = dt_utc(2026, 5, 25, 12, 0); // noon UTC

View File

@@ -0,0 +1,301 @@
//! Optional CDP-level SSRF guard for headless-browser navigations.
//!
//! The reqwest clients get a DNS-filtering resolver (see
//! [`crate::crawler::safety::SafeResolver`]) so an image/API host that
//! resolves to an internal IP is refused at connect time. Chromium, however,
//! does its **own** DNS and connection handling, so that resolver can't see
//! browser navigations. Without a second guard, a scraped chapter page that
//! `302`-redirects the browser to `http://127.0.0.1:5432/` (or an
//! attacker-owned hostname that resolves to `169.254.169.254`) would be
//! loaded and parsed as if it were catalog content.
//!
//! This module installs CDP `Fetch` interception on a page so **every** request
//! it issues — the main-frame Document, its redirects, *and every subresource*
//! (`<img>`, `fetch()`/XHR, media, …) — is re-validated through the same
//! [`ensure_public_target`] + resolved-IP check the reqwest paths use, failing
//! any that target an internal address. Intercepting only the Document would
//! leave a scraped page free to pull `<img src="http://169.254.169.254/…">` or
//! `fetch('http://postgres:5432')` straight past the guard, so no resource type
//! is exempt. (WebSocket handshakes are not surfaced by CDP `Fetch`, so `ws://`
//! internal targets remain out of this hook's reach — the reqwest-layer
//! resolver does not see them either; documented as a known gap.)
//!
//! **Opt-in / default-off.** Enabling `Fetch` means every intercepted request
//! *must* be resolved by a live handler or the navigation hangs, so this is a
//! fragile hook in the crawler's critical path. It ships behind
//! `CRAWLER_SSRF_INTERCEPT` (default `false`) and the wiring has **not** been
//! exercised against a real Chromium in CI — validate with a manual crawl
//! before enabling in production. When disabled, [`open_page`] is byte-for-byte
//! the previous `browser.new_page(url)` behavior. When enabled, the guard is
//! **fail-closed**: if interception can't be installed, [`open_page`] closes the
//! blank page and returns the error rather than navigating unguarded.
use std::net::IpAddr;
use std::sync::atomic::{AtomicBool, Ordering};
use chromiumoxide::browser::Browser;
use chromiumoxide::cdp::browser_protocol::fetch::{
ContinueRequestParams, EnableParams, EventRequestPaused, FailRequestParams, RequestPattern,
RequestStage,
};
use chromiumoxide::cdp::browser_protocol::network::ErrorReason;
use chromiumoxide::error::Result as CdpResult;
use chromiumoxide::Page;
use futures_util::StreamExt;
use reqwest::Url;
use crate::crawler::safety::{ensure_public_target, is_private_ip};
/// Process-wide toggle, set once at startup from `CRAWLER_SSRF_INTERCEPT`.
/// A single boot-time flag (rather than threading a param through every
/// crawler navigation signature) keeps the off-path a no-op.
static ENABLED: AtomicBool = AtomicBool::new(false);
pub fn set_enabled(on: bool) {
ENABLED.store(on, Ordering::Relaxed);
}
pub fn is_enabled() -> bool {
ENABLED.load(Ordering::Relaxed)
}
/// What to do with a navigation URL, decided without DNS where possible so the
/// security-critical branching is unit-testable.
#[derive(Debug, PartialEq, Eq)]
pub(crate) enum Verdict {
/// Safe to continue (non-network scheme, or a public IP literal).
Allow,
/// Refuse — bad scheme handled elsewhere, localhost, or a private IP literal.
Block,
/// A hostname that must be resolved to decide (DNS-rebinding check).
ResolveHost { host: String, port: u16 },
}
/// Pure decision over a URL string. `data:` / `blob:` / `about:` and other
/// non-http(s) schemes are allowed (not a network-SSRF vector); http(s) with a
/// private/loopback/localhost literal is blocked; an http(s) hostname needs
/// resolution.
pub(crate) fn verdict(url: &str) -> Verdict {
let Ok(parsed) = Url::parse(url) else {
// Unparseable — let Chromium reject it; not our call to make.
return Verdict::Allow;
};
match parsed.scheme() {
"http" | "https" => {}
// Non-http(s) subresource schemes: block the ones that can reach the
// local filesystem or pivot to another protocol; continue the rest
// (data:/blob:/about: and Chrome-internal schemes) which legitimate page
// loads depend on — blocking those would wedge navigation.
"file" | "ftp" | "gopher" => return Verdict::Block,
_ => return Verdict::Allow,
}
// Literal private IP / localhost / missing host → block (string check).
if ensure_public_target(url).is_err() {
return Verdict::Block;
}
match parsed.host_str() {
Some(host) if !is_ip_literal(host) => Verdict::ResolveHost {
host: host.to_string(),
port: parsed.port_or_known_default().unwrap_or(80),
},
// Public IP literal (ensure_public_target already passed it).
_ => Verdict::Allow,
}
}
/// Whether `host` (as reqwest's `host_str()` yields it) is an IP literal.
/// IPv6 literals arrive bracketed (`[::1]`), which don't parse as `IpAddr`
/// directly — strip the brackets first.
fn is_ip_literal(host: &str) -> bool {
let unbracketed = host
.strip_prefix('[')
.and_then(|s| s.strip_suffix(']'))
.unwrap_or(host);
unbracketed.parse::<IpAddr>().is_ok()
}
/// Async decision: resolves hostnames and blocks if any resolved address is
/// private (DNS rebinding). A resolution failure is *not* treated as blocked —
/// Chromium won't be able to connect either, so there's nothing to exfiltrate.
pub(crate) async fn is_blocked(url: &str) -> bool {
match verdict(url) {
Verdict::Allow => false,
Verdict::Block => true,
Verdict::ResolveHost { host, port } => match tokio::net::lookup_host((host.as_str(), port))
.await
{
Ok(addrs) => addrs.map(|a| a.ip()).any(|ip| is_private_ip(&ip)),
Err(_) => false,
},
}
}
/// Open a page for navigation. With interception off, this is exactly
/// `browser.new_page(url)`. With it on, the page is created blank (no network),
/// the navigation guard is installed, and only then does it navigate — so the
/// initial request and any redirects pass through the guard.
pub async fn open_page(browser: &Browser, url: &str) -> CdpResult<Page> {
if !is_enabled() {
return browser.new_page(url).await;
}
let page = browser.new_page("about:blank").await?;
if let Err(e) = install_navigation_guard(&page).await {
// Fail closed: an unguarded page could be redirected (or pull a
// subresource) to an internal target, so refuse to navigate. Close the
// blank page and surface the error so the caller aborts this fetch.
tracing::warn!(url = %url, error = %e, "SSRF navigation guard failed to install; aborting navigation (fail-closed)");
let _ = page.close().await;
return Err(e);
}
page.goto(url).await?;
Ok(page)
}
/// The `Fetch` interception patterns to register. A single pattern with **no**
/// `resource_type` constraint matches every request the page makes — Document,
/// Image, Fetch/XHR, media, everything — so subresources to internal targets are
/// re-validated too, not only the main-frame navigation. Restricting this to
/// `ResourceType::Document` (the previous behavior) left every `<img>`/`fetch()`
/// subresource unguarded, which is the SSRF hole this closes. Pinned to the
/// request stage so each request is paused once, before it leaves the browser.
fn interception_patterns() -> Vec<RequestPattern> {
vec![RequestPattern::builder()
.request_stage(RequestStage::Request)
.build()]
}
/// Enable `Fetch` for all requests on `page` and spawn a task that
/// continues/fails each paused request per [`is_blocked`].
async fn install_navigation_guard(page: &Page) -> CdpResult<()> {
page.execute(EnableParams {
patterns: Some(interception_patterns()),
handle_auth_requests: None,
})
.await?;
let mut paused = page.event_listener::<EventRequestPaused>().await?;
let handler_page = page.clone();
tokio::spawn(async move {
while let Some(ev) = paused.next().await {
let request_id = ev.request_id.clone();
let outcome = if is_blocked(&ev.request.url).await {
tracing::warn!(
url = %ev.request.url,
"SSRF guard blocked browser navigation to an internal target"
);
handler_page
.execute(FailRequestParams::new(request_id, ErrorReason::BlockedByClient))
.await
.map(|_| ())
} else {
handler_page
.execute(ContinueRequestParams::new(request_id))
.await
.map(|_| ())
};
if let Err(e) = outcome {
// The page/session is gone (page closed) — the stream will end
// too; stop handling so the task exits rather than spins.
tracing::debug!(error = %e, "fetch interceptor: continue/fail failed, ending handler");
break;
}
}
});
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn verdict_allows_non_network_schemes() {
for url in ["about:blank", "data:text/html,hi", "blob:abc", "chrome://version"] {
assert_eq!(verdict(url), Verdict::Allow, "{url} should be allowed");
}
}
#[test]
fn verdict_blocks_dangerous_non_http_schemes() {
// file:/ftp:/gopher: can reach the local filesystem or pivot protocols;
// block them explicitly rather than falling through to Allow.
for url in ["file:///etc/passwd", "ftp://internal/x", "gopher://internal:70/1"] {
assert_eq!(verdict(url), Verdict::Block, "{url} should be blocked");
}
}
#[test]
fn verdict_blocks_private_ip_literals_and_localhost() {
for url in [
"http://127.0.0.1/",
"http://10.0.0.1/",
"http://192.168.1.1:5432/",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://localhost/",
"https://[fd00::1]/",
] {
assert_eq!(verdict(url), Verdict::Block, "{url} should be blocked");
}
}
#[test]
fn verdict_allows_public_ip_literals() {
for url in ["http://93.184.216.34/", "https://[2606:4700:4700::1111]/"] {
assert_eq!(verdict(url), Verdict::Allow, "{url} should be allowed");
}
}
#[test]
fn verdict_defers_hostnames_to_resolution() {
assert_eq!(
verdict("https://cdn.example.com/img.jpg"),
Verdict::ResolveHost {
host: "cdn.example.com".to_string(),
port: 443
}
);
assert_eq!(
verdict("http://catalog.test:8080/list"),
Verdict::ResolveHost {
host: "catalog.test".to_string(),
port: 8080
}
);
}
#[tokio::test]
async fn is_blocked_true_for_private_literal_no_dns() {
assert!(is_blocked("http://169.254.169.254/").await);
assert!(!is_blocked("http://93.184.216.34/").await);
// Non-network scheme is always allowed through.
assert!(!is_blocked("about:blank").await);
}
#[test]
fn interception_covers_all_resource_types() {
// Regression guard for the SSRF subresource hole: the CDP Fetch pattern
// must NOT be constrained to Document, or `<img>`/`fetch()`/XHR
// subresources to internal targets slip past `is_blocked`. A pattern
// with no `resource_type` set intercepts every request type.
let patterns = interception_patterns();
assert_eq!(patterns.len(), 1);
assert!(
patterns[0].resource_type.is_none(),
"interception must cover all resource types (subresources included), not just Document"
);
assert_eq!(patterns[0].request_stage, Some(RequestStage::Request));
}
#[test]
fn enabled_toggle_roundtrips() {
// Global; restore afterwards so other tests see the default.
let prev = is_enabled();
set_enabled(true);
assert!(is_enabled());
set_enabled(false);
assert!(!is_enabled());
set_enabled(prev);
}
}

View File

@@ -132,14 +132,17 @@ fn backoff_for(attempts: i32) -> Duration {
/// `Skipped`. The slot frees again once the previous job leaves the
/// in-flight states (done/failed/dead), so a re-enqueue after a force
/// refetch succeeds.
pub async fn enqueue(pool: &PgPool, payload: &JobPayload) -> sqlx::Result<EnqueueResult> {
pub async fn enqueue<'e, E>(executor: E, payload: &JobPayload) -> sqlx::Result<EnqueueResult>
where
E: sqlx::PgExecutor<'e>,
{
let json = serde_json::to_value(payload).expect("JobPayload is always serializable");
let id: Option<Uuid> = sqlx::query_scalar(
"INSERT INTO crawler_jobs (payload) VALUES ($1) \
ON CONFLICT DO NOTHING RETURNING id",
)
.bind(json)
.fetch_optional(pool)
.fetch_optional(executor)
.await?;
Ok(match id {
Some(id) => EnqueueResult::Inserted(id),
@@ -422,7 +425,10 @@ pub async fn release(
/// the original worker becomes a no-op. Returns the attempt to
/// `pending` and refunds the retry attempt (the operator click
/// isn't a job-level failure).
pub async fn release_unowned(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
pub async fn release_unowned<'e, E>(executor: E, lease_id: Uuid) -> sqlx::Result<()>
where
E: sqlx::PgExecutor<'e>,
{
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'pending', leased_until = NULL, \
@@ -432,7 +438,7 @@ pub async fn release_unowned(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()>
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.execute(pool)
.execute(executor)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
@@ -479,22 +485,43 @@ pub async fn reclaim_orphaned(pool: &PgPool) -> sqlx::Result<u64> {
Ok(result.rows_affected())
}
/// Delete `done` jobs whose `updated_at` is older than `retention_days`
/// days. `0` disables the reaper without touching the table. Returns the
/// number of rows removed.
pub async fn reap_done(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
/// Delete **terminal** jobs (`done` or `dead`) whose `updated_at` is older
/// than `retention_days` days. Both states are end-of-life — `done`
/// succeeded, `dead` exhausted its retries — and neither is ever leased
/// again, so reaping only `done` left `dead` rows to accumulate forever.
/// `pending` / `running` are active and never touched. `0` disables the
/// reaper without touching the table. Returns the number of rows removed.
pub async fn reap_terminal(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
if retention_days == 0 {
return Ok(0);
}
let result = sqlx::query(
"DELETE FROM crawler_jobs \
WHERE state = 'done' \
AND updated_at < now() - ($1::bigint || ' days')::interval",
)
.bind(retention_days as i64)
.execute(pool)
.await?;
Ok(result.rows_affected())
// Delete in bounded batches rather than one statement. A single unbatched
// DELETE over an unbounded backlog takes a long-held lock and one huge
// transaction; batching keeps each delete short (index-backed by
// crawler_jobs_terminal_reap_idx, 0040) so the reaper never pins the table
// (audit M5). Loop until a batch comes back short.
const BATCH: i64 = 5_000;
let mut total: u64 = 0;
loop {
let result = sqlx::query(
"DELETE FROM crawler_jobs \
WHERE ctid IN ( \
SELECT ctid FROM crawler_jobs \
WHERE state IN ('done', 'dead') \
AND updated_at < now() - ($1::bigint || ' days')::interval \
LIMIT $2 )",
)
.bind(retention_days as i64)
.bind(BATCH)
.execute(pool)
.await?;
let n = result.rows_affected();
total += n;
if n < BATCH as u64 {
break;
}
}
Ok(total)
}
#[cfg(test)]

View File

@@ -19,6 +19,7 @@ pub mod content;
pub mod daemon;
pub mod detect;
pub mod diff;
pub mod intercept;
pub mod jobs;
pub mod nav;
pub mod pipeline;

View File

@@ -85,6 +85,28 @@ pub async fn wait_for_selector(
/// under a second.
pub const SELECTOR_TIMEOUT: Duration = Duration::from_secs(10);
/// Run `body` to completion, then run `close` — on *every* exit path,
/// including an early `?` / `return` inside `body`. chromiumoxide's `Page`
/// does **not** close its CDP target on drop, so a fetch helper that opens a
/// page with `browser.new_page(...)` and then bails via `?` on a nav /
/// content-read error would leak a browser tab for the process's lifetime.
/// Wrapping the fallible work in `close_after` — with `close` built from a
/// clone of the page — guarantees the tab is closed regardless of how `body`
/// returns.
///
/// Both arguments are pre-built futures, so this stays generic over the
/// body's return type (the anyhow and `PageError` fetch paths both use it)
/// and references no browser types — which also lets it be unit-tested
/// without standing up a real `Page`.
pub(crate) async fn close_after<R>(
close: impl std::future::Future<Output = ()>,
body: impl std::future::Future<Output = R>,
) -> R {
let result = body.await;
close.await;
result
}
impl NavError {
/// Does this navigation error indicate the underlying Chromium
/// process has died or its CDP connection has dropped? Used by the
@@ -150,6 +172,41 @@ mod tests {
assert!(result.is_err(), "expected Elapsed on a hung future");
}
#[tokio::test]
async fn close_after_runs_close_and_returns_value_on_ok() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let closed = Arc::new(AtomicBool::new(false));
let c = closed.clone();
let out: anyhow::Result<i32> = close_after(
async move { c.store(true, Ordering::SeqCst) },
async { Ok(7) },
)
.await;
assert_eq!(out.unwrap(), 7);
assert!(closed.load(Ordering::SeqCst), "close must run on the happy path");
}
#[tokio::test]
async fn close_after_runs_close_even_when_body_errs() {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::Arc;
let closed = Arc::new(AtomicBool::new(false));
let c = closed.clone();
// This is the leak the fix targets: the body bails before it could
// close the page itself, and `close_after` must still close it.
let out: anyhow::Result<()> =
close_after(async move { c.store(true, Ordering::SeqCst) }, async {
anyhow::bail!("nav failed")
})
.await;
assert!(out.is_err());
assert!(
closed.load(Ordering::SeqCst),
"the page must be closed even when the body returns Err"
);
}
#[test]
fn nav_error_timeout_message_includes_duration() {
let e = NavError::Timeout(Duration::from_secs(30));
@@ -164,8 +221,7 @@ mod tests {
#[test]
fn anyhow_with_nav_timeout_in_chain_is_flagged() {
let inner: Result<(), NavError> = Err(NavError::Timeout(NAV_TIMEOUT));
let outer = inner.unwrap_err();
let outer = NavError::Timeout(NAV_TIMEOUT);
let wrapped: anyhow::Error =
anyhow::Error::new(outer).context("wait for chapter nav");
assert!(anyhow_looks_browser_dead(&wrapped));

View File

@@ -44,6 +44,25 @@ impl RateLimiter {
}
}
/// Once the per-host map grows past this many entries, the next `wait_for`
/// sweeps out hosts idle longer than [`IDLE_EVICT_AFTER`]. A long crawl that
/// touches thousands of distinct CDN shards would otherwise retain one bucket
/// per host for the daemon's whole lifetime. Far above any single source's
/// real host count, so the sweep is rare.
const MAX_TRACKED_HOSTS: usize = 1024;
/// A host bucket untouched for at least this long is evicted on the next
/// over-cap sweep. Generous: a host still being crawled is touched every
/// `interval`, so only genuinely-finished hosts age out.
const IDLE_EVICT_AFTER: Duration = Duration::from_secs(3600);
/// A host's bucket plus when it was last used, so idle entries can be evicted.
#[derive(Debug)]
struct HostEntry {
limiter: Arc<Mutex<RateLimiter>>,
last_used: Instant,
}
/// Per-host rate limiter map. The outer `Mutex<HashMap>` is held only
/// during the entry-or-insert + Arc clone; the per-host `Mutex<RateLimiter>`
/// is held during the actual `wait().await`. So N workers calling
@@ -54,7 +73,7 @@ impl RateLimiter {
pub struct HostRateLimiters {
default_interval: Duration,
overrides: HashMap<String, Duration>,
map: Mutex<HashMap<String, Arc<Mutex<RateLimiter>>>>,
map: Mutex<HashMap<String, HostEntry>>,
}
impl HostRateLimiters {
@@ -82,20 +101,37 @@ impl HostRateLimiters {
.ok_or_else(|| anyhow::anyhow!("no host in url: {url}"))?;
let limiter = {
let mut map = self.map.lock().await;
map.entry(host.clone())
.or_insert_with(|| {
let interval = self
.overrides
.get(&host)
.copied()
.unwrap_or(self.default_interval);
Arc::new(Mutex::new(RateLimiter::new(interval)))
})
.clone()
let now = Instant::now();
// Bound the map: when it grows past the soft cap, drop hosts that
// have been idle past the TTL. Only fires over-cap, so the common
// path is a plain lookup. A host still being crawled is touched
// every `interval` and so never ages out.
if map.len() >= MAX_TRACKED_HOSTS {
map.retain(|_, e| now.duration_since(e.last_used) < IDLE_EVICT_AFTER);
}
let entry = map.entry(host.clone()).or_insert_with(|| {
let interval = self
.overrides
.get(&host)
.copied()
.unwrap_or(self.default_interval);
HostEntry {
limiter: Arc::new(Mutex::new(RateLimiter::new(interval))),
last_used: now,
}
});
entry.last_used = now;
entry.limiter.clone()
};
limiter.lock().await.wait().await;
Ok(())
}
/// Number of host buckets currently tracked. Test/observability hook.
#[cfg(test)]
pub(crate) async fn tracked_hosts(&self) -> usize {
self.map.lock().await.len()
}
}
// `host_of` was duplicated across session/rate_limit/pipeline; the
@@ -165,6 +201,46 @@ mod tests {
);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_evict_idle_hosts_over_cap() {
// Fill the map to the soft cap with distinct hosts (first call to a
// fresh host never sleeps, so this is fast even under real time).
let rl = HostRateLimiters::new(Duration::from_millis(1));
for i in 0..MAX_TRACKED_HOSTS {
rl.wait_for(&format!("https://host{i}.example/x")).await.unwrap();
}
assert_eq!(rl.tracked_hosts().await, MAX_TRACKED_HOSTS);
// Let every tracked host age past the idle TTL, then touch one new
// host: the over-cap sweep should evict all the idle ones, leaving
// only the freshly-inserted entry.
tokio::time::sleep(IDLE_EVICT_AFTER + Duration::from_secs(1)).await;
rl.wait_for("https://newcomer.example/y").await.unwrap();
assert_eq!(
rl.tracked_hosts().await,
1,
"idle hosts should be swept once the map is over the cap"
);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_keep_recently_used_hosts() {
// A host touched within the TTL must survive a sweep so active crawls
// aren't reset. Fill to the cap, then re-touch one host right before
// adding a newcomer that triggers the sweep.
let rl = HostRateLimiters::new(Duration::from_millis(1));
for i in 0..MAX_TRACKED_HOSTS {
rl.wait_for(&format!("https://host{i}.example/x")).await.unwrap();
}
tokio::time::sleep(IDLE_EVICT_AFTER + Duration::from_secs(1)).await;
// Re-touch host0 so it's recent again.
rl.wait_for("https://host0.example/x").await.unwrap();
// Newcomer triggers the sweep (map is at cap+1 conceptually).
rl.wait_for("https://newcomer.example/y").await.unwrap();
// host0 (recent) + newcomer survive; the rest aged out.
assert_eq!(rl.tracked_hosts().await, 2);
}
#[tokio::test(start_paused = true)]
async fn host_rate_limiters_honor_overrides() {
let rl = HostRateLimiters::new(Duration::from_millis(1000))

View File

@@ -81,6 +81,8 @@ pub struct RealResyncService {
pub rate: Arc<HostRateLimiters>,
pub download_allowlist: DownloadAllowlist,
pub max_image_bytes: usize,
/// Per-chapter image-count cap (see `CrawlerConfig::max_images_per_chapter`).
pub max_images_per_chapter: usize,
pub tor: Option<Arc<TorController>>,
}
@@ -133,28 +135,7 @@ impl ResyncService for RealResyncService {
.await
.with_context(|| format!("fetch_manga during resync of {manga_id}"))?;
// Partial-render guard: same logic as run_metadata_pass.
let source_id = source.id();
if !manga.chapters.is_empty() || {
let prior = repo::crawler::live_chapter_count_for_source_manga(
&self.db,
source_id,
&source_manga_key,
)
.await
.unwrap_or(0);
prior == 0
} {
// Either the new fetch surfaced chapters, or there were
// none before either — chapter sync is safe to run.
} else {
tracing::warn!(
%manga_id,
source_url = %source_url,
"resync_manga: fetch returned empty chapters but prior count > 0; skipping chapter sync to avoid soft-drop"
);
}
let upsert = repo::crawler::upsert_manga_from_source(
&self.db,
source_id,
@@ -199,6 +180,11 @@ impl ResyncService for RealResyncService {
)
.await
.unwrap_or(0);
// Partial-render guard (same logic as run_metadata_pass): only sync
// chapters when the fetch surfaced some, or the manga never had any.
// A fetch that returned empty while a prior count exists is almost
// certainly a partial render — skip the sync so we don't soft-drop the
// real chapters.
if !manga.chapters.is_empty() || prior_chapter_count == 0 {
match repo::crawler::sync_manga_chapters(
&self.db,
@@ -221,6 +207,12 @@ impl ResyncService for RealResyncService {
"resync_manga: chapter sync failed"
),
}
} else {
tracing::warn!(
%manga_id,
source_url = %source_url,
"resync_manga: fetch returned empty chapters but prior count > 0; skipping chapter sync to avoid soft-drop"
);
}
drop(lease);
@@ -256,6 +248,7 @@ impl ResyncService for RealResyncService {
true,
&self.download_allowlist,
self.max_image_bytes,
self.max_images_per_chapter,
self.tor.as_deref(),
// Admin resync isn't a daemon worker slot — no live status.
None,
@@ -277,6 +270,12 @@ impl ResyncService for RealResyncService {
SyncOutcome::SessionExpired => {
anyhow::bail!("source session expired — operator must refresh PHPSESSID")
}
// Unreachable here: resync already holds its own browser lease and
// `sync_chapter_content` never acquires one, so it can't report the
// browser unavailable. Handled defensively for exhaustiveness.
SyncOutcome::BrowserUnavailable => {
anyhow::bail!("crawler browser unavailable")
}
}
}
}

View File

@@ -31,11 +31,13 @@
//! URL string, and the byte accumulator is keyed off a generic stream.
//! Easy to unit-test without a live network or browser.
use std::net::IpAddr;
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
use std::sync::Arc;
use anyhow::{bail, Context};
use bytes::BytesMut;
use futures_util::StreamExt;
use reqwest::dns::{Addrs, Name, Resolve, Resolving};
use reqwest::Url;
/// Default per-image download cap. A page image is generally <2 MiB;
@@ -179,7 +181,7 @@ fn ensure_public_target_inner(raw_url: &str) -> Result<Url, UrlSafetyError> {
Ok(url)
}
fn is_private_ip(ip: &IpAddr) -> bool {
pub(crate) fn is_private_ip(ip: &IpAddr) -> bool {
match ip {
IpAddr::V4(v4) => {
v4.is_loopback()
@@ -193,11 +195,14 @@ fn is_private_ip(ip: &IpAddr) -> bool {
|| v4.octets()[0] == 0
}
IpAddr::V6(v6) => {
// IPv4-mapped IPv6 (::ffff:0:0/96): unwrap to the embedded
// IPv4 and recurse so `::ffff:127.0.0.1` is caught by the
// IPv4 loopback check rather than passing through.
// `Ipv6Addr::is_loopback()` only matches `::1` exactly.
if let Some(v4) = v6.to_ipv4_mapped() {
// Any IPv6 form that *embeds* an IPv4 address (mapped
// `::ffff:0:0/96`, compatible `::/96`, NAT64 `64:ff9b::/96`,
// 6to4 `2002::/16`) is unwrapped and re-checked as its IPv4 —
// otherwise `::127.0.0.1` / `2002:7f00:1::` / `64:ff9b::7f00:1`
// would smuggle an internal IPv4 past the check (the audit's
// IPv6-embedding gap). `Ipv6Addr::is_loopback()` only matches
// `::1` exactly, so these embeddings need explicit handling.
if let Some(v4) = embedded_ipv4(v6) {
return is_private_ip(&IpAddr::V4(v4));
}
v6.is_loopback()
@@ -210,6 +215,114 @@ fn is_private_ip(ip: &IpAddr) -> bool {
}
}
/// Extract the IPv4 address embedded in an IPv6 literal, for every
/// transitional encoding that can carry one: IPv4-mapped (`::ffff:0:0/96`),
/// IPv4-compatible (`::/96`, deprecated but still routable via some stacks),
/// NAT64 (`64:ff9b::/96`), and 6to4 (`2002::/16`). Returns `None` for a
/// native IPv6 address. Callers recurse into [`is_private_ip`] on the result
/// so a private IPv4 can't hide inside an IPv6 literal.
fn embedded_ipv4(v6: &Ipv6Addr) -> Option<Ipv4Addr> {
let seg = v6.segments();
let low32 = |a: u16, b: u16| Ipv4Addr::new((a >> 8) as u8, (a & 0xff) as u8, (b >> 8) as u8, (b & 0xff) as u8);
// ::ffff:0:0/96 (mapped) and ::/96 (compatible) — top 96 bits zero,
// except mapped which has 0xffff at seg[5]. to_ipv4() covers both.
if seg[0..5] == [0, 0, 0, 0, 0] && (seg[5] == 0 || seg[5] == 0xffff) {
return Some(low32(seg[6], seg[7]));
}
// NAT64 64:ff9b::/96
if seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0] {
return Some(low32(seg[6], seg[7]));
}
// 6to4 2002::/16 — embedded IPv4 is bits 16..48 (seg[1], seg[2]).
if seg[0] == 0x2002 {
return Some(low32(seg[1], seg[2]));
}
None
}
/// A `reqwest::dns::Resolve` that performs normal system resolution, then
/// drops any resolved address in a private / loopback / link-local / metadata
/// range. Installed on every crawler + analysis reqwest client so a hostname
/// that resolves to an internal IP (DNS rebinding: an attacker-owned domain
/// with an `A` record for `169.254.169.254` or `10.x`) can never be connected
/// to — closing the TOCTOU gap that the string-only `ensure_public_target`
/// check leaves open. Fires per connection, so it also guards redirect hops.
#[derive(Debug, Default)]
pub struct SafeResolver;
/// Partition resolved addresses into the public ones, rejecting when nothing
/// survives. Split out from the async `resolve` so the security-critical
/// filter is unit-testable without real DNS.
fn retain_public_addrs(
host: &str,
addrs: impl Iterator<Item = SocketAddr>,
) -> Result<Vec<SocketAddr>, BlockedResolution> {
let public: Vec<SocketAddr> = addrs.filter(|a| !is_private_ip(&a.ip())).collect();
if public.is_empty() {
return Err(BlockedResolution(host.to_string()));
}
Ok(public)
}
#[derive(Debug, thiserror::Error)]
#[error("host {0} resolved only to private/blocked addresses")]
struct BlockedResolution(String);
impl Resolve for SafeResolver {
fn resolve(&self, name: Name) -> Resolving {
Box::pin(async move {
let host = name.as_str().to_string();
// Port 0: reqwest overrides it with the URL's port after
// resolution (same convention as reqwest's default GaiResolver).
let resolved = tokio::net::lookup_host((host.as_str(), 0))
.await
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
let public = retain_public_addrs(&host, resolved)
.map_err(|e| Box::new(e) as Box<dyn std::error::Error + Send + Sync>)?;
Ok(Box::new(public.into_iter()) as Addrs)
})
}
}
/// Shared [`SafeResolver`] for wiring into `ClientBuilder::dns_resolver`.
pub fn safe_dns_resolver() -> Arc<SafeResolver> {
Arc::new(SafeResolver)
}
/// Whether the DNS-rebinding [`SafeResolver`] should be attached to a crawler
/// reqwest client given the configured proxy (if any).
///
/// The resolver only guards targets that *reqwest itself* resolves:
/// - **No proxy** (direct): reqwest resolves the target — attach.
/// - **`http(s)://` proxy**: reqwest still resolves the target host locally and
/// issues a `CONNECT`, so a hostname resolving to a private IP must be
/// refused — attach.
/// - **`socks5://` / `socks5h://` / `socks4://` (Tor)**: the *proxy* resolves
/// the target; reqwest only ever resolves the proxy's own host, which
/// legitimately lives on a private Docker IP (e.g. `tor` → 172.x). Attaching
/// the resolver there rejects every fetch for zero security gain (a SOCKS
/// proxy can't route into internal ranges anyway) — do **not** attach.
///
/// This narrows commit 134ab54, which dropped the resolver for *any* proxy, back
/// to SOCKS-only so the http(s)-proxy path keeps its DNS-rebinding guard.
pub fn should_attach_safe_resolver(proxy: Option<&str>) -> bool {
match proxy {
None => true,
Some(p) => !is_socks_proxy(p),
}
}
/// True when `proxy`'s scheme is a SOCKS variant (`socks4`, `socks5`,
/// `socks5h`). A scheme-less value (reqwest treats it as HTTP) is not SOCKS.
fn is_socks_proxy(proxy: &str) -> bool {
proxy
.split_once("://")
.map(|(scheme, _)| scheme)
.unwrap_or("")
.to_ascii_lowercase()
.starts_with("socks")
}
#[derive(Debug, thiserror::Error, PartialEq, Eq)]
pub enum UrlSafetyError {
#[error("URL is not parseable")]
@@ -226,6 +339,73 @@ pub enum UrlSafetyError {
HostNotAllowed(String),
}
/// Maximum number of redirects the crawler will follow before giving up.
/// Matches reqwest's historical default; every hop is re-validated by
/// [`check_redirect_hop`], so the cap is a belt-and-braces stop against a
/// redirect loop rather than the primary SSRF defence.
pub const MAX_REDIRECTS: usize = 10;
/// Why a redirect hop was refused.
#[derive(Debug, thiserror::Error)]
pub enum RedirectError {
#[error("redirect chain exceeded {0} hops")]
TooManyHops(usize),
#[error("redirect target rejected: {0}")]
Unsafe(#[from] UrlSafetyError),
}
/// Decide whether a single redirect hop is safe to follow.
///
/// `is_safe_url` only inspects the *initial* URL a caller hands to reqwest;
/// without re-validation an allowlisted CDN that answers `302 ->
/// http://169.254.169.254/...` or `-> http://127.0.0.1:5432/` would be
/// followed transparently (reqwest's default policy follows up to 10
/// redirects). This re-runs the full allowlist + private-IP + scheme check on
/// each hop and enforces [`MAX_REDIRECTS`]. `completed_hops` is the number of
/// URLs already visited in the chain (reqwest's `attempt.previous().len()`).
pub fn check_redirect_hop(
next_url: &str,
completed_hops: usize,
allow: &DownloadAllowlist,
) -> Result<(), RedirectError> {
if completed_hops >= MAX_REDIRECTS {
return Err(RedirectError::TooManyHops(completed_hops));
}
is_safe_url(next_url, allow)?;
Ok(())
}
/// Build a reqwest redirect policy that re-validates every hop against the
/// download allowlist (see [`check_redirect_hop`]). Use for the crawler image
/// clients, which fetch attacker-influenced URLs.
pub fn safe_redirect_policy(allow: DownloadAllowlist) -> reqwest::redirect::Policy {
reqwest::redirect::Policy::custom(move |attempt| {
let hops = attempt.previous().len();
match check_redirect_hop(attempt.url().as_str(), hops, &allow) {
Ok(()) => attempt.follow(),
Err(e) => attempt.error(e),
}
})
}
/// Build a reqwest redirect policy that re-validates every hop with
/// [`ensure_public_target`] (scheme + private-IP, no allowlist). Use for
/// single-endpoint clients (the analysis vision endpoint / its probe) where
/// there is no per-deployment allowlist but a redirect into the deployment's
/// internal network must still be refused.
pub fn public_redirect_policy() -> reqwest::redirect::Policy {
reqwest::redirect::Policy::custom(move |attempt| {
let hops = attempt.previous().len();
if hops >= MAX_REDIRECTS {
return attempt.error(RedirectError::TooManyHops(hops));
}
match ensure_public_target(attempt.url().as_str()) {
Ok(()) => attempt.follow(),
Err(e) => attempt.error(RedirectError::Unsafe(e)),
}
})
}
/// Drain a byte stream into a single buffer, bailing out as soon as
/// the running total exceeds `max_bytes`. Generic over the stream so
/// it's testable without a live HTTP response.
@@ -327,6 +507,26 @@ mod tests {
DownloadAllowlist::new().allow(host)
}
#[test]
fn rejects_alternate_ipv4_encodings_of_loopback() {
// The `url` crate normalizes special-scheme IPv4 hosts per WHATWG, so
// decimal/hex/octal/short forms all collapse to 127.0.0.1 and must be
// rejected. This normalization is load-bearing (it's what stops these
// encodings smuggling past the literal-IP check) but was untested — pin
// it so a future URL-parser swap can't silently reopen the hole.
for url in [
"http://2130706433/", // decimal 127.0.0.1
"http://0x7f000001/", // hex
"http://0177.0.0.1/", // octal first octet
"http://127.1/", // short form
] {
assert!(
ensure_public_target(url).is_err(),
"{url} normalizes to loopback and must be rejected"
);
}
}
#[test]
fn allow_any_admits_arbitrary_public_host() {
// Operators who can't pre-enumerate a numbered-CDN fleet
@@ -542,6 +742,79 @@ mod tests {
assert!(matches!(err, UrlSafetyError::PrivateIp(_)));
}
#[test]
fn is_private_ip_unwraps_embedded_ipv4_encodings() {
// Every IPv6 encoding that can smuggle an internal IPv4 must be
// caught. The audit flagged compatible ::/96, NAT64, and 6to4 as
// gaps past the original mapped-only handling.
for s in [
"::ffff:127.0.0.1", // IPv4-mapped loopback
"::127.0.0.1", // IPv4-compatible loopback (was a gap)
"::ffff:10.1.2.3", // mapped RFC1918
"::10.1.2.3", // compatible RFC1918 (was a gap)
"64:ff9b::7f00:1", // NAT64 of 127.0.0.1 (was a gap)
"64:ff9b::a01:203", // NAT64 of 10.1.2.3
"2002:7f00:1::", // 6to4 of 127.0.0.1 (was a gap)
"2002:a01:203::", // 6to4 of 10.1.2.3
"2002:a9fe:a9fe::", // 6to4 of 169.254.169.254 (metadata)
] {
let ip: IpAddr = s.parse().unwrap();
assert!(is_private_ip(&ip), "{s} must be flagged private");
}
}
#[test]
fn is_private_ip_allows_public_embedded_and_native_ipv6() {
// A public IPv4 embedded in IPv6, and a native public IPv6, must
// NOT be flagged — the unwrap only blocks when the embedded v4 is
// itself private.
for s in [
"::ffff:8.8.8.8", // mapped public
"2002:808:808::", // 6to4 of 8.8.8.8 (public)
"2606:4700:4700::1111", // native public (Cloudflare)
] {
let ip: IpAddr = s.parse().unwrap();
assert!(!is_private_ip(&ip), "{s} must be allowed");
}
}
#[test]
fn retain_public_addrs_drops_private_and_errors_when_all_private() {
use std::net::{Ipv4Addr, SocketAddr};
let pub_addr = SocketAddr::from((Ipv4Addr::new(93, 184, 216, 34), 0));
let loopback = SocketAddr::from((Ipv4Addr::new(127, 0, 0, 1), 0));
let metadata = SocketAddr::from((Ipv4Addr::new(169, 254, 169, 254), 0));
// Mixed result keeps only the public address.
let kept =
retain_public_addrs("mixed.example", [pub_addr, loopback, metadata].into_iter())
.expect("public address survives");
assert_eq!(kept, vec![pub_addr]);
// All-private (DNS rebinding to internal) is rejected outright.
let err =
retain_public_addrs("rebind.attacker", [loopback, metadata].into_iter()).unwrap_err();
assert!(err.to_string().contains("rebind.attacker"));
}
#[test]
fn safe_resolver_attaches_except_for_socks_proxies() {
// Direct + http(s) proxies: reqwest resolves the target, so the
// DNS-rebinding guard must be attached.
assert!(should_attach_safe_resolver(None));
assert!(should_attach_safe_resolver(Some("http://proxy.internal:8080")));
assert!(should_attach_safe_resolver(Some("https://proxy.internal:8080")));
assert!(should_attach_safe_resolver(Some("HTTP://Proxy:8080")));
// A scheme-less proxy is treated as HTTP by reqwest — keep the guard.
assert!(should_attach_safe_resolver(Some("proxy.internal:8080")));
// SOCKS variants (incl. Tor): the proxy resolves, so attaching the
// resolver would reject every fetch for zero gain.
assert!(!should_attach_safe_resolver(Some("socks5://tor:9050")));
assert!(!should_attach_safe_resolver(Some("socks5h://tor:9050")));
assert!(!should_attach_safe_resolver(Some("socks4://x:1080")));
assert!(!should_attach_safe_resolver(Some("SOCKS5://Tor:9050")));
}
#[test]
fn safe_url_blocks_non_http_schemes() {
let allow = allow_just("anywhere");
@@ -578,6 +851,54 @@ mod tests {
assert!(is_safe_url("https://CDN.EXAMPLE.com/x.jpg", &allow).is_ok());
}
// --- redirect-hop re-validation (SSRF via 3xx) ---
#[test]
fn redirect_hop_allows_listed_public_target() {
let allow = allow_just("cdn.example.com");
assert!(check_redirect_hop("https://cdn.example.com/next.jpg", 1, &allow).is_ok());
}
#[test]
fn redirect_hop_blocks_private_ip_target() {
// The core SSRF case: an allowlisted CDN 302s to the cloud metadata
// service / an intra-compose port. Must be refused mid-chain.
let allow = allow_just("cdn.example.com");
for url in ["http://169.254.169.254/", "http://127.0.0.1:5432/", "http://10.0.0.1/"] {
let err = check_redirect_hop(url, 1, &allow).unwrap_err();
assert!(
matches!(err, RedirectError::Unsafe(UrlSafetyError::PrivateIp(_))),
"expected PrivateIp for {url}, got {err:?}"
);
}
}
#[test]
fn redirect_hop_blocks_off_allowlist_public_host() {
// Per the strict policy: a redirect to an unlisted *public* host is
// also refused (allow_any covers the numbered-CDN case instead).
let allow = allow_just("cdn.example.com");
let err = check_redirect_hop("https://evil.example.org/x", 1, &allow).unwrap_err();
assert!(matches!(err, RedirectError::Unsafe(UrlSafetyError::HostNotAllowed(_))));
}
#[test]
fn redirect_hop_blocks_bad_scheme_target() {
let allow = DownloadAllowlist::allow_any();
let err = check_redirect_hop("file:///etc/passwd", 1, &allow).unwrap_err();
assert!(matches!(err, RedirectError::Unsafe(UrlSafetyError::BadScheme(_))));
}
#[test]
fn redirect_hop_caps_chain_length() {
let allow = allow_just("cdn.example.com");
// A safe target is still refused once the hop cap is reached, so a
// redirect loop can't spin forever.
let err = check_redirect_hop("https://cdn.example.com/x", MAX_REDIRECTS, &allow)
.unwrap_err();
assert!(matches!(err, RedirectError::TooManyHops(n) if n == MAX_REDIRECTS));
}
#[tokio::test]
async fn accumulate_capped_returns_full_body_under_cap() {
let chunks: Vec<Result<bytes::Bytes, std::io::Error>> = vec![

View File

@@ -285,25 +285,39 @@ where
}
async fn fetch_probe_html(browser: &Browser, probe_url: &str) -> anyhow::Result<String> {
let page = browser
.new_page(probe_url)
// Guard the probe navigation for parity with the list/detail and
// chapter-content paths — the probe URL is operator-controlled, but
// keeping every `new_page` behind the same SSRF check avoids a gap if
// the URL ever becomes attacker-influenced.
crate::crawler::safety::ensure_public_target(probe_url)
.with_context(|| format!("refuse to navigate unsafe probe URL {probe_url}"))?;
let page = crate::crawler::intercept::open_page(browser, probe_url)
.await
.with_context(|| format!("open probe page {probe_url}"))?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for nav on probe")?;
// Best-effort wait for the layout marker. Timeout is fine — the
// probe classifier handles a missing `#logo` as Transient anyway,
// and the verify loop retries on Transient.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"#logo",
crate::crawler::nav::SELECTOR_TIMEOUT,
// Close the tab on every exit path — a `?` on wait_for_nav / content()
// would otherwise leak it (chromiumoxide doesn't close on drop).
let closer = page.clone();
crate::crawler::nav::close_after(
async move {
closer.close().await.ok();
},
async {
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for nav on probe")?;
// Best-effort wait for the layout marker. Timeout is fine — the
// probe classifier handles a missing `#logo` as Transient anyway,
// and the verify loop retries on Transient.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"#logo",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
page.content().await.context("read probe html")
},
)
.await;
let html = page.content().await.context("read probe html")?;
page.close().await.ok();
Ok(html)
.await
}
#[cfg(test)]

View File

@@ -20,7 +20,8 @@ use super::{
use crate::crawler::detect::{
has_logo_sentinel, is_broken_page_body, retry_on_transient_with_hook, PageError,
};
use crate::crawler::nav::{wait_for_nav, wait_for_selector, NavError, SELECTOR_TIMEOUT};
use crate::crawler::nav::{close_after, wait_for_nav, wait_for_selector, NavError, SELECTOR_TIMEOUT};
use crate::crawler::safety::ensure_public_target;
/// `sources.id` value for this Source impl. Exposed as a const so the
/// daemon can look up per-source state (e.g. the recovery flag) before
@@ -220,6 +221,19 @@ const LIST_PAGE_MARKER: &str = "#left_side .pic_list .updatesli";
const DETAIL_PAGE_CHAPTERS_MARKER: &str = "#chapter_table td h4 a.chico";
const DETAIL_PAGE_LAYOUT_MARKER: &str = "#logo";
/// Refuse to point the headless browser at a private/internal target.
/// The list/detail URLs driven through [`navigate`] originate from
/// scraped hrefs (base URL, pagination, and detail links harvested from
/// listings), so a hostile or compromised source could otherwise steer
/// Chromium at `http://169.254.169.254/`, `http://postgres:5432/`, etc.
/// and read the response body as an SSRF oracle. Mirrors the
/// chapter-content guard in [`crate::crawler::content`].
fn guard_navigate_url(url: &str) -> Result<(), PageError> {
ensure_public_target(url).map_err(|e| {
PageError::Other(anyhow::anyhow!("refuse to navigate unsafe URL {url}: {e}"))
})
}
/// Single point of rate-limited navigation. Every Source request goes
/// through here, so the per-host limiter map is the only knob that
/// controls per-origin RPS. Also the choke point for transient-page
@@ -237,32 +251,39 @@ async fn navigate(
url: &str,
marker: &str,
) -> Result<String, PageError> {
guard_navigate_url(url)?;
ctx.rate.wait_for(url).await?;
let page = ctx
.browser
.new_page(url)
let page = crate::crawler::intercept::open_page(ctx.browser, url)
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
match wait_for_nav(&page).await {
Ok(()) => {}
Err(NavError::Timeout(_)) => {
page.close().await.ok();
return Err(PageError::transient("nav timeout"));
}
Err(NavError::Cdp(e)) => {
page.close().await.ok();
return Err(PageError::Other(anyhow::Error::from(e)));
}
}
// Best-effort wait for the page-type marker. We deliberately
// discard a timeout here — see fn-level doc.
let _ = wait_for_selector(&page, marker, SELECTOR_TIMEOUT).await;
let html = page
.content()
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
page.close().await.ok();
classify_navigate_html(html)
// Close the tab on every exit path — the previous code closed on the two
// nav-error branches but leaked it on a content() read error.
let closer = page.clone();
close_after(
async move {
closer.close().await.ok();
},
async {
match wait_for_nav(&page).await {
Ok(()) => {}
Err(NavError::Timeout(_)) => {
return Err(PageError::transient("nav timeout"));
}
Err(NavError::Cdp(e)) => {
return Err(PageError::Other(anyhow::Error::from(e)));
}
}
// Best-effort wait for the page-type marker. We deliberately
// discard a timeout here — see fn-level doc.
let _ = wait_for_selector(&page, marker, SELECTOR_TIMEOUT).await;
let html = page
.content()
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
classify_navigate_html(html)
},
)
.await
}
/// Classify a fetched body. The broken-page template is universal across
@@ -1107,4 +1128,37 @@ mod tests {
.expect("metadata-only parse must not require chapter table");
assert!(manga.chapters.is_empty());
}
#[test]
fn navigate_guard_rejects_private_and_internal_targets() {
// The SSRF guard `navigate` runs before opening any headless page.
// A scraped listing/detail href pointing at cloud metadata, an
// internal service, or the loopback interface must be refused.
for bad in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:5432/",
"http://postgres:5432/", // resolves to a private range name, but…
"http://[::1]/",
"http://10.0.0.5/",
"file:///etc/passwd",
] {
// `postgres` is a bare hostname, not an IP literal, so the
// literal-IP guard alone passes it — assert only the cases the
// guard is designed to catch (IP literals + bad schemes).
if bad.contains("postgres") {
assert!(guard_navigate_url(bad).is_ok(), "bare hostname passes literal check");
continue;
}
assert!(
guard_navigate_url(bad).is_err(),
"expected {bad} to be refused before navigation"
);
}
}
#[test]
fn navigate_guard_allows_public_targets() {
assert!(guard_navigate_url("https://target.example/manga/foo").is_ok());
assert!(guard_navigate_url("https://8.8.8.8/").is_ok());
}
}

View File

@@ -12,4 +12,6 @@ pub struct ApiToken {
pub token_hash: Vec<u8>,
pub created_at: DateTime<Utc>,
pub last_used_at: Option<DateTime<Utc>>,
/// When the token stops authenticating. `None` = never expires.
pub expires_at: Option<DateTime<Utc>>,
}

View File

@@ -11,6 +11,7 @@ pub mod page;
pub mod page_analysis;
pub mod page_tag;
pub mod patch;
pub mod reaction;
pub mod read_progress;
pub mod session;
pub mod storage_stats;

View File

@@ -0,0 +1,40 @@
use serde::{Deserialize, Serialize};
use uuid::Uuid;
/// A user's private taste signal on a manga. Stored as text in
/// `manga_reactions.reaction` (CHECK-constrained), so we map to/from a
/// `&str` at the repo layer rather than deriving a Postgres enum type.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Reaction {
Like,
Dislike,
}
impl Reaction {
pub fn as_str(self) -> &'static str {
match self {
Reaction::Like => "like",
Reaction::Dislike => "dislike",
}
}
/// Parse a stored/inbound value. Returns `None` for anything outside the
/// closed vocabulary (the DB CHECK guarantees stored rows are valid; this
/// also guards the inbound API body).
pub fn parse(s: &str) -> Option<Reaction> {
match s {
"like" => Some(Reaction::Like),
"dislike" => Some(Reaction::Dislike),
_ => None,
}
}
}
/// Response shape for `GET /me/reactions/:manga_id` — the current user's
/// reaction on one manga, or `null` when they haven't reacted.
#[derive(Debug, Clone, Serialize)]
pub struct MangaReaction {
pub manga_id: Uuid,
pub reaction: Option<Reaction>,
}

View File

@@ -24,8 +24,19 @@ pub struct ReadProgressSummary {
/// `None` when the chapter was deleted after this row was written
/// (FK ON DELETE SET NULL on `chapter_id`).
pub chapter_number: Option<i32>,
/// Page count of the last-read chapter (`None` when unknown / deleted).
/// Lets a client distinguish a finished series (read to the last page,
/// nothing new) from one still in progress — e.g. to keep finished
/// series off a "Continue reading" shelf.
pub chapter_page_count: Option<i32>,
pub page: i32,
pub updated_at: DateTime<Utc>,
/// How many chapters sit past the reader's last-read chapter (by
/// chapter number) — a personal "new since you last read" count.
/// `0` when the reader is caught up or when `chapter_number` is
/// unknown (manga-level progress or a deleted chapter), so we never
/// claim chapters are new when we can't place the reader.
pub new_chapters_count: i64,
}
/// Returned by `GET /me/read-progress/:manga_id`. Same shape as
@@ -40,6 +51,11 @@ pub struct ReadProgressForManga {
pub chapter_number: Option<i32>,
pub page: i32,
pub updated_at: DateTime<Utc>,
/// Distinct chapter numbers past the last-read chapter — the detail
/// page's authoritative "new since last read" count (computed over all
/// chapters, not the client's paginated list). `0` when caught up or
/// the position is unknown. See [`ReadProgressSummary::new_chapters_count`].
pub new_chapters_count: i64,
}
#[derive(Debug, Clone, Deserialize)]

View File

@@ -207,4 +207,68 @@ mod tests {
"text_search_not_yet_supported"
);
}
#[test]
fn status_mapping_is_stable() {
// Pin the code -> HTTP status mapping so a refactor of into_response
// can't silently change a status (e.g. 501 -> 500). The 501 NotImplemented
// path and a deterministic 429 Retry-After were previously unexercised
// (audit GAP B).
let cases: Vec<(AppError, StatusCode)> = vec![
(AppError::NotFound, StatusCode::NOT_FOUND),
(AppError::InvalidInput("x".into()), StatusCode::BAD_REQUEST),
(AppError::Unauthenticated, StatusCode::UNAUTHORIZED),
(AppError::Forbidden, StatusCode::FORBIDDEN),
(AppError::Conflict("x".into()), StatusCode::CONFLICT),
(AppError::PayloadTooLarge("x".into()), StatusCode::PAYLOAD_TOO_LARGE),
(
AppError::UnsupportedMediaType("x".into()),
StatusCode::UNSUPPORTED_MEDIA_TYPE,
),
(
AppError::ServiceUnavailable("x".into()),
StatusCode::SERVICE_UNAVAILABLE,
),
(
AppError::ValidationFailed {
message: "x".into(),
details: json!({}),
},
StatusCode::UNPROCESSABLE_ENTITY,
),
(
AppError::NotImplemented {
code: "c",
message: "m",
},
StatusCode::NOT_IMPLEMENTED,
),
(
AppError::TooManyRequests {
retry_after_secs: None,
},
StatusCode::TOO_MANY_REQUESTS,
),
(
AppError::Other(anyhow::anyhow!("boom")),
StatusCode::INTERNAL_SERVER_ERROR,
),
];
for (err, want) in cases {
let code = err.code();
assert_eq!(err.into_response().status(), want, "status for code {code}");
}
// A 429 with retry_after carries the Retry-After header deterministically
// (without needing a live rate limiter).
let resp = AppError::TooManyRequests {
retry_after_secs: Some(7),
}
.into_response();
assert_eq!(resp.status(), StatusCode::TOO_MANY_REQUESTS);
assert_eq!(
resp.headers().get(axum::http::header::RETRY_AFTER).unwrap(),
"7"
);
}
}

View File

@@ -214,7 +214,7 @@ pub async fn list_mangas_with_sync_state(
let search_pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
// sqlx::Type → text: bind the snake_case representation manually so
// the SQL can compare it as text without an explicit cast.
@@ -235,7 +235,7 @@ pub async fn list_mangas_with_sync_state(
(SELECT MAX(last_seen_at) FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL) AS latest_seen_at
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
WHERE ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
)
SELECT * FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)
@@ -261,7 +261,7 @@ pub async fn list_mangas_with_sync_state(
WITH classified AS (
SELECT {case} AS sync_state
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
WHERE ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
)
SELECT COUNT(*) FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)

View File

@@ -2,6 +2,7 @@
//! token; the raw value is shown to the user once at creation and never
//! stored.
use chrono::{DateTime, Utc};
use sqlx::PgPool;
use uuid::Uuid;
@@ -13,28 +14,48 @@ pub async fn create(
user_id: Uuid,
name: &str,
token_hash: &[u8],
expires_at: Option<DateTime<Utc>>,
) -> AppResult<ApiToken> {
let row = sqlx::query_as::<_, ApiToken>(
r#"
INSERT INTO api_tokens (user_id, name, token_hash)
VALUES ($1, $2, $3)
RETURNING id, user_id, name, token_hash, created_at, last_used_at
INSERT INTO api_tokens (user_id, name, token_hash, expires_at)
VALUES ($1, $2, $3, $4)
RETURNING id, user_id, name, token_hash, created_at, last_used_at, expires_at
"#,
)
.bind(user_id)
.bind(name)
.bind(token_hash)
.bind(expires_at)
.fetch_one(pool)
.await?;
Ok(row)
}
/// The caller's tokens, newest first. `token_hash` is `#[serde(skip)]` on the
/// domain type, so returning the full row never leaks the secret.
pub async fn list_for_user(pool: &PgPool, user_id: Uuid) -> AppResult<Vec<ApiToken>> {
let rows = sqlx::query_as::<_, ApiToken>(
r#"
SELECT id, user_id, name, token_hash, created_at, last_used_at, expires_at
FROM api_tokens
WHERE user_id = $1
ORDER BY created_at DESC
"#,
)
.bind(user_id)
.fetch_all(pool)
.await?;
Ok(rows)
}
pub async fn find_active(pool: &PgPool, token_hash: &[u8]) -> AppResult<Option<ApiToken>> {
let row = sqlx::query_as::<_, ApiToken>(
r#"
SELECT id, user_id, name, token_hash, created_at, last_used_at
SELECT id, user_id, name, token_hash, created_at, last_used_at, expires_at
FROM api_tokens
WHERE token_hash = $1
AND (expires_at IS NULL OR expires_at > now())
"#,
)
.bind(token_hash)

View File

@@ -81,7 +81,7 @@ pub async fn list(
SELECT id, name, created_at
FROM authors
WHERE $1::text IS NULL
OR name ILIKE '%' || $1 || '%'
OR name ILIKE '%' || $4 || '%' ESCAPE '\'
OR name % $1
ORDER BY CASE WHEN $1::text IS NULL THEN 0 ELSE similarity(name, $1) END DESC,
lower(name) ASC
@@ -91,6 +91,9 @@ pub async fn list(
.bind(search)
.bind(limit)
.bind(offset)
// $4: LIKE-escaped term for the ILIKE branch so `%`/`_` match literally; the
// trigram/similarity branches keep the raw $1.
.bind(search.map(crate::repo::escape_like))
.fetch_all(pool)
.await?;
Ok(rows)

View File

@@ -6,13 +6,23 @@ use uuid::Uuid;
use crate::domain::{Bookmark, BookmarkSummary};
use crate::error::{AppError, AppResult};
/// Add a bookmark, idempotently. Returns the bookmark plus whether it was
/// newly created (`true`) or already existed (`false`), so the handler can
/// answer 201 vs 200. Re-adding an existing bookmark is a no-op success rather
/// than a 409 — matching the idempotent collection semantics, so the UI doesn't
/// show a false "Could not add bookmark" toast when the manga is already saved.
///
/// Uniqueness is per `(user_id, manga_id, chapter_id)` — enforced by the 0001
/// constraint for chapter-level rows and the 0004 partial index for manga-level
/// (NULL chapter) rows. `page` is not part of the key, so the existing row is
/// returned unchanged (its page is not overwritten).
pub async fn create(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
chapter_id: Option<Uuid>,
page: Option<i32>,
) -> AppResult<Bookmark> {
) -> AppResult<(Bookmark, bool)> {
let result = sqlx::query_as::<_, Bookmark>(
r#"
INSERT INTO bookmarks (user_id, manga_id, chapter_id, page)
@@ -28,10 +38,27 @@ pub async fn create(
.await;
match result {
Ok(b) => Ok(b),
Err(sqlx::Error::Database(ref db_err)) if db_err.is_unique_violation() => Err(
AppError::Conflict("bookmark already exists for this manga/chapter".into()),
),
Ok(b) => Ok((b, true)),
Err(sqlx::Error::Database(ref db_err)) if db_err.is_unique_violation() => {
// A bookmark for this (user, manga, chapter) already exists — fetch
// and return it. `IS NOT DISTINCT FROM` matches a NULL chapter_id
// (manga-level bookmark) as well as a concrete one, covering both
// uniqueness paths with a single lookup.
let existing = sqlx::query_as::<_, Bookmark>(
r#"
SELECT id, user_id, manga_id, chapter_id, page, created_at
FROM bookmarks
WHERE user_id = $1 AND manga_id = $2
AND chapter_id IS NOT DISTINCT FROM $3
"#,
)
.bind(user_id)
.bind(manga_id)
.bind(chapter_id)
.fetch_one(pool)
.await?;
Ok((existing, false))
}
Err(e) => Err(AppError::Database(e)),
}
}

View File

@@ -12,14 +12,19 @@ pub async fn list_for_manga(
limit: i64,
offset: i64,
) -> AppResult<Vec<Chapter>> {
// Display order = source-site order reversed. The crawler stamps
// `source_index` = position in the source DOM (0 = first = newest
// on this site, see migration 0021), so DESC puts the oldest
// chapter first and keeps the site's variant grouping and the
// placement of non-numeric entries (e.g. "notice. : Officials")
// intact. NULLS LAST keeps user-uploaded chapters (no source row)
// and rows that pre-date the migration below crawled rows; the
// (number, created_at) tail then orders them deterministically.
// Display order. Crawled chapters carry `source_index` = position in the
// source DOM (0 = newest on this site, migration 0021); we display them
// reversed (oldest first) via the `-source_index` key, which keeps the
// site's variant grouping and non-numeric entries (e.g. "notice.") in the
// spot the site placed them — NOT clustered at number 0.
//
// User-uploaded chapters have no `source_index`. Instead of dumping them
// after every crawled chapter (the old `NULLS LAST` bug, which put an
// uploaded chapter 5 after crawled chapter 100), each is slotted by NUMBER:
// just before the crawled chapter with the smallest number greater than it,
// so it interleaves. An upload newer than every crawled chapter goes last;
// when there are no crawled chapters at all, uploads fall back to the
// `number, created_at` tail.
let rows = sqlx::query_as::<_, Chapter>(
r#"
SELECT id, manga_id, number, title, page_count, created_at,
@@ -31,7 +36,22 @@ pub async fn list_for_manga(
FROM pages p WHERE p.chapter_id = chapters.id)::bigint AS size_bytes
FROM chapters
WHERE manga_id = $1
ORDER BY source_index DESC NULLS LAST, number ASC, created_at ASC
ORDER BY
CASE
WHEN source_index IS NOT NULL THEN (-source_index)::float8
ELSE COALESCE(
(SELECT MIN(-c2.source_index)::float8 - 0.5
FROM chapters c2
WHERE c2.manga_id = chapters.manga_id
AND c2.source_index IS NOT NULL
AND c2.number > chapters.number),
(SELECT COALESCE(MAX(-c3.source_index), 0)::float8 + 0.5
FROM chapters c3
WHERE c3.manga_id = chapters.manga_id
AND c3.source_index IS NOT NULL)
)
END,
number ASC, created_at ASC
LIMIT $2 OFFSET $3
"#,
)

View File

@@ -187,7 +187,7 @@ pub async fn list_ops(
}
/// Delete metric rows older than `retention_days`. `0` disables the reaper
/// (returns 0 without touching the table). Mirrors `jobs::reap_done`.
/// (returns 0 without touching the table). Mirrors `jobs::reap_terminal`.
pub async fn reap(pool: &PgPool, retention_days: u32) -> sqlx::Result<u64> {
if retention_days == 0 {
return Ok(0);

View File

@@ -247,10 +247,13 @@ async fn sync_genres(
let genre_id = match existing {
Some((id,)) => id,
None => {
// Conflict on lower(name) (0038) not name, so a racing insert of
// a differently-cased variant resolves to the existing row
// instead of raising a unique violation.
let (id,): (Uuid,) = sqlx::query_as(
r#"
INSERT INTO genres (name) VALUES ($1)
ON CONFLICT (name) DO UPDATE SET name = genres.name
ON CONFLICT (lower(name)) DO UPDATE SET name = genres.name
RETURNING id
"#,
)
@@ -720,7 +723,7 @@ pub async fn list_dead_jobs(
offset: i64,
) -> sqlx::Result<(Vec<DeadJob>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<DeadJob> = sqlx::query_as(
@@ -743,7 +746,7 @@ pub async fn list_dead_jobs(
LEFT JOIN chapters c ON c.id = (cj.payload->>'chapter_id')::uuid
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state = 'dead'
AND ($1::text IS NULL OR m.title ILIKE $1 OR cj.payload->>'title' ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\' OR cj.payload->>'title' ILIKE $1 ESCAPE '\')
ORDER BY cj.updated_at DESC
LIMIT $2 OFFSET $3
"#,
@@ -761,7 +764,7 @@ pub async fn list_dead_jobs(
LEFT JOIN chapters c ON c.id = (cj.payload->>'chapter_id')::uuid
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state = 'dead'
AND ($1::text IS NULL OR m.title ILIKE $1 OR cj.payload->>'title' ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\' OR cj.payload->>'title' ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)
@@ -797,7 +800,7 @@ pub async fn list_active_jobs(
offset: i64,
) -> sqlx::Result<(Vec<ActiveJob>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<ActiveJob> = sqlx::query_as(
@@ -817,7 +820,7 @@ pub async fn list_active_jobs(
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state IN ('pending','running')
AND cj.payload->>'kind' = 'sync_chapter_content'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
ORDER BY (cj.state = 'running') DESC, cj.scheduled_at, cj.created_at
LIMIT $2 OFFSET $3
"#,
@@ -836,7 +839,7 @@ pub async fn list_active_jobs(
LEFT JOIN mangas m ON m.id = c.manga_id
WHERE cj.state IN ('pending','running')
AND cj.payload->>'kind' = 'sync_chapter_content'
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)
@@ -903,9 +906,8 @@ pub struct JobHistoryFilter<'a> {
/// manga/chapter/page context (best-effort) so the table can label rows.
/// Returns the page slice plus the filtered total for pagination.
///
/// History depth is bounded by the done-job reaper (`reap_done`): completed
/// jobs older than the retention window are gone. Terminal `dead` jobs
/// persist until requeued.
/// History depth is bounded by the terminal-job reaper (`reap_terminal`):
/// `done` and `dead` jobs older than the retention window are gone.
pub async fn list_job_history(
pool: &PgPool,
filter: JobHistoryFilter<'_>,
@@ -914,7 +916,7 @@ pub async fn list_job_history(
) -> sqlx::Result<(Vec<JobHistoryRow>, i64)> {
let search_pat = filter
.search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
// The same FROM/JOIN/WHERE drives both the page slice and the count, so
@@ -952,7 +954,7 @@ pub async fn list_job_history(
) cm ON true
WHERE ($1::text IS NULL OR cj.state = $1)
AND ($2::text IS NULL OR cj.payload->>'kind' = $2)
AND ($3::text IS NULL OR m.title ILIKE $3 OR cj.payload->>'title' ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\' OR cj.payload->>'title' ILIKE $3 ESCAPE '\')
ORDER BY cj.updated_at DESC
LIMIT $4 OFFSET $5
"#,
@@ -974,7 +976,7 @@ pub async fn list_job_history(
LEFT JOIN mangas m ON m.id = COALESCE(c.manga_id, (cj.payload->>'manga_id')::uuid)
WHERE ($1::text IS NULL OR cj.state = $1)
AND ($2::text IS NULL OR cj.payload->>'kind' = $2)
AND ($3::text IS NULL OR m.title ILIKE $3 OR cj.payload->>'title' ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\' OR cj.payload->>'title' ILIKE $3 ESCAPE '\')
"#,
)
.bind(filter.state)
@@ -1019,7 +1021,7 @@ pub async fn list_missing_cover_mangas(
offset: i64,
) -> sqlx::Result<(Vec<MissingCoverRow>, i64)> {
let search_pat = search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items: Vec<MissingCoverRow> = sqlx::query_as(
@@ -1031,7 +1033,7 @@ pub async fn list_missing_cover_mangas(
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL
)
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
ORDER BY m.updated_at DESC
LIMIT $2 OFFSET $3
"#,
@@ -1050,7 +1052,7 @@ pub async fn list_missing_cover_mangas(
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL
)
AND ($1::text IS NULL OR m.title ILIKE $1)
AND ($1::text IS NULL OR m.title ILIKE $1 ESCAPE '\')
"#,
)
.bind(&search_pat)

View File

@@ -10,7 +10,7 @@ use uuid::Uuid;
use crate::domain::manga::{Manga, MangaCard, MangaDetail};
use crate::error::{AppError, AppResult};
use crate::repo;
use crate::repo::{self, escape_like};
/// Status values mirror the CHECK constraint in 0009. Centralized so
/// the API layer can validate uploads against the same vocabulary.
@@ -110,13 +110,13 @@ fn manga_cols(alias: &str) -> String {
/// true.
const FILTER_WHERE: &str = r#"
($1::text IS NULL
OR title ILIKE '%' || $1 || '%'
OR title ILIKE '%' || $8 || '%' ESCAPE '\'
OR title % $1
OR EXISTS (
SELECT 1 FROM manga_authors ma
JOIN authors a ON a.id = ma.author_id
WHERE ma.manga_id = mangas.id
AND (a.name ILIKE '%' || $1 || '%' OR a.name % $1)
AND (a.name ILIKE '%' || $8 || '%' ESCAPE '\' OR a.name % $1)
)
)
AND ($2::text IS NULL OR status = $2)
@@ -144,17 +144,13 @@ const FILTER_WHERE: &str = r#"
AND NOT EXISTS (
SELECT 1 FROM unnest($6::text[]) AS req(w)
WHERE NOT EXISTS (
SELECT 1 FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mangas.id AND pw.warning = req.w
SELECT 1 FROM manga_content_warnings mcw
WHERE mcw.manga_id = mangas.id AND mcw.warning = req.w
)
)
AND NOT EXISTS (
SELECT 1 FROM page_content_warnings pw
JOIN pages p ON p.id = pw.page_id
JOIN chapters c ON c.id = p.chapter_id
WHERE c.manga_id = mangas.id AND pw.warning = ANY($7::text[])
SELECT 1 FROM manga_content_warnings mcw
WHERE mcw.manga_id = mangas.id AND mcw.warning = ANY($7::text[])
)
"#;
@@ -163,7 +159,7 @@ const FILTER_WHERE: &str = r#"
/// (`mangas_created_at_idx`, `mangas_updated_at_idx`, `mangas_title_lower_idx`)
/// and the trigram GIN indexes (0005_search.sql, 0009_manga_metadata.sql) keep
/// the search filter cheap as the library grows.
pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i64)> {
pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, Option<i64>)> {
// Both `col` and `dir` are interpolated from hard-coded enums, never from
// request input, so this is not a SQL injection seam. The trailing `id` is
// a stable tie-break that keeps pagination deterministic across rows with
@@ -172,16 +168,11 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
SortField::Created => "created_at",
SortField::Updated => "updated_at",
SortField::Title => "lower(title)",
// Sorts on the alphabetically-first attached author. As an ORDER BY
// key this correlated subquery is evaluated per filter-matching row
// before LIMIT applies, so it scales worse than the indexed date/title
// sorts — revisit with a LATERAL join or precomputed sort-name column
// if the library grows large.
SortField::Author => {
"(SELECT min(lower(a.name)) \
FROM manga_authors ma JOIN authors a ON a.id = ma.author_id \
WHERE ma.manga_id = mangas.id)"
}
// Sorts on the alphabetically-first attached author. Precomputed into
// `mangas.sort_author` (migration 0037, maintained by triggers on
// manga_authors) and index-backed by `mangas_sort_author_idx`, so this
// is a plain column read rather than a per-row correlated subquery.
SortField::Author => "sort_author",
};
let dir = match query.order {
SortOrder::Asc => "ASC",
@@ -205,11 +196,15 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
FROM mangas
WHERE {FILTER_WHERE}
ORDER BY {order_by}
LIMIT $8 OFFSET $9
LIMIT $9 OFFSET $10
"#,
cols = manga_cols(""),
);
// $8 is the LIKE-escaped search term used by the ILIKE branches so `%`/`_`
// in the term match literally; the trigram `%` branches keep the raw $1.
let search_escaped = search.map(escape_like);
let rows = sqlx::query_as::<_, Manga>(&list_sql)
.bind(search)
.bind(status)
@@ -218,27 +213,40 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.bind(&search_escaped)
.bind(query.limit)
.bind(query.offset)
.fetch_all(pool)
.await?;
let count_sql = format!(
r#"
SELECT count(*) FROM mangas
WHERE {FILTER_WHERE}
"#
);
let (total,): (i64,) = sqlx::query_as(&count_sql)
.bind(search)
.bind(status)
.bind(&query.author_ids)
.bind(&query.genre_ids)
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.fetch_one(pool)
.await?;
// The count reuses FILTER_WHERE — up to five correlated NOT EXISTS/unnest
// subqueries (plus a page_content_warnings join under CW filters) over the
// whole filtered set. Recomputing it on every page made pagination scale
// with catalog size. It doesn't change as the caller walks pages, so
// compute it once on the first page (offset 0) and return None thereafter;
// the pagination envelope serialises that as `total: null`.
let total = if query.offset == 0 {
let count_sql = format!(
r#"
SELECT count(*) FROM mangas
WHERE {FILTER_WHERE}
"#
);
let (total,): (i64,) = sqlx::query_as(&count_sql)
.bind(search)
.bind(status)
.bind(&query.author_ids)
.bind(&query.genre_ids)
.bind(&query.tag_ids)
.bind(&query.cw_include)
.bind(&query.cw_exclude)
.bind(&search_escaped)
.fetch_one(pool)
.await?;
Some(total)
} else {
None
};
Ok((rows, total))
}
@@ -249,7 +257,7 @@ pub async fn list(pool: &PgPool, query: &ListQuery) -> AppResult<(Vec<Manga>, i6
pub async fn list_cards(
pool: &PgPool,
query: &ListQuery,
) -> AppResult<(Vec<MangaCard>, i64)> {
) -> AppResult<(Vec<MangaCard>, Option<i64>)> {
let (rows, total) = list(pool, query).await?;
let cards = cards_from_rows(pool, rows).await?;
Ok((cards, total))
@@ -307,6 +315,73 @@ pub async fn list_similar(
cards_from_rows(pool, rows).await
}
/// Content-based "Recommended for you": rank mangas by weighted tag overlap
/// with the user's taste. Signals: explicit like = +1.0, bookmark = +0.5,
/// dislike = -1.0 (a reaction overrides a bookmark on the same manga). Per
/// tag we sum those weights into an affinity, then score each candidate by
/// the sum of its tags' affinities, normalized by the candidate's tag count
/// (same anti-tag-stuffing rationale as `list_similar`). Candidates the user
/// already reacted to, bookmarked, or read are excluded; net-negative
/// candidates (dominated by disliked-tag affinity) are dropped, so a dislike
/// down-ranks rather than the manga being hidden from normal browse. No
/// signals → empty. Reuses `cards_from_rows` for author/genre hydration.
pub async fn list_recommendations(
pool: &PgPool,
user_id: Uuid,
limit: i64,
) -> AppResult<Vec<MangaCard>> {
let sql = format!(
r#"
WITH signals AS (
SELECT s.manga_id,
CASE WHEN r.reaction = 'dislike' THEN -1.0
WHEN r.reaction = 'like' THEN 1.0
ELSE 0.5 END AS weight
FROM (
SELECT manga_id FROM manga_reactions WHERE user_id = $1
UNION
SELECT manga_id FROM bookmarks WHERE user_id = $1
) s
LEFT JOIN manga_reactions r
ON r.user_id = $1 AND r.manga_id = s.manga_id
),
tag_affinity AS (
SELECT mt.tag_id, SUM(sig.weight) AS affinity
FROM signals sig
JOIN manga_tags mt ON mt.manga_id = sig.manga_id
GROUP BY mt.tag_id
)
SELECT {cols}
FROM manga_tags cand
JOIN tag_affinity ta ON ta.tag_id = cand.tag_id
JOIN mangas m ON m.id = cand.manga_id
WHERE cand.manga_id NOT IN (SELECT manga_id FROM signals)
AND cand.manga_id NOT IN (
SELECT manga_id FROM read_progress WHERE user_id = $1
)
GROUP BY m.id
HAVING SUM(ta.affinity) > 0
ORDER BY
SUM(ta.affinity)
/ (SELECT count(*) FROM manga_tags WHERE manga_id = m.id) DESC,
SUM(ta.affinity) DESC,
m.updated_at DESC,
lower(m.title) ASC,
m.id
LIMIT $2
"#,
cols = manga_cols("m"),
);
let rows = sqlx::query_as::<_, Manga>(&sql)
.bind(user_id)
.bind(limit)
.fetch_all(pool)
.await?;
cards_from_rows(pool, rows).await
}
/// Hydrate a batch of `Manga` rows into `MangaCard`s by attaching their
/// authors and genres in two batched round-trips. The input order is
/// preserved (callers rely on this to keep list/ranking order), so we

View File

@@ -13,6 +13,7 @@ pub mod manga;
pub mod page;
pub mod page_analysis;
pub mod page_tag;
pub mod reaction;
pub mod read_progress;
pub mod session;
pub mod storage_stats;
@@ -20,3 +21,45 @@ pub mod tag;
pub mod upload_history;
pub mod user;
pub mod user_preferences;
/// Escape the LIKE/ILIKE metacharacters (`%`, `_`, and the escape char `\`
/// itself) in a user-supplied search term so they match literally rather than
/// as wildcards. Pair the resulting value with `ESCAPE '\'` in the SQL — a
/// single backslash, which under `standard_conforming_strings` (Postgres
/// default) is one backslash in a single-quoted literal.
///
/// This is a search-correctness fix, not an injection fix: every term is
/// already a bound parameter, so `%`/`_` can never break out of the string —
/// they were just silently acting as wildcards (`50%` matching everything,
/// `a_b` matching `axb`). Callers that build a substring pattern wrap the
/// escaped term themselves, e.g. `format!("%{}%", escape_like(term))`.
pub(crate) fn escape_like(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for ch in s.chars() {
if matches!(ch, '\\' | '%' | '_') {
out.push('\\');
}
out.push(ch);
}
out
}
#[cfg(test)]
mod tests {
use super::escape_like;
#[test]
fn escapes_wildcards_and_the_escape_char() {
assert_eq!(escape_like("50%"), r"50\%");
assert_eq!(escape_like("a_b"), r"a\_b");
assert_eq!(escape_like(r"back\slash"), r"back\\slash");
// A already-escaped-looking input is double-escaped so it stays literal.
assert_eq!(escape_like(r"\%"), r"\\\%");
}
#[test]
fn leaves_ordinary_text_untouched() {
assert_eq!(escape_like("naruto"), "naruto");
assert_eq!(escape_like(""), "");
}
}

View File

@@ -78,13 +78,30 @@ pub enum EnqueueForPageOutcome {
/// with `force=true`, running an UPDATE that flips the existing pending
/// row's `force` flag to `true`; (3) reporting which path ran so the
/// caller can audit accurately.
/// Pool convenience wrapper for [`enqueue_for_page_conn`]. Use this from
/// callers that don't need to share a transaction (chapter upload, the
/// crawler). The admin force-reanalyze handler uses the `_conn` form so the
/// enqueue and its `admin_audit` row commit together.
pub async fn enqueue_for_page(
pool: &PgPool,
page_id: Uuid,
force: bool,
) -> AppResult<EnqueueForPageOutcome> {
let mut conn = pool.acquire().await?;
enqueue_for_page_conn(&mut conn, page_id, force).await
}
/// Enqueue (or force-upgrade) an `analyze_page` job on a caller-supplied
/// connection, so the admin handler can run it inside the same transaction as
/// its audit insert. The body is a sequence of single statements, each
/// reborrowing `&mut *conn`.
pub async fn enqueue_for_page_conn(
conn: &mut sqlx::PgConnection,
page_id: Uuid,
force: bool,
) -> AppResult<EnqueueForPageOutcome> {
use crate::crawler::jobs::EnqueueResult;
match jobs::enqueue(pool, &JobPayload::AnalyzePage { page_id, force }).await? {
match jobs::enqueue(&mut *conn, &JobPayload::AnalyzePage { page_id, force }).await? {
EnqueueResult::Inserted(_) => Ok(EnqueueForPageOutcome::Inserted),
EnqueueResult::Skipped if !force => Ok(EnqueueForPageOutcome::AlreadyEnqueued),
EnqueueResult::Skipped => {
@@ -106,7 +123,7 @@ pub async fn enqueue_for_page(
RETURNING id, state",
)
.bind(page_id.to_string())
.fetch_all(pool)
.fetch_all(&mut *conn)
.await?;
if upgraded.is_empty() {
// Race: between the skipped INSERT and the UPDATE the
@@ -115,7 +132,9 @@ pub async fn enqueue_for_page(
// would succeed. Retry once to close the race without
// unbounded loops.
return Ok(
match jobs::enqueue(pool, &JobPayload::AnalyzePage { page_id, force }).await? {
match jobs::enqueue(&mut *conn, &JobPayload::AnalyzePage { page_id, force })
.await?
{
EnqueueResult::Inserted(_) => EnqueueForPageOutcome::Inserted,
EnqueueResult::Skipped => EnqueueForPageOutcome::AlreadyEnqueued,
},
@@ -139,7 +158,7 @@ pub async fn enqueue_for_page(
// the original's ack would clobber it.
for (id, state) in &upgraded {
if state == "running" {
let _ = jobs::release_unowned(pool, *id).await;
let _ = jobs::release_unowned(&mut *conn, *id).await;
}
}
Ok(EnqueueForPageOutcome::UpgradedToForce)
@@ -168,11 +187,14 @@ pub enum ReenqueueScope {
/// skip-if-done net would no-op them). Pages with a pending/running
/// `analyze_page` job are always skipped so repeated calls don't pile up
/// duplicates. Returns the number of jobs enqueued.
pub async fn enqueue_pages(
pool: &PgPool,
pub async fn enqueue_pages<'e, E>(
executor: E,
scope: ReenqueueScope,
only_unanalyzed: bool,
) -> AppResult<u64> {
) -> AppResult<u64>
where
E: sqlx::PgExecutor<'e>,
{
// Scope predicate; the bound uuid (when present) is always $2.
let scope_clause = match scope {
ReenqueueScope::All => "",
@@ -206,7 +228,7 @@ pub async fn enqueue_pages(
ReenqueueScope::All => query,
ReenqueueScope::Manga(id) | ReenqueueScope::Chapter(id) => query.bind(id),
};
Ok(query.execute(pool).await?.rows_affected())
Ok(query.execute(executor).await?.rows_affected())
}
/// Per-manga analysis coverage for the admin overview. Only mangas that
@@ -219,6 +241,10 @@ pub async fn manga_coverage(
limit: i64,
offset: i64,
) -> AppResult<(Vec<MangaCoverage>, i64)> {
// LIKE-escape so `%`/`_` in the title filter match literally. No trigram
// branch here, so binding the escaped term directly (rather than appending a
// second param) is safe — $1 feeds only the ILIKE.
let search = search.map(crate::repo::escape_like);
let rows = sqlx::query_as::<_, MangaCoverage>(
r#"
SELECT m.id AS manga_id, m.title,
@@ -228,13 +254,13 @@ pub async fn manga_coverage(
JOIN chapters c ON c.manga_id = m.id
JOIN pages p ON p.chapter_id = c.id
LEFT JOIN page_analysis pa ON pa.page_id = p.id
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%')
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%' ESCAPE '\')
GROUP BY m.id, m.title
ORDER BY lower(m.title), m.id
LIMIT $2 OFFSET $3
"#,
)
.bind(search)
.bind(&search)
.bind(limit)
.bind(offset)
.fetch_all(pool)
@@ -247,12 +273,12 @@ pub async fn manga_coverage(
FROM mangas m
JOIN chapters c ON c.manga_id = m.id
JOIN pages p ON p.chapter_id = c.id
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%')
WHERE ($1::text IS NULL OR m.title ILIKE '%' || $1 || '%' ESCAPE '\')
GROUP BY m.id
) x
"#,
)
.bind(search)
.bind(&search)
.fetch_one(pool)
.await?;
Ok((rows, total))
@@ -355,7 +381,7 @@ pub async fn list_history(
) -> AppResult<(Vec<AnalysisHistoryRow>, i64)> {
let search_pat = filter
.search
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items = sqlx::query_as::<_, AnalysisHistoryRow>(
@@ -380,7 +406,7 @@ pub async fn list_history(
WHERE pa.status <> 'pending'
AND ($1::text IS NULL OR pa.status = $1)
AND ($2::bool IS FALSE OR pa.is_nsfw)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\')
ORDER BY pa.analyzed_at DESC NULLS LAST, pa.page_id
LIMIT $4 OFFSET $5
"#,
@@ -403,7 +429,7 @@ pub async fn list_history(
WHERE pa.status <> 'pending'
AND ($1::text IS NULL OR pa.status = $1)
AND ($2::bool IS FALSE OR pa.is_nsfw)
AND ($3::text IS NULL OR m.title ILIKE $3)
AND ($3::text IS NULL OR m.title ILIKE $3 ESCAPE '\')
"#,
)
.bind(filter.status)

View File

@@ -120,27 +120,11 @@ pub async fn list_for_page(
Ok(rows.into_iter().map(|(t,)| t).collect())
}
/// Escape a string for use as a LIKE pattern fragment: `%`, `_`, and
/// `\` get a leading backslash so they're matched literally rather
/// than as wildcards / escapes. The matching queries below pair this
/// with `ESCAPE '\'` for explicitness — a single backslash, since the
/// SQL lives in a raw string and Postgres treats `\\` in a single-
/// quoted literal as one backslash under `standard_conforming_strings`.
///
/// The public API rejects `%`/`_`/`\` in `normalize_tag` before
/// they reach this repo, so this is defence-in-depth — a future
/// internal caller (worker, CLI) that bypasses the normalizer can't
/// turn a prefix filter into a wildcard search by accident.
fn escape_like_prefix(s: &str) -> String {
let mut out = String::with_capacity(s.len());
for ch in s.chars() {
if matches!(ch, '\\' | '%' | '_') {
out.push('\\');
}
out.push(ch);
}
out
}
// LIKE-escaping the autocomplete prefix is defence-in-depth: the public API
// already rejects `%`/`_`/`\` in `normalize_tag` before they reach this repo, so
// a stray wildcard can only arrive from a future internal caller (worker, CLI)
// that bypasses the normalizer. Shared with the other search sites.
use crate::repo::escape_like as escape_like_prefix;
/// Paged list of `user_id`'s tagged pages, with breadcrumb. When
/// `tag_filter` is `Some(_)`, restrict to that exact tag (used by the

View File

@@ -0,0 +1,66 @@
//! Per-user manga reaction (like/dislike) persistence.
use sqlx::PgPool;
use uuid::Uuid;
use crate::domain::reaction::Reaction;
use crate::error::{AppError, AppResult};
/// Insert-or-overwrite the user's reaction on this manga (like ↔ dislike).
/// A foreign-key violation (unknown manga) maps to `NotFound` so the API
/// returns 404 rather than 500 — mirrors `read_progress::upsert`.
pub async fn upsert(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
reaction: Reaction,
) -> AppResult<()> {
sqlx::query(
r#"
INSERT INTO manga_reactions (user_id, manga_id, reaction, created_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id, manga_id) DO UPDATE
SET reaction = EXCLUDED.reaction,
created_at = now()
"#,
)
.bind(user_id)
.bind(manga_id)
.bind(reaction.as_str())
.execute(pool)
.await
.map_err(|e| match e {
sqlx::Error::Database(ref db_err) if db_err.is_foreign_key_violation() => {
AppError::NotFound
}
other => AppError::Database(other),
})?;
Ok(())
}
/// Remove the user's reaction on this manga. Idempotent — clearing a
/// non-existent reaction is a no-op.
pub async fn clear(pool: &PgPool, user_id: Uuid, manga_id: Uuid) -> AppResult<()> {
sqlx::query("DELETE FROM manga_reactions WHERE user_id = $1 AND manga_id = $2")
.bind(user_id)
.bind(manga_id)
.execute(pool)
.await?;
Ok(())
}
/// The user's current reaction on this manga, or `None` if unset.
pub async fn get(
pool: &PgPool,
user_id: Uuid,
manga_id: Uuid,
) -> AppResult<Option<Reaction>> {
let row: Option<(String,)> = sqlx::query_as(
"SELECT reaction FROM manga_reactions WHERE user_id = $1 AND manga_id = $2",
)
.bind(user_id)
.bind(manga_id)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|(s,)| Reaction::parse(&s)))
}

View File

@@ -80,7 +80,17 @@ pub async fn get_for_manga(
rp.chapter_id,
c.number AS chapter_number,
rp.page,
rp.updated_at
rp.updated_at,
-- Distinct chapter numbers past the reader's last-read chapter
-- (see list_for_user for the non-unique-number rationale). 0
-- when the last-read chapter is unknown.
(
SELECT count(DISTINCT c2.number)
FROM chapters c2
WHERE c2.manga_id = rp.manga_id
AND c.number IS NOT NULL
AND c2.number > c.number
) AS new_chapters_count
FROM read_progress rp
LEFT JOIN chapters c ON c.id = rp.chapter_id
WHERE rp.user_id = $1 AND rp.manga_id = $2
@@ -131,8 +141,23 @@ pub async fn list_for_user(
m.cover_image_path AS manga_cover_image_path,
rp.chapter_id,
c.number AS chapter_number,
c.page_count AS chapter_page_count,
rp.page,
rp.updated_at
rp.updated_at,
-- Personal "new since last read": how many distinct chapter
-- numbers sit past the reader's last-read chapter.
-- COUNT(DISTINCT number) — not COUNT(*) — because
-- (manga_id, number) is non-unique (scanlations share a
-- number, migration 0013), so raw rows would over-count. When
-- `c.number` is NULL (manga-level progress or a deleted
-- chapter) the predicate matches no rows, yielding 0.
(
SELECT count(DISTINCT c2.number)
FROM chapters c2
WHERE c2.manga_id = rp.manga_id
AND c.number IS NOT NULL
AND c2.number > c.number
) AS new_chapters_count
FROM read_progress rp
JOIN mangas m ON m.id = rp.manga_id
LEFT JOIN chapters c ON c.id = rp.chapter_id

View File

@@ -65,3 +65,15 @@ pub async fn delete_by_token_hash(pool: &PgPool, token_hash: &[u8]) -> AppResult
.await?;
Ok(())
}
/// Delete every session whose `expires_at` has passed, returning the number
/// reaped. `find_active` already refuses expired sessions, so this only
/// reclaims storage — running it on any schedule (or not at all) is safe.
/// Backed by `sessions_expires_idx` (0002). Called by the periodic reaper in
/// `app::build`.
pub async fn delete_expired(pool: &PgPool) -> AppResult<u64> {
let result = sqlx::query("DELETE FROM sessions WHERE expires_at <= now()")
.execute(pool)
.await?;
Ok(result.rows_affected())
}

View File

@@ -124,7 +124,7 @@ pub async fn list(
SELECT id, name, created_at
FROM tags
WHERE $1::text IS NULL
OR name ILIKE '%' || $1 || '%'
OR name ILIKE '%' || $3 || '%' ESCAPE '\'
OR name % $1
ORDER BY CASE WHEN $1::text IS NULL THEN 0 ELSE similarity(name, $1) END DESC,
lower(name) ASC
@@ -133,6 +133,9 @@ pub async fn list(
)
.bind(search)
.bind(limit)
// $3: LIKE-escaped term for the ILIKE branch so `%`/`_` match literally; the
// trigram/similarity branches keep the raw $1.
.bind(search.map(crate::repo::escape_like))
.fetch_all(pool)
.await?;
Ok(rows)

View File

@@ -110,7 +110,7 @@ pub async fn list_for_user(
});
}
// Newest first; trim to limit after the merge.
entries.sort_by(|a, b| b.created_at().cmp(&a.created_at()));
entries.sort_by_key(|b| std::cmp::Reverse(b.created_at()));
entries.truncate(limit as usize);
let (manga_total, chapter_total): (i64, i64) = sqlx::query_as(

View File

@@ -106,14 +106,14 @@ pub async fn list_with_total(
let pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.map(|s| format!("%{}%", crate::repo::escape_like(s.trim())))
.filter(|p| p.len() > 2);
let items = sqlx::query_as::<_, User>(
r#"
SELECT id, username, password_hash, created_at, is_admin
FROM users
WHERE ($1::text IS NULL OR username ILIKE $1)
WHERE ($1::text IS NULL OR username ILIKE $1 ESCAPE '\')
ORDER BY username
LIMIT $2 OFFSET $3
"#,
@@ -125,7 +125,7 @@ pub async fn list_with_total(
.await?;
let total: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM users WHERE ($1::text IS NULL OR username ILIKE $1)",
"SELECT COUNT(*) FROM users WHERE ($1::text IS NULL OR username ILIKE $1 ESCAPE '\\')",
)
.bind(&pat)
.fetch_one(pool)
@@ -157,8 +157,10 @@ pub async fn set_is_admin_unchecked(pool: &PgPool, id: Uuid, value: bool) -> App
/// - If a row already exists: flip `is_admin` to true if needed; **never**
/// touch the existing `password_hash`. Lets the operator rotate the
/// admin password through the UI without env-var conflict.
///
/// Wrapped in a transaction so a concurrent `register` for the same
/// username can't slip an INSERT between the SELECT and UPDATE/INSERT.
///
/// Set `is_admin` on a user with full safety checks: rejects self-demote,
/// rejects demoting the only remaining admin (under `ADMIN_INVARIANT_LOCK_KEY`
/// to close the parallel-demote race), and writes an `admin_audit` row
@@ -379,7 +381,7 @@ pub async fn bootstrap_admin(
.await?;
}
None => {
let hash = crate::auth::password::hash_password(password)?;
let hash = crate::auth::password::hash_password_async(password.to_string()).await?;
sqlx::query("INSERT INTO users (username, password_hash, is_admin) VALUES ($1, $2, true)")
.bind(username)
.bind(&hash)

View File

@@ -25,7 +25,7 @@ use serde::{Deserialize, Serialize};
use crate::analysis::prompt::{
GROUNDING_PROMPT_DEFAULT, OCR_PROMPT_DEFAULT, SYSTEM_PROMPT_DEFAULT,
};
use crate::config::{AnalysisConfig, CrawlerConfig, ResponseFormat};
use crate::config::{AnalysisBackend, AnalysisConfig, CrawlerConfig, ResponseFormat};
use crate::crawler::safety::DownloadAllowlist;
/// `app_settings.key` for the crawler group.
@@ -33,6 +33,28 @@ pub const KEY_CRAWLER: &str = "crawler";
/// `app_settings.key` for the analysis group.
pub const KEY_ANALYSIS: &str = "analysis";
// Upper bounds on numeric settings. These are sanity caps, not tuned limits —
// they keep a fat-fingered (or CSRF-injected) value from spawning thousands of
// workers, demanding gigabyte buffers, or sending out-of-range sampling
// params. Generous enough that no realistic deployment hits them.
const MAX_WORKERS: u64 = 64;
const MAX_CHAPTER_WORKERS: u64 = 64;
const MAX_ANALYSIS_MAX_TOKENS: u32 = 1_000_000;
const MAX_ANALYSIS_SLICES: u64 = 1024;
/// 1 GiB — far above any real page, well below "exhaust the host".
const MAX_ANALYSIS_IMAGE_BYTES: u64 = 1024 * 1024 * 1024;
/// OpenAI-compatible sampling ranges (the analysis endpoint speaks that API).
const MAX_TEMPERATURE: f64 = 2.0;
const FREQUENCY_PENALTY_MIN: f64 = -2.0;
const FREQUENCY_PENALTY_MAX: f64 = 2.0;
/// Max manga-detail fetches per metadata pass. `0` stays special-cased as
/// "unlimited"; this only bounds an explicit positive value.
const MAX_MANGA_LIMIT: u64 = 1_000_000;
/// Upper bound (seconds) shared by every timeout knob — 24h. A timeout
/// longer than a day is almost certainly a fat-fingered value (e.g. ms
/// mistaken for s) and would wedge a worker for far too long.
const MAX_TIMEOUT_SECS: u64 = 86_400;
/// One field-level validation failure, surfaced to the UI per input.
#[derive(Debug, Clone, Serialize, PartialEq, Eq)]
pub struct FieldError {
@@ -151,6 +173,8 @@ impl CrawlerSettings {
};
if self.chapter_workers < 1 {
errs.push("chapter_workers", "must be at least 1");
} else if self.chapter_workers > MAX_CHAPTER_WORKERS {
errs.push("chapter_workers", format!("must be at most {MAX_CHAPTER_WORKERS}"));
}
if let Some(url) = self.start_url.as_deref().map(str::trim).filter(|s| !s.is_empty()) {
// SSRF defence: Url::parse alone admits http://169.254.169.254
@@ -164,6 +188,16 @@ impl CrawlerSettings {
}
if self.job_timeout_secs < 1 {
errs.push("job_timeout_secs", "must be at least 1 second");
} else if self.job_timeout_secs > MAX_TIMEOUT_SECS {
errs.push("job_timeout_secs", format!("must be at most {MAX_TIMEOUT_SECS} seconds"));
}
if self.idle_timeout_secs > MAX_TIMEOUT_SECS {
errs.push("idle_timeout_secs", format!("must be at most {MAX_TIMEOUT_SECS} seconds"));
}
// 0 is intentionally "unlimited"; only an explicit positive value is
// capped.
if self.manga_limit > MAX_MANGA_LIMIT {
errs.push("manga_limit", format!("must be at most {MAX_MANGA_LIMIT}"));
}
if !errs.is_empty() {
@@ -220,6 +254,9 @@ impl CrawlerSettings {
.map(str::to_string),
download_allowlist,
max_image_bytes: self.max_image_bytes as usize,
// Env-only safety cap, not a runtime-editable setting — preserve
// it from the base so a settings reload keeps the boot value.
max_images_per_chapter: base.max_images_per_chapter,
manga_limit: self.manga_limit as usize,
job_timeout: Duration::from_secs(self.job_timeout_secs.max(1)),
metadata_max_consecutive_failures: self.metadata_max_consecutive_failures,
@@ -232,6 +269,7 @@ impl CrawlerSettings {
tor_control_cookie_path: base.tor_control_cookie_path.clone(),
tor_recircuit_max_attempts: base.tor_recircuit_max_attempts,
browser: base.browser.clone(),
ssrf_intercept: base.ssrf_intercept,
})
}
}
@@ -354,7 +392,18 @@ impl AnalysisSettings {
if self.workers < 1 {
errs.push("workers", "must be at least 1");
} else if self.workers > MAX_WORKERS {
errs.push("workers", format!("must be at most {MAX_WORKERS}"));
}
// The endpoint/model are vision-only knobs — the OCR backend never
// dials a URL or sends a model id. While vision is dormant
// (`base.backend == Ocr`), skip the live-worker SSRF gate and the
// model-required check so an OCR operator can enable the worker
// without a vision endpoint/model (the OCR settings UI doesn't even
// expose them). The basic malformed-URL sanity check below stays
// unconditional. Re-enabling vision restores the full gate via the
// `base.backend == Vision` predicate.
let vision_active = base.backend == AnalysisBackend::Vision;
let trimmed_endpoint = self.endpoint.trim();
if trimmed_endpoint.is_empty() {
errs.push("endpoint", "must be a valid absolute URL");
@@ -362,7 +411,7 @@ impl AnalysisSettings {
// Always reject obviously-malformed URLs (matches prior behaviour).
let _ = e;
errs.push("endpoint", "must be a valid absolute URL");
} else if self.enabled {
} else if self.enabled && vision_active {
// SSRF + API-key exfiltration defence — only enforced when the
// worker is actually live. Worker attaches an env-managed bearer
// token to every call; without this check, an admin (or one
@@ -379,23 +428,40 @@ impl AnalysisSettings {
errs.push("endpoint", url_safety_message(&e));
}
}
if self.enabled && self.model.trim().is_empty() {
if self.enabled && vision_active && self.model.trim().is_empty() {
errs.push("model", "required when analysis is enabled");
}
if self.max_tokens < 1 {
errs.push("max_tokens", "must be at least 1");
} else if self.max_tokens > MAX_ANALYSIS_MAX_TOKENS {
errs.push("max_tokens", format!("must be at most {MAX_ANALYSIS_MAX_TOKENS}"));
}
if self.request_timeout_secs < 1 {
errs.push("request_timeout_secs", "must be at least 1 second");
} else if self.request_timeout_secs > MAX_TIMEOUT_SECS {
errs.push(
"request_timeout_secs",
format!("must be at most {MAX_TIMEOUT_SECS} seconds"),
);
}
if self.job_timeout_secs < 1 {
errs.push("job_timeout_secs", "must be at least 1 second");
} else if self.job_timeout_secs > MAX_TIMEOUT_SECS {
errs.push(
"job_timeout_secs",
format!("must be at most {MAX_TIMEOUT_SECS} seconds"),
);
}
if self.max_pixels < 1 {
errs.push("max_pixels", "must be at least 1");
}
if self.max_image_bytes < 1 {
errs.push("max_image_bytes", "must be greater than 0");
} else if self.max_image_bytes > MAX_ANALYSIS_IMAGE_BYTES {
errs.push(
"max_image_bytes",
format!("must be at most {MAX_ANALYSIS_IMAGE_BYTES} (1 GiB)"),
);
}
if !(0.0..=0.9).contains(&self.slice_overlap) {
errs.push("slice_overlap", "must be between 0.0 and 0.9");
@@ -408,9 +474,17 @@ impl AnalysisSettings {
}
if self.max_slices < 1 {
errs.push("max_slices", "must be at least 1");
} else if self.max_slices > MAX_ANALYSIS_SLICES {
errs.push("max_slices", format!("must be at most {MAX_ANALYSIS_SLICES}"));
}
if self.temperature < 0.0 {
errs.push("temperature", "must be 0 or greater");
if !(0.0..=MAX_TEMPERATURE).contains(&self.temperature) {
errs.push("temperature", format!("must be between 0 and {MAX_TEMPERATURE}"));
}
if !(FREQUENCY_PENALTY_MIN..=FREQUENCY_PENALTY_MAX).contains(&self.frequency_penalty) {
errs.push(
"frequency_penalty",
format!("must be between {FREQUENCY_PENALTY_MIN} and {FREQUENCY_PENALTY_MAX}"),
);
}
let response_format = match ResponseFormat::parse_strict(&self.response_format) {
Some(rf) => rf,
@@ -464,6 +538,8 @@ impl AnalysisSettings {
backend: base.backend,
ocr_detection_model: base.ocr_detection_model.clone(),
ocr_recognition_model: base.ocr_recognition_model.clone(),
// Env-only decompression-bomb decode cap, carried from the base.
ocr_max_decode_pixels: base.ocr_max_decode_pixels,
})
}
}
@@ -495,11 +571,13 @@ mod tests {
#[test]
fn crawler_round_trips_through_dto() {
let mut base = CrawlerConfig::default();
base.start_url = Some("https://example.com/".to_string());
base.tz = Tz::Europe__Berlin;
base.chapter_workers = 3;
base.cookie_domain = Some("example.com".to_string());
let base = CrawlerConfig {
start_url: Some("https://example.com/".to_string()),
tz: Tz::Europe__Berlin,
chapter_workers: 3,
cookie_domain: Some("example.com".to_string()),
..Default::default()
};
let dto = CrawlerSettings::from_config(&base);
let back = dto.to_config(&base).expect("valid");
assert_eq!(back.tz, Tz::Europe__Berlin);
@@ -525,10 +603,12 @@ mod tests {
#[test]
fn crawler_overlay_preserves_env_only_fields() {
let mut base = CrawlerConfig::default();
base.proxy = Some("socks5://127.0.0.1:9050".to_string());
base.tor_control_password = Some("secret".to_string());
base.phpsessid = Some("abc123".to_string());
let base = CrawlerConfig {
proxy: Some("socks5://127.0.0.1:9050".to_string()),
tor_control_password: Some("secret".to_string()),
phpsessid: Some("abc123".to_string()),
..Default::default()
};
// A DTO that knows nothing about the env-only fields.
let dto = CrawlerSettings {
rate_ms: 2000,
@@ -632,8 +712,10 @@ mod tests {
#[test]
fn analysis_captures_env_prompt_override() {
let mut base = AnalysisConfig::default();
base.system_prompt = "custom env prompt".to_string();
let base = AnalysisConfig {
system_prompt: "custom env prompt".to_string(),
..Default::default()
};
let dto = AnalysisSettings::from_config(&base);
assert_eq!(dto.system_prompt.as_deref(), Some("custom env prompt"));
}
@@ -657,8 +739,10 @@ mod tests {
#[test]
fn analysis_overlay_preserves_api_key() {
let mut base = AnalysisConfig::default();
base.api_key = Some("sk-secret".to_string());
let base = AnalysisConfig {
api_key: Some("sk-secret".to_string()),
..Default::default()
};
let dto = AnalysisSettings::from_config(&base);
assert_eq!(dto.to_config(&base).unwrap().api_key.as_deref(), Some("sk-secret"));
}
@@ -695,13 +779,115 @@ mod tests {
}
}
#[test]
fn analysis_rejects_over_upper_bounds() {
// Sanity caps: an absurdly large worker count / buffer / token budget
// and out-of-range sampling params are all refused so a fat-fingered
// or CSRF-injected value can't exhaust the host or break the upstream
// API contract.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
workers: MAX_WORKERS + 1,
max_tokens: MAX_ANALYSIS_MAX_TOKENS + 1,
max_slices: MAX_ANALYSIS_SLICES + 1,
max_image_bytes: MAX_ANALYSIS_IMAGE_BYTES + 1,
temperature: MAX_TEMPERATURE + 0.5,
frequency_penalty: FREQUENCY_PENALTY_MAX + 0.5,
request_timeout_secs: MAX_TIMEOUT_SECS + 1,
job_timeout_secs: MAX_TIMEOUT_SECS + 1,
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
for f in [
"workers",
"max_tokens",
"max_slices",
"max_image_bytes",
"temperature",
"frequency_penalty",
"request_timeout_secs",
"job_timeout_secs",
] {
assert!(fields.contains(&f), "missing upper-bound error for {f}");
}
}
#[test]
fn crawler_rejects_over_upper_bounds() {
// manga_limit ceiling + timeout caps. manga_limit=0 stays "unlimited"
// and is asserted valid by the round-trip tests above.
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
manga_limit: MAX_MANGA_LIMIT + 1,
job_timeout_secs: MAX_TIMEOUT_SECS + 1,
idle_timeout_secs: MAX_TIMEOUT_SECS + 1,
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
for f in ["manga_limit", "job_timeout_secs", "idle_timeout_secs"] {
assert!(fields.contains(&f), "missing upper-bound error for {f}");
}
}
#[test]
fn crawler_allows_unlimited_manga_limit() {
// 0 means "no cap" and must stay valid.
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
manga_limit: 0,
..CrawlerSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn analysis_rejects_negative_frequency_penalty_below_range() {
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
frequency_penalty: FREQUENCY_PENALTY_MIN - 0.5,
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"frequency_penalty"));
}
#[test]
fn analysis_accepts_in_range_sampling_params() {
// A normal config with mid-range sampling values stays valid.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
temperature: 0.7,
frequency_penalty: 0.3,
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn crawler_rejects_over_chapter_worker_cap() {
let base = CrawlerConfig::default();
let dto = CrawlerSettings {
chapter_workers: MAX_CHAPTER_WORKERS + 1,
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"chapter_workers"));
}
#[test]
fn analysis_endpoint_rejects_ip_literal_attacks_when_enabled() {
// The vision worker bearer-attaches an env secret to every call;
// a hostile/CSRF-able admin must NOT be able to point endpoint at
// cloud metadata, loopback services, or RFC1918 hosts — when the
// worker is enabled (toggling enabled=true later re-runs this gate).
let base = AnalysisConfig::default();
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
for url in [
"http://169.254.169.254/v1/chat/completions",
"http://127.0.0.1:5432/",
@@ -725,7 +911,10 @@ mod tests {
// The documented default — docker DNS name resolving to a private IP
// at runtime — must still validate, because the bearer recipient
// identity is the operator-chosen hostname, not the underlying IP.
let base = AnalysisConfig::default();
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
for url in [
"http://mangalord-vision:8000/v1/chat/completions",
"https://api.openai.com/v1/chat/completions",
@@ -756,7 +945,12 @@ mod tests {
#[test]
fn analysis_requires_model_only_when_enabled() {
let base = AnalysisConfig::default();
// Vision base: the model id is required only when the vision worker
// is actually live (see the OCR carve-out below).
let base = AnalysisConfig {
backend: AnalysisBackend::Vision,
..Default::default()
};
let disabled = AnalysisSettings {
enabled: false,
model: "".to_string(),
@@ -772,6 +966,28 @@ mod tests {
assert!(errs.errors.iter().any(|e| e.field == "model"));
}
#[test]
fn analysis_ocr_backend_enable_skips_vision_endpoint_and_model_validation() {
// With the OCR backend active the worker never dials the vision
// endpoint nor sends a model id, so enabling analysis must NOT
// validate those vision-only fields. The default base carries the
// dev-localhost endpoint (which the SSRF gate would otherwise reject)
// and an empty model — under OCR, enabling is still valid. Without
// this carve-out an OCR operator can't turn the worker on, since the
// OCR settings UI doesn't expose endpoint/model to fix.
let base = AnalysisConfig::default(); // backend = Ocr
let dto = AnalysisSettings {
enabled: true,
endpoint: "http://localhost:8000/v1/chat/completions".to_string(),
model: "".to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(
dto.to_config(&base).is_ok(),
"OCR-enabled config must not fail on vision endpoint/model"
);
}
#[test]
fn dtos_serialize_to_json_and_back() {
let c = CrawlerSettings::default();

View File

@@ -95,6 +95,11 @@ impl Storage for LocalStorage {
match fs::read(&path).await {
Ok(b) => Ok(b),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Err(StorageError::NotFound),
// A key resolving to a directory isn't a stored blob; `fs::read`
// fails with EISDIR. Treat it as absent, mirroring `size`.
Err(e) if e.kind() == std::io::ErrorKind::IsADirectory => {
Err(StorageError::NotFound)
}
Err(e) => Err(e.into()),
}
}
@@ -108,7 +113,14 @@ impl Storage for LocalStorage {
}
Err(e) => return Err(e.into()),
};
let size_bytes = file.metadata().await?.len();
let meta = file.metadata().await?;
// Opening a directory succeeds on Unix, but it isn't a stored blob:
// its inode "size" is meaningless and ReaderStream would fail mid-read.
// Treat it as absent, mirroring `size` and `get`.
if !meta.is_file() {
return Err(StorageError::NotFound);
}
let size_bytes = meta.len();
// 64 KiB chunks: small enough that a few-MB page emits many frames
// (so streaming is observable), large enough to keep syscalls cheap.
let stream = ReaderStream::with_capacity(file, 64 * 1024);
@@ -127,6 +139,23 @@ impl Storage for LocalStorage {
}
}
async fn rename(&self, from: &str, to: &str) -> Result<(), StorageError> {
let from_path = self.resolve(from)?;
let to_path = self.resolve(to)?;
// Create the destination parent so a rename into a not-yet-existing
// chapter directory succeeds. `from` and `to` share the storage
// root (same filesystem), so this is a cheap atomic metadata move,
// not a copy.
if let Some(parent) = to_path.parent() {
fs::create_dir_all(parent).await?;
}
match fs::rename(&from_path, &to_path).await {
Ok(()) => Ok(()),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Err(StorageError::NotFound),
Err(e) => Err(e.into()),
}
}
async fn exists(&self, key: &str) -> Result<bool, StorageError> {
let path: &Path = &self.resolve(key)?;
Ok(fs::try_exists(path).await?)
@@ -227,6 +256,21 @@ mod tests {
assert!(matches!(s.size("adir").await, Err(StorageError::NotFound)));
}
#[tokio::test]
async fn get_on_directory_is_not_found() {
// A key resolving to a directory isn't a stored blob. `fs::read` on a
// dir errors with EISDIR (not NotFound), and `File::open` on a dir
// succeeds on Unix then streams garbage — both must surface NotFound.
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
std::fs::create_dir(dir.path().join("adir")).unwrap();
assert!(matches!(s.get("adir").await, Err(StorageError::NotFound)));
assert!(matches!(
s.get_stream("adir").await.err(),
Some(StorageError::NotFound)
));
}
#[tokio::test]
async fn put_stream_writes_full_body_and_removes_temp_on_error() {
use bytes::Bytes;
@@ -273,6 +317,34 @@ mod tests {
assert_eq!(entries, vec!["ok.bin"]);
}
#[tokio::test]
async fn rename_moves_blob_and_creates_destination_dirs() {
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
s.put("staging/up/0000.png", b"page-bytes").await.unwrap();
// Destination dir doesn't exist yet — rename must create it.
s.rename("staging/up/0000.png", "mangas/m/chapters/c/pages/0001.png")
.await
.unwrap();
assert!(!s.exists("staging/up/0000.png").await.unwrap(), "source gone");
assert_eq!(
s.get("mangas/m/chapters/c/pages/0001.png").await.unwrap(),
b"page-bytes"
);
}
#[tokio::test]
async fn rename_missing_source_is_not_found() {
let dir = tempdir().unwrap();
let s = LocalStorage::new(dir.path());
assert!(matches!(
s.rename("staging/nope.png", "dest/x.png").await,
Err(StorageError::NotFound)
));
}
#[tokio::test]
async fn get_stream_emits_multiple_chunks_for_large_files() {
use futures_util::StreamExt as _;

View File

@@ -82,6 +82,27 @@ pub trait Storage: Send + Sync {
async fn delete(&self, key: &str) -> Result<(), StorageError>;
async fn exists(&self, key: &str) -> Result<bool, StorageError>;
/// Move a blob from `from` to `to`, overwriting any existing blob at
/// `to`. Returns `NotFound` if `from` doesn't exist. The chapter
/// upload path uses this to promote a staged page to its final,
/// chapter-scoped key once the chapter row (and thus its id) exists —
/// so pages can be streamed to storage as their multipart parts
/// arrive, without buffering the whole chapter in memory.
///
/// The default implementation streams `from` to `to` and deletes the
/// source, so backends without a native move still satisfy the
/// contract. LocalStorage overrides it with a filesystem rename (an
/// atomic metadata op within a mount); a future S3Storage would
/// override with a server-side copy + delete.
async fn rename(&self, from: &str, to: &str) -> Result<(), StorageError> {
use futures_util::StreamExt as _;
let StreamingFile { stream, .. } = self.get_stream(from).await?;
let mapped: PutByteStream<'_> = Box::pin(stream.map(|r| r.map_err(StorageError::Io)));
self.put_stream(to, mapped).await?;
self.delete(from).await?;
Ok(())
}
/// Size in bytes of the blob at `key`, `NotFound` when it doesn't
/// exist. Cheap metadata lookup (local: `fs::metadata`; a future
/// `S3Storage`: HEAD object). Used by the cover-capture path and the

View File

@@ -6,7 +6,12 @@
//! whitelist with 415. Filename and extension never reach the storage
//! key — we derive both from the sniffed type.
use axum::extract::multipart::Field;
use uuid::Uuid;
use crate::api::mangas::map_multipart_error;
use crate::error::{AppError, AppResult};
use crate::storage::Storage;
#[derive(Debug, Clone)]
pub struct UploadedImage {
@@ -15,6 +20,100 @@ pub struct UploadedImage {
pub ext: &'static str,
}
/// A page image written to a temporary staging key during a chapter
/// upload, awaiting promotion to its final chapter-scoped key once the
/// chapter row (and thus its id) exists. Carries only the small metadata
/// the caller needs — never the image bytes.
#[derive(Debug, Clone)]
pub struct StagedImage {
pub staging_key: String,
pub mime: &'static str,
pub ext: &'static str,
pub size_bytes: i64,
}
/// The staging-key prefix. Blobs left here by a failed or abandoned upload
/// are orphans a future reaper can sweep; a successful upload renames every
/// staged page out of this prefix.
pub const STAGING_PREFIX: &str = "staging";
/// Upper bound on a multipart `metadata` JSON part, enforced as bytes arrive
/// (via [`read_capped`]). Manga/chapter metadata is title + a few short lists +
/// a description — kilobytes at most. Without this, `metadata` was the one
/// remaining part read with the unbounded `Field::bytes()`, letting a client
/// buffer up to the whole 200 MiB request body in memory as a single JSON blob
/// before any validation ran. 64 KiB is generous headroom over any legitimate
/// payload while keeping the worst case tiny.
pub const MAX_METADATA_BYTES: usize = 64 * 1024;
/// Read one multipart image part and write it straight to a staging key,
/// returning only its metadata. The per-file byte cap is enforced as bytes
/// arrive, so an oversized part is rejected (413) without being fully
/// buffered, and at most one page's bytes are held in memory at a time —
/// the whole chapter is never buffered, unlike the previous
/// read-all-then-persist path.
pub async fn stage_image_part(
storage: &dyn Storage,
field: Field<'_>,
upload_id: Uuid,
seq: usize,
max_size: usize,
field_name: &str,
) -> AppResult<StagedImage> {
let bytes = read_capped(field, max_size, field_name).await?;
// Reuse the shared sniff + whitelist check; it re-verifies the size cap
// (already enforced above) and derives mime/ext from magic bytes.
let img = parse_image(bytes, max_size, field_name)?;
let staging_key = format!("{STAGING_PREFIX}/{}/{:04}.{}", upload_id.simple(), seq, img.ext);
storage.put(&staging_key, &img.bytes).await?;
Ok(StagedImage {
staging_key,
mime: img.mime,
ext: img.ext,
size_bytes: img.bytes.len() as i64,
})
}
/// Read one multipart part fully into memory, enforcing the per-file byte cap
/// as chunks arrive so an oversized part is rejected (413) **without being fully
/// buffered**. Use for parts whose bytes the caller needs in hand — e.g. the
/// cover image, which is written to a manga-scoped key. Page parts should prefer
/// [`stage_image_part`], which streams straight to storage.
///
/// Replaces the `Field::bytes()` path, which buffered the entire field before
/// any size check ran, letting an attacker allocate an arbitrarily large body
/// before the cap kicked in.
pub async fn read_capped(
mut field: Field<'_>,
max_size: usize,
field_name: &str,
) -> AppResult<Vec<u8>> {
let mut bytes: Vec<u8> = Vec::new();
while let Some(chunk) = field.chunk().await.map_err(map_multipart_error)? {
push_capped(&mut bytes, &chunk, max_size, field_name)?;
}
Ok(bytes)
}
/// Append `chunk` to `buf`, rejecting with 413 as soon as the running total
/// would exceed `max_size` — so the oversized chunk is never copied in. Split
/// out from the read loop so the cap logic is unit-testable without a live
/// multipart field.
fn push_capped(
buf: &mut Vec<u8>,
chunk: &[u8],
max_size: usize,
field_name: &str,
) -> AppResult<()> {
if buf.len().saturating_add(chunk.len()) > max_size {
return Err(AppError::PayloadTooLarge(format!(
"{field_name} exceeds {max_size}-byte cap"
)));
}
buf.extend_from_slice(chunk);
Ok(())
}
pub fn parse_image(bytes: Vec<u8>, max_size: usize, field_name: &str) -> AppResult<UploadedImage> {
if bytes.len() > max_size {
return Err(AppError::PayloadTooLarge(format!(
@@ -114,4 +213,30 @@ mod tests {
assert!(matches!(err, AppError::PayloadTooLarge(_)));
assert_eq!(err.code(), "payload_too_large");
}
#[test]
fn push_capped_accumulates_under_cap() {
let mut buf = Vec::new();
push_capped(&mut buf, b"hello ", 100, "cover").unwrap();
push_capped(&mut buf, b"world", 100, "cover").unwrap();
assert_eq!(buf, b"hello world");
}
#[test]
fn push_capped_rejects_before_copying_oversized_chunk() {
// The whole point: the offending chunk must NOT be appended — an
// oversized part is rejected without buffering it.
let mut buf = vec![0u8; 90];
let err = push_capped(&mut buf, &[0u8; 20], 100, "cover").unwrap_err();
assert!(matches!(err, AppError::PayloadTooLarge(_)));
assert_eq!(err.code(), "payload_too_large");
assert_eq!(buf.len(), 90, "buffer must not grow past the cap");
}
#[test]
fn push_capped_allows_exactly_at_cap() {
let mut buf = vec![0u8; 90];
push_capped(&mut buf, &[0u8; 10], 100, "cover").unwrap();
assert_eq!(buf.len(), 100);
}
}

View File

@@ -56,6 +56,7 @@ fn ocr_dispatcher(
storage,
engine: StubOcrEngine::new(lines),
max_image_bytes: 8 * 1024 * 1024,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(1)),
}
}
@@ -98,6 +99,36 @@ async fn dispatch_persists_ocr_lines_and_search_doc(pool: PgPool) {
assert!(has_doc, "search_doc must be populated from OCR text");
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_rejects_page_image_over_the_byte_cap(pool: PgPool) {
let dir = TempDir::new().unwrap();
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(dir.path()));
let key = "mangas/x/big.png";
// 4 KiB on disk, 1 KiB cap — the streamed read must bail on the cap
// rather than buffering the whole blob and OCR-ing it.
storage.put(key, &vec![0u8; 4096]).await.unwrap();
let page_id = seed_page(&pool, key).await;
let dispatcher = OcrAnalyzeDispatcher {
db: pool.clone(),
storage: Arc::clone(&storage),
engine: StubOcrEngine::new(&["should not run"]),
max_image_bytes: 1024,
ocr_permits: Arc::new(tokio::sync::Semaphore::new(1)),
};
let err = dispatcher.dispatch(page_id).await.unwrap_err();
assert!(
err.chain().any(|c| c.to_string().contains("cap")),
"expected an over-cap error, got: {err:#}"
);
// A rejected page must not land an analysis row.
assert!(repo::page_analysis::load(&pool, page_id)
.await
.unwrap()
.is_none());
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_missing_page_is_noop(pool: PgPool) {
let dir = TempDir::new().unwrap();

View File

@@ -74,6 +74,38 @@ async fn analyze_job_count(pool: &PgPool) -> i64 {
.unwrap()
}
#[sqlx::test(migrations = "./migrations")]
async fn force_analyze_enqueue_rolls_back_with_its_transaction(pool: PgPool) {
// The admin force-reanalyze handler runs enqueue_for_page_conn + the audit
// insert in one transaction. Prove the enqueue genuinely participates in
// that tx: when the tx is abandoned (the path taken if the audit insert
// fails and the `?` propagates), no job survives — so the audit trail can
// never miss a force-reanalyze that actually landed.
let page_id = seed_pages(&pool, 1).await[0];
let mut tx = pool.begin().await.unwrap();
let outcome =
mangalord::repo::page_analysis::enqueue_for_page_conn(&mut tx, page_id, true)
.await
.unwrap();
assert!(matches!(
outcome,
mangalord::repo::page_analysis::EnqueueForPageOutcome::Inserted
));
// Visible inside the open transaction…
let in_tx: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM crawler_jobs WHERE payload->>'kind' = 'analyze_page'",
)
.fetch_one(&mut *tx)
.await
.unwrap();
assert_eq!(in_tx, 1);
// …but gone once the tx rolls back instead of committing.
tx.rollback().await.unwrap();
assert_eq!(analyze_job_count(&pool).await, 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_upload_enqueues_one_analysis_job_per_page(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
@@ -157,6 +189,51 @@ async fn reenqueue_returns_503_when_analysis_disabled(pool: PgPool) {
assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_rejects_a_malformed_body(pool: PgPool) {
// A present-but-malformed JSON body must be a 422 — NOT silently coerced to
// an absent body, which would run the default full "All" scope the caller
// never asked for and enqueue jobs across the whole library.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_raw_with_cookie(
"/api/v1/admin/analysis/reenqueue",
"{ not valid json",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNPROCESSABLE_ENTITY);
assert_eq!(analyze_job_count(&pool).await, 0, "a rejected body enqueues nothing");
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_treats_empty_body_as_default_all_scope(pool: PgPool) {
// An absent/empty body is still valid: it means the default "All" scope.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_raw_with_cookie(
"/api/v1/admin/analysis/reenqueue",
"",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["enqueued"], 2, "empty body backfills the whole library");
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_backfills_existing_pages(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());
@@ -194,6 +271,38 @@ async fn reenqueue_backfills_existing_pages(pool: PgPool) {
assert_eq!(analyze_job_count(&pool).await, 3);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_writes_audit_row_atomically_with_jobs(pool: PgPool) {
// The enqueue and its admin_audit row are committed in one transaction,
// so a successful re-enqueue always leaves both the jobs AND exactly one
// matching audit row — the audit trail can't miss an enqueue that landed.
let h = common::harness_with_analysis(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
seed_pages(&pool, 2).await;
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/analysis/reenqueue",
json!({ "only_unanalyzed": true }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(analyze_job_count(&pool).await, 2);
let audit: serde_json::Value = sqlx::query_scalar(
"SELECT payload FROM admin_audit WHERE action = 'analysis_reenqueue'",
)
.fetch_one(&pool)
.await
.expect("exactly one analysis_reenqueue audit row committed with the jobs");
assert_eq!(audit["enqueued"], 2);
assert_eq!(audit["only_unanalyzed"], true);
}
#[sqlx::test(migrations = "./migrations")]
async fn reenqueue_scoped_to_manga(pool: PgPool) {
let h = common::harness_with_analysis(pool.clone());

View File

@@ -279,7 +279,7 @@ async fn backfill_fills_unmeasured_pages_and_covers_idempotently(pool: PgPool) {
let body2 = common::body_json(resp2).await;
assert_eq!(body2["pages"].as_i64().unwrap(), 0);
assert_eq!(body2["covers"].as_i64().unwrap(), 0);
assert_eq!(body2["more_remaining"].as_bool().unwrap(), false);
assert!(!body2["more_remaining"].as_bool().unwrap());
}
#[sqlx::test(migrations = "./migrations")]
@@ -362,7 +362,7 @@ async fn backfill_caps_per_run_and_reports_more_remaining(pool: PgPool) {
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
// Capped at 20_000 attempts this run; one row left → run again.
assert_eq!(body["more_remaining"].as_bool().unwrap(), true);
assert!(body["more_remaining"].as_bool().unwrap());
assert_eq!(body["missing"].as_i64().unwrap(), 20_000);
assert_eq!(body["pages"].as_i64().unwrap(), 0);
}
@@ -403,9 +403,8 @@ async fn backfill_at_exact_cap_is_not_more_remaining(pool: PgPool) {
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["missing"].as_i64().unwrap(), 20_000);
assert_eq!(
body["more_remaining"].as_bool().unwrap(),
false,
assert!(
!body["more_remaining"].as_bool().unwrap(),
"exactly-cap backlog drains in one run; no follow-up needed"
);
}

View File

@@ -147,6 +147,46 @@ async fn list_filters_by_substring_search(pool: PgPool) {
assert_eq!(body["page"]["total"], 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn search_treats_like_wildcards_literally(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin_name, cookie, _) = seed_admin(&pool, &h.app).await;
// Two usernames differing only at one position (underscores are legal in
// usernames). `_` is a LIKE single-char wildcard: unescaped, `%a_b%` matches
// BOTH; escaped, only the literal "a_b" username. Admin user search has no
// trigram OR, so length/similarity don't matter here.
for username in ["axbfindme", "a_bfindme"] {
let resp = h
.app
.clone()
.oneshot(common::post_json(
"/api/v1/auth/register",
json!({ "username": username, "password": "hunter2hunter2" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
}
let resp = h
.app
.oneshot(common::get_with_cookie(
"/api/v1/admin/users?search=a_b",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(
items.len(),
1,
"the `_` must match literally: only a_bfindme, not axbfindme"
);
assert_eq!(items[0]["username"], "a_bfindme");
}
// ---- self-protection -------------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
@@ -381,10 +421,11 @@ async fn delete_writes_audit_row(pool: PgPool) {
.unwrap();
assert_eq!(resp.status(), StatusCode::NO_CONTENT);
let rows: Vec<(Option<Uuid>, String, String, Option<Uuid>, serde_json::Value)> =
sqlx::query_as(
"SELECT actor_user_id, action, target_kind, target_id, payload FROM admin_audit",
)
// (actor_user_id, action, target_kind, target_id, payload)
type AuditRow = (Option<Uuid>, String, String, Option<Uuid>, serde_json::Value);
let rows: Vec<AuditRow> = sqlx::query_as(
"SELECT actor_user_id, action, target_kind, target_id, payload FROM admin_audit",
)
.fetch_all(&pool)
.await
.unwrap();

View File

@@ -9,6 +9,73 @@ fn creds(username: &str) -> serde_json::Value {
json!({ "username": username, "password": "hunter2hunter2" })
}
/// A login request carrying a chosen `X-Forwarded-For`. Empty password so the
/// handler consumes a rate-limit token then short-circuits with 400 before any
/// argon2/DB work — the burst drains near-instantly regardless of CI load.
fn login_from(xff: &str) -> axum::http::Request<axum::body::Body> {
axum::http::Request::builder()
.method("POST")
.uri("/api/v1/auth/login")
.header(header::CONTENT_TYPE, "application/json")
.header("x-forwarded-for", xff)
.body(axum::body::Body::from(
json!({ "username": "victim", "password": "" }).to_string(),
))
.unwrap()
}
// The per-IP rate limiter's soundness hinges on one gate: X-Forwarded-For is
// honored ONLY when AUTH_TRUSTED_PROXY is set. These two tests pin both sides of
// that gate at the request level (the pure parser is unit-tested separately).
#[sqlx::test(migrations = "./migrations")]
async fn trusted_proxy_gives_each_forwarded_ip_its_own_bucket(pool: PgPool) {
let h = common::harness_with_auth_rate_limit_proxy(pool, 1, 2, true);
// Drain IP A's bucket (per_sec=1, burst=2) until it 429s.
let mut a_saw_429 = false;
for _ in 0..8 {
let resp = h.app.clone().oneshot(login_from("203.0.113.10")).await.unwrap();
if resp.status() == StatusCode::TOO_MANY_REQUESTS {
a_saw_429 = true;
break;
}
}
assert!(a_saw_429, "IP A must be rate-limited after draining its own bucket");
// A different forwarded IP has an independent bucket — its first hit is not 429.
let resp_b = h.app.clone().oneshot(login_from("198.51.100.20")).await.unwrap();
assert_ne!(
resp_b.status(),
StatusCode::TOO_MANY_REQUESTS,
"a distinct X-Forwarded-For hop must get its own bucket, not A's drained one"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn untrusted_proxy_ignores_forwarded_ip_and_shares_one_bucket(pool: PgPool) {
let h = common::harness_with_auth_rate_limit_proxy(pool, 1, 2, false);
// Every request carries a DISTINCT (spoofed) X-Forwarded-For, but the backend
// doesn't trust it — so they all fall back to the single global bucket, which
// drains and starts returning 429. If XFF were wrongly honored here, each
// unique IP would get a fresh bucket and none of these would ever 429.
let mut saw_429 = false;
for i in 0..12 {
let resp = h
.app
.clone()
.oneshot(login_from(&format!("10.0.0.{i}")))
.await
.unwrap();
if resp.status() == StatusCode::TOO_MANY_REQUESTS {
saw_429 = true;
break;
}
}
assert!(
saw_429,
"with trusted_proxy off, spoofed X-Forwarded-For must NOT dodge the shared limiter"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn register_creates_user_and_sets_session_cookie(pool: PgPool) {
let h = common::harness(pool);
@@ -170,6 +237,26 @@ async fn login_rejects_wrong_password(pool: PgPool) {
assert_eq!(body["error"]["code"], "unauthenticated");
}
#[sqlx::test(migrations = "./migrations")]
async fn login_rejects_oversized_password_before_argon2(pool: PgPool) {
// A multi-KB password on the login path must be rejected as malformed
// input (400) rather than fed to argon2 — otherwise every attempt is a
// CPU-DoS. The account need not even exist; the guard is input-shape only.
let h = common::harness(pool);
let giant = "a".repeat(5000);
let resp = h
.app
.oneshot(common::post_json(
"/api/v1/auth/login",
json!({ "username": "alice", "password": giant }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "invalid_input");
}
#[sqlx::test(migrations = "./migrations")]
async fn login_rejects_unknown_user(pool: PgPool) {
let h = common::harness(pool);
@@ -521,6 +608,156 @@ async fn create_and_use_bot_token(pool: PgPool) {
assert_eq!(resp.status(), StatusCode::OK);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_tokens_returns_callers_tokens_scoped_and_without_hash(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
// Mint two tokens for this user, one with an expiry.
for body in [
json!({ "name": "no-expiry" }),
json!({ "name": "expiring", "expires_in_days": 30 }),
] {
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/auth/tokens",
body,
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
}
// A second user's token must NOT appear in the first user's list.
let (_, other) = common::register_user(&h.app).await;
let _ = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/auth/tokens",
json!({ "name": "someone-elses" }),
&other,
))
.await
.unwrap();
let resp = h
.app
.oneshot(common::get_with_cookie("/api/v1/auth/tokens", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(items.len(), 2, "only the caller's two tokens");
let names: Vec<&str> = items.iter().map(|t| t["name"].as_str().unwrap()).collect();
assert!(names.contains(&"no-expiry") && names.contains(&"expiring"));
// Raw secret / hash must never appear, but expiry metadata must.
for t in items {
assert!(t.get("token_hash").is_none(), "token_hash must be absent");
assert!(t.get("bearer").is_none(), "raw bearer only shown at creation");
assert!(t.get("expires_at").is_some(), "expiry metadata present");
}
}
#[sqlx::test(migrations = "./migrations")]
async fn bot_token_with_future_expiry_authenticates(pool: PgPool) {
// A token minted with expires_in_days is still active before its
// expiry, and the response echoes a non-null expires_at.
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/auth/tokens",
json!({ "name": "ci-bot", "expires_in_days": 30 }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let body = common::body_json(resp).await;
assert!(
body["expires_at"].is_string(),
"expires_at should be set, got {}",
body["expires_at"]
);
let bearer = body["bearer"].as_str().unwrap().to_string();
let resp = h
.app
.oneshot(common::get_with_bearer("/api/v1/auth/me", &bearer))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
#[sqlx::test(migrations = "./migrations")]
async fn expired_bot_token_is_rejected(pool: PgPool) {
use chrono::{Duration, Utc};
use mangalord::auth::token::generate_token;
let h = common::harness(pool.clone());
common::register_user(&h.app).await;
let user_id: uuid::Uuid = sqlx::query_scalar("SELECT id FROM users LIMIT 1")
.fetch_one(&pool)
.await
.unwrap();
// Hand-craft a token that expired an hour ago.
let (raw, hash) = generate_token();
let expires_at = Utc::now() - Duration::hours(1);
sqlx::query(
"INSERT INTO api_tokens (user_id, name, token_hash, expires_at) \
VALUES ($1, 'stale', $2, $3)",
)
.bind(user_id)
.bind(&hash[..])
.bind(expires_at)
.execute(&pool)
.await
.unwrap();
let resp = h
.app
.oneshot(common::get_with_bearer("/api/v1/auth/me", &raw))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "unauthenticated");
}
#[sqlx::test(migrations = "./migrations")]
async fn create_token_rejects_out_of_range_expiry(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
for days in [0, -5, 100_000] {
let resp = h
.app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/auth/tokens",
json!({ "name": "bad", "expires_in_days": days }),
&cookie,
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::UNPROCESSABLE_ENTITY,
"expires_in_days={days} should be rejected"
);
}
}
#[sqlx::test(migrations = "./migrations")]
async fn user_a_cannot_delete_user_b_token(pool: PgPool) {
let h = common::harness(pool);

View File

@@ -30,6 +30,47 @@ fn first_author_id(manga: &Value) -> String {
manga["authors"][0]["id"].as_str().unwrap().to_string()
}
#[sqlx::test(migrations = "./migrations")]
async fn search_treats_like_wildcards_literally(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
// Long author names differing only at one position, short search term — so
// the trigram OR (which keeps the raw term by design) stays under threshold
// and the ILIKE branch is what decides. Unescaped `%a_b%` matches both the
// "axb" and "a_b" names; escaped, only the literal "a_b" name.
create_manga(
&h.app,
&cookie,
json!({ "title": "M1", "authors": ["The Quick Brown Fox axb Jumps Over"] }),
)
.await;
create_manga(
&h.app,
&cookie,
json!({ "title": "M2", "authors": ["The Quick Brown Fox a_b Jumps Over"] }),
)
.await;
let resp = h
.app
.oneshot(common::get("/api/v1/authors?search=a_b"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let names: Vec<&str> = body
.as_array()
.unwrap()
.iter()
.map(|a| a["name"].as_str().unwrap())
.collect();
assert_eq!(
names,
vec!["The Quick Brown Fox a_b Jumps Over"],
"the `_` in the search term must match literally, not as a wildcard"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_returns_name_and_manga_count(pool: PgPool) {
let h = common::harness(pool);

View File

@@ -52,7 +52,7 @@ async fn create_then_list_returns_only_own(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn create_returns_409_on_duplicate_manga_level(pool: PgPool) {
async fn create_is_idempotent_on_duplicate_manga_level(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
@@ -64,12 +64,26 @@ async fn create_returns_409_on_duplicate_manga_level(pool: PgPool) {
&cookie,
)
};
// First add creates (201); re-adding is an idempotent no-op (200) that
// returns the SAME bookmark rather than a 409 — collections behave this way
// too, so the UI shouldn't surface a false "Could not add bookmark" error.
let first = h.app.clone().oneshot(make()).await.unwrap();
assert_eq!(first.status(), StatusCode::CREATED);
let second = h.app.oneshot(make()).await.unwrap();
assert_eq!(second.status(), StatusCode::CONFLICT);
let body = common::body_json(second).await;
assert_eq!(body["error"]["code"], "conflict");
let first_body = common::body_json(first).await;
let second = h.app.clone().oneshot(make()).await.unwrap();
assert_eq!(second.status(), StatusCode::OK);
let second_body = common::body_json(second).await;
assert_eq!(second_body["id"], first_body["id"], "same bookmark returned");
// Exactly one row exists.
let list = h
.app
.oneshot(common::get_with_cookie("/api/v1/me/bookmarks", &cookie))
.await
.unwrap();
let body = common::body_json(list).await;
assert_eq!(body["items"].as_array().unwrap().len(), 1);
}
#[sqlx::test(migrations = "./migrations")]
@@ -225,13 +239,16 @@ async fn concurrent_manga_bookmarks_serialised_by_unique_index(pool: PgPool) {
let (s1, s2) = tokio::join!(f1, f2);
let statuses = [s1.unwrap(), s2.unwrap()];
// The unique index serialises the two inserts: one wins with 201, the other
// hits the violation and — now idempotent — returns the existing bookmark
// with 200 rather than a 409.
assert!(
statuses.contains(&StatusCode::CREATED),
"expected one winner with 201, got {statuses:?}"
);
assert!(
statuses.contains(&StatusCode::CONFLICT),
"expected one loser with 409 (the partial unique index), got {statuses:?}"
statuses.contains(&StatusCode::OK),
"expected the loser to return 200 (idempotent), got {statuses:?}"
);
}

View File

@@ -8,6 +8,8 @@ use uuid::Uuid;
#[allow(unused_imports)]
use serde_json as _;
use common::MultipartBuilder;
async fn seed_manga(h: &common::Harness, cookie: &str, title: &str) -> Uuid {
common::seed_manga_via_api(&h.app, cookie, title).await
}
@@ -146,6 +148,49 @@ async fn list_chapters_returned_in_number_order(pool: PgPool) {
assert_eq!(body["items"][1]["title"], serde_json::Value::Null);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_interleaves_uploaded_and_crawled_chapters_by_number(pool: PgPool) {
// Mixed source: crawled chapters carry a `source_index`; an uploaded
// chapter has none. The uploaded chapter must interleave by NUMBER, not sort
// after every crawled chapter (the old `source_index DESC NULLS LAST` bug
// dumped uploaded chapter 2 to the end, giving [1, 3, 2]).
let h = common::harness(pool.clone());
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = seed_manga(&h, &cookie, "Berserk").await;
// Two crawled chapters (1 and 3) with DOM positions — newest-first, so
// chapter 3 is at source_index 0 and chapter 1 at source_index 1.
let c1 = seed_chapter(&pool, manga_id, 1, Some("Crawled One")).await;
let c3 = seed_chapter(&pool, manga_id, 3, Some("Crawled Three")).await;
sqlx::query("UPDATE chapters SET source_index = 1 WHERE id = $1")
.bind(c1)
.execute(&pool)
.await
.unwrap();
sqlx::query("UPDATE chapters SET source_index = 0 WHERE id = $1")
.bind(c3)
.execute(&pool)
.await
.unwrap();
// Uploaded chapter 2 — no source_index.
seed_chapter(&pool, manga_id, 2, Some("Uploaded Two")).await;
let resp = h
.app
.oneshot(common::get(&format!("/api/v1/mangas/{manga_id}/chapters")))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let numbers: Vec<i64> = body["items"]
.as_array()
.unwrap()
.iter()
.map(|c| c["number"].as_i64().unwrap())
.collect();
assert_eq!(numbers, vec![1, 2, 3], "uploaded chapter 2 must slot between 1 and 3");
}
#[sqlx::test(migrations = "./migrations")]
async fn list_chapters_returns_404_for_unknown_manga(pool: PgPool) {
let h = common::harness(pool);
@@ -272,3 +317,68 @@ async fn list_pages_returns_404_for_unknown_chapter(pool: PgPool) {
.unwrap();
assert_eq!(resp.status(), StatusCode::NOT_FOUND);
}
#[sqlx::test(migrations = "./migrations")]
async fn non_owner_can_upload_chapter(pool: PgPool) {
// Contract lock: chapter upload is INTENTIONALLY open to any
// authenticated user, not just the manga's creator. A user who did not
// create the manga can still contribute a chapter (community
// contributions), unlike manga-record edits which gate on ownership.
// If this ever needs to become owner-only, that's a deliberate change —
// this test (and the doc comment on `api::chapters::create`) should be
// updated together, not silently broken.
let h = common::harness(pool);
// Owner creates the manga.
let (_owner, owner_cookie) = common::register_user(&h.app).await;
let manga_id = seed_manga(&h, &owner_cookie, "Berserk").await;
// A different, non-owner user uploads a chapter to it.
let (_other, other_cookie) = common::register_user(&h.app).await;
let resp = h
.app
.clone()
.oneshot(common::post_multipart_with_cookie(
&format!("/api/v1/mangas/{manga_id}/chapters"),
MultipartBuilder::new()
.add_json("metadata", json!({ "number": 1, "title": "Contributed" }))
.add_file("page", "1.png", "image/png", &common::fake_png_bytes()),
&other_cookie,
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::CREATED,
"a non-owner authenticated user must be able to upload a chapter"
);
let body = common::body_json(resp).await;
assert_eq!(body["number"], 1);
assert_eq!(body["title"], "Contributed");
assert_eq!(body["page_count"], 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_upload_rejects_oversized_metadata_part(pool: PgPool) {
// The chapter `metadata` JSON part is capped as bytes arrive, just like the
// manga one, so a client can't buffer a huge JSON blob in memory. A ~200 KiB
// title blows the metadata cap before any page is staged.
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = seed_manga(&h, &cookie, "Berserk").await;
let huge = "A".repeat(200 * 1024);
let resp = h
.app
.oneshot(common::post_multipart_with_cookie(
&format!("/api/v1/mangas/{manga_id}/chapters"),
MultipartBuilder::new()
.add_json("metadata", json!({ "number": 1, "title": huge }))
.add_file("page", "1.png", "image/png", &common::fake_png_bytes()),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::PAYLOAD_TOO_LARGE);
}

View File

@@ -0,0 +1,74 @@
mod common;
use axum::http::StatusCode;
use tower::ServiceExt;
/// Write a blob straight into the harness storage root at `key`, bypassing the
/// upload handlers — the only way to land a key whose extension resolves to the
/// `application/octet-stream` fallback (uploads always mint image extensions).
fn write_blob(h: &common::Harness, key: &str, bytes: &[u8]) {
let path = h._storage_dir.path().join(key);
std::fs::create_dir_all(path.parent().unwrap()).unwrap();
std::fs::write(path, bytes).unwrap();
}
#[sqlx::test(migrations = "./migrations")]
async fn octet_stream_blobs_are_served_as_attachment(pool: sqlx::PgPool) {
// A blob with an unknown extension serves as application/octet-stream. Such a
// body could be crafted HTML/JS, so it must never render inline: force a
// download with Content-Disposition: attachment (nosniff is already set).
let h = common::harness(pool);
write_blob(&h, "misc/blob.bin", b"\x00\x01not-an-image");
let resp = h
.app
.oneshot(common::get("/api/v1/files/misc/blob.bin"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(
resp.headers().get("content-type").unwrap(),
"application/octet-stream"
);
assert_eq!(
resp.headers().get("content-disposition").unwrap(),
"attachment"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn image_blobs_are_served_inline(pool: sqlx::PgPool) {
// Regression guard: known image types keep rendering inline (no attachment
// disposition), so covers/pages still display in the reader.
let h = common::harness(pool);
write_blob(&h, "misc/pic.png", &common::fake_png_bytes());
let resp = h
.app
.oneshot(common::get("/api/v1/files/misc/pic.png"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(resp.headers().get("content-type").unwrap(), "image/png");
assert!(
resp.headers().get("content-disposition").is_none(),
"images must render inline, not download"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn corrupt_image_thumbnail_falls_back_to_original_not_500(pool: sqlx::PgPool) {
// A blob with valid PNG magic bytes but an undecodable body passes upload's
// magic-byte sniff; a ?w= thumbnail request then tries to decode it. That
// must not 500 the reader — fall back to streaming the original bytes.
let h = common::harness(pool);
write_blob(&h, "misc/corrupt.png", &common::fake_png_bytes());
let resp = h
.app
.oneshot(common::get("/api/v1/files/misc/corrupt.png?w=320"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK, "corrupt thumbnail must not 500");
assert_eq!(resp.headers().get("content-type").unwrap(), "image/png");
}

View File

@@ -164,6 +164,199 @@ async fn list_is_per_user_only(pool: PgPool) {
assert_eq!(body["items"], json!([]));
}
#[sqlx::test(migrations = "./migrations")]
async fn list_reports_new_chapters_since_last_read(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
// Three chapters exist; the reader is on chapter 1, so chapters 2 and
// 3 are "new since last read".
let ch1 = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 3).await;
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "chapter_id": ch1, "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/read-progress", &cookie))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["items"][0]["new_chapters_count"], 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_reports_last_read_chapter_page_count(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
// seed_chapter uploads a single page, so page_count is 1.
let ch1 = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "chapter_id": ch1, "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/read-progress", &cookie))
.await
.unwrap();
let body = common::body_json(resp).await;
// Exposes the last-read chapter's page count so a client can tell a
// finished series (on the last page) from one still in progress.
assert_eq!(body["items"][0]["chapter_page_count"], 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_new_chapters_count_is_zero_at_latest_chapter(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let ch2 = seed_chapter(&h.app, &cookie, manga_id, 2).await;
// Caught up to the last chapter → nothing newer.
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "chapter_id": ch2, "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/read-progress", &cookie))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["items"][0]["new_chapters_count"], 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_new_chapters_count_is_zero_without_a_read_chapter(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
// Progress recorded without a chapter (manga-level) — we can't place
// the reader among the chapters, so we don't claim any are "new".
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/read-progress", &cookie))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["items"][0]["new_chapters_count"], 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_new_chapters_count_dedupes_same_numbered_scanlations(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
// (manga_id, number) is non-unique — two scanlations of chapter 2.
// "New since last read" should count distinct chapter numbers past the
// reader (2, 3 → 2), not raw rows (2, 2, 3 → 3).
let ch1 = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 3).await;
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "chapter_id": ch1, "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/read-progress", &cookie))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["items"][0]["new_chapters_count"], 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_single_manga_reports_new_chapters_count(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let ch1 = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
// Duplicate scanlation of chapter 3 must not inflate the count.
let _ = seed_chapter(&h.app, &cookie, manga_id, 3).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 3).await;
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "chapter_id": ch1, "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie(
&format!("/api/v1/me/read-progress/{manga_id}"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
// Distinct numbers past chapter 1: {2, 3} → 2.
assert_eq!(body["new_chapters_count"], 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_single_manga_new_chapters_count_zero_without_read_chapter(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 1).await;
let _ = seed_chapter(&h.app, &cookie, manga_id, 2).await;
// Manga-level progress (no chapter) → position unknown → 0.
let _ = upsert_progress(
&h.app,
&cookie,
json!({ "manga_id": manga_id.to_string(), "page": 1 }),
)
.await;
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie(
&format!("/api/v1/me/read-progress/{manga_id}"),
&cookie,
))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(body["new_chapters_count"], 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_single_manga_returns_404_when_unread(pool: PgPool) {
let h = common::harness(pool);

View File

@@ -127,6 +127,46 @@ async fn list_filters_by_content_warning(pool: PgPool) {
assert_eq!(ids, want);
}
#[sqlx::test(migrations = "./migrations")]
async fn denormalized_warnings_track_chapter_deletion(pool: PgPool) {
// The denormalized manga_content_warnings table must stay in sync when the
// underlying pages disappear. Deleting the only chapter (which cascade-
// deletes its pages and their page_content_warnings) must drop the manga
// from the include filter.
let h = common::harness(pool.clone());
let gory = seed_manga(&pool, "Gory", &[&["gore"]]).await;
// Present in the denorm table and matched by the filter.
let count: i64 =
sqlx::query_scalar("SELECT count(*) FROM manga_content_warnings WHERE manga_id = $1")
.bind(gory)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(count, 1, "warning denormalized on insert");
assert_eq!(list_ids(&h.app, "cw_include=gore").await, vec![gory.to_string()]);
// Delete the chapter → cascade removes pages + page_content_warnings →
// trigger recomputes the (now empty) set for this manga.
sqlx::query("DELETE FROM chapters WHERE manga_id = $1")
.bind(gory)
.execute(&pool)
.await
.unwrap();
let count: i64 =
sqlx::query_scalar("SELECT count(*) FROM manga_content_warnings WHERE manga_id = $1")
.bind(gory)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(count, 0, "warning removed after chapter (and pages) deleted");
assert!(
list_ids(&h.app, "cw_include=gore").await.is_empty(),
"manga no longer matches the include filter"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_rejects_unknown_warning(pool: PgPool) {
let h = common::harness(pool.clone());

View File

@@ -99,6 +99,66 @@ async fn list_returns_total_count_independent_of_pagination(pool: PgPool) {
assert_eq!(body["page"]["total"], 3);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_total_is_computed_only_on_the_first_page(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
for title in ["One Piece", "Berserk", "Vinland Saga"] {
seed(&h.app, &cookie, title).await;
}
// Page 1 (offset 0): total is the full population.
let body0 = common::body_json(
h.app
.clone()
.oneshot(common::get("/api/v1/mangas?limit=2&offset=0"))
.await
.unwrap(),
)
.await;
assert_eq!(body0["page"]["total"], 3);
// Page 2 (offset 2): total is omitted (null) — the correlated count is
// not recomputed on every page. Items still paginate correctly.
let body1 = common::body_json(
h.app
.oneshot(common::get("/api/v1/mangas?limit=2&offset=2"))
.await
.unwrap(),
)
.await;
assert!(
body1["page"]["total"].is_null(),
"total should be null past the first page, got {}",
body1["page"]["total"]
);
assert_eq!(body1["items"].as_array().unwrap().len(), 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn search_treats_like_wildcards_literally(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
// Two long titles differing only at one position. `_` is a LIKE single-char
// wildcard: unescaped, `%a_b%` matches BOTH ("axb" and "a_b"); escaped, only
// the literal "a_b" title matches. The titles are long and the term short so
// trigram similarity stays under threshold — the ILIKE branch decides.
seed(&h.app, &cookie, "The Quick Brown Fox Jumps axb Over The Lazy Dog").await;
seed(&h.app, &cookie, "The Quick Brown Fox Jumps a_b Over The Lazy Dog").await;
let resp = h
.app
.oneshot(common::get("/api/v1/mangas?search=a_b"))
.await
.unwrap();
let body = common::body_json(resp).await;
assert_eq!(
title_list(&body),
vec!["The Quick Brown Fox Jumps a_b Over The Lazy Dog"],
"the `_` in the search term must match literally, not as a wildcard"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn search_via_trigram_tolerates_typos(pool: PgPool) {
let h = common::harness(pool);
@@ -446,7 +506,7 @@ async fn list_tie_break_orders_equal_keys_by_ascending_id(pool: PgPool) {
// Paginating one row at a time reproduces that exact sequence — no overlap,
// no gap — which only holds because the id tie-break makes the order total.
let paged: Vec<String> = vec![page(1, 0).await, page(1, 1).await, page(1, 2).await]
let paged: Vec<String> = [page(1, 0).await, page(1, 1).await, page(1, 2).await]
.iter()
.map(|b| b["items"][0]["title"].as_str().unwrap().to_string())
.collect();

View File

@@ -1,6 +1,7 @@
mod common;
use axum::http::StatusCode;
use http_body_util::BodyExt;
use serde_json::{json, Value};
use sqlx::PgPool;
use tower::ServiceExt;
@@ -45,6 +46,65 @@ fn cover_form(bytes: &[u8]) -> MultipartBuilder {
MultipartBuilder::new().add_file("cover", "cover.bin", "application/octet-stream", bytes)
}
/// A real, decodable PNG (unlike `fake_png_bytes`, which is only magic bytes)
/// so the thumbnail endpoint has something to resize.
fn real_png(width: u32, height: u32) -> Vec<u8> {
let mut buf = std::io::Cursor::new(Vec::new());
image::RgbImage::from_pixel(width, height, image::Rgb([10, 120, 200]))
.write_to(&mut buf, image::ImageFormat::Png)
.unwrap();
buf.into_inner()
}
#[sqlx::test(migrations = "./migrations")]
async fn files_serves_downscaled_thumbnail_variant(pool: PgPool) {
let h = harness(pool);
let (_, cookie) = register_user(&h.app).await;
let manga = create_manga_with_cover(&h.app, &cookie, "Thumb", None).await;
let id = id_of(&manga);
// Upload a real 800x400 PNG cover.
let resp = h
.app
.clone()
.oneshot(put_multipart_with_cookie(
&format!("/api/v1/mangas/{id}/cover"),
cover_form(&real_png(800, 400)),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let key = format!("mangas/{id}/cover.png");
// ?w=320 serves a 320px-wide variant with the source content-type.
let resp = h
.app
.clone()
.oneshot(get(&format!("/api/v1/files/{key}?w=320")))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
assert_eq!(
resp.headers().get("content-type").unwrap().to_str().unwrap(),
"image/png"
);
let body = resp.into_body().collect().await.unwrap().to_bytes();
let thumb = image::load_from_memory(&body).unwrap();
assert_eq!(thumb.width(), 320, "served a 320px-wide variant");
assert_eq!(thumb.height(), 160, "aspect ratio preserved");
// The un-parametrised request still serves the full-resolution original.
let resp = h
.app
.clone()
.oneshot(get(&format!("/api/v1/files/{key}")))
.await
.unwrap();
let full = resp.into_body().collect().await.unwrap().to_bytes();
assert_eq!(image::load_from_memory(&full).unwrap().width(), 800);
}
#[sqlx::test(migrations = "./migrations")]
async fn put_cover_sets_path_when_none_existed(pool: PgPool) {
let h = harness(pool);

View File

@@ -326,6 +326,54 @@ async fn patch_updates_status_authors_and_genres(pool: PgPool) {
assert_eq!(body["genres"][0]["name"], "Drama");
}
#[sqlx::test(migrations = "./migrations")]
async fn patch_authors_updates_precomputed_sort_author(pool: PgPool) {
// The precomputed mangas.sort_author (used by ?sort=author) must track
// author edits: changing the alphabetically-first author reorders the
// author sort. Two mangas whose relative author order flips after a PATCH.
let h = common::harness(pool.clone());
let (_, cookie) = common::register_user(&h.app).await;
let a = id_of(&create_manga(&h.app, &cookie, json!({ "title": "A", "authors": ["Zeta"] })).await);
let _b = id_of(&create_manga(&h.app, &cookie, json!({ "title": "B", "authors": ["Mid"] })).await);
// Initially: Mid < Zeta, so B before A.
let ids = list_titles(&h.app, "sort=author&order=asc").await;
assert_eq!(ids, vec!["B", "A"]);
// Re-author A to "Alpha", which now sorts first.
let resp = h
.app
.clone()
.oneshot(common::patch_json_with_cookie(
&format!("/api/v1/mangas/{a}"),
json!({ "authors": ["Alpha"] }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
// Now A (Alpha) sorts before B (Mid) — proving sort_author was recomputed.
let ids = list_titles(&h.app, "sort=author&order=asc").await;
assert_eq!(ids, vec!["A", "B"]);
}
async fn list_titles(app: &axum::Router, query: &str) -> Vec<String> {
let resp = app
.clone()
.oneshot(common::get(&format!("/api/v1/mangas?{query}")))
.await
.unwrap();
let body = common::body_json(resp).await;
body["items"]
.as_array()
.unwrap()
.iter()
.map(|m| m["title"].as_str().unwrap().to_string())
.collect()
}
#[sqlx::test(migrations = "./migrations")]
async fn patch_404_on_unknown_id(pool: PgPool) {
let h = common::harness(pool);
@@ -879,3 +927,25 @@ async fn bearer_authed_admin_cannot_edit_null_uploader(pool: PgPool) {
);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_rejects_oversized_metadata_part(pool: PgPool) {
// The metadata JSON part is capped well below the 200 MiB request limit so a
// client can't force the server to buffer a huge JSON blob in memory before
// any validation runs. A ~200 KiB description blows the metadata cap.
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let huge = "A".repeat(200 * 1024);
let resp = h
.app
.oneshot(common::post_multipart_with_cookie(
"/api/v1/mangas",
MultipartBuilder::new()
.add_json("metadata", json!({ "title": "Big", "description": huge })),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::PAYLOAD_TOO_LARGE);
}

View File

@@ -149,6 +149,56 @@ async fn private_mode_blocks_register_even_when_self_register_enabled(pool: PgPo
assert_eq!(body["error"]["code"], "forbidden");
}
#[sqlx::test(migrations = "./migrations")]
async fn private_mode_serves_files_as_private_not_public_cacheable(pool: PgPool) {
// In private mode /files is auth-gated, so a served blob must NOT carry a
// `public` Cache-Control: a shared cache / CDN would otherwise store it and
// hand it to anonymous clients, defeating the gate. It should be `private`.
//
// Register via a public harness on the shared pool so the session exists,
// then upload + fetch a cover through the private harness (same storage).
let public = common::harness(pool.clone());
let (_, cookie) = common::register_user(&public.app).await;
let private = common::harness_with_private_mode(pool);
let form = common::MultipartBuilder::new()
.add_json("metadata", json!({ "title": "Secret Library" }))
.add_file("cover", "cover.png", "image/png", &common::fake_png_bytes());
let created = private
.app
.clone()
.oneshot(common::post_multipart_with_cookie(
"/api/v1/mangas",
form,
&cookie,
))
.await
.unwrap();
assert_eq!(created.status(), StatusCode::CREATED);
let body = common::body_json(created).await;
let key = body["cover_image_path"].as_str().unwrap().to_string();
let resp = private
.app
.oneshot(common::get_with_cookie(
&format!("/api/v1/files/{key}"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let cc = resp
.headers()
.get(axum::http::header::CACHE_CONTROL)
.unwrap()
.to_str()
.unwrap();
assert!(
cc.starts_with("private") && !cc.contains("public"),
"private-mode blobs must be `private`, not `public`; got: {cc}"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn auth_config_reports_private_mode_and_effective_self_register(pool: PgPool) {
let h = common::harness_with_private_mode(pool);

View File

@@ -0,0 +1,133 @@
mod common;
use axum::http::StatusCode;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
async fn put_reaction(
app: &axum::Router,
cookie: &str,
manga_id: Uuid,
reaction: &str,
) -> StatusCode {
let resp = app
.clone()
.oneshot(common::put_json_with_cookie(
&format!("/api/v1/mangas/{manga_id}/reaction"),
json!({ "reaction": reaction }),
cookie,
))
.await
.unwrap();
resp.status()
}
async fn get_reaction(app: &axum::Router, cookie: &str, manga_id: Uuid) -> serde_json::Value {
let resp = app
.clone()
.oneshot(common::get_with_cookie(
&format!("/api/v1/me/reactions/{manga_id}"),
cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
common::body_json(resp).await
}
#[sqlx::test(migrations = "./migrations")]
async fn put_creates_and_reads_back(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
assert_eq!(put_reaction(&h.app, &cookie, manga_id, "like").await, StatusCode::OK);
let body = get_reaction(&h.app, &cookie, manga_id).await;
assert_eq!(body["reaction"], "like");
assert_eq!(body["manga_id"], manga_id.to_string());
}
#[sqlx::test(migrations = "./migrations")]
async fn put_toggles_like_to_dislike(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let _ = put_reaction(&h.app, &cookie, manga_id, "like").await;
assert_eq!(put_reaction(&h.app, &cookie, manga_id, "dislike").await, StatusCode::OK);
assert_eq!(get_reaction(&h.app, &cookie, manga_id).await["reaction"], "dislike");
}
#[sqlx::test(migrations = "./migrations")]
async fn delete_clears_the_reaction(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let _ = put_reaction(&h.app, &cookie, manga_id, "like").await;
let resp = h
.app
.clone()
.oneshot(common::delete_with_cookie(
&format!("/api/v1/mangas/{manga_id}/reaction"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::NO_CONTENT);
assert_eq!(get_reaction(&h.app, &cookie, manga_id).await["reaction"], serde_json::Value::Null);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_unset_returns_null(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
assert_eq!(get_reaction(&h.app, &cookie, manga_id).await["reaction"], serde_json::Value::Null);
}
#[sqlx::test(migrations = "./migrations")]
async fn reactions_are_per_user(pool: PgPool) {
let h = common::harness(pool);
let (_, a) = common::register_user(&h.app).await;
let (_, b) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &a, "Berserk").await;
let _ = put_reaction(&h.app, &a, manga_id, "like").await;
// B sees no reaction of their own.
assert_eq!(get_reaction(&h.app, &b, manga_id).await["reaction"], serde_json::Value::Null);
}
#[sqlx::test(migrations = "./migrations")]
async fn put_unknown_manga_is_404(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
assert_eq!(
put_reaction(&h.app, &cookie, Uuid::new_v4(), "like").await,
StatusCode::NOT_FOUND
);
}
#[sqlx::test(migrations = "./migrations")]
async fn put_invalid_reaction_is_422(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
assert_eq!(
put_reaction(&h.app, &cookie, manga_id, "meh").await,
StatusCode::UNPROCESSABLE_ENTITY
);
}
#[sqlx::test(migrations = "./migrations")]
async fn get_requires_authentication(pool: PgPool) {
let h = common::harness(pool);
let resp = h
.app
.clone()
.oneshot(common::get(&format!("/api/v1/me/reactions/{}", Uuid::new_v4())))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}

View File

@@ -0,0 +1,182 @@
mod common;
use axum::http::StatusCode;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
async fn attach_tag(app: &axum::Router, cookie: &str, manga_id: Uuid, name: &str) {
let resp = app
.clone()
.oneshot(common::post_json_with_cookie(
&format!("/api/v1/mangas/{manga_id}/tags"),
json!({ "name": name }),
cookie,
))
.await
.unwrap();
assert!(resp.status().is_success(), "attach_tag: {}", resp.status());
}
async fn set_reaction(app: &axum::Router, cookie: &str, manga_id: Uuid, reaction: &str) {
let resp = app
.clone()
.oneshot(common::put_json_with_cookie(
&format!("/api/v1/mangas/{manga_id}/reaction"),
json!({ "reaction": reaction }),
cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
async fn bookmark(app: &axum::Router, cookie: &str, manga_id: Uuid) {
let resp = app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/bookmarks",
json!({ "manga_id": manga_id.to_string() }),
cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
}
async fn mark_read(app: &axum::Router, cookie: &str, manga_id: Uuid) {
let resp = app
.clone()
.oneshot(common::put_json_with_cookie(
"/api/v1/me/read-progress",
json!({ "manga_id": manga_id.to_string(), "page": 1 }),
cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
}
/// Recommended manga titles, in ranked order.
async fn recommend(app: &axum::Router, cookie: &str) -> Vec<String> {
let resp = app
.clone()
.oneshot(common::get_with_cookie("/api/v1/me/recommendations", cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
body["items"]
.as_array()
.unwrap()
.iter()
.map(|m| m["title"].as_str().unwrap().to_string())
.collect()
}
#[sqlx::test(migrations = "./migrations")]
async fn likes_drive_recommendations(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let liked = common::seed_manga_via_api(&h.app, &cookie, "Liked").await;
let similar = common::seed_manga_via_api(&h.app, &cookie, "Similar").await;
let unrelated = common::seed_manga_via_api(&h.app, &cookie, "Unrelated").await;
attach_tag(&h.app, &cookie, liked, "action").await;
attach_tag(&h.app, &cookie, similar, "action").await;
attach_tag(&h.app, &cookie, unrelated, "sports").await;
set_reaction(&h.app, &cookie, liked, "like").await;
let recs = recommend(&h.app, &cookie).await;
assert!(recs.contains(&"Similar".to_string()), "recs: {recs:?}");
assert!(!recs.contains(&"Unrelated".to_string()), "recs: {recs:?}");
// The liked manga itself is not recommended back.
assert!(!recs.contains(&"Liked".to_string()), "recs: {recs:?}");
}
#[sqlx::test(migrations = "./migrations")]
async fn dislike_downranks_shared_tags(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let liked = common::seed_manga_via_api(&h.app, &cookie, "Liked").await;
let disliked = common::seed_manga_via_api(&h.app, &cookie, "Disliked").await;
let good = common::seed_manga_via_api(&h.app, &cookie, "Good").await;
let bad = common::seed_manga_via_api(&h.app, &cookie, "Bad").await;
attach_tag(&h.app, &cookie, liked, "action").await;
attach_tag(&h.app, &cookie, good, "action").await; // shares the liked tag
attach_tag(&h.app, &cookie, disliked, "gore").await;
attach_tag(&h.app, &cookie, bad, "gore").await; // shares the disliked tag
set_reaction(&h.app, &cookie, liked, "like").await;
set_reaction(&h.app, &cookie, disliked, "dislike").await;
let recs = recommend(&h.app, &cookie).await;
assert!(recs.contains(&"Good".to_string()), "recs: {recs:?}");
// Net-negative (disliked-tag) candidate is dropped.
assert!(!recs.contains(&"Bad".to_string()), "recs: {recs:?}");
}
#[sqlx::test(migrations = "./migrations")]
async fn bookmark_counts_as_half_a_like(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let liked = common::seed_manga_via_api(&h.app, &cookie, "Liked").await;
let booked = common::seed_manga_via_api(&h.app, &cookie, "Booked").await;
let from_like = common::seed_manga_via_api(&h.app, &cookie, "FromLike").await;
let from_bookmark = common::seed_manga_via_api(&h.app, &cookie, "FromBookmark").await;
attach_tag(&h.app, &cookie, liked, "tliked").await;
attach_tag(&h.app, &cookie, from_like, "tliked").await; // affinity 1.0
attach_tag(&h.app, &cookie, booked, "tbooked").await;
attach_tag(&h.app, &cookie, from_bookmark, "tbooked").await; // affinity 0.5
set_reaction(&h.app, &cookie, liked, "like").await;
bookmark(&h.app, &cookie, booked).await;
let recs = recommend(&h.app, &cookie).await;
// Both recommended, but the like-derived one outranks the bookmark-derived.
let i_like = recs.iter().position(|t| t == "FromLike");
let i_book = recs.iter().position(|t| t == "FromBookmark");
assert!(i_like.is_some() && i_book.is_some(), "recs: {recs:?}");
assert!(i_like < i_book, "like should outrank bookmark; recs: {recs:?}");
}
#[sqlx::test(migrations = "./migrations")]
async fn excludes_already_seen(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let liked = common::seed_manga_via_api(&h.app, &cookie, "Liked").await;
let fresh = common::seed_manga_via_api(&h.app, &cookie, "Fresh").await;
let already_read = common::seed_manga_via_api(&h.app, &cookie, "AlreadyRead").await;
for m in [liked, fresh, already_read] {
attach_tag(&h.app, &cookie, m, "action").await;
}
set_reaction(&h.app, &cookie, liked, "like").await;
mark_read(&h.app, &cookie, already_read).await;
let recs = recommend(&h.app, &cookie).await;
assert!(recs.contains(&"Fresh".to_string()), "recs: {recs:?}");
assert!(!recs.contains(&"AlreadyRead".to_string()), "recs: {recs:?}");
}
#[sqlx::test(migrations = "./migrations")]
async fn empty_without_signals(pool: PgPool) {
let h = common::harness(pool);
let (_, cookie) = common::register_user(&h.app).await;
let m = common::seed_manga_via_api(&h.app, &cookie, "Whatever").await;
attach_tag(&h.app, &cookie, m, "action").await;
assert!(recommend(&h.app, &cookie).await.is_empty());
}
#[sqlx::test(migrations = "./migrations")]
async fn requires_authentication(pool: PgPool) {
let h = common::harness(pool);
let resp = h
.app
.clone()
.oneshot(common::get("/api/v1/me/recommendations"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}

View File

@@ -279,7 +279,7 @@ async fn tag_autocomplete_returns_matches_ordered_by_similarity(pool: PgPool) {
.iter()
.map(|t| t["name"].as_str().unwrap())
.collect();
assert!(names.iter().any(|n| *n == "Mystery"));
assert!(names.iter().any(|n| *n == "Murder Mystery"));
assert!(!names.iter().any(|n| *n == "Comedy"));
assert!(names.contains(&"Mystery"));
assert!(names.contains(&"Murder Mystery"));
assert!(!names.contains(&"Comedy"));
}

View File

@@ -172,6 +172,13 @@ async fn files_endpoint_streams_in_multiple_frames(pool: PgPool) {
resp.headers().get("x-content-type-options").unwrap(),
"nosniff"
);
// Blobs are content-addressed by unguessable, immutable keys, so they're
// safe to cache forever — this is what makes the reader's page + next-
// chapter preloading actually hit cache instead of re-downloading.
assert_eq!(
resp.headers().get(header::CACHE_CONTROL).unwrap(),
"public, max-age=31536000, immutable"
);
let mut body = resp.into_body();
let mut frames = 0usize;
@@ -328,6 +335,42 @@ async fn create_chapter_rejects_when_no_pages_with_422(pool: PgPool) {
assert!(body["error"]["details"]["page"].is_string());
}
#[sqlx::test(migrations = "./migrations")]
async fn create_chapter_rejects_over_page_cap_with_413(pool: PgPool) {
// Cap of 2 pages: a 3-page upload is refused once the third `page` part
// arrives, before any chapter row is written.
let h = common::harness_with_page_cap(pool.clone(), 2);
let (_, cookie) = common::register_user(&h.app).await;
let manga_id = common::seed_manga_via_api(&h.app, &cookie, "Berserk").await;
let resp = h
.app
.clone()
.oneshot(common::post_multipart_with_cookie(
&format!("/api/v1/mangas/{manga_id}/chapters"),
MultipartBuilder::new()
.add_json("metadata", json!({ "number": 1 }))
.add_file("page", "1.png", "image/png", &common::fake_png_bytes())
.add_file("page", "2.png", "image/png", &common::fake_png_bytes())
.add_file("page", "3.png", "image/png", &common::fake_png_bytes()),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::PAYLOAD_TOO_LARGE);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "payload_too_large");
// Nothing persisted — the cap trips before the chapter transaction.
let (chapter_count,): (i64,) =
sqlx::query_as("SELECT count(*) FROM chapters WHERE manga_id = $1")
.bind(manga_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(chapter_count, 0, "over-cap upload must not create a chapter");
}
#[sqlx::test(migrations = "./migrations")]
async fn create_chapter_rejects_renamed_non_image_page(pool: PgPool) {
let h = common::harness(pool);

View File

@@ -79,6 +79,7 @@ fn harness_with_auth_config(
// exercise without producing tens of MBs of bytes.
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter: 2000,
},
auth_limiter,
// Default harness has no crawler daemon wired up; admin resync
@@ -147,6 +148,27 @@ pub fn harness_with_auth_rate_limit(
harness_with_auth_config(pool, storage, storage_dir, auth)
}
/// Like [`harness_with_auth_rate_limit`] but also sets whether the backend
/// trusts `X-Forwarded-For` (`AUTH_TRUSTED_PROXY`). Used to prove the per-IP
/// rate-limit gate: with `trusted_proxy` on, distinct XFF hops get independent
/// buckets; with it off, XFF is ignored and everything shares the global bucket.
pub fn harness_with_auth_rate_limit_proxy(
pool: PgPool,
per_sec: u32,
burst: u32,
trusted_proxy: bool,
) -> Harness {
let storage_dir = tempfile::tempdir().expect("tempdir");
let storage = Arc::new(LocalStorage::new(storage_dir.path()));
let auth = AuthConfig {
cookie_secure: false,
trusted_proxy,
rate_limit: mangalord::auth::rate_limit::RateLimitConfig { per_sec, burst },
..AuthConfig::default()
};
harness_with_auth_config(pool, storage, storage_dir, auth)
}
/// Like [`harness`] but slots a caller-supplied [`ResyncService`] stub
/// into `AppState.resync`. Used by the admin resync tests so the
/// endpoint path is exercised without standing up a real Chromium.
@@ -170,6 +192,7 @@ pub fn harness_with_resync(
upload: UploadConfig {
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter: 2000,
},
auth_limiter,
runtime,
@@ -203,6 +226,7 @@ pub fn harness_with_analysis(pool: PgPool) -> Harness {
upload: UploadConfig {
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter: 2000,
},
auth_limiter,
runtime: Arc::new(RuntimeControls::new(true)),
@@ -218,6 +242,39 @@ pub fn harness_with_analysis(pool: PgPool) -> Harness {
}
}
/// Like [`harness`] but with a low `max_pages_per_chapter` so the chapter
/// upload page-count cap is cheap to exercise.
pub fn harness_with_page_cap(pool: PgPool, max_pages_per_chapter: usize) -> Harness {
let storage_dir = tempfile::tempdir().expect("tempdir");
let storage = Arc::new(LocalStorage::new(storage_dir.path()));
let auth = AuthConfig {
cookie_secure: false,
..AuthConfig::default()
};
let auth_limiter = Arc::new(AuthRateLimiter::new(auth.rate_limit));
let state = AppState {
db: pool,
storage,
auth,
upload: UploadConfig {
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter,
},
auth_limiter,
runtime: Arc::new(RuntimeControls::new(false)),
reloader: None,
crawler_base: CrawlerConfig::default(),
analysis_base: AnalysisConfig::default(),
admin_allowed_origins: Arc::new(vec![TEST_ORIGIN.to_string()]),
analysis_events: Arc::new(mangalord::analysis::events::AnalysisEvents::new()),
};
Harness {
app: router(state),
_storage_dir: storage_dir,
}
}
/// A [`DaemonReloader`] stub that records the configs it was asked to apply
/// and flips the shared analysis gate, without spawning any real daemon. Lets
/// settings tests assert that a `PUT` triggers a reload with the converted
@@ -265,6 +322,7 @@ pub fn harness_with_settings_reloader(pool: PgPool) -> (Harness, Arc<StubReloade
upload: UploadConfig {
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter: 2000,
},
auth_limiter,
runtime,
@@ -300,6 +358,7 @@ pub fn harness_with_admin_origins(pool: PgPool, origins: Vec<String>) -> Harness
upload: UploadConfig {
max_request_bytes: 4 * 1024 * 1024,
max_file_bytes: 256 * 1024,
max_pages_per_chapter: 2000,
},
auth_limiter,
runtime: Arc::new(RuntimeControls::new(false)),
@@ -376,6 +435,12 @@ impl Storage for FailingStorage {
async fn size(&self, key: &str) -> Result<u64, StorageError> {
self.inner.size(key).await
}
// Delegate straight to the inner filesystem rename — the fault
// injection counts `put`/`put_stream` only, so promoting a staged page
// to its final key never spuriously trips the injected failure.
async fn rename(&self, from: &str, to: &str) -> Result<(), StorageError> {
self.inner.rename(from, to).await
}
}
pub async fn body_json(response: axum::response::Response) -> serde_json::Value {
@@ -431,6 +496,20 @@ pub fn post_json_with_cookie(
.unwrap()
}
/// Like [`post_json_with_cookie`] but sends a raw (possibly malformed) body,
/// for exercising body-parse error handling. Content-Type stays JSON and the
/// CSRF Origin is attached so the request reaches the handler.
pub fn post_raw_with_cookie(uri: &str, body: &str, cookie: &str) -> Request<Body> {
Request::builder()
.method("POST")
.uri(uri)
.header(header::CONTENT_TYPE, "application/json")
.header(header::COOKIE, cookie)
.header(header::ORIGIN, TEST_ORIGIN)
.body(Body::from(body.to_string()))
.unwrap()
}
/// Same as [`post_json_with_cookie`] but also attaches `Origin` (and
/// optionally `Referer`) headers. Used by the admin CSRF tests to drive
/// the cross-origin reject + allowed-origin accept paths.

View File

@@ -62,6 +62,43 @@ async fn headless_browser_can_navigate_and_read_title() {
handle.close().await.expect("close cleanly");
}
/// Smoke-test the opt-in SSRF navigation guard (`CRAWLER_SSRF_INTERCEPT`).
/// With interception ON, a normal (allowed) navigation must still complete —
/// i.e. enabling CDP `Fetch` and installing the request handler must NOT wedge
/// page loads. This is the regression the interceptor's fragility risks; it
/// can only be exercised with a real Chromium, hence `#[ignore]`.
///
/// (The private-target *blocking* path is covered by the pure-logic unit tests
/// in `crawler::intercept` — `verdict` / `is_blocked` — which don't need a
/// browser.)
#[tokio::test]
#[ignore = "downloads Chromium; run with --ignored"]
async fn ssrf_interception_does_not_wedge_allowed_navigation() {
use mangalord::crawler::intercept;
const PAGE: &str =
"data:text/html,<html><head><title>Guarded%20OK</title></head><body></body></html>";
intercept::set_enabled(true);
let handle = browser::launch(LaunchOptions::headless())
.await
.expect("launch headless chromium");
// Route through the guarded opener (blank page -> Fetch.enable -> handler
// -> goto). If the handler failed to continue the navigation, this would
// hang until the test harness times out.
let page = intercept::open_page(handle.browser(), PAGE)
.await
.expect("guarded open_page");
page.wait_for_navigation().await.expect("wait for navigation");
let title = page.get_title().await.expect("get title");
assert_eq!(title.as_deref(), Some("Guarded OK"));
handle.close().await.expect("close cleanly");
intercept::set_enabled(false);
}
/// Live end-to-end: navigate to a real page, get the rendered HTML, and
/// parse it with `scraper`. ipify.org renders the visitor's public IP
/// into the page DOM, so a successful run proves browser → render →

View File

@@ -104,6 +104,20 @@ impl ChapterDispatcher for FailingDispatcher {
}
}
/// Always reports the browser unavailable — models a Chromium outage where
/// `acquire()` fails. The job must be returned to `pending` WITHOUT burning an
/// attempt, so an outage doesn't chew the backlog to `dead`.
struct BrowserUnavailableDispatcher {
seen: AtomicUsize,
}
#[async_trait::async_trait]
impl ChapterDispatcher for BrowserUnavailableDispatcher {
async fn dispatch(&self, _payload: JobPayload) -> anyhow::Result<SyncOutcome> {
self.seen.fetch_add(1, Ordering::AcqRel);
Ok(SyncOutcome::BrowserUnavailable)
}
}
/// Never completes — used to verify the worker's outer dispatch timeout.
struct HangingDispatcher {
seen: AtomicUsize,
@@ -204,6 +218,58 @@ async fn shutdown_mid_dispatch_releases_lease_without_burning_attempt(pool: PgPo
);
}
#[sqlx::test(migrations = "./migrations")]
async fn browser_unavailable_releases_lease_without_burning_attempt(pool: PgPool) {
// During a browser outage the dispatcher reports BrowserUnavailable. The
// worker must return the job to `pending` with the attempt refunded
// (attempts stays 0) instead of ack-failing it toward `dead`, so the whole
// pending backlog survives the outage.
enqueue_chapter_job(&pool).await;
let dispatcher = Arc::new(BrowserUnavailableDispatcher {
seen: AtomicUsize::new(0),
});
let session_expired = Arc::new(std::sync::atomic::AtomicBool::new(false));
let cancel = CancellationToken::new();
let handle = daemon::spawn(
pool.clone(),
cancel.clone(),
make_cfg(None, dispatcher.clone(), session_expired, 1),
);
// Wait until the dispatcher has been invoked at least once (the job was
// leased and deferred).
let mut dispatched = false;
for _ in 0..40 {
if dispatcher.seen.load(Ordering::Acquire) >= 1 {
dispatched = true;
break;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
assert!(dispatched, "dispatcher must have been invoked");
handle.shutdown().await;
assert_eq!(
count_state(&pool, "dead").await,
0,
"browser outage must not dead-letter the job"
);
assert_eq!(
count_state(&pool, "pending").await,
1,
"job returns to pending after a browser-unavailable outcome"
);
let attempts: i32 = sqlx::query_scalar("SELECT attempts FROM crawler_jobs")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
attempts, 0,
"browser-unavailable must refund the lease attempt (no burn)"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn workers_drain_jobs_through_dispatcher(pool: PgPool) {
enqueue_chapter_job(&pool).await;

View File

@@ -718,42 +718,49 @@ async fn release_returns_to_pending_and_undoes_attempt_increment(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn reap_done_deletes_old_rows_keeps_fresh(pool: PgPool) {
// Two done rows: one old (updated_at 10 days ago), one fresh.
let old_id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
let fresh_id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
async fn reap_terminal_deletes_old_done_and_dead_keeps_fresh_and_active(pool: PgPool) {
// Helper: enqueue a fresh pending job and return its id.
async fn enqueue_one(pool: &PgPool) -> Uuid {
match jobs::enqueue(pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
}
}
let old_done = enqueue_one(&pool).await;
let old_dead = enqueue_one(&pool).await;
let fresh_done = enqueue_one(&pool).await;
let fresh_dead = enqueue_one(&pool).await;
let old_pending = enqueue_one(&pool).await;
// Old terminal rows (10 days) in both terminal states — both must reap.
sqlx::query("UPDATE crawler_jobs SET state='done', updated_at = now() - interval '10 days' WHERE id = $1")
.bind(old_id)
.execute(&pool)
.await
.unwrap();
.bind(old_done).execute(&pool).await.unwrap();
sqlx::query("UPDATE crawler_jobs SET state='dead', updated_at = now() - interval '10 days' WHERE id = $1")
.bind(old_dead).execute(&pool).await.unwrap();
// Fresh terminal rows — inside the retention window, kept.
sqlx::query("UPDATE crawler_jobs SET state='done' WHERE id = $1")
.bind(fresh_id)
.execute(&pool)
.await
.unwrap();
.bind(fresh_done).execute(&pool).await.unwrap();
sqlx::query("UPDATE crawler_jobs SET state='dead' WHERE id = $1")
.bind(fresh_dead).execute(&pool).await.unwrap();
// Old but still active (pending) — never reaped regardless of age.
sqlx::query("UPDATE crawler_jobs SET updated_at = now() - interval '10 days' WHERE id = $1")
.bind(old_pending).execute(&pool).await.unwrap();
let deleted = jobs::reap_done(&pool, 7).await.unwrap();
assert_eq!(deleted, 1);
let deleted = jobs::reap_terminal(&pool, 7).await.unwrap();
assert_eq!(deleted, 2, "both old done and old dead rows are reaped");
let remaining: Vec<Uuid> = sqlx::query_scalar("SELECT id FROM crawler_jobs ORDER BY id")
let mut remaining: Vec<Uuid> = sqlx::query_scalar("SELECT id FROM crawler_jobs")
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(remaining, vec![fresh_id], "only fresh row remains");
remaining.sort();
let mut expected = vec![fresh_done, fresh_dead, old_pending];
expected.sort();
assert_eq!(remaining, expected, "fresh terminal + active-pending rows survive");
}
#[sqlx::test(migrations = "./migrations")]
@@ -840,7 +847,7 @@ async fn lease_ties_on_scheduled_at_break_by_created_at(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn reap_done_zero_is_a_no_op(pool: PgPool) {
async fn reap_terminal_zero_is_a_no_op(pool: PgPool) {
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
@@ -854,7 +861,7 @@ async fn reap_done_zero_is_a_no_op(pool: PgPool) {
.await
.unwrap();
let deleted = jobs::reap_done(&pool, 0).await.unwrap();
let deleted = jobs::reap_terminal(&pool, 0).await.unwrap();
assert_eq!(deleted, 0);
assert_eq!(job_count(&pool).await, 1);
}

View File

@@ -674,6 +674,110 @@ async fn arbitrary_genres_from_source_get_inserted(pool: PgPool) {
assert_eq!(webtoons_count.0, 1, "case-insensitive lookup reuses the existing row");
}
#[sqlx::test(migrations = "./migrations")]
async fn genre_dedup_survives_a_manga_linked_to_two_variants(pool: PgPool) {
// Regression for a migration-0038 collision: a manga linked to TWO
// non-canonical case-variants of one genre. A repoint-UPDATE would set both
// rows to the canonical id in one statement -> manga_genres PK violation ->
// migration rollback -> boot failure. #[sqlx::test] migrates a clean DB, so
// recreate the dirty pre-index state and replay 0038's healing statements;
// they must complete and leave exactly the one canonical link. (Mirrors the
// healing SQL in migrations/0038_genres_name_lower_unique.sql.)
let manga = Uuid::new_v4();
sqlx::query("INSERT INTO mangas (id, title) VALUES ($1, 'T')")
.bind(manga)
.execute(&pool)
.await
.unwrap();
// Drop the live guard so we can plant case-variant genres.
sqlx::query("DROP INDEX genres_name_lower_uniq").execute(&pool).await.unwrap();
let keep = Uuid::parse_str("00000000-0000-0000-0000-000000000001").unwrap();
let dup2 = Uuid::parse_str("00000000-0000-0000-0000-000000000002").unwrap();
let dup3 = Uuid::parse_str("00000000-0000-0000-0000-000000000003").unwrap();
for (id, name) in [(keep, "Zzz"), (dup2, "zzz"), (dup3, "ZZZ")] {
sqlx::query("INSERT INTO genres (id, name) VALUES ($1, $2)")
.bind(id)
.bind(name)
.execute(&pool)
.await
.unwrap();
}
// The manga links to the two NON-canonical variants, not the canonical one.
for gid in [dup2, dup3] {
sqlx::query("INSERT INTO manga_genres (manga_id, genre_id) VALUES ($1, $2)")
.bind(manga)
.bind(gid)
.execute(&pool)
.await
.unwrap();
}
// Step 1 — collision-proof canonical-link backfill (the crux of the fix).
sqlx::query(
"INSERT INTO manga_genres (manga_id, genre_id) \
SELECT DISTINCT mg.manga_id, k.keep_id \
FROM manga_genres mg JOIN genres g ON mg.genre_id = g.id \
JOIN (SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id \
FROM genres GROUP BY lower(name)) k ON lower(g.name) = k.lname \
WHERE g.id <> k.keep_id \
ON CONFLICT (manga_id, genre_id) DO NOTHING",
)
.execute(&pool)
.await
.expect("collision-proof INSERT must not raise a manga_genres PK violation");
// Step 2 — drop the non-canonical links.
sqlx::query(
"DELETE FROM manga_genres mg USING genres g, \
(SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id \
FROM genres GROUP BY lower(name)) k \
WHERE mg.genre_id = g.id AND lower(g.name) = k.lname AND g.id <> k.keep_id",
)
.execute(&pool)
.await
.unwrap();
// Step 3 — remove orphaned duplicate genres.
sqlx::query(
"DELETE FROM genres g USING \
(SELECT lower(name) AS lname, (array_agg(id ORDER BY id))[1] AS keep_id \
FROM genres GROUP BY lower(name)) k \
WHERE lower(g.name) = k.lname AND g.id <> k.keep_id",
)
.execute(&pool)
.await
.unwrap();
// Exactly the canonical link remains, and the guard re-creates cleanly.
let links: Vec<Uuid> =
sqlx::query_scalar("SELECT genre_id FROM manga_genres WHERE manga_id = $1")
.bind(manga)
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(links, vec![keep], "manga keeps exactly the canonical genre link");
sqlx::query("CREATE UNIQUE INDEX genres_name_lower_uniq ON genres (lower(name))")
.execute(&pool)
.await
.expect("no residual case-variant duplicates remain");
}
#[sqlx::test(migrations = "./migrations")]
async fn genres_reject_case_variant_duplicates_at_the_db(pool: PgPool) {
// The sequential pre-check in sync_genres dedups the common case, but two
// concurrent inserts could each miss it. The lower(name) unique index (0038)
// is the race backstop: a case variant of an existing genre must be rejected
// by the DB itself. "Action" is seeded by 0009.
let dup = sqlx::query("INSERT INTO genres (name) VALUES ('action')")
.execute(&pool)
.await;
let err = dup.expect_err("a case-variant of a seeded genre must violate the unique index");
let msg = err.to_string();
assert!(
msg.contains("genres_name_lower_uniq") || msg.contains("unique") || msg.contains("duplicate"),
"expected a unique-violation, got: {msg}"
);
}
/// User-attached tags (rows with non-NULL `added_by` in `manga_tags`)
/// must survive a crawler upsert. The crawler owns source-attached tags
/// (added_by IS NULL); user attachments are owned by the user who made
@@ -1175,7 +1279,7 @@ async fn list_for_manga_returns_source_order_reversed(pool: PgPool) {
}
#[sqlx::test(migrations = "./migrations")]
async fn list_for_manga_places_null_source_index_last(pool: PgPool) {
async fn list_for_manga_interleaves_null_source_index_by_number(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
@@ -1184,23 +1288,24 @@ async fn list_for_manga_places_null_source_index_last(pool: PgPool) {
.await
.unwrap();
// Crawled chapters get source_index 0 and 1; the upload path leaves
// it NULL. NULLS LAST plus the (number, created_at) tail means the
// upload sits after both crawled rows even though its number is in
// the middle.
// Crawled chapters, newest-first in the source DOM (so chapter 3 is at
// source_index 0 and chapter 1 at source_index 1). Reversed for display that
// is [Ch.1, Ch.3]. The uploaded chapter 2 must slot BETWEEN them by number,
// not fall to the end — that was the bug (NULLS LAST dumped it last
// regardless of its number).
let crawled = vec![
SourceChapterRef {
source_chapter_key: "a".into(),
number: 1,
title: Some("Ch.1".into()),
url: "https://x.example/foo/a".into(),
},
SourceChapterRef {
source_chapter_key: "b".into(),
number: 3,
title: Some("Ch.3".into()),
url: "https://x.example/foo/b".into(),
},
SourceChapterRef {
source_chapter_key: "a".into(),
number: 1,
title: Some("Ch.1".into()),
url: "https://x.example/foo/a".into(),
},
];
crawler::sync_manga_chapters(&pool, "target", up.manga_id, &crawled)
.await
@@ -1220,11 +1325,11 @@ async fn list_for_manga_places_null_source_index_last(pool: PgPool) {
assert_eq!(
titles,
vec![
"Ch.3".to_string(),
"Ch.1".to_string(),
"User upload Ch.2".to_string(),
"Ch.3".to_string(),
],
"crawled rows ordered by reversed source_index; user upload \
(NULL source_index) falls through to the end",
"uploaded chapter (NULL source_index) interleaves by number between the \
crawled rows instead of falling to the end",
);
}

View File

@@ -0,0 +1,52 @@
mod common;
use chrono::{Duration, Utc};
use sqlx::PgPool;
use mangalord::repo;
#[sqlx::test(migrations = "./migrations")]
async fn delete_expired_removes_only_lapsed_sessions(pool: PgPool) {
let user = repo::user::create(&pool, "reaper-subject", "x").await.unwrap();
// One session already lapsed, one still valid.
let expired = repo::session::create(
&pool,
user.id,
b"expired-token-hash-00000000000000",
Utc::now() - Duration::hours(1),
)
.await
.unwrap();
let active = repo::session::create(
&pool,
user.id,
b"active-token-hash-000000000000000",
Utc::now() + Duration::hours(1),
)
.await
.unwrap();
let removed = repo::session::delete_expired(&pool).await.unwrap();
assert_eq!(removed, 1, "exactly the one lapsed session is reaped");
// The active session is untouched and still resolvable; the expired one is
// gone from the table entirely (find_active already ignored it, but now the
// row is reclaimed too).
assert!(
repo::session::find_active(&pool, b"active-token-hash-000000000000000")
.await
.unwrap()
.is_some(),
"active session survives the sweep"
);
let remaining: i64 = sqlx::query_scalar("SELECT count(*) FROM sessions")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(remaining, 1);
// Sweeping again is a no-op now that nothing is expired.
assert_eq!(repo::session::delete_expired(&pool).await.unwrap(), 0);
let _ = (expired, active);
}

View File

@@ -23,6 +23,12 @@ services:
# takes the whole stack offline. Match the policy already set on
# tor / docker-socket-proxy / vision-manager.
restart: unless-stopped
# Bound Postgres so a runaway query/backlog can't consume all host RAM.
# Tune to your host via .env (should comfortably exceed shared_buffers +
# work_mem * max_connections).
mem_limit: ${POSTGRES_MEM_LIMIT:-1g}
security_opt:
- no-new-privileges:true
tor:
# SOCKS5 proxy for the crawler, plus a control port so the backend
@@ -71,12 +77,19 @@ services:
BIND_ADDRESS: 0.0.0.0:8080
STORAGE_DIR: /var/lib/mangalord/storage
RUST_LOG: ${RUST_LOG:-info,mangalord=debug}
# Postgres connection-pool sizing — see .env.example for context.
DB_MAX_CONNECTIONS: ${DB_MAX_CONNECTIONS:-20}
DB_ACQUIRE_TIMEOUT_SECS: ${DB_ACQUIRE_TIMEOUT_SECS:-10}
# Auth / cookies — see .env.example for context.
COOKIE_SECURE: ${COOKIE_SECURE:-true}
COOKIE_DOMAIN: ${COOKIE_DOMAIN:-}
SESSION_TTL_DAYS: ${SESSION_TTL_DAYS:-30}
AUTH_RATE_PER_SEC: ${AUTH_RATE_PER_SEC:-5}
AUTH_RATE_BURST: ${AUTH_RATE_BURST:-10}
# The SvelteKit container is the single trusted hop in front of the
# backend and stamps the real client IP, so per-IP rate limiting is on
# by default here (unlike the backend's safe-by-default `false`).
AUTH_TRUSTED_PROXY: ${AUTH_TRUSTED_PROXY:-true}
# CORS — same-origin by default; populate when serving the API on
# a different host than the frontend.
CORS_ALLOWED_ORIGINS: ${CORS_ALLOWED_ORIGINS:-}
@@ -96,6 +109,7 @@ services:
# Upload limits.
MAX_REQUEST_BYTES: ${MAX_REQUEST_BYTES:-209715200}
MAX_FILE_BYTES: ${MAX_FILE_BYTES:-20971520}
MAX_PAGES_PER_CHAPTER: ${MAX_PAGES_PER_CHAPTER:-2000}
# Crawler boot seeds (first-boot only; switch to dashboard-editable
# once the app_settings row exists).
CRAWLER_START_URL: ${CRAWLER_START_URL:-}
@@ -104,6 +118,7 @@ services:
CRAWLER_JOB_TIMEOUT_SECS: ${CRAWLER_JOB_TIMEOUT_SECS:-600}
CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES: ${CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES:-10}
CRAWLER_BROWSER_RESTART_THRESHOLD: ${CRAWLER_BROWSER_RESTART_THRESHOLD:-3}
CRAWLER_SSRF_INTERCEPT: ${CRAWLER_SSRF_INTERCEPT:-true}
# Crawler daemon schedule + retention. CRAWLER_DAEMON=false keeps
# the in-process scheduler off; the dashboard force-resync still works.
CRAWLER_DAEMON: ${CRAWLER_DAEMON:-true}
@@ -124,6 +139,7 @@ services:
CRAWLER_DOWNLOAD_ALLOWLIST: ${CRAWLER_DOWNLOAD_ALLOWLIST:-}
CRAWLER_ALLOW_ANY_HOST: ${CRAWLER_ALLOW_ANY_HOST:-false}
CRAWLER_MAX_IMAGE_BYTES: ${CRAWLER_MAX_IMAGE_BYTES:-33554432}
CRAWLER_MAX_IMAGES_PER_CHAPTER: ${CRAWLER_MAX_IMAGES_PER_CHAPTER:-2000}
# System-chromium override for the crawler. Leave blank to use the
# bundled fetcher; set to e.g. /usr/bin/chromium-headless-shell on
# arm64 deployments. Pair with `--build-arg INSTALL_CHROMIUM=true`
@@ -145,8 +161,10 @@ services:
CRAWLER_TOR_CONTROL_COOKIE_PATH: ${CRAWLER_TOR_CONTROL_COOKIE_PATH:-}
# Analysis worker — boot seeds + env-only API key.
ANALYSIS_ENABLED: ${ANALYSIS_ENABLED:-false}
# Engine selection (deploy-time): `ocr` (in-process ocrs, default) or
# `vision` (local LLM). Model paths default to the image-baked /models.
# Engine selection (deploy-time): `ocr` (in-process ocrs, default).
# The `vision` (local LLM) backend is temporarily disabled — the worker
# forces OCR regardless, so a `vision` override here is currently inert.
# Model paths default to the image-baked /models.
ANALYSIS_BACKEND: ${ANALYSIS_BACKEND:-ocr}
OCRS_DETECTION_MODEL: ${OCRS_DETECTION_MODEL:-/models/text-detection.rten}
OCRS_RECOGNITION_MODEL: ${OCRS_RECOGNITION_MODEL:-/models/text-recognition.rten}
@@ -165,6 +183,7 @@ services:
ANALYSIS_TALL_ASPECT: ${ANALYSIS_TALL_ASPECT:-1.6}
ANALYSIS_MAX_SLICES: ${ANALYSIS_MAX_SLICES:-16}
ANALYSIS_MAX_IMAGE_BYTES: ${ANALYSIS_MAX_IMAGE_BYTES:-8388608}
ANALYSIS_OCR_MAX_DECODE_PIXELS: ${ANALYSIS_OCR_MAX_DECODE_PIXELS:-100000000}
ANALYSIS_RESPONSE_FORMAT: ${ANALYSIS_RESPONSE_FORMAT:-json_schema}
ANALYSIS_FREQUENCY_PENALTY: ${ANALYSIS_FREQUENCY_PENALTY:-0.3}
ANALYSIS_TEMPERATURE: ${ANALYSIS_TEMPERATURE:-0.0}
@@ -184,6 +203,13 @@ services:
expose:
- "8080"
restart: unless-stopped
# Bound the heaviest consumer: the backend hosts the OCR/analysis worker and
# drives a headless Chromium crawler, either of which can balloon and
# OOM-kill the host (taking Postgres with it) with no limit. Tune to your
# host via .env — this default is a generous ceiling, not a target.
mem_limit: ${BACKEND_MEM_LIMIT:-4g}
security_opt:
- no-new-privileges:true
frontend:
build: ./frontend
@@ -200,8 +226,17 @@ services:
# to 300000 (5 min) in hooks.server.ts; raise/lower via .env.
BACKEND_PROXY_TIMEOUT_MS: ${BACKEND_PROXY_TIMEOUT_MS:-300000}
ports:
- "3000:3000"
# Bind to loopback by default so SvelteKit (plain HTTP; COOKIE_SECURE=true
# expects TLS in front) is reachable ONLY by a host-local reverse proxy,
# not cleartext over the LAN/internet. A host-external TLS terminator can
# set FRONTEND_PUBLISH_ADDR=0.0.0.0 (or a specific interface) in .env.
- "${FRONTEND_PUBLISH_ADDR:-127.0.0.1}:3000:3000"
restart: unless-stopped
# Cap runaway memory (SvelteKit is light, but bound it anyway) and forbid
# privilege escalation. Tune the limit to your host via .env.
mem_limit: ${FRONTEND_MEM_LIMIT:-512m}
security_opt:
- no-new-privileges:true
# ----- Vision autoscaling (profile: ai) -----------------------------------
# Two extra containers that idle-stop the mangalord-vision (llama.cpp)

10
frontend/csp-config.js Normal file
View File

@@ -0,0 +1,10 @@
// Shared CSP constants, kept dependency-free so both svelte.config.js (loaded by
// Node when Vite starts) and the vitest drift-guard test (jsdom env) can import
// it without pulling in adapter-node / esbuild.
// sha256 of the inline theme <script> in src/app.html, base64-encoded, wrapped
// for the CSP `script-src` allowlist. SvelteKit's kit.csp hash mode does not
// cover app.html template scripts, so this is pinned by hand. src/csp-theme-hash.test.ts
// recomputes it from app.html and fails if it drifts.
export const THEME_SCRIPT_HASH =
"'sha256-qj6Oim9siqow/Su+v47sZHJeJci+M/RQwGXn53eSULQ='";

View File

@@ -1,4 +1,4 @@
import { test, expect, type Page } from '@playwright/test';
import { test, expect, type Page } from './fixtures';
// E2E for the admin Analysis section: coverage overview + badges, drill
// manga → chapter → page, the page-detail modal, and the enqueue actions.
@@ -196,6 +196,21 @@ async function mockAdmin(page: Page, cap: Captured) {
})
})
);
// The metrics tab also pulls a per-bucket time series for the trend
// charts. Registered after the aggregate `metrics**` route so it wins
// for the more specific `/metrics/series` path.
await page.route('**/api/v1/admin/analysis/metrics/series**', (r) =>
r.fulfill({
status: 200,
contentType: 'application/json',
body: JSON.stringify({
buckets: [
{ t: '2026-06-12T00:00:00Z', n: 1600, ok: 1580, failed: 20, avg_ms: 2300 },
{ t: '2026-06-13T00:00:00Z', n: 1604, ok: 1579, failed: 25, avg_ms: 2500 }
]
})
})
);
// Default: keep the SSE connection pending (no events) so tests that
// don't care about live updates don't trigger reconnect churn. The
@@ -264,11 +279,12 @@ test.describe('/admin/analysis', () => {
await expect(
page.getByTestId('admin-analysis-detail-status')
).toContainText('Analyzed');
await expect(modal).toContainText('model test-model');
// OCR-only backend: the detail surfaces the extracted OCR lines (with
// their kind) — tags / scene / NSFW belong to the dormant vision
// backend and are intentionally not shown here.
await expect(modal).toContainText('OCR text');
await expect(modal).toContainText('Hello there');
await expect(modal).toContainText('action');
await expect(page.getByTestId('admin-analysis-detail-warnings')).toContainText(
'gore'
);
});
test('live SSE events drive the indicator and activity ticker', async ({
@@ -367,8 +383,9 @@ test.describe('/admin/analysis', () => {
await page.getByTestId('admin-analysis-tab-history').click();
const row = page.getByTestId(`analysis-history-row-${pageDone}`);
await expect(row).toContainText('Berserk · Ch 1 · p1');
await expect(row).toContainText('NSFW');
await expect(row).toContainText('test-model'); // model column
await expect(row).toContainText('2.4s'); // duration column
// (No NSFW flag: the active OCR backend extracts text only.)
await row.click();
await expect(page.getByTestId('admin-analysis-detail')).toBeVisible();
@@ -377,16 +394,22 @@ test.describe('/admin/analysis', () => {
);
});
test('metrics tab shows aggregate timing + by-model', async ({ page }) => {
test('metrics tab shows aggregate timing + trend charts', async ({ page }) => {
const cap: Captured = { reenqueue: null, analyzeCalls: 0 };
await mockAdmin(page, cap);
await page.setViewportSize(DESKTOP);
await page.goto('/admin/analysis');
await page.getByTestId('admin-analysis-tab-metrics').click();
// Aggregate tiles for the selected window.
await expect(page.getByTestId('analysis-metrics-n')).toContainText('3204');
await expect(page.getByTestId('analysis-metrics-avg')).toContainText('2.4s');
await expect(page.getByTestId('analysis-metrics')).toContainText('qwen2-vl-7b');
const metrics = page.getByTestId('analysis-metrics');
await expect(metrics).toContainText('Success');
await expect(metrics).toContainText('99%'); // 3159/3204
await expect(metrics).toContainText('45'); // failed
// The per-bucket trend charts render from the series endpoint.
await expect(page.getByTestId('analysis-metrics-charts')).toBeVisible();
});
test('queue an unanalyzed page from its detail modal', async ({ page }) => {

View File

@@ -1,4 +1,4 @@
import { test, expect, type Page } from '@playwright/test';
import { test, expect, type Page } from './fixtures';
// E2E for the admin Crawler "History" tab: the Live/History toggle, the
// searchable/filterable job log, and inline requeue of a dead job. The

Some files were not shown because too many files have changed in this diff Show More