Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The analysis worker's flat 1s idle sleep meant an empty queue fired a lease query
per worker per second forever (audit). Add idle_backoff (1s..30s, reset on lease),
mirroring the crawler. Unit-tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reap_terminal was one unbatched DELETE with no updated_at index — a full scan and
long-held lock over a large backlog (audit M5). Migration 0040 adds a partial
index on (updated_at) WHERE state IN ('done','dead'); reap_terminal now deletes in
bounded 5k batches. Existing tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lease predicate's `state='running' AND leased_until < now()` arm had no
index, so every lease poll seq-scanned the whole crawler_jobs table (audit H2).
migration 0039 adds a partial index on (leased_until) WHERE state='running'.
Behavior-neutral; 29 lease tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A blob with valid magic but a corrupt body passed upload then 500'd on a ?w=
thumbnail decode (audit). Fall back to serve_original (streams without decoding)
instead. Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Option<Json<T>> collapses a malformed body to None, so a bad reenqueue body
silently ran the default full "All" scope (audit). Parse raw bytes: empty =
default All, malformed = 422. Tested both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
put_cover/delete_cover mutated storage before the DB pointer, so a transient DB
failure could leave the row pointing at a deleted blob. Reorder to
put->set-DB->delete-old and clear-DB->delete-blob, so a failure only orphans a
blob, never dangles the pointer. Happy path covered by the 16 existing cover tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retargeting the modal mid-load could resolve responses out of order, overwriting
the newer target's membership with the older one's (audit FE-2). Route load()
through the tested CancellableLoader so a superseded result is dropped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Arrows/j/k/Home/End bubbled to the reader's window handler while a
jump/settings/action/collections/tags/context overlay was open, paging the
chapter behind it (FE-1); native select/contenteditable weren't exempt (FE-4).
New unit-tested isTypingTarget() + an overlay-open early return fix both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
genres had only case-sensitive name UNIQUE while authors/tags use
UNIQUE (lower(name)); sync_genres could race two case variants into duplicate
rows (audit DB-3). Migration 0038 pre-dedups then adds
UNIQUE (lower(name)); sync_genres upserts ON CONFLICT (lower(name)). Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deterministic per-page storage keys mean a re-crawl overwrites the image but the
pages row survives the upsert, so analysis keyed to page_id (page_analysis/
search_doc, OCR, auto-tags, content-warnings) stayed stale forever and
force=false re-enqueue never re-analyzed it — poisoning search + the derived
manga_content_warnings (audit DB-1). persist_pages now detects conflict-updated
pages via xmax and clears their four derived tables in the same tx (the
content-warnings DELETE fires the mcw_* triggers), returning them to unanalyzed
so they re-derive. Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Harvested source_url (external content) rendered as an admin <a href> without
scheme validation — a javascript:/data: link would execute on click (audit M2).
safeExternalHref() gates to http(s); CrawlerHistoryTable/DeadJobsTable fall back
to plain text otherwise. Tested in safeHref.test.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two integration tests pinning both sides of the gate: trusted_proxy=true gives
each forwarded IP its own bucket; trusted_proxy=false ignores spoofed XFF and
shares the global bucket. Closes audit gap M7. Test-only, no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
constant_time_eq (and the `subtle` dependency) was never wired into any
comparison — the module doc wrongly claimed token comparison used it, but lookup
is an indexed DB equality on the 256-bit SHA-256 hash, so there's no in-process
secret compare to time-attack. Delete the dead fn, its test, and the unused
`subtle` dep; rewrite the doc. No behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sessions table only grew — find_active ignored expired rows but nothing
deleted them, so lapsed sessions accumulated indefinitely.
- repo::session::delete_expired: indexed DELETE (sessions_expires_idx) returning
the reaped count.
- app::build spawns a detached hourly reaper (SESSION_GC_INTERVAL) that calls it,
independent of crawler/analysis config.
Test: delete_expired_removes_only_lapsed_sessions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /files handler served the application/octet-stream fallback with no
Content-Disposition, letting a browser render it inline — a stored-XSS vector if
the blob is crafted HTML/JS. Add Content-Disposition: attachment for that type
(belt to the existing nosniff); known image types stay inline.
Test (new tests/api_files.rs): octet-stream → attachment; png → inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `metadata` JSON part in the manga-create and chapter-upload handlers was
read with `Field::bytes()`, buffering the whole part before any check — a client
could send a multi-hundred-MB metadata blob (up to the 200 MiB request limit) as
one allocation. Image parts were already streamed with a cap; metadata was the
last unbounded read.
Cap it at a shared upload::MAX_METADATA_BYTES (64 KiB) via the read_capped
streaming loop (413 on overflow), and remove the now-unused read_field_bytes
helper so the unbounded path can't return.
Tests: manga + chapter oversized-metadata parts are rejected with 413.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`get` (fs::read → EISDIR) and `get_stream` (File::open succeeds on a Unix
directory, then streams garbage that fails mid-read) both mishandled a key
resolving to a directory. Mirror `size`'s existing guard: `get` maps
IsADirectory to NotFound; `get_stream` checks metadata.is_file() after open.
Test: get_on_directory_is_not_found.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `%` or `_` in a search term silently acted as a LIKE wildcard rather than
matching literally (`50%` matched everything, `a_b` matched `axb`). Not
injection — terms are bound — but a search-correctness bug across every
user-facing ILIKE site.
- repo::escape_like: shared pub(crate) escaper (promoted from page_tag), unit-tested.
- Trigram-entangled sites (manga, tag, author) append a separate escaped param
used only by the ILIKE branch; the trigram/similarity branches keep the raw term.
- Pattern-built sites (admin manga list, admin users, analysis coverage + history,
crawler search incl. JSONB payload title) escape inside format!("%{}%", ..) and
pair each ILIKE with ESCAPE '\'.
Tests: escape_like unit tests + integration tests on public manga search, author
autocomplete, and admin user search proving `_` matches literally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document responses carried no CSP, X-Frame-Options, Referrer-Policy, or
Permissions-Policy — leaving clickjacking and zero script-injection defense
in depth (security audit T4).
- svelte.config.js: enable kit.csp hash mode. SvelteKit hashes its own inline
hydration scripts; the inline theme <script> in app.html isn't part of
%sveltekit.head%, so its sha256 is pinned by hand (csp-config.js) and added
to script-src alongside object-src 'none', base-uri 'self', frame-ancestors
'none'. Styles stay unconstrained (Svelte emits dynamic inline style= attrs).
- hooks.server.ts: applySecurityHeaders() sets X-Frame-Options: DENY,
Referrer-Policy: strict-origin-when-cross-origin, X-Content-Type-Options:
nosniff, and a Permissions-Policy locking down unused features, only when the
response hasn't already set them.
- src/csp-theme-hash.test.ts: drift guard against editing the theme script
without updating THEME_SCRIPT_HASH.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
listAllChapters used `page.total ?? all.length` as the loop bound. The
/chapters endpoint always returns `total: null`, so a full 200-row first
page broke the loop immediately — the reader dead-ended past chapter 200.
Stop on a short page instead; only trust `total` when actually reported.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several frontend-only commits bumped both manifests but staged only the
frontend, leaving backend/Cargo.toml behind. Bring it back in lockstep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The network-error wrapping added in the error-boundary work also caught
deliberate AbortController cancellations and turned them into a network_error
ApiError, breaking callers that distinguish "cancelled" from "failed" (the
cancellable admin fetches). Rethrow DOMException AbortError unchanged; only
genuine network failures become ApiError(status 0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Overlays didn't lock body scroll, so the page behind a modal/sheet scrolled
under it. Add a ref-counted scroll lock (so stacked overlays don't prematurely
release it) and hold it via an effect in Modal and Sheet while open, restoring
on close or unmount. Focus-trapping was already handled.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a partial upload failure (manga created, a chapter failed), re-submitting
re-ran createManga and spawned a duplicate manga. Track the created manga id:
a retry skips creation, re-sends only the not-yet-done chapters, disables the
now-saved title, relabels the button "Retry failed chapters", and shows a note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drag-dropping page images then clicking a nav link silently discarded the
whole form. Add a reusable guardUnsavedChanges helper (SvelteKit beforeNavigate
confirm + native beforeunload) and wire it into the manga upload, manga edit,
and chapter upload forms. The dirty check returns false while submitting so the
post-save redirect isn't prompted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both lists fetched a single capped page (100 / 200) with no pager, so entries
past the cap were silently unreachable. Add a reusable LoadMore component that
seeds from the streamed first page (preserving the auth gate + inline error
handling) and fetches further offset-based pages on demand, and wire it into
the bookmarks and collections pages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reader fetched only the first 200 chapters, so a deep chapter (#250) fell
outside the window and its prev/next chevrons resolved to null — a dead end —
and the chapter dropdown was truncated. Add listAllChapters, which pages
through the API's 200-row cap, and use it in the reader loader.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Chapters/Mangas aggregation views accept an OCR ?text= filter and the
page search intersects tags, but the client dropped both: any text query
collapsed every view into a flat page search that discarded the selected tag
and grouping. Add text to AggregateOptions, thread it into
listTaggedChapters/listTaggedMangas, and rework the search loader so a
tag-scoped view runs the aggregation (optionally text-filtered) and the page
search passes the selected tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A load() that rethrew (e.g. the profile overview) fell through to SvelteKit's
bare default error page, and a connection-refused fetch rejected as a raw
TypeError that blanked the view. Wrap fetch network failures in the client as
ApiError(status 0, network_error) so callers handle them uniformly, and add a
root +error.svelte with a friendly message plus Try again / Go home actions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The admin checkbox fired an immediate, irreversible privilege change with no
confirmation, and on cancel/failure the optimistic DOM flip stuck while the
server state was unchanged. Add a confirm() gate (like Delete) and make the
checkbox controlled by u.is_admin: onchange reverts the DOM immediately and it
only flips once the server confirms — so a cancelled or failed toggle never
misrepresents the real role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md claimed re-uploading a chapter unconditionally drops saved-page
references. In fact the crawler re-fetch (content::persist_pages) upserts by
(chapter_id, page_number) with ON CONFLICT DO UPDATE, preserving pages.id, so
collections/page-tags survive a re-fetch. Saves drop only when a pages row is
actually deleted (chapter deletion, or a user delete-then-recreate). Migration
0023's text is left untouched (sqlx checksums applied migrations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first guard block computed the partial-render condition but had an empty
if-body, so it never skipped anything — the real skip is the chapter-sync
guard further down. Delete the dead block and move its observability warning
into the live guard's else branch, eliminating the data-loss trap without
changing behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
try_acquire used .expect on lock(), so a single panic while holding either
limiter mutex would poison it and make every subsequent auth request re-panic
— one blip became a persistent auth outage. Recover the guard with
unwrap_or_else(|e| e.into_inner()) so the limiter keeps serving.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heartbeat renew failures were only logged, so under DB flakiness a job could
be leased twice concurrently — the original worker kept crawling past a lapsed
lease while another re-leased it. Track consecutive renew failures and, after
MAX_HEARTBEAT_RENEW_FAILURES, signal the worker (via a CancellationToken) to
abandon the in-flight dispatch instead of working a job it no longer owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An idle worker re-issued a row-locking lease query every 1s per worker with
no backoff. Replace the flat sleep with idle_backoff: 1s, 2s, 4s, … capped at
30s, reset to 0 the moment a job is leased, so a quiet daemon stops hammering
the crawler_jobs table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.
Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cover multipart path called Field::bytes(), buffering the entire field
into memory before parse_image checked max_file_bytes — an oversized cover
was fully allocated before rejection. Add read_capped/push_capped, which
reject with 413 as soon as the running total exceeds the cap (the offending
chunk is never copied in), and route both cover handlers and stage_image_part
through it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commit 134ab54 dropped the safe DNS resolver for any configured proxy, but
that guard is only pointless for SOCKS proxies (where the proxy resolves the
target). On an http(s):// proxy reqwest still resolves the target locally, so
a hostname resolving to a private IP would be reachable. Narrow the exemption
to SOCKS-only via should_attach_safe_resolver.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CDP Fetch interceptor only paused main-frame Document requests, so a
scraped page's <img>/fetch()/XHR subresources to internal targets
(169.254.169.254, postgres:5432, RFC1918) reached those hosts unguarded.
Drop the ResourceType::Document constraint so every request runs through
is_blocked. Also make open_page fail-closed: if the guard can't install,
close the blank page and return the error instead of navigating unguarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ff4ca96 wired SafeResolver (drops any host resolving to a private IP) into
all four crawler/analysis reqwest clients. That broke every proxied fetch:
with a socks5h:// proxy the target is resolved by Tor, so reqwest only ever
resolves the proxy's OWN host — `tor`, on a private Docker IP (172.x) — which
the resolver then rejects ("SOCKS error: failed to create underlying
connection"). 100% of crawler image downloads failed. The vision clients
broke the same way against `mangalord-vision` (latent; analysis was off).
Fix: attach the resolver only on the crawler's direct (unproxied) path, where
it genuinely guards DNS rebinding. Behind a proxy it adds no protection (Tor
can't reach internal ranges and the target is invisible to reqwest) so it's
omitted. Drop it from the vision clients entirely — that endpoint is an
operator-configured internal service that must resolve to a private IP.
Security posture is >= the pre-ff4ca96 baseline (which had no resolver at all).
is_private_ip / retain_public_addrs and their tests are unchanged.
Bump 0.124.13.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single global token bucket let one attacker at the sustained rate 429
every user's login/register/change-password. Key buckets by client IP: the
SvelteKit proxy now stamps the real peer address onto X-Forwarded-For
(overriding any client-supplied value, anti-spoof), and axum reads it via a
ClientIp extractor — but only when AUTH_TRUSTED_PROXY is set, else it falls
back to the shared bucket (today's behavior). The per-IP map is bounded
(10k IPs, idle buckets pruned) so a spoofed-IP spray can't grow it.
AUTH_TRUSTED_PROXY defaults false (safe for a directly-exposed backend);
compose sets it true since the proxy is the single trusted hop.
Bump to 0.124.12.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.
Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.
Bump to 0.124.11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SSRF guard only checked the host string, so an attacker-owned domain
resolving to 169.254.169.254 / 10.x (DNS rebinding, TOCTOU) bypassed it.
Add a reqwest dns::Resolve (SafeResolver) that drops resolved addresses in
private ranges, wired into all four crawler/analysis clients — it fires per
connection so it also covers redirect hops. Also close the is_private_ip
IPv6-embedding gaps (IPv4-compatible ::/96, NAT64 64:ff9b::/96, 6to4
2002::/16 all now unwrap to the embedded IPv4).
Bump to 0.124.10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
