Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The analysis worker's flat 1s idle sleep meant an empty queue fired a lease query
per worker per second forever (audit). Add idle_backoff (1s..30s, reset on lease),
mirroring the crawler. Unit-tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
reap_terminal was one unbatched DELETE with no updated_at index — a full scan and
long-held lock over a large backlog (audit M5). Migration 0040 adds a partial
index on (updated_at) WHERE state IN ('done','dead'); reap_terminal now deletes in
bounded 5k batches. Existing tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The lease predicate's `state='running' AND leased_until < now()` arm had no
index, so every lease poll seq-scanned the whole crawler_jobs table (audit H2).
migration 0039 adds a partial index on (leased_until) WHERE state='running'.
Behavior-neutral; 29 lease tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A blob with valid magic but a corrupt body passed upload then 500'd on a ?w=
thumbnail decode (audit). Fall back to serve_original (streams without decoding)
instead. Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Option<Json<T>> collapses a malformed body to None, so a bad reenqueue body
silently ran the default full "All" scope (audit). Parse raw bytes: empty =
default All, malformed = 422. Tested both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
put_cover/delete_cover mutated storage before the DB pointer, so a transient DB
failure could leave the row pointing at a deleted blob. Reorder to
put->set-DB->delete-old and clear-DB->delete-blob, so a failure only orphans a
blob, never dangles the pointer. Happy path covered by the 16 existing cover tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retargeting the modal mid-load could resolve responses out of order, overwriting
the newer target's membership with the older one's (audit FE-2). Route load()
through the tested CancellableLoader so a superseded result is dropped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Arrows/j/k/Home/End bubbled to the reader's window handler while a
jump/settings/action/collections/tags/context overlay was open, paging the
chapter behind it (FE-1); native select/contenteditable weren't exempt (FE-4).
New unit-tested isTypingTarget() + an overlay-open early return fix both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
genres had only case-sensitive name UNIQUE while authors/tags use
UNIQUE (lower(name)); sync_genres could race two case variants into duplicate
rows (audit DB-3). Migration 0038 pre-dedups then adds
UNIQUE (lower(name)); sync_genres upserts ON CONFLICT (lower(name)). Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deterministic per-page storage keys mean a re-crawl overwrites the image but the
pages row survives the upsert, so analysis keyed to page_id (page_analysis/
search_doc, OCR, auto-tags, content-warnings) stayed stale forever and
force=false re-enqueue never re-analyzed it — poisoning search + the derived
manga_content_warnings (audit DB-1). persist_pages now detects conflict-updated
pages via xmax and clears their four derived tables in the same tx (the
content-warnings DELETE fires the mcw_* triggers), returning them to unanalyzed
so they re-derive. Tested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Harvested source_url (external content) rendered as an admin <a href> without
scheme validation — a javascript:/data: link would execute on click (audit M2).
safeExternalHref() gates to http(s); CrawlerHistoryTable/DeadJobsTable fall back
to plain text otherwise. Tested in safeHref.test.ts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two integration tests pinning both sides of the gate: trusted_proxy=true gives
each forwarded IP its own bucket; trusted_proxy=false ignores spoofed XFF and
shares the global bucket. Closes audit gap M7. Test-only, no behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
constant_time_eq (and the `subtle` dependency) was never wired into any
comparison — the module doc wrongly claimed token comparison used it, but lookup
is an indexed DB equality on the 256-bit SHA-256 hash, so there's no in-process
secret compare to time-attack. Delete the dead fn, its test, and the unused
`subtle` dep; rewrite the doc. No behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sessions table only grew — find_active ignored expired rows but nothing
deleted them, so lapsed sessions accumulated indefinitely.
- repo::session::delete_expired: indexed DELETE (sessions_expires_idx) returning
the reaped count.
- app::build spawns a detached hourly reaper (SESSION_GC_INTERVAL) that calls it,
independent of crawler/analysis config.
Test: delete_expired_removes_only_lapsed_sessions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /files handler served the application/octet-stream fallback with no
Content-Disposition, letting a browser render it inline — a stored-XSS vector if
the blob is crafted HTML/JS. Add Content-Disposition: attachment for that type
(belt to the existing nosniff); known image types stay inline.
Test (new tests/api_files.rs): octet-stream → attachment; png → inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `metadata` JSON part in the manga-create and chapter-upload handlers was
read with `Field::bytes()`, buffering the whole part before any check — a client
could send a multi-hundred-MB metadata blob (up to the 200 MiB request limit) as
one allocation. Image parts were already streamed with a cap; metadata was the
last unbounded read.
Cap it at a shared upload::MAX_METADATA_BYTES (64 KiB) via the read_capped
streaming loop (413 on overflow), and remove the now-unused read_field_bytes
helper so the unbounded path can't return.
Tests: manga + chapter oversized-metadata parts are rejected with 413.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`get` (fs::read → EISDIR) and `get_stream` (File::open succeeds on a Unix
directory, then streams garbage that fails mid-read) both mishandled a key
resolving to a directory. Mirror `size`'s existing guard: `get` maps
IsADirectory to NotFound; `get_stream` checks metadata.is_file() after open.
Test: get_on_directory_is_not_found.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `%` or `_` in a search term silently acted as a LIKE wildcard rather than
matching literally (`50%` matched everything, `a_b` matched `axb`). Not
injection — terms are bound — but a search-correctness bug across every
user-facing ILIKE site.
- repo::escape_like: shared pub(crate) escaper (promoted from page_tag), unit-tested.
- Trigram-entangled sites (manga, tag, author) append a separate escaped param
used only by the ILIKE branch; the trigram/similarity branches keep the raw term.
- Pattern-built sites (admin manga list, admin users, analysis coverage + history,
crawler search incl. JSONB payload title) escape inside format!("%{}%", ..) and
pair each ILIKE with ESCAPE '\'.
Tests: escape_like unit tests + integration tests on public manga search, author
autocomplete, and admin user search proving `_` matches literally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document responses carried no CSP, X-Frame-Options, Referrer-Policy, or
Permissions-Policy — leaving clickjacking and zero script-injection defense
in depth (security audit T4).
- svelte.config.js: enable kit.csp hash mode. SvelteKit hashes its own inline
hydration scripts; the inline theme <script> in app.html isn't part of
%sveltekit.head%, so its sha256 is pinned by hand (csp-config.js) and added
to script-src alongside object-src 'none', base-uri 'self', frame-ancestors
'none'. Styles stay unconstrained (Svelte emits dynamic inline style= attrs).
- hooks.server.ts: applySecurityHeaders() sets X-Frame-Options: DENY,
Referrer-Policy: strict-origin-when-cross-origin, X-Content-Type-Options:
nosniff, and a Permissions-Policy locking down unused features, only when the
response hasn't already set them.
- src/csp-theme-hash.test.ts: drift guard against editing the theme script
without updating THEME_SCRIPT_HASH.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
listAllChapters used `page.total ?? all.length` as the loop bound. The
/chapters endpoint always returns `total: null`, so a full 200-row first
page broke the loop immediately — the reader dead-ended past chapter 200.
Stop on a short page instead; only trust `total` when actually reported.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several frontend-only commits bumped both manifests but staged only the
frontend, leaving backend/Cargo.toml behind. Bring it back in lockstep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The admin checkbox fired an immediate, irreversible privilege change with no
confirmation, and on cancel/failure the optimistic DOM flip stuck while the
server state was unchanged. Add a confirm() gate (like Delete) and make the
checkbox controlled by u.is_admin: onchange reverts the DOM immediately and it
only flips once the server confirms — so a cancelled or failed toggle never
misrepresents the real role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first guard block computed the partial-render condition but had an empty
if-body, so it never skipped anything — the real skip is the chapter-sync
guard further down. Delete the dead block and move its observability warning
into the live guard's else branch, eliminating the data-loss trap without
changing behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
try_acquire used .expect on lock(), so a single panic while holding either
limiter mutex would poison it and make every subsequent auth request re-panic
— one blip became a persistent auth outage. Recover the guard with
unwrap_or_else(|e| e.into_inner()) so the limiter keeps serving.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heartbeat renew failures were only logged, so under DB flakiness a job could
be leased twice concurrently — the original worker kept crawling past a lapsed
lease while another re-leased it. Track consecutive renew failures and, after
MAX_HEARTBEAT_RENEW_FAILURES, signal the worker (via a CancellationToken) to
abandon the in-flight dispatch instead of working a job it no longer owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An idle worker re-issued a row-locking lease query every 1s per worker with
no backoff. Replace the flat sleep with idle_backoff: 1s, 2s, 4s, … capped at
30s, reset to 0 the moment a job is leased, so a quiet daemon stops hammering
the crawler_jobs table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.
Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cover multipart path called Field::bytes(), buffering the entire field
into memory before parse_image checked max_file_bytes — an oversized cover
was fully allocated before rejection. Add read_capped/push_capped, which
reject with 413 as soon as the running total exceeds the cap (the offending
chunk is never copied in), and route both cover handlers and stage_image_part
through it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commit 134ab54 dropped the safe DNS resolver for any configured proxy, but
that guard is only pointless for SOCKS proxies (where the proxy resolves the
target). On an http(s):// proxy reqwest still resolves the target locally, so
a hostname resolving to a private IP would be reachable. Narrow the exemption
to SOCKS-only via should_attach_safe_resolver.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CDP Fetch interceptor only paused main-frame Document requests, so a
scraped page's <img>/fetch()/XHR subresources to internal targets
(169.254.169.254, postgres:5432, RFC1918) reached those hosts unguarded.
Drop the ResourceType::Document constraint so every request runs through
is_blocked. Also make open_page fail-closed: if the guard can't install,
close the blank page and return the error instead of navigating unguarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ff4ca96 wired SafeResolver (drops any host resolving to a private IP) into
all four crawler/analysis reqwest clients. That broke every proxied fetch:
with a socks5h:// proxy the target is resolved by Tor, so reqwest only ever
resolves the proxy's OWN host — `tor`, on a private Docker IP (172.x) — which
the resolver then rejects ("SOCKS error: failed to create underlying
connection"). 100% of crawler image downloads failed. The vision clients
broke the same way against `mangalord-vision` (latent; analysis was off).
Fix: attach the resolver only on the crawler's direct (unproxied) path, where
it genuinely guards DNS rebinding. Behind a proxy it adds no protection (Tor
can't reach internal ranges and the target is invisible to reqwest) so it's
omitted. Drop it from the vision clients entirely — that endpoint is an
operator-configured internal service that must resolve to a private IP.
Security posture is >= the pre-ff4ca96 baseline (which had no resolver at all).
is_private_ip / retain_public_addrs and their tests are unchanged.
Bump 0.124.13.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single global token bucket let one attacker at the sustained rate 429
every user's login/register/change-password. Key buckets by client IP: the
SvelteKit proxy now stamps the real peer address onto X-Forwarded-For
(overriding any client-supplied value, anti-spoof), and axum reads it via a
ClientIp extractor — but only when AUTH_TRUSTED_PROXY is set, else it falls
back to the shared bucket (today's behavior). The per-IP map is bounded
(10k IPs, idle buckets pruned) so a spoofed-IP spray can't grow it.
AUTH_TRUSTED_PROXY defaults false (safe for a directly-exposed backend);
compose sets it true since the proxy is the single trusted hop.
Bump to 0.124.12.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.
Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.
Bump to 0.124.11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SSRF guard only checked the host string, so an attacker-owned domain
resolving to 169.254.169.254 / 10.x (DNS rebinding, TOCTOU) bypassed it.
Add a reqwest dns::Resolve (SafeResolver) that drops resolved addresses in
private ranges, wired into all four crawler/analysis clients — it fires per
connection so it also covers redirect hops. Also close the is_private_ip
IPv6-embedding gaps (IPv4-compatible ::/96, NAT64 64:ff9b::/96, 6to4
2002::/16 all now unwrap to the embedded IPv4).
Bump to 0.124.10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The immutable Cache-Control added for the reader preload work was
`public` unconditionally. Under PRIVATE_MODE, /files is auth-gated, so a
shared cache/CDN would store the blob and serve it to unauthenticated
clients, defeating the gate. Emit `private, ...` when private_mode is on,
`public, ...` otherwise.
Found in the security audit; regression from 0.124.3. Bump to 0.124.9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The new DB_MAX_CONNECTIONS / DB_ACQUIRE_TIMEOUT_SECS knobs were read by the
backend but never added to .env.example or docker-compose.yml, so an operator
setting them in .env would see no effect (caught by the config meta-tests).
Document both in .env.example and wire them into backend.environment.
Bump to 0.124.8.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /mangas recomputed count(*) over FILTER_WHERE (up to five correlated
subqueries) on every page, so pagination cost scaled with catalog size. The
total doesn't change as a caller walks pages, so compute it once at offset 0
and return null thereafter (the envelope already serialises total as
Option<i64>; the frontend types it number|null and the library route ignores
it).
Bump to 0.124.7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.
Bump to 0.124.6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
new_page() opened a tab that was only closed on the happy path; a `?` on
wait_for_nav / content() leaked it (chromiumoxide doesn't close on drop),
so errored fetches accumulated tabs in the shared browser. Add a generic,
unit-tested `close_after(close, body)` helper and wrap the three fetch
sites (chapter content, session probe, source navigate) so the tab is
closed on every exit path — including navigate's content()-read branch,
which previously leaked.
Bump to 0.124.5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single Postgres pool (shared by HTTP handlers and the crawler/analysis
daemons) hardcoded max_connections=10 with no acquire_timeout, so under
saturation callers hung on the driver's silent 30s default. Add a DbConfig
(DB_MAX_CONNECTIONS default 20, DB_ACQUIRE_TIMEOUT_SECS default 10, clamped
to >=1) and apply both in PgPoolOptions.
Bump to 0.124.4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Page and cover blobs are content-addressed by unguessable, immutable keys,
but /files sent no Cache-Control, so the browser refetched on every reader
open — defeating the whole-chapter and next-chapter preloading (and re-
proxying every byte through the node server in prod). Send
`Cache-Control: public, max-age=31536000, immutable`.
Bump to 0.124.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuous mode rendered every page (up to 2000) with loading="eager",
fetching and decoding the whole chapter up front — enough to OOM a mobile
tab on a long chapter. Eager-load only a leading window ([0..=eagerThrough],
covering the deep-link target + a lead, min 6 pages) and lazy-load the rest;
the browser fetches them as the reader scrolls near. Unloaded lazy pages
reserve height (page width x ~1.4) so they stay distinct scroll targets for
the progress IntersectionObserver, cleared on load.
Keeps the DOM dense so the scroll-to-?page=N effect, progress observer, and
next-chapter preload trigger are all unchanged. Rewrote reader-preload-window
spec to assert the eager/lazy attribute partition + deep-link scroll landing.
Bump to 0.124.2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Argon2id hashing (~15-50ms, 19 MiB) ran synchronously inside async
handlers, stalling every task sharing those runtime worker threads under
concurrent auth load. Add spawn_blocking wrappers hash_password_async /
verify_password_async and switch all async call sites; sync primitives
stay for the login timing-equaliser and unit tests. Bump to 0.124.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
