The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The first guard block computed the partial-render condition but had an empty
if-body, so it never skipped anything — the real skip is the chapter-sync
guard further down. Delete the dead block and move its observability warning
into the live guard's else branch, eliminating the data-loss trap without
changing behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
try_acquire used .expect on lock(), so a single panic while holding either
limiter mutex would poison it and make every subsequent auth request re-panic
— one blip became a persistent auth outage. Recover the guard with
unwrap_or_else(|e| e.into_inner()) so the limiter keeps serving.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heartbeat renew failures were only logged, so under DB flakiness a job could
be leased twice concurrently — the original worker kept crawling past a lapsed
lease while another re-leased it. Track consecutive renew failures and, after
MAX_HEARTBEAT_RENEW_FAILURES, signal the worker (via a CancellationToken) to
abandon the in-flight dispatch instead of working a job it no longer owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An idle worker re-issued a row-locking lease query every 1s per worker with
no backoff. Replace the flat sleep with idle_backoff: 1s, 2s, 4s, … capped at
30s, reset to 0 the moment a job is leased, so a quiet daemon stops hammering
the crawler_jobs table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.
Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cover multipart path called Field::bytes(), buffering the entire field
into memory before parse_image checked max_file_bytes — an oversized cover
was fully allocated before rejection. Add read_capped/push_capped, which
reject with 413 as soon as the running total exceeds the cap (the offending
chunk is never copied in), and route both cover handlers and stage_image_part
through it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Commit 134ab54 dropped the safe DNS resolver for any configured proxy, but
that guard is only pointless for SOCKS proxies (where the proxy resolves the
target). On an http(s):// proxy reqwest still resolves the target locally, so
a hostname resolving to a private IP would be reachable. Narrow the exemption
to SOCKS-only via should_attach_safe_resolver.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CDP Fetch interceptor only paused main-frame Document requests, so a
scraped page's <img>/fetch()/XHR subresources to internal targets
(169.254.169.254, postgres:5432, RFC1918) reached those hosts unguarded.
Drop the ResourceType::Document constraint so every request runs through
is_blocked. Also make open_page fail-closed: if the guard can't install,
close the blank page and return the error instead of navigating unguarded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ff4ca96 wired SafeResolver (drops any host resolving to a private IP) into
all four crawler/analysis reqwest clients. That broke every proxied fetch:
with a socks5h:// proxy the target is resolved by Tor, so reqwest only ever
resolves the proxy's OWN host — `tor`, on a private Docker IP (172.x) — which
the resolver then rejects ("SOCKS error: failed to create underlying
connection"). 100% of crawler image downloads failed. The vision clients
broke the same way against `mangalord-vision` (latent; analysis was off).
Fix: attach the resolver only on the crawler's direct (unproxied) path, where
it genuinely guards DNS rebinding. Behind a proxy it adds no protection (Tor
can't reach internal ranges and the target is invisible to reqwest) so it's
omitted. Drop it from the vision clients entirely — that endpoint is an
operator-configured internal service that must resolve to a private IP.
Security posture is >= the pre-ff4ca96 baseline (which had no resolver at all).
is_private_ip / retain_public_addrs and their tests are unchanged.
Bump 0.124.13.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A single global token bucket let one attacker at the sustained rate 429
every user's login/register/change-password. Key buckets by client IP: the
SvelteKit proxy now stamps the real peer address onto X-Forwarded-For
(overriding any client-supplied value, anti-spoof), and axum reads it via a
ClientIp extractor — but only when AUTH_TRUSTED_PROXY is set, else it falls
back to the shared bucket (today's behavior). The per-IP map is bounded
(10k IPs, idle buckets pruned) so a spoofed-IP spray can't grow it.
AUTH_TRUSTED_PROXY defaults false (safe for a directly-exposed backend);
compose sets it true since the proxy is the single trusted hop.
Bump to 0.124.12.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.
Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.
Bump to 0.124.11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SSRF guard only checked the host string, so an attacker-owned domain
resolving to 169.254.169.254 / 10.x (DNS rebinding, TOCTOU) bypassed it.
Add a reqwest dns::Resolve (SafeResolver) that drops resolved addresses in
private ranges, wired into all four crawler/analysis clients — it fires per
connection so it also covers redirect hops. Also close the is_private_ip
IPv6-embedding gaps (IPv4-compatible ::/96, NAT64 64:ff9b::/96, 6to4
2002::/16 all now unwrap to the embedded IPv4).
Bump to 0.124.10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The immutable Cache-Control added for the reader preload work was
`public` unconditionally. Under PRIVATE_MODE, /files is auth-gated, so a
shared cache/CDN would store the blob and serve it to unauthenticated
clients, defeating the gate. Emit `private, ...` when private_mode is on,
`public, ...` otherwise.
Found in the security audit; regression from 0.124.3. Bump to 0.124.9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /mangas recomputed count(*) over FILTER_WHERE (up to five correlated
subqueries) on every page, so pagination cost scaled with catalog size. The
total doesn't change as a caller walks pages, so compute it once at offset 0
and return null thereafter (the envelope already serialises total as
Option<i64>; the frontend types it number|null and the library route ignores
it).
Bump to 0.124.7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.
Bump to 0.124.6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
new_page() opened a tab that was only closed on the happy path; a `?` on
wait_for_nav / content() leaked it (chromiumoxide doesn't close on drop),
so errored fetches accumulated tabs in the shared browser. Add a generic,
unit-tested `close_after(close, body)` helper and wrap the three fetch
sites (chapter content, session probe, source navigate) so the tab is
closed on every exit path — including navigate's content()-read branch,
which previously leaked.
Bump to 0.124.5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single Postgres pool (shared by HTTP handlers and the crawler/analysis
daemons) hardcoded max_connections=10 with no acquire_timeout, so under
saturation callers hung on the driver's silent 30s default. Add a DbConfig
(DB_MAX_CONNECTIONS default 20, DB_ACQUIRE_TIMEOUT_SECS default 10, clamped
to >=1) and apply both in PgPoolOptions.
Bump to 0.124.4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Page and cover blobs are content-addressed by unguessable, immutable keys,
but /files sent no Cache-Control, so the browser refetched on every reader
open — defeating the whole-chapter and next-chapter preloading (and re-
proxying every byte through the node server in prod). Send
`Cache-Control: public, max-age=31536000, immutable`.
Bump to 0.124.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Argon2id hashing (~15-50ms, 19 MiB) ran synchronously inside async
handlers, stalling every task sharing those runtime worker threads under
concurrent auth load. Add spawn_blocking wrappers hash_password_async /
verify_password_async and switch all async call sites; sync primitives
stay for the login timing-equaliser and unit tests. Bump to 0.124.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /me/recommendations: rank mangas by weighted tag overlap with the
user's taste — explicit like +1.0, bookmark +0.5, dislike −1.0 (reaction
overrides bookmark). Per-tag affinities are summed, candidates scored by
their tags' affinity normalized by tag count (anti-tag-stuffing, à la
list_similar), with already-reacted/bookmarked/read mangas excluded and
net-negative candidates dropped (dislike down-ranks, not browse-hides).
Reuses manga_cols/cards_from_rows. Integration tests cover like-driven recs,
dislike down-rank, bookmark half-weight ordering, seen-exclusion, empty
cold-start, and auth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a private per-user reaction resource: a manga_reactions table (one row
per user+manga, like/dislike, cascades) and endpoints PUT/DELETE
/mangas/:id/reaction plus GET /me/reactions/:manga_id, wired on the existing
per-user seam (mirrors bookmarks/read_progress). Unknown manga → 404 via the
FK-violation mapping; an invalid reaction value → 422. Reactions are never
exposed publicly — this is a taste signal for the upcoming content-based
recommendations. Integration tests cover set/toggle/clear/read, per-user
isolation, and the 404/422/401 paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A "Continue reading" shelf listing series the reader already finished (read
to the last page, nothing new) is odd. Expose the last-read chapter's
page_count on the read-progress list, and filter the shelf to drop entries
that are caught up (on the last page of the latest chapter with no new
chapters). Mid-chapter and has-new-chapters entries stay; the full history
view is unaffected. Finished detection is a pure, unit-tested helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
new_chapters_count over-counted when scanlations share a chapter number
((manga_id, number) is non-unique, migration 0013): COUNT(*) tallied rows,
so duplicate-numbered chapters inflated the badge. Switch both the list and
per-manga read-progress queries to COUNT(DISTINCT number), and drop the dead
COALESCE (an empty set counts as 0, and the NULL-number case is handled by
the predicate). Fix the misleading comment.
Expose new_chapters_count on GET /me/read-progress/:manga_id and drive the
detail page's "N new" badge from it, instead of recomputing over the
paginated chapter list — which under-counted series past 50 chapters and
disagreed with the homepage shelf. Read markers stay client-side over the
loaded chapters and are documented as by-number (all we persist).
Adds duplicate-scanlation tests at both endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `new_chapters_count` to each `GET /me/read-progress` item: how many
chapters sit past the reader's last-read chapter, computed by a correlated
subquery on chapter number. It is 0 when the reader is caught up, and 0
when the last-read chapter is unknown (manga-level progress or a deleted
chapter) so we never over-claim. (manga_id, number) is non-unique, so this
counts positions past the reader rather than distinct files.
Backs the personal "new since last read" indicator on the upcoming
Continue-reading shelf. Tests cover the mid-series, caught-up, and
no-read-chapter cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The retention reaper deleted only `state = 'done'` rows, so terminal
`dead` jobs (retries exhausted) accumulated in crawler_jobs forever —
slow unbounded table growth. Both states are end-of-life and never
leased again.
Rename `reap_done` → `reap_terminal` and broaden the filter to
`state IN ('done','dead')`; `pending`/`running` stay untouched. Test
extended to assert old done + old dead both reap while fresh terminal and
old-but-active rows survive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vision prep pass decoded pages with bare `image::load_from_memory`,
which applies no allocation bound. `max_pixels` only downscales after a
full decode, so a tiny WebP/JPEG header declaring huge dimensions could
OOM the blocking worker — the same decompression-bomb the OCR backend
already guards against. Manga pages are commonly WebP/JPEG, where the
format's own self-limits are weaker than PNG's.
Route the decode through `decode_within`, applying an `image::Limits`
alloc cap sized from the shared `ocr_max_decode_pixels`
(ANALYSIS_OCR_MAX_DECODE_PIXELS). Add real-WebP bomb coverage asserting
both the helper and the end-to-end Undecodable fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR dispatcher acquired a borrowed semaphore permit, then ran the
CPU-bound inference in `spawn_blocking`. On cancellation (graceful
shutdown) the future dropped the permit while the detached blocking task
kept running, briefly oversubscribing the blocking pool past
ANALYSIS_WORKERS.
Extract `run_ocr_blocking`, which acquires an *owned* permit and moves it
into the blocking task so the concurrency slot's lifetime matches the
actual work. Covered by a cancellation-invariant test asserting the
permit stays held after the caller future is aborted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-image byte cap bounds each download but not the count, so a
hostile or compromised reader page listing thousands of <img> tags could
drive an unbounded disk fill. Add `CRAWLER_MAX_IMAGES_PER_CHAPTER`
(CrawlerConfig::max_images_per_chapter, default 2000, 0 = disabled) and
reject an over-cap chapter with a failed ack (exponential backoff) rather
than downloading it.
Threaded through sync_chapter_content and its three call sites (daemon
dispatcher, admin resync, CLI); enforced via `image_count_over_cap` right
after parse. The cap is an env-only safety knob, preserved across settings
reloads.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-image download cap subtracted the constant SNIFF_PREFIX_BYTES (64)
from the budget instead of the bytes actually drained into the prefix.
Because the prefix loop appends whole chunks, a single ~16 KiB first chunk
fills the 64-byte sniff window in one drain, so the tail could then add
another near-full cap — storing up to ~2× max_image_bytes per page.
Capture prefix.len() before the prefix is moved into the stream and budget
the tail as `max_image_bytes - prefix_len` via a small `remaining_after_prefix`
helper (unit-tested for the overshoot and saturation cases).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`navigate()` (list/detail/pagination) and the session probe called
`Browser::new_page()` without the SSRF check that already guards the
chapter-content path, so a hostile or compromised scraped source could
serve `<a href="http://169.254.169.254/…">` / `http://postgres:5432/` in a
listing and use the in-container Chromium as a read oracle.
Add `guard_navigate_url` (reusing `ensure_public_target`) at the top of
`navigate`, before rate-limiting or opening a page, and apply the same
check to `fetch_probe_html`. Covers base URL, pagination, and detail links.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:
- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
literal `Err`).
- test helpers: build configs via struct-update syntax instead of
`Default::default()` + field reassignment; add a type alias for a
complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.
Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
