The sessions table only grew — find_active ignored expired rows but nothing
deleted them, so lapsed sessions accumulated indefinitely.
- repo::session::delete_expired: indexed DELETE (sessions_expires_idx) returning
the reaped count.
- app::build spawns a detached hourly reaper (SESSION_GC_INTERVAL) that calls it,
independent of crawler/analysis config.
Test: delete_expired_removes_only_lapsed_sessions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The /files handler served the application/octet-stream fallback with no
Content-Disposition, letting a browser render it inline — a stored-XSS vector if
the blob is crafted HTML/JS. Add Content-Disposition: attachment for that type
(belt to the existing nosniff); known image types stay inline.
Test (new tests/api_files.rs): octet-stream → attachment; png → inline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The `metadata` JSON part in the manga-create and chapter-upload handlers was
read with `Field::bytes()`, buffering the whole part before any check — a client
could send a multi-hundred-MB metadata blob (up to the 200 MiB request limit) as
one allocation. Image parts were already streamed with a cap; metadata was the
last unbounded read.
Cap it at a shared upload::MAX_METADATA_BYTES (64 KiB) via the read_capped
streaming loop (413 on overflow), and remove the now-unused read_field_bytes
helper so the unbounded path can't return.
Tests: manga + chapter oversized-metadata parts are rejected with 413.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `%` or `_` in a search term silently acted as a LIKE wildcard rather than
matching literally (`50%` matched everything, `a_b` matched `axb`). Not
injection — terms are bound — but a search-correctness bug across every
user-facing ILIKE site.
- repo::escape_like: shared pub(crate) escaper (promoted from page_tag), unit-tested.
- Trigram-entangled sites (manga, tag, author) append a separate escaped param
used only by the ILIKE branch; the trigram/similarity branches keep the raw term.
- Pattern-built sites (admin manga list, admin users, analysis coverage + history,
crawler search incl. JSONB payload title) escape inside format!("%{}%", ..) and
pair each ILIKE with ESCAPE '\'.
Tests: escape_like unit tests + integration tests on public manga search, author
autocomplete, and admin user search proving `_` matches literally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The account page pointed users at a "bot-token list" that didn't exist: there
was no GET endpoint, no UI route, and the client type lacked expiry. Add
GET /v1/auth/tokens (caller-scoped, token_hash never serialised), extend the
auth client with listTokens/expires_at and createToken expiry, and add a
/profile/tokens page to list, create (revealing the raw bearer once), and
revoke tokens. Link the account-page copy to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add /files/{key}?w=<px>: the endpoint decodes JPEG/PNG sources, downscales to
a width snapped to a small allow-list (aspect-preserving, never upscaling),
caches the result under a thumbs/ prefix, and serves it — so cover grids
download ~KB instead of the 1–5 MB original. Cover handlers purge cached
variants when a cover (whose key is reused) is replaced or deleted. Frontend
grids use new thumbUrl/thumbSrcset helpers (MangaCard with responsive srcset).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
?sort=author used a correlated min(lower(a.name)) subquery as the ORDER BY
key, evaluated per filter-matching row before LIMIT. Materialize it into
mangas.sort_author (indexed by mangas_sort_author_idx), maintained by triggers
on manga_authors and backfilled in the migration, so the sort is a plain
indexed column read. Author names are immutable, so only join changes matter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The content-warning include/exclude filter (and its count) ran a correlated
page_content_warnings->pages->chapters join per candidate manga — O(mangas *
pages) on every filtered list. Add a manga_content_warnings(manga_id, warning)
table holding each manga's deduped warning union, maintained by triggers that
recompute an affected manga's set (delete+reinsert of the tiny five-label set,
robust across page/chapter/manga cascade deletes), backfilled in the migration.
The filter is now a single indexed lookup.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Uploaded chapters have no source_index, and ORDER BY source_index DESC NULLS
LAST sorted every one of them after all crawled chapters regardless of number
— an uploaded chapter 5 landed after crawled chapter 100. Slot each uploaded
chapter by number, just before the crawled chapter with the smallest greater
number, so it interleaves. Crawled rows keep their reversed source-DOM order,
preserving the deliberate placement of non-numeric entries (migration 0021).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-adding an already-bookmarked manga returned 409, which the UI surfaced as
a false "Could not add bookmark" toast — collections are idempotent, bookmarks
weren't. On a unique violation, fetch and return the existing bookmark, and
answer 200 (vs 201 for a fresh insert). The frontend toggle already treats a
successful POST as done, so the false toast disappears with no client change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A job's attempt is incremented at lease time, then the dispatcher acquires
the headless browser. When the browser was down or mid-restart, acquire()
failed and the error was ack_failed — so during an outage the whole pending
backlog was chewed to `dead` within minutes while workers hot-looped.
Add SyncOutcome::BrowserUnavailable: the dispatcher returns it instead of
propagating the acquire error, and the daemon releases the lease (refunding
the attempt, like the cancel/session paths) and backs off, so an outage is
treated as an infrastructure blip rather than per-job failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.
Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.
Bump to 0.124.11.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The immutable Cache-Control added for the reader preload work was
`public` unconditionally. Under PRIVATE_MODE, /files is auth-gated, so a
shared cache/CDN would store the blob and serve it to unauthenticated
clients, defeating the gate. Emit `private, ...` when private_mode is on,
`public, ...` otherwise.
Found in the security audit; regression from 0.124.3. Bump to 0.124.9.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /mangas recomputed count(*) over FILTER_WHERE (up to five correlated
subqueries) on every page, so pagination cost scaled with catalog size. The
total doesn't change as a caller walks pages, so compute it once at offset 0
and return null thereafter (the envelope already serialises total as
Option<i64>; the frontend types it number|null and the library route ignores
it).
Bump to 0.124.7.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.
Bump to 0.124.6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Page and cover blobs are content-addressed by unguessable, immutable keys,
but /files sent no Cache-Control, so the browser refetched on every reader
open — defeating the whole-chapter and next-chapter preloading (and re-
proxying every byte through the node server in prod). Send
`Cache-Control: public, max-age=31536000, immutable`.
Bump to 0.124.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /me/recommendations: rank mangas by weighted tag overlap with the
user's taste — explicit like +1.0, bookmark +0.5, dislike −1.0 (reaction
overrides bookmark). Per-tag affinities are summed, candidates scored by
their tags' affinity normalized by tag count (anti-tag-stuffing, à la
list_similar), with already-reacted/bookmarked/read mangas excluded and
net-negative candidates dropped (dislike down-ranks, not browse-hides).
Reuses manga_cols/cards_from_rows. Integration tests cover like-driven recs,
dislike down-rank, bookmark half-weight ordering, seen-exclusion, empty
cold-start, and auth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a private per-user reaction resource: a manga_reactions table (one row
per user+manga, like/dislike, cascades) and endpoints PUT/DELETE
/mangas/:id/reaction plus GET /me/reactions/:manga_id, wired on the existing
per-user seam (mirrors bookmarks/read_progress). Unknown manga → 404 via the
FK-violation mapping; an invalid reaction value → 422. Reactions are never
exposed publicly — this is a taste signal for the upcoming content-based
recommendations. Integration tests cover set/toggle/clear/read, per-user
isolation, and the 404/422/401 paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A "Continue reading" shelf listing series the reader already finished (read
to the last page, nothing new) is odd. Expose the last-read chapter's
page_count on the read-progress list, and filter the shelf to drop entries
that are caught up (on the last page of the latest chapter with no new
chapters). Mid-chapter and has-new-chapters entries stay; the full history
view is unaffected. Finished detection is a pure, unit-tested helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
new_chapters_count over-counted when scanlations share a chapter number
((manga_id, number) is non-unique, migration 0013): COUNT(*) tallied rows,
so duplicate-numbered chapters inflated the badge. Switch both the list and
per-manga read-progress queries to COUNT(DISTINCT number), and drop the dead
COALESCE (an empty set counts as 0, and the NULL-number case is handled by
the predicate). Fix the misleading comment.
Expose new_chapters_count on GET /me/read-progress/:manga_id and drive the
detail page's "N new" badge from it, instead of recomputing over the
paginated chapter list — which under-counted series past 50 chapters and
disagreed with the homepage shelf. Read markers stay client-side over the
loaded chapters and are documented as by-number (all we persist).
Adds duplicate-scanlation tests at both endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `new_chapters_count` to each `GET /me/read-progress` item: how many
chapters sit past the reader's last-read chapter, computed by a correlated
subquery on chapter number. It is 0 when the reader is caught up, and 0
when the last-read chapter is unknown (manga-level progress or a deleted
chapter) so we never over-claim. (manga_id, number) is non-unique, so this
counts positions past the reader rather than distinct files.
Backs the personal "new since last read" indicator on the upcoming
Continue-reading shelf. Tests cover the mid-series, caught-up, and
no-read-chapter cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The retention reaper deleted only `state = 'done'` rows, so terminal
`dead` jobs (retries exhausted) accumulated in crawler_jobs forever —
slow unbounded table growth. Both states are end-of-life and never
leased again.
Rename `reap_done` → `reap_terminal` and broaden the filter to
`state IN ('done','dead')`; `pending`/`running` stay untouched. Test
extended to assert old done + old dead both reap while fresh terminal and
old-but-active rows survive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:
- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
literal `Err`).
- test helpers: build configs via struct-update syntax instead of
`Default::default()` + field reassignment; add a type alias for a
complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.
Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sort the manga catalog by created / updated / title / author with an
independent asc/desc direction control. An omitted `order` defaults per field
(dates newest-first, text A→Z), matching the UI, so a bare `?sort=<field>`
means the same thing in the browser and over the API; `sort=recent` remains a
back-compat alias for `created`.
- Backend: SortField/SortOrder parsed with validation (structured 422 on bad
input), per-field default_order, NULLS LAST only on the nullable author key,
and migration 0033 indexing mangas(updated_at DESC, id) to back the default
sort and its id tie-break.
- Frontend: catalog sort field + direction (SegmentedControl) on desktop and a
mobile bottom sheet; pure helpers in $lib/mangaSort; keyboard-accessible
direction control; visible Direction labels and a sheet "Done" button.
- Tests: backend integration coverage (defaults, alias, invalid input,
ascending-id tie-break), frontend unit + e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.
Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.11 followup. The UPDATE matched pending+running but the worker
holds the pre-update `force=false` in an in-memory local — so on the
running branch the row flipped but the worker's skip-if-done still
acked done without re-analyzing. Now `jobs::release` any running row
the UPDATE matched, so the worker's ack_done no-ops (state guard) and
a fresh lease picks up the updated payload.
New test mints the running-state shape, asserts state=pending +
force=true + attempts=0 post-click. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.10 followup from the rereview. The fix lacked a test for the
specific axis it changed (admin authority via cookie only). New test
mints a bearer for a promoted admin and asserts PATCH/PUT-cover/DELETE-
cover all 403 on NULL-uploader rows. Mutation-confirmed.
Also: `MultipartBuilder::finalize` now `pub` for bearer-multipart use;
`admin_csrf_guard` rustdoc updated to describe the post-0.87.10
cookie-precedence rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four 0.87.6 follow-ups from the adversarial review:
1. **Migration 0031 pre-dedup.** Demote all-but-lowest-id duplicate
`analyze_page` rows in `(pending|running)` to `dead` before
creating the unique index, with a curator-recoverable last_error
marker. Without this, `sqlx::migrate!` would refuse to boot on
any dirty production DB.
2. **`enqueue_for_page(force=true)` collision.** The partial unique
index used to silently swallow force requests when a
`force=false` job was already pending. Repo function now upgrades
the pending row's `force` flag in place (or falls through to
re-INSERT if the sibling drained mid-call), and reports an
`EnqueueForPageOutcome` for accurate auditing.
3. **`record_duration` gate test.** New test seeds a `done` row with
known duration, force-re-analyzes with failing dispatcher +
max_attempts=3 (non-terminal), asserts duration_ms wasn't
overwritten.
4. **`bookmark.rs` PK comment correction.** Use `b.id DESC` instead
of the wrong `manga_id`; PK is actually `id`, and migration 0004
allows both chapter-level and manga-level bookmarks on the same
manga.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to 0.87.2 + 0.87.3 from the adversarial review:
1. **CSRF cookie-ride.** Bearer-bypass branch checked Authorization
header presence only — an attacker page could mint `Authorization:
Bearer junk` and ride the still-attached session cookie. Bypass now
requires bearer AND no cookie. Drive the cookie-name comparison off
`SESSION_COOKIE_NAME` and split on `=` (not prefix).
2. **`require_can_edit` admin path open to bearer.** Now reads admin
authority off an `Option<CurrentSessionUser>` extractor — None on
bearer-only requests. Owner-match path still accepts bearer; admin
path is cookie-gated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`require_can_edit` matched `Some(owner) != caller → Forbidden` but let
`None` through. Every crawler row (`repo::crawler::upsert_manga`) has
NULL `uploaded_by`, so any signed-in user could PATCH /api/v1/mangas/<id>
and rewrite the catalog — `update`, `put_cover`, and `delete_cover`
were all in scope. With self-registration on by default, the path was:
register → mass-edit catalog + delete cover blobs from storage.
The original carve-out at the comment said "Once an admin role lands the
NULL case can flip to admin-only." The admin role landed in migration
0018; flipping it now. New rule:
* `Some(owner)` and owner == caller → ok (unchanged)
* `Some(_)` and caller is admin → ok (admin moderation)
* `Some(_)` otherwise → 403 (unchanged)
* `None` and caller is admin → ok (NEW — operator can curate)
* `None` otherwise → 403 (was: ok — the bug)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four high/medium findings from the audit, plus their tests:
1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
`AnalysisSettings::endpoint` validated only with `Url::parse`. A
hostile or CSRF-able admin could repoint at `169.254.169.254`
(cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
— and the vision worker bearer-attaches an env-managed API key to
every call. Extract `ensure_public_target` from `safety::is_safe_url`
(allowlist-free public-host check: scheme + private-IP literal +
localhost) and wire it into both fields. Docker DNS names
(`mangalord-vision`) keep passing because they're hostnames, not
IP literals. The analysis check is gated on `enabled=true` so the
dev-default `localhost:8000` stays usable until the worker is
actually turned on (toggling enabled=true later re-runs the gate).
2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
silently skipped the entire CSRF check, so an operator forgetting
the env var shipped an unguarded admin surface. Cookie-auth POSTs
now fail-closed when the allowlist is empty AND when neither
`Origin` nor `Referer` accompanies the request. Two carve-outs:
`Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
and requests with NO session cookie at all bypass to let the auth
extractor return a clean 401 instead of a confusing 403.
3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
would exfiltrate page-image bytes + the bearer key through an
upstream proxy. Same for the readiness probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.
Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.
Closes the Phase-1 observability set (audit log · health checks · trends).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.
GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).
Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.
Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
