Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CancellableLoader.run() previously swallowed AbortError but propagated
every other rejection. A predecessor whose fetch survived its abort
(timing race, or fn that ignores signal) and then 5xx'd would bubble
the error into the caller's catch, clearing the successor's spinner
and writing its `error` field — exactly the flicker the loader was
supposed to prevent.
Strengthen the contract: a call that no longer owns this.current
returns null regardless of how it died. Symmetric with the existing
late-resolve suppression at the success path. The loadCoverage catch
in admin/analysis/+page.svelte now only fires when this call genuinely
failed AND still owns the loader — so clearing the spinner there is
sound.
Three new unit tests pin all three contract corners: a superseded
predecessor's non-Abort rejection resolves to null; a standalone call
with no successor still throws normally; the AbortError swallow on a
still-owning call (synthetic AbortError from fn with no successor or
cancel()) remains protected, since the new supersede check would
otherwise have eaten every existing AbortError-path test and left
that branch as silently-removable dead code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.16 followup. Extract the four debounce-race invariants
(predecessor abort, late-result suppression, AbortError swallow,
ownership-aware finally) from `loadCoverage` into a reusable
`CancellableLoader` helper at $lib/util/cancellable, with 9 unit
tests covering each. Page rewrite uses the helper.
Also corrects: hooks.server.test.ts (added Node-fetch rejection
shape for already-aborted entry; removed redundant tick); admin.test.ts
(AbortError-propagation now uses pre-aborted signal + mockRejectedValueOnce
mirroring real Node fetch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three frontend findings (medium):
1. **`hooks.server.ts` didn't chain client-disconnect into the upstream
fetch.** Closing an admin tab left the backend's `broadcast::Receiver`
+ spawned tokio task running until next backend restart. Listen for
`event.request.signal.abort` and forward to the timeout controller.
2. **`admin/analysis/+page.svelte` missing visibility/pagehide/pageshow
handlers.** Mobile Safari throttles SSE in background tabs; bfcache
leaves a dead connection on back-nav. Apply the close/open-stream
pattern from `admin/crawler/+page.svelte`.
3. **`loadCoverage` had no AbortController.** Debounced search
keystrokes raced. Per-loader controller plumbed through
`getAnalysisMangaCoverage`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.
GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).
Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.
Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The admin overview dashboard polls /admin/overview every 30s. Its
`manga_stats` aggregate evaluated MANGA_SYNC_STATE_CASE over the WHOLE
library (14.5k mangas) with no LIMIT, and that CASE runs a correlated
EXISTS over crawler_jobs. With no index on the `sync_chapter_list`
manga_id path, Postgres seqscanned all in-flight jobs *per manga* —
O(mangas x jobs). The disabled-analysis backlog (9.5k pending
`analyze_page` rows that can never match the sync-kind filter) inflated
every scan, so each call took 6-7 min. The 30s poll stacked ~10 of them
concurrently → load 11, Postgres at ~100%.
EXPLAIN ANALYZE on the live DB: the rewrite drops the query from 6-7 min
to ~1.5s (per-manga crawler_jobs seqscan → single index-backed pass).
Fixes:
- Rewrite `manga_stats` to collect the in-flight manga-id set in ONE pass
over the sync jobs (index-backed), then classify via a hash semi-join —
O(mangas + jobs), immune to the analyze_page backlog size.
- Add partial index crawler_jobs_sync_chapter_list_manga_idx (migration
0029) covering the sync_chapter_list manga_id path; mirrors the existing
sync_manga index (0020). Also speeds the per-row Mangas tab listing.
- statement_timeout backstop (5s) on the aggregate so a pathological plan
can never pin a backend for minutes again.
- Single-flight + 10s TTL cache on the /admin/overview handler so
concurrent pollers coalesce instead of stacking.
- Slow the dashboard poll 30s -> 60s (server-cached now anyway).
The in_progress rule in the rewrite is kept in lockstep with the first
arm of MANGA_SYNC_STATE_CASE (documented in both places).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Analysis section now reflects worker progress in real time:
- Opens an EventSource on the live stream while mounted; a "Live" pill
reflects connection state and reconnects on drop (probes after repeated
failures so a lost session logs out).
- Applies events incrementally: enqueued marks in-scope loaded pages
queued; started flips a chip to a pulsing "analyzing"; completed flips
it green and live-bumps the manga + chapter coverage badges (guarded so
no double count); failed flips it red. A compact activity ticker shows
the last few events.
- Server numbers stay authoritative: re-loading the overview or a page
grid clears the matching live overrides.
API client: analysisStatusStreamUrl() + AnalysisEvent type.
Tests: vitest for the stream URL; Playwright asserts SSE frames flow into
the ticker (existing coverage/drill/queue specs get a default quiet
stream). svelte-check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks /admin/analysis from a blind enqueue form into a coverage browser:
- Loads a paginated overview of mangas with a CoverageBadge
(Full/Partial/None, analyzed/total) per row; debounced search filters it.
- Drill manga → chapters (coverage badge each) → page grid (chips colored
done/queued/failed/none) with a legend.
- Click a page chip → Modal showing the full result: status, model, time,
NSFW + content-warning chips, scene description, tags, and OCR lines
(labeled by kind). Unanalyzed/failed pages get a "Queue this page"
action (force re-analyze).
- Keeps "Queue all" + per-manga/chapter enqueue and the include-analyzed
toggle; enqueue refreshes the open chapter's grid so queued pages show.
API client: getAnalysisMangaCoverage / ChapterCoverage / ChapterPages /
PageDetail + types. New CoverageBadge component.
Tests: vitest for the four clients; Playwright for overview+badges+queue-all,
drill+inspect, and queue-from-detail. svelte-check + build clean; 266 vitest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the raw manga-id / chapter-id UUID inputs with a real picker:
- Search a manga (debounced, reuses listAdminMangas) → result rows show
title + chapter count; "Queue manga" enqueues the whole manga.
- Expand a result to load its chapters (listAdminChapters) and "Queue
chapter" enqueues a single chapter.
- Keeps the "Queue all (whole library)" action and a single global
"include already-analyzed" toggle applied to every action; per-action
busy state + a result message naming what was queued.
Tests: rewrote the Playwright spec to drive the search → manga/chapter
flow (whole library, search+queue manga, expand+queue chapter with
include-analyzed). svelte-check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the scoped re-enqueue and per-page force-analyze in the UI:
- api/admin: reenqueueAnalysis({onlyUnanalyzed, mangaId, chapterId}) and
analyzePage(pageId).
- Reader PageContextMenu gains an admin-only "Queue for analysis" item
(canAnalyze/onAnalyzePage/analyzeState props) wired to the force-analyze
endpoint with inline busy/done/error feedback; reset per page.
- New /admin/analysis dashboard section + nav tab: scope selector
(whole library / one manga / one chapter), id input, and an
"include already-analyzed" toggle (default off), with busy/notice/error.
Tests: vitest for the two clients; Playwright for the context-menu action
(admin vs non-admin) and the admin section (scope + include toggle +
validation). svelte-check + build clean; 262 vitest, 10 new e2e green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundle of small UI/UX fixes plus a build hygiene tweak.
* List pagination — Home (`/`) and `/authors/[id]` silently capped at
the backend default of 50 with no UI to advance. New reusable
`Pager.svelte` (Prev/Next + numbered with ellipsis), URL-synced
`?page=N`, and filter/search/sort reset to page 1 so users aren't
stranded on an out-of-range page. Count label now shows a range
("Showing 51–100 of 237").
* Stale page title — Pages without a `<svelte:head><title>` left the
document title at whatever the last manga / author / collection page
set it to. Move static-route titles into a route-id → title map in
the root layout and invert every dynamic title to brand-first
(`Mangalord | {X}`) for consistency.
* Admin filter bar — `/admin/mangas` search input had `flex: 1` and
ballooned across the row, shoving the sync-state select + Search
button to the far right. Cap at 24rem, vertical-align the row, and
promote the previously aria-only "Sync state" label to visible text.
* Build hygiene — `backend/target` had grown to 68 GiB. Cleaned and
added `[profile.dev] debug = "line-tables-only"` (and `[profile.test]`
too) to cut future dev builds by ~50–70% while keeping line numbers
in backtraces.
Also: configure vitest to resolve Svelte's browser entry so
`@testing-library/svelte` can mount components in jsdom — needed for
the new `Pager.svelte.test.ts`.
Bump 0.48.0 -> 0.49.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with the ALLOW_SELF_REGISTER toggle from 0.42.0: admins can mint
accounts regardless of the toggle state, so a closed-membership
deployment still has a working enrollment path. The endpoint accepts
{ username, password, is_admin? } so admins can mint co-admins in one
call (avoiding a separate promote + extra audit row for the common
"invite a co-admin" flow).
Implementation:
- POST /api/v1/admin/users guarded by RequireAdmin
- Reuses validate_username / validate_password from api::auth (made
pub(crate)) so the admin path can never produce an account self-
register would reject and vice versa
- repo::user::admin_create_user wraps INSERT + admin_audit insert in
a single tx — same "audit reflects what committed" semantics as the
existing admin_safe_* fns
- Audit row: action="create_user", payload={username, is_admin}
Frontend:
- createAdminUser() in lib/api/admin.ts
- /admin/users grows a collapsible "Create user" form above the table
(username, password, "Make admin" checkbox). Errors surface inline;
the list reloads on success.
Backend tests: 7 new, including the headline
`create_user_works_even_when_self_register_disabled` that pins the
admin-create path is NOT gated by the public toggle.
Addresses the security-audit findings on top of the admin feature stack:
M1: /admin/mangas/:id/chapters now paginates (default limit 200, max 500).
A long-runner with thousands of chapters would otherwise produce a multi-MB
response with that many scalar subqueries per row — admin-only but a real
stall risk on one expand-click. Adds explicit pagination tests for the cap
and offset; frontend renders a "Showing first N of M" hint when the cap
clips the result.
L1: repo::user::set_is_admin renamed to set_is_admin_unchecked with a
doc-comment pointing at admin_safe_set_is_admin for production use. The
short name was a footgun — a future contributor reaching for it would
silently bypass self-protection, the last-admin invariant, and the audit
log. Used only by integration-test setup; production code goes through
the admin_safe_* paths.
CSRF posture: build_session_cookie carries a comment that the
SameSite=Lax default is the project's CSRF defense for state-changing
mutations and breaks the instant anyone adds a side-effecting GET under
/admin/*. Spells out what to do then (Strict + explicit token check).
Test counts: 43 backend admin tests + 12 vitest admin tests all green;
svelte-check 0/0 across 446 files.
Adds the SvelteKit /admin route tree backed by the admin endpoints
landed in PR 1-4. Pages: Overview (alerts + summary cards), Users
(list / promote-demote / delete), Mangas (list with sync state +
expandable per-chapter state), System (live disk/mem/cpu bars,
refreshing every 5s).
Security model: the backend's RequireAdmin extractor is the actual
boundary. /admin/+layout.ts calls getSystemStats() at load and
translates the response — 401 → redirect to /login, 403 → throw
SvelteKit error(403) which renders the framework error page. The
header's "Admin" link is hidden unless `session.user?.is_admin`,
but that's UX only.
Carries `is_admin: boolean` through to the frontend User TS type so
the header check works and so admin tables can show role per row.
Vitest covers lib/api/admin.ts (10 tests: list/delete/PATCH for
users, sync-state filter for mangas, nested chapter route, system
disk-nullable case). Playwright is intentionally deferred until the
routes stabilise — admin UI is operator-only and changes shape often
in v0.
