Three 0.87.9 follow-ups:
1. hooks.server.ts: two new tests covering the
already-aborted-at-entry and mid-flight client-disconnect branches
of the abort chain. Mutation-confirmed.
2. lib/api/admin.ts: two new tests pinning the
getAnalysisMangaCoverage(init?.signal) pass-through and the
AbortError-rejection propagation the dashboard's debounce-race
handler relies on. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.
Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.
Closes the Phase-1 observability set (audit log · health checks · trends).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.
GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).
Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.
Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Analysis section now reflects worker progress in real time:
- Opens an EventSource on the live stream while mounted; a "Live" pill
reflects connection state and reconnects on drop (probes after repeated
failures so a lost session logs out).
- Applies events incrementally: enqueued marks in-scope loaded pages
queued; started flips a chip to a pulsing "analyzing"; completed flips
it green and live-bumps the manga + chapter coverage badges (guarded so
no double count); failed flips it red. A compact activity ticker shows
the last few events.
- Server numbers stay authoritative: re-loading the overview or a page
grid clears the matching live overrides.
API client: analysisStatusStreamUrl() + AnalysisEvent type.
Tests: vitest for the stream URL; Playwright asserts SSE frames flow into
the ticker (existing coverage/drill/queue specs get a default quiet
stream). svelte-check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks /admin/analysis from a blind enqueue form into a coverage browser:
- Loads a paginated overview of mangas with a CoverageBadge
(Full/Partial/None, analyzed/total) per row; debounced search filters it.
- Drill manga → chapters (coverage badge each) → page grid (chips colored
done/queued/failed/none) with a legend.
- Click a page chip → Modal showing the full result: status, model, time,
NSFW + content-warning chips, scene description, tags, and OCR lines
(labeled by kind). Unanalyzed/failed pages get a "Queue this page"
action (force re-analyze).
- Keeps "Queue all" + per-manga/chapter enqueue and the include-analyzed
toggle; enqueue refreshes the open chapter's grid so queued pages show.
API client: getAnalysisMangaCoverage / ChapterCoverage / ChapterPages /
PageDetail + types. New CoverageBadge component.
Tests: vitest for the four clients; Playwright for overview+badges+queue-all,
drill+inspect, and queue-from-detail. svelte-check + build clean; 266 vitest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the scoped re-enqueue and per-page force-analyze in the UI:
- api/admin: reenqueueAnalysis({onlyUnanalyzed, mangaId, chapterId}) and
analyzePage(pageId).
- Reader PageContextMenu gains an admin-only "Queue for analysis" item
(canAnalyze/onAnalyzePage/analyzeState props) wired to the force-analyze
endpoint with inline busy/done/error feedback; reset per page.
- New /admin/analysis dashboard section + nav tab: scope selector
(whole library / one manga / one chapter), id input, and an
"include already-analyzed" toggle (default off), with busy/notice/error.
Tests: vitest for the two clients; Playwright for the context-menu action
(admin vs non-admin) and the admin section (scope + include toggle +
validation). svelte-check + build clean; 262 vitest, 10 new e2e green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tick cover-backfill pass to the crawler daemon so mangas whose
cover download failed on first attempt get retried — the metadata pass's
early-stop optimisation otherwise prevents the walk from revisiting them.
Adds admin-only POST /admin/mangas/:id/resync and POST /admin/chapters/:id/resync
that refetch metadata + cover (or chapter content with force_refetch) from the
crawler source synchronously and return the refreshed row. Surfaced in the
UI as "Force resync" buttons on the manga detail and reader pages,
admin-only via session.user.is_admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with the ALLOW_SELF_REGISTER toggle from 0.42.0: admins can mint
accounts regardless of the toggle state, so a closed-membership
deployment still has a working enrollment path. The endpoint accepts
{ username, password, is_admin? } so admins can mint co-admins in one
call (avoiding a separate promote + extra audit row for the common
"invite a co-admin" flow).
Implementation:
- POST /api/v1/admin/users guarded by RequireAdmin
- Reuses validate_username / validate_password from api::auth (made
pub(crate)) so the admin path can never produce an account self-
register would reject and vice versa
- repo::user::admin_create_user wraps INSERT + admin_audit insert in
a single tx — same "audit reflects what committed" semantics as the
existing admin_safe_* fns
- Audit row: action="create_user", payload={username, is_admin}
Frontend:
- createAdminUser() in lib/api/admin.ts
- /admin/users grows a collapsible "Create user" form above the table
(username, password, "Make admin" checkbox). Errors surface inline;
the list reloads on success.
Backend tests: 7 new, including the headline
`create_user_works_even_when_self_register_disabled` that pins the
admin-create path is NOT gated by the public toggle.
Addresses the security-audit findings on top of the admin feature stack:
M1: /admin/mangas/:id/chapters now paginates (default limit 200, max 500).
A long-runner with thousands of chapters would otherwise produce a multi-MB
response with that many scalar subqueries per row — admin-only but a real
stall risk on one expand-click. Adds explicit pagination tests for the cap
and offset; frontend renders a "Showing first N of M" hint when the cap
clips the result.
L1: repo::user::set_is_admin renamed to set_is_admin_unchecked with a
doc-comment pointing at admin_safe_set_is_admin for production use. The
short name was a footgun — a future contributor reaching for it would
silently bypass self-protection, the last-admin invariant, and the audit
log. Used only by integration-test setup; production code goes through
the admin_safe_* paths.
CSRF posture: build_session_cookie carries a comment that the
SameSite=Lax default is the project's CSRF defense for state-changing
mutations and breaks the instant anyone adds a side-effecting GET under
/admin/*. Spells out what to do then (Strict + explicit token check).
Test counts: 43 backend admin tests + 12 vitest admin tests all green;
svelte-check 0/0 across 446 files.
Adds the SvelteKit /admin route tree backed by the admin endpoints
landed in PR 1-4. Pages: Overview (alerts + summary cards), Users
(list / promote-demote / delete), Mangas (list with sync state +
expandable per-chapter state), System (live disk/mem/cpu bars,
refreshing every 5s).
Security model: the backend's RequireAdmin extractor is the actual
boundary. /admin/+layout.ts calls getSystemStats() at load and
translates the response — 401 → redirect to /login, 403 → throw
SvelteKit error(403) which renders the framework error page. The
header's "Admin" link is hidden unless `session.user?.is_admin`,
but that's UX only.
Carries `is_admin: boolean` through to the frontend User TS type so
the header check works and so admin tables can show role per row.
Vitest covers lib/api/admin.ts (10 tests: list/delete/PATCH for
users, sync-state filter for mangas, nested chapter route, system
disk-nullable case). Playwright is intentionally deferred until the
routes stabilise — admin UI is operator-only and changes shape often
in v0.
