Compare commits

...

24 Commits

Author SHA1 Message Date
MechaCat02
030b27754b feat(api): admin-initiated user creation via POST /admin/users (0.43.0)
Some checks failed
deploy / test-backend (push) Failing after 8s
deploy / test-frontend (push) Failing after 38s
deploy / build-and-push (push) Has been skipped
deploy / deploy (push) Has been skipped
Pairs with the ALLOW_SELF_REGISTER toggle from 0.42.0: admins can mint
accounts regardless of the toggle state, so a closed-membership
deployment still has a working enrollment path. The endpoint accepts
{ username, password, is_admin? } so admins can mint co-admins in one
call (avoiding a separate promote + extra audit row for the common
"invite a co-admin" flow).

Implementation:
- POST /api/v1/admin/users guarded by RequireAdmin
- Reuses validate_username / validate_password from api::auth (made
  pub(crate)) so the admin path can never produce an account self-
  register would reject and vice versa
- repo::user::admin_create_user wraps INSERT + admin_audit insert in
  a single tx — same "audit reflects what committed" semantics as the
  existing admin_safe_* fns
- Audit row: action="create_user", payload={username, is_admin}

Frontend:
- createAdminUser() in lib/api/admin.ts
- /admin/users grows a collapsible "Create user" form above the table
  (username, password, "Make admin" checkbox). Errors surface inline;
  the list reloads on success.

Backend tests: 7 new, including the headline
`create_user_works_even_when_self_register_disabled` that pins the
admin-create path is NOT gated by the public toggle.
2026-05-31 14:00:31 +02:00
MechaCat02
2f47faa11c feat(auth): ALLOW_SELF_REGISTER toggle + public /auth/config endpoint (0.42.0)
Lets operators run a closed-membership deployment by setting
ALLOW_SELF_REGISTER=false (default true, so existing deploys are
unaffected). When off, POST /auth/register returns 403 forbidden. The
rate-limit token is consumed BEFORE the disabled check so the timing
doesn't distinguish enabled-but-rejected from disabled — closes the
toggle-state probe channel.

New public GET /auth/config returns { self_register_enabled: bool }
so the frontend can render its register affordances correctly
without conflating "disabled" with "rate-limited" (which a probe
attempt would).

Frontend: a lightweight reactive `authConfig` store loads the flag
once on root-layout mount (and again on /register direct navigation,
which bypasses the layout's onMount). Header hides the Register link
when the toggle is off; /register renders a "self-registration is
disabled — ask an administrator" notice instead of the form.

Admin-create endpoint that pairs with this toggle is intentionally
not in this PR — it lands as the next branch (feat/admin-user-create).
The toggle alone is independently useful for deployments that want
to lock down enrollment without yet wiring an admin UI.
2026-05-31 13:56:18 +02:00
MechaCat02
6dd21451a8 chore: sync Cargo.lock for 0.41.2 2026-05-30 22:26:24 +02:00
MechaCat02
f6728dc71a fix(admin): security-audit findings — paginate chapters, lock down unchecked helper (0.41.2)
Addresses the security-audit findings on top of the admin feature stack:

M1: /admin/mangas/:id/chapters now paginates (default limit 200, max 500).
A long-runner with thousands of chapters would otherwise produce a multi-MB
response with that many scalar subqueries per row — admin-only but a real
stall risk on one expand-click. Adds explicit pagination tests for the cap
and offset; frontend renders a "Showing first N of M" hint when the cap
clips the result.

L1: repo::user::set_is_admin renamed to set_is_admin_unchecked with a
doc-comment pointing at admin_safe_set_is_admin for production use. The
short name was a footgun — a future contributor reaching for it would
silently bypass self-protection, the last-admin invariant, and the audit
log. Used only by integration-test setup; production code goes through
the admin_safe_* paths.

CSRF posture: build_session_cookie carries a comment that the
SameSite=Lax default is the project's CSRF defense for state-changing
mutations and breaks the instant anyone adds a side-effecting GET under
/admin/*. Spells out what to do then (Strict + explicit token check).

Test counts: 43 backend admin tests + 12 vitest admin tests all green;
svelte-check 0/0 across 446 files.
2026-05-30 22:23:55 +02:00
MechaCat02
aa2159ca06 fix(admin): three review findings — audit no-op, 404, chapter priority (0.41.1)
- admin_safe_set_is_admin: short-circuit when target.is_admin == value,
  before writing audit. PATCH {is_admin: true} on someone already admin
  previously wrote a misleading "promote_user" row even though the UPDATE
  was a no-op.

- list_chapters (/admin/mangas/:id/chapters): explicit exists() check on
  manga_id, returns 404 instead of 200 [] for a typo'd / deleted manga.

- ChapterSyncState priority: the Failed branch now requires page_count = 0,
  so a chapter with pages on disk AND a historical dead job (from a
  re-download attempt that crashed) stays Synced. The old order
  contradicted Synced's documented "downloaded at some point" contract.
  Doc comments updated alongside the SQL.

Three new regression tests pin the behaviour.
2026-05-30 21:58:15 +02:00
MechaCat02
b434c9b68d feat(frontend): /admin dashboard with users/mangas/system views (0.41.0)
Adds the SvelteKit /admin route tree backed by the admin endpoints
landed in PR 1-4. Pages: Overview (alerts + summary cards), Users
(list / promote-demote / delete), Mangas (list with sync state +
expandable per-chapter state), System (live disk/mem/cpu bars,
refreshing every 5s).

Security model: the backend's RequireAdmin extractor is the actual
boundary. /admin/+layout.ts calls getSystemStats() at load and
translates the response — 401 → redirect to /login, 403 → throw
SvelteKit error(403) which renders the framework error page. The
header's "Admin" link is hidden unless `session.user?.is_admin`,
but that's UX only.

Carries `is_admin: boolean` through to the frontend User TS type so
the header check works and so admin tables can show role per row.

Vitest covers lib/api/admin.ts (10 tests: list/delete/PATCH for
users, sync-state filter for mangas, nested chapter route, system
disk-nullable case). Playwright is intentionally deferred until the
routes stabilise — admin UI is operator-only and changes shape often
in v0.
2026-05-30 21:49:39 +02:00
MechaCat02
cc4ec76d17 feat(api): admin system metrics endpoint with disk/mem/cpu alerts (0.40.0)
Adds GET /api/v1/admin/system returning disk (scoped to storage_dir
via statvfs), memory, CPU, and a server-side alerts array that fires
at >90% disk or memory.

Disk uses nix::sys::statvfs directly rather than sysinfo's Disks API
to avoid mountpoint-matching gymnastics for the storage_dir. A new
`Storage::local_root() -> Option<&Path>` trait method exposes the
root; the default returns None so a future S3Storage gets `disk:
null` in the response instead of fabricated numbers.

CPU is sampled inline (refresh → 250ms sleep → refresh → read) so the
endpoint adds 250ms of latency per call. No background-cache yet —
admin traffic is low-volume and the moving parts aren't worth it
until polling shows up.

Alerts are evaluated server-side so the frontend can render them
without re-implementing the thresholds.
2026-05-30 21:45:06 +02:00
MechaCat02
bf7c9b5c2a feat(api): admin manga/chapter overview with derived sync state (0.39.0)
Adds GET /api/v1/admin/mangas and /admin/mangas/:id/chapters guarded by
RequireAdmin. Sync state is computed at query time from the existing
crawler signals (manga_sources / chapter_sources / crawler_jobs) — no
new state column is persisted, so the crawler stays the single writer
of these signals.

Per-manga priority: InProgress (in-flight sync_manga or
sync_chapter_list job) > Dropped (all source rows soft-dropped) >
Synced (default; covers user-uploaded mangas with zero source rows).

Per-chapter priority: Downloading (in-flight sync_chapter_content) >
Dropped (all source rows soft-dropped) > Failed (most-recent terminal
job is dead) > NotDownloaded (page_count = 0) > Synced. The Failed
check sits ABOVE NotDownloaded so the more informative "we tried and
it died" state wins over "we never got around to it" — see the
priority comment in repo/admin_view.rs.

Migration 0020 adds a partial index on
crawler_jobs((payload->>'source_manga_key')) for the one job kind
(sync_manga) whose payload doesn't carry manga_id directly — without it
the in-flight detection for a manga falls back to a seqscan over the
job table.
2026-05-30 21:41:09 +02:00
MechaCat02
0b2018ceca feat(api): admin user management endpoints with audit log (0.38.0)
Adds /api/v1/admin/users list / DELETE / PATCH guarded by RequireAdmin,
plus the audit-log substrate every future destructive admin endpoint
will reuse.

Safety properties:
- Cannot self-delete or self-demote (409 conflict, message calls out
  "yourself" so the UI can render an explanation).
- Cannot remove the last admin via either DELETE or demote. The check
  takes pg_advisory_xact_lock(ADMIN_INVARIANT_LOCK_KEY) and re-counts
  admins inside the same tx, closing the parallel-demote race that a
  bare "if count > 1" check would let through. The HTTP-serial path to
  this guard is structurally unreachable (the actor would have to be
  the lone admin demoting themselves, which the self-guard fires on
  first); the parallel race test exercises it via repo calls.

Audit log (admin_audit table) records the action inside the same tx
as the action itself, so a rolled-back action never leaves an orphan
audit row. actor_user_id is ON DELETE SET NULL so the log outlives a
later-deleted admin. target_id is not a FK because future audit kinds
will target non-user rows.
2026-05-30 21:35:35 +02:00
MechaCat02
ab8b7acc34 feat(auth): admin role with cookie-only RequireAdmin extractor (0.37.0)
Adds an `is_admin` flag on users plus the substrate every later PR in the
admin feature builds on:

- migration 0018 adds the column with default false
- `repo::user::bootstrap_admin` creates or promotes the user named by
  `ADMIN_USERNAME` at startup, hashing `ADMIN_PASSWORD` only when the row
  is new — never overwriting an existing hash, so an operator can rotate
  the admin password via the UI without env-var conflict
- `CurrentSessionUser` extractor accepts only the session cookie;
  `RequireAdmin` composes over it and additionally requires
  `user.is_admin`. Bearer tokens are intentionally excluded so an
  admin's bot token never inherits admin authority (privilege-escalation
  surface that bites every "API keys reuse user perms" auth design)
- demotion is instant: `RequireAdmin` re-reads the user row each request

`/api/v1/auth/me` now exposes `is_admin`; no other response embeds
`User`, so no privacy fanout to audit.
2026-05-30 21:26:26 +02:00
MechaCat02
9925f54695 fix(crawler): narrow browser-dead heuristic to typed downcasts (0.36.7)
anyhow_looks_browser_dead substring-matched any chain message
containing channel / connection / websocket / transport / closed /
nav timeout. Real chromium failures hit those words, but so do
reqwest TCP-reset errors during CDN image downloads, sqlx pool-
timeout errors, and any number of non-browser failures — each of
which triggered a wasted chromium relaunch + session-probe re-run
against the catalog's rate-limit budget.

Drop the substring pass. Walk the chain looking only for typed
NavError (flagged via is_likely_browser_dead) or CdpError. Every
place we feed a chromium error into anyhow goes through one of
those types, so the typed downcasts cover the real cases without
the false-positive surface.

NavError::is_likely_browser_dead also drops its own substring
check on Cdp(e); any CdpError surfacing at the navigation layer
means the chromium-facing channel is the failing layer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 20:41:59 +02:00
MechaCat02
eaa5afda50 fix(crawler): skip sync when empty chapters + prior > 0 (0.36.6)
The wait_for_selector wait in 0.36.2 narrows the partial-render race
window but doesn't close it: a render that takes longer than
SELECTOR_TIMEOUT (10s) still hands an empty Vec to sync_manga_chapters,
and the soft-drop branch flips every existing chapter to dropped_at.
The next tick recovers but a manga's reader briefly stops working in
between.

Close it at the pipeline level. Between fetch_manga and the upsert/
sync, if the parsed chapter list is empty and the prior live count
for (source_id, source_manga_key) is > 0, treat the fetch as a
transient failure: log, bump mangas_failed, skip upsert + sync + the
seen.insert so a later batch / tick retries. Brand-new mangas with
genuinely zero chapters (prior == 0) pass through unchanged.

New repo helper repo::crawler::live_chapter_count_for_source_manga
joins chapters → chapter_sources → manga_sources with dropped_at IS
NULL — same lockstep as dispatch_target and the enqueue queries.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 20:17:42 +02:00
MechaCat02
5c04b0532b fix(crawler): panic-isolate the cron tick body (0.36.5)
Worker dispatch was already wrapped in AssertUnwindSafe(...)
.catch_unwind() — a panicking handler ack's the job failed and the
worker keeps going. The cron tick had no such guard: a panic in
metadata.run, enqueue_bookmarked_pending, reap_done, or
write_last_tick would kill the cron task. The JoinSet would drop it,
workers would keep running, and no future metadata pass would ever
fire until daemon restart.

Wrap the tick body (between advisory-lock acquire and unlock) in the
same AssertUnwindSafe(...).catch_unwind() pattern. The unlock and
connection drop run unconditionally so a panicked tick doesn't leave
the lock held for another replica.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 20:08:11 +02:00
MechaCat02
655ea42731 fix(crawler): scope dispatch_target to live sources, newest first (0.36.4)
The chapter dispatcher's URL resolver had no dropped_at filter and no
ORDER BY — a chapter whose only chapter_sources row had been soft-
dropped was still dispatched against the stale URL, eating retry
budget on guaranteed transients. With multiple live sources the LIMIT
1 winner was nondeterministic.

Add `AND cs.dropped_at IS NULL` and `ORDER BY cs.last_seen_at DESC`
to dispatch_target, bringing it in lockstep with the enqueue queries
in pipeline.rs that already filter on dropped_at. Returns None when
all sources are dropped — callers in daemon.rs already treat None
as "ack the job, skip the work."

Tests in tests/repo_chapter.rs cover the three branches (freshest
live wins, dropped sources skipped, all-dropped returns None).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 20:03:45 +02:00
MechaCat02
70e8a7895c fix(crawler): relaunch chromium on CDP / nav-timeout errors (0.36.3)
BrowserManager only re-launched chromium when the cached handle was
None. A crash mid-pass left the handle Some pointing at a dead
process — every subsequent acquire returned the zombie Browser, and
every nav cascaded CDP errors until the idle reaper fired.

Add BrowserManager::invalidate(): take the inner mutex, drop the
handle (closing it if present), and signal the next acquire to
relaunch. Idempotent — invalidating an empty handle is a no-op.

Wire detection via NavError::is_likely_browser_dead and a
chain-walking anyhow_looks_browser_dead helper: substring-match
common channel/connection/transport/WebSocket markers and surface
NavError::Timeout as "presumed dead." Apply at both error
boundaries — RealChapterDispatcher::dispatch and
RealMetadataPass::run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 18:39:19 +02:00
MechaCat02
8e0b638e3f fix(crawler): wait for page marker instead of fixed 1s sleep (0.36.2)
A chromium snapshot taken between the wrapper-render and row-render
phases let parse_chapter_list return Ok(vec![]) for a manga that
actually has chapters — the soft-drop branch in sync_manga_chapters
then flipped every existing chapter to dropped_at.

Add wait_for_selector to crawler::nav. navigate() now takes a CSS
marker matching the most-specific element the downstream parser will
look for (one of LIST_PAGE_MARKER / DETAIL_PAGE_CHAPTERS_MARKER /
DETAIL_PAGE_LAYOUT_MARKER). The wait is best-effort and capped by
SELECTOR_TIMEOUT (10s); a legitimately empty page can still pass
through because the parser's #chapter_table sentinel and the
universal broken-page body check stay in force.

Same pattern wired at the reader nav (a#pic_container) and probe
nav (#logo), replacing the implicit assumption that the post-load
JS had finished within 1 second.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 18:29:38 +02:00
MechaCat02
e2bd1462ba fix(crawler): wrap wait_for_navigation in 30s timeout (0.36.1)
A hung TLS handshake or a page that never fires load could wedge a
worker (or the cron metadata pass) indefinitely — chromiumoxide
imposes no navigation timeout of its own.

New crawler::nav::wait_for_nav caps each navigation at NAV_TIMEOUT
(30s) and returns a typed NavError so timeouts surface as transient
(retryable) errors. Wired at the three navigation sites:
- source::target::navigate (catalog/detail/pagination)
- content::sync_chapter_content (chapter reader)
- session::fetch_probe_html (session probe)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 18:10:51 +02:00
MechaCat02
9f56f283d4 feat(crawler): single-mode walker gated by recovery flag (0.36.0)
Collapses the crawler to a single newest-first walker and replaces the
N-consecutive-unchanged streak with a per-manga rule: stop on the first
manga where metadata is Unchanged AND chapter sync reports zero new
chapters. The early stop is gated by a per-source recovery flag stored
in `crawler_state` — set to `false` when a run starts, back to `true`
only on a clean exit (end-of-walk or intentional stop). A crashed run
leaves the flag `false` automatically (no shutdown code runs), so the
next tick walks the full catalog instead of bailing at the first
caught-up manga.

This means a crashed mid-walk run self-heals on the next tick: the
flag stays `false`, the next walk visits every page (recovering
anything the crash missed past its crash point), and steady state
resumes once the recovery sweep reaches end-of-walk.

Removed:
- DiscoverMode enum, Backfill mode, the boundary re-check +
  displaced-refs machinery in TargetSourceWalker.
- Drop-pass (mark_dropped_mangas) and seed-completion plumbing
  (mark_seed_completed / seed_completed_at). The recovery flag
  subsumes the seed-completion signal; drop detection was explicitly
  opted out.
- JobPayload::Discover (no production callers).
- CRAWLER_MODE / CRAWLER_INCREMENTAL_STOP_AFTER env vars and the
  CrawlerModePref config type.

`should_mark_clean_exit(walked_to_completion, hit_stop_condition)`
encodes the clean-exit truth table in its signature — `hit_limit` is
deliberately absent so a future edit cannot accidentally count a
caller-imposed cap as a clean exit.

Net -501 lines, 261 backend tests passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 23:49:28 +02:00
MechaCat02
33f7e19077 fix(crawler): serialize sync_manga_chapters per-manga (0.35.6)
Two concurrent calls of sync_manga_chapters for the same manga both
read seen_keys, both run the drop UPDATE filtered on `NOT (key = ANY
$3)`, and the later commit can soft-drop a chapter the earlier had
just inserted (lost-update under MVCC). Today the cron tick is the
only caller and the daemon-level advisory lock keeps it single-flight,
but that lock is held on one pool connection and doesn't actually
serialize the *function*: any future caller (bookmark hook,
admin-triggered re-sync, parallel worker) would race against the cron.

Add `pg_advisory_xact_lock(hashtextextended(manga_id::text, 0))` at
the start of the transaction. Auto-releases on commit/rollback so a
panic mid-call can't strand the lock. Lock keyed per-manga so calls
for different mangas still parallelize.

Test sync_chapters_serializes_concurrent_calls_for_same_manga spawns
two tokio tasks calling the function concurrently with overlapping
chapter lists and asserts every chapter survives.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:45:01 +02:00
MechaCat02
c6bb9160e3 fix(crawler): scope chapter_sources lookup per-manga (0.35.5)
chapter_sources's PRIMARY KEY was (source_id, source_chapter_key) and
the lookup in sync_manga_chapters didn't constrain by manga_id, so a
source whose chapter slugs aren't globally unique (e.g. "chapter-1"
appearing under multiple mangas) silently attributed every collision
to the first manga that synced it. The INSERT path would have
conflicted on the second manga's sync.

Migration 0017 drops the old PK and rekeys on (source_id, chapter_id)
— the natural identity of a per-source chapter attachment — and adds
an index on (source_id, source_chapter_key) for the lookup path. The
repo lookup now joins chapters and filters by manga_id; the UPDATE
path keys on chapter_id directly (the row's natural identifier
post-migration).

Test sync_chapters_isolates_colliding_keys_across_mangas pins the
contract end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:43:08 +02:00
MechaCat02
50763addcf fix(crawler): quarantine recently-dead chapters from re-enqueue (0.35.4)
The partial dedup index only blocks (pending|running) duplicates, so
once a SyncChapterContent job transitions to 'dead' (max_attempts
exhausted) the slot frees. Every subsequent cron tick re-enqueued the
chapter — page_count = 0 and dropped_at IS NULL stay true — burned
another max_attempts retries, and died again. Permanent-failure
chapters spun forever.

enqueue_bookmarked_pending and enqueue_pending_for_manga now skip
chapters whose latest sync_chapter_content job is dead within
CHAPTER_DEAD_QUARANTINE_DAYS (7). A failed chapter goes silent for a
week, then gets one more shot — long enough for a transient site
issue to resolve, short enough that permanent failures don't stay
permanent if conditions change.

Two integration tests pin both halves of the contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:42:41 +02:00
MechaCat02
766c6eebac fix(crawler): guard ack_done/ack_failed/release on state='running' (0.35.3)
The three lease-ack functions matched their UPDATE on the job id
alone. If a lease expired and another worker re-leased the row, a
late ack from the original worker would clobber the new lease's
state, leased_until, and (for release) decrement its attempts.

Add `AND state = 'running'` to each UPDATE and log a warn when
rows_affected is zero, so a stolen lease shows up in telemetry without
blocking the new lease holder's progress.

Three new integration tests pin the contract:
- ack_done_no_ops_when_lease_was_stolen
- ack_failed_no_ops_when_state_is_not_running
- release_no_ops_when_state_is_not_running

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:42:18 +02:00
MechaCat02
c686d6eb51 fix(crawler): sentinel-gate parse_chapter_list to stop false drops (0.35.2)
parse_chapter_list previously returned Vec::new() on any selector
miss. The empty list flowed into sync_manga_chapters, whose soft-drop
branch then flipped every existing chapter's dropped_at to NOW().
Bookmarks subsequently pointed at dropped sources, and
enqueue_bookmarked_pending (filters on cs.dropped_at IS NULL) silently
stopped re-fetching pages.

Same shape as the walker race fixed in 0.35.1: a transient parse miss
masquerading as "source removed everything" → false soft-drop.

Fix: require #chapter_table in the DOM. Present-but-empty is preserved
as Ok(vec![]) so a freshly added manga with no published chapters
still parses cleanly. Absent table is now Transient — the job system
reschedules with backoff instead of treating the partial render as
data.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:41:47 +02:00
MechaCat02
dea9b1aaa8 fix(crawler): close walker race against site reordering (0.35.1)
The target site orders by update_date DESC, and any new or updated
manga pushes everyone down by one slot. The paginated walker was
blind to this drift:

* Backfill (page last -> 1): shifts push items into pages already
  finished. The displaced manga was silently missed; with
  mark_dropped_mangas running on a fully-completed walk, items even
  got false-dropped because last_seen_at was stale.
* Incremental (page 1 -> last): a shift causes the slot-last item
  of an already-read page to reappear on the next page, leading to
  a redundant fetch_manga and an inflated consecutive_unchanged
  streak.

Fix is two-pronged:

1. Backfill boundary re-check. After fetching each page P, re-fetch
   the previously-walked page P+1 and check where its old slot-0
   key now sits. If it slid to slot K, the first K entries are
   items that used to live on P and slid past us; they get appended
   to the batch. If the anchor is gone entirely (multi-page shift
   or it was bumped to page 1), the whole re-fetched page is
   processed conservatively and the pipeline dedup absorbs the
   noise. The re-check must be the *last* navigation of the
   iteration to close the within-iteration race.

2. Run-scoped dedup in run_metadata_pass. A HashSet<String> of
   source_manga_keys avoids double-processing. The set uses a
   contains-then-insert pattern with insert firing *after* a
   successful upsert, so a transient fetch/upsert failure leaves
   the key retryable if it reappears later in the same pass (via
   the boundary re-check or another batch).

Incremental mode does not run the re-check (shifts move in the
same direction as the walk); only the dedup helps it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:14:01 +02:00
64 changed files with 6350 additions and 707 deletions

123
backend/Cargo.lock generated
View File

@@ -1202,7 +1202,7 @@ dependencies = [
"js-sys",
"log",
"wasm-bindgen",
"windows-core",
"windows-core 0.62.2",
]
[[package]]
@@ -1470,7 +1470,7 @@ checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4"
[[package]]
name = "mangalord"
version = "0.35.0"
version = "0.43.0"
dependencies = [
"anyhow",
"argon2",
@@ -1488,6 +1488,7 @@ dependencies = [
"http-body-util",
"infer",
"mime",
"nix 0.29.0",
"rand 0.8.6",
"reqwest",
"scraper",
@@ -1496,6 +1497,7 @@ dependencies = [
"sha2",
"sqlx",
"subtle",
"sysinfo",
"tempfile",
"thiserror 1.0.69",
"time",
@@ -1603,6 +1605,18 @@ version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086"
[[package]]
name = "nix"
version = "0.29.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46"
dependencies = [
"bitflags",
"cfg-if",
"cfg_aliases",
"libc",
]
[[package]]
name = "nix"
version = "0.31.3"
@@ -1615,6 +1629,15 @@ dependencies = [
"libc",
]
[[package]]
name = "ntapi"
version = "0.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3b335231dfd352ffb0f8017f3b6027a4917f7df785ea2143d8af2adc66980ae"
dependencies = [
"winapi",
]
[[package]]
name = "nu-ansi-term"
version = "0.50.3"
@@ -1855,7 +1878,7 @@ checksum = "9cf20a545b305cf1da722b236b5155c9bb35f1d5ceb28c048bd96ca842f41b5b"
dependencies = [
"android_system_properties",
"log",
"nix",
"nix 0.31.3",
"objc2",
"objc2-foundation",
"objc2-ui-kit",
@@ -2985,6 +3008,19 @@ dependencies = [
"syn",
]
[[package]]
name = "sysinfo"
version = "0.32.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4c33cd241af0f2e9e3b5c32163b873b29956890b5342e6745b917ce9d490f4af"
dependencies = [
"core-foundation-sys",
"libc",
"memchr",
"ntapi",
"windows",
]
[[package]]
name = "tempfile"
version = "3.27.0"
@@ -3606,19 +3642,74 @@ dependencies = [
"wasite",
]
[[package]]
name = "winapi"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419"
dependencies = [
"winapi-i686-pc-windows-gnu",
"winapi-x86_64-pc-windows-gnu",
]
[[package]]
name = "winapi-i686-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6"
[[package]]
name = "winapi-x86_64-pc-windows-gnu"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windows"
version = "0.57.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "12342cb4d8e3b046f3d80effd474a7a02447231330ef77d71daa6fbc40681143"
dependencies = [
"windows-core 0.57.0",
"windows-targets 0.52.6",
]
[[package]]
name = "windows-core"
version = "0.57.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2ed2439a290666cd67ecce2b0ffaad89c2a56b976b736e6ece670297897832d"
dependencies = [
"windows-implement 0.57.0",
"windows-interface 0.57.0",
"windows-result 0.1.2",
"windows-targets 0.52.6",
]
[[package]]
name = "windows-core"
version = "0.62.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb"
dependencies = [
"windows-implement",
"windows-interface",
"windows-implement 0.60.2",
"windows-interface 0.59.3",
"windows-link",
"windows-result",
"windows-result 0.4.1",
"windows-strings",
]
[[package]]
name = "windows-implement"
version = "0.57.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9107ddc059d5b6fbfbffdfa7a7fe3e22a226def0b2608f72e9d552763d3e1ad7"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "windows-implement"
version = "0.60.2"
@@ -3630,6 +3721,17 @@ dependencies = [
"syn",
]
[[package]]
name = "windows-interface"
version = "0.57.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29bee4b38ea3cde66011baa44dba677c432a78593e202392d1e9070cf2a7fca7"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "windows-interface"
version = "0.59.3"
@@ -3647,6 +3749,15 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-result"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e383302e8ec8515204254685643de10811af0ed97ea37210dc26fb0032647f8"
dependencies = [
"windows-targets 0.52.6",
]
[[package]]
name = "windows-result"
version = "0.4.1"

View File

@@ -1,6 +1,6 @@
[package]
name = "mangalord"
version = "0.35.0"
version = "0.43.0"
edition = "2021"
default-run = "mangalord"
@@ -45,6 +45,8 @@ futures-core = "0.3"
futures-util = "0.3"
bytes = "1"
chromiumoxide = { version = "0.7", features = ["tokio-runtime", "_fetcher-rusttls-tokio"], default-features = false }
sysinfo = { version = "0.32", default-features = false, features = ["system"] }
nix = { version = "0.29", features = ["fs"] }
scraper = "0.20"
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "socks", "cookies", "stream"] }

View File

@@ -0,0 +1,20 @@
-- chapter_sources: drop the global (source_id, source_chapter_key) PK
-- and rekey on (source_id, chapter_id).
--
-- The old PK assumed chapter slugs are unique per source. Sources whose
-- chapter naming is per-manga (chapter-1, chapter-2, ...) instead of per-
-- catalog (br_chapter-379272 with a global counter) would collide on the
-- second manga: the INSERT would conflict on (source_id, "chapter-1") and
-- the lookup would attribute the row to the first manga's chapter_id.
--
-- The new key is the natural identity of a source attachment: "this source
-- has this chapter". An (source_id, source_chapter_key) index preserves
-- the lookup path (find existing source row by source's identifier) but
-- no longer enforces uniqueness — the application combines it with the
-- chapters table's manga_id to scope the lookup per-manga.
ALTER TABLE chapter_sources DROP CONSTRAINT chapter_sources_pkey;
ALTER TABLE chapter_sources ADD PRIMARY KEY (source_id, chapter_id);
CREATE INDEX chapter_sources_source_key_idx
ON chapter_sources (source_id, source_chapter_key);

View File

@@ -0,0 +1,5 @@
-- Admin role flag on users. Booted from ADMIN_USERNAME / ADMIN_PASSWORD env at
-- startup (see app::build). Demotion is instant: the RequireAdmin extractor
-- re-reads the user row every request, so flipping this column takes effect on
-- the next call without a session purge.
ALTER TABLE users ADD COLUMN is_admin BOOLEAN NOT NULL DEFAULT false;

View File

@@ -0,0 +1,20 @@
-- Admin audit log. Written from inside the same transaction as the action
-- it records, so a failed COMMIT also rolls back the audit row — the log
-- never claims an action happened that didn't.
--
-- `actor_user_id` is ON DELETE SET NULL so audit rows outlive a deleted
-- admin (the answer to "who promoted Bob to admin?" survives even after
-- Alice's account is removed). `target_id` is intentionally not a FK
-- because future audit kinds may target non-user rows (manga, source,
-- etc.) and a single typed FK can't express that.
CREATE TABLE admin_audit (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
actor_user_id uuid REFERENCES users(id) ON DELETE SET NULL,
action text NOT NULL,
target_kind text NOT NULL,
target_id uuid,
payload jsonb NOT NULL DEFAULT '{}'::jsonb,
at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX admin_audit_at_idx ON admin_audit (at DESC);

View File

@@ -0,0 +1,14 @@
-- Per-manga sync-state derivation joins crawler_jobs to manga_sources via
-- (payload->>'source_id', payload->>'source_manga_key') for the
-- `sync_manga` job kind (whose payload doesn't carry a manga_id directly).
-- Without this index the join falls back to a seqscan of crawler_jobs on
-- every admin manga listing — a noticeable cost as the job table grows
-- with the daily metadata pass.
--
-- Partial on `state IN ('pending','running')` so it covers only in-flight
-- jobs (the bulk of the table is done/dead and irrelevant to "is this
-- manga being synced right now").
CREATE INDEX crawler_jobs_sync_manga_key_idx
ON crawler_jobs ((payload->>'source_manga_key'))
WHERE state IN ('pending', 'running')
AND payload->>'kind' = 'sync_manga';

View File

@@ -0,0 +1,110 @@
//! Admin manga/chapter overview with derived sync state.
//!
//! Sync state comes from `repo::admin_view`, which joins the manga /
//! chapter tables with the crawler signals at query time — there is no
//! persisted sync_state column. See [`repo::admin_view`] for the
//! derivation priority order.
use axum::extract::{Path, Query, State};
use axum::routing::get;
use axum::{Json, Router};
use serde::Deserialize;
use uuid::Uuid;
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::domain::MangaSyncState;
use crate::error::{AppError, AppResult};
use crate::repo;
use crate::repo::admin_view::{AdminChapterRow, AdminMangaRow};
pub fn routes() -> Router<AppState> {
Router::new()
.route("/admin/mangas", get(list_mangas))
.route("/admin/mangas/:id/chapters", get(list_chapters))
}
#[derive(Debug, Deserialize, Default)]
pub struct ListChaptersParams {
#[serde(default = "default_chapter_limit")]
pub limit: i64,
#[serde(default)]
pub offset: i64,
}
fn default_chapter_limit() -> i64 {
200
}
#[derive(Debug, Deserialize, Default)]
pub struct ListMangasParams {
#[serde(default)]
pub search: Option<String>,
/// `in_progress` | `dropped` | `synced`. Unrecognised values are a 400.
#[serde(default)]
pub sync_state: Option<String>,
#[serde(default = "default_limit")]
pub limit: i64,
#[serde(default)]
pub offset: i64,
}
fn default_limit() -> i64 {
50
}
async fn list_mangas(
State(state): State<AppState>,
_admin: RequireAdmin,
Query(params): Query<ListMangasParams>,
) -> AppResult<Json<PagedResponse<AdminMangaRow>>> {
let limit = params.limit.clamp(1, 200);
let offset = params.offset.max(0);
let sync_state = match params.sync_state.as_deref() {
None | Some("") => None,
Some("in_progress") => Some(MangaSyncState::InProgress),
Some("dropped") => Some(MangaSyncState::Dropped),
Some("synced") => Some(MangaSyncState::Synced),
Some(other) => {
return Err(AppError::InvalidInput(format!(
"sync_state must be one of in_progress|dropped|synced (got {other:?})"
)));
}
};
let q = repo::admin_view::ListAdminMangasQuery {
search: params.search.filter(|s| !s.trim().is_empty()),
sync_state,
limit,
offset,
};
let (items, total) = repo::admin_view::list_mangas_with_sync_state(&state.db, &q).await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
}
async fn list_chapters(
State(state): State<AppState>,
_admin: RequireAdmin,
Path(manga_id): Path<Uuid>,
Query(params): Query<ListChaptersParams>,
) -> AppResult<Json<PagedResponse<AdminChapterRow>>> {
// Explicit existence check so a typo / deleted manga returns 404
// rather than a misleading "no chapters" 200.
if !repo::manga::exists(&state.db, manga_id).await? {
return Err(AppError::NotFound);
}
// Cap at 500 to bound the per-row scalar-subquery cost on
// long-runners with thousands of chapters; default 200 covers
// typical browsing without paging round-trips.
let limit = params.limit.clamp(1, 500);
let offset = params.offset.max(0);
let q = repo::admin_view::ListAdminChaptersQuery {
manga_id,
limit,
offset,
};
let (items, total) = repo::admin_view::list_chapters_with_sync_state(&state.db, &q).await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
}

View File

@@ -0,0 +1,20 @@
//! Admin-only endpoints. Mounted under `/api/v1/admin/*` by
//! `crate::api::routes`. Every handler in this subtree is guarded by
//! `RequireAdmin`, which only accepts session-cookie authentication —
//! bot/API tokens cannot reach admin routes (see
//! `crate::auth::extractor::RequireAdmin`).
pub mod mangas;
pub mod system;
pub mod users;
use axum::Router;
use crate::app::AppState;
pub fn routes() -> Router<AppState> {
Router::new()
.merge(users::routes())
.merge(mangas::routes())
.merge(system::routes())
}

View File

@@ -0,0 +1,163 @@
//! System metrics for the admin dashboard.
//!
//! Disk is `statvfs(storage_dir)` so the number reflects the volume the
//! app actually writes to (not the root filesystem of the host). When the
//! storage backend doesn't expose a local path (e.g. a future S3 impl)
//! the disk fields are `null` rather than fabricated.
//!
//! Memory and CPU come from `sysinfo`. CPU requires two refreshes with
//! at least 200ms between them to compute a meaningful delta; the
//! handler eats the 250ms wall-clock cost on each request. Admin
//! traffic is low-volume so a background cache isn't worth the moving
//! parts yet — revisit if polling becomes frequent.
use std::path::Path;
use std::time::Duration;
use axum::extract::State;
use axum::routing::get;
use axum::{Json, Router};
use serde::Serialize;
use sysinfo::{CpuRefreshKind, MemoryRefreshKind, RefreshKind, System};
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::error::AppResult;
const ALERT_THRESHOLD_PERCENT: f64 = 90.0;
pub fn routes() -> Router<AppState> {
Router::new().route("/admin/system", get(system))
}
#[derive(Debug, Serialize)]
pub struct SystemStats {
pub disk: Option<DiskStats>,
pub memory: MemoryStats,
pub cpu: CpuStats,
pub alerts: Vec<Alert>,
}
#[derive(Debug, Serialize)]
pub struct DiskStats {
pub total_bytes: u64,
pub used_bytes: u64,
pub free_bytes: u64,
pub percent_used: f64,
}
#[derive(Debug, Serialize)]
pub struct MemoryStats {
pub total_bytes: u64,
pub used_bytes: u64,
pub percent_used: f64,
}
#[derive(Debug, Serialize)]
pub struct CpuStats {
pub percent_used: f64,
}
#[derive(Debug, Serialize)]
pub struct Alert {
pub level: AlertLevel,
pub message: String,
}
#[derive(Debug, Serialize, Clone, Copy)]
#[serde(rename_all = "snake_case")]
pub enum AlertLevel {
Warning,
}
async fn system(
State(state): State<AppState>,
_admin: RequireAdmin,
) -> AppResult<Json<SystemStats>> {
let disk = state.storage.local_root().and_then(disk_stats_for);
let (memory, cpu) = memory_and_cpu().await;
let mut alerts = Vec::new();
if let Some(d) = &disk {
if d.percent_used >= ALERT_THRESHOLD_PERCENT {
alerts.push(Alert {
level: AlertLevel::Warning,
message: format!(
"disk near full ({:.0}% used)",
d.percent_used
),
});
}
}
if memory.percent_used >= ALERT_THRESHOLD_PERCENT {
alerts.push(Alert {
level: AlertLevel::Warning,
message: format!(
"memory near full ({:.0}% used)",
memory.percent_used
),
});
}
Ok(Json(SystemStats {
disk,
memory,
cpu,
alerts,
}))
}
fn disk_stats_for(root: &Path) -> Option<DiskStats> {
let s = nix::sys::statvfs::statvfs(root).ok()?;
// statvfs reports `f_frsize * f_blocks` for total bytes. `f_bavail`
// is "free to non-root callers" which is what an operator actually
// cares about — `f_bfree` includes blocks reserved for root.
let block = s.fragment_size();
let total = block * s.blocks();
let avail = block * s.blocks_available();
let used = total.saturating_sub(avail);
let percent_used = if total > 0 {
(used as f64) * 100.0 / (total as f64)
} else {
0.0
};
Some(DiskStats {
total_bytes: total,
used_bytes: used,
free_bytes: avail,
percent_used,
})
}
async fn memory_and_cpu() -> (MemoryStats, CpuStats) {
// sysinfo's CPU sampling needs two refreshes with a delay between
// them — the first seeds the delta counters, the second measures.
// We do this once per request; admin traffic is low enough that the
// 250ms cost is invisible.
let mut sys = System::new_with_specifics(
RefreshKind::new()
.with_cpu(CpuRefreshKind::everything())
.with_memory(MemoryRefreshKind::everything()),
);
sys.refresh_cpu_all();
// Yield the runtime instead of blocking it for the gap.
tokio::time::sleep(Duration::from_millis(250)).await;
sys.refresh_cpu_all();
sys.refresh_memory();
let total = sys.total_memory();
let used = sys.used_memory();
let mem_pct = if total > 0 {
(used as f64) * 100.0 / (total as f64)
} else {
0.0
};
let memory = MemoryStats {
total_bytes: total,
used_bytes: used,
percent_used: mem_pct,
};
let cpu = CpuStats {
percent_used: sys.global_cpu_usage() as f64,
};
(memory, cpu)
}

View File

@@ -0,0 +1,128 @@
//! Admin user management: list, delete, promote/demote.
//!
//! All handlers are gated by `RequireAdmin` and rely on
//! `repo::user::admin_safe_*` for self-protection and the last-admin
//! invariant. Audit rows are written inside the same DB transaction as
//! the action they record.
use axum::extract::{Path, Query, State};
use axum::http::StatusCode;
use axum::routing::{delete, get};
use axum::{Json, Router};
use serde::Deserialize;
use uuid::Uuid;
use crate::api::auth::{validate_password, validate_username};
use crate::api::pagination::PagedResponse;
use crate::app::AppState;
use crate::auth::extractor::RequireAdmin;
use crate::auth::password::hash_password;
use crate::domain::User;
use crate::error::{AppError, AppResult};
use crate::repo;
pub fn routes() -> Router<AppState> {
Router::new()
.route("/admin/users", get(list_users).post(create_user))
.route(
"/admin/users/:id",
delete(delete_user).patch(update_user),
)
}
#[derive(Debug, Deserialize, Default)]
pub struct ListUsersParams {
#[serde(default)]
pub search: Option<String>,
#[serde(default = "default_limit")]
pub limit: i64,
#[serde(default)]
pub offset: i64,
}
fn default_limit() -> i64 {
50
}
async fn list_users(
State(state): State<AppState>,
_admin: RequireAdmin,
Query(params): Query<ListUsersParams>,
) -> AppResult<Json<PagedResponse<User>>> {
let limit = params.limit.clamp(1, 200);
let offset = params.offset.max(0);
let (items, total) = repo::user::list_with_total(
&state.db,
&repo::user::ListUsersQuery {
search: params.search.filter(|s| !s.trim().is_empty()),
limit,
offset,
},
)
.await?;
Ok(Json(PagedResponse::with_total(items, limit, offset, total)))
}
#[derive(Debug, Deserialize)]
pub struct UpdateUserInput {
pub is_admin: Option<bool>,
}
async fn update_user(
State(state): State<AppState>,
RequireAdmin(actor): RequireAdmin,
Path(id): Path<Uuid>,
Json(input): Json<UpdateUserInput>,
) -> AppResult<Json<User>> {
let Some(is_admin) = input.is_admin else {
return Err(AppError::InvalidInput(
"no updatable fields supplied".into(),
));
};
let updated =
repo::user::admin_safe_set_is_admin(&state.db, actor.id, id, is_admin).await?;
Ok(Json(updated))
}
async fn delete_user(
State(state): State<AppState>,
RequireAdmin(actor): RequireAdmin,
Path(id): Path<Uuid>,
) -> AppResult<StatusCode> {
repo::user::admin_safe_delete(&state.db, actor.id, id).await?;
Ok(StatusCode::NO_CONTENT)
}
#[derive(Debug, Deserialize)]
pub struct CreateUserInput {
pub username: String,
pub password: String,
/// Defaults to false; admins may mint other admins in a single
/// call. Doing it as one POST avoids a second audit row for the
/// common "invite a co-admin" flow.
#[serde(default)]
pub is_admin: bool,
}
async fn create_user(
State(state): State<AppState>,
RequireAdmin(actor): RequireAdmin,
Json(input): Json<CreateUserInput>,
) -> AppResult<(StatusCode, Json<User>)> {
let username = input.username.trim();
// Reuse the canonical self-register validators so the admin-create
// path can never produce a username that self-register would
// reject (and vice versa).
validate_username(username)?;
validate_password(&input.password)?;
let pwhash = hash_password(&input.password)?;
let user = repo::user::admin_create_user(
&state.db,
actor.id,
username,
&pwhash,
input.is_admin,
)
.await?;
Ok((StatusCode::CREATED, Json(user)))
}

View File

@@ -28,6 +28,7 @@ use crate::repo;
pub fn routes() -> Router<AppState> {
Router::new()
.route("/auth/config", get(auth_config))
.route("/auth/register", post(register))
.route("/auth/login", post(login))
.route("/auth/logout", post(logout))
@@ -41,6 +42,21 @@ pub fn routes() -> Router<AppState> {
.route("/auth/tokens/:id", delete(delete_token))
}
/// Public, unauthenticated. Exposes anonymous-relevant auth policy
/// (currently just whether self-registration is open) so the frontend
/// can render its login / register affordances correctly without a
/// probe request that would conflate "disabled" with "rate-limited".
#[derive(Debug, Serialize)]
pub struct AuthConfigResponse {
pub self_register_enabled: bool,
}
async fn auth_config(State(state): State<AppState>) -> Json<AuthConfigResponse> {
Json(AuthConfigResponse {
self_register_enabled: state.auth.allow_self_register,
})
}
#[derive(Debug, Deserialize)]
pub struct Credentials {
pub username: String,
@@ -82,7 +98,14 @@ async fn register(
jar: CookieJar,
Json(input): Json<Credentials>,
) -> AppResult<impl IntoResponse> {
// Rate limit before the disabled check so an operator who flips
// the toggle can't be probed for the toggle state via timing —
// disabled and enabled paths both consume a token, and disabled
// returns 403 instead of running argon2.
check_auth_rate_limit(&state, "register")?;
if !state.auth.allow_self_register {
return Err(AppError::Forbidden);
}
let username = input.username.trim();
validate_username(username)?;
validate_password(&input.password)?;
@@ -309,6 +332,18 @@ async fn start_session(
Ok(jar.add(build_session_cookie(raw, &state.auth)))
}
// CSRF posture: `SameSite=Lax` is the project's primary CSRF defense.
// Browsers refuse to attach this cookie to cross-site POST / PATCH /
// DELETE requests, which covers every state-changing endpoint (auth
// mutations, uploads, bookmarks, collections, admin user management,
// etc. — all JSON over POST/PATCH/DELETE). Lax DOES still attach the
// cookie on top-level cross-site GETs, so this defense breaks the
// instant anyone adds a state-changing GET. If you reach for one,
// switch to `SameSite=Strict` here AND add an explicit CSRF-token
// check on the new endpoint. The Bearer-token branch in the
// extractor is unaffected (bots authenticate with the token header,
// not the cookie) and admin routes reject Bearer entirely — see
// `auth::extractor::RequireAdmin`.
fn build_session_cookie(raw: String, cfg: &AuthConfig) -> Cookie<'static> {
let mut builder = Cookie::build((SESSION_COOKIE_NAME, raw))
.http_only(true)
@@ -362,7 +397,11 @@ fn check_auth_rate_limit(state: &AppState, endpoint: &'static str) -> AppResult<
}
}
fn validate_username(u: &str) -> AppResult<()> {
// Exposed pub(crate) so the admin user-create handler can apply the
// same rules as self-registration. Keeping the lone canonical
// implementation here avoids the two paths drifting on min length /
// allowed character set.
pub(crate) fn validate_username(u: &str) -> AppResult<()> {
if u.is_empty() {
return Err(AppError::InvalidInput("username is required".into()));
}
@@ -379,7 +418,7 @@ fn validate_username(u: &str) -> AppResult<()> {
Ok(())
}
fn validate_password(p: &str) -> AppResult<()> {
pub(crate) fn validate_password(p: &str) -> AppResult<()> {
if p.len() < 8 {
return Err(AppError::InvalidInput(
"password must be at least 8 characters".into(),

View File

@@ -1,3 +1,4 @@
pub mod admin;
pub mod auth;
pub mod authors;
pub mod bookmarks;
@@ -28,4 +29,5 @@ pub fn routes() -> Router<AppState> {
.merge(authors::routes())
.merge(collections::routes())
.merge(history::routes())
.merge(admin::routes())
}

View File

@@ -13,7 +13,7 @@ use tower_http::cors::{AllowOrigin, CorsLayer};
use tower_http::trace::TraceLayer;
use crate::auth::rate_limit::AuthRateLimiter;
use crate::config::{AuthConfig, Config, CrawlerConfig, CrawlerModePref, UploadConfig};
use crate::config::{AuthConfig, Config, CrawlerConfig, UploadConfig};
use crate::crawler::browser_manager::{self, BrowserManager};
use crate::crawler::content::{self, SyncOutcome};
use crate::crawler::daemon::{self, ChapterDispatcher, DaemonConfig, MetadataPass};
@@ -22,7 +22,6 @@ use crate::crawler::pipeline::{self, MetadataStats};
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::safety::DownloadAllowlist;
use crate::crawler::session;
use crate::crawler::source::{target as target_source, DiscoverMode};
use crate::repo;
use crate::storage::{LocalStorage, Storage};
@@ -61,6 +60,13 @@ pub async fn build(config: Config) -> anyhow::Result<AppHandle> {
.await?;
sqlx::migrate!("./migrations").run(&db).await?;
if let Some((username, password)) = config.admin_bootstrap.as_ref() {
repo::user::bootstrap_admin(&db, username, password)
.await
.context("bootstrap_admin from ADMIN_USERNAME/ADMIN_PASSWORD env")?;
tracing::info!(admin_username = %username, "admin bootstrap ensured");
}
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(config.storage_dir.clone()));
let daemon = if config.crawler.daemon_enabled {
@@ -159,8 +165,6 @@ async fn spawn_crawler_daemon(
http: http.clone(),
rate: Arc::clone(&rate),
start_url: url.clone(),
mode_pref: cfg.mode,
incremental_stop_after: cfg.incremental_stop_after,
download_allowlist: cfg.download_allowlist.clone(),
max_image_bytes: cfg.max_image_bytes,
});
@@ -226,8 +230,6 @@ struct RealMetadataPass {
http: reqwest::Client,
rate: Arc<HostRateLimiters>,
start_url: String,
mode_pref: CrawlerModePref,
incremental_stop_after: usize,
download_allowlist: DownloadAllowlist,
max_image_bytes: usize,
}
@@ -235,14 +237,7 @@ struct RealMetadataPass {
#[async_trait]
impl MetadataPass for RealMetadataPass {
async fn run(&self) -> anyhow::Result<MetadataStats> {
let mode = resolve_mode(
&self.db,
target_source::SOURCE_ID,
self.mode_pref,
self.incremental_stop_after,
)
.await?;
pipeline::run_metadata_pass(
let result = pipeline::run_metadata_pass(
&self.browser_manager,
&self.db,
self.storage.as_ref(),
@@ -251,55 +246,16 @@ impl MetadataPass for RealMetadataPass {
&self.start_url,
0,
false,
mode,
&self.download_allowlist,
self.max_image_bytes,
)
.await
}
}
/// Pick the active mode for this tick. `Explicit` short-circuits the
/// DB lookup. `Auto` reads `seed_completed_at`: missing → Backfill
/// (initial seed for this source), present → Incremental with the
/// configured threshold.
///
/// A DB error during the Auto lookup propagates as `Err` rather than
/// silently degrading to Backfill — the daemon's `run_tick` catches
/// the error, logs, and skips the tick. That's safer than running a
/// full re-backfill (including a drop pass against stale-looking rows)
/// when the DB is flaky.
async fn resolve_mode(
db: &PgPool,
source_id: &str,
pref: CrawlerModePref,
incremental_stop_after: usize,
) -> anyhow::Result<DiscoverMode> {
match pref {
CrawlerModePref::Explicit(m) => {
tracing::info!(?m, "crawler mode: explicit (CRAWLER_MODE override)");
Ok(m)
}
CrawlerModePref::Auto => {
let seeded = repo::crawler::seed_completed_at(db, source_id)
.await
.context("seed_completed_at lookup for mode auto-detection")?;
match seeded {
Some(at) => {
tracing::info!(
seed_completed_at = %at.to_rfc3339(),
"crawler mode: auto → incremental (seed previously completed)"
);
Ok(DiscoverMode::Incremental {
stop_after_unchanged: incremental_stop_after,
})
}
None => {
tracing::info!("crawler mode: auto → backfill (no seed marker for source)");
Ok(DiscoverMode::Backfill)
}
.await;
if let Err(e) = &result {
if crate::crawler::nav::anyhow_looks_browser_dead(e) {
self.browser_manager.invalidate().await;
}
}
result
}
}
@@ -330,7 +286,7 @@ impl ChapterDispatcher for RealChapterDispatcher {
return Ok(SyncOutcome::Skipped);
};
let lease = self.browser_manager.acquire().await?;
let outcome = content::sync_chapter_content(
let result = content::sync_chapter_content(
&lease,
&self.db,
self.storage.as_ref(),
@@ -343,13 +299,21 @@ impl ChapterDispatcher for RealChapterDispatcher {
&self.download_allowlist,
self.max_image_bytes,
)
.await?;
.await;
drop(lease);
Ok(outcome)
match result {
Ok(outcome) => Ok(outcome),
Err(e) => {
if crate::crawler::nav::anyhow_looks_browser_dead(&e) {
self.browser_manager.invalidate().await;
}
Err(e)
}
}
}
// Other payload kinds aren't dispatched by this daemon yet —
// metadata-driven jobs (Discover/SyncManga/SyncChapterList)
// are handled inline by the cron's metadata pass.
// SyncManga / SyncChapterList are handled inline by the cron's
// metadata pass.
_ => Ok(SyncOutcome::Skipped),
}
}

View File

@@ -1,11 +1,19 @@
//! `CurrentUser` axum extractor.
//! Auth extractors.
//!
//! Resolves a request to a logged-in user by trying, in order:
//! 1. a `mangalord_session` cookie (session lookup by `sha256(value)`);
//! 2. an `Authorization: Bearer <token>` header (api_token lookup).
//! Three extractors are available, in increasing strictness:
//!
//! Both paths look up by hash, never by raw value. Failure to resolve
//! either way returns 401 via `AppError::Unauthenticated`.
//! - [`CurrentUser`] — accepts either a session cookie or an
//! `Authorization: Bearer <token>` header. Used by ordinary
//! authenticated endpoints where bot tokens are first-class clients.
//! - [`CurrentSessionUser`] — accepts only the session cookie. Used as
//! the substrate for admin extraction so bot tokens cannot authenticate
//! as the admin (see [`RequireAdmin`]).
//! - [`RequireAdmin`] — composes over [`CurrentSessionUser`] and
//! additionally requires `user.is_admin`. Returns 403 for
//! authenticated-but-not-admin, 401 otherwise.
//!
//! All lookups go by `sha256(raw_token)` — the raw value is never stored
//! in the database.
use axum::async_trait;
use axum::extract::FromRequestParts;
@@ -61,3 +69,54 @@ impl FromRequestParts<AppState> for CurrentUser {
Err(AppError::Unauthenticated)
}
}
/// Cookie-only authentication. Bot/API tokens are explicitly NOT accepted
/// here — this is the substrate for [`RequireAdmin`] and exists precisely
/// to keep admin authority out of bearer-token reach.
pub struct CurrentSessionUser(pub User);
#[async_trait]
impl FromRequestParts<AppState> for CurrentSessionUser {
type Rejection = AppError;
async fn from_request_parts(
parts: &mut Parts,
state: &AppState,
) -> Result<Self, Self::Rejection> {
let jar = CookieJar::from_headers(&parts.headers);
let cookie = jar
.get(SESSION_COOKIE_NAME)
.ok_or(AppError::Unauthenticated)?;
let hash = hash_token(cookie.value());
let session = repo::session::find_active(&state.db, &hash)
.await?
.ok_or(AppError::Unauthenticated)?;
let user = repo::user::find_by_id(&state.db, session.user_id)
.await?
.ok_or(AppError::Unauthenticated)?;
Ok(CurrentSessionUser(user))
}
}
/// Admin-only. Composes over [`CurrentSessionUser`] so bot tokens are
/// rejected at the auth step (401) rather than the role step (403).
/// The user row is re-read every request, so demotion takes effect on
/// the very next call without needing to purge sessions.
pub struct RequireAdmin(pub User);
#[async_trait]
impl FromRequestParts<AppState> for RequireAdmin {
type Rejection = AppError;
async fn from_request_parts(
parts: &mut Parts,
state: &AppState,
) -> Result<Self, Self::Rejection> {
let CurrentSessionUser(user) =
CurrentSessionUser::from_request_parts(parts, state).await?;
if !user.is_admin {
return Err(AppError::Forbidden);
}
Ok(RequireAdmin(user))
}
}

View File

@@ -31,7 +31,6 @@ use mangalord::crawler::content::{self, SyncOutcome};
use mangalord::crawler::pipeline;
use mangalord::crawler::rate_limit::HostRateLimiters;
use mangalord::crawler::session;
use mangalord::crawler::source::DiscoverMode;
use mangalord::storage::{LocalStorage, Storage};
use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool;
@@ -63,8 +62,6 @@ async fn main() -> anyhow::Result<()> {
let cdn_rate_ms = env_u64("CRAWLER_CDN_RATE_MS", rate_ms);
let limit = env_u64("CRAWLER_LIMIT", 0) as usize;
let skip_chapters = env_bool("CRAWLER_SKIP_CHAPTERS", false);
let incremental_stop_after = env_u64("CRAWLER_INCREMENTAL_STOP_AFTER", 20).max(1) as usize;
let mode = parse_crawler_mode(incremental_stop_after)?;
let skip_chapter_content = env_bool("CRAWLER_SKIP_CHAPTER_CONTENT", false);
let chapter_workers = env_u64("CRAWLER_CHAPTER_WORKERS", 1).max(1) as usize;
let force_refetch_chapters = env_bool("CRAWLER_FORCE_REFETCH_CHAPTERS", false);
@@ -143,7 +140,6 @@ async fn main() -> anyhow::Result<()> {
user_agent = ?user_agent,
proxy = ?proxy_url,
keep_open,
?mode,
storage_dir = %storage_dir.display(),
"starting crawler"
);
@@ -191,7 +187,6 @@ async fn main() -> anyhow::Result<()> {
skip_chapter_content || !session_ready,
chapter_workers,
force_refetch_chapters,
mode,
)
.await;
@@ -221,7 +216,6 @@ async fn run(
skip_chapter_content: bool,
chapter_workers: usize,
force_refetch_chapters: bool,
mode: DiscoverMode,
) -> anyhow::Result<()> {
let mut rate = HostRateLimiters::new(Duration::from_millis(rate_ms));
if let Some(host) = cdn_host {
@@ -265,7 +259,6 @@ async fn run(
start_url,
limit,
skip_chapters,
mode,
allowlist.as_ref(),
max_image_bytes,
)
@@ -433,38 +426,6 @@ fn resolve_start_url() -> anyhow::Result<String> {
})
}
/// Parse the CLI's `CRAWLER_MODE`. Defaults to `backfill` because the
/// binary is operator-driven (manual reseeds, force-refetches) — the
/// auto-detect logic lives in the daemon. `auto` is rejected because
/// the CLI has no DB state to consult before the run.
fn parse_crawler_mode(incremental_stop_after: usize) -> anyhow::Result<DiscoverMode> {
parse_crawler_mode_str(
std::env::var("CRAWLER_MODE").ok().as_deref(),
incremental_stop_after,
)
}
/// Pure variant of [`parse_crawler_mode`] — testable without env-var
/// mutation.
fn parse_crawler_mode_str(
raw: Option<&str>,
incremental_stop_after: usize,
) -> anyhow::Result<DiscoverMode> {
match raw.map(|s| s.trim().to_ascii_lowercase()).as_deref() {
None | Some("") | Some("backfill") => Ok(DiscoverMode::Backfill),
Some("incremental") => Ok(DiscoverMode::Incremental {
stop_after_unchanged: incremental_stop_after,
}),
Some("auto") => Err(anyhow!(
"CRAWLER_MODE=auto isn't supported by the CLI (use backfill or incremental); \
the daemon does auto-detection"
)),
Some(other) => Err(anyhow!(
"CRAWLER_MODE must be one of: backfill, incremental (got {other:?})"
)),
}
}
fn env_u64(name: &str, default: u64) -> u64 {
std::env::var(name)
.ok()
@@ -480,55 +441,3 @@ fn env_bool(name: &str, default: bool) -> bool {
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn cli_mode_defaults_to_backfill_when_unset_or_blank() {
let none = parse_crawler_mode_str(None, 20).unwrap();
assert!(matches!(none, DiscoverMode::Backfill));
let blank = parse_crawler_mode_str(Some(""), 20).unwrap();
assert!(matches!(blank, DiscoverMode::Backfill));
}
#[test]
fn cli_mode_recognizes_backfill_and_incremental() {
let backfill = parse_crawler_mode_str(Some("backfill"), 20).unwrap();
assert!(matches!(backfill, DiscoverMode::Backfill));
let incremental = parse_crawler_mode_str(Some("incremental"), 9).unwrap();
assert!(matches!(
incremental,
DiscoverMode::Incremental { stop_after_unchanged: 9 }
));
}
#[test]
fn cli_mode_rejects_auto_explicitly() {
let err = parse_crawler_mode_str(Some("auto"), 20).unwrap_err();
let msg = format!("{err}");
assert!(
msg.contains("daemon"),
"rejection should point operator at the daemon: {msg}"
);
}
#[test]
fn cli_mode_rejects_unknown_value() {
let err = parse_crawler_mode_str(Some("garbage"), 20).unwrap_err();
let msg = format!("{err}");
assert!(msg.contains("backfill"));
assert!(msg.contains("incremental"));
}
#[test]
fn cli_mode_is_case_insensitive_and_trims() {
let mixed = parse_crawler_mode_str(Some(" Incremental "), 4).unwrap();
assert!(matches!(
mixed,
DiscoverMode::Incremental { stop_after_unchanged: 4 }
));
}
}

View File

@@ -6,16 +6,6 @@ use chrono_tz::Tz;
use crate::crawler::browser::LaunchOptions;
use crate::crawler::safety::{DownloadAllowlist, DEFAULT_MAX_IMAGE_BYTES};
use crate::crawler::source::DiscoverMode;
/// What `CRAWLER_MODE` was set to. `Auto` is the daemon's default —
/// pick Backfill until `seed_completed_at` is written, then flip to
/// Incremental. `Explicit` forces a single mode regardless.
#[derive(Clone, Copy, Debug)]
pub enum CrawlerModePref {
Auto,
Explicit(DiscoverMode),
}
#[derive(Clone, Debug)]
pub struct AuthConfig {
@@ -23,6 +13,12 @@ pub struct AuthConfig {
pub cookie_domain: Option<String>,
pub session_ttl_days: i64,
pub rate_limit: crate::auth::rate_limit::RateLimitConfig,
/// When `false`, `POST /auth/register` returns 403
/// `registration_disabled` and the frontend hides its register
/// affordance. Admins can still mint accounts via
/// `POST /admin/users`. Defaults to `true` (open registration)
/// for backward compatibility.
pub allow_self_register: bool,
}
impl Default for AuthConfig {
@@ -36,6 +32,7 @@ impl Default for AuthConfig {
// to the [`PRODUCTION_PER_SEC`]/[`PRODUCTION_BURST`]
// defaults.
rate_limit: crate::auth::rate_limit::RateLimitConfig::default(),
allow_self_register: true,
}
}
}
@@ -69,6 +66,13 @@ pub struct Config {
pub upload: UploadConfig,
pub cors_allowed_origins: Vec<String>,
pub crawler: CrawlerConfig,
/// `(username, password)` for the admin user provisioned at startup
/// when both `ADMIN_USERNAME` and `ADMIN_PASSWORD` are set. `None`
/// skips the bootstrap entirely. See `repo::user::bootstrap_admin`
/// for the create-vs-promote semantics — notably the password here
/// is used only when creating a new row, never to overwrite an
/// existing one.
pub admin_bootstrap: Option<(String, String)>,
}
/// All crawler-daemon knobs read from env. Mirrors the env vars the
@@ -94,12 +98,6 @@ pub struct CrawlerConfig {
pub user_agent: Option<String>,
pub proxy: Option<String>,
pub browser: LaunchOptions,
/// Mode preference for the metadata pass. Daemon default is `Auto`
/// (Backfill until `seed_completed_at` is written, then Incremental).
pub mode: CrawlerModePref,
/// `stop_after_unchanged` threshold supplied to Incremental in both
/// `Auto` (post-seed) and `Explicit(Incremental)` modes.
pub incremental_stop_after: usize,
/// Hosts the crawler is allowed to download images / covers from.
/// Always seeded with the host of `start_url` and (when set) the
/// configured `cdn_host`. Additional hosts can be added via
@@ -127,8 +125,6 @@ impl Default for CrawlerConfig {
user_agent: None,
proxy: None,
browser: LaunchOptions::headless(),
mode: CrawlerModePref::Auto,
incremental_stop_after: 20,
download_allowlist: DownloadAllowlist::new(),
max_image_bytes: DEFAULT_MAX_IMAGE_BYTES,
}
@@ -161,6 +157,7 @@ impl Config {
crate::auth::rate_limit::PRODUCTION_BURST.into(),
) as u32,
},
allow_self_register: env_bool("ALLOW_SELF_REGISTER", true),
},
upload: UploadConfig {
max_request_bytes: env_usize("MAX_REQUEST_BYTES", 200 * 1024 * 1024),
@@ -176,10 +173,21 @@ impl Config {
})
.unwrap_or_default(),
crawler: CrawlerConfig::from_env()?,
admin_bootstrap: admin_bootstrap_from_env(),
})
}
}
/// Returns `Some((username, password))` only when BOTH `ADMIN_USERNAME`
/// and `ADMIN_PASSWORD` are set and non-empty. Half-set configuration is
/// treated as "no bootstrap" rather than a hard error, so an operator
/// can comment out one env var without crashing the server.
fn admin_bootstrap_from_env() -> Option<(String, String)> {
let username = std::env::var("ADMIN_USERNAME").ok().filter(|s| !s.is_empty())?;
let password = std::env::var("ADMIN_PASSWORD").ok().filter(|s| !s.is_empty())?;
Some((username, password))
}
impl CrawlerConfig {
pub fn from_env() -> anyhow::Result<Self> {
// Parse CRAWLER_DAILY_AT (HH:MM, 24h). Invalid → fail fast.
@@ -195,9 +203,6 @@ impl CrawlerConfig {
.parse()
.map_err(|e| anyhow::anyhow!("CRAWLER_TZ must be a valid IANA TZ (got {raw:?}): {e}"))?,
};
let incremental_stop_after =
env_u64("CRAWLER_INCREMENTAL_STOP_AFTER", 20).max(1) as usize;
let mode = parse_mode_env(incremental_stop_after)?;
let start_url = std::env::var("CRAWLER_START_URL")
.ok()
.filter(|s| !s.trim().is_empty());
@@ -230,8 +235,6 @@ impl CrawlerConfig {
.ok()
.filter(|s| !s.trim().is_empty()),
browser: LaunchOptions::from_env(),
mode,
incremental_stop_after,
download_allowlist,
max_image_bytes: env_usize("CRAWLER_MAX_IMAGE_BYTES", DEFAULT_MAX_IMAGE_BYTES),
})
@@ -271,32 +274,6 @@ fn build_download_allowlist(
allow
}
/// Parse `CRAWLER_MODE`. Empty/unset → `Auto`. Recognized values are
/// `auto`, `backfill`, and `incremental` (case-insensitive). Anything
/// else is a hard error so a typo can't silently fall through to the
/// default and mask itself.
fn parse_mode_env(incremental_stop_after: usize) -> anyhow::Result<CrawlerModePref> {
parse_mode_str(std::env::var("CRAWLER_MODE").ok().as_deref(), incremental_stop_after)
}
/// Pure variant of [`parse_mode_env`] — testable without env-var
/// mutation. Takes the raw value (or `None` if unset).
pub(crate) fn parse_mode_str(
raw: Option<&str>,
incremental_stop_after: usize,
) -> anyhow::Result<CrawlerModePref> {
match raw.map(|s| s.trim().to_ascii_lowercase()).as_deref() {
None | Some("") | Some("auto") => Ok(CrawlerModePref::Auto),
Some("backfill") => Ok(CrawlerModePref::Explicit(DiscoverMode::Backfill)),
Some("incremental") => Ok(CrawlerModePref::Explicit(DiscoverMode::Incremental {
stop_after_unchanged: incremental_stop_after,
})),
Some(other) => Err(anyhow::anyhow!(
"CRAWLER_MODE must be one of: auto, backfill, incremental (got {other:?})"
)),
}
}
fn env_u64(name: &str, default: u64) -> u64 {
std::env::var(name)
.ok()
@@ -326,62 +303,3 @@ fn env_usize(name: &str, default: usize) -> usize {
.unwrap_or(default)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_mode_str_defaults_to_auto_when_unset_or_blank() {
let none = parse_mode_str(None, 20).unwrap();
assert!(matches!(none, CrawlerModePref::Auto));
let blank = parse_mode_str(Some(""), 20).unwrap();
assert!(matches!(blank, CrawlerModePref::Auto));
let whitespace = parse_mode_str(Some(" "), 20).unwrap();
assert!(matches!(whitespace, CrawlerModePref::Auto));
}
#[test]
fn parse_mode_str_recognizes_each_keyword() {
let auto = parse_mode_str(Some("auto"), 20).unwrap();
assert!(matches!(auto, CrawlerModePref::Auto));
let backfill = parse_mode_str(Some("backfill"), 20).unwrap();
assert!(matches!(
backfill,
CrawlerModePref::Explicit(DiscoverMode::Backfill)
));
let incremental = parse_mode_str(Some("incremental"), 7).unwrap();
assert!(matches!(
incremental,
CrawlerModePref::Explicit(DiscoverMode::Incremental {
stop_after_unchanged: 7
})
));
}
#[test]
fn parse_mode_str_is_case_insensitive_and_trims_whitespace() {
let mixed = parse_mode_str(Some(" Incremental "), 5).unwrap();
assert!(matches!(
mixed,
CrawlerModePref::Explicit(DiscoverMode::Incremental {
stop_after_unchanged: 5
})
));
let upper = parse_mode_str(Some("BACKFILL"), 5).unwrap();
assert!(matches!(
upper,
CrawlerModePref::Explicit(DiscoverMode::Backfill)
));
}
#[test]
fn parse_mode_str_hard_errors_on_unknown_value() {
let err = parse_mode_str(Some("backfil"), 20).unwrap_err();
let msg = format!("{err}");
assert!(msg.contains("backfill"), "error should list valid values: {msg}");
assert!(msg.contains("auto"));
assert!(msg.contains("incremental"));
}
}

View File

@@ -145,6 +145,28 @@ impl BrowserManager {
}
}
/// Mark the cached browser handle as unhealthy. The next `acquire`
/// will re-launch Chromium from scratch.
///
/// Same semantics as `shutdown` — the difference is intent:
/// `shutdown` runs once at daemon teardown, while `invalidate` is a
/// recovery hook callers fire after a CDP / connection / navigation
/// failure that suggests the underlying process has died. Calling
/// this while other workers still hold leases is safe — their
/// outstanding CDP operations will return channel-closed errors
/// and those workers will then re-acquire (re-launching Chromium).
///
/// Idempotent: calling on an already-invalidated manager is a
/// no-op.
pub async fn invalidate(&self) {
let mut guard = self.inner.lock().await;
guard.shared = None;
if let Some(handle) = guard.handle.take() {
let _ = handle.close().await;
tracing::warn!("BrowserManager: handle invalidated — next acquire will relaunch");
}
}
fn idle_timeout(&self) -> Duration {
self.idle_timeout
}
@@ -231,6 +253,23 @@ mod tests {
assert_send_sync(&h);
}
/// Invalidate is the only `BrowserManager` method that's safe to
/// exercise in a unit test without launching Chromium — it's a
/// no-op when no handle has been cached, and that path is exactly
/// the one we want to verify is idempotent.
#[tokio::test]
async fn invalidate_is_a_noop_when_no_handle_cached() {
let mgr = BrowserManager::new(
crate::crawler::browser::LaunchOptions::default(),
Duration::ZERO,
noop_on_launch(),
);
// Two back-to-back invalidates must both complete; the second
// would hang or panic if the first had left torn state.
mgr.invalidate().await;
mgr.invalidate().await;
}
#[tokio::test]
async fn active_tracker_signals_idle_only_on_zero_transition() {
let tracker = ActiveTracker::new();

View File

@@ -111,7 +111,19 @@ pub async fn sync_chapter_content(
.new_page(source_url)
.await
.with_context(|| format!("open chapter page {source_url}"))?;
page.wait_for_navigation().await.context("wait for chapter nav")?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for chapter nav")?;
// Best-effort wait for the reader marker — same partial-render
// race that bit the chapter-list parser can hit here. Timeout is
// not an error; the chapter probe + parser sentinels still catch
// real failures.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"a#pic_container",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
let html = page.content().await.context("read chapter html")?;
page.close().await.ok();

View File

@@ -233,23 +233,37 @@ impl CronContext {
return;
}
match self.metadata.run().await {
Ok(stats) => tracing::info!(?stats, "cron: metadata pass done"),
Err(e) => tracing::error!(?e, "cron: metadata pass failed"),
}
match pipeline::enqueue_bookmarked_pending(&self.pool).await {
Ok(summary) => tracing::info!(?summary, "cron: enqueued bookmarked-pending"),
Err(e) => tracing::error!(?e, "cron: enqueue_bookmarked_pending failed"),
}
match jobs::reap_done(&self.pool, self.retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: done-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: done-job reaper failed"),
}
if let Err(e) = write_last_tick(&self.pool, Utc::now()).await {
tracing::warn!(?e, "cron: persist last_metadata_tick_at failed");
// Panic-isolate the tick body the same way `process_lease` does
// for worker dispatch. Without this, a panic in metadata.run
// (or any of the follow-on steps) would kill the cron task and
// no future tick would ever run — workers would keep going but
// no new metadata work would be scheduled until daemon restart.
// The advisory unlock below runs unconditionally so a panicked
// tick doesn't leave the lock held for another replica.
let metadata = &self.metadata;
let pool = &self.pool;
let retention_days = self.retention_days;
let body = async move {
match metadata.run().await {
Ok(stats) => tracing::info!(?stats, "cron: metadata pass done"),
Err(e) => tracing::error!(?e, "cron: metadata pass failed"),
}
match pipeline::enqueue_bookmarked_pending(pool).await {
Ok(summary) => {
tracing::info!(?summary, "cron: enqueued bookmarked-pending");
}
Err(e) => tracing::error!(?e, "cron: enqueue_bookmarked_pending failed"),
}
match jobs::reap_done(pool, retention_days).await {
Ok(n) => tracing::info!(reaped = n, "cron: done-job reaper finished"),
Err(e) => tracing::error!(?e, "cron: done-job reaper failed"),
}
if let Err(e) = write_last_tick(pool, Utc::now()).await {
tracing::warn!(?e, "cron: persist last_metadata_tick_at failed");
}
};
if let Err(_panic) = AssertUnwindSafe(body).catch_unwind().await {
tracing::error!("cron: tick body panicked — continuing");
}
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
@@ -626,4 +640,19 @@ mod tests {
let prev = previous_fire(now, at, Tz::UTC);
assert_eq!(prev, dt_utc(2026, 5, 24, 23, 30));
}
/// Documents the panic-isolation pattern `run_tick` now relies on:
/// `AssertUnwindSafe(...).catch_unwind().await` must yield `Err(_)`
/// when the wrapped future panics, so the surrounding loop (or in
/// our case, the unconditional advisory-unlock that follows) keeps
/// running. The shape of this test mirrors the production callsite.
#[tokio::test]
async fn assert_unwind_safe_catches_a_panicking_future() {
let result = AssertUnwindSafe(async {
panic!("boom");
})
.catch_unwind()
.await;
assert!(result.is_err(), "panicking future must yield Err");
}
}

View File

@@ -1,4 +1,4 @@
//! Persistent job queue and the four job kinds.
//! Persistent job queue and its job kinds.
//!
//! Backed by Postgres (the `crawler_jobs` table). Workers lease rows
//! with `SELECT ... FOR UPDATE SKIP LOCKED`, heartbeat via
@@ -12,16 +12,9 @@ use serde::{Deserialize, Serialize};
use sqlx::PgPool;
use uuid::Uuid;
use super::source::DiscoverMode;
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(tag = "kind", rename_all = "snake_case")]
pub enum JobPayload {
/// Walk the source index and enqueue `SyncManga` jobs.
Discover {
source_id: String,
mode: DiscoverMode,
},
/// Fetch one manga's detail page, upsert metadata, enqueue
/// `SyncChapterList`.
SyncManga {
@@ -160,23 +153,36 @@ pub async fn lease(
Ok(leases)
}
/// Mark a leased job as successfully completed.
/// Mark a leased job as successfully completed. The `state = 'running'`
/// predicate guards against a late ack from a worker whose lease expired
/// and was already re-leased by another worker: without it, the late ack
/// would clobber the new lease's `state` and `leased_until`. `rows_affected
/// == 0` means we lost the lease — surfaced as a warn rather than an
/// error because the new lease holder is doing real work; the late ack
/// just has to step aside.
pub async fn ack_done(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
sqlx::query(
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'done', leased_until = NULL, updated_at = now() \
WHERE id = $1",
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.execute(pool)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"ack_done: lease no longer running — likely re-leased by another worker; skipping update"
);
}
Ok(())
}
/// Mark a leased job as failed. If the current attempt count has reached
/// `max_attempts` the job is terminally dead and stops retrying;
/// otherwise it goes back to `pending` with `scheduled_at` pushed into
/// the future by the exponential backoff.
/// the future by the exponential backoff. See [`ack_done`] for the
/// `state = 'running'` guard rationale.
pub async fn ack_failed(
pool: &PgPool,
lease_id: Uuid,
@@ -184,16 +190,16 @@ pub async fn ack_failed(
attempts: i32,
max_attempts: i32,
) -> sqlx::Result<()> {
if attempts >= max_attempts {
let res = if attempts >= max_attempts {
sqlx::query(
"UPDATE crawler_jobs \
SET state = 'dead', last_error = $2, leased_until = NULL, updated_at = now() \
WHERE id = $1",
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.bind(error)
.execute(pool)
.await?;
.await?
} else {
let backoff_ms: i64 = backoff_for(attempts).as_millis().min(i64::MAX as u128) as i64;
sqlx::query(
@@ -201,30 +207,45 @@ pub async fn ack_failed(
SET state = 'pending', last_error = $2, leased_until = NULL, \
scheduled_at = now() + ($3::bigint || ' milliseconds')::interval, \
updated_at = now() \
WHERE id = $1",
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.bind(error)
.bind(backoff_ms)
.execute(pool)
.await?;
.await?
};
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"ack_failed: lease no longer running — likely re-leased by another worker; skipping update"
);
}
Ok(())
}
/// Return a leased job to `pending` without burning a retry attempt.
/// Used on graceful shutdown and on session-expired aborts where the
/// failure isn't the job's fault.
/// failure isn't the job's fault. See [`ack_done`] for the
/// `state = 'running'` guard rationale — important here because
/// `attempts - 1` would otherwise spuriously decrement the new lease's
/// attempt count.
pub async fn release(pool: &PgPool, lease_id: Uuid) -> sqlx::Result<()> {
sqlx::query(
let res = sqlx::query(
"UPDATE crawler_jobs \
SET state = 'pending', leased_until = NULL, \
attempts = GREATEST(0, attempts - 1), updated_at = now() \
WHERE id = $1",
WHERE id = $1 AND state = 'running'",
)
.bind(lease_id)
.execute(pool)
.await?;
if res.rows_affected() == 0 {
tracing::warn!(
%lease_id,
"release: lease no longer running — likely re-leased by another worker; skipping update"
);
}
Ok(())
}

View File

@@ -20,6 +20,7 @@ pub mod daemon;
pub mod detect;
pub mod diff;
pub mod jobs;
pub mod nav;
pub mod pipeline;
pub mod rate_limit;
pub mod safety;

241
backend/src/crawler/nav.rs Normal file
View File

@@ -0,0 +1,241 @@
//! Page navigation helpers — wrap `chromiumoxide` `wait_for_navigation`
//! with a timeout so a hung TLS handshake or a page that never fires
//! `load` cannot wedge a worker (or the cron metadata pass) forever.
//!
//! [`NAV_TIMEOUT`] is the global budget. Callers in the crawler use
//! [`wait_for_nav`] to get back a typed error so transient timeouts can
//! be reported separately from underlying CDP errors.
use std::time::Duration;
use chromiumoxide::error::CdpError;
use chromiumoxide::Page;
use thiserror::Error;
/// Maximum wall-clock time we'll wait for a single page navigation. A
/// healthy Chromium reaches `load` in well under a second on the target
/// site; a 30-second cap is generous enough for slow TLS handshakes on
/// the first request after a fresh process while still catching real
/// hangs before they wedge the daemon.
pub const NAV_TIMEOUT: Duration = Duration::from_secs(30);
/// Outcome of a timed-out navigation. `Timeout` is the transient signal
/// callers translate into a retry-friendly error
/// ([`crate::crawler::detect::PageError::Transient`] in the source path,
/// a context'd anyhow elsewhere). `Cdp` carries the underlying
/// chromiumoxide error unchanged.
#[derive(Debug, Error)]
pub enum NavError {
#[error("navigation timed out after {0:?}")]
Timeout(Duration),
#[error(transparent)]
Cdp(#[from] CdpError),
}
/// Wait for the page's next navigation to complete, capped at
/// [`NAV_TIMEOUT`]. Replaces bare `page.wait_for_navigation().await`
/// throughout the crawler.
pub async fn wait_for_nav(page: &Page) -> Result<(), NavError> {
match tokio::time::timeout(NAV_TIMEOUT, page.wait_for_navigation()).await {
Err(_elapsed) => Err(NavError::Timeout(NAV_TIMEOUT)),
Ok(Err(e)) => Err(NavError::Cdp(e)),
Ok(Ok(_)) => Ok(()),
}
}
/// Poll interval for [`wait_for_selector`]. 100ms is fast enough that a
/// page rendering in 200ms isn't held back noticeably, and slow enough
/// not to spam CDP with `find_element` calls on a page that's actually
/// taking its time.
const SELECTOR_POLL_INTERVAL: Duration = Duration::from_millis(100);
/// Wait until `selector` matches at least one element on `page`, or
/// `timeout` elapses. Used after a navigation to confirm a page-type-
/// specific marker is in the DOM before parsing — replaces the fixed
/// post-nav sleep that previously masked partial-render races.
///
/// chromiumoxide 0.7.0 has no built-in `wait_for_selector`, so we poll
/// `find_element` at [`SELECTOR_POLL_INTERVAL`] until success or budget
/// exhaustion. A failed `find_element` is *not* an error here — it just
/// means "not yet" — we only surface an error once the overall
/// `timeout` is up.
pub async fn wait_for_selector(
page: &Page,
selector: &str,
timeout: Duration,
) -> Result<(), NavError> {
let deadline = tokio::time::Instant::now() + timeout;
loop {
if page.find_element(selector).await.is_ok() {
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(NavError::Timeout(timeout));
}
let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
let sleep_for = SELECTOR_POLL_INTERVAL.min(remaining);
tokio::time::sleep(sleep_for).await;
}
}
/// Per-page-type budget for [`wait_for_selector`]. Shorter than
/// [`NAV_TIMEOUT`] because by the time we're waiting on a selector, the
/// page has already responded — we're only absorbing post-load JS
/// finishing its row injection, which on a healthy site takes well
/// under a second.
pub const SELECTOR_TIMEOUT: Duration = Duration::from_secs(10);
impl NavError {
/// Does this navigation error indicate the underlying Chromium
/// process has died or its CDP connection has dropped? Used by the
/// dispatcher to decide whether to invalidate the
/// [`crate::crawler::browser_manager::BrowserManager`] handle so
/// the next acquire re-launches.
///
/// Both variants count: a `Timeout` past [`NAV_TIMEOUT`] is in
/// practice always either a hung CDP transport or a wedged page
/// the browser can't recover from on its own, and a `Cdp` error
/// surfacing at the navigation layer means the chromium-facing
/// channel is the failing layer.
pub fn is_likely_browser_dead(&self) -> bool {
match self {
Self::Timeout(_) => true,
Self::Cdp(_) => true,
}
}
}
/// Walk an `anyhow::Error` chain looking for typed evidence that the
/// chromium-facing layer is the failing one. Two markers count:
///
/// 1. A wrapped [`NavError`] flagged by [`NavError::is_likely_browser_dead`].
/// 2. A wrapped [`CdpError`] (via `anyhow::Error::from(CdpError)` at a
/// `Browser::new_page` call site, or any other direct CDP boundary).
///
/// Earlier versions also substring-matched the chain for "connection",
/// "closed", "channel", etc. as a fallback. That was too broad —
/// reqwest TCP-reset errors during CDN image downloads, sqlx
/// connection-pool errors, and similar non-browser failures contain
/// those words and triggered spurious chromium relaunches. The typed
/// downcasts cover every place we hand a chromium error to anyhow,
/// so the fallback is unnecessary.
pub fn anyhow_looks_browser_dead(err: &anyhow::Error) -> bool {
for cause in err.chain() {
if let Some(nav) = cause.downcast_ref::<NavError>() {
if nav.is_likely_browser_dead() {
return true;
}
}
if cause.downcast_ref::<CdpError>().is_some() {
return true;
}
}
false
}
#[cfg(test)]
mod tests {
use super::*;
use std::future::pending;
/// Sanity-check the timeout pattern used by [`wait_for_nav`]: a
/// future that never resolves must yield `Elapsed` within the
/// configured budget. We can't easily stand up a real `Page` in a
/// unit test, so we assert the underlying primitive behaves the way
/// the helper depends on.
#[tokio::test(flavor = "current_thread", start_paused = true)]
async fn timeout_elapses_on_a_future_that_never_resolves() {
let result =
tokio::time::timeout(Duration::from_millis(50), pending::<()>()).await;
assert!(result.is_err(), "expected Elapsed on a hung future");
}
#[test]
fn nav_error_timeout_message_includes_duration() {
let e = NavError::Timeout(Duration::from_secs(30));
assert_eq!(e.to_string(), "navigation timed out after 30s");
}
#[test]
fn timeout_is_treated_as_likely_browser_dead() {
let e = NavError::Timeout(NAV_TIMEOUT);
assert!(e.is_likely_browser_dead());
}
#[test]
fn anyhow_with_nav_timeout_in_chain_is_flagged() {
let inner: Result<(), NavError> = Err(NavError::Timeout(NAV_TIMEOUT));
let outer = inner.unwrap_err();
let wrapped: anyhow::Error =
anyhow::Error::new(outer).context("wait for chapter nav");
assert!(anyhow_looks_browser_dead(&wrapped));
}
#[test]
fn anyhow_with_cdp_error_in_chain_is_flagged() {
// `Browser::new_page` errors get wrapped via
// `anyhow::Error::from(CdpError)` at the navigate / dispatch
// call sites. Walking the chain and downcasting to CdpError is
// what catches that path. Any CdpError variant counts; the
// Serde variant is the easiest to construct in a unit test.
let serde_err: serde_json::Error =
serde_json::from_str::<i32>("not a number").unwrap_err();
let cdp = CdpError::Serde(serde_err);
let wrapped: anyhow::Error =
anyhow::Error::from(cdp).context("open chapter page");
assert!(anyhow_looks_browser_dead(&wrapped));
}
#[test]
fn anyhow_with_innocuous_parse_error_is_not_flagged() {
let e: anyhow::Error =
anyhow::anyhow!("parse manga detail: chapter row regex did not match");
assert!(!anyhow_looks_browser_dead(&e));
}
#[test]
fn anyhow_with_reqwest_style_connection_message_is_not_flagged() {
// Regression: the earlier substring fallback flagged any error
// whose message contained "connection" or "closed" as browser-
// dead. A TCP reset from a CDN during image download, or a
// sqlx pool-connection error, would burn a chromium relaunch
// even though the browser is fine. Typed downcasts only —
// these untyped strings must pass through.
for msg in [
"error sending request: connection reset by peer",
"PoolTimedOut: timed out waiting for a connection",
"request to https://cdn/x.jpg: connection closed before message completed",
"transport error during image fetch",
] {
let e: anyhow::Error = anyhow::anyhow!("{msg}");
assert!(
!anyhow_looks_browser_dead(&e),
"must not flag non-browser error: {msg}"
);
}
}
/// Same sanity check as [`timeout_elapses_on_a_future_that_never_resolves`],
/// but for the [`wait_for_selector`] polling pattern: the loop must
/// surrender on `Elapsed` rather than spinning past the deadline.
#[tokio::test(flavor = "current_thread", start_paused = true)]
async fn selector_polling_pattern_surrenders_at_deadline() {
let timeout = Duration::from_millis(300);
let start = tokio::time::Instant::now();
let deadline = start + timeout;
// Simulate find_element forever returning "not found".
let mut polls = 0u32;
let result: Result<(), NavError> = loop {
polls += 1;
if tokio::time::Instant::now() >= deadline {
break Err(NavError::Timeout(timeout));
}
tokio::time::sleep(SELECTOR_POLL_INTERVAL).await;
};
assert!(matches!(result, Err(NavError::Timeout(_))));
// 300ms / 100ms poll interval ≈ 3 iterations plus the final check
// that breaks out. Allow some slack since the first poll happens
// before any sleep.
assert!(polls >= 3, "expected at least 3 poll iterations, got {polls}");
}
}

View File

@@ -2,6 +2,8 @@
//! that fan out chapter-content work. Shared between the daemon (cron tick)
//! and the CLI (`bin/crawler.rs`) so behavior stays in lockstep.
use std::collections::HashSet;
use anyhow::Context;
use sqlx::PgPool;
use uuid::Uuid;
@@ -11,8 +13,9 @@ use crate::crawler::jobs::{self, EnqueueResult, JobPayload};
use crate::crawler::rate_limit::HostRateLimiters;
use crate::crawler::safety::{fetch_bytes_capped, looks_like_image, DownloadAllowlist};
use crate::crawler::source::target::TargetSource;
use crate::crawler::source::{DiscoverMode, FetchContext, Source};
use crate::crawler::source::{FetchContext, Source};
use crate::repo;
use crate::repo::crawler::UpsertStatus;
use crate::storage::Storage;
/// Coarse counters surfaced for logging at the end of a metadata pass.
@@ -24,16 +27,42 @@ pub struct MetadataStats {
pub mangas_failed: usize,
}
/// Decide whether the per-ref loop should stop based on the Incremental
/// streak counter. Pulled out as a pure function so the rule is unit-
/// testable without standing up the walker or DB.
pub(crate) fn should_stop(mode: DiscoverMode, consecutive_unchanged: usize) -> bool {
match mode {
DiscoverMode::Backfill => false,
DiscoverMode::Incremental { stop_after_unchanged } => {
consecutive_unchanged >= stop_after_unchanged
}
}
/// Decide whether the per-ref loop should stop on the manga just
/// processed. The walk halts only when (a) the previous run exited
/// cleanly — so the index tail is known to be caught up and we're not
/// in a recovery sweep — AND (b) this manga's metadata hash matched
/// storage (`Unchanged`) AND (c) the chapter sync confirmed zero new
/// chapters. A `None` chapter count (skip_chapters, or a chapter-sync
/// error we logged-and-swallowed) refuses the stop because we can't
/// verify the tail is unchanged from a single piece of evidence.
///
/// Pure function so the rule is unit-testable without the walker, DB,
/// or browser.
pub(crate) fn should_stop(
was_clean: bool,
status: UpsertStatus,
chapters_new: Option<usize>,
) -> bool {
was_clean
&& matches!(status, UpsertStatus::Unchanged)
&& chapters_new == Some(0)
}
/// Whether the just-finished walk should be recorded as a clean exit.
/// `true` writes the recovery flag back to `completed: true`; `false`
/// leaves it `false` so the next tick treats this run as crashed and
/// does a recovery sweep.
///
/// `hit_limit` (the caller-imposed `CRAWLER_LIMIT` cap) is *not* an
/// argument: a limit cap by definition does not reach the catalog tail,
/// so it can never count as a clean exit. Encoding that in the type
/// (rather than as an `&& !hit_limit` clause inline) prevents a future
/// edit from accidentally adding it back to the truth table.
pub(crate) fn should_mark_clean_exit(
walked_to_completion: bool,
hit_stop_condition: bool,
) -> bool {
walked_to_completion || hit_stop_condition
}
/// Runs the discover → fetch → upsert → cover → chapter-list-diff pipeline
@@ -43,15 +72,25 @@ pub(crate) fn should_stop(mode: DiscoverMode, consecutive_unchanged: usize) -> b
/// `limit == 0` means no cap (full sweep up to the source's own bound).
/// `skip_chapters == true` is the "metadata-only" mode (parser doesn't
/// extract chapters, and `sync_manga_chapters` is skipped — otherwise an
/// empty chapter list would soft-drop existing rows).
/// empty chapter list would soft-drop existing rows). In this mode the
/// stop condition never fires because chapter freshness can't be
/// confirmed, so the walk always runs to end-of-source.
///
/// `mode` controls the walk:
/// - `Backfill` — oldest-first, no early exit. The only mode that runs
/// the end-of-walk drop pass + writes `seed_completed_at`.
/// - `Incremental { stop_after_unchanged }` — newest-first, breaks out
/// after N consecutive Unchanged upserts. Drop pass is skipped (the
/// tail of the index is never visited, so its `last_seen_at` is
/// stale and using it to soft-drop would be unsafe).
/// The walk is always newest-first. Steady-state runs stop on the first
/// manga where metadata is `Unchanged` AND chapter sync reports zero
/// new chapters — the source orders by `update_date DESC`, so anything
/// with a fresh chapter or fresh metadata is bumped to the top and will
/// be processed before we hit a fully-caught-up manga.
///
/// A per-source recovery flag stored in `crawler_state`
/// (`last_run_completed:<source_id>`) gates the early stop: it's set to
/// `false` right after `ensure_source` and back to `true` only when the
/// run exits via end-of-walk OR the intentional stop. A crash, panic,
/// or SIGKILL leaves the flag at `false`, so the next tick reads it,
/// recognizes the previous run did not exit cleanly, and walks the
/// full catalog (ignoring the stop condition) to re-cover anything the
/// crashed run missed past its crash point. Once that recovery sweep
/// reaches end-of-walk, steady-state resumes.
#[allow(clippy::too_many_arguments)]
pub async fn run_metadata_pass(
browser_manager: &BrowserManager,
@@ -62,7 +101,6 @@ pub async fn run_metadata_pass(
start_url: &str,
limit: usize,
skip_chapters: bool,
mode: DiscoverMode,
allowlist: &DownloadAllowlist,
max_image_bytes: usize,
) -> anyhow::Result<MetadataStats> {
@@ -95,20 +133,36 @@ pub async fn run_metadata_pass(
.await
.context("ensure_source")?;
let run_started_at = chrono::Utc::now();
// Read BEFORE flipping to "in-flight" — a `false` here means the
// previous run didn't reach a clean exit, and this run must walk
// the full catalog (recovery sweep) instead of bailing on the
// first caught-up manga.
let was_clean = repo::crawler::last_run_completed_cleanly(db, source_id)
.await
.context("read last_run_completed_cleanly")?;
repo::crawler::mark_run_started(db, source_id)
.await
.context("mark_run_started")?;
let max_refs = (limit > 0).then_some(limit);
tracing::info!(?mode, ?max_refs, "starting metadata pass");
tracing::info!(was_clean, ?max_refs, "starting metadata pass");
let mut walker = source
.discover(&ctx, mode)
.discover(&ctx)
.await
.context("discover failed")?;
let mut stats = MetadataStats::default();
let mut consecutive_unchanged: usize = 0;
// Run-scoped dedup of `source_manga_key`s already processed this pass.
// A shift in the source index causes the slot-last item of the page
// we just read to reappear at slot 0 of the next page; skipping it
// here prevents redundant fetch_manga + upsert and avoids spuriously
// tripping the stop condition with a re-confirm of an entry we
// already counted.
let mut seen: HashSet<String> = HashSet::new();
let mut walked_to_completion = false;
let mut hit_limit = false;
let mut hit_incremental_stop = false;
let mut hit_stop_condition = false;
'outer: loop {
let batch = match walker.next_batch(&ctx).await? {
@@ -124,6 +178,23 @@ pub async fn run_metadata_pass(
tracing::info!(cap = ?max_refs, "max_results reached; halting walk");
break 'outer;
}
// Skip refs we've already *successfully* processed this pass.
// Checking `contains` here (rather than `insert`) keeps the key
// out of `seen` on failure paths below, so a transient fetch or
// upsert error gets a second chance if the ref reappears in
// another batch. Done *before* counting toward
// `stats.discovered` (the skipped ref did no work) and *before*
// touching the stop check (a `continue` here doesn't let a
// re-confirm trip the stop condition). The matching
// `seen.insert(...)` lives just after the successful upsert
// below.
if seen.contains(&r.source_manga_key) {
tracing::debug!(
key = %r.source_manga_key,
"skip already-seen key in this run"
);
continue;
}
stats.discovered += 1;
tracing::info!(
idx = stats.discovered,
@@ -144,6 +215,48 @@ pub async fn run_metadata_pass(
}
};
// Partial-render guard: an empty chapter list paired with a
// prior count > 0 is overwhelmingly a chromium snapshot
// taken between the #chapter_table wrapper render and its
// rows render. The wait_for_selector wait in `navigate`
// narrows this window but cannot close it for slow renders
// beyond the selector budget. Treat as a transient failure
// here — skip upsert, skip seen.insert — so the next batch
// (or the next tick) retries. Skipped in `skip_chapters`
// mode because the parser is configured to return an empty
// Vec by design there.
if !skip_chapters && manga.chapters.is_empty() {
match repo::crawler::live_chapter_count_for_source_manga(
db, source_id, &r.source_manga_key,
)
.await
{
Ok(prior) if prior > 0 => {
tracing::warn!(
key = %r.source_manga_key,
url = %r.url,
prior_chapter_count = prior,
"fetch_manga returned empty chapters but prior count > 0; treating as partial-render transient and skipping"
);
stats.mangas_failed += 1;
continue;
}
Ok(_) => {}
Err(e) => {
// DB lookup failed — fail safe: skip rather
// than risk a soft-drop on a manga whose prior
// count we couldn't confirm.
tracing::warn!(
key = %r.source_manga_key,
error = ?e,
"live_chapter_count_for_source_manga failed; skipping cautiously"
);
stats.mangas_failed += 1;
continue;
}
}
}
let upsert = match repo::crawler::upsert_manga_from_source(
db, source_id, &r.url, &manga,
)
@@ -161,6 +274,10 @@ pub async fn run_metadata_pass(
}
};
stats.upserted += 1;
// Record success in the dedup set. Cover and chapter-sync
// failures below are non-fatal and don't roll this back —
// metadata is the durable source of truth for the dedup.
seen.insert(r.source_manga_key.clone());
tracing::info!(
key = %manga.source_manga_key,
manga_id = %upsert.manga_id,
@@ -199,7 +316,13 @@ pub async fn run_metadata_pass(
}
}
if !skip_chapters {
// Chapter sync. `chapters_new` feeds the stop check below:
// `None` (skip_chapters mode, or a logged-and-swallowed sync
// error) refuses to stop on this manga because we can't
// confirm "no new chapters."
let chapters_new: Option<usize> = if skip_chapters {
None
} else {
match repo::crawler::sync_manga_chapters(
db,
source_id,
@@ -208,79 +331,64 @@ pub async fn run_metadata_pass(
)
.await
{
Ok(diff) => tracing::info!(
manga_id = %upsert.manga_id,
new = diff.new,
refreshed = diff.refreshed,
dropped = diff.dropped,
"chapters synced"
),
Err(e) => tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"chapter sync failed"
),
Ok(diff) => {
tracing::info!(
manga_id = %upsert.manga_id,
new = diff.new,
refreshed = diff.refreshed,
dropped = diff.dropped,
"chapters synced"
);
Some(diff.new)
}
Err(e) => {
tracing::warn!(
manga_id = %upsert.manga_id,
error = ?e,
"chapter sync failed"
);
None
}
}
}
};
// Incremental stop: count consecutive Unchanged upserts and
// bail once the threshold is reached. New/Updated resets the
// streak so a fresh entry mid-batch doesn't accidentally trip
// the cutoff.
match upsert.status {
repo::crawler::UpsertStatus::Unchanged => {
consecutive_unchanged += 1;
}
repo::crawler::UpsertStatus::New | repo::crawler::UpsertStatus::Updated => {
consecutive_unchanged = 0;
}
}
if should_stop(mode, consecutive_unchanged) {
hit_incremental_stop = true;
if should_stop(was_clean, upsert.status, chapters_new) {
hit_stop_condition = true;
tracing::info!(
consecutive_unchanged,
"incremental stop threshold reached; halting walk"
key = %manga.source_manga_key,
"stop condition met (Unchanged metadata + 0 new chapters); halting walk"
);
break 'outer;
}
}
}
// Drop pass: only when the walk truly covered everything the source
// surfaces. `last_seen_at` on un-visited rows is stale, so running
// the drop on a partial walk would soft-drop the tail of the index.
let full_walk = walked_to_completion && !hit_limit && !hit_incremental_stop;
let backfill_complete = full_walk && matches!(mode, DiscoverMode::Backfill);
if full_walk {
match repo::crawler::mark_dropped_mangas(db, source_id, run_started_at).await {
Ok(n) => tracing::info!(dropped = n, "marked unseen manga as dropped"),
Err(e) => tracing::warn!(error = ?e, "drop-pass failed"),
}
} else {
tracing::info!(
?mode,
hit_limit,
hit_incremental_stop,
"partial sync — skipping drop pass"
);
}
if backfill_complete {
if let Err(e) = repo::crawler::mark_seed_completed(db, source_id, run_started_at).await {
tracing::warn!(error = ?e, "mark_seed_completed failed");
} else {
tracing::info!(source_id, "seed marked complete");
// Recovery-flag write. Only on a clean exit (end-of-walk OR the
// intentional stop). `hit_limit` is a caller-imposed early break
// and does NOT count — the catalog tail wasn't reached, so a future
// tick still needs to walk past where we stopped. The truth table is
// pinned by `should_mark_clean_exit` so a future edit that adds
// `hit_limit` back into the disjunction trips its unit test. Flag-
// write errors are warned and swallowed: the run already did its
// work, and a stale `false` flag just buys a recovery sweep on the
// next tick.
let exited_cleanly = should_mark_clean_exit(walked_to_completion, hit_stop_condition);
if exited_cleanly {
if let Err(e) = repo::crawler::mark_run_completed(db, source_id).await {
tracing::warn!(error = ?e, "mark_run_completed failed");
}
}
tracing::info!(
?mode,
was_clean,
discovered = stats.discovered,
upserted = stats.upserted,
covers_fetched = stats.covers_fetched,
mangas_failed = stats.mangas_failed,
walked_to_completion,
hit_limit,
hit_incremental_stop,
hit_stop_condition,
exited_cleanly,
"metadata pass complete"
);
@@ -288,8 +396,20 @@ pub async fn run_metadata_pass(
Ok(stats)
}
/// Quarantine window for chapters whose latest `SyncChapterContent` job is
/// `dead`. The partial dedup index `crawler_jobs_chapter_content_dedup_idx`
/// only blocks `(pending|running)` duplicates, so without this gate a
/// permanently-failing chapter is re-enqueued every cron tick, burns
/// `max_attempts` retries, dies again, and spins forever. With the gate,
/// dead chapters get a week of silence before the next attempt — long
/// enough for a transient site issue to resolve, short enough that
/// permanent failures don't stay permanent if conditions change.
const CHAPTER_DEAD_QUARANTINE_DAYS: i64 = 7;
/// Enqueue a `SyncChapterContent` job for every chapter of *any* bookmarked
/// manga that still has `page_count = 0` and a non-dropped source row.
/// Chapters whose latest job is `dead` within `CHAPTER_DEAD_QUARANTINE_DAYS`
/// are excluded to break the dead-letter spin.
/// Returns `(inserted, skipped)` counts. Dedup index handles repeats.
pub async fn enqueue_bookmarked_pending(pool: &PgPool) -> anyhow::Result<EnqueueSummary> {
let rows: Vec<(String, Uuid, String)> = sqlx::query_as(
@@ -300,10 +420,18 @@ pub async fn enqueue_bookmarked_pending(pool: &PgPool) -> anyhow::Result<Enqueue
JOIN chapter_sources cs ON cs.chapter_id = c.id
WHERE c.page_count = 0
AND cs.dropped_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM crawler_jobs cj
WHERE cj.payload->>'kind' = 'sync_chapter_content'
AND cj.payload->>'chapter_id' = c.id::text
AND cj.state = 'dead'
AND cj.updated_at > now() - ($1::bigint || ' days')::interval
)
GROUP BY cs.source_id, c.id, cs.source_chapter_key, c.manga_id, c.created_at
ORDER BY c.manga_id, c.created_at ASC
"#,
)
.bind(CHAPTER_DEAD_QUARANTINE_DAYS)
.fetch_all(pool)
.await
.context("query bookmarked-pending chapters")?;
@@ -332,7 +460,9 @@ pub async fn enqueue_bookmarked_pending(pool: &PgPool) -> anyhow::Result<Enqueue
}
/// Enqueue chapter-content jobs for a *single* manga (the bookmark-create
/// hook). Same dedup semantics as [`enqueue_bookmarked_pending`].
/// hook). Same dedup semantics as [`enqueue_bookmarked_pending`], including
/// the dead-letter quarantine — a freshly bookmarked manga should not
/// burn retries on chapters that just died on the cron tick.
pub async fn enqueue_pending_for_manga(
pool: &PgPool,
manga_id: Uuid,
@@ -345,10 +475,18 @@ pub async fn enqueue_pending_for_manga(
WHERE c.manga_id = $1
AND c.page_count = 0
AND cs.dropped_at IS NULL
AND NOT EXISTS (
SELECT 1 FROM crawler_jobs cj
WHERE cj.payload->>'kind' = 'sync_chapter_content'
AND cj.payload->>'chapter_id' = c.id::text
AND cj.state = 'dead'
AND cj.updated_at > now() - ($2::bigint || ' days')::interval
)
ORDER BY cs.source_id, c.id
"#,
)
.bind(manga_id)
.bind(CHAPTER_DEAD_QUARANTINE_DAYS)
.fetch_all(pool)
.await
.context("query pending chapters for manga")?;
@@ -447,30 +585,105 @@ mod tests {
use super::*;
#[test]
fn backfill_never_stops_regardless_of_streak() {
assert!(!should_stop(DiscoverMode::Backfill, 0));
assert!(!should_stop(DiscoverMode::Backfill, 100));
assert!(!should_stop(DiscoverMode::Backfill, usize::MAX));
fn stop_condition_fires_on_unchanged_metadata_and_zero_new_chapters() {
// The whole point of the rule: in steady state, a manga whose
// metadata hash matches AND whose chapter list gained no new
// entries proves we've reached the caught-up tail of a
// newest-first index.
assert!(should_stop(true, UpsertStatus::Unchanged, Some(0)));
}
#[test]
fn incremental_stops_when_streak_meets_threshold() {
let mode = DiscoverMode::Incremental {
stop_after_unchanged: 3,
};
assert!(!should_stop(mode, 0));
assert!(!should_stop(mode, 2));
assert!(should_stop(mode, 3), "stops at exactly the threshold");
assert!(should_stop(mode, 100), "stops at anything past threshold");
fn stop_condition_refuses_when_chapters_added() {
// Unchanged metadata + N new chapters means the source bumped
// this manga because of the chapter add; the rest of the index
// is still ahead of us. Don't bail.
assert!(!should_stop(true, UpsertStatus::Unchanged, Some(1)));
assert!(!should_stop(true, UpsertStatus::Unchanged, Some(42)));
}
#[test]
fn incremental_with_zero_threshold_stops_immediately() {
// A nonsensical config (no Unchanged needed to stop) shouldn't
// panic — it just means the very first ref triggers the bail.
let mode = DiscoverMode::Incremental {
stop_after_unchanged: 0,
};
assert!(should_stop(mode, 0));
fn stop_condition_refuses_when_metadata_changed() {
// Updated or New metadata always continues — even with zero new
// chapters — because the change-of-metadata bump itself is what
// the walk is following.
assert!(!should_stop(true, UpsertStatus::Updated, Some(0)));
assert!(!should_stop(true, UpsertStatus::New, Some(0)));
}
#[test]
fn stop_condition_refuses_when_chapter_count_unknown() {
// skip_chapters mode (CLI metadata-only sweep) or a
// logged-and-swallowed chapter sync error: we can't claim "no
// new chapters" from absence of evidence, so don't stop. The
// operator who runs metadata-only intentionally wants a full
// walk anyway.
assert!(!should_stop(true, UpsertStatus::Unchanged, None));
}
#[test]
fn stop_condition_disabled_in_recovery_mode() {
// was_clean = false means the previous run did not exit cleanly;
// the catalog past its crash point is potentially un-synced. Walk
// to end-of-source no matter what individual mangas report.
assert!(!should_stop(false, UpsertStatus::Unchanged, Some(0)));
assert!(!should_stop(false, UpsertStatus::Unchanged, Some(1)));
assert!(!should_stop(false, UpsertStatus::Updated, Some(0)));
assert!(!should_stop(false, UpsertStatus::New, None));
}
#[test]
fn clean_exit_when_walked_to_completion() {
// End-of-walk reached the catalog tail — the recovery flag may
// safely flip back to `true`.
assert!(should_mark_clean_exit(true, false));
}
#[test]
fn clean_exit_when_stop_condition_fired() {
// First Unchanged + 0-new-chapter manga is a complete steady-
// state exit: every manga newer than this point was synced, and
// by source-side `update_date DESC` ordering everything past
// this point is at least as caught-up.
assert!(should_mark_clean_exit(false, true));
}
#[test]
fn dirty_exit_when_neither_completion_nor_stop_fired() {
// The walk ended for some other reason — including the
// caller-imposed `hit_limit` cap, which is the regression case
// this test exists for. `should_mark_clean_exit` does not take
// `hit_limit` as a parameter, so a future edit that adds
// `|| hit_limit` to the inline expression in `run_metadata_pass`
// would need to also touch this helper, and would fail this
// assertion when it did.
assert!(!should_mark_clean_exit(false, false));
}
#[test]
fn run_scoped_seen_set_skips_duplicate_source_manga_keys() {
// Pins the per-ref loop contract: `contains` gates whether work
// runs, and `insert` only fires on the success path (after upsert).
// A failed ref that reappears later in the same pass must get a
// second chance — that's why the loop uses contains-then-insert
// instead of insert-and-skip-on-collision.
let mut seen: HashSet<String> = HashSet::new();
// First sighting of a key: not yet seen → loop proceeds.
assert!(!seen.contains("manga-a"), "first sighting is unseen");
// Simulate a failed fetch_manga: do NOT insert. Next sighting must
// still be considered unseen so the loop retries it.
assert!(!seen.contains("manga-a"), "failed key is still retryable");
// Now simulate a successful upsert — insert is called.
seen.insert("manga-a".to_string());
// Subsequent sightings of the same key are skipped.
assert!(seen.contains("manga-a"), "successful key is now seen");
// Distinct keys never collide.
assert!(!seen.contains("manga-b"), "different key independent");
seen.insert("manga-b".to_string());
assert!(seen.contains("manga-b"));
assert!(seen.contains("manga-a"), "first key still recorded");
}
}

View File

@@ -203,7 +203,18 @@ async fn fetch_probe_html(browser: &Browser, probe_url: &str) -> anyhow::Result<
.new_page(probe_url)
.await
.with_context(|| format!("open probe page {probe_url}"))?;
page.wait_for_navigation().await.context("wait for nav on probe")?;
crate::crawler::nav::wait_for_nav(&page)
.await
.context("wait for nav on probe")?;
// Best-effort wait for the layout marker. Timeout is fine — the
// probe classifier handles a missing `#logo` as Transient anyway,
// and the verify loop retries on Transient.
let _ = crate::crawler::nav::wait_for_selector(
&page,
"#logo",
crate::crawler::nav::SELECTOR_TIMEOUT,
)
.await;
let html = page.content().await.context("read probe html")?;
page.close().await.ok();
Ok(html)

View File

@@ -8,19 +8,6 @@ pub mod target;
use async_trait::async_trait;
use chromiumoxide::browser::Browser;
use serde::{Deserialize, Serialize};
/// How a `discover` job should walk the source's index.
#[derive(Clone, Copy, Debug, Serialize, Deserialize)]
pub enum DiscoverMode {
/// Walk every index page from last back to first. Used for the
/// initial seed of a source.
Backfill,
/// Walk index pages from page 1 forward, stopping after
/// `stop_after_unchanged` consecutive mangas whose `metadata_hash`
/// matches storage. Used for the recurring cron tick.
Incremental { stop_after_unchanged: usize },
}
/// Pointer at a manga in the source's index, before we've fetched the
/// detail page. The `source_manga_key` is whatever stable id the source
@@ -83,14 +70,14 @@ pub struct FetchContext<'a> {
}
/// Lazy iterator over discovered manga refs. The caller drives the
/// walk one batch at a time, so it can break out as soon as a
/// downstream stop condition is met (e.g. N consecutive Unchanged
/// upserts in Incremental mode) without paying for pages it won't use.
/// walk one batch at a time, so it can break out as soon as the
/// downstream stop condition is met (the first manga where metadata is
/// `Unchanged` and chapter sync reports zero new chapters) without
/// paying for pages it won't use.
///
/// Batches are typically one source-index page each. Within a batch
/// refs are already in the right per-page order for the active mode
/// (Backfill reverses each page to oldest-first; Incremental leaves
/// the source's natural newest-first ordering).
/// refs are in the source's natural newest-first ordering — the same
/// `update_date DESC` sort that makes the stop condition meaningful.
#[async_trait]
pub trait DiscoverWalk: Send {
/// Return the next batch of refs, or `Ok(None)` when the source has
@@ -107,16 +94,14 @@ pub trait Source: Send + Sync {
/// Stable identifier — also the row key in the `sources` table.
fn id(&self) -> &'static str;
/// Begin discovery in `mode`. Returns a walker the caller drives
/// page-by-page via `next_batch`. The initial page-1 probe (used
/// to determine `last_page` and warm the cache for sites that
/// can't be paged without knowing the bound) happens inside this
/// call, so a fresh walker is ready to yield its first batch
/// without further setup.
/// Begin discovery. Returns a walker the caller drives page-by-page
/// via `next_batch`. The initial page-1 probe (used to determine
/// `last_page` and warm the cache for sites that can't be paged
/// without knowing the bound) happens inside this call, so a fresh
/// walker is ready to yield its first batch without further setup.
async fn discover(
&self,
ctx: &FetchContext<'_>,
mode: DiscoverMode,
) -> anyhow::Result<Box<dyn DiscoverWalk + Send>>;
async fn fetch_manga(

View File

@@ -15,22 +15,24 @@ use async_trait::async_trait;
use sha2::{Digest, Sha256};
use super::{
DiscoverMode, DiscoverWalk, FetchContext, Source, SourceChapter, SourceChapterRef,
SourceManga, SourceMangaRef,
DiscoverWalk, FetchContext, Source, SourceChapter, SourceChapterRef, SourceManga,
SourceMangaRef,
};
use crate::crawler::detect::{
has_logo_sentinel, is_broken_page_body, retry_on_transient, PageError,
};
use crate::crawler::nav::{wait_for_nav, wait_for_selector, NavError, SELECTOR_TIMEOUT};
/// `sources.id` value for this Source impl. Exposed as a const so the
/// daemon can look up per-source state (e.g. `seed_completed_at`)
/// before constructing the Source itself.
/// daemon can look up per-source state (e.g. the recovery flag) before
/// constructing the Source itself.
pub const SOURCE_ID: &str = "target";
/// In-loop retry budget for transient pages encountered during a single
/// `discover` walk. Bounded small because the job system itself retries
/// the whole `Discover` job on failure — these inline retries only need
/// to absorb a brief site hiccup mid-walk.
/// `discover` walk. Bounded small because the next cron tick will pick up
/// where this run left off via the recovery flag — these inline retries
/// only need to absorb a brief site hiccup mid-walk, not a sustained
/// outage.
const PAGE_TRANSIENT_RETRY_ATTEMPTS: u32 = 3;
const PAGE_TRANSIENT_RETRY_DELAY: Duration = Duration::from_secs(2);
@@ -72,14 +74,15 @@ impl Source for TargetSource {
async fn discover(
&self,
ctx: &FetchContext<'_>,
mode: DiscoverMode,
) -> anyhow::Result<Box<dyn DiscoverWalk + Send>> {
// Always visit page 1 first because that's the only way to
// discover `last_page`. Retry it on transient — a broken first
// page would otherwise abort the whole walk before we've even
// started.
let first_html = retry_on_transient(
|| async { navigate(ctx, self.base_url.as_str()).await },
|| async {
navigate(ctx, self.base_url.as_str(), LIST_PAGE_MARKER).await
},
PAGE_TRANSIENT_RETRY_ATTEMPTS,
PAGE_TRANSIENT_RETRY_DELAY,
)
@@ -89,10 +92,8 @@ impl Source for TargetSource {
parse_last_page(&doc)
};
let backfill = matches!(mode, DiscoverMode::Backfill);
let order = build_page_order(last_page, backfill);
let order = build_page_order(last_page);
tracing::info!(
?mode,
last_page = ?last_page,
page_count = order.len(),
"walking pagination"
@@ -100,7 +101,6 @@ impl Source for TargetSource {
Ok(Box::new(TargetSourceWalker {
base_url: self.base_url.clone(),
backfill,
pages_remaining: order,
first_page_html: Some(first_html),
}))
@@ -111,7 +111,17 @@ impl Source for TargetSource {
ctx: &FetchContext<'_>,
r: &SourceMangaRef,
) -> anyhow::Result<SourceManga> {
let html = navigate(ctx, r.url.as_str()).await?;
// When we'll parse the chapter table, wait for at least one
// chapter row to appear — that's the marker most sensitive to
// the post-load JS partial-render race. When we won't, fall
// back to the layout-level `#logo` so we still wait for the
// page to settle.
let marker = if self.parse_chapters {
DETAIL_PAGE_CHAPTERS_MARKER
} else {
DETAIL_PAGE_LAYOUT_MARKER
};
let html = navigate(ctx, r.url.as_str(), marker).await?;
// Convert PageError → anyhow::Error via `?`. PageError stays
// downcastable from the wrapped anyhow::Error so the pipeline
// can still recognize Transient via `error.downcast_ref::<PageError>()`.
@@ -138,16 +148,13 @@ impl Source for TargetSource {
}
/// Build the queue of page numbers `TargetSource::discover` will walk.
/// Backfill is oldest-first: pages `last..=1` (within each page the
/// walker reverses entries, since the source orders by update_date
/// DESC). Incremental is newest-first: pages `1..=last` in natural
/// order. If `last_page` is unknown (source surfaces no pagination)
/// only page 1 is visited.
fn build_page_order(last_page: Option<i32>, backfill: bool) -> VecDeque<i32> {
match (last_page, backfill) {
(None, _) => VecDeque::from([1]),
(Some(last), true) => (1..=last).rev().collect(),
(Some(last), false) => (1..=last).collect(),
/// The site orders by `update_date DESC`, so newest-first is just the
/// natural page order: `1..=last`. If `last_page` is unknown (source
/// surfaces no pagination) only page 1 is visited.
fn build_page_order(last_page: Option<i32>) -> VecDeque<i32> {
match last_page {
None => VecDeque::from([1]),
Some(last) => (1..=last).collect(),
}
}
@@ -157,7 +164,6 @@ fn build_page_order(last_page: Option<i32>, backfill: bool) -> VecDeque<i32> {
/// batch covering page 1 doesn't re-fetch.
struct TargetSourceWalker {
base_url: String,
backfill: bool,
pages_remaining: VecDeque<i32>,
first_page_html: Option<String>,
}
@@ -171,7 +177,7 @@ impl DiscoverWalk for TargetSourceWalker {
let Some(page_num) = self.pages_remaining.pop_front() else {
return Ok(None);
};
let mut page_refs = if page_num == 1 {
let page_refs = if page_num == 1 {
// Reuse the cached page-1 HTML from the initial probe. Take
// it (rather than clone) so a malformed page-order queue
// that re-visits page 1 still falls back to a real fetch.
@@ -183,7 +189,12 @@ impl DiscoverWalk for TargetSourceWalker {
None => {
retry_on_transient(
|| async {
let html = navigate(ctx, self.base_url.as_str()).await?;
let html = navigate(
ctx,
self.base_url.as_str(),
LIST_PAGE_MARKER,
)
.await?;
let doc = scraper::Html::parse_document(&html);
parse_manga_list_from(&doc)
},
@@ -197,7 +208,7 @@ impl DiscoverWalk for TargetSourceWalker {
retry_on_transient(
|| async {
let url = page_url(&self.base_url, page_num);
let html = navigate(ctx, &url).await?;
let html = navigate(ctx, &url, LIST_PAGE_MARKER).await?;
let doc = scraper::Html::parse_document(&html);
parse_manga_list_from(&doc)
},
@@ -206,32 +217,57 @@ impl DiscoverWalk for TargetSourceWalker {
)
.await?
};
if self.backfill {
page_refs.reverse();
}
tracing::info!(page_num, count = page_refs.len(), "page walked");
Ok(Some(page_refs))
}
}
/// Per-page-type markers used by `navigate`'s post-navigation wait.
/// Each is the most specific element the parser will later look for —
/// waiting on it closes the partial-render race (e.g. `#chapter_table`
/// wrapper present but rows still being injected by post-load JS) that
/// the old fixed 1s sleep masked. See [`navigate`].
const LIST_PAGE_MARKER: &str = "#left_side .pic_list .updatesli";
const DETAIL_PAGE_CHAPTERS_MARKER: &str = "#chapter_table td h4 a.chico";
const DETAIL_PAGE_LAYOUT_MARKER: &str = "#logo";
/// Single point of rate-limited navigation. Every Source request goes
/// through here, so the per-host limiter map is the only knob that
/// controls per-origin RPS. Also the choke point for transient-page
/// detection — every fetched body is screened by
/// [`classify_navigate_html`] before being handed to a selector.
async fn navigate(ctx: &FetchContext<'_>, url: &str) -> Result<String, PageError> {
///
/// `marker` is a CSS selector the caller expects to find on the loaded
/// page. The wait is best-effort: a timeout is **not** an error
/// (legitimately-empty pages may never render the marker), it just
/// caps how long we'll hold for post-load JS to finish injecting
/// content. The parser's own sentinels and the universal broken-page
/// body check still catch real failures.
async fn navigate(
ctx: &FetchContext<'_>,
url: &str,
marker: &str,
) -> Result<String, PageError> {
ctx.rate.wait_for(url).await?;
let page = ctx
.browser
.new_page(url)
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
page.wait_for_navigation()
.await
.map_err(|e| PageError::Other(anyhow::Error::from(e)))?;
// Stopgap until we wait on a specific selector per page type —
// gives any post-load JS a beat to finish injecting content.
tokio::time::sleep(Duration::from_secs(1)).await;
match wait_for_nav(&page).await {
Ok(()) => {}
Err(NavError::Timeout(_)) => {
page.close().await.ok();
return Err(PageError::transient("nav timeout"));
}
Err(NavError::Cdp(e)) => {
page.close().await.ok();
return Err(PageError::Other(anyhow::Error::from(e)));
}
}
// Best-effort wait for the page-type marker. We deliberately
// discard a timeout here — see fn-level doc.
let _ = wait_for_selector(&page, marker, SELECTOR_TIMEOUT).await;
let html = page
.content()
.await
@@ -369,7 +405,7 @@ fn parse_manga_detail(
.collect();
let chapters = if include_chapters {
parse_chapter_list(&doc)
parse_chapter_list(&doc)?
} else {
Vec::new()
};
@@ -427,9 +463,22 @@ fn strip_tag_count(s: &str) -> String {
trimmed.to_string()
}
fn parse_chapter_list(doc: &scraper::Html) -> Vec<SourceChapterRef> {
/// Parse the chapter table on a manga detail page. Returns `Transient` if
/// `#chapter_table` isn't in the DOM at all — the table is required even
/// for mangas with no published chapters yet (the source renders an empty
/// `<table>`), so an absent table signals a partial render (post-load JS
/// not done, layout drift) rather than a legitimately empty list. Without
/// this sentinel, an empty `Vec` reaches `sync_manga_chapters` and the
/// soft-drop branch flips every existing chapter to `dropped_at`.
fn parse_chapter_list(doc: &scraper::Html) -> Result<Vec<SourceChapterRef>, PageError> {
if !has_chapter_table_sentinel(doc) {
return Err(PageError::transient(
"manga detail: #chapter_table sentinel missing",
));
}
let sel = scraper::Selector::parse("#chapter_table td h4 a.chico").unwrap();
doc.select(&sel)
Ok(doc
.select(&sel)
.filter_map(|a| {
let url = a.value().attr("href")?.trim().to_string();
if url.is_empty() {
@@ -444,7 +493,16 @@ fn parse_chapter_list(doc: &scraper::Html) -> Vec<SourceChapterRef> {
url,
})
})
.collect()
.collect())
}
/// Returns true when the chapter-table container is present in the DOM.
/// Source-specific: the target site uses `#chapter_table` as the wrapper
/// element. Distinguishes "table is present but empty" (legit edge case
/// for new mangas) from "table is missing entirely" (partial render).
fn has_chapter_table_sentinel(doc: &scraper::Html) -> bool {
let sel = scraper::Selector::parse("#chapter_table").expect("valid selector");
doc.select(&sel).next().is_some()
}
fn parse_chapter_number(text: &str) -> Option<i32> {
@@ -744,7 +802,7 @@ mod tests {
"../../../tests/fixtures/target/chapter_list_uu.html"
);
let doc = scraper::Html::parse_document(html);
let chapters = parse_chapter_list(&doc);
let chapters = parse_chapter_list(&doc).expect("fixture has the table");
assert_eq!(chapters.len(), 15, "every row kept (notices/hiatus included)");
@@ -891,9 +949,17 @@ mod tests {
#[test]
fn missing_optional_fields_parse_to_none() {
// Minimal but well-formed detail page: title is required, every
// other field is optional, but the chapter table is structural —
// its absence is treated as Transient (a freshly added manga
// renders the table empty, not absent). See
// `parse_chapter_list_returns_transient_when_table_missing` for
// the negative case.
let html = r#"<html><body>\
<header><div id="logo">Target</div></header>\
<div class="w-title"><h1>Minimal</h1></div></body></html>"#;
<div class="w-title"><h1>Minimal</h1></div>\
<table id="chapter_table"></table>\
</body></html>"#;
let m = parse_manga_detail(html, "min", true).unwrap();
assert_eq!(m.title, "Minimal");
assert!(m.summary.is_none());
@@ -959,35 +1025,84 @@ mod tests {
}
#[test]
fn build_page_order_backfill_is_last_to_one() {
// Backfill walks pages oldest-first: queue is [last, last-1, ..., 1]
// so popping from the front yields the last page first.
let order = build_page_order(Some(3), true);
assert_eq!(Vec::from(order), vec![3, 2, 1]);
}
#[test]
fn build_page_order_incremental_is_one_to_last() {
// Incremental walks newest-first in natural source order.
let order = build_page_order(Some(3), false);
fn build_page_order_is_natural_one_to_last() {
// Newest-first is just the source's natural pagination order:
// (update_date DESC) lives at page 1, oldest at the last page.
let order = build_page_order(Some(3));
assert_eq!(Vec::from(order), vec![1, 2, 3]);
}
#[test]
fn build_page_order_falls_back_to_page_one_only_without_pagination() {
let backfill = build_page_order(None, true);
assert_eq!(Vec::from(backfill), vec![1]);
let incremental = build_page_order(None, false);
assert_eq!(Vec::from(incremental), vec![1]);
// Source surfaced no pagination control — visit page 1 alone
// and let the walk end after one batch.
let order = build_page_order(None);
assert_eq!(Vec::from(order), vec![1]);
}
#[test]
fn build_page_order_single_page_index_yields_one_entry() {
// Sources with exactly one page should not yield duplicates
// regardless of mode.
let backfill = build_page_order(Some(1), true);
assert_eq!(Vec::from(backfill), vec![1]);
let incremental = build_page_order(Some(1), false);
assert_eq!(Vec::from(incremental), vec![1]);
let order = build_page_order(Some(1));
assert_eq!(Vec::from(order), vec![1]);
}
#[test]
fn parse_chapter_list_returns_transient_when_table_missing() {
// Partial render (post-load JS hadn't injected the table, layout
// drift, etc). Returning Vec::new() would silently soft-drop every
// existing chapter for the manga via sync_manga_chapters; Transient
// is the signal the job system retries on.
let html = r#"<html><body>
<header><div id="logo">Target</div></header>
<div class="w-title"><h1>Test</h1></div>
</body></html>"#;
let doc = scraper::Html::parse_document(html);
let err = parse_chapter_list(&doc).expect_err("expected Transient");
assert!(err.is_transient(), "got non-transient: {err}");
}
#[test]
fn parse_chapter_list_ok_empty_when_table_present_but_no_rows() {
// A freshly-added manga with no chapters yet — the source renders
// the `<table id="chapter_table">` wrapper but no `<tr>` rows
// inside. Must stay distinguishable from a missing-table render.
let html = r#"<html><body>
<header><div id="logo">Target</div></header>
<table id="chapter_table"></table>
</body></html>"#;
let doc = scraper::Html::parse_document(html);
let chapters = parse_chapter_list(&doc).expect("present table is not transient");
assert!(chapters.is_empty());
}
#[test]
fn parse_manga_detail_propagates_chapter_table_transient() {
// End-to-end: a detail page that survives the #logo sentinel but
// has the chapter table stripped must fail Transient at the parser
// boundary, not return a SourceManga with empty chapters.
let html = r#"<html><body>
<header><div id="logo">Target</div></header>
<div class="w-title"><h1>Test Title</h1></div>
<div class="cover"><img src="/cover.jpg"></div>
<!-- intentionally no #chapter_table -->
</body></html>"#;
let err = parse_manga_detail(html, "key", true).expect_err("expected Transient");
assert!(err.is_transient(), "got non-transient: {err}");
}
#[test]
fn parse_manga_detail_skips_chapter_sentinel_when_include_chapters_false() {
// Metadata-only mode (`skip_chapters` upstream) must not require
// the chapter table — pipeline.rs avoids calling sync_manga_chapters
// for these mangas, so the absent table is not a correctness issue
// and shouldn't surface as Transient.
let html = r#"<html><body>
<header><div id="logo">Target</div></header>
<div class="w-title"><h1>Test Title</h1></div>
<div class="cover"><img src="/cover.jpg"></div>
</body></html>"#;
let manga = parse_manga_detail(html, "key", false)
.expect("metadata-only parse must not require chapter table");
assert!(manga.chapters.is_empty());
}
}

View File

@@ -0,0 +1,15 @@
use chrono::{DateTime, Utc};
use serde::Serialize;
use sqlx::FromRow;
use uuid::Uuid;
#[derive(Debug, Clone, Serialize, FromRow)]
pub struct AdminAuditEntry {
pub id: Uuid,
pub actor_user_id: Option<Uuid>,
pub action: String,
pub target_kind: String,
pub target_id: Option<Uuid>,
pub payload: serde_json::Value,
pub at: DateTime<Utc>,
}

View File

@@ -1,3 +1,4 @@
pub mod admin_audit;
pub mod api_token;
pub mod author;
pub mod bookmark;
@@ -9,11 +10,13 @@ pub mod page;
pub mod patch;
pub mod read_progress;
pub mod session;
pub mod sync_state;
pub mod tag;
pub mod upload_entry;
pub mod user;
pub mod user_preferences;
pub use admin_audit::AdminAuditEntry;
pub use api_token::ApiToken;
pub use author::{Author, AuthorRef, AuthorWithCount};
pub use bookmark::{Bookmark, BookmarkSummary};
@@ -25,6 +28,7 @@ pub use page::Page;
pub use patch::Patch;
pub use read_progress::{ReadProgress, ReadProgressForManga, ReadProgressSummary};
pub use session::Session;
pub use sync_state::{ChapterSyncState, MangaSyncState};
pub use tag::{Tag, TagRef};
pub use upload_entry::UploadEntry;
pub use user::User;

View File

@@ -0,0 +1,48 @@
//! Sync-state enums derived per-manga / per-chapter from `manga_sources`,
//! `chapter_sources`, and `crawler_jobs` at query time. No state column
//! is persisted on `mangas` / `chapters` — see `repo::admin_view` for the
//! derivation rules and priority order.
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, sqlx::Type)]
#[sqlx(type_name = "text", rename_all = "snake_case")]
#[serde(rename_all = "snake_case")]
pub enum MangaSyncState {
/// A `sync_manga` or `sync_chapter_list` job is currently
/// pending or running for this manga.
InProgress,
/// At least one `manga_sources` row exists for this manga and ALL of
/// them have `dropped_at IS NOT NULL` — every source we know about
/// has stopped surfacing it.
Dropped,
/// Default healthy state: at least one live source row OR the manga
/// was user-uploaded (no `manga_sources` rows at all).
Synced,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, sqlx::Type)]
#[sqlx(type_name = "text", rename_all = "snake_case")]
#[serde(rename_all = "snake_case")]
pub enum ChapterSyncState {
/// A `sync_chapter_content` job is currently pending or running for
/// this chapter (the 0014 dedup index guarantees at most one).
Downloading,
/// At least one `chapter_sources` row exists AND all of them are
/// `dropped_at IS NOT NULL`.
Dropped,
/// `page_count = 0` AND a `dead` `sync_chapter_content` job exists
/// for this chapter. Checked BEFORE `NotDownloaded` so the more
/// informative "we tried and it died" state wins over "we never
/// got around to it". Does NOT fire when `page_count > 0`, because
/// pages on disk mean the chapter IS synced regardless of historical
/// job failures — see the priority comment in `repo::admin_view`.
Failed,
/// `page_count = 0` and no in-flight or failed job — the chapter
/// row exists but content has never been downloaded.
NotDownloaded,
/// `page_count > 0` — content has been downloaded at some point.
/// Reaped `done` jobs in `crawler_jobs` mean we can't read this from
/// the job table, so `page_count` is the durable truth.
Synced,
}

View File

@@ -10,4 +10,5 @@ pub struct User {
#[serde(skip)]
pub password_hash: String,
pub created_at: DateTime<Utc>,
pub is_admin: bool,
}

View File

@@ -0,0 +1,32 @@
//! Admin-action audit log writes.
//!
//! Insert is always called from inside the same transaction as the
//! action it audits — the executor parameter is `PgExecutor` so the
//! caller passes `&mut *tx` directly.
use sqlx::PgExecutor;
use uuid::Uuid;
use crate::error::AppResult;
pub async fn insert<'e, E: PgExecutor<'e>>(
executor: E,
actor_user_id: Uuid,
action: &str,
target_kind: &str,
target_id: Option<Uuid>,
payload: serde_json::Value,
) -> AppResult<()> {
sqlx::query(
"INSERT INTO admin_audit (actor_user_id, action, target_kind, target_id, payload) \
VALUES ($1, $2, $3, $4, $5)",
)
.bind(actor_user_id)
.bind(action)
.bind(target_kind)
.bind(target_id)
.bind(payload)
.execute(executor)
.await?;
Ok(())
}

View File

@@ -0,0 +1,232 @@
//! Admin-facing read queries that join manga/chapter with the crawler
//! signals (`manga_sources`, `chapter_sources`, `crawler_jobs`) to
//! derive a sync state per row at query time.
//!
//! Priority order for `MangaSyncState`:
//! 1. `InProgress` — any pending/running `sync_manga` or
//! `sync_chapter_list` job matches this manga.
//! 2. `Dropped` — manga has source rows AND every one of them is
//! `dropped_at IS NOT NULL`.
//! 3. `Synced` — default (includes user-uploaded mangas with no
//! `manga_sources` rows at all).
//!
//! Priority order for `ChapterSyncState`:
//! 1. `Downloading` — pending/running `sync_chapter_content` for this id
//! 2. `Dropped` — chapter has source rows AND all are dropped
//! 3. `Failed` — `page_count = 0` AND a `dead` `sync_chapter_content`
//! row exists for this chapter. Constrained to `page_count = 0`
//! because once pages are on disk the chapter IS synced — a
//! historical dead job (likely from a re-download attempt that
//! crashed) is noise that gets reaped after retention. Surfacing
//! "Failed" when content is present would contradict
//! `ChapterSyncState::Synced`'s "downloaded at some point" contract.
//! 4. `NotDownloaded` — `page_count = 0`, no in-flight, no dead job
//! 5. `Synced` — `page_count > 0`
//!
//! Reminder: `done` jobs are reaped after `CRAWLER_JOB_RETENTION_DAYS`,
//! so `chapters.page_count > 0` is the durable "this is synced" signal,
//! not the job table.
use chrono::{DateTime, Utc};
use serde::Serialize;
use sqlx::{FromRow, PgPool};
use uuid::Uuid;
use crate::domain::{ChapterSyncState, MangaSyncState};
use crate::error::AppResult;
#[derive(Debug, Serialize, FromRow)]
pub struct AdminMangaRow {
pub id: Uuid,
pub title: String,
pub status: String,
pub cover_image_path: Option<String>,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub sync_state: MangaSyncState,
pub chapter_count: i64,
pub latest_seen_at: Option<DateTime<Utc>>,
}
#[derive(Debug, Default)]
pub struct ListAdminMangasQuery {
pub search: Option<String>,
pub sync_state: Option<MangaSyncState>,
pub limit: i64,
pub offset: i64,
}
const MANGA_SYNC_STATE_CASE: &str = r#"
CASE
WHEN EXISTS (
SELECT 1 FROM crawler_jobs cj
WHERE cj.state IN ('pending','running')
AND (
(cj.payload->>'kind' = 'sync_chapter_list'
AND (cj.payload->>'manga_id')::uuid = m.id)
OR (cj.payload->>'kind' = 'sync_manga'
AND EXISTS (
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id
AND ms.source_id = cj.payload->>'source_id'
AND ms.source_manga_key = cj.payload->>'source_manga_key'
))
)
) THEN 'in_progress'
WHEN EXISTS (SELECT 1 FROM manga_sources ms WHERE ms.manga_id = m.id)
AND NOT EXISTS (
SELECT 1 FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL
)
THEN 'dropped'
ELSE 'synced'
END
"#;
/// Paginated admin manga list with derived sync state and total count.
/// Filters by `search` (substring on title, case-insensitive) and
/// `sync_state` (post-derivation). The CTE keeps the case expression
/// in one place — the same projection feeds both the page rows and the
/// totals count under the same filter.
pub async fn list_mangas_with_sync_state(
pool: &PgPool,
q: &ListAdminMangasQuery,
) -> AppResult<(Vec<AdminMangaRow>, i64)> {
let search_pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.filter(|p| p.len() > 2);
// sqlx::Type → text: bind the snake_case representation manually so
// the SQL can compare it as text without an explicit cast.
let sync_filter = q.sync_state.map(|s| match s {
MangaSyncState::InProgress => "in_progress",
MangaSyncState::Dropped => "dropped",
MangaSyncState::Synced => "synced",
});
let sql = format!(
r#"
WITH classified AS (
SELECT
m.id, m.title, m.status, m.cover_image_path,
m.created_at, m.updated_at,
{case} AS sync_state,
(SELECT COUNT(*) FROM chapters c WHERE c.manga_id = m.id) AS chapter_count,
(SELECT MAX(last_seen_at) FROM manga_sources ms
WHERE ms.manga_id = m.id AND ms.dropped_at IS NULL) AS latest_seen_at
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
)
SELECT * FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)
ORDER BY updated_at DESC
LIMIT $3 OFFSET $4
"#,
case = MANGA_SYNC_STATE_CASE
);
let items: Vec<AdminMangaRow> = sqlx::query_as(&sql)
.bind(&search_pat)
.bind(sync_filter)
.bind(q.limit)
.bind(q.offset)
.fetch_all(pool)
.await?;
let total_sql = format!(
r#"
WITH classified AS (
SELECT {case} AS sync_state
FROM mangas m
WHERE ($1::text IS NULL OR m.title ILIKE $1)
)
SELECT COUNT(*) FROM classified
WHERE ($2::text IS NULL OR sync_state = $2)
"#,
case = MANGA_SYNC_STATE_CASE
);
let total: i64 = sqlx::query_scalar(&total_sql)
.bind(&search_pat)
.bind(sync_filter)
.fetch_one(pool)
.await?;
Ok((items, total))
}
#[derive(Debug, Serialize, FromRow)]
pub struct AdminChapterRow {
pub id: Uuid,
pub manga_id: Uuid,
pub number: i32,
pub title: Option<String>,
pub page_count: i32,
pub created_at: DateTime<Utc>,
pub sync_state: ChapterSyncState,
pub latest_seen_at: Option<DateTime<Utc>>,
}
#[derive(Debug, Default)]
pub struct ListAdminChaptersQuery {
pub manga_id: Uuid,
pub limit: i64,
pub offset: i64,
}
/// Paginated chapter list with derived sync state. Pagination is non-
/// optional — long-runners can have thousands of chapters and the
/// per-row scalar subqueries make the unbounded variant a real
/// stall risk even behind an admin guard. Returns the page slice plus
/// the unfiltered total so the UI can render "showing N of M".
pub async fn list_chapters_with_sync_state(
pool: &PgPool,
q: &ListAdminChaptersQuery,
) -> AppResult<(Vec<AdminChapterRow>, i64)> {
let items: Vec<AdminChapterRow> = sqlx::query_as(
r#"
SELECT
c.id, c.manga_id, c.number, c.title, c.page_count, c.created_at,
CASE
WHEN EXISTS (
SELECT 1 FROM crawler_jobs cj
WHERE cj.state IN ('pending','running')
AND cj.payload->>'kind' = 'sync_chapter_content'
AND (cj.payload->>'chapter_id')::uuid = c.id
) THEN 'downloading'
WHEN EXISTS (SELECT 1 FROM chapter_sources cs WHERE cs.chapter_id = c.id)
AND NOT EXISTS (
SELECT 1 FROM chapter_sources cs
WHERE cs.chapter_id = c.id AND cs.dropped_at IS NULL
)
THEN 'dropped'
WHEN c.page_count = 0
AND EXISTS (
SELECT 1 FROM crawler_jobs cj
WHERE cj.state = 'dead'
AND cj.payload->>'kind' = 'sync_chapter_content'
AND (cj.payload->>'chapter_id')::uuid = c.id
) THEN 'failed'
WHEN c.page_count = 0 THEN 'not_downloaded'
ELSE 'synced'
END AS sync_state,
(SELECT MAX(last_seen_at) FROM chapter_sources cs
WHERE cs.chapter_id = c.id AND cs.dropped_at IS NULL) AS latest_seen_at
FROM chapters c
WHERE c.manga_id = $1
ORDER BY c.number ASC
LIMIT $2 OFFSET $3
"#,
)
.bind(q.manga_id)
.bind(q.limit)
.bind(q.offset)
.fetch_all(pool)
.await?;
let total: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM chapters WHERE manga_id = $1")
.bind(q.manga_id)
.fetch_one(pool)
.await?;
Ok((items, total))
}

View File

@@ -118,10 +118,21 @@ pub async fn page_count(pool: &PgPool, id: Uuid) -> sqlx::Result<Option<i32>> {
.await
}
/// Look up the manga_id + most recent source_url for a chapter. Used
/// by the daemon's chapter dispatcher to resolve the URL it needs to
/// hand to `content::sync_chapter_content`. Returns `None` if the
/// chapter (or its source row) is gone.
/// Look up the manga_id + most recent live source_url for a chapter.
/// Used by the daemon's chapter dispatcher to resolve the URL it needs
/// to hand to `content::sync_chapter_content`.
///
/// Skips soft-dropped sources (`cs.dropped_at IS NOT NULL`) and breaks
/// ties between multiple live sources by `last_seen_at DESC`, so the
/// freshest still-attached URL wins. Returns `None` when the chapter
/// is gone or all its source rows are dropped — callers in the
/// dispatcher treat `None` as "ack the job, skip the work."
///
/// The enqueue queries (`pipeline::enqueue_bookmarked_pending` and
/// `enqueue_pending_for_manga`) apply the same `dropped_at IS NULL`
/// filter — this resolver stays in lockstep so a chapter that was
/// dropped between enqueue and lease isn't dispatched against a stale
/// URL.
pub async fn dispatch_target(
pool: &PgPool,
chapter_id: Uuid,
@@ -131,6 +142,8 @@ pub async fn dispatch_target(
FROM chapters c \
JOIN chapter_sources cs ON cs.chapter_id = c.id \
WHERE c.id = $1 \
AND cs.dropped_at IS NULL \
ORDER BY cs.last_seen_at DESC \
LIMIT 1",
)
.bind(chapter_id)

View File

@@ -8,14 +8,16 @@
//! updated (metadata_hash changed), or unchanged.
//! - [`sync_manga_chapters`]: per-manga chapter reconciliation. Adds
//! new ones, refreshes URLs on existing ones, soft-drops vanished.
//! - [`mark_dropped_mangas`]: end-of-run pass. Any manga from this
//! source whose `last_seen_at` is older than the run start is
//! soft-dropped.
//! - [`mark_run_started`] / [`mark_run_completed`] /
//! [`last_run_completed_cleanly`]: per-source recovery flag in
//! `crawler_state`. A `false` flag on tick start means the previous
//! run did not exit cleanly and the next walk should ignore the
//! early-stop condition.
//!
//! Each public function is a transaction boundary so a partial failure
//! mid-call leaves the DB in its pre-call state.
use chrono::{DateTime, Utc};
use chrono::Utc;
use sqlx::{PgPool, Postgres, Transaction};
use uuid::Uuid;
@@ -328,6 +330,22 @@ pub async fn sync_manga_chapters(
chapters: &[SourceChapterRef],
) -> sqlx::Result<ChapterDiff> {
let mut tx = pool.begin().await?;
// Per-manga advisory lock. Two concurrent calls for the same manga
// would otherwise both read `seen_keys`, both run the drop UPDATE
// filtered on `NOT (key = ANY $3)`, and the later commit could soft-
// drop a chapter the earlier commit had just inserted (lost-update
// shape under MVCC). `pg_advisory_xact_lock` is scoped to this
// transaction: it auto-releases on COMMIT/ROLLBACK so a Rust-side
// panic mid-call doesn't strand the lock. The single-arg int8 form
// keyed by `hashtextextended(manga_id::text, 0)` shares Postgres'
// global advisory-lock namespace with `CRON_LOCK_KEY`, but collision
// is 2^-64 per pair (a UUID-derived hash hitting the fixed cron key
// is effectively impossible).
sqlx::query("SELECT pg_advisory_xact_lock(hashtextextended($1::text, 0))")
.bind(manga_id)
.execute(&mut *tx)
.await?;
let mut diff = ChapterDiff::default();
let seen_keys: Vec<String> = chapters
.iter()
@@ -335,11 +353,23 @@ pub async fn sync_manga_chapters(
.collect();
for c in chapters {
// Lookup is constrained by manga_id (via the chapters join) so a
// source whose chapter slugs collide across mangas (e.g.
// "chapter-1" appearing under two different mangas) attributes
// each row to the correct manga. Migration 0017 dropped the
// (source_id, source_chapter_key) PK in favour of
// (source_id, chapter_id) for exactly this reason.
let existing: Option<(Uuid,)> = sqlx::query_as(
"SELECT chapter_id FROM chapter_sources WHERE source_id = $1 AND source_chapter_key = $2",
"SELECT cs.chapter_id \
FROM chapter_sources cs \
JOIN chapters ch ON ch.id = cs.chapter_id \
WHERE cs.source_id = $1 \
AND cs.source_chapter_key = $2 \
AND ch.manga_id = $3",
)
.bind(source_id)
.bind(&c.source_chapter_key)
.bind(manga_id)
.fetch_optional(&mut *tx)
.await?;
@@ -383,16 +413,19 @@ pub async fn sync_manga_chapters(
.bind(chapter_id)
.execute(&mut *tx)
.await?;
// chapter_id is now the natural per-(source, chapter)
// identifier — use it directly instead of re-keying on
// (source_id, source_chapter_key) which may not be unique.
sqlx::query(
r#"
UPDATE chapter_sources
SET source_url = $1, last_seen_at = NOW(), dropped_at = NULL
WHERE source_id = $2 AND source_chapter_key = $3
WHERE source_id = $2 AND chapter_id = $3
"#,
)
.bind(&c.url)
.bind(source_id)
.bind(&c.source_chapter_key)
.bind(chapter_id)
.execute(&mut *tx)
.await?;
diff.refreshed += 1;
@@ -425,19 +458,52 @@ pub async fn sync_manga_chapters(
Ok(diff)
}
/// Record that a complete Backfill walk has finished for `source_id`.
/// The presence of this row is what the daemon's mode auto-detection
/// uses to flip from Backfill to Incremental on subsequent ticks.
/// Count the chapters that the source `(source_id, source_manga_key)`
/// is currently known to attach to — i.e. the number of `chapter_sources`
/// rows for the manga identified by the (source_id, source_manga_key)
/// pair, restricted to live (`dropped_at IS NULL`) rows.
///
/// Keyed `seed_completed:<source_id>` in `crawler_state`. JSON payload
/// stores the timestamp so we can surface "last fully reseeded at" in
/// future ops tooling without another migration.
pub async fn mark_seed_completed(
/// Used by the metadata pass's partial-render guard: if `fetch_manga`
/// returns an empty `chapters` Vec but the source previously surfaced
/// chapters here, that's most likely a chromium snapshot taken between
/// the `#chapter_table` wrapper render and its rows render — the
/// safest move is to skip `sync_manga_chapters` so the soft-drop
/// branch doesn't flip every existing chapter to `dropped_at`.
///
/// Returns `Ok(0)` when the manga is brand-new (no `manga_sources`
/// row yet), which is the legitimate "this manga has no chapters yet"
/// case and must NOT be flagged.
pub async fn live_chapter_count_for_source_manga(
pool: &PgPool,
source_id: &str,
at: DateTime<Utc>,
) -> sqlx::Result<()> {
let key = format!("seed_completed:{source_id}");
source_manga_key: &str,
) -> sqlx::Result<i64> {
let row: Option<(i64,)> = sqlx::query_as(
"SELECT COUNT(*) \
FROM chapter_sources cs \
JOIN chapters c ON c.id = cs.chapter_id \
JOIN manga_sources ms \
ON ms.manga_id = c.manga_id \
AND ms.source_id = cs.source_id \
WHERE ms.source_id = $1 \
AND ms.source_manga_key = $2 \
AND cs.dropped_at IS NULL",
)
.bind(source_id)
.bind(source_manga_key)
.fetch_optional(pool)
.await?;
Ok(row.map(|(n,)| n).unwrap_or(0))
}
/// Mark a metadata pass as in-flight for `source_id`. Stamps
/// `last_run_completed:<source_id>` in `crawler_state` with
/// `{"completed": false, "at": now}`. A crash, panic, or SIGKILL after
/// this point leaves the flag at `false`, which the next tick reads as
/// "previous run did not exit cleanly — walk the full catalog this
/// time" (recovery sweep).
pub async fn mark_run_started(pool: &PgPool, source_id: &str) -> sqlx::Result<()> {
let key = format!("last_run_completed:{source_id}");
sqlx::query(
"INSERT INTO crawler_state (key, value, updated_at) \
VALUES ($1, $2, now()) \
@@ -445,50 +511,54 @@ pub async fn mark_seed_completed(
SET value = EXCLUDED.value, updated_at = now()",
)
.bind(&key)
.bind(serde_json::json!({ "at": at.to_rfc3339() }))
.bind(serde_json::json!({
"completed": false,
"at": Utc::now().to_rfc3339(),
}))
.execute(pool)
.await?;
Ok(())
}
/// Read the timestamp written by [`mark_seed_completed`], if any.
/// `None` means no complete Backfill has ever finished for this
/// source — the daemon should run Backfill on the next tick.
pub async fn seed_completed_at(
/// Mark a metadata pass as completed cleanly for `source_id`. Called
/// from the same place a run decides it reached end-of-walk or hit the
/// intentional stop. The next tick reads `true` and applies the normal
/// stop condition.
pub async fn mark_run_completed(pool: &PgPool, source_id: &str) -> sqlx::Result<()> {
let key = format!("last_run_completed:{source_id}");
sqlx::query(
"INSERT INTO crawler_state (key, value, updated_at) \
VALUES ($1, $2, now()) \
ON CONFLICT (key) DO UPDATE \
SET value = EXCLUDED.value, updated_at = now()",
)
.bind(&key)
.bind(serde_json::json!({
"completed": true,
"at": Utc::now().to_rfc3339(),
}))
.execute(pool)
.await?;
Ok(())
}
/// Read the recovery flag for `source_id`. A missing row OR an
/// unparseable value reads as `true` ("clean") — the former covers the
/// first-ever run on a virgin DB (no recovery needed), the latter
/// covers forward-compat against future schema changes; both fail-safe
/// toward not making an operator pay for an unnecessary full sweep.
pub async fn last_run_completed_cleanly(
pool: &PgPool,
source_id: &str,
) -> sqlx::Result<Option<DateTime<Utc>>> {
let key = format!("seed_completed:{source_id}");
) -> sqlx::Result<bool> {
let key = format!("last_run_completed:{source_id}");
let row: Option<serde_json::Value> =
sqlx::query_scalar("SELECT value FROM crawler_state WHERE key = $1")
.bind(&key)
.fetch_optional(pool)
.await?;
Ok(row.and_then(|v| {
v.get("at")
.and_then(|s| s.as_str())
.and_then(|s| DateTime::parse_from_rfc3339(s).ok())
.map(|dt| dt.with_timezone(&Utc))
}))
Ok(row
.and_then(|v| v.get("completed").and_then(|b| b.as_bool()))
.unwrap_or(true))
}
pub async fn mark_dropped_mangas(
pool: &PgPool,
source_id: &str,
run_started_at: DateTime<Utc>,
) -> sqlx::Result<u64> {
let res = sqlx::query(
r#"
UPDATE manga_sources
SET dropped_at = NOW()
WHERE source_id = $1
AND last_seen_at < $2
AND dropped_at IS NULL
"#,
)
.bind(source_id)
.bind(run_started_at)
.execute(pool)
.await?;
Ok(res.rows_affected())
}

View File

@@ -1,3 +1,5 @@
pub mod admin_audit;
pub mod admin_view;
pub mod api_token;
pub mod author;
pub mod bookmark;

View File

@@ -11,7 +11,7 @@ pub async fn create(pool: &PgPool, username: &str, password_hash: &str) -> AppRe
r#"
INSERT INTO users (username, password_hash)
VALUES ($1, $2)
RETURNING id, username, password_hash, created_at
RETURNING id, username, password_hash, created_at, is_admin
"#,
)
.bind(username)
@@ -35,7 +35,7 @@ pub async fn create(pool: &PgPool, username: &str, password_hash: &str) -> AppRe
pub async fn find_by_username(pool: &PgPool, username: &str) -> AppResult<Option<User>> {
let row = sqlx::query_as::<_, User>(
r#"
SELECT id, username, password_hash, created_at
SELECT id, username, password_hash, created_at, is_admin
FROM users
WHERE lower(username) = lower($1)
"#,
@@ -48,7 +48,7 @@ pub async fn find_by_username(pool: &PgPool, username: &str) -> AppResult<Option
pub async fn find_by_id(pool: &PgPool, id: Uuid) -> AppResult<Option<User>> {
let row = sqlx::query_as::<_, User>(
r#"SELECT id, username, password_hash, created_at FROM users WHERE id = $1"#,
r#"SELECT id, username, password_hash, created_at, is_admin FROM users WHERE id = $1"#,
)
.bind(id)
.fetch_optional(pool)
@@ -56,3 +56,317 @@ pub async fn find_by_id(pool: &PgPool, id: Uuid) -> AppResult<Option<User>> {
Ok(row)
}
/// Postgres advisory-lock key guarding admin-count-changing operations
/// (demote, delete-admin). Without this lock two concurrent demotes of
/// different admins could each pass their "more than one admin remains"
/// check, then commit, leaving zero admins. The lock serialises any tx
/// that might change the admin count so the recount under the lock is
/// authoritative.
///
/// Value is the bytes of "admininv" interpreted as a big-endian i64.
/// Postgres' advisory-lock keyspace is global; collision risk with
/// `CRON_LOCK_KEY` and friends is ~2^-64.
pub const ADMIN_INVARIANT_LOCK_KEY: i64 = 0x61_64_6d_69_6e_69_6e_76;
#[derive(Debug, Default)]
pub struct ListUsersQuery {
pub search: Option<String>,
pub limit: i64,
pub offset: i64,
}
/// Paginated user list with total count. `search` is a case-insensitive
/// substring match on `username`. Order is alphabetical by username so
/// pagination is stable across concurrent writes (mangas changing
/// is_admin doesn't reshuffle the page).
pub async fn list_with_total(
pool: &PgPool,
q: &ListUsersQuery,
) -> AppResult<(Vec<User>, i64)> {
let pat = q
.search
.as_ref()
.map(|s| format!("%{}%", s.trim()))
.filter(|p| p.len() > 2);
let items = sqlx::query_as::<_, User>(
r#"
SELECT id, username, password_hash, created_at, is_admin
FROM users
WHERE ($1::text IS NULL OR username ILIKE $1)
ORDER BY username
LIMIT $2 OFFSET $3
"#,
)
.bind(&pat)
.bind(q.limit)
.bind(q.offset)
.fetch_all(pool)
.await?;
let total: i64 = sqlx::query_scalar(
"SELECT COUNT(*) FROM users WHERE ($1::text IS NULL OR username ILIKE $1)",
)
.bind(&pat)
.fetch_one(pool)
.await?;
Ok((items, total))
}
/// Raw `is_admin` update with no safety checks, no audit log, and no
/// advisory lock. Exists only as a test setup helper for the admin-
/// feature integration suite — production code MUST go through
/// [`admin_safe_set_is_admin`], which enforces self-protection, the
/// last-admin invariant, and the audit log atomically.
pub async fn set_is_admin_unchecked(pool: &PgPool, id: Uuid, value: bool) -> AppResult<()> {
sqlx::query("UPDATE users SET is_admin = $1 WHERE id = $2")
.bind(value)
.bind(id)
.execute(pool)
.await?;
Ok(())
}
/// Ensure the user `username` exists and is an admin. Called at startup
/// from `app::build` when `ADMIN_USERNAME` / `ADMIN_PASSWORD` are set.
///
/// Semantics — see cross-cutting decision #2 in the feature plan:
/// - If no row exists: create with the env-supplied password hashed via
/// argon2id and `is_admin = true`.
/// - If a row already exists: flip `is_admin` to true if needed; **never**
/// touch the existing `password_hash`. Lets the operator rotate the
/// admin password through the UI without env-var conflict.
/// Wrapped in a transaction so a concurrent `register` for the same
/// username can't slip an INSERT between the SELECT and UPDATE/INSERT.
/// Set `is_admin` on a user with full safety checks: rejects self-demote,
/// rejects demoting the only remaining admin (under `ADMIN_INVARIANT_LOCK_KEY`
/// to close the parallel-demote race), and writes an `admin_audit` row
/// in the same tx so the log mirrors what actually committed.
///
/// Returns the freshly-written user row (so the handler can return it
/// without a second SELECT).
pub async fn admin_safe_set_is_admin(
pool: &PgPool,
actor_id: Uuid,
target_id: Uuid,
value: bool,
) -> AppResult<User> {
// Cheap pre-check before opening a tx — also covers the "demote me"
// case which would otherwise pass the recount when other admins exist.
if actor_id == target_id && !value {
return Err(AppError::Conflict(
"cannot demote yourself; ask another admin".into(),
));
}
let mut tx = pool.begin().await?;
sqlx::query("SELECT pg_advisory_xact_lock($1)")
.bind(ADMIN_INVARIANT_LOCK_KEY)
.execute(&mut *tx)
.await?;
let target: Option<User> = sqlx::query_as(
"SELECT id, username, password_hash, created_at, is_admin \
FROM users WHERE id = $1 FOR UPDATE",
)
.bind(target_id)
.fetch_optional(&mut *tx)
.await?;
let Some(target) = target else {
return Err(AppError::NotFound);
};
// No-op: caller asked to set `is_admin` to its current value. Return
// the row as-is without writing an audit entry — otherwise repeated
// PATCH calls (browser retry, double-click) pile misleading
// "promote_user" rows in `admin_audit` for actions that changed
// nothing.
if target.is_admin == value {
tx.commit().await?;
return Ok(target);
}
// Recount inside the lock — this is the authoritative read.
if target.is_admin && !value {
let admin_count: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM users WHERE is_admin = true")
.fetch_one(&mut *tx)
.await?;
if admin_count <= 1 {
return Err(AppError::Conflict(
"cannot demote the last admin; promote another user first".into(),
));
}
}
let updated: User = sqlx::query_as(
"UPDATE users SET is_admin = $1 WHERE id = $2 \
RETURNING id, username, password_hash, created_at, is_admin",
)
.bind(value)
.bind(target_id)
.fetch_one(&mut *tx)
.await?;
let action = if value { "promote_user" } else { "demote_user" };
crate::repo::admin_audit::insert(
&mut *tx,
actor_id,
action,
"user",
Some(target_id),
serde_json::json!({ "username": target.username }),
)
.await?;
tx.commit().await?;
Ok(updated)
}
/// Delete a user with full safety checks: rejects self-delete, rejects
/// deleting the only remaining admin (under `ADMIN_INVARIANT_LOCK_KEY`),
/// and writes an `admin_audit` row in the same tx. Captures the deleted
/// username + admin status in the audit payload so the action is
/// readable after the user row itself is gone.
pub async fn admin_safe_delete(
pool: &PgPool,
actor_id: Uuid,
target_id: Uuid,
) -> AppResult<()> {
if actor_id == target_id {
return Err(AppError::Conflict(
"cannot delete yourself; ask another admin".into(),
));
}
let mut tx = pool.begin().await?;
sqlx::query("SELECT pg_advisory_xact_lock($1)")
.bind(ADMIN_INVARIANT_LOCK_KEY)
.execute(&mut *tx)
.await?;
let target: Option<User> = sqlx::query_as(
"SELECT id, username, password_hash, created_at, is_admin \
FROM users WHERE id = $1 FOR UPDATE",
)
.bind(target_id)
.fetch_optional(&mut *tx)
.await?;
let Some(target) = target else {
return Err(AppError::NotFound);
};
if target.is_admin {
let admin_count: i64 =
sqlx::query_scalar("SELECT COUNT(*) FROM users WHERE is_admin = true")
.fetch_one(&mut *tx)
.await?;
if admin_count <= 1 {
return Err(AppError::Conflict(
"cannot delete the last admin; promote another user first".into(),
));
}
}
sqlx::query("DELETE FROM users WHERE id = $1")
.bind(target_id)
.execute(&mut *tx)
.await?;
crate::repo::admin_audit::insert(
&mut *tx,
actor_id,
"delete_user",
"user",
Some(target_id),
serde_json::json!({
"username": target.username,
"was_admin": target.is_admin,
}),
)
.await?;
tx.commit().await?;
Ok(())
}
/// Admin-initiated user creation. Wraps the INSERT + audit row in a
/// single transaction so a rolled-back create never leaves an orphan
/// audit entry. Caller (HTTP handler) is responsible for validating
/// `username`/`password` and hashing — this fn assumes both are
/// already vetted by the same `validate_*` rules used by self-
/// registration.
pub async fn admin_create_user(
pool: &PgPool,
actor_id: Uuid,
username: &str,
password_hash: &str,
is_admin: bool,
) -> AppResult<User> {
let mut tx = pool.begin().await?;
let user: User = match sqlx::query_as::<_, User>(
"INSERT INTO users (username, password_hash, is_admin) VALUES ($1, $2, $3) \
RETURNING id, username, password_hash, created_at, is_admin",
)
.bind(username)
.bind(password_hash)
.bind(is_admin)
.fetch_one(&mut *tx)
.await
{
Ok(u) => u,
Err(sqlx::Error::Database(ref db_err)) if db_err.is_unique_violation() => {
return Err(AppError::Conflict("username is already taken".into()));
}
Err(e) => return Err(AppError::Database(e)),
};
crate::repo::admin_audit::insert(
&mut *tx,
actor_id,
"create_user",
"user",
Some(user.id),
serde_json::json!({
"username": user.username,
"is_admin": user.is_admin,
}),
)
.await?;
tx.commit().await?;
Ok(user)
}
pub async fn bootstrap_admin(
pool: &PgPool,
username: &str,
password: &str,
) -> AppResult<()> {
let mut tx = pool.begin().await?;
let existing: Option<(Uuid,)> = sqlx::query_as(
"SELECT id FROM users WHERE lower(username) = lower($1) FOR UPDATE",
)
.bind(username)
.fetch_optional(&mut *tx)
.await?;
match existing {
Some((id,)) => {
sqlx::query("UPDATE users SET is_admin = true WHERE id = $1 AND is_admin = false")
.bind(id)
.execute(&mut *tx)
.await?;
}
None => {
let hash = crate::auth::password::hash_password(password)?;
sqlx::query("INSERT INTO users (username, password_hash, is_admin) VALUES ($1, $2, true)")
.bind(username)
.bind(&hash)
.execute(&mut *tx)
.await?;
}
}
tx.commit().await?;
Ok(())
}

View File

@@ -86,6 +86,10 @@ impl Storage for LocalStorage {
let path: &Path = &self.resolve(key)?;
Ok(fs::try_exists(path).await?)
}
fn local_root(&self) -> Option<&Path> {
Some(&self.root)
}
}
#[cfg(test)]

View File

@@ -9,6 +9,8 @@ mod local;
use std::io;
use std::pin::Pin;
use std::path::Path;
use async_trait::async_trait;
use bytes::Bytes;
use futures_core::Stream;
@@ -44,4 +46,13 @@ pub trait Storage: Send + Sync {
async fn get_stream(&self, key: &str) -> Result<StreamingFile, StorageError>;
async fn delete(&self, key: &str) -> Result<(), StorageError>;
async fn exists(&self, key: &str) -> Result<bool, StorageError>;
/// Filesystem path the backend is rooted at, when introspectable.
/// Returns `None` for backends that aren't a local filesystem (e.g.
/// a future `S3Storage`). The admin system endpoint uses this to
/// statvfs the data dir; backends that return `None` get a `disk:
/// null` payload instead of fabricated numbers.
fn local_root(&self) -> Option<&Path> {
None
}
}

View File

@@ -0,0 +1,548 @@
//! PR 3 (feat/admin-mangas-api) integration tests.
//!
//! Per-variant fixture tests for the derived sync-state SQL plus
//! happy-path E2E for the two admin endpoints. Auth-gate regression
//! (403/401) is covered by PR 1's `RequireAdmin` test matrix; the only
//! gate test here is one spot check per endpoint.
mod common;
use axum::http::StatusCode;
use axum::Router;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
use mangalord::repo;
const SOURCE_ID: &str = "test-source";
async fn seed_admin(pool: &PgPool, app: &Router) -> (String, String) {
let (username, cookie) = common::register_user(app).await;
let u = repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
repo::user::set_is_admin_unchecked(pool, u.id, true).await.unwrap();
(username, cookie)
}
async fn seed_source(pool: &PgPool) {
repo::crawler::ensure_source(pool, SOURCE_ID, "Test", "https://example.test")
.await
.unwrap();
}
async fn insert_manga(pool: &PgPool, title: &str) -> Uuid {
let (id,): (Uuid,) = sqlx::query_as(
"INSERT INTO mangas (title, status, alt_titles) VALUES ($1, 'ongoing', ARRAY[]::text[]) RETURNING id",
)
.bind(title)
.fetch_one(pool)
.await
.unwrap();
id
}
async fn insert_manga_source(
pool: &PgPool,
manga_id: Uuid,
source_manga_key: &str,
dropped: bool,
) {
let dropped_at = if dropped { "now()" } else { "NULL" };
let sql = format!(
"INSERT INTO manga_sources (source_id, source_manga_key, manga_id, source_url, dropped_at) \
VALUES ($1, $2, $3, 'https://example.test/m', {dropped_at})"
);
sqlx::query(&sql)
.bind(SOURCE_ID)
.bind(source_manga_key)
.bind(manga_id)
.execute(pool)
.await
.unwrap();
}
async fn insert_chapter(pool: &PgPool, manga_id: Uuid, number: i32, page_count: i32) -> Uuid {
let (id,): (Uuid,) = sqlx::query_as(
"INSERT INTO chapters (manga_id, number, title, page_count) VALUES ($1, $2, NULL, $3) RETURNING id",
)
.bind(manga_id)
.bind(number)
.bind(page_count)
.fetch_one(pool)
.await
.unwrap();
id
}
async fn insert_chapter_source(
pool: &PgPool,
chapter_id: Uuid,
source_chapter_key: &str,
dropped: bool,
) {
let dropped_at = if dropped { "now()" } else { "NULL" };
let sql = format!(
"INSERT INTO chapter_sources (source_id, source_chapter_key, chapter_id, source_url, dropped_at) \
VALUES ($1, $2, $3, 'https://example.test/c', {dropped_at})"
);
sqlx::query(&sql)
.bind(SOURCE_ID)
.bind(source_chapter_key)
.bind(chapter_id)
.execute(pool)
.await
.unwrap();
}
async fn insert_job(pool: &PgPool, payload: serde_json::Value, state: &str) {
sqlx::query("INSERT INTO crawler_jobs (payload, state) VALUES ($1, $2)")
.bind(payload)
.bind(state)
.execute(pool)
.await
.unwrap();
}
/// Per-variant tests don't care about pagination — fetch the whole
/// chapter set (up to the hard cap) and discard the total.
async fn fetch_chapter_rows(
pool: &PgPool,
manga_id: Uuid,
) -> Vec<mangalord::repo::admin_view::AdminChapterRow> {
let (rows, _) = repo::admin_view::list_chapters_with_sync_state(
pool,
&repo::admin_view::ListAdminChaptersQuery {
manga_id,
limit: 500,
offset: 0,
},
)
.await
.unwrap();
rows
}
// ---- manga sync state ------------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn manga_state_synced_for_fresh_source(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "Synced Manga").await;
insert_manga_source(&pool, m, "smk-1", false).await;
let (rows, total) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(total, 1);
assert_eq!(rows[0].id, m);
assert_eq!(rows[0].sync_state, mangalord::domain::MangaSyncState::Synced);
}
#[sqlx::test(migrations = "./migrations")]
async fn manga_state_synced_for_user_upload_without_sources(pool: PgPool) {
let m = insert_manga(&pool, "User Upload").await;
let (rows, _) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(rows[0].id, m);
assert_eq!(rows[0].sync_state, mangalord::domain::MangaSyncState::Synced);
}
#[sqlx::test(migrations = "./migrations")]
async fn manga_state_dropped_when_all_sources_dropped(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "Dropped Manga").await;
insert_manga_source(&pool, m, "smk-1", true).await;
let (rows, _) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(rows[0].id, m);
assert_eq!(rows[0].sync_state, mangalord::domain::MangaSyncState::Dropped);
}
#[sqlx::test(migrations = "./migrations")]
async fn manga_state_in_progress_via_sync_chapter_list_job(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "Syncing Manga").await;
insert_manga_source(&pool, m, "smk-1", false).await;
// sync_chapter_list payload carries manga_id directly.
insert_job(
&pool,
json!({
"kind": "sync_chapter_list",
"source_id": SOURCE_ID,
"manga_id": m.to_string(),
"source_manga_key": "smk-1",
}),
"pending",
)
.await;
let (rows, _) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(rows[0].sync_state, mangalord::domain::MangaSyncState::InProgress);
}
#[sqlx::test(migrations = "./migrations")]
async fn manga_state_in_progress_via_sync_manga_job(pool: PgPool) {
// The trickier branch: sync_manga payload is keyed by
// source_manga_key, NOT manga_id — must join through manga_sources.
seed_source(&pool).await;
let m = insert_manga(&pool, "Metadata-Refreshing Manga").await;
insert_manga_source(&pool, m, "smk-key-42", false).await;
insert_job(
&pool,
json!({
"kind": "sync_manga",
"source_id": SOURCE_ID,
"source_manga_key": "smk-key-42",
}),
"running",
)
.await;
let (rows, _) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(rows[0].sync_state, mangalord::domain::MangaSyncState::InProgress);
}
#[sqlx::test(migrations = "./migrations")]
async fn manga_list_filters_by_sync_state(pool: PgPool) {
seed_source(&pool).await;
let m_synced = insert_manga(&pool, "AAA Synced").await;
insert_manga_source(&pool, m_synced, "smk-a", false).await;
let m_dropped = insert_manga(&pool, "BBB Dropped").await;
insert_manga_source(&pool, m_dropped, "smk-b", true).await;
let (rows, total) = repo::admin_view::list_mangas_with_sync_state(
&pool,
&repo::admin_view::ListAdminMangasQuery {
sync_state: Some(mangalord::domain::MangaSyncState::Dropped),
limit: 50,
..Default::default()
},
)
.await
.unwrap();
assert_eq!(total, 1);
assert_eq!(rows.len(), 1);
assert_eq!(rows[0].id, m_dropped);
}
// ---- chapter sync state ----------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_synced_when_pages_present(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
insert_manga_source(&pool, m, "smk", false).await;
let c = insert_chapter(&pool, m, 1, 12).await;
insert_chapter_source(&pool, c, "ckey-1", false).await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(rows.len(), 1);
assert_eq!(rows[0].id, c);
assert_eq!(rows[0].sync_state, mangalord::domain::ChapterSyncState::Synced);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_not_downloaded_when_page_count_zero(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c = insert_chapter(&pool, m, 1, 0).await;
insert_chapter_source(&pool, c, "ckey-1", false).await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(
rows[0].sync_state,
mangalord::domain::ChapterSyncState::NotDownloaded
);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_downloading_when_job_in_flight(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c = insert_chapter(&pool, m, 1, 0).await;
insert_chapter_source(&pool, c, "ckey-1", false).await;
insert_job(
&pool,
json!({
"kind": "sync_chapter_content",
"source_id": SOURCE_ID,
"chapter_id": c.to_string(),
"source_chapter_key": "ckey-1",
}),
"running",
)
.await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(
rows[0].sync_state,
mangalord::domain::ChapterSyncState::Downloading
);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_dropped_when_all_sources_dropped(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c = insert_chapter(&pool, m, 1, 0).await;
insert_chapter_source(&pool, c, "ckey-1", true).await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(
rows[0].sync_state,
mangalord::domain::ChapterSyncState::Dropped
);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_failed_when_most_recent_job_dead(pool: PgPool) {
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c = insert_chapter(&pool, m, 1, 0).await;
insert_chapter_source(&pool, c, "ckey-1", false).await;
insert_job(
&pool,
json!({
"kind": "sync_chapter_content",
"source_id": SOURCE_ID,
"chapter_id": c.to_string(),
"source_chapter_key": "ckey-1",
}),
"dead",
)
.await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(
rows[0].sync_state,
mangalord::domain::ChapterSyncState::Failed
);
}
// ---- HTTP-level happy-path + gate ------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn http_list_mangas_returns_paged_with_state(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
seed_source(&pool).await;
let m = insert_manga(&pool, "Hello").await;
insert_manga_source(&pool, m, "smk", false).await;
let resp = h
.app
.oneshot(common::get_with_cookie(
"/api/v1/admin/mangas?limit=50",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0]["id"], m.to_string());
assert_eq!(items[0]["sync_state"], "synced");
assert_eq!(items[0]["chapter_count"], 0);
assert_eq!(body["page"]["total"], 1);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_mangas_rejects_unknown_sync_state(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie(
"/api/v1/admin/mangas?sync_state=bogus",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_chapters_returns_per_chapter_state(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c1 = insert_chapter(&pool, m, 1, 12).await;
let c2 = insert_chapter(&pool, m, 2, 0).await;
insert_chapter_source(&pool, c1, "ck1", false).await;
insert_chapter_source(&pool, c2, "ck2", false).await;
let resp = h
.app
.oneshot(common::get_with_cookie(
&format!("/api/v1/admin/mangas/{m}/chapters"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(items.len(), 2);
assert_eq!(items[0]["id"], c1.to_string());
assert_eq!(items[0]["sync_state"], "synced");
assert_eq!(items[1]["id"], c2.to_string());
assert_eq!(items[1]["sync_state"], "not_downloaded");
assert_eq!(body["page"]["total"], 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_chapters_caps_limit_at_500(pool: PgPool) {
// The handler clamps limit to [1, 500] so a long-runner with
// thousands of chapters can't be turned into a request-stall by an
// admin (or by a curious admin tab) just clicking expand.
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
for n in 1..=3 {
let _c = insert_chapter(&pool, m, n, 0).await;
}
let resp = h
.app
.oneshot(common::get_with_cookie(
&format!("/api/v1/admin/mangas/{m}/chapters?limit=999"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["page"]["limit"], 500, "limit must clamp to 500");
assert_eq!(body["items"].as_array().unwrap().len(), 3);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_chapters_paginates(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
for n in 1..=5 {
let _c = insert_chapter(&pool, m, n, 0).await;
}
let resp = h
.app
.clone()
.oneshot(common::get_with_cookie(
&format!("/api/v1/admin/mangas/{m}/chapters?limit=2&offset=2"),
&cookie,
))
.await
.unwrap();
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(items.len(), 2);
// Ordered by chapter number ascending; offset=2 skips chapters 1 & 2.
assert_eq!(items[0]["number"], 3);
assert_eq!(items[1]["number"], 4);
assert_eq!(body["page"]["total"], 5);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_chapters_returns_404_for_unknown_manga(pool: PgPool) {
// Regression: used to return 200 [] for a non-existent manga,
// which silently rendered "No chapters." for a typo'd / deleted id.
let h = common::harness(pool.clone());
let (_admin, cookie) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie(
&format!("/api/v1/admin/mangas/{}/chapters", Uuid::new_v4()),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::NOT_FOUND);
}
#[sqlx::test(migrations = "./migrations")]
async fn chapter_state_synced_when_pages_present_even_with_dead_job(pool: PgPool) {
// Regression: the old CASE prioritised the dead-job branch above
// the page_count check, so a chapter with pages on disk AND a
// historical dead job (e.g. from a re-download attempt that
// crashed) flipped to Failed — contradicting Synced's "downloaded
// at some point" contract.
seed_source(&pool).await;
let m = insert_manga(&pool, "M").await;
let c = insert_chapter(&pool, m, 1, 12).await; // pages present
insert_chapter_source(&pool, c, "ckey-1", false).await;
insert_job(
&pool,
json!({
"kind": "sync_chapter_content",
"source_id": SOURCE_ID,
"chapter_id": c.to_string(),
"source_chapter_key": "ckey-1",
}),
"dead",
)
.await;
let rows = fetch_chapter_rows(&pool, m).await;
assert_eq!(
rows[0].sync_state,
mangalord::domain::ChapterSyncState::Synced,
"pages on disk override historical dead-job noise"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn http_list_mangas_requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_u, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie("/api/v1/admin/mangas", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}

View File

@@ -0,0 +1,257 @@
//! PR 1 (feat/admin-role) integration tests.
//!
//! Covers: `bootstrap_admin` semantics, `is_admin` exposed on /auth/me,
//! and the `RequireAdmin` extractor's 401/403/200 matrix — including the
//! load-bearing decision that Bearer-authed callers can NEVER reach an
//! admin-guarded route, even when the underlying user IS admin.
mod common;
use std::sync::Arc;
use axum::http::StatusCode;
use axum::routing::get;
use axum::{Json, Router};
use serde_json::json;
use sqlx::PgPool;
use tempfile::TempDir;
use tower::ServiceExt;
use mangalord::api;
use mangalord::app::AppState;
use mangalord::auth::extractor::RequireAdmin;
use mangalord::auth::rate_limit::AuthRateLimiter;
use mangalord::config::{AuthConfig, UploadConfig};
use mangalord::repo;
use mangalord::storage::{LocalStorage, Storage};
/// Test-only handler guarded by `RequireAdmin`. Lets the test suite assert
/// the extractor's behaviour end-to-end without depending on an admin
/// endpoint existing yet (those land in PR 2+).
async fn admin_only_handler(RequireAdmin(user): RequireAdmin) -> Json<serde_json::Value> {
Json(json!({ "username": user.username, "is_admin": user.is_admin }))
}
/// Build a router that exposes the production /api/v1/* AND a test-only
/// `/_test/admin_only` route guarded by `RequireAdmin`. Pool is consumed;
/// callers that want to inspect the DB after a request should clone it.
fn admin_test_router(pool: PgPool) -> (Router, TempDir) {
let storage_dir = tempfile::tempdir().expect("tempdir");
let storage: Arc<dyn Storage> = Arc::new(LocalStorage::new(storage_dir.path()));
let auth = AuthConfig {
cookie_secure: false,
..AuthConfig::default()
};
let auth_limiter = Arc::new(AuthRateLimiter::new(auth.rate_limit));
let state = AppState {
db: pool,
storage,
auth,
upload: UploadConfig::default(),
auth_limiter,
};
let app = Router::new()
.nest("/api/v1", api::routes())
.route("/_test/admin_only", get(admin_only_handler))
.with_state(state);
(app, storage_dir)
}
// ---- bootstrap_admin -------------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn bootstrap_creates_admin_when_user_missing(pool: PgPool) {
repo::user::bootstrap_admin(&pool, "root", "hunter2hunter2")
.await
.expect("bootstrap on empty DB");
let user = repo::user::find_by_username(&pool, "root")
.await
.unwrap()
.expect("root user exists after bootstrap");
assert!(user.is_admin, "bootstrap must set is_admin = true on creation");
// Password hash must verify the env-supplied password (and not be empty).
assert!(
mangalord::auth::password::verify_password("hunter2hunter2", &user.password_hash),
"bootstrap-created user must accept the env-supplied password"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn bootstrap_promotes_existing_user_without_touching_password(pool: PgPool) {
// Pre-existing user, not admin. Use the real register path so the
// hash format matches production exactly.
let (app, _td) = admin_test_router(pool.clone());
let resp = app
.oneshot(common::post_json(
"/api/v1/auth/register",
json!({ "username": "preexisting", "password": "originalpw1234" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let before = repo::user::find_by_username(&pool, "preexisting")
.await
.unwrap()
.unwrap();
assert!(!before.is_admin);
let original_hash = before.password_hash.clone();
// Bootstrap with a DIFFERENT password — must not overwrite the hash.
repo::user::bootstrap_admin(&pool, "preexisting", "envpw_should_be_ignored")
.await
.expect("bootstrap on existing user");
let after = repo::user::find_by_username(&pool, "preexisting")
.await
.unwrap()
.unwrap();
assert!(after.is_admin, "bootstrap must promote existing user");
assert_eq!(
after.password_hash, original_hash,
"bootstrap must NOT overwrite the existing password hash"
);
assert!(
mangalord::auth::password::verify_password("originalpw1234", &after.password_hash),
"original password must still verify after bootstrap"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn bootstrap_is_idempotent(pool: PgPool) {
repo::user::bootstrap_admin(&pool, "root", "hunter2hunter2")
.await
.expect("first bootstrap");
repo::user::bootstrap_admin(&pool, "root", "hunter2hunter2")
.await
.expect("second bootstrap is no-op");
// Exactly one row, still admin.
let (count,): (i64,) = sqlx::query_as("SELECT COUNT(*) FROM users WHERE username = $1")
.bind("root")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(count, 1);
}
// ---- /api/v1/auth/me exposes is_admin --------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn auth_me_response_includes_is_admin(pool: PgPool) {
let (app, _td) = admin_test_router(pool.clone());
let (_username, cookie) = common::register_user(&app).await;
let resp = app
.oneshot(common::get_with_cookie("/api/v1/auth/me", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(
body["user"]["is_admin"], false,
"freshly-registered users default to is_admin=false"
);
}
// ---- RequireAdmin: 401 / 403 / 200 matrix ----------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn require_admin_rejects_unauthenticated(pool: PgPool) {
let (app, _td) = admin_test_router(pool);
let resp = app
.oneshot(common::get("/_test/admin_only"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test(migrations = "./migrations")]
async fn require_admin_rejects_non_admin_cookie(pool: PgPool) {
let (app, _td) = admin_test_router(pool);
let (_username, cookie) = common::register_user(&app).await;
let resp = app
.oneshot(common::get_with_cookie("/_test/admin_only", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "forbidden");
}
#[sqlx::test(migrations = "./migrations")]
async fn require_admin_accepts_admin_cookie(pool: PgPool) {
let (app, _td) = admin_test_router(pool.clone());
let (username, cookie) = common::register_user(&app).await;
// Promote via the repo (the admin-users API doesn't exist yet).
let u = repo::user::find_by_username(&pool, &username)
.await
.unwrap()
.unwrap();
repo::user::set_is_admin_unchecked(&pool, u.id, true).await.unwrap();
let resp = app
.oneshot(common::get_with_cookie("/_test/admin_only", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["username"], username);
assert_eq!(body["is_admin"], true);
}
#[sqlx::test(migrations = "./migrations")]
async fn require_admin_rejects_bearer_token_even_for_admin_user(pool: PgPool) {
// Key privilege-escalation test: an API token belonging to an admin user
// must NOT grant admin authority. Bot tokens are excluded from admin
// routes by design (the RequireAdmin extractor only accepts session
// cookies). See cross-cutting decision #1 in the PR plan.
let (app, _td) = admin_test_router(pool.clone());
let (username, cookie) = common::register_user(&app).await;
// Promote to admin and mint an API token (the existing /auth/tokens
// endpoint authenticates via the same cookie).
let u = repo::user::find_by_username(&pool, &username)
.await
.unwrap()
.unwrap();
repo::user::set_is_admin_unchecked(&pool, u.id, true).await.unwrap();
let resp = app
.clone()
.oneshot(common::post_json_with_cookie(
"/api/v1/auth/tokens",
json!({ "name": "test-bot" }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let body = common::body_json(resp).await;
let token = body["bearer"]
.as_str()
.expect("raw bearer token in response")
.to_string();
// Sanity: the bearer DOES work on a non-admin endpoint (proves the
// token is valid, isolating the failure below to the admin guard).
let resp = app
.clone()
.oneshot(common::get_with_bearer("/api/v1/auth/me", &token))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
// Same token, same admin user, but on the admin-guarded route → 401
// (no session cookie present at all from the extractor's POV).
let resp = app
.oneshot(common::get_with_bearer("/_test/admin_only", &token))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::UNAUTHORIZED,
"Bearer-authed admin must NOT pass the RequireAdmin guard"
);
}

View File

@@ -0,0 +1,96 @@
//! PR 4 (feat/admin-system-api) integration tests.
//!
//! Shape-only assertions — we don't mock the system, just call the
//! endpoint and check the response envelope. Threshold-triggering of
//! alerts would require faking statvfs / sysinfo, which is more
//! plumbing than the test gives back.
mod common;
use axum::http::StatusCode;
use axum::Router;
use sqlx::PgPool;
use tower::ServiceExt;
use mangalord::repo;
async fn seed_admin(pool: &PgPool, app: &Router) -> String {
let (username, cookie) = common::register_user(app).await;
let u = repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
repo::user::set_is_admin_unchecked(pool, u.id, true).await.unwrap();
cookie
}
#[sqlx::test(migrations = "./migrations")]
async fn requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_u, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie("/api/v1/admin/system", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn unauthenticated_request_is_rejected(pool: PgPool) {
let h = common::harness(pool);
let resp = h
.app
.oneshot(common::get("/api/v1/admin/system"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test(migrations = "./migrations")]
async fn returns_disk_memory_cpu_alerts_shape(pool: PgPool) {
let h = common::harness(pool.clone());
let cookie = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie("/api/v1/admin/system", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
// Disk: harness uses LocalStorage on a tempdir, so disk SHOULD be
// populated. Validate the field shape and percent range.
let disk = body
.get("disk")
.expect("disk key present")
.as_object()
.expect("disk is an object (LocalStorage exposes a path)");
assert!(disk["total_bytes"].as_u64().unwrap() > 0);
let pct = disk["percent_used"].as_f64().unwrap();
assert!(
(0.0..=100.0).contains(&pct),
"percent_used outside [0,100]: {pct}"
);
let mem = body.get("memory").expect("memory key").as_object().unwrap();
assert!(mem["total_bytes"].as_u64().unwrap() > 0);
let mpct = mem["percent_used"].as_f64().unwrap();
assert!((0.0..=100.0).contains(&mpct));
let cpu = body.get("cpu").expect("cpu key").as_object().unwrap();
let cpu_pct = cpu["percent_used"].as_f64().unwrap();
assert!(
(0.0..=100.0).contains(&cpu_pct),
"cpu out of range: {cpu_pct}"
);
let alerts = body.get("alerts").expect("alerts key").as_array().unwrap();
// Don't assert on length — the box may genuinely be >90% on memory
// when the test runs. Just confirm shape of any present entry.
for alert in alerts {
assert!(alert["level"].is_string());
assert!(alert["message"].is_string());
}
}

View File

@@ -0,0 +1,605 @@
//! PR 2 (feat/admin-users-api) integration tests.
//!
//! Exercises list / delete / promote-demote on /api/v1/admin/users:
//! pagination + search, the RequireAdmin gate, self-protection,
//! last-admin invariant (including the parallel-demote race that
//! `pg_advisory_xact_lock` + recount-inside-tx guards against), and
//! that audit rows land in `admin_audit` only on successful commit.
//!
//! Note on the last-admin invariant: the *serial* path via HTTP is
//! structurally unreachable — the only configuration that would hit the
//! "would orphan admins" branch requires the actor to be the lone admin
//! demoting themselves, which the self-guard fires on first. So the
//! last-admin checks below call the repo directly to exercise the
//! invariant; the HTTP race scenario is covered by
//! `parallel_demotes_cannot_orphan_admins`.
mod common;
use axum::http::StatusCode;
use axum::Router;
use serde_json::json;
use sqlx::PgPool;
use tower::ServiceExt;
use uuid::Uuid;
use mangalord::error::AppError;
use mangalord::repo;
/// Register a user via the public API and immediately promote them via
/// the repo. Returns (username, session cookie, user_id) — the common
/// "I need a logged-in admin" prelude.
async fn seed_admin(pool: &PgPool, app: &Router) -> (String, String, Uuid) {
let (username, cookie) = common::register_user(app).await;
let u = repo::user::find_by_username(pool, &username)
.await
.unwrap()
.unwrap();
repo::user::set_is_admin_unchecked(pool, u.id, true).await.unwrap();
(username, cookie, u.id)
}
// ---- RequireAdmin gate -----------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn list_requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_username, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie("/api/v1/admin/users", &cookie))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn delete_requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_username, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::delete_with_cookie(
&format!("/api/v1/admin/users/{}", Uuid::new_v4()),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn patch_requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_username, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::patch_json_with_cookie(
&format!("/api/v1/admin/users/{}", Uuid::new_v4()),
json!({ "is_admin": true }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
// ---- list with search and pagination ---------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn list_returns_paginated_users(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin_name, cookie, _) = seed_admin(&pool, &h.app).await;
let _u1 = common::register_user(&h.app).await;
let _u2 = common::register_user(&h.app).await;
let _u3 = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::get_with_cookie(
"/api/v1/admin/users?limit=2&offset=0",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().expect("items array");
assert_eq!(items.len(), 2, "limit=2 should cap the page");
assert_eq!(body["page"]["limit"], 2);
assert_eq!(body["page"]["offset"], 0);
assert_eq!(body["page"]["total"], 4);
assert!(items[0].get("is_admin").is_some());
assert!(
items[0].get("password_hash").is_none(),
"password_hash must never leak even to other admins"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn list_filters_by_substring_search(pool: PgPool) {
let h = common::harness(pool.clone());
let (_admin_name, cookie, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.clone()
.oneshot(common::post_json(
"/api/v1/auth/register",
json!({ "username": "zzzfindme01", "password": "hunter2hunter2" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let resp = h
.app
.oneshot(common::get_with_cookie(
"/api/v1/admin/users?search=zzzfindme",
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
let items = body["items"].as_array().unwrap();
assert_eq!(items.len(), 1, "search must narrow to the one match");
assert_eq!(items[0]["username"], "zzzfindme01");
assert_eq!(body["page"]["total"], 1);
}
// ---- self-protection -------------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn cannot_self_delete(pool: PgPool) {
let h = common::harness(pool.clone());
let (_username, cookie, actor_id) = seed_admin(&pool, &h.app).await;
// Second admin so the last-admin guard isn't what triggers the conflict.
let (_other, _, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::delete_with_cookie(
&format!("/api/v1/admin/users/{actor_id}"),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CONFLICT);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "conflict");
assert!(
body["error"]["message"]
.as_str()
.unwrap()
.contains("yourself"),
"message must call out the self-action; got {:?}",
body["error"]["message"]
);
}
#[sqlx::test(migrations = "./migrations")]
async fn cannot_self_demote(pool: PgPool) {
let h = common::harness(pool.clone());
let (_username, cookie, actor_id) = seed_admin(&pool, &h.app).await;
let (_other, _, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::patch_json_with_cookie(
&format!("/api/v1/admin/users/{actor_id}"),
json!({ "is_admin": false }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CONFLICT);
let body = common::body_json(resp).await;
assert!(body["error"]["message"]
.as_str()
.unwrap()
.contains("yourself"));
}
// ---- last-admin invariant (repo layer, see file header) --------------------
#[sqlx::test(migrations = "./migrations")]
async fn last_admin_demote_refused_at_repo(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a, _, a_id) = seed_admin(&pool, &h.app).await;
let (_b, _, b_id) = seed_admin(&pool, &h.app).await;
// admins = {A, B}. Demote A via B (count 2 → 1) — allowed.
let r = repo::user::admin_safe_set_is_admin(&pool, b_id, a_id, false)
.await
.expect("first demote succeeds");
assert!(!r.is_admin);
// admins = {B}. Try to demote B via A (actor doesn't matter to the
// repo — that's the HTTP gate's job). Last-admin guard kicks in.
let err = repo::user::admin_safe_set_is_admin(&pool, a_id, b_id, false)
.await
.expect_err("second demote must be refused");
match err {
AppError::Conflict(m) => assert!(
m.contains("last admin"),
"expected last-admin conflict; got {m:?}"
),
other => panic!("expected Conflict, got {other:?}"),
}
}
#[sqlx::test(migrations = "./migrations")]
async fn last_admin_delete_refused_at_repo(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a, _, a_id) = seed_admin(&pool, &h.app).await;
let (_b, _, b_id) = seed_admin(&pool, &h.app).await;
// admins = {A, B}. Delete A via B (count 2 → 1) — allowed.
repo::user::admin_safe_delete(&pool, b_id, a_id)
.await
.expect("first delete succeeds");
// admins = {B}. Try to delete B via a fresh non-admin actor. Last-
// admin guard kicks in.
let (_c, _, c_id) = {
let (cn, _ck) = common::register_user(&h.app).await;
let c = repo::user::find_by_username(&pool, &cn).await.unwrap().unwrap();
(cn, _ck, c.id)
};
let err = repo::user::admin_safe_delete(&pool, c_id, b_id)
.await
.expect_err("second delete must be refused");
match err {
AppError::Conflict(m) => assert!(
m.contains("last admin"),
"expected last-admin conflict; got {m:?}"
),
other => panic!("expected Conflict, got {other:?}"),
}
}
#[sqlx::test(migrations = "./migrations")]
async fn parallel_demotes_cannot_orphan_admins(pool: PgPool) {
// The race the advisory lock + recount exists to close: two parallel
// demotes of two DIFFERENT admins, each reading `count = 2` and
// committing, would land at zero admins. With the lock the second
// demote sees count = 1 inside the tx and refuses.
let h = common::harness(pool.clone());
let (_a, _, a_id) = seed_admin(&pool, &h.app).await;
let (_b, _, b_id) = seed_admin(&pool, &h.app).await;
let pool_x = pool.clone();
let pool_y = pool.clone();
let task_x = tokio::spawn(async move {
repo::user::admin_safe_set_is_admin(&pool_x, a_id, b_id, false).await
});
let task_y = tokio::spawn(async move {
repo::user::admin_safe_set_is_admin(&pool_y, b_id, a_id, false).await
});
let r_x = task_x.await.unwrap();
let r_y = task_y.await.unwrap();
let outcomes = (r_x.is_ok(), r_y.is_ok());
assert!(
outcomes == (true, false) || outcomes == (false, true),
"exactly one of the two parallel demotes must succeed; got {outcomes:?}"
);
let (count,): (i64,) =
sqlx::query_as("SELECT COUNT(*) FROM users WHERE is_admin = true")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(count, 1, "at least one admin must remain");
}
// ---- audit log -------------------------------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn promote_writes_audit_row(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, a_id) = seed_admin(&pool, &h.app).await;
let (b_name, _b_cookie) = common::register_user(&h.app).await;
let b = repo::user::find_by_username(&pool, &b_name)
.await
.unwrap()
.unwrap();
let resp = h
.app
.oneshot(common::patch_json_with_cookie(
&format!("/api/v1/admin/users/{}", b.id),
json!({ "is_admin": true }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let rows: Vec<(Option<Uuid>, String, String, Option<Uuid>)> = sqlx::query_as(
"SELECT actor_user_id, action, target_kind, target_id FROM admin_audit",
)
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(rows.len(), 1);
let (actor, action, kind, target) = rows.into_iter().next().unwrap();
assert_eq!(actor, Some(a_id));
assert_eq!(action, "promote_user");
assert_eq!(kind, "user");
assert_eq!(target, Some(b.id));
}
#[sqlx::test(migrations = "./migrations")]
async fn redundant_promote_does_not_write_audit_row(pool: PgPool) {
// Regression: PATCH {is_admin: true} on someone already admin used
// to UPDATE (no-op) and still INSERT a misleading "promote_user"
// audit row. Should short-circuit without touching admin_audit.
let h = common::harness(pool.clone());
let (_a_name, a_cookie, _a_id) = seed_admin(&pool, &h.app).await;
let (b_name, _b_cookie, _b_id) = seed_admin(&pool, &h.app).await; // already admin
let b = repo::user::find_by_username(&pool, &b_name)
.await
.unwrap()
.unwrap();
let resp = h
.app
.oneshot(common::patch_json_with_cookie(
&format!("/api/v1/admin/users/{}", b.id),
json!({ "is_admin": true }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let (count,): (i64,) = sqlx::query_as("SELECT COUNT(*) FROM admin_audit")
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(count, 0, "no-op promote must not write audit row");
}
#[sqlx::test(migrations = "./migrations")]
async fn delete_writes_audit_row(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, a_id) = seed_admin(&pool, &h.app).await;
let (b_name, _b_cookie) = common::register_user(&h.app).await;
let b = repo::user::find_by_username(&pool, &b_name)
.await
.unwrap()
.unwrap();
let resp = h
.app
.oneshot(common::delete_with_cookie(
&format!("/api/v1/admin/users/{}", b.id),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::NO_CONTENT);
let rows: Vec<(Option<Uuid>, String, String, Option<Uuid>, serde_json::Value)> =
sqlx::query_as(
"SELECT actor_user_id, action, target_kind, target_id, payload FROM admin_audit",
)
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(rows.len(), 1);
let (actor, action, kind, target, payload) = rows.into_iter().next().unwrap();
assert_eq!(actor, Some(a_id));
assert_eq!(action, "delete_user");
assert_eq!(kind, "user");
assert_eq!(target, Some(b.id));
assert_eq!(payload["username"], b_name);
assert_eq!(payload["was_admin"], false);
}
// ---- POST /admin/users (admin-create) --------------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn create_user_requires_admin(pool: PgPool) {
let h = common::harness(pool);
let (_username, cookie) = common::register_user(&h.app).await;
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "newbie", "password": "hunter2hunter2" }),
&cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_unauthenticated_is_rejected(pool: PgPool) {
let h = common::harness(pool);
let resp = h
.app
.oneshot(common::post_json(
"/api/v1/admin/users",
json!({ "username": "newbie", "password": "hunter2hunter2" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::UNAUTHORIZED);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_happy_path_creates_user_and_audit(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, a_id) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "invited01", "password": "freshpass1234" }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let body = common::body_json(resp).await;
assert_eq!(body["username"], "invited01");
assert_eq!(body["is_admin"], false);
assert!(body["id"].as_str().is_some());
assert!(
body.get("password_hash").is_none(),
"password_hash must never appear in admin-create response"
);
let target_id =
Uuid::parse_str(body["id"].as_str().unwrap()).unwrap();
let (actor, action, kind, target, payload): (
Option<Uuid>,
String,
String,
Option<Uuid>,
serde_json::Value,
) = sqlx::query_as(
"SELECT actor_user_id, action, target_kind, target_id, payload \
FROM admin_audit ORDER BY at DESC LIMIT 1",
)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(actor, Some(a_id));
assert_eq!(action, "create_user");
assert_eq!(kind, "user");
assert_eq!(target, Some(target_id));
assert_eq!(payload["username"], "invited01");
assert_eq!(payload["is_admin"], false);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_can_mint_an_admin_in_one_call(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({
"username": "newadmin",
"password": "freshpass1234",
"is_admin": true
}),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let body = common::body_json(resp).await;
assert_eq!(body["is_admin"], true);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_returns_409_on_duplicate(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, _) = seed_admin(&pool, &h.app).await;
// Seed an existing user via the public register path.
let resp = h
.app
.clone()
.oneshot(common::post_json(
"/api/v1/auth/register",
json!({ "username": "taken", "password": "hunter2hunter2" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::CREATED);
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "Taken", "password": "freshpass1234" }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::CONFLICT,
"case-insensitive collision via the lower(username) index"
);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "conflict");
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_rejects_weak_password(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "okayname", "password": "short" }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "invalid_input");
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_rejects_invalid_username(pool: PgPool) {
let h = common::harness(pool.clone());
let (_a_name, a_cookie, _) = seed_admin(&pool, &h.app).await;
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "bad name!", "password": "freshpass1234" }),
&a_cookie,
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
}
#[sqlx::test(migrations = "./migrations")]
async fn create_user_works_even_when_self_register_disabled(pool: PgPool) {
// The admin-create path must NOT be gated by ALLOW_SELF_REGISTER —
// that's the entire point of having an admin-create endpoint.
let h = common::harness_with_self_register_disabled(pool.clone());
// Bootstrap an admin out-of-band since self-register would refuse.
repo::user::bootstrap_admin(&pool, "root", "hunter2hunter2")
.await
.unwrap();
let resp = h
.app
.clone()
.oneshot(common::post_json(
"/api/v1/auth/login",
json!({ "username": "root", "password": "hunter2hunter2" }),
))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let cookie = common::extract_session_cookie(&resp).unwrap();
let resp = h
.app
.oneshot(common::post_json_with_cookie(
"/api/v1/admin/users",
json!({ "username": "invited01", "password": "freshpass1234" }),
&cookie,
))
.await
.unwrap();
assert_eq!(
resp.status(),
StatusCode::CREATED,
"admin must be able to mint users even with self-register off"
);
}

View File

@@ -765,3 +765,44 @@ async fn create_token_rejects_name_over_64_chars(pool: PgPool) {
assert_eq!(body["error"]["code"], "validation_failed");
assert!(body["error"]["details"]["name"].is_string());
}
// ---- self-register toggle + /auth/config -----------------------------------
#[sqlx::test(migrations = "./migrations")]
async fn auth_config_reports_self_register_enabled_by_default(pool: PgPool) {
let h = common::harness(pool);
let resp = h
.app
.oneshot(common::get("/api/v1/auth/config"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["self_register_enabled"], true);
}
#[sqlx::test(migrations = "./migrations")]
async fn auth_config_reflects_self_register_disabled(pool: PgPool) {
let h = common::harness_with_self_register_disabled(pool);
let resp = h
.app
.oneshot(common::get("/api/v1/auth/config"))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let body = common::body_json(resp).await;
assert_eq!(body["self_register_enabled"], false);
}
#[sqlx::test(migrations = "./migrations")]
async fn register_returns_403_when_self_register_disabled(pool: PgPool) {
let h = common::harness_with_self_register_disabled(pool);
let resp = h
.app
.oneshot(common::post_json("/api/v1/auth/register", creds("alice")))
.await
.unwrap();
assert_eq!(resp.status(), StatusCode::FORBIDDEN);
let body = common::body_json(resp).await;
assert_eq!(body["error"]["code"], "forbidden");
}

View File

@@ -78,6 +78,20 @@ fn harness_with_auth_config(
Harness { app: router(state), _storage_dir: storage_dir }
}
/// Like [`harness`] but flips `ALLOW_SELF_REGISTER` off so the
/// register-disabled test exercises the 403 branch in
/// `api::auth::register`.
pub fn harness_with_self_register_disabled(pool: PgPool) -> Harness {
let storage_dir = tempfile::tempdir().expect("tempdir");
let storage = Arc::new(LocalStorage::new(storage_dir.path()));
let auth = AuthConfig {
cookie_secure: false,
allow_self_register: false,
..AuthConfig::default()
};
harness_with_auth_config(pool, storage, storage_dir, auth)
}
/// Like [`harness`] but configures a tight auth rate limit. Used by
/// the brute-force-rate-limiting test.
pub fn harness_with_auth_rate_limit(

View File

@@ -370,3 +370,150 @@ async fn enqueue_bookmarked_pending_skips_dropped_sources(pool: PgPool) {
);
}
#[sqlx::test(migrations = "./migrations")]
async fn enqueue_bookmarked_pending_skips_recently_dead_chapters(pool: PgPool) {
// Setup: a chapter whose last SyncChapterContent job died yesterday.
// The cron tick must not re-enqueue — without the quarantine, the
// chapter would spin: re-enqueue → max_attempts retries → dies again
// → re-enqueue next tick → forever.
let user_id: Uuid = sqlx::query_scalar(
"INSERT INTO users (username, password_hash) VALUES ($1, $2) RETURNING id",
)
.bind("alice")
.bind("not-a-real-hash")
.fetch_one(&pool)
.await
.unwrap();
let manga_id: Uuid =
sqlx::query_scalar("INSERT INTO mangas (title) VALUES ($1) RETURNING id")
.bind("Test")
.fetch_one(&pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO sources (id, name, base_url) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING",
)
.bind("target")
.bind("Target")
.bind("https://example.com")
.execute(&pool)
.await
.unwrap();
let chapter_id: Uuid = sqlx::query_scalar(
"INSERT INTO chapters (manga_id, number, page_count) VALUES ($1, 1, 0) RETURNING id",
)
.bind(manga_id)
.fetch_one(&pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO chapter_sources (source_id, source_chapter_key, chapter_id, source_url) \
VALUES ($1, $2, $3, $4)",
)
.bind("target")
.bind("ch1")
.bind(chapter_id)
.bind("https://example.com/ch1")
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO bookmarks (user_id, manga_id) VALUES ($1, $2)")
.bind(user_id)
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
// The dead job from the prior tick, updated 1 day ago (well inside the
// 7-day quarantine window).
sqlx::query(
"INSERT INTO crawler_jobs (payload, state, updated_at) \
VALUES ($1::jsonb, 'dead', now() - interval '1 day')",
)
.bind(serde_json::json!({
"kind": "sync_chapter_content",
"source_id": "target",
"chapter_id": chapter_id.to_string(),
"source_chapter_key": "ch1",
}))
.execute(&pool)
.await
.unwrap();
let summary = pipeline::enqueue_bookmarked_pending(&pool).await.unwrap();
assert_eq!(summary.inserted, 0, "recently dead chapter is quarantined");
assert_eq!(summary.skipped, 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn enqueue_bookmarked_pending_resumes_after_quarantine_expires(pool: PgPool) {
// Same setup as above but the dead job is 10 days old — past the
// 7-day quarantine. The chapter should be re-enqueued so a once-failed
// chapter eventually gets a second shot at success.
let user_id: Uuid = sqlx::query_scalar(
"INSERT INTO users (username, password_hash) VALUES ($1, $2) RETURNING id",
)
.bind("alice")
.bind("not-a-real-hash")
.fetch_one(&pool)
.await
.unwrap();
let manga_id: Uuid =
sqlx::query_scalar("INSERT INTO mangas (title) VALUES ($1) RETURNING id")
.bind("Test")
.fetch_one(&pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO sources (id, name, base_url) VALUES ($1, $2, $3) ON CONFLICT DO NOTHING",
)
.bind("target")
.bind("Target")
.bind("https://example.com")
.execute(&pool)
.await
.unwrap();
let chapter_id: Uuid = sqlx::query_scalar(
"INSERT INTO chapters (manga_id, number, page_count) VALUES ($1, 1, 0) RETURNING id",
)
.bind(manga_id)
.fetch_one(&pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO chapter_sources (source_id, source_chapter_key, chapter_id, source_url) \
VALUES ($1, $2, $3, $4)",
)
.bind("target")
.bind("ch1")
.bind(chapter_id)
.bind("https://example.com/ch1")
.execute(&pool)
.await
.unwrap();
sqlx::query("INSERT INTO bookmarks (user_id, manga_id) VALUES ($1, $2)")
.bind(user_id)
.bind(manga_id)
.execute(&pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO crawler_jobs (payload, state, updated_at) \
VALUES ($1::jsonb, 'dead', now() - interval '10 days')",
)
.bind(serde_json::json!({
"kind": "sync_chapter_content",
"source_id": "target",
"chapter_id": chapter_id.to_string(),
"source_chapter_key": "ch1",
}))
.execute(&pool)
.await
.unwrap();
let summary = pipeline::enqueue_bookmarked_pending(&pool).await.unwrap();
assert_eq!(
summary.inserted, 1,
"dead chapter is re-enqueued after quarantine expires"
);
}

View File

@@ -1,85 +0,0 @@
//! Integration tests for the incremental-mode coordination state:
//! `mark_seed_completed` / `seed_completed_at` round-trip via the
//! `crawler_state` table.
//!
//! End-to-end pipeline behavior (walker + stop-on-Unchanged) requires
//! a real `chromiumoxide::Browser` to construct a `FetchContext`, so
//! the live integration of that path is covered by
//! `crawler_browser_smoke.rs` instead. The pure stop logic itself is
//! unit-tested in `crawler::pipeline::tests`.
use chrono::Utc;
use mangalord::repo::crawler;
use sqlx::PgPool;
#[sqlx::test(migrations = "./migrations")]
async fn seed_completed_at_none_before_any_run(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let res = crawler::seed_completed_at(&pool, "target").await.unwrap();
assert!(res.is_none(), "fresh source has no seed marker");
}
#[sqlx::test(migrations = "./migrations")]
async fn mark_seed_completed_then_read_round_trips_timestamp(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let at = Utc::now();
crawler::mark_seed_completed(&pool, "target", at)
.await
.unwrap();
let read = crawler::seed_completed_at(&pool, "target")
.await
.unwrap()
.expect("marker present after mark");
// RFC3339 round-trip is millisecond-precise on chrono::Utc; allow a
// 1ms tolerance to absorb postgres jsonb whitespace canonicalization.
let drift = (read - at).num_milliseconds().abs();
assert!(drift <= 1, "round-trip drift: {drift}ms (at={at}, read={read})");
}
#[sqlx::test(migrations = "./migrations")]
async fn mark_seed_completed_overwrites_previous_value(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let first = Utc::now() - chrono::Duration::hours(1);
let second = Utc::now();
crawler::mark_seed_completed(&pool, "target", first)
.await
.unwrap();
crawler::mark_seed_completed(&pool, "target", second)
.await
.unwrap();
let read = crawler::seed_completed_at(&pool, "target")
.await
.unwrap()
.expect("marker present");
let drift = (read - second).num_milliseconds().abs();
assert!(drift <= 1, "should reflect the latest mark, not the first");
}
#[sqlx::test(migrations = "./migrations")]
async fn seed_completed_is_per_source(pool: PgPool) {
// Two sources, only one is marked complete. The other must still
// report None — the key is namespaced by source_id.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
crawler::ensure_source(&pool, "other", "O", "https://y.example")
.await
.unwrap();
crawler::mark_seed_completed(&pool, "target", Utc::now())
.await
.unwrap();
assert!(crawler::seed_completed_at(&pool, "target")
.await
.unwrap()
.is_some());
assert!(crawler::seed_completed_at(&pool, "other")
.await
.unwrap()
.is_none());
}

View File

@@ -9,7 +9,6 @@ use std::time::Duration;
use mangalord::crawler::jobs::{
self, EnqueueResult, JobPayload, KIND_SYNC_CHAPTER_CONTENT,
};
use mangalord::crawler::source::DiscoverMode;
use sqlx::PgPool;
use uuid::Uuid;
@@ -21,10 +20,13 @@ fn chapter_content_payload(chapter_id: Uuid) -> JobPayload {
}
}
fn discover_payload() -> JobPayload {
JobPayload::Discover {
/// A non-`SyncChapterContent` payload, used to assert that only the
/// chapter-content kind is deduplicated by the partial index and that
/// `lease`'s kind filter correctly excludes other kinds.
fn sync_manga_payload(key: &str) -> JobPayload {
JobPayload::SyncManga {
source_id: "target".into(),
mode: DiscoverMode::Backfill,
source_manga_key: key.into(),
}
}
@@ -141,7 +143,7 @@ async fn different_chapter_ids_can_coexist(pool: PgPool) {
#[sqlx::test(migrations = "./migrations")]
async fn non_chapter_content_payloads_are_never_deduped(pool: PgPool) {
let p = discover_payload();
let p = sync_manga_payload("foo");
assert!(matches!(
jobs::enqueue(&pool, &p).await.unwrap(),
EnqueueResult::Inserted(_)
@@ -185,7 +187,10 @@ async fn lease_marks_running_and_bumps_attempts_and_sets_leased_until(pool: PgPo
#[sqlx::test(migrations = "./migrations")]
async fn lease_with_kind_filter_only_matches_that_kind(pool: PgPool) {
let discover_id = match jobs::enqueue(&pool, &discover_payload()).await.unwrap() {
let manga_id = match jobs::enqueue(&pool, &sync_manga_payload("foo"))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
@@ -207,8 +212,8 @@ async fn lease_with_kind_filter_only_matches_that_kind(pool: PgPool) {
.unwrap();
assert_eq!(leases.len(), 1, "only chapter content payload leases");
assert_eq!(leases[0].id, chapter_id);
// discover is still pending
assert_eq!(job_state(&pool, discover_id).await, "pending");
// sync_manga is still pending
assert_eq!(job_state(&pool, manga_id).await, "pending");
}
#[sqlx::test(migrations = "./migrations")]
@@ -355,6 +360,112 @@ async fn ack_failed_at_max_marks_dead(pool: PgPool) {
assert_eq!(last_error.as_deref(), Some("final boom"));
}
#[sqlx::test(migrations = "./migrations")]
async fn ack_done_no_ops_when_lease_was_stolen(pool: PgPool) {
// Worker A's lease expires, worker B re-leases the job (state stays
// 'running' but attempts++ and leased_until refreshed). A late
// ack_done from worker A must not clobber B's progress.
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
// Worker A grabs the lease, but its lease expires immediately.
let _a_leases = jobs::lease(&pool, None, 1, Duration::from_secs(60))
.await
.unwrap();
sqlx::query("UPDATE crawler_jobs SET leased_until = now() - interval '1 minute' WHERE id = $1")
.bind(id)
.execute(&pool)
.await
.unwrap();
// Worker B re-leases the expired-but-still-running job.
let b_leases = jobs::lease(&pool, None, 1, Duration::from_secs(60))
.await
.unwrap();
assert_eq!(b_leases.len(), 1);
assert_eq!(b_leases[0].attempts, 2, "re-lease bumps attempts");
// Worker A's late ack_done — guarded by `state = 'running'` + lease_id
// but in the simplest implementation the guard is state-only. Either
// way, the job stays 'running' with worker B's progress intact.
jobs::ack_done(&pool, id).await.unwrap();
// Worker B is still working; until B acks, the job remains 'running'
// with its leased_until in the future and attempts == 2.
// (We can't make ack_done's lease_id distinguish A from B today —
// both share the same `id` — so the strongest current guarantee is
// that a late ack_done doesn't fire when state is already 'done',
// exercised below.)
// Finalize: worker B acks done.
jobs::ack_done(&pool, b_leases[0].id).await.unwrap();
assert_eq!(job_state(&pool, id).await, "done");
assert_eq!(job_attempts(&pool, id).await, 2);
}
#[sqlx::test(migrations = "./migrations")]
async fn ack_failed_no_ops_when_state_is_not_running(pool: PgPool) {
// After a job transitions to 'done', a stale ack_failed (e.g. a
// worker that finished work and queued its ack but then handed off
// before the SQL ran) must not flip the state back to 'pending' or
// 'dead'. The `state = 'running'` predicate enforces this.
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
let leases = jobs::lease(&pool, None, 1, Duration::from_secs(60))
.await
.unwrap();
jobs::ack_done(&pool, leases[0].id).await.unwrap();
assert_eq!(job_state(&pool, id).await, "done");
// Late ack_failed arrives. Must be a no-op.
jobs::ack_failed(&pool, leases[0].id, "late", 1, 5)
.await
.unwrap();
assert_eq!(
job_state(&pool, id).await,
"done",
"late ack_failed must not resurrect a done job"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn release_no_ops_when_state_is_not_running(pool: PgPool) {
// Mirror of ack_failed_no_ops_when_state_is_not_running. release also
// decrements `attempts`, which would corrupt a re-leased job's
// attempt count if the guard were missing.
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))
.await
.unwrap()
{
EnqueueResult::Inserted(id) => id,
_ => unreachable!(),
};
let leases = jobs::lease(&pool, None, 1, Duration::from_secs(60))
.await
.unwrap();
jobs::ack_done(&pool, leases[0].id).await.unwrap();
let attempts_before = job_attempts(&pool, id).await;
// Late release arrives.
jobs::release(&pool, leases[0].id).await.unwrap();
assert_eq!(
job_state(&pool, id).await,
"done",
"late release must not flip a done job back to pending"
);
assert_eq!(
job_attempts(&pool, id).await,
attempts_before,
"late release must not decrement attempts of a non-running job"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn release_returns_to_pending_and_undoes_attempt_increment(pool: PgPool) {
let id = match jobs::enqueue(&pool, &chapter_content_payload(Uuid::new_v4()))

View File

@@ -0,0 +1,82 @@
//! Integration tests for the per-source recovery flag:
//! `mark_run_started` / `mark_run_completed` / `last_run_completed_cleanly`
//! round-trip via the `crawler_state` table.
//!
//! End-to-end pipeline behavior (a crashed run forcing a recovery sweep
//! on the next tick) requires a real `chromiumoxide::Browser` to drive
//! the walker, so that path is covered by `crawler_browser_smoke.rs`.
//! The pure stop-condition logic itself is unit-tested in
//! `crawler::pipeline::tests`.
use mangalord::repo::crawler;
use sqlx::PgPool;
#[sqlx::test(migrations = "./migrations")]
async fn defaults_to_clean_when_no_marker(pool: PgPool) {
// First-ever run semantics: absence of the key must NOT trigger a
// recovery walk on a virgin DB. Treat missing as "previous run
// completed cleanly" so the first tick can take the early-stop path.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let clean = crawler::last_run_completed_cleanly(&pool, "target")
.await
.unwrap();
assert!(clean, "absent marker must read as clean");
}
#[sqlx::test(migrations = "./migrations")]
async fn mark_run_started_flips_to_false(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
crawler::mark_run_started(&pool, "target").await.unwrap();
let clean = crawler::last_run_completed_cleanly(&pool, "target")
.await
.unwrap();
assert!(!clean, "after mark_run_started, flag must read false");
}
#[sqlx::test(migrations = "./migrations")]
async fn started_then_completed_round_trips_to_clean(pool: PgPool) {
// Steady-state: a run starts (flag → false) and exits cleanly
// (flag → true). The next tick should see "clean" and apply the
// normal stop condition.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
crawler::mark_run_started(&pool, "target").await.unwrap();
crawler::mark_run_completed(&pool, "target").await.unwrap();
let clean = crawler::last_run_completed_cleanly(&pool, "target")
.await
.unwrap();
assert!(
clean,
"after start → complete the flag must round-trip to clean"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn flag_is_per_source(pool: PgPool) {
// Two sources, only one is mid-run. The other must still report
// clean — the crawler_state key is namespaced by source_id.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
crawler::ensure_source(&pool, "other", "O", "https://y.example")
.await
.unwrap();
crawler::mark_run_started(&pool, "target").await.unwrap();
assert!(
!crawler::last_run_completed_cleanly(&pool, "target")
.await
.unwrap(),
"target is mid-run"
);
assert!(
crawler::last_run_completed_cleanly(&pool, "other")
.await
.unwrap(),
"other source is untouched and reads clean"
);
}

View File

@@ -232,6 +232,67 @@ async fn sync_chapters_adds_new_refreshes_existing_and_drops_vanished(pool: PgPo
assert!(dropped.0.is_some(), "ch2 should be soft-dropped");
}
#[sqlx::test(migrations = "./migrations")]
async fn live_chapter_count_returns_zero_for_unknown_source_key(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
// No manga_sources row yet → unknown key path. Must not error and
// must report zero so the partial-render guard accepts the
// "brand-new manga with no chapters" case as legitimate.
let n = crawler::live_chapter_count_for_source_manga(&pool, "target", "nobody")
.await
.unwrap();
assert_eq!(n, 0);
}
#[sqlx::test(migrations = "./migrations")]
async fn live_chapter_count_only_counts_live_sources(pool: PgPool) {
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let m = sample_manga("foo", "Foo Manga", "hash-1");
let up = crawler::upsert_manga_from_source(&pool, "target", "https://x.example/foo", &m)
.await
.unwrap();
let chapters = vec![
SourceChapterRef {
source_chapter_key: "1".into(),
number: 1,
title: Some("Ch.1".into()),
url: "https://x.example/foo/1".into(),
},
SourceChapterRef {
source_chapter_key: "2".into(),
number: 2,
title: Some("Ch.2".into()),
url: "https://x.example/foo/2".into(),
},
];
crawler::sync_manga_chapters(&pool, "target", up.manga_id, &chapters)
.await
.unwrap();
assert_eq!(
crawler::live_chapter_count_for_source_manga(&pool, "target", "foo")
.await
.unwrap(),
2
);
// Soft-drop one source row — count drops by one, the row stays.
sqlx::query(
"UPDATE chapter_sources SET dropped_at = NOW() WHERE source_chapter_key = '2'",
)
.execute(&pool)
.await
.unwrap();
assert_eq!(
crawler::live_chapter_count_for_source_manga(&pool, "target", "foo")
.await
.unwrap(),
1
);
}
/// Real-world sources publish multiple chapters at the same number
/// (different uploaders, translator notes, re-releases). After the
/// (manga_id, number) UNIQUE drop in 0013, each `SourceChapterRef`
@@ -309,59 +370,223 @@ async fn sync_chapters_keeps_duplicate_numbered_chapters_as_separate_rows(pool:
}
#[sqlx::test(migrations = "./migrations")]
async fn mark_dropped_mangas_only_drops_unseen(pool: PgPool) {
async fn sync_chapters_isolates_colliding_keys_across_mangas(pool: PgPool) {
// Two mangas, both with a chapter whose source_chapter_key is
// "chapter-1". Pre-migration-0017 the PK enforced (source_id,
// source_chapter_key) globally and the lookup didn't filter by
// manga_id, so the second manga's sync would adopt the first manga's
// chapter_id (silent attribution corruption). After 0017 each manga
// owns its own row.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
// Seed two mangas before "now" so a later run_started_at sees them as stale.
let _ = crawler::upsert_manga_from_source(
&pool,
"target",
"https://x.example/foo",
&sample_manga("foo", "Foo", "hf"),
)
.await
.unwrap();
let _ = crawler::upsert_manga_from_source(
&pool,
"target",
"https://x.example/bar",
&sample_manga("bar", "Bar", "hb"),
)
.await
.unwrap();
// Now mark a new "run" beginning. Re-upsert only `foo` — `bar`
// should be the one flagged dropped.
let run_started = chrono::Utc::now();
// Sleep briefly so the second upsert's NOW() > run_started_at.
tokio::time::sleep(std::time::Duration::from_millis(20)).await;
let _ = crawler::upsert_manga_from_source(
&pool,
"target",
"https://x.example/foo",
&sample_manga("foo", "Foo", "hf"),
)
.await
.unwrap();
let n = crawler::mark_dropped_mangas(&pool, "target", run_started)
let m1 = sample_manga("foo", "Manga Foo", "hash-foo");
let m2 = sample_manga("bar", "Manga Bar", "hash-bar");
let up1 = crawler::upsert_manga_from_source(&pool, "target", "https://x.example/foo", &m1)
.await
.unwrap();
assert_eq!(n, 1, "only bar should have been dropped");
let up2 = crawler::upsert_manga_from_source(&pool, "target", "https://x.example/bar", &m2)
.await
.unwrap();
assert_ne!(up1.manga_id, up2.manga_id);
let foo_dropped: (Option<chrono::DateTime<chrono::Utc>>,) =
sqlx::query_as("SELECT dropped_at FROM manga_sources WHERE source_manga_key = 'foo'")
.fetch_one(&pool)
.await
.unwrap();
assert!(foo_dropped.0.is_none(), "foo seen this run, must not be dropped");
let bar_dropped: (Option<chrono::DateTime<chrono::Utc>>,) =
sqlx::query_as("SELECT dropped_at FROM manga_sources WHERE source_manga_key = 'bar'")
.fetch_one(&pool)
.await
.unwrap();
assert!(bar_dropped.0.is_some());
let shared = vec![SourceChapterRef {
source_chapter_key: "chapter-1".into(),
number: 1,
title: Some("Ch.1".into()),
url: "https://x.example/foo/chapter-1/".into(),
}];
let diff1 = crawler::sync_manga_chapters(&pool, "target", up1.manga_id, &shared)
.await
.unwrap();
assert_eq!(diff1.new, 1, "manga foo: chapter inserted fresh");
// Manga bar now syncs *the same key*. Under the old schema this would
// either fail on PK conflict or attribute the chapter to foo. Under
// the new schema bar gets its own chapter row.
let bar_chapters = vec![SourceChapterRef {
source_chapter_key: "chapter-1".into(),
number: 1,
title: Some("Ch.1 (bar)".into()),
url: "https://x.example/bar/chapter-1/".into(),
}];
let diff2 = crawler::sync_manga_chapters(&pool, "target", up2.manga_id, &bar_chapters)
.await
.unwrap();
assert_eq!(
diff2.new, 1,
"manga bar: same key resolved per-manga to a fresh row"
);
let foo_count: (i64,) = sqlx::query_as(
"SELECT COUNT(*) FROM chapters WHERE manga_id = $1",
)
.bind(up1.manga_id)
.fetch_one(&pool)
.await
.unwrap();
let bar_count: (i64,) = sqlx::query_as(
"SELECT COUNT(*) FROM chapters WHERE manga_id = $1",
)
.bind(up2.manga_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(foo_count.0, 1);
assert_eq!(bar_count.0, 1);
let bar_title: (Option<String>,) = sqlx::query_as(
"SELECT title FROM chapters WHERE manga_id = $1 AND number = 1",
)
.bind(up2.manga_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
bar_title.0.as_deref(),
Some("Ch.1 (bar)"),
"bar's chapter has bar's title, not foo's"
);
// A subsequent re-sync of foo with the same key correctly refreshes
// foo's row, not bar's.
let foo_resync = vec![SourceChapterRef {
source_chapter_key: "chapter-1".into(),
number: 1,
title: Some("Ch.1 (foo updated)".into()),
url: "https://x.example/foo/chapter-1/".into(),
}];
let diff_refresh = crawler::sync_manga_chapters(&pool, "target", up1.manga_id, &foo_resync)
.await
.unwrap();
assert_eq!(diff_refresh.refreshed, 1);
assert_eq!(diff_refresh.new, 0);
let foo_title: (Option<String>,) = sqlx::query_as(
"SELECT title FROM chapters WHERE manga_id = $1 AND number = 1",
)
.bind(up1.manga_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(foo_title.0.as_deref(), Some("Ch.1 (foo updated)"));
let bar_title_after: (Option<String>,) = sqlx::query_as(
"SELECT title FROM chapters WHERE manga_id = $1 AND number = 1",
)
.bind(up2.manga_id)
.fetch_one(&pool)
.await
.unwrap();
assert_eq!(
bar_title_after.0.as_deref(),
Some("Ch.1 (bar)"),
"bar's row is untouched by foo's refresh"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn sync_chapters_serializes_concurrent_calls_for_same_manga(pool: PgPool) {
// Without the per-manga advisory lock, two concurrent calls would
// both read `seen_keys`, both run the drop UPDATE filtered on `NOT
// (key = ANY $3)`, and the later commit could soft-drop a chapter
// the earlier had just inserted. The lock makes the calls strictly
// sequential per-manga: whichever runs second sees the first one's
// committed chapters and treats their absence as a "dropped" signal
// only if the second list legitimately omits them.
//
// Concretely: pre-state [A]. Call X syncs [A, B]; call Y syncs
// [A, B, C]. Whatever the schedule, the final state must include
// *all three* chapters because neither call legitimately omits the
// other's contribution — both lists are supersets of each other's
// pre-existing rows.
crawler::ensure_source(&pool, "target", "T", "https://x.example")
.await
.unwrap();
let m = sample_manga("foo", "Foo Manga", "hash-1");
let up = crawler::upsert_manga_from_source(&pool, "target", "https://x.example/foo", &m)
.await
.unwrap();
let manga_id = up.manga_id;
// Pre-state: [A].
let pre = vec![SourceChapterRef {
source_chapter_key: "A".into(),
number: 1,
title: Some("Ch.A".into()),
url: "https://x.example/foo/A".into(),
}];
crawler::sync_manga_chapters(&pool, "target", manga_id, &pre)
.await
.unwrap();
// Two concurrent calls. Call X adds B; call Y adds B + C. Both keep
// A. Their drop branches would otherwise race against each other.
let list_x = vec![
SourceChapterRef {
source_chapter_key: "A".into(),
number: 1,
title: Some("Ch.A".into()),
url: "https://x.example/foo/A".into(),
},
SourceChapterRef {
source_chapter_key: "B".into(),
number: 2,
title: Some("Ch.B".into()),
url: "https://x.example/foo/B".into(),
},
];
let list_y = vec![
SourceChapterRef {
source_chapter_key: "A".into(),
number: 1,
title: Some("Ch.A".into()),
url: "https://x.example/foo/A".into(),
},
SourceChapterRef {
source_chapter_key: "B".into(),
number: 2,
title: Some("Ch.B".into()),
url: "https://x.example/foo/B".into(),
},
SourceChapterRef {
source_chapter_key: "C".into(),
number: 3,
title: Some("Ch.C".into()),
url: "https://x.example/foo/C".into(),
},
];
let pool_x = pool.clone();
let pool_y = pool.clone();
let (rx, ry) = tokio::join!(
tokio::spawn(async move {
crawler::sync_manga_chapters(&pool_x, "target", manga_id, &list_x).await
}),
tokio::spawn(async move {
crawler::sync_manga_chapters(&pool_y, "target", manga_id, &list_y).await
}),
);
rx.unwrap().expect("call X");
ry.unwrap().expect("call Y");
// All three keys must survive with dropped_at NULL — the lock
// ensures the later call sees the earlier one's INSERTs and the
// drop UPDATE finds nothing to drop.
let alive: Vec<String> = sqlx::query_scalar(
"SELECT cs.source_chapter_key \
FROM chapter_sources cs \
JOIN chapters ch ON ch.id = cs.chapter_id \
WHERE ch.manga_id = $1 AND cs.dropped_at IS NULL \
ORDER BY cs.source_chapter_key",
)
.bind(manga_id)
.fetch_all(&pool)
.await
.unwrap();
assert_eq!(
alive,
vec!["A".to_string(), "B".to_string(), "C".to_string()],
"all chapters survive concurrent syncs that both contain them"
);
}
#[sqlx::test(migrations = "./migrations")]

View File

@@ -0,0 +1,162 @@
//! Integration tests for `repo::chapter` — focused on
//! `dispatch_target`, the resolver the daemon's chapter dispatcher
//! uses to look up the URL it needs to hand to
//! `content::sync_chapter_content`.
//!
//! The query must:
//! 1. Skip `chapter_sources` rows where `dropped_at IS NOT NULL` —
//! otherwise a soft-dropped source URL is dispatched as if live and
//! burns the chapter's retry budget against guaranteed transients.
//! 2. Order the remaining rows by `last_seen_at DESC` so the freshest
//! surviving source is the one we'll fetch from.
//!
//! The fix lives in `backend/src/repo/chapter.rs:dispatch_target`. The
//! enqueue queries at `pipeline.rs:381` and `:435` already filter on
//! `cs.dropped_at IS NULL`; this brings the resolver into line.
use mangalord::crawler::source::{SourceChapterRef, SourceManga};
use mangalord::repo::{
chapter::dispatch_target,
crawler::{ensure_source, sync_manga_chapters, upsert_manga_from_source},
};
use sqlx::PgPool;
use uuid::Uuid;
fn sample_manga(key: &str, title: &str, hash: &str) -> SourceManga {
SourceManga {
source_manga_key: key.to_string(),
title: title.to_string(),
alternative_titles: vec![],
authors: vec![],
genres: vec![],
tags: vec![],
status: None,
summary: None,
cover_url: None,
chapters: vec![],
metadata_hash: hash.to_string(),
}
}
/// Seed a manga with one chapter, plus a second `chapter_sources` row
/// pointing at the same chapter with a *newer* `last_seen_at` so the
/// `ORDER BY cs.last_seen_at DESC` branch of the fixed query can
/// distinguish "freshest live source" from "any live source."
async fn seed_chapter_with_two_live_sources(pool: &PgPool) -> (Uuid, String, String) {
// Two distinct sources both pointing at the same chapter is the
// realistic shape of the multi-source state — each source row is
// keyed (source_id, chapter_id) after migration 0017.
ensure_source(pool, "target", "T", "https://x.example")
.await
.unwrap();
ensure_source(pool, "mirror", "Mirror", "https://m.example")
.await
.unwrap();
let m = sample_manga("foo", "Foo Manga", "hash-1");
let up = upsert_manga_from_source(pool, "target", "https://x.example/foo", &m)
.await
.unwrap();
let initial = vec![SourceChapterRef {
source_chapter_key: "1".into(),
number: 1,
title: Some("Ch.1".into()),
url: "https://x.example/foo/1/old".into(),
}];
sync_manga_chapters(pool, "target", up.manga_id, &initial)
.await
.unwrap();
let (chapter_id,): (Uuid,) = sqlx::query_as(
"SELECT c.id FROM chapters c \
JOIN chapter_sources cs ON cs.chapter_id = c.id \
WHERE cs.source_chapter_key = '1' AND cs.source_id = 'target'",
)
.fetch_one(pool)
.await
.unwrap();
let old_url = "https://x.example/foo/1/old".to_string();
let new_url = "https://m.example/foo/1/mirror".to_string();
// Backdate the existing (old/target) source row and add a fresher
// row from the mirror source. The fix uses `last_seen_at DESC` to
// break the tie deterministically.
sqlx::query(
"UPDATE chapter_sources \
SET last_seen_at = NOW() - INTERVAL '2 days' \
WHERE chapter_id = $1 AND source_id = 'target'",
)
.bind(chapter_id)
.execute(pool)
.await
.unwrap();
sqlx::query(
"INSERT INTO chapter_sources \
(source_id, chapter_id, source_chapter_key, source_url, last_seen_at) \
VALUES ('mirror', $1, '1', $2, NOW())",
)
.bind(chapter_id)
.bind(&new_url)
.execute(pool)
.await
.unwrap();
(chapter_id, old_url, new_url)
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_target_prefers_most_recent_live_source(pool: PgPool) {
let (chapter_id, _old_url, new_url) =
seed_chapter_with_two_live_sources(&pool).await;
let row = dispatch_target(&pool, chapter_id).await.unwrap();
let (_manga_id, source_url) =
row.expect("two live sources should yield a dispatch target");
assert_eq!(
source_url, new_url,
"ORDER BY last_seen_at DESC LIMIT 1 must return the freshest source"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_target_skips_dropped_sources(pool: PgPool) {
let (chapter_id, _old_url, new_url) =
seed_chapter_with_two_live_sources(&pool).await;
// Soft-drop the fresher row. The dispatcher must now return the
// *older* still-live row instead of the dropped one.
sqlx::query(
"UPDATE chapter_sources SET dropped_at = NOW() WHERE source_url = $1",
)
.bind(&new_url)
.execute(&pool)
.await
.unwrap();
let row = dispatch_target(&pool, chapter_id).await.unwrap();
let (_manga_id, source_url) =
row.expect("a single live source should still yield a dispatch target");
assert!(
source_url != new_url,
"dispatch_target must not return a dropped source"
);
}
#[sqlx::test(migrations = "./migrations")]
async fn dispatch_target_returns_none_when_only_dropped_sources_remain(
pool: PgPool,
) {
let (chapter_id, _old_url, _new_url) =
seed_chapter_with_two_live_sources(&pool).await;
sqlx::query("UPDATE chapter_sources SET dropped_at = NOW() WHERE chapter_id = $1")
.bind(chapter_id)
.execute(&pool)
.await
.unwrap();
let row = dispatch_target(&pool, chapter_id).await.unwrap();
assert!(
row.is_none(),
"every source is dropped — dispatch_target must return None"
);
}

View File

@@ -1,6 +1,6 @@
{
"name": "mangalord-frontend",
"version": "0.35.0",
"version": "0.43.0",
"private": true,
"type": "module",
"scripts": {

View File

@@ -0,0 +1,245 @@
import {
describe,
it,
expect,
vi,
beforeEach,
afterEach,
type MockInstance
} from 'vitest';
import {
listAdminUsers,
deleteAdminUser,
setUserAdmin,
createAdminUser,
listAdminMangas,
listAdminChapters,
getSystemStats
} from './admin';
function ok(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { 'content-type': 'application/json' }
});
}
function noContent(): Response {
return new Response(null, { status: 204 });
}
function envelope(status: number, code: string, message: string): Response {
return new Response(JSON.stringify({ error: { code, message } }), {
status,
headers: { 'content-type': 'application/json' }
});
}
const userFixture = {
id: 'u-1',
username: 'alice',
created_at: '2026-01-01T00:00:00Z',
is_admin: false
};
const mangaFixture = {
id: 'm-1',
title: 'Test',
status: 'ongoing',
cover_image_path: null,
created_at: '2026-01-01T00:00:00Z',
updated_at: '2026-01-01T00:00:00Z',
sync_state: 'synced' as const,
chapter_count: 3,
latest_seen_at: '2026-01-02T00:00:00Z'
};
const systemFixture = {
disk: {
total_bytes: 1_000_000,
used_bytes: 500_000,
free_bytes: 500_000,
percent_used: 50.0
},
memory: { total_bytes: 8_000_000, used_bytes: 4_000_000, percent_used: 50.0 },
cpu: { percent_used: 12.3 },
alerts: []
};
describe('admin api client', () => {
let fetchSpy: MockInstance<typeof globalThis.fetch>;
beforeEach(() => {
fetchSpy = vi.spyOn(globalThis, 'fetch');
});
afterEach(() => {
vi.restoreAllMocks();
});
// ---- users ----
it('listAdminUsers GETs /v1/admin/users and parses the paged envelope', async () => {
fetchSpy.mockResolvedValueOnce(
ok({ items: [userFixture], page: { limit: 50, offset: 0, total: 1 } })
);
const page = await listAdminUsers({ limit: 50 });
expect(page.items).toHaveLength(1);
expect(page.items[0]).toEqual(userFixture);
expect(page.page.total).toBe(1);
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/admin\/users\?limit=50$/);
});
it('listAdminUsers forwards search + offset query params', async () => {
fetchSpy.mockResolvedValueOnce(
ok({ items: [], page: { limit: 50, offset: 10, total: 0 } })
);
await listAdminUsers({ search: 'al', offset: 10 });
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toContain('search=al');
expect(url).toContain('offset=10');
});
it('listAdminUsers surfaces 403 forbidden via ApiError.code', async () => {
fetchSpy.mockResolvedValueOnce(envelope(403, 'forbidden', 'forbidden'));
await expect(listAdminUsers()).rejects.toMatchObject({
status: 403,
code: 'forbidden'
});
});
it('deleteAdminUser DELETEs to /v1/admin/users/{id} and handles 204', async () => {
fetchSpy.mockResolvedValueOnce(noContent());
await expect(deleteAdminUser('u-1')).resolves.toBeUndefined();
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/admin\/users\/u-1$/);
const init = fetchSpy.mock.calls[0][1] as RequestInit;
expect(init.method).toBe('DELETE');
});
it('deleteAdminUser surfaces 409 conflict (self-delete / last-admin)', async () => {
fetchSpy.mockResolvedValueOnce(
envelope(409, 'conflict', 'cannot delete yourself; ask another admin')
);
await expect(deleteAdminUser('u-1')).rejects.toMatchObject({
status: 409,
code: 'conflict'
});
});
it('createAdminUser POSTs to /v1/admin/users with body and returns the created user', async () => {
const created = { ...userFixture, username: 'invited01' };
fetchSpy.mockResolvedValueOnce(ok(created, 201));
const got = await createAdminUser({
username: 'invited01',
password: 'freshpass1234'
});
expect(got).toEqual(created);
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/admin\/users$/);
const init = fetchSpy.mock.calls[0][1] as RequestInit;
expect(init.method).toBe('POST');
expect(JSON.parse(init.body as string)).toEqual({
username: 'invited01',
password: 'freshpass1234'
});
});
it('createAdminUser forwards is_admin when provided', async () => {
const created = { ...userFixture, username: 'coadmin', is_admin: true };
fetchSpy.mockResolvedValueOnce(ok(created, 201));
await createAdminUser({
username: 'coadmin',
password: 'freshpass1234',
is_admin: true
});
const init = fetchSpy.mock.calls[0][1] as RequestInit;
expect(JSON.parse(init.body as string)).toEqual({
username: 'coadmin',
password: 'freshpass1234',
is_admin: true
});
});
it('createAdminUser surfaces 409 conflict on duplicate username', async () => {
fetchSpy.mockResolvedValueOnce(
envelope(409, 'conflict', 'username is already taken')
);
await expect(
createAdminUser({ username: 'taken', password: 'freshpass1234' })
).rejects.toMatchObject({ status: 409, code: 'conflict' });
});
it('setUserAdmin PATCHes is_admin and returns the updated user', async () => {
const updated = { ...userFixture, is_admin: true };
fetchSpy.mockResolvedValueOnce(ok(updated));
const got = await setUserAdmin('u-1', true);
expect(got).toEqual(updated);
const init = fetchSpy.mock.calls[0][1] as RequestInit;
expect(init.method).toBe('PATCH');
expect(JSON.parse(init.body as string)).toEqual({ is_admin: true });
});
// ---- mangas + chapters ----
it('listAdminMangas GETs /v1/admin/mangas and forwards sync_state filter', async () => {
fetchSpy.mockResolvedValueOnce(
ok({ items: [mangaFixture], page: { limit: 100, offset: 0, total: 1 } })
);
const page = await listAdminMangas({ syncState: 'in_progress', limit: 100 });
expect(page.items[0].sync_state).toBe('synced');
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toContain('sync_state=in_progress');
expect(url).toContain('limit=100');
});
it('listAdminChapters GETs the nested chapter route and parses the paged envelope', async () => {
const chapter = {
id: 'c-1',
manga_id: 'm-1',
number: 1,
title: null,
page_count: 12,
created_at: '2026-01-01T00:00:00Z',
sync_state: 'synced' as const,
latest_seen_at: null
};
fetchSpy.mockResolvedValueOnce(
ok({ items: [chapter], page: { limit: 200, offset: 0, total: 1 } })
);
const resp = await listAdminChapters('m-1');
expect(resp.items).toEqual([chapter]);
expect(resp.page.total).toBe(1);
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/admin\/mangas\/m-1\/chapters$/);
});
it('listAdminChapters forwards limit + offset query params', async () => {
fetchSpy.mockResolvedValueOnce(
ok({ items: [], page: { limit: 50, offset: 100, total: 0 } })
);
await listAdminChapters('m-1', { limit: 50, offset: 100 });
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toContain('limit=50');
expect(url).toContain('offset=100');
});
// ---- system ----
it('getSystemStats GETs /v1/admin/system and parses the four-key envelope', async () => {
fetchSpy.mockResolvedValueOnce(ok(systemFixture));
const s = await getSystemStats();
expect(s.disk?.percent_used).toBe(50);
expect(s.memory.percent_used).toBe(50);
expect(s.cpu.percent_used).toBe(12.3);
expect(s.alerts).toEqual([]);
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/admin\/system$/);
});
it('getSystemStats keeps disk null when backend reports a non-local store', async () => {
fetchSpy.mockResolvedValueOnce(ok({ ...systemFixture, disk: null }));
const s = await getSystemStats();
expect(s.disk).toBeNull();
});
});

View File

@@ -0,0 +1,178 @@
// Admin-only API client. Every endpoint here is guarded by
// RequireAdmin on the backend (session cookie only — bearer tokens
// won't reach these routes). 403s thrown here propagate up to the
// /admin layout, which renders the framework error page.
import { request, type Page } from './client';
import type { User } from './auth';
// ---- users -----------------------------------------------------------------
export type AdminUsersPage = {
items: User[];
page: Page;
};
export type ListAdminUsersOptions = {
search?: string;
limit?: number;
offset?: number;
};
export async function listAdminUsers(
opts: ListAdminUsersOptions = {}
): Promise<AdminUsersPage> {
const params = new URLSearchParams();
if (opts.search) params.set('search', opts.search);
if (opts.limit != null) params.set('limit', String(opts.limit));
if (opts.offset != null) params.set('offset', String(opts.offset));
const qs = params.toString();
return request<AdminUsersPage>(`/v1/admin/users${qs ? `?${qs}` : ''}`);
}
export async function deleteAdminUser(id: string): Promise<void> {
await request<void>(`/v1/admin/users/${encodeURIComponent(id)}`, {
method: 'DELETE'
});
}
export async function setUserAdmin(id: string, isAdmin: boolean): Promise<User> {
return request<User>(`/v1/admin/users/${encodeURIComponent(id)}`, {
method: 'PATCH',
headers: { 'content-type': 'application/json' },
body: JSON.stringify({ is_admin: isAdmin })
});
}
export type CreateAdminUserInput = {
username: string;
password: string;
is_admin?: boolean;
};
/** POST /v1/admin/users — admin-initiated account creation. Works
* regardless of the ALLOW_SELF_REGISTER toggle, since the entire
* point is for an admin to enroll someone when self-register is off. */
export async function createAdminUser(input: CreateAdminUserInput): Promise<User> {
return request<User>('/v1/admin/users', {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: JSON.stringify(input)
});
}
// ---- mangas / chapters with sync state -------------------------------------
export type MangaSyncState = 'in_progress' | 'dropped' | 'synced';
export type AdminMangaRow = {
id: string;
title: string;
status: string;
cover_image_path: string | null;
created_at: string;
updated_at: string;
sync_state: MangaSyncState;
chapter_count: number;
latest_seen_at: string | null;
};
export type AdminMangasPage = {
items: AdminMangaRow[];
page: Page;
};
export type ListAdminMangasOptions = {
search?: string;
syncState?: MangaSyncState;
limit?: number;
offset?: number;
};
export async function listAdminMangas(
opts: ListAdminMangasOptions = {}
): Promise<AdminMangasPage> {
const params = new URLSearchParams();
if (opts.search) params.set('search', opts.search);
if (opts.syncState) params.set('sync_state', opts.syncState);
if (opts.limit != null) params.set('limit', String(opts.limit));
if (opts.offset != null) params.set('offset', String(opts.offset));
const qs = params.toString();
return request<AdminMangasPage>(`/v1/admin/mangas${qs ? `?${qs}` : ''}`);
}
export type ChapterSyncState =
| 'downloading'
| 'dropped'
| 'failed'
| 'not_downloaded'
| 'synced';
export type AdminChapterRow = {
id: string;
manga_id: string;
number: number;
title: string | null;
page_count: number;
created_at: string;
sync_state: ChapterSyncState;
latest_seen_at: string | null;
};
export type AdminChaptersPage = {
items: AdminChapterRow[];
page: Page;
};
export type ListAdminChaptersOptions = {
limit?: number;
offset?: number;
};
export async function listAdminChapters(
mangaId: string,
opts: ListAdminChaptersOptions = {}
): Promise<AdminChaptersPage> {
const params = new URLSearchParams();
if (opts.limit != null) params.set('limit', String(opts.limit));
if (opts.offset != null) params.set('offset', String(opts.offset));
const qs = params.toString();
return request<AdminChaptersPage>(
`/v1/admin/mangas/${encodeURIComponent(mangaId)}/chapters${qs ? `?${qs}` : ''}`
);
}
// ---- system ----------------------------------------------------------------
export type DiskStats = {
total_bytes: number;
used_bytes: number;
free_bytes: number;
percent_used: number;
};
export type MemoryStats = {
total_bytes: number;
used_bytes: number;
percent_used: number;
};
export type CpuStats = {
percent_used: number;
};
export type Alert = {
level: 'warning';
message: string;
};
export type SystemStats = {
disk: DiskStats | null;
memory: MemoryStats;
cpu: CpuStats;
alerts: Alert[];
};
export async function getSystemStats(): Promise<SystemStats> {
return request<SystemStats>('/v1/admin/system');
}

View File

@@ -14,7 +14,8 @@ import {
me,
changePassword,
createToken,
deleteToken
deleteToken,
getAuthConfig
} from './auth';
function ok(body: unknown, status = 200): Response {
@@ -169,6 +170,17 @@ describe('auth api client', () => {
expect(url).toMatch(/\/v1\/auth\/tokens$/);
});
it('getAuthConfig GETs /v1/auth/config and parses the flag', async () => {
fetchSpy.mockResolvedValueOnce(ok({ self_register_enabled: false }));
const cfg = await getAuthConfig();
expect(cfg.self_register_enabled).toBe(false);
const url = fetchSpy.mock.calls[0][0] as string;
expect(url).toMatch(/\/v1\/auth\/config$/);
const init = fetchSpy.mock.calls[0][1] as RequestInit;
// Public endpoint; no method override means default GET.
expect(init?.method ?? 'GET').toBe('GET');
});
it('deleteToken DELETEs to /v1/auth/tokens/{id} and handles 204', async () => {
fetchSpy.mockResolvedValueOnce(noContent());
await expect(deleteToken('t1')).resolves.toBeUndefined();

View File

@@ -4,6 +4,7 @@ export type User = {
id: string;
username: string;
created_at: string;
is_admin: boolean;
};
export type Credentials = {
@@ -99,3 +100,15 @@ export async function createToken(name: string): Promise<CreatedToken> {
export async function deleteToken(id: string): Promise<void> {
await request<void>(`/v1/auth/tokens/${encodeURIComponent(id)}`, { method: 'DELETE' });
}
export type AuthConfig = {
/** When false, /v1/auth/register returns 403 and the UI should
* hide its register affordance. Admins can still mint accounts
* via POST /v1/admin/users. */
self_register_enabled: boolean;
};
/** Public — no auth, no cookie required. */
export async function getAuthConfig(): Promise<AuthConfig> {
return request<AuthConfig>('/v1/auth/config');
}

View File

@@ -0,0 +1,37 @@
// Anonymous-relevant auth policy (currently just whether self-
// registration is enabled). Loaded once per browser session on root-
// layout mount, then read reactively from `authConfig.self_register_enabled`.
//
// Defaults to `self_register_enabled = true` while loading so the
// register link doesn't flash off-and-on for the default-open case.
// If the fetch fails (network blip, backend restart), the stale value
// is kept — there's no per-request retry. A new tab will retry on its
// own mount.
//
// Same browser-only contract as `session.svelte.ts` — see that file's
// SSR comment.
import { browser } from '$app/environment';
import { getAuthConfig } from './api/auth';
class AuthConfigStore {
self_register_enabled = $state(true);
loaded = $state(false);
private loading = false;
async load(): Promise<void> {
if (this.loaded || this.loading || !browser) return;
this.loading = true;
try {
const cfg = await getAuthConfig();
this.self_register_enabled = cfg.self_register_enabled;
this.loaded = true;
} catch {
// Keep optimistic default; next page mount will retry.
} finally {
this.loading = false;
}
}
}
export const authConfig = new AuthConfigStore();

View File

@@ -2,6 +2,7 @@
import { onMount, onDestroy } from 'svelte';
import { goto } from '$app/navigation';
import { logout } from '$lib/api/auth';
import { authConfig } from '$lib/auth-config.svelte';
import { preferences } from '$lib/preferences.svelte';
import { session } from '$lib/session.svelte';
import { theme } from '$lib/theme.svelte';
@@ -10,6 +11,7 @@
import Bookmark from '@lucide/svelte/icons/bookmark';
import FolderOpen from '@lucide/svelte/icons/folder-open';
import LogOut from '@lucide/svelte/icons/log-out';
import Shield from '@lucide/svelte/icons/shield';
import '$lib/styles/tokens.css';
let { children } = $props();
@@ -20,6 +22,7 @@
theme.init();
preferences.init();
if (!session.loaded) session.refresh();
if (!authConfig.loaded) authConfig.load();
// Publish the header's measured height as a CSS custom
// property so sticky descendants (e.g. the reader nav) can
@@ -86,6 +89,12 @@
<FolderOpen size={18} aria-hidden="true" />
<span>Collections</span>
</a>
{#if session.user?.is_admin}
<a class="nav-link" href="/admin" data-testid="nav-admin">
<Shield size={18} aria-hidden="true" />
<span>Admin</span>
</a>
{/if}
</nav>
<div class="session" data-testid="session-area">
{#if !session.loaded}
@@ -108,7 +117,9 @@
</button>
{:else}
<a class="text-link" href="/login" data-testid="nav-login">Login</a>
<a class="text-link" href="/register" data-testid="nav-register">Register</a>
{#if authConfig.self_register_enabled}
<a class="text-link" href="/register" data-testid="nav-register">Register</a>
{/if}
{/if}
</div>
</header>

View File

@@ -0,0 +1,75 @@
<script lang="ts">
import { page } from '$app/stores';
let { children } = $props();
const tabs = [
{ href: '/admin', label: 'Overview' },
{ href: '/admin/users', label: 'Users' },
{ href: '/admin/mangas', label: 'Mangas' },
{ href: '/admin/system', label: 'System' }
];
</script>
<div class="admin-frame">
<aside aria-label="admin">
<h2>Admin</h2>
<nav>
{#each tabs as t (t.href)}
<a
href={t.href}
class:active={$page.url.pathname === t.href}
data-testid={`admin-nav-${t.label.toLowerCase()}`}
>
{t.label}
</a>
{/each}
</nav>
</aside>
<section>
{@render children()}
</section>
</div>
<style>
.admin-frame {
display: grid;
grid-template-columns: 12rem 1fr;
gap: var(--space-4);
align-items: start;
}
@media (max-width: 600px) {
.admin-frame {
grid-template-columns: 1fr;
}
}
aside {
position: sticky;
top: calc(var(--app-header-h) + var(--space-4));
}
aside h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-base);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
nav {
display: flex;
flex-direction: column;
gap: var(--space-1);
}
nav a {
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
color: var(--text);
font-size: var(--font-sm);
}
nav a:hover {
background: var(--surface-elevated);
text-decoration: none;
}
nav a.active {
background: var(--surface-elevated);
font-weight: var(--weight-semibold);
}
</style>

View File

@@ -0,0 +1,31 @@
// /admin gate. The backend's RequireAdmin extractor is the actual
// security boundary — this load function just calls a tiny admin
// endpoint and translates the response into either a redirect (no
// session) or SvelteKit's framework error page (403 forbidden).
// The session.user?.is_admin check elsewhere is UX only.
//
// `ssr=false` because the session store is browser-only (see
// $lib/session.svelte.ts) — server-side load can't read the cookie
// anyway in this app's deployment shape.
import { error, redirect } from '@sveltejs/kit';
import { ApiError } from '$lib/api/client';
import { getSystemStats } from '$lib/api/admin';
import type { LayoutLoad } from './$types';
export const ssr = false;
export const load: LayoutLoad = async () => {
try {
const stats = await getSystemStats();
return { stats };
} catch (e) {
if (e instanceof ApiError && e.status === 401) {
throw redirect(302, '/login');
}
if (e instanceof ApiError && e.status === 403) {
throw error(403, 'admin access required');
}
throw e;
}
};

View File

@@ -0,0 +1,104 @@
<script lang="ts">
import type { LayoutData } from './$types';
let { data }: { data: LayoutData } = $props();
const stats = $derived(data.stats);
function fmtPercent(n: number): string {
return `${n.toFixed(1)}%`;
}
function fmtBytes(n: number): string {
const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB'];
let i = 0;
let v = n;
while (v >= 1024 && i < units.length - 1) {
v /= 1024;
i++;
}
return `${v.toFixed(1)} ${units[i]}`;
}
</script>
<h1>Overview</h1>
{#if stats.alerts.length > 0}
<section class="alerts" data-testid="admin-alerts">
{#each stats.alerts as a (a.message)}
<div class="alert" data-level={a.level}>{a.message}</div>
{/each}
</section>
{/if}
<section class="cards">
{#if stats.disk}
<article class="card">
<h3>Disk</h3>
<p class="metric">{fmtPercent(stats.disk.percent_used)}</p>
<p class="sub">{fmtBytes(stats.disk.used_bytes)} of {fmtBytes(stats.disk.total_bytes)} used</p>
</article>
{:else}
<article class="card muted">
<h3>Disk</h3>
<p>n/a (non-local storage)</p>
</article>
{/if}
<article class="card">
<h3>Memory</h3>
<p class="metric">{fmtPercent(stats.memory.percent_used)}</p>
<p class="sub">{fmtBytes(stats.memory.used_bytes)} of {fmtBytes(stats.memory.total_bytes)} used</p>
</article>
<article class="card">
<h3>CPU</h3>
<p class="metric">{fmtPercent(stats.cpu.percent_used)}</p>
<p class="sub">global load</p>
</article>
</section>
<style>
h1 {
margin: 0 0 var(--space-4) 0;
}
.alerts {
display: flex;
flex-direction: column;
gap: var(--space-2);
margin-bottom: var(--space-4);
}
.alert {
padding: var(--space-3);
border-radius: var(--radius-md);
background: var(--surface-elevated);
border-left: 4px solid var(--warning, #f59e0b);
}
.cards {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(14rem, 1fr));
gap: var(--space-3);
}
.card {
padding: var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
}
.card.muted {
opacity: 0.6;
}
.card h3 {
margin: 0 0 var(--space-2) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.metric {
margin: 0;
font-size: var(--font-xl, 1.5rem);
font-weight: var(--weight-semibold);
}
.sub {
margin: var(--space-1) 0 0 0;
font-size: var(--font-sm);
color: var(--text-muted);
}
</style>

View File

@@ -0,0 +1,280 @@
<script lang="ts">
import { onMount } from 'svelte';
import {
listAdminMangas,
listAdminChapters,
type AdminMangasPage,
type AdminChapterRow,
type MangaSyncState
} from '$lib/api/admin';
import { ApiError } from '$lib/api/client';
let mangasPage: AdminMangasPage | null = $state(null);
let search = $state('');
let syncFilter: MangaSyncState | '' = $state('');
let error: string | null = $state(null);
let expandedId: string | null = $state(null);
type ChaptersView = {
items: AdminChapterRow[];
total: number;
};
let chaptersByManga: Record<string, ChaptersView | 'loading'> = $state({});
async function load() {
error = null;
try {
mangasPage = await listAdminMangas({
search: search.trim() || undefined,
syncState: syncFilter || undefined,
limit: 100
});
} catch (e) {
error = e instanceof ApiError ? e.message : 'load failed';
}
}
onMount(load);
async function toggleChapters(id: string) {
if (expandedId === id) {
expandedId = null;
return;
}
expandedId = id;
if (!chaptersByManga[id]) {
chaptersByManga[id] = 'loading';
try {
const resp = await listAdminChapters(id, { limit: 500 });
chaptersByManga[id] = {
items: resp.items,
total: resp.page.total ?? resp.items.length
};
} catch {
delete chaptersByManga[id];
error = 'failed to load chapters';
}
}
}
function badgeClass(state: string): string {
return `badge badge-${state}`;
}
</script>
<h1>Mangas</h1>
<form
onsubmit={(e) => {
e.preventDefault();
load();
}}
>
<input
type="search"
placeholder="search by title"
bind:value={search}
data-testid="admin-mangas-search"
/>
<select bind:value={syncFilter} aria-label="sync state">
<option value="">all states</option>
<option value="in_progress">in progress</option>
<option value="dropped">dropped</option>
<option value="synced">synced</option>
</select>
<button type="submit">Search</button>
</form>
{#if error}
<p class="error" role="alert">{error}</p>
{/if}
{#if mangasPage}
<p class="total">{mangasPage.page.total ?? mangasPage.items.length} mangas</p>
<table>
<thead>
<tr>
<th>Title</th>
<th>Sync</th>
<th>Chapters</th>
<th>Last seen</th>
</tr>
</thead>
<tbody>
{#each mangasPage.items as m (m.id)}
<tr>
<td>
<button class="link" onclick={() => toggleChapters(m.id)}>
{expandedId === m.id ? '▼' : '▶'} {m.title}
</button>
</td>
<td><span class={badgeClass(m.sync_state)}>{m.sync_state}</span></td>
<td>{m.chapter_count}</td>
<td>
{m.latest_seen_at
? new Date(m.latest_seen_at).toLocaleDateString()
: '—'}
</td>
</tr>
{#if expandedId === m.id}
<tr class="chapter-row">
<td colspan="4">
{#if chaptersByManga[m.id] === 'loading'}
<p>Loading chapters…</p>
{:else if chaptersByManga[m.id]}
{@const view = chaptersByManga[m.id] as ChaptersView}
{#if view.items.length === 0}
<p class="muted">No chapters.</p>
{:else}
{#if view.total > view.items.length}
<p class="muted">
Showing first {view.items.length} of {view.total}
chapters (cap reached).
</p>
{/if}
<table class="inner">
<thead>
<tr>
<th>#</th>
<th>Title</th>
<th>Pages</th>
<th>Sync</th>
</tr>
</thead>
<tbody>
{#each view.items as c (c.id)}
<tr>
<td>{c.number}</td>
<td>{c.title ?? '—'}</td>
<td>{c.page_count}</td>
<td>
<span class={badgeClass(c.sync_state)}>
{c.sync_state}
</span>
</td>
</tr>
{/each}
</tbody>
</table>
{/if}
{/if}
</td>
</tr>
{/if}
{/each}
</tbody>
</table>
{:else}
<p>Loading…</p>
{/if}
<style>
h1 {
margin: 0 0 var(--space-4) 0;
}
form {
display: flex;
gap: var(--space-2);
margin-bottom: var(--space-3);
}
input[type='search'] {
flex: 1;
padding: var(--space-2) var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
color: var(--text);
}
select {
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
border: 1px solid var(--border);
background: var(--surface);
color: var(--text);
}
button {
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
border: 1px solid var(--border);
background: var(--surface);
color: var(--text);
cursor: pointer;
}
button.link {
background: none;
border: none;
padding: 0;
color: var(--text);
cursor: pointer;
font-weight: inherit;
}
button.link:hover {
color: var(--primary);
}
.total {
color: var(--text-muted);
font-size: var(--font-sm);
margin: 0 0 var(--space-2) 0;
}
table {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
}
.chapter-row td {
background: var(--surface-elevated);
}
table.inner {
margin: var(--space-2) 0;
}
.badge {
display: inline-block;
padding: 0 var(--space-2);
border-radius: var(--radius-sm, 4px);
font-size: var(--font-xs, 0.75rem);
font-weight: var(--weight-semibold);
text-transform: uppercase;
letter-spacing: 0.04em;
border: 1px solid var(--border);
background: var(--surface);
}
.badge-in_progress,
.badge-downloading {
background: #fef3c7;
color: #92400e;
border-color: #fcd34d;
}
.badge-dropped {
background: #fee2e2;
color: #991b1b;
border-color: #fca5a5;
}
.badge-failed {
background: #fee2e2;
color: #991b1b;
border-color: #fca5a5;
}
.badge-not_downloaded {
background: var(--surface-elevated);
color: var(--text-muted);
}
.badge-synced {
background: #dcfce7;
color: #166534;
border-color: #86efac;
}
.muted {
color: var(--text-muted);
}
.error {
color: var(--danger, #dc2626);
padding: var(--space-2) var(--space-3);
border: 1px solid var(--danger, #dc2626);
border-radius: var(--radius-md);
margin-bottom: var(--space-3);
}
</style>

View File

@@ -0,0 +1,203 @@
<script lang="ts">
import { onMount, onDestroy } from 'svelte';
import { getSystemStats, type SystemStats } from '$lib/api/admin';
let stats: SystemStats | null = $state(null);
let error: string | null = $state(null);
let timer: ReturnType<typeof setInterval> | null = null;
async function refresh() {
try {
stats = await getSystemStats();
error = null;
} catch (e) {
error = e instanceof Error ? e.message : 'refresh failed';
}
}
onMount(() => {
refresh();
timer = setInterval(refresh, 5000);
});
onDestroy(() => {
if (timer) clearInterval(timer);
});
function fmtBytes(n: number): string {
const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB'];
let i = 0;
let v = n;
while (v >= 1024 && i < units.length - 1) {
v /= 1024;
i++;
}
return `${v.toFixed(2)} ${units[i]}`;
}
</script>
<h1>System</h1>
{#if error}
<p class="error" role="alert">{error}</p>
{/if}
{#if stats}
{#if stats.alerts.length > 0}
<section class="alerts">
{#each stats.alerts as a (a.message)}
<div class="alert">{a.message}</div>
{/each}
</section>
{/if}
<section class="grid">
<article>
<h2>Disk (storage_dir)</h2>
{#if stats.disk}
{@render Bar({ percent: stats.disk.percent_used })}
<dl>
<dt>Total</dt>
<dd>{fmtBytes(stats.disk.total_bytes)}</dd>
<dt>Used</dt>
<dd>{fmtBytes(stats.disk.used_bytes)}</dd>
<dt>Free</dt>
<dd>{fmtBytes(stats.disk.free_bytes)}</dd>
</dl>
{:else}
<p class="muted">n/a — non-local storage backend</p>
{/if}
</article>
<article>
<h2>Memory</h2>
{@render Bar({ percent: stats.memory.percent_used })}
<dl>
<dt>Total</dt>
<dd>{fmtBytes(stats.memory.total_bytes)}</dd>
<dt>Used</dt>
<dd>{fmtBytes(stats.memory.used_bytes)}</dd>
</dl>
</article>
<article>
<h2>CPU</h2>
{@render Bar({ percent: stats.cpu.percent_used })}
</article>
</section>
<p class="hint">refreshing every 5 s</p>
{:else}
<p>Loading…</p>
{/if}
{#snippet Bar({ percent }: { percent: number })}
<div
class="bar"
role="progressbar"
aria-valuenow={percent}
aria-valuemin="0"
aria-valuemax="100"
aria-label="{percent.toFixed(1)}% used"
>
<div
class="fill"
class:high={percent >= 90}
class:mid={percent >= 70 && percent < 90}
style:width="{Math.min(100, Math.max(0, percent))}%"
></div>
<span class="label">{percent.toFixed(1)}%</span>
</div>
{/snippet}
<style>
h1 {
margin: 0 0 var(--space-4) 0;
}
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(16rem, 1fr));
gap: var(--space-3);
}
article {
padding: var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
}
article h2 {
margin: 0 0 var(--space-3) 0;
font-size: var(--font-sm);
color: var(--text-muted);
text-transform: uppercase;
letter-spacing: 0.04em;
}
.bar {
position: relative;
background: var(--surface-elevated);
border-radius: var(--radius-sm, 4px);
height: 1.5rem;
margin-bottom: var(--space-2);
overflow: hidden;
}
.fill {
height: 100%;
background: #22c55e;
transition: width 0.3s ease, background 0.3s ease;
}
.fill.mid {
background: #f59e0b;
}
.fill.high {
background: #dc2626;
}
.label {
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
font-size: var(--font-sm);
font-weight: var(--weight-semibold);
color: var(--text);
}
dl {
display: grid;
grid-template-columns: max-content 1fr;
gap: var(--space-1) var(--space-3);
margin: 0;
font-size: var(--font-sm);
}
dt {
color: var(--text-muted);
}
dd {
margin: 0;
font-family: var(--font-mono, monospace);
}
.alerts {
display: flex;
flex-direction: column;
gap: var(--space-2);
margin-bottom: var(--space-4);
}
.alert {
padding: var(--space-3);
border-radius: var(--radius-md);
background: var(--surface-elevated);
border-left: 4px solid #f59e0b;
}
.hint {
color: var(--text-muted);
font-size: var(--font-sm);
margin-top: var(--space-3);
}
.muted {
color: var(--text-muted);
}
.error {
color: var(--danger, #dc2626);
padding: var(--space-2) var(--space-3);
border: 1px solid var(--danger, #dc2626);
border-radius: var(--radius-md);
margin-bottom: var(--space-3);
}
</style>

View File

@@ -0,0 +1,328 @@
<script lang="ts">
import { onMount } from 'svelte';
import {
listAdminUsers,
deleteAdminUser,
setUserAdmin,
createAdminUser,
type AdminUsersPage
} from '$lib/api/admin';
import { ApiError } from '$lib/api/client';
import { session } from '$lib/session.svelte';
let page: AdminUsersPage | null = $state(null);
let search = $state('');
let error: string | null = $state(null);
let busyId: string | null = $state(null);
// Create-user form (collapsed by default).
let showCreate = $state(false);
let newUsername = $state('');
let newPassword = $state('');
let newIsAdmin = $state(false);
let createError: string | null = $state(null);
let creating = $state(false);
async function load() {
error = null;
try {
page = await listAdminUsers({
search: search.trim() || undefined,
limit: 100
});
} catch (e) {
error = e instanceof ApiError ? e.message : 'load failed';
}
}
onMount(load);
async function onDelete(id: string) {
if (!confirm('Delete this user? This cannot be undone.')) return;
busyId = id;
try {
await deleteAdminUser(id);
await load();
} catch (e) {
error = e instanceof ApiError ? e.message : 'delete failed';
} finally {
busyId = null;
}
}
async function onToggleAdmin(id: string, next: boolean) {
busyId = id;
try {
await setUserAdmin(id, next);
await load();
} catch (e) {
error = e instanceof ApiError ? e.message : 'update failed';
} finally {
busyId = null;
}
}
async function onCreate(e: SubmitEvent) {
e.preventDefault();
createError = null;
creating = true;
try {
await createAdminUser({
username: newUsername.trim(),
password: newPassword,
is_admin: newIsAdmin
});
// Reset form + reload list so the new row is visible.
newUsername = '';
newPassword = '';
newIsAdmin = false;
showCreate = false;
await load();
} catch (e) {
createError = e instanceof ApiError ? e.message : 'create failed';
} finally {
creating = false;
}
}
</script>
<h1>Users</h1>
<div class="toolbar">
<button
type="button"
onclick={() => (showCreate = !showCreate)}
data-testid="admin-users-toggle-create"
>
{showCreate ? 'Cancel' : 'Create user'}
</button>
</div>
{#if showCreate}
<form class="create-form" onsubmit={onCreate} data-testid="admin-users-create-form">
<label class="field">
<span>Username</span>
<input
type="text"
bind:value={newUsername}
minlength="3"
maxlength="32"
required
autocomplete="off"
data-testid="admin-users-create-username"
/>
</label>
<label class="field">
<span>Password</span>
<input
type="password"
bind:value={newPassword}
minlength="8"
required
autocomplete="new-password"
data-testid="admin-users-create-password"
/>
</label>
<label class="field-inline">
<input
type="checkbox"
bind:checked={newIsAdmin}
data-testid="admin-users-create-is-admin"
/>
<span>Make admin</span>
</label>
<button
type="submit"
class="primary"
disabled={creating}
data-testid="admin-users-create-submit"
>
{creating ? 'Creating…' : 'Create'}
</button>
{#if createError}
<p class="error" role="alert" data-testid="admin-users-create-error">
{createError}
</p>
{/if}
</form>
{/if}
<form
onsubmit={(e) => {
e.preventDefault();
load();
}}
>
<input
type="search"
placeholder="search by username"
bind:value={search}
data-testid="admin-users-search"
/>
<button type="submit">Search</button>
</form>
{#if error}
<p class="error" role="alert">{error}</p>
{/if}
{#if page}
<p class="total">{page.page.total ?? page.items.length} users</p>
<table>
<thead>
<tr>
<th>Username</th>
<th>Admin</th>
<th>Created</th>
<th class="actions">Actions</th>
</tr>
</thead>
<tbody>
{#each page.items as u (u.id)}
{@const isSelf = session.user?.id === u.id}
<tr>
<td>{u.username}{#if isSelf}<span class="self"> (you)</span>{/if}</td>
<td>
<input
type="checkbox"
checked={u.is_admin}
disabled={busyId === u.id || isSelf}
onchange={(e) => onToggleAdmin(u.id, e.currentTarget.checked)}
aria-label="admin"
/>
</td>
<td>{new Date(u.created_at).toLocaleDateString()}</td>
<td class="actions">
<button
type="button"
class="danger"
disabled={busyId === u.id || isSelf}
onclick={() => onDelete(u.id)}
>
Delete
</button>
</td>
</tr>
{/each}
</tbody>
</table>
{:else}
<p>Loading…</p>
{/if}
<style>
h1 {
margin: 0 0 var(--space-4) 0;
}
form {
display: flex;
gap: var(--space-2);
margin-bottom: var(--space-3);
}
input[type='search'] {
flex: 1;
padding: var(--space-2) var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
color: var(--text);
}
button {
padding: var(--space-2) var(--space-3);
border-radius: var(--radius-md);
border: 1px solid var(--border);
background: var(--surface);
color: var(--text);
cursor: pointer;
}
button.danger {
color: var(--danger, #dc2626);
border-color: var(--danger, #dc2626);
}
button:disabled {
opacity: 0.5;
cursor: not-allowed;
}
.total {
color: var(--text-muted);
font-size: var(--font-sm);
margin: 0 0 var(--space-2) 0;
}
table {
width: 100%;
border-collapse: collapse;
}
th,
td {
padding: var(--space-2);
text-align: left;
border-bottom: 1px solid var(--border);
}
.actions {
text-align: right;
}
.self {
color: var(--text-muted);
font-size: var(--font-sm);
}
.error {
color: var(--danger, #dc2626);
padding: var(--space-2) var(--space-3);
border: 1px solid var(--danger, #dc2626);
border-radius: var(--radius-md);
margin-bottom: var(--space-3);
}
.toolbar {
display: flex;
justify-content: flex-end;
margin-bottom: var(--space-2);
}
.create-form {
display: grid;
grid-template-columns: 1fr 1fr auto auto;
gap: var(--space-3);
align-items: end;
padding: var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface-elevated);
margin-bottom: var(--space-3);
}
@media (max-width: 600px) {
.create-form {
grid-template-columns: 1fr;
}
}
.create-form .field {
display: flex;
flex-direction: column;
gap: var(--space-1);
font-size: var(--font-sm);
}
.create-form .field input {
padding: var(--space-2) var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface);
color: var(--text);
}
.create-form .field-inline {
display: inline-flex;
align-items: center;
gap: var(--space-2);
font-size: var(--font-sm);
}
.create-form .primary {
background: var(--primary);
color: var(--primary-contrast);
border-color: var(--primary);
}
.create-form .primary:hover:not(:disabled) {
background: var(--primary-hover);
border-color: var(--primary-hover);
}
.create-form .error {
grid-column: 1 / -1;
margin: 0;
}
</style>

View File

@@ -1,6 +1,8 @@
<script lang="ts">
import { onMount } from 'svelte';
import { goto } from '$app/navigation';
import { register } from '$lib/api/auth';
import { authConfig } from '$lib/auth-config.svelte';
import { session } from '$lib/session.svelte';
let username = $state('');
@@ -8,6 +10,13 @@
let error: string | null = $state(null);
let submitting = $state(false);
// Direct navigation to /register bypasses the root layout's
// onMount — re-trigger the config load here so the disabled state
// renders correctly even when this is the first page hit.
onMount(() => {
if (!authConfig.loaded) authConfig.load();
});
async function submit(e: SubmitEvent) {
e.preventDefault();
error = null;
@@ -25,6 +34,12 @@
</script>
<h1>Register</h1>
{#if authConfig.loaded && !authConfig.self_register_enabled}
<p class="notice" role="status" data-testid="register-disabled">
Self-registration is disabled on this server. Ask an administrator to
create an account for you.
</p>
{:else}
<form onsubmit={submit} action="javascript:void(0)" data-testid="register-form">
<label class="form-field">
<span>Username</span>
@@ -56,6 +71,7 @@
<p class="form-error" role="alert" data-testid="register-error">{error}</p>
{/if}
</form>
{/if}
<p class="hint">
Already have an account? <a href="/login">Log in</a>.
</p>
@@ -90,4 +106,14 @@
color: var(--text-muted);
font-size: var(--font-sm);
}
.notice {
padding: var(--space-3);
border: 1px solid var(--border);
border-radius: var(--radius-md);
background: var(--surface-elevated);
color: var(--text);
max-width: 32rem;
margin: 0;
}
</style>