Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds GET /api/v1/admin/system returning disk (scoped to storage_dir
via statvfs), memory, CPU, and a server-side alerts array that fires
at >90% disk or memory.
Disk uses nix::sys::statvfs directly rather than sysinfo's Disks API
to avoid mountpoint-matching gymnastics for the storage_dir. A new
`Storage::local_root() -> Option<&Path>` trait method exposes the
root; the default returns None so a future S3Storage gets `disk:
null` in the response instead of fabricated numbers.
CPU is sampled inline (refresh → 250ms sleep → refresh → read) so the
endpoint adds 250ms of latency per call. No background-cache yet —
admin traffic is low-volume and the moving parts aren't worth it
until polling shows up.
Alerts are evaluated server-side so the frontend can render them
without re-implementing the thresholds.
Five fixes bundled into one release:
- preserve user-attached tags across crawler upserts
(repo::crawler::sync_tags now scopes to added_by IS NULL; orphaned
attachments from deleted users are reaped as crawler-owned)
- gate manga PATCH and cover endpoints on uploaded_by (require_can_edit
in api::mangas; non-NULL uploaded_by must match the caller)
- equalise login response time across user-existence branches
(run argon2 against a OnceLock-cached dummy hash on the no-user
branch so timing doesn't leak username existence)
- crawler download defences (SSRF allowlist of host literals
including IPv4-mapped IPv6 ranges, 32 MiB streamed size cap,
reject non-whitelisted image types, three-way chapter-probe
classifier replaces the binary #avatar_menu check)
- tighten validation and clean up dead unload path
(attach_tag + create_token enforce 64-char caps; LocalStorage
rejects NUL bytes explicitly; reader flushFinalProgress drops
the always-405 sendBeacon path)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tightens three tests whose names overstated what they checked:
- `login_succeeds_and_rotates_session` now asserts the login cookie
differs from the registration cookie, and that the registration
cookie is still valid after login (the documented contract).
- `storage::local::rejects_path_traversal` exercises three extra
rejection paths the existing implementation already handled but the
tests didn't probe: `a/./b`, the single-segment `.`, and the empty
segment `a//b`.
- `create_and_use_bot_token` asserts that `token_hash` is *absent*
from the response (`get(...).is_none()`), not just `is_null()`,
which would have accepted an explicit `"token_hash": null` payload
too.
Adds four coverage cases that the audit flagged as missing:
- `me_rejects_expired_session` — hand-craft a session row with
`expires_at = now() - 1h`, hit `/auth/me` with the matching cookie,
expect 401 + `unauthenticated`. Proves the extractor's
`expires_at > now()` filter is wired.
- `concurrent_manga_bookmarks_serialised_by_unique_index` — spawn two
POSTs in parallel for the same `(user, manga, chapter=null)`,
assert one wins (201) and one collides (409) via the partial unique
index from migration 0004.
- `bookmark_create_accepts_bearer_token` — mint a bot token and POST
/bookmarks with `Authorization: Bearer`, asserting `CurrentUser`
resolves identically to the cookie path on a write endpoint (not
just `/auth/me`).
- Three new unit tests on `app::cors_layer` covering the allowlist
(origin reflected, credentials true), a foreign origin (no
allow-origin header emitted), and the same-origin default (empty
allowlist emits no CORS headers at all).
`cors_layer` is `pub(crate)` now so the tests in `app::tests` can
reach it; the function itself is unchanged.
No version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Migration 0003_pages.sql adds a `pages` table (id, chapter_id,
page_number, storage_key, content_type) with a unique (chapter_id,
page_number). New table because chapter pages can have different MIME
types per page; reconstructing keys from a single template would
break the moment a chapter mixes png and jpg pages.
- `domain::Page` + `repo::page` (create + list_for_chapter).
- The chapter upload handler now inserts one page row per part as it
writes the bytes to storage.
- GET /api/v1/mangas/{id}/chapters/{n}/pages returns `{pages: [...]}`
with the storage_key clients need to construct image URLs. 404 if
the manga or chapter doesn't exist; reads are public.
Storage trait grows `get_stream(&str) -> StreamingFile` returning a
`Pin<Box<dyn Stream<Item = io::Result<Bytes>> + Send>>` + size. The
local backend implements via `tokio::fs::File` + `tokio_util::io::
ReaderStream` with a 64 KiB chunk size. GET /api/v1/files/*key now
streams via `axum::body::Body::from_stream` instead of buffering — the
test asserts a 200 KiB file emits >1 frame end-to-end through the
router.
Frontend:
- lib/api/client.ts gains `fileUrl(key)` so components don't
reconstruct the `/api/v1/files/...` path manually.
- lib/api/chapters.ts gains `ChapterPage` type + `getChapterPages` (the
type is named ChapterPage to avoid colliding with `Page` from
client.ts, which is the pagination envelope).
- /manga/[id]/+page.svelte: overview with cover, title, author,
description, chapter list, and a disabled bookmark control (real
bookmarking lands in feat/bookmarks). Responsive at 640 px.
- /manga/[id]/chapter/[n]/+page.svelte: paginated reader. Current page
loads eagerly; next page is preloaded in a hidden img so navigation
feels instant. Keyboard handler maps ArrowRight/j/Space → next,
ArrowLeft/k → prev, Home/End → first/last; skips when the user is
typing in an input. Focus ring on the prev/next buttons.
- SSR is disabled on both routes via `export const ssr = false` so the
client-only fetch flow doesn't need to be replicated server-side; the
routes are interactive features, not SEO surfaces.
- E2E (e2e/reader.spec.ts): overview shows the title/cover/chapter
list; reader pages through three pages via ArrowRight, j, k, and
ArrowLeft, and the preload img holds the page-2 src on initial load.
Lockstep version bump to 0.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
