MechaCat02 d6ac648ac9 feat(admin): crawler observability dashboard + reliability hardening (0.55.0)
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.

Backend:

- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
  into status / control / dead_jobs / backlog modules. SSE stream
  composes in-memory status with DB-derived queue counts, memoizes
  the counts for 1s and debounces watch pokes for 250ms (~10x QPS
  reduction per subscriber). One-shot GET /admin/crawler shares the
  same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
  (409 Conflict on overlapping click); browser restart goes through
  the coordinated_restart gate (drain + relaunch + auto-clear of the
  sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
  validation, never logged, audit row carries SHA-256 fingerprint).
  Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
  scope=all requires confirm:true; DISTINCT ON dedup keeps the
  partial unique index from rejecting requeues for chapters with
  multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
  panics; last-writer-wins on cover so concurrent dispatches don't
  clobber each other's slot. Pure functions (should_stop /
  should_mark_clean_exit / should_abort_pass) with named regression
  tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
  per-job timeout, circuit breaker on consecutive failures, BrowserManager
  coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
  LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
  Pages stream through with peak memory ~one HTTP chunk + 64-byte
  sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
  and crawler_jobs_dead_idx, both ordered by updated_at DESC to
  match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
  on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
  admin_no_store_guard (Cache-Control: no-store on admin
  responses), audit rows carry per-scope target_id.

Frontend:

- /admin/crawler page decomposed into lib/components/crawler/
  (11 components: ProgressBar, SearchBar, CrawlerHero,
  CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
  MissingCoversTable, DeadJobsTable, RestartConfirmModal,
  RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
  orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
  pageshow (BFCache); after 5 consecutive errors probes the status
  endpoint so a 401 routes through the global on401Hook instead of
  infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
  AbortControllers; refresh after a control action only runs when
  the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
  sync_state locally (no full chapter-list refetch); proper
  aria-label. Requeue-all gets its own confirm modal; both confirm
  modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
  for Accept: text/event-stream; pure shouldBypassProxyTimeout
  helper covered by unit tests.

Config / docs:

- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
  CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
  CRAWLER_BROWSER_RESTART_THRESHOLD.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-06 18:49:56 +02:00
2026-05-16 21:05:16 +02:00

Mangalord

A self-hosted manga and comics reader. Users browse, search, read, bookmark, and upload manga. The same HTTP API is consumed by the SvelteKit web UI and by external bots — there is no separate "bot API" — so anything you can do in the browser you can also do over curl with a bearer token.

Stack

  • Backend: Rust, axum 0.7, sqlx 0.8, Postgres 16. Argon2id passwords, HttpOnly SameSite=Lax session cookies, bearer tokens for bots.
  • Frontend: SvelteKit 2 with Svelte 5 runes, TypeScript.
  • Storage: pluggable Storage trait. LocalStorage ships in-tree; S3 and friends are first-class extension slots.
  • Deploy: a single host via Docker Compose.

Quick start

cp .env.example .env
docker compose up --build
Service URL
Frontend (and API browser) http://localhost:3000
API health http://localhost:3000/api/v1/health

The browser only ever talks to the frontend container on :3000. SvelteKit's hooks.server.ts reverse-proxies /api/* to the backend service over docker's internal network, so cookies stay same-origin and you don't need to publish the backend port or configure CORS to get a working deploy.

If you want to hit the backend directly (bot scripts, ops debugging), publish its port by editing the backend service in docker-compose.yml — change expose to ports: ["8080:8080"].

The first boot runs the migrations automatically. From there:

  1. Open http://localhost:3000, click Register, create a user.
  2. Click Upload to create a manga (optionally with a cover) and upload a chapter.
  3. Open the manga, click the chapter, read with j/k or the arrow keys.

Local development

For fast iteration, run Postgres in Docker and the backend + frontend natively:

docker compose -f docker-compose.dev.yml up -d

# Backend (separate shell)
cd backend
export DATABASE_URL=postgres://mangalord:mangalord@localhost:5432/mangalord
cargo run

# Frontend (separate shell)
cd frontend
npm install
npm run dev   # serves on http://localhost:5173, proxies /api to :8080

Port note: docker-compose.dev.yml exposes Postgres on 5432. If another Postgres is already bound to that port (or another Docker container is), stop it first or edit the host-port mapping in the dev compose file. Backend tests need a real Postgres that the test user can CREATEDB on; the dev container works because the mangalord role is the cluster superuser.

Tests

This project is developed test-first. Three levels:

# Backend: unit (in-module) + integration (#[sqlx::test] against a real DB)
cd backend && cargo test
cd backend && cargo clippy --all-targets -- -D warnings

# Frontend: Vitest unit tests and svelte-check
cd frontend && npm test
cd frontend && npm run check

# End-to-end: Playwright (mocks the backend at the page level by default)
cd frontend && npm run test:e2e

Backend integration tests provision a fresh, migrated database per test via sqlx::test, so they require DATABASE_URL to point at a Postgres where the configured user has CREATEDB.

Playwright starts the SvelteKit dev server on a dedicated port (5174) so it doesn't reuse whatever you may have running on 5173. To point Playwright at an already-running deployment instead, set E2E_BASE_URL=http://your.host.

API surface

Everything is namespaced under /api/v1/. /api/* outside the version prefix is reserved for future versioning.

Read endpoints (public)

Method Path Description
GET /api/v1/health Liveness probe.
GET `/api/v1/mangas?search=&sort=recent title&limit=&offset=`
GET /api/v1/mangas/{id} Single manga.
GET /api/v1/mangas/{id}/chapters Paginated chapter list, ordered by number.
GET /api/v1/mangas/{id}/chapters/{n} Single chapter.
GET /api/v1/mangas/{id}/chapters/{n}/pages Page metadata (storage keys + MIME).
GET /api/v1/files/{key} Streams a blob (cover or page). See Capability URLs below.

Write endpoints (require authentication)

Method Path Description
POST /api/v1/auth/register {username,password} → user + sets session cookie.
POST /api/v1/auth/login Same shape; rotates to a fresh session.
POST /api/v1/auth/logout Invalidates the session, clears the cookie.
GET /api/v1/auth/me Current user, or 401 if anonymous.
PATCH /api/v1/auth/me/password {current_password,new_password} → 401 on wrong current; on success deletes all of the user's sessions and mints a fresh one for the caller. Bot tokens stay (use DELETE /tokens/{id} to revoke).
POST /api/v1/auth/tokens {name} → bot bearer token (returned once).
DELETE /api/v1/auth/tokens/{id} Revoke a bot token (owner-only).
POST /api/v1/mangas Multipart: metadata JSON + optional cover image.
POST /api/v1/mangas/{id}/chapters Multipart: metadata JSON + N page image parts.
POST /api/v1/bookmarks {manga_id, chapter_id?, page?}.
DELETE /api/v1/bookmarks/{id} Remove a bookmark (owner-only).
GET /api/v1/me/bookmarks?limit=&offset= The caller's bookmarks.

Envelopes

Errors — every non-2xx response:

{ "error": { "code": "snake_case", "message": "human readable", "details": { "field?": "..." } } }

details is only present on validation_failed (422). code is the stable contract; the message is for humans and may change.

HTTP code
400 invalid_input, bad_file_key
401 unauthenticated
403 forbidden
404 not_found
409 conflict
413 payload_too_large
415 unsupported_media_type
422 validation_failed
500 internal_error

Lists — every paginated endpoint:

{
  "items": [ /* ... */ ],
  "page": { "limit": 50, "offset": 0, "total": 137 }
}

total is a number when the endpoint computes it (currently /mangas), otherwise null.

Auth quick-start

Browser

The frontend handles this for you: register → cookie set → writes work. Cookies are HttpOnly; SameSite=Lax; Secure (configurable via COOKIE_SECURE).

Capability URLs

GET /api/v1/files/{key} is unauthenticated by design: reads stay public so embedded <img> tags and external readers work without round-tripping a cookie or token. The keys themselves are unguessable — they're scoped under v4 UUIDs (mangas/{manga-uuid}/cover.png, mangas/{manga-uuid}/chapters/{chapter-uuid}/pages/0001.png) — so a leaked URL leaks at most the one file. Treat the URLs as bearer tokens for the bytes they reference and don't paste them into public chat. A future private-libraries feature would gate this endpoint behind a per-key ownership check; the architectural seam is documented in backend/src/api/files.rs.

Bots / scripts (bearer tokens)

The frontend on :3000 proxies /api/* through to the backend, so any URL below works against http://localhost:3000 in the default compose deploy. If you publish the backend port directly, swap in http://localhost:8080.

# 1. Log in once via cookies (or register).
curl -sb -c cookies.txt -X POST http://localhost:3000/api/v1/auth/login \
  -H 'content-type: application/json' \
  -d '{"username":"alice","password":"hunter2hunter2"}'

# 2. Mint a long-lived bot token. The `bearer` value is shown ONCE.
curl -sb cookies.txt -X POST http://localhost:3000/api/v1/auth/tokens \
  -H 'content-type: application/json' \
  -d '{"name":"ci-bot"}'
# → { "id": "...", "name": "ci-bot", "bearer": "raw-token-here", ... }

# 3. Use the bearer from anywhere.
curl -H 'Authorization: Bearer raw-token-here' \
     http://localhost:3000/api/v1/auth/me

Tokens are stored hashed (sha256) at rest; the raw value never leaves the response that created it. Revoke with DELETE /api/v1/auth/tokens/{id} while authenticated as the owner.

Configuration

All variables can be set in .env (for docker compose) or your shell (for cargo run). See .env.example for the full list with explanations.

Variable Default Purpose
DATABASE_URL required Postgres connection string.
BIND_ADDRESS 0.0.0.0:8080 Backend listen address.
STORAGE_DIR ./data/storage Local-storage root.
RUST_LOG info,mangalord=debug tracing-subscriber env filter.
COOKIE_SECURE true Secure flag on the session cookie.
COOKIE_DOMAIN (none) Scope the session cookie to a parent domain.
SESSION_TTL_DAYS 30 Session lifetime.
CORS_ALLOWED_ORIGINS (empty → same-origin) Comma-separated origin allowlist.
MAX_REQUEST_BYTES 209715200 (200 MiB) Hard cap on multipart request size.
MAX_FILE_BYTES 20971520 (20 MiB) Cap on a single image part.
BACKEND_URL http://backend:8080 Where the frontend's /api/* proxy points.

Deployment

For real hosts:

  • Front Mangalord with a TLS terminator (Caddy, nginx, traefik). Point :443 at the frontend container on :3000 — the SvelteKit proxy handles /api/* internally, so you only need one upstream. With same-origin routing you can leave CORS_ALLOWED_ORIGINS empty and the session cookie's SameSite=Lax does its job.
  • Set a strong Postgres password in .env before the first docker compose up. The defaults are fine for local dev only.
  • Keep COOKIE_SECURE=true behind HTTPS. Browsers drop Secure cookies on plain HTTP; the dev compose accepts COOKIE_SECURE=false for that case.
  • Watch RUST_LOG if you're noisy with debug — drop the ,mangalord=debug suffix in production to log at info for the app's spans.

Backup and restore

Two stateful volumes:

  • postgres-data — all metadata (users, mangas, chapters, pages, bookmarks).
  • storage-data — the actual cover and page bytes.

A consistent backup needs both. See docs/backup.md for step-by-step procedures and a restore drill.

Project layout

backend/      # axum + sqlx, see backend/src/ for the hexagonal layout
frontend/     # SvelteKit 2 / Svelte 5
docs/         # operator docs (backup procedure, etc.)
CLAUDE.md     # contributor playbook

The contributor guidelines (TDD, Conventional Commits, lockstep SemVer, hexagonal seams) are in CLAUDE.md.

Description
No description provided
Readme 7.7 MiB
Languages
Rust 57.1%
TypeScript 21.1%
Svelte 19.8%
Shell 0.9%
JavaScript 0.4%
Other 0.7%