The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds GET /api/v1/admin/system returning disk (scoped to storage_dir
via statvfs), memory, CPU, and a server-side alerts array that fires
at >90% disk or memory.
Disk uses nix::sys::statvfs directly rather than sysinfo's Disks API
to avoid mountpoint-matching gymnastics for the storage_dir. A new
`Storage::local_root() -> Option<&Path>` trait method exposes the
root; the default returns None so a future S3Storage gets `disk:
null` in the response instead of fabricated numbers.
CPU is sampled inline (refresh → 250ms sleep → refresh → read) so the
endpoint adds 250ms of latency per call. No background-cache yet —
admin traffic is low-volume and the moving parts aren't worth it
until polling shows up.
Alerts are evaluated server-side so the frontend can render them
without re-implementing the thresholds.
Five fixes bundled into one release:
- preserve user-attached tags across crawler upserts
(repo::crawler::sync_tags now scopes to added_by IS NULL; orphaned
attachments from deleted users are reaped as crawler-owned)
- gate manga PATCH and cover endpoints on uploaded_by (require_can_edit
in api::mangas; non-NULL uploaded_by must match the caller)
- equalise login response time across user-existence branches
(run argon2 against a OnceLock-cached dummy hash on the no-user
branch so timing doesn't leak username existence)
- crawler download defences (SSRF allowlist of host literals
including IPv4-mapped IPv6 ranges, 32 MiB streamed size cap,
reject non-whitelisted image types, three-way chapter-probe
classifier replaces the binary #avatar_menu check)
- tighten validation and clean up dead unload path
(attach_tag + create_token enforce 64-char caps; LocalStorage
rejects NUL bytes explicitly; reader flushFinalProgress drops
the always-405 sendBeacon path)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tightens three tests whose names overstated what they checked:
- `login_succeeds_and_rotates_session` now asserts the login cookie
differs from the registration cookie, and that the registration
cookie is still valid after login (the documented contract).
- `storage::local::rejects_path_traversal` exercises three extra
rejection paths the existing implementation already handled but the
tests didn't probe: `a/./b`, the single-segment `.`, and the empty
segment `a//b`.
- `create_and_use_bot_token` asserts that `token_hash` is *absent*
from the response (`get(...).is_none()`), not just `is_null()`,
which would have accepted an explicit `"token_hash": null` payload
too.
Adds four coverage cases that the audit flagged as missing:
- `me_rejects_expired_session` — hand-craft a session row with
`expires_at = now() - 1h`, hit `/auth/me` with the matching cookie,
expect 401 + `unauthenticated`. Proves the extractor's
`expires_at > now()` filter is wired.
- `concurrent_manga_bookmarks_serialised_by_unique_index` — spawn two
POSTs in parallel for the same `(user, manga, chapter=null)`,
assert one wins (201) and one collides (409) via the partial unique
index from migration 0004.
- `bookmark_create_accepts_bearer_token` — mint a bot token and POST
/bookmarks with `Authorization: Bearer`, asserting `CurrentUser`
resolves identically to the cookie path on a write endpoint (not
just `/auth/me`).
- Three new unit tests on `app::cors_layer` covering the allowlist
(origin reflected, credentials true), a foreign origin (no
allow-origin header emitted), and the same-origin default (empty
allowlist emits no CORS headers at all).
`cors_layer` is `pub(crate)` now so the tests in `app::tests` can
reach it; the function itself is unchanged.
No version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Migration 0003_pages.sql adds a `pages` table (id, chapter_id,
page_number, storage_key, content_type) with a unique (chapter_id,
page_number). New table because chapter pages can have different MIME
types per page; reconstructing keys from a single template would
break the moment a chapter mixes png and jpg pages.
- `domain::Page` + `repo::page` (create + list_for_chapter).
- The chapter upload handler now inserts one page row per part as it
writes the bytes to storage.
- GET /api/v1/mangas/{id}/chapters/{n}/pages returns `{pages: [...]}`
with the storage_key clients need to construct image URLs. 404 if
the manga or chapter doesn't exist; reads are public.
Storage trait grows `get_stream(&str) -> StreamingFile` returning a
`Pin<Box<dyn Stream<Item = io::Result<Bytes>> + Send>>` + size. The
local backend implements via `tokio::fs::File` + `tokio_util::io::
ReaderStream` with a 64 KiB chunk size. GET /api/v1/files/*key now
streams via `axum::body::Body::from_stream` instead of buffering — the
test asserts a 200 KiB file emits >1 frame end-to-end through the
router.
Frontend:
- lib/api/client.ts gains `fileUrl(key)` so components don't
reconstruct the `/api/v1/files/...` path manually.
- lib/api/chapters.ts gains `ChapterPage` type + `getChapterPages` (the
type is named ChapterPage to avoid colliding with `Page` from
client.ts, which is the pagination envelope).
- /manga/[id]/+page.svelte: overview with cover, title, author,
description, chapter list, and a disabled bookmark control (real
bookmarking lands in feat/bookmarks). Responsive at 640 px.
- /manga/[id]/chapter/[n]/+page.svelte: paginated reader. Current page
loads eagerly; next page is preloaded in a hidden img so navigation
feels instant. Keyboard handler maps ArrowRight/j/Space → next,
ArrowLeft/k → prev, Home/End → first/last; skips when the user is
typing in an input. Focus ring on the prev/next buttons.
- SSR is disabled on both routes via `export const ssr = false` so the
client-only fetch flow doesn't need to be replicated server-side; the
routes are interactive features, not SEO surfaces.
- E2E (e2e/reader.spec.ts): overview shows the title/cover/chapter
list; reader pages through three pages via ArrowRight, j, k, and
ArrowLeft, and the preload img holds the page-2 src on initial load.
Lockstep version bump to 0.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
