Broadcasts analysis progress so the dashboard updates live:
- analysis::events: AnalysisEvents broadcaster + AnalysisEvent
(Enqueued / Started / Completed / Failed), carrying the
manga/chapter/page breadcrumb.
- The worker daemon resolves each page's breadcrumb (repo::page::locate)
and publishes Started before dispatch and Completed/Failed after.
- The admin reenqueue publishes Enqueued (scoped by manga/chapter).
- GET /v1/admin/analysis/status/stream — SSE (RequireAdmin) forwarding
each event as a named `analysis` frame; broadcast lag emits a `lagged`
frame. AppState carries the always-present events bus.
Tests: worker publishes started+completed (with breadcrumb) and failed;
SSE route is admin-gated (403 non-admin) and returns text/event-stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the analyze_page job kind into the page-create paths and adds admin
controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled,
config::AnalysisConfig):
- Chapter upload (api::chapters) enqueues one analyze_page job per page
after the tx commits (best-effort; failures logged, never fatal).
- Crawler content sync (persist_pages → RETURNING id) enqueues per
persisted page when the daemon dispatcher's analysis_enabled flag is set;
threaded through sync_chapter_content (CLI/resync pass false).
- repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT,
skipping done pages (only_unanalyzed) and existing pending jobs.
- New admin endpoints (RequireAdmin, 503 when disabled, audited):
POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force).
Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages
enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503,
non-admin 403) and force-analyze (force flag, 404).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tick cover-backfill pass to the crawler daemon so mangas whose
cover download failed on first attempt get retried — the metadata pass's
early-stop optimisation otherwise prevents the walk from revisiting them.
Adds admin-only POST /admin/mangas/:id/resync and POST /admin/chapters/:id/resync
that refetch metadata + cover (or chapter content with force_refetch) from the
crawler source synchronously and return the refreshed row. Surfaced in the
UI as "Force resync" buttons on the manga detail and reader pages,
admin-only via session.user.is_admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When `PRIVATE_MODE=true`, every API path except a small allowlist
(`/health`, `/auth/{config,login,logout,register}`) requires a valid
session cookie or bearer token — anonymous reads are rejected with
401. Self-registration is force-disabled in private mode regardless
of `ALLOW_SELF_REGISTER`, so a locked-down instance flips with a
single switch (admins still mint accounts via `POST /admin/users`).
The backend gate is a tower middleware that reuses the existing
`CurrentUser` extractor, so the cookie + bearer paths cannot drift
from per-handler auth. `/auth/config` now exposes the flag plus the
effective `self_register_enabled` value so the frontend can render
the navbar correctly on the first paint.
On the frontend, a new universal root `+layout.ts` fetches the
config and redirects anonymous visitors to `/login?next=<path>`
before page-specific loads fire. The redirect is UX only — the
backend middleware is the source of truth, so crafted requests
still 401.
Defaults stay public (`PRIVATE_MODE=false`); existing deployments
need no env change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets operators run a closed-membership deployment by setting
ALLOW_SELF_REGISTER=false (default true, so existing deploys are
unaffected). When off, POST /auth/register returns 403 forbidden. The
rate-limit token is consumed BEFORE the disabled check so the timing
doesn't distinguish enabled-but-rejected from disabled — closes the
toggle-state probe channel.
New public GET /auth/config returns { self_register_enabled: bool }
so the frontend can render its register affordances correctly
without conflating "disabled" with "rate-limited" (which a probe
attempt would).
Frontend: a lightweight reactive `authConfig` store loads the flag
once on root-layout mount (and again on /register direct navigation,
which bypasses the layout's onMount). Header hides the Register link
when the toggle is off; /register renders a "self-registration is
disabled — ask an administrator" notice instead of the form.
Admin-create endpoint that pairs with this toggle is intentionally
not in this PR — it lands as the next branch (feat/admin-user-create).
The toggle alone is independently useful for deployments that want
to lock down enrollment without yet wiring an admin UI.
Three features bundled into one release:
- rate-limit /auth/login, /register, /me/password (token bucket,
5 req/sec sustained with 10-request burst by default; 429 +
Retry-After header on hit; tracing::warn! per hit so operators
see attack patterns; AUTH_RATE_PER_SEC / AUTH_RATE_BURST env knobs)
- handle SIGTERM for graceful container stops (replaces bare
ctrl_c() with a select over ctrl_c + SignalKind::terminate() so
docker compose stop runs the daemon shutdown path instead of
letting Chromium leak past SIGKILL)
- clear session.user on 401 from any API call (setOn401Hook in
api/client.ts, registered from session.svelte.ts gated on
$app/environment::browser so the SSR bundle never installs it;
fixes "logged in but no bookmarks/collections" mid-session
expiry state)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds PUT /mangas/:id/cover (multipart) and DELETE /mangas/:id/cover so
covers can be replaced or cleared after creation, and wires a dedicated
/manga/[id]/edit SvelteKit route that combines the existing PATCH with
the new cover endpoints. Cover PUT cleans up the old blob when the
extension changes, swallowing StorageError::NotFound so a manually-gone
file doesn't surface as a 404 to the client. Edit link on the manga
detail page is gated on session.user, matching the auth posture of the
underlying handlers.
Also pins the local-dev port story via loadEnv() in vite.config.ts so
VITE_PORT / BACKEND_URL from a (gitignored) .env keep the dev URL
stable across runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-user reading progress and uploader attribution.
Schema (migration 0011): `read_progress` table (one row per (user,
manga); chapter_id nullable on chapter delete) and nullable
`uploaded_by` columns on mangas + chapters with partial indexes
scoped to non-null rows.
Endpoints (all `/me/*`, auth-scoped):
- PUT `/v1/me/read-progress` upserts. FK violations + cross-manga
chapter ids both surface as 4xx (404 / 422) so the API can't be
used to write logically invalid rows.
- GET `/v1/me/read-progress` paged newest-first list.
- GET `/v1/me/read-progress/:manga_id` enriched with chapter_number
for the manga page's Continue CTA.
- DELETE `/v1/me/read-progress/:manga_id` idempotent.
- GET `/v1/me/uploads` interleaved manga + chapter uploads as a
tagged union; limit-only pagination.
Existing manga + chapter upload handlers stamp `uploaded_by`.
Frontend:
- Reader emits progress on mount + page change (debounce) and via
IntersectionObserver in continuous mode. High-water mark is seeded
from the persisted server value so re-opening a chapter doesn't
regress to page 1. Tab close survives via `sendBeacon` (fallback
`keepalive` fetch); SPA navigation flushes via regular fetch.
- Manga detail page shows "Continue reading Chapter N — page M"
above the chapters list, working even for mangas with >50
chapters.
- New `/profile/history` tab with reading history (clear-per-row,
inline error on failure) and uploads (mangas + chapters mixed
chronologically with type-aware rendering).
171 backend tests (incl. 16 history tests covering ownership, FK
race, cross-link guard, chapter SET NULL behaviour) and 97 frontend
tests + svelte-check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the pre-1.0 password-change story flagged by the audit. Browser
users and bot owners both go through PATCH /api/v1/auth/me/password
with the current + new password in the body.
Implementation in `api::auth::change_password`:
- CurrentUser-gated: 401 if unauthenticated.
- Verifies current_password against the stored argon2 hash. Wrong
current → 401 unauthenticated, matching the login contract.
- new_password runs through the same `validate_password` used at
registration (≥8 chars). Weak → 400 invalid_input.
- On success, wraps the swap in a single transaction:
- UPDATE users.password_hash with a fresh argon2 hash.
- DELETE every session for this user (signs out other devices —
any cookie stolen before the change is dead now).
- INSERT a new session and mint a fresh cookie so the caller stays
logged in.
- 204 + Set-Cookie on success.
Bot tokens (api_tokens) are intentionally left alone. They're explicit
opt-in credentials that the user can already audit and revoke
individually via DELETE /auth/tokens/{id}; rotating them on every
password change would surprise CI scripts.
Repo refactor: `repo::session::create` accepts `impl PgExecutor<'_>`
(same pattern feat/uploads used for chapters), and a new
`session::delete_all_for_user` covers the "sign out everywhere"
write. The existing `delete_by_token_hash` (used by logout) is
unchanged.
Coverage in tests/api_auth.rs (4 cases):
- change_password_rotates_sessions_and_swaps_credentials — happy path
asserts the new cookie differs from the original, that both the
original cookie AND a second-device cookie become invalid, that the
new cookie keeps working, that login with the old password fails
(401) and login with the new password succeeds.
- change_password_rejects_wrong_current_with_401 — wrong current
password returns 401 unauthenticated.
- change_password_rejects_weak_new_password — new_password "short"
returns 400 invalid_input.
- change_password_requires_authentication — no cookie returns 401.
README updated with the new endpoint in the auth table.
Lockstep version bump to 0.10.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously a storage failure mid-chapter-upload left a partial chapter
row pointing at a `page_count` that didn't match what was on disk, plus
any successfully-inserted page rows. Same shape for a manga create
where the cover put or cover_image_path UPDATE failed after the manga
row was already inserted.
Fix at the DB layer: open `pool.begin()` at the start of the create,
do all DB writes against `&mut *tx`, commit only after the full
sequence succeeds. If anything before commit fails, the transaction is
rolled back on drop and the DB stays consistent. Bytes already written
to storage on a rolled-back transaction become orphans on disk; a
future reaper can sweep them, and we prioritise DB consistency over
storage tidiness in this branch.
- repo::manga::create / set_cover_image_path: signature changed to
`impl PgExecutor<'_>` so handlers can pass either `&PgPool` or
`&mut *tx`. set_cover_image_path is new — replaces the inline
`UPDATE` in the manga upload handler so the call site stays
consistent.
- repo::chapter::create / set_page_count: same shape.
- repo::page::create: same.
- api::mangas::create and api::chapters::create both open a
transaction around their DB writes; storage puts happen inside the
transaction window (since they must precede the page-row insert), so
a failed put aborts before commit.
New integration test (api_uploads::chapter_upload_rolls_back_when_
storage_fails_mid_loop) uses a `FailingStorage` helper that errors on
the N-th `put`. With N=1 (page 2 fails), the handler returns 500 and
the chapter + page tables stay empty.
`harness_with_failing_storage` is exposed alongside the existing
`harness` so future tests can reuse it for other fault-injection
cases.
Lockstep version bump to 0.9.3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/v1/mangas and POST /api/v1/mangas/{id}/chapters now accept
multipart/form-data, gated by CurrentUser:
- /mangas: required `metadata` part (NewManga JSON) + optional `cover`
image part.
- /mangas/{id}/chapters: required `metadata` (NewChapter JSON) + one or
more `page` parts ordered by arrival. Returns 404 if the parent manga
doesn't exist, 409 on duplicate (manga_id, number).
MIME is sniffed via the `infer` crate (magic bytes), not the
client-supplied filename or Content-Type. Whitelist:
jpeg / png / webp / gif / avif. Anything else → 415
unsupported_media_type. The stored key's extension is derived from the
sniffed type so a "page1.png" that's actually a JPEG lands as `.jpg`.
Size cap is two-layer:
- Request body cap (config.max_request_bytes, default 200 MiB) enforced
by axum's DefaultBodyLimit before the handler sees the request.
- Per-image-part cap (config.max_file_bytes, default 20 MiB) enforced
after reading the part, so a single oversized image can't pass even
if the total request fits.
Storage keys follow the layout documented in CLAUDE.md:
- mangas/{manga_id}/cover.{ext}
- mangas/{manga_id}/chapters/{chapter_id}/pages/{nnnn}.{ext} (1-indexed).
AppError grows PayloadTooLarge/UnsupportedMediaType/ValidationFailed
(413 / 415 / 422). ValidationFailed carries a `details` JSON object the
client can use to highlight bad fields (e.g. {"title":"required"}).
Top-level matching in code() stays exhaustive.
Backend coverage in tests/api_uploads.rs (10 cases):
- create_manga_with_cover_stores_image — file is reachable via
/api/v1/files/{key} with the right Content-Type.
- create_manga_without_cover_leaves_path_null.
- create_manga_rejects_non_image_cover_with_415 — PDF claimed as png.
- create_manga_rejects_oversized_cover_with_413.
- create_chapter_with_pages_stores_each — extension derived from
sniffed MIME, files reachable in arrival order.
- create_chapter_rejects_when_no_pages_with_422 — details.page set.
- create_chapter_rejects_renamed_non_image_page → 415.
- create_chapter_returns_409_on_duplicate_number.
- create_chapter_requires_authentication → 401.
- create_chapter_under_unknown_manga_is_404.
Existing tests/api_mangas.rs is migrated to multipart; the create
response is now 201 Created. tests/common::MultipartBuilder builds the
body by hand so the test crate stays free of HTTP-client deps.
Frontend lib/api/mangas.ts: createManga now sends FormData (metadata +
optional cover Blob). Browser fills in the boundary header automatically.
Vitest asserts the FormData structure via FileReader (jsdom doesn't
implement Blob.text()).
E2E tests wait for the post-hydration nav-login link before
interacting with the login form, fixing a flake where pre-hydration
clicks would submit via the browser default and bypass our handler.
Lockstep version bump to 0.5.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full auth flow. Reads stay public; writes (currently only POST
/api/v1/mangas) require a CurrentUser. Both browsers and bot scripts hit
the same endpoints — they just present credentials differently.
Migration 0002_auth.sql introduces users.password_hash, a sessions
table, and an api_tokens table. Sessions and api_tokens store only
sha256(raw_token) — the raw value lives in the cookie or the
Authorization header.
New endpoints under /api/v1/auth/:
- POST /register — argon2id hash, creates a session, sets cookie.
- POST /login — verifies, rotates to a fresh session (old ones expire
naturally so other devices stay signed in).
- POST /logout — deletes the server-side session row + clears the
cookie via Max-Age=0.
- GET /me — current user via the new CurrentUser extractor.
- POST /tokens — issue a bot bearer token; raw value returned exactly
once at creation.
- DELETE /tokens/{id} — owner-only: 404 if unknown, 403 if it exists
but belongs to another user, 204 on success.
The CurrentUser axum extractor resolves cookie first, then
Authorization: Bearer; failure → AppError::Unauthenticated (401). New
AppError variants Unauthenticated/Forbidden/Conflict carry the matching
envelope codes; the top-level match in `code()` stays exhaustive.
Backend integration coverage in tests/api_auth.rs: register sets a
HttpOnly SameSite=Lax cookie and never leaks password_hash; duplicate
username → 409; weak password → 400; login rotates the cookie; wrong
password / unknown user → 401; /me with vs without cookie; logout
invalidates the cookie; bot-token roundtrip via Bearer; user A cannot
delete user B's token (403); unknown delete → 404.
Frontend:
- lib/api/auth.ts — typed wrappers; me() returns null on 401.
- lib/session.svelte.ts — per-tab user state with a seq counter to
guard against an in-flight /me clobbering a fresh setUser.
- lib/api/client.ts — request<T> returns undefined for 204.
- routes/login + routes/register — forms with action="javascript:void(0)"
so the no-JS path is a no-op (avoids the hydration-race where a
pre-attach click would submit via the browser default).
- routes/+layout.svelte — session-aware nav: spinner → user + Logout,
or Login / Register.
- e2e/auth-flow.spec.ts — login flips the layout, logout flips back;
bad credentials surface the API error message.
Config grows AuthConfig (cookie_secure, cookie_domain, session_ttl_days)
and CORS_ALLOWED_ORIGINS. CORS middleware is mounted in app::build and
stays a no-op (same-origin) until origins are listed.
Lockstep version bump to 0.3.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move every handler from /api/* to /api/v1/*. /api/* is now reserved for
future versioning.
Standardise the error response shape across the API as
{"error": {"code": "snake_case", "message": "..."}}. AppError gains a
`code()` whose top-level variants are matched exhaustively without a
wildcard — new variants are a compile error until coded. 500-class
responses always emit the fixed "internal error" string and log the
real cause via tracing only.
Lock in the list pagination envelope as {"items": [...], "page": {
"limit", "offset", "total"}} and apply it to GET /api/v1/mangas. `total`
serialises as null until feat/list-search-polish lands an indexed count.
The frontend client parses the envelope into ApiError.code with an
http_error fallback for non-JSON bodies. listMangas now returns the
paged shape; the root route consumes .items. New client.test.ts covers
envelope parsing and the fallback paths.
Lockstep version bump to 0.2.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
