The backend now boots an internal crawler daemon that runs a daily
metadata pass (CRAWLER_DAILY_AT in CRAWLER_TZ, advisory-lock guarded
for multi-replica safety) and drains SyncChapterContent jobs from
crawler_jobs through a worker pool. Chromium launches lazily on first
job and is torn down after CRAWLER_IDLE_TIMEOUT_S seconds of inactivity.
Modules:
- crawler::browser_manager — lazy-launch / idle-teardown wrapper
around browser::Handle, with an on_launch hook that re-injects
PHPSESSID on every fresh Chromium spawn.
- crawler::pipeline — run_metadata_pass (the shared discover/upsert
/cover/sync-chapters loop) and the enqueue_bookmarked_pending helper
used by the cron tick.
- crawler::daemon — cron task + worker pool, behind two trait seams
(MetadataPass, ChapterDispatcher) so tests can inject stubs without
standing up Chromium or a live source.
Behavior:
- CRAWLER_DAEMON=false skips daemon spawn entirely (default for tests).
- Catch-up tick fires on startup if the last persisted slot was missed.
- A SyncOutcome::SessionExpired sets a sticky AtomicBool; workers
idle until operator restart with a refreshed PHPSESSID.
- Worker dispatch wrapped in catch_unwind so a panicking handler
marks the job failed instead of taking down the worker.
- Migration 0015 adds a small crawler_state k-v table for the
last_metadata_tick_at watermark.
Dep additions: chrono-tz (IANA TZ parsing).
CLI (bin/crawler) reuses pipeline::run_metadata_pass and now holds
the browser via BrowserManager so the on_launch session injection
flow stays in one place. Inline chapter-content sync semantics are
unchanged — the queue is for the daemon, force-refetches and manual
backfills still bypass it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds enqueue / lease / ack_done / ack_failed / release / reap_done on
crawler::jobs, backed by the existing crawler_jobs table. lease() uses
a single FOR UPDATE SKIP LOCKED CTE that also re-claims stale running
rows (crashed-worker recovery), and ack_failed applies an exponential
backoff capped at 1h before retrying.
Migration 0014 adds a partial unique index on
(payload->>'chapter_id') restricted to (pending|running)
sync_chapter_content jobs, so producers can just
INSERT ... ON CONFLICT DO NOTHING without racing each other. The slot
frees again the moment the job leaves the in-flight states, so a
future force-refetch can re-enqueue.
Library-only — no daemon, no API hook. Those land in the next two
phases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real-world sources publish multiple chapters at the same number:
different scanlators ("Ch.52 from bloomingdale" + "Ch.52 from mina"),
translator notices and farewells, alt-translations. The (manga_id,
number) UNIQUE constraint from 0001 silently collapsed all of those
into a single row via the upsert path in repo::crawler. Migration 0013
drops the constraint; sync_manga_chapters now plain-INSERTs each
SourceChapterRef so every parsed chapter survives as its own row.
Identity moves from the (manga_id, number) tuple to the chapter UUID:
- `GET /api/v1/mangas/:manga_id/chapters/:chapter_id` (replaces :number)
- `GET /api/v1/mangas/:manga_id/chapters/:chapter_id/pages`
- `repo::chapter::find_by_id_in_manga` (replaces find_by_manga_and_number)
- Frontend reader route renamed to `/manga/[id]/chapter/[chapter_id]`
- Chapter links throughout (manga page list, continue-reading CTA,
reader prev/next, history rows, bookmark cards) use chapter.id
- API clients getChapter/getChapterPages take a chapter id string
read_progress + bookmarks already FK chapter_id; they only enrich with
chapter_number for display, which is preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-user reading progress and uploader attribution.
Schema (migration 0011): `read_progress` table (one row per (user,
manga); chapter_id nullable on chapter delete) and nullable
`uploaded_by` columns on mangas + chapters with partial indexes
scoped to non-null rows.
Endpoints (all `/me/*`, auth-scoped):
- PUT `/v1/me/read-progress` upserts. FK violations + cross-manga
chapter ids both surface as 4xx (404 / 422) so the API can't be
used to write logically invalid rows.
- GET `/v1/me/read-progress` paged newest-first list.
- GET `/v1/me/read-progress/:manga_id` enriched with chapter_number
for the manga page's Continue CTA.
- DELETE `/v1/me/read-progress/:manga_id` idempotent.
- GET `/v1/me/uploads` interleaved manga + chapter uploads as a
tagged union; limit-only pagination.
Existing manga + chapter upload handlers stamp `uploaded_by`.
Frontend:
- Reader emits progress on mount + page change (debounce) and via
IntersectionObserver in continuous mode. High-water mark is seeded
from the persisted server value so re-opening a chapter doesn't
regress to page 1. Tab close survives via `sendBeacon` (fallback
`keepalive` fetch); SPA navigation flushes via regular fetch.
- Manga detail page shows "Continue reading Chapter N — page M"
above the chapters list, working even for mangas with >50
chapters.
- New `/profile/history` tab with reading history (clear-per-row,
inline error on failure) and uploads (mangas + chapters mixed
chronologically with type-aware rendering).
171 backend tests (incl. 16 history tests covering ownership, FK
race, cross-link guard, chapter SET NULL behaviour) and 97 frontend
tests + svelte-check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-owned named lists of mangas with an add-to-collection modal on
the manga page and dedicated /collections and /collections/:id pages.
- Schema (0010): `collections` (per-user case-insensitive name
uniqueness) + `collection_mangas` join with cascade FKs.
- Endpoints: full CRUD on `/v1/collections`, idempotent add/remove
for `/v1/collections/:id/mangas`, and `/v1/mangas/:id/my-collections`
for the modal's pre-checked state. Owner-mismatch surfaces as 404
(not 403) so the API doesn't disclose collection existence to
non-owners; the frontend funnels 401 to /login. Three-state PATCH
via a new shared `domain::patch::Patch<T>` lets clients distinguish
"leave alone", "clear", and "set" for description.
- Frontend: reusable `Modal` component (focus trap, opt-in
backdrop close, ESC) and `AddToCollectionModal` with optimistic
toggling that's race-safe under fast clicks. /collections page
renders cover-collage cards; /collections/:id is editable with
per-card remove. Top nav gets a Collections link.
155 backend tests (incl. 21 collection tests covering ownership,
idempotence, sample-cover enrichment, three-state PATCH, FK race);
88 frontend tests; svelte-check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds first-class manga metadata across the stack:
- **Status** (ongoing / completed), **alternative titles**, normalized
**multi-author** support, **curated genres** (13 seeded), and
**free-form user tags** (case-insensitive, globally shared). Each is
modelled as its own table joined to mangas; `mangas.author` is
backfilled into `authors` + `manga_authors` and dropped.
- New endpoints: `PATCH /v1/mangas/:id` (three-state `description`),
`POST/DELETE /v1/mangas/:id/tags[/:tag_id]`, `GET /v1/genres`,
`GET /v1/tags?search=`.
- `GET /v1/mangas` now returns `MangaCard` (with authors + genres
batched in) and supports `?status=`, `?author_id=`, `?genre_id=`,
`?tag_id=` filters — AND across facets, with empty-array no-op
semantics for the unnest primitive.
- `GET /v1/mangas/:id` returns the enriched `MangaDetail` with tags.
- Frontend: reusable `Chip` component; manga detail page renders
authors as chips linking to `/authors/:id` (Phase 2), a status
badge, alt titles, genres, and tags with inline add/remove (only
the attacher sees remove); upload form supports multi-author /
multi-genre / alt titles / status; search page gets a collapsible
URL-synced filter panel with keyboard-navigable tag autocomplete.
- 126 backend tests (incl. AND-across-facets primitive, case-insens
author/tag de-dup, transactional create rollback, PATCH semantics
for missing / null / set on description); 72 frontend tests +
svelte-check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a vertical-scroll continuous mode to the reader alongside the
existing single-page mode. A segmented toggle in the reader top bar
switches between them; in continuous mode a gap selector
(None/Small/Medium/Large → 0/12/32/64px) controls the spacing
between stacked pages. Settings page mirrors the same controls.
Backend: new user_preferences table (one row per user, lazily
inserted, ON DELETE CASCADE) and GET/PATCH /api/v1/auth/me/preferences
gated by the existing CurrentUser extractor. Allowed values are
enforced both by API validation and table-level CHECK constraints.
Eight integration tests cover defaults, persistence, partial
updates, validation errors, auth, per-user isolation, and cascade.
Frontend: a new preferences store mirrors the theme-store pattern
with a localStorage shadow so anonymous browsers get a consistent
experience and logged-in users don't flash defaults while the
server response is in flight. Server values that the frontend
doesn't recognize (forward-compat) are ignored rather than poisoning
the UI; non-401 PATCH errors revert the optimistic local update;
logout clears the shadow so user A's settings don't follow user B
on a shared browser.
In continuous mode native scrolling handles Space/PageDown/arrows;
Home/End remain wired and call scrollIntoView() so jumping to chapter
bounds stays one keystroke. Single-page mode (chevrons, arrow-key
pagination, next-page preload) is unchanged.
Versions bumped 0.13.0 → 0.14.0 in lockstep.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four small follow-ups from the 0.9.0 audit, none of them
user-visible:
- Migration 0007 drops `chapters_manga_idx`. The 0001 schema declared
both `UNIQUE (manga_id, number)` and `CREATE INDEX chapters_manga_idx
ON (manga_id, number)`, but Postgres maintains an identical index
for the unique constraint automatically — the explicit one was just
paying for a second per-write update. Query plans are unchanged
because the planner already preferred the constraint's index.
- `upload::parse_image` sniffs from the first 64 bytes instead of the
full image buffer. `infer` only looks at magic bytes anyway, so
scanning 20 MiB is wasted work. Functionally identical; cheaper in
the hot path.
- AVIF was on the whitelist but had no test fixture. New `avif_bytes`
helper produces a minimal `ftyp avif` header that `infer` recognises,
and a new `accepts_avif` unit test covers the path end-to-end.
- Frontend `request()` sets `credentials: 'include'`. Same-origin
callers see no change (default was already `'same-origin'`), but the
first user who configures `CORS_ALLOWED_ORIGINS` for a cross-origin
deployment gets working cookies without having to chase a runtime
ApiError trail.
No version bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related correctness fixes from the audit:
- Username uniqueness was case-sensitive (`username text UNIQUE`), so
"Alice" and "alice" could both register and then race on login.
Migration 0006 adds a unique index on `lower(username)`; the
existing constraint is kept (overlapping but cheap) to avoid a
destructive migration on any deployments that may already exist.
`repo::user::find_by_username` now matches on `lower(username) =
lower($1)` so login is case-insensitive against the same index.
Test: registering "alice" then "Alice" returns 409 conflict; login
with "ALICE" succeeds against the existing user.
- `POST /api/v1/bookmarks` silently accepted `page: 0` and `page: -1`
even though both are nonsense for a 1-indexed page number. Reject
with 422 `validation_failed` and `details.page` populated, matching
the pattern used for missing-metadata / empty-title elsewhere. Test
covers both 0 and -1.
Lockstep version bump to 0.9.4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Migration 0005_search.sql enables pg_trgm and adds GIN indexes
(gin_trgm_ops) on mangas.title and on mangas.author (partial, WHERE
author IS NOT NULL).
- repo::manga::list keeps the existing substring (ILIKE) clause and
adds the `%` operator on title + author so the search tolerates typos
('narto' → 'Naruto'). Both branches share the trgm index. A second
count(*) query (same WHERE clause, indexed) yields the total without
scanning twice in any meaningful sense.
- New ListSort enum (Recent / Title) interpolated into ORDER BY from a
hard-coded match — never from request input, so the format!() is not
a SQL-injection seam. Default stays Recent (created_at DESC).
- api::mangas accepts `?sort=recent|title` (snake_case) via serde and
returns `page.total` as a number instead of null.
- api::pagination::PagedResponse gains a `with_total` constructor.
Backend coverage in tests/api_mangas.rs (4 new cases plus the existing
list_is_empty_initially updated to assert total: 0):
- list_returns_total_count_independent_of_pagination — limit=2 with 3
rows returns 2 items and total=3.
- search_via_trigram_tolerates_typos — `?search=narto` finds Naruto.
- list_sort_title_orders_alphabetically — three out-of-order inserts
come back A→Z.
- search_reflects_filtered_total — search narrows total to 1.
Frontend:
- lib/api/mangas.ts gains a `MangaSort` type and threads `sort` through
listMangas's query-string builder.
- Home page renders a "Sort" select (Recent / Title A→Z) that re-runs
the list query, and shows "Showing N of M" when total is present.
Lockstep version bump to 0.8.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Migration 0004_bookmarks_unique.sql adds a partial unique index on
(user_id, manga_id) WHERE chapter_id IS NULL. The 0001 UNIQUE
constraint over (user_id, manga_id, chapter_id) doesn't block dupes
when chapter_id is NULL under Postgres's default NULLS DISTINCT, so a
user could otherwise bookmark the same manga twice at the manga
level. Chapter-level dupes are still caught by the 0001 constraint.
- repo::bookmark with create / list_for_user / find_owner / delete.
create catches the 23505 unique violation and surfaces it as
AppError::Conflict so handlers return a clean 409.
- POST /api/v1/bookmarks { manga_id, chapter_id?, page? } — CurrentUser
required. Pre-validates the manga exists (404 if not) and, when
chapter_id is supplied, that the chapter belongs to that manga (also
404), so FK violations can't bubble up as 500s.
- DELETE /api/v1/bookmarks/{id} — owner-only. 404 if unknown, 403 if it
exists for another user, 204 on success. Idempotent: deleting an
already-deleted bookmark is 404, not 500.
- GET /api/v1/me/bookmarks — paged envelope, sorted by created_at DESC,
scoped to the current user so the URL itself can't be used to peek at
someone else's bookmarks.
Integration coverage in tests/api_bookmarks.rs (9 cases): create+list
returns only own; duplicate manga-level bookmark → 409; unknown manga
→ 404; unauthenticated POST → 401; user A cannot delete user B's
bookmark (403); unknown delete → 404; double-delete → 404, not 500;
/me/bookmarks requires auth; paged envelope shape on empty list.
Frontend:
- lib/api/bookmarks.ts with createBookmark / deleteBookmark /
listMyBookmarks. listMyBookmarksOrEmpty wraps the 401 case so pages
can render anonymously without try/catch boilerplate.
- /manga/[id] overview: pre-loads the user's bookmark list in its load
function and renders either:
- "★ Bookmarked" / "☆ Bookmark" toggle with aria-pressed when authed;
click POSTs or DELETEs and mutates a local working copy of the
bookmark list (optimistic UI without re-fetching);
- or a "Sign in to bookmark" link for anonymous users.
- /bookmarks page lists the current user's bookmarks (chapter-level
bookmarks link into the reader, manga-level back to the overview).
Anonymous users see a sign-in prompt instead of a 401 page.
E2E in e2e/bookmarks.spec.ts (3 cases): authed toggle round-trip
(bookmark, see in /bookmarks list, unbookmark); anonymous user gets the
sign-in CTA on the overview; anonymous /bookmarks shows the sign-in
prompt. Existing reader.spec.ts updated for the new
bookmark-signin/toggle test IDs.
Lockstep version bump to 0.7.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Migration 0003_pages.sql adds a `pages` table (id, chapter_id,
page_number, storage_key, content_type) with a unique (chapter_id,
page_number). New table because chapter pages can have different MIME
types per page; reconstructing keys from a single template would
break the moment a chapter mixes png and jpg pages.
- `domain::Page` + `repo::page` (create + list_for_chapter).
- The chapter upload handler now inserts one page row per part as it
writes the bytes to storage.
- GET /api/v1/mangas/{id}/chapters/{n}/pages returns `{pages: [...]}`
with the storage_key clients need to construct image URLs. 404 if
the manga or chapter doesn't exist; reads are public.
Storage trait grows `get_stream(&str) -> StreamingFile` returning a
`Pin<Box<dyn Stream<Item = io::Result<Bytes>> + Send>>` + size. The
local backend implements via `tokio::fs::File` + `tokio_util::io::
ReaderStream` with a 64 KiB chunk size. GET /api/v1/files/*key now
streams via `axum::body::Body::from_stream` instead of buffering — the
test asserts a 200 KiB file emits >1 frame end-to-end through the
router.
Frontend:
- lib/api/client.ts gains `fileUrl(key)` so components don't
reconstruct the `/api/v1/files/...` path manually.
- lib/api/chapters.ts gains `ChapterPage` type + `getChapterPages` (the
type is named ChapterPage to avoid colliding with `Page` from
client.ts, which is the pagination envelope).
- /manga/[id]/+page.svelte: overview with cover, title, author,
description, chapter list, and a disabled bookmark control (real
bookmarking lands in feat/bookmarks). Responsive at 640 px.
- /manga/[id]/chapter/[n]/+page.svelte: paginated reader. Current page
loads eagerly; next page is preloaded in a hidden img so navigation
feels instant. Keyboard handler maps ArrowRight/j/Space → next,
ArrowLeft/k → prev, Home/End → first/last; skips when the user is
typing in an input. Focus ring on the prev/next buttons.
- SSR is disabled on both routes via `export const ssr = false` so the
client-only fetch flow doesn't need to be replicated server-side; the
routes are interactive features, not SEO surfaces.
- E2E (e2e/reader.spec.ts): overview shows the title/cover/chapter
list; reader pages through three pages via ArrowRight, j, k, and
ArrowLeft, and the preload img holds the page-2 src on initial load.
Lockstep version bump to 0.6.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full auth flow. Reads stay public; writes (currently only POST
/api/v1/mangas) require a CurrentUser. Both browsers and bot scripts hit
the same endpoints — they just present credentials differently.
Migration 0002_auth.sql introduces users.password_hash, a sessions
table, and an api_tokens table. Sessions and api_tokens store only
sha256(raw_token) — the raw value lives in the cookie or the
Authorization header.
New endpoints under /api/v1/auth/:
- POST /register — argon2id hash, creates a session, sets cookie.
- POST /login — verifies, rotates to a fresh session (old ones expire
naturally so other devices stay signed in).
- POST /logout — deletes the server-side session row + clears the
cookie via Max-Age=0.
- GET /me — current user via the new CurrentUser extractor.
- POST /tokens — issue a bot bearer token; raw value returned exactly
once at creation.
- DELETE /tokens/{id} — owner-only: 404 if unknown, 403 if it exists
but belongs to another user, 204 on success.
The CurrentUser axum extractor resolves cookie first, then
Authorization: Bearer; failure → AppError::Unauthenticated (401). New
AppError variants Unauthenticated/Forbidden/Conflict carry the matching
envelope codes; the top-level match in `code()` stays exhaustive.
Backend integration coverage in tests/api_auth.rs: register sets a
HttpOnly SameSite=Lax cookie and never leaks password_hash; duplicate
username → 409; weak password → 400; login rotates the cookie; wrong
password / unknown user → 401; /me with vs without cookie; logout
invalidates the cookie; bot-token roundtrip via Bearer; user A cannot
delete user B's token (403); unknown delete → 404.
Frontend:
- lib/api/auth.ts — typed wrappers; me() returns null on 401.
- lib/session.svelte.ts — per-tab user state with a seq counter to
guard against an in-flight /me clobbering a fresh setUser.
- lib/api/client.ts — request<T> returns undefined for 204.
- routes/login + routes/register — forms with action="javascript:void(0)"
so the no-JS path is a no-op (avoids the hydration-race where a
pre-attach click would submit via the browser default).
- routes/+layout.svelte — session-aware nav: spinner → user + Logout,
or Login / Register.
- e2e/auth-flow.spec.ts — login flips the layout, logout flips back;
bad credentials surface the API error message.
Config grows AuthConfig (cookie_secure, cookie_domain, session_ttl_days)
and CORS_ALLOWED_ORIGINS. CORS middleware is mounted in app::build and
stays a no-op (same-origin) until origins are listed.
Lockstep version bump to 0.3.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
