Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lets operators run a closed-membership deployment by setting
ALLOW_SELF_REGISTER=false (default true, so existing deploys are
unaffected). When off, POST /auth/register returns 403 forbidden. The
rate-limit token is consumed BEFORE the disabled check so the timing
doesn't distinguish enabled-but-rejected from disabled — closes the
toggle-state probe channel.
New public GET /auth/config returns { self_register_enabled: bool }
so the frontend can render its register affordances correctly
without conflating "disabled" with "rate-limited" (which a probe
attempt would).
Frontend: a lightweight reactive `authConfig` store loads the flag
once on root-layout mount (and again on /register direct navigation,
which bypasses the layout's onMount). Header hides the Register link
when the toggle is off; /register renders a "self-registration is
disabled — ask an administrator" notice instead of the form.
Admin-create endpoint that pairs with this toggle is intentionally
not in this PR — it lands as the next branch (feat/admin-user-create).
The toggle alone is independently useful for deployments that want
to lock down enrollment without yet wiring an admin UI.
Adds an `is_admin` flag on users plus the substrate every later PR in the
admin feature builds on:
- migration 0018 adds the column with default false
- `repo::user::bootstrap_admin` creates or promotes the user named by
`ADMIN_USERNAME` at startup, hashing `ADMIN_PASSWORD` only when the row
is new — never overwriting an existing hash, so an operator can rotate
the admin password via the UI without env-var conflict
- `CurrentSessionUser` extractor accepts only the session cookie;
`RequireAdmin` composes over it and additionally requires
`user.is_admin`. Bearer tokens are intentionally excluded so an
admin's bot token never inherits admin authority (privilege-escalation
surface that bites every "API keys reuse user perms" auth design)
- demotion is instant: `RequireAdmin` re-reads the user row each request
`/api/v1/auth/me` now exposes `is_admin`; no other response embeds
`User`, so no privacy fanout to audit.
Collapses the crawler to a single newest-first walker and replaces the
N-consecutive-unchanged streak with a per-manga rule: stop on the first
manga where metadata is Unchanged AND chapter sync reports zero new
chapters. The early stop is gated by a per-source recovery flag stored
in `crawler_state` — set to `false` when a run starts, back to `true`
only on a clean exit (end-of-walk or intentional stop). A crashed run
leaves the flag `false` automatically (no shutdown code runs), so the
next tick walks the full catalog instead of bailing at the first
caught-up manga.
This means a crashed mid-walk run self-heals on the next tick: the
flag stays `false`, the next walk visits every page (recovering
anything the crash missed past its crash point), and steady state
resumes once the recovery sweep reaches end-of-walk.
Removed:
- DiscoverMode enum, Backfill mode, the boundary re-check +
displaced-refs machinery in TargetSourceWalker.
- Drop-pass (mark_dropped_mangas) and seed-completion plumbing
(mark_seed_completed / seed_completed_at). The recovery flag
subsumes the seed-completion signal; drop detection was explicitly
opted out.
- JobPayload::Discover (no production callers).
- CRAWLER_MODE / CRAWLER_INCREMENTAL_STOP_AFTER env vars and the
CrawlerModePref config type.
`should_mark_clean_exit(walked_to_completion, hit_stop_condition)`
encodes the clean-exit truth table in its signature — `hit_limit` is
deliberately absent so a future edit cannot accidentally count a
caller-imposed cap as a clean exit.
Net -501 lines, 261 backend tests passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three features bundled into one release:
- rate-limit /auth/login, /register, /me/password (token bucket,
5 req/sec sustained with 10-request burst by default; 429 +
Retry-After header on hit; tracing::warn! per hit so operators
see attack patterns; AUTH_RATE_PER_SEC / AUTH_RATE_BURST env knobs)
- handle SIGTERM for graceful container stops (replaces bare
ctrl_c() with a select over ctrl_c + SignalKind::terminate() so
docker compose stop runs the daemon shutdown path instead of
letting Chromium leak past SIGKILL)
- clear session.user on 401 from any API call (setOn401Hook in
api/client.ts, registered from session.svelte.ts gated on
$app/environment::browser so the SSR bundle never installs it;
fixes "logged in but no bookmarks/collections" mid-session
expiry state)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Five fixes bundled into one release:
- preserve user-attached tags across crawler upserts
(repo::crawler::sync_tags now scopes to added_by IS NULL; orphaned
attachments from deleted users are reaped as crawler-owned)
- gate manga PATCH and cover endpoints on uploaded_by (require_can_edit
in api::mangas; non-NULL uploaded_by must match the caller)
- equalise login response time across user-existence branches
(run argon2 against a OnceLock-cached dummy hash on the no-user
branch so timing doesn't leak username existence)
- crawler download defences (SSRF allowlist of host literals
including IPv4-mapped IPv6 ranges, 32 MiB streamed size cap,
reject non-whitelisted image types, three-way chapter-probe
classifier replaces the binary #avatar_menu check)
- tighten validation and clean up dead unload path
(attach_tag + create_token enforce 64-char caps; LocalStorage
rejects NUL bytes explicitly; reader flushFinalProgress drops
the always-405 sendBeacon path)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Daemon now auto-detects mode per source: Backfill until the first
full walk records `seed_completed:<source>` in `crawler_state`, then
Incremental (newest-first, stops after N consecutive Unchanged
upserts). `CRAWLER_MODE` overrides to a fixed mode; CLI rejects
`auto` since it has no pre-run DB state.
`Source::discover` returns a lazy `DiscoverWalk` so Incremental can
break out mid-walk without prefetching pages. The drop pass and seed
marker are now gated on a true full walk — fixes a latent soft-drop
of the index tail under partial sweeps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The backend now boots an internal crawler daemon that runs a daily
metadata pass (CRAWLER_DAILY_AT in CRAWLER_TZ, advisory-lock guarded
for multi-replica safety) and drains SyncChapterContent jobs from
crawler_jobs through a worker pool. Chromium launches lazily on first
job and is torn down after CRAWLER_IDLE_TIMEOUT_S seconds of inactivity.
Modules:
- crawler::browser_manager — lazy-launch / idle-teardown wrapper
around browser::Handle, with an on_launch hook that re-injects
PHPSESSID on every fresh Chromium spawn.
- crawler::pipeline — run_metadata_pass (the shared discover/upsert
/cover/sync-chapters loop) and the enqueue_bookmarked_pending helper
used by the cron tick.
- crawler::daemon — cron task + worker pool, behind two trait seams
(MetadataPass, ChapterDispatcher) so tests can inject stubs without
standing up Chromium or a live source.
Behavior:
- CRAWLER_DAEMON=false skips daemon spawn entirely (default for tests).
- Catch-up tick fires on startup if the last persisted slot was missed.
- A SyncOutcome::SessionExpired sets a sticky AtomicBool; workers
idle until operator restart with a refreshed PHPSESSID.
- Worker dispatch wrapped in catch_unwind so a panicking handler
marks the job failed instead of taking down the worker.
- Migration 0015 adds a small crawler_state k-v table for the
last_metadata_tick_at watermark.
Dep additions: chrono-tz (IANA TZ parsing).
CLI (bin/crawler) reuses pipeline::run_metadata_pass and now holds
the browser via BrowserManager so the on_launch session injection
flow stays in one place. Inline chapter-content sync semantics are
unchanged — the queue is for the daemon, force-refetches and manual
backfills still bypass it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
POST /api/v1/mangas and POST /api/v1/mangas/{id}/chapters now accept
multipart/form-data, gated by CurrentUser:
- /mangas: required `metadata` part (NewManga JSON) + optional `cover`
image part.
- /mangas/{id}/chapters: required `metadata` (NewChapter JSON) + one or
more `page` parts ordered by arrival. Returns 404 if the parent manga
doesn't exist, 409 on duplicate (manga_id, number).
MIME is sniffed via the `infer` crate (magic bytes), not the
client-supplied filename or Content-Type. Whitelist:
jpeg / png / webp / gif / avif. Anything else → 415
unsupported_media_type. The stored key's extension is derived from the
sniffed type so a "page1.png" that's actually a JPEG lands as `.jpg`.
Size cap is two-layer:
- Request body cap (config.max_request_bytes, default 200 MiB) enforced
by axum's DefaultBodyLimit before the handler sees the request.
- Per-image-part cap (config.max_file_bytes, default 20 MiB) enforced
after reading the part, so a single oversized image can't pass even
if the total request fits.
Storage keys follow the layout documented in CLAUDE.md:
- mangas/{manga_id}/cover.{ext}
- mangas/{manga_id}/chapters/{chapter_id}/pages/{nnnn}.{ext} (1-indexed).
AppError grows PayloadTooLarge/UnsupportedMediaType/ValidationFailed
(413 / 415 / 422). ValidationFailed carries a `details` JSON object the
client can use to highlight bad fields (e.g. {"title":"required"}).
Top-level matching in code() stays exhaustive.
Backend coverage in tests/api_uploads.rs (10 cases):
- create_manga_with_cover_stores_image — file is reachable via
/api/v1/files/{key} with the right Content-Type.
- create_manga_without_cover_leaves_path_null.
- create_manga_rejects_non_image_cover_with_415 — PDF claimed as png.
- create_manga_rejects_oversized_cover_with_413.
- create_chapter_with_pages_stores_each — extension derived from
sniffed MIME, files reachable in arrival order.
- create_chapter_rejects_when_no_pages_with_422 — details.page set.
- create_chapter_rejects_renamed_non_image_page → 415.
- create_chapter_returns_409_on_duplicate_number.
- create_chapter_requires_authentication → 401.
- create_chapter_under_unknown_manga_is_404.
Existing tests/api_mangas.rs is migrated to multipart; the create
response is now 201 Created. tests/common::MultipartBuilder builds the
body by hand so the test crate stays free of HTTP-client deps.
Frontend lib/api/mangas.ts: createManga now sends FormData (metadata +
optional cover Blob). Browser fills in the boundary header automatically.
Vitest asserts the FormData structure via FileReader (jsdom doesn't
implement Blob.text()).
E2E tests wait for the post-hydration nav-login link before
interacting with the login form, fixing a flake where pre-hydration
clicks would submit via the browser default and bypass our handler.
Lockstep version bump to 0.5.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full auth flow. Reads stay public; writes (currently only POST
/api/v1/mangas) require a CurrentUser. Both browsers and bot scripts hit
the same endpoints — they just present credentials differently.
Migration 0002_auth.sql introduces users.password_hash, a sessions
table, and an api_tokens table. Sessions and api_tokens store only
sha256(raw_token) — the raw value lives in the cookie or the
Authorization header.
New endpoints under /api/v1/auth/:
- POST /register — argon2id hash, creates a session, sets cookie.
- POST /login — verifies, rotates to a fresh session (old ones expire
naturally so other devices stay signed in).
- POST /logout — deletes the server-side session row + clears the
cookie via Max-Age=0.
- GET /me — current user via the new CurrentUser extractor.
- POST /tokens — issue a bot bearer token; raw value returned exactly
once at creation.
- DELETE /tokens/{id} — owner-only: 404 if unknown, 403 if it exists
but belongs to another user, 204 on success.
The CurrentUser axum extractor resolves cookie first, then
Authorization: Bearer; failure → AppError::Unauthenticated (401). New
AppError variants Unauthenticated/Forbidden/Conflict carry the matching
envelope codes; the top-level match in `code()` stays exhaustive.
Backend integration coverage in tests/api_auth.rs: register sets a
HttpOnly SameSite=Lax cookie and never leaks password_hash; duplicate
username → 409; weak password → 400; login rotates the cookie; wrong
password / unknown user → 401; /me with vs without cookie; logout
invalidates the cookie; bot-token roundtrip via Bearer; user A cannot
delete user B's token (403); unknown delete → 404.
Frontend:
- lib/api/auth.ts — typed wrappers; me() returns null on 401.
- lib/session.svelte.ts — per-tab user state with a seq counter to
guard against an in-flight /me clobbering a fresh setUser.
- lib/api/client.ts — request<T> returns undefined for 204.
- routes/login + routes/register — forms with action="javascript:void(0)"
so the no-JS path is a no-op (avoids the hydration-race where a
pre-attach click would submit via the browser default).
- routes/+layout.svelte — session-aware nav: spinner → user + Logout,
or Login / Register.
- e2e/auth-flow.spec.ts — login flips the layout, logout flips back;
bad credentials surface the API error message.
Config grows AuthConfig (cookie_secure, cookie_domain, session_ttl_days)
and CORS_ALLOWED_ORIGINS. CORS middleware is mounted in app::build and
stays a no-op (same-origin) until origins are listed.
Lockstep version bump to 0.3.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
