Add a tiny privileged sidecar (vision-manager/) that polls the analysis
backlog in Postgres and starts/stops the mangalord-vision container by name:
start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle.
It is the single owner of the vision lifecycle and the only component with
Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST
only) on an internal-only network, so the internet-facing backend never
touches the socket.
Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is
unaffected). The manager debounces the stop, gates the first request on
GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB
box, and owns its own idle timer (leak-safe if the backend dies). Backlog
query uses the real schema (state + payload->>'kind'); a read-only DB role
(readonly-role.sql) keeps it off the backend creds.
Bump 0.80.0 -> 0.81.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CLI binary already capped runs at CRAWLER_LIMIT mangas, but the
daemon's RealMetadataPass passed a hardcoded `0` (no cap) to
`pipeline::run_metadata_pass`, so the env var was silently ignored once
the daemon took over the metadata pass.
Adds `manga_limit` to `CrawlerConfig`, reads it from `CRAWLER_LIMIT`
(default 0 = no cap), and threads it through `RealMetadataPass::run`
so a daemon-driven sweep stops at the same boundary as a CLI run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dockurr/tor's stock entrypoint binds the control port to localhost
(unreachable from a sibling container), refuses to run as a
non-default user (its setup chowns dirs and su-execs down to its
`tor` user, both requiring root), and skips its own
HashedControlPassword injection whenever the user's torrc declares
a ControlPort. The combination meant the original cookie-via-shared-
volume design couldn't work without fighting the image.
This commit:
- Adds tor/entrypoint.sh, a small wrapper that hashes $PASSWORD
with `tor --hash-password`, appends the hash to a writable copy
of /etc/tor/torrc, then execs tor. Container runs as root only
for that bring-up; the torrc's `User tor` directive drops privs
after port binding.
- Adds a healthcheck on the tor service that gates downstream
containers on both 9050 + 9051 actually listening (was
service_started, which fires before tor finishes bootstrap).
- Loosens MaxCircuitDirtiness 60 → 600. The 60s value would have
rotated mid-chapter for any chapter with > ~50 images, which is
exactly the kind of fingerprint we're trying to avoid.
- Wires TOR_CONTROL_PASSWORD as a REQUIRED .env var on both sides
(PASSWORD on tor, CRAWLER_TOR_CONTROL_PASSWORD on backend).
docker-compose.yml fails fast if unset.
- Removes the tor-data shared volume on backend (cookie auth is no
longer the default; operators wanting cookie can mount it back).
- Documents the pivot + the cookie-vs-password tradeoff in
.env.example.
End-to-end validated: `docker compose up -d tor`, then
`printf 'AUTHENTICATE "test"\r\nSIGNAL NEWNYM\r\nQUIT\r\n' | nc tor 9051`
returns three `250 OK` lines.
Audit ref: #2, #3, #6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `tor` service to the compose stack (dockurr/tor) with a torrc
tuned for the crawler — SOCKS5 on 9050 with IsolateDestAddr +
IsolateDestPort so NEWNYM picks up promptly, control port on 9051
with cookie auth, MaxCircuitDirtiness 60.
Backend defaults CRAWLER_PROXY → socks5h://tor:9050 and
CRAWLER_TOR_CONTROL_URL → tcp://tor:9051 so TOR + recircuit are on
out-of-the-box. Operators can override both to empty in .env to opt
out without removing the service.
The tor-data named volume is mounted ro on the backend so it can read
/var/lib/tor/control_auth_cookie; CookieAuthFileGroupReadable handles
the permissions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Handle::close aborts its chromiumoxide driver task when another
Arc<Browser> outlives the call, so shutdown returns instead of
hanging on a stream that never terminates. Generic close_or_abort
helper with regression tests covering both Arc paths.
- daemon.shutdown() is wrapped in a 5s timeout in main as defense
in depth.
- Default RUST_LOG silences chromiumoxide::conn / chromiumoxide::handler
WS-deserialize ERROR spam.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three features bundled into one release:
- rate-limit /auth/login, /register, /me/password (token bucket,
5 req/sec sustained with 10-request burst by default; 429 +
Retry-After header on hit; tracing::warn! per hit so operators
see attack patterns; AUTH_RATE_PER_SEC / AUTH_RATE_BURST env knobs)
- handle SIGTERM for graceful container stops (replaces bare
ctrl_c() with a select over ctrl_c + SignalKind::terminate() so
docker compose stop runs the daemon shutdown path instead of
letting Chromium leak past SIGKILL)
- clear session.user on 401 from any API call (setOn401Hook in
api/client.ts, registered from session.svelte.ts gated on
$app/environment::browser so the SSR bundle never installs it;
fixes "logged in but no bookmarks/collections" mid-session
expiry state)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Five fixes bundled into one release:
- preserve user-attached tags across crawler upserts
(repo::crawler::sync_tags now scopes to added_by IS NULL; orphaned
attachments from deleted users are reaped as crawler-owned)
- gate manga PATCH and cover endpoints on uploaded_by (require_can_edit
in api::mangas; non-NULL uploaded_by must match the caller)
- equalise login response time across user-existence branches
(run argon2 against a OnceLock-cached dummy hash on the no-user
branch so timing doesn't leak username existence)
- crawler download defences (SSRF allowlist of host literals
including IPv4-mapped IPv6 ranges, 32 MiB streamed size cap,
reject non-whitelisted image types, three-way chapter-probe
classifier replaces the binary #avatar_menu check)
- tighten validation and clean up dead unload path
(attach_tag + create_token enforce 64-char caps; LocalStorage
rejects NUL bytes explicitly; reader flushFinalProgress drops
the always-405 sendBeacon path)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The SvelteKit proxy was only stripping host + content-length; the rest
of RFC 7230 §6.1 (connection, keep-alive, proxy-authenticate,
proxy-authorization, te, trailer, transfer-encoding, upgrade) leaked
through to axum. Axum doesn't emit them so the impact is theoretical,
but the proxy should be RFC-conformant. Also adds an AbortController
with a configurable 60s timeout (BACKEND_PROXY_TIMEOUT_MS) so a
wedged backend can't hang the browser request indefinitely — failures
surface as the standard 502 upstream_unavailable envelope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- .gitea/workflows/deploy.yml: trigger on pull_request to main so PRs
get test feedback; gate build-and-push + deploy on push events so
PRs only run the test jobs (no registry push, no SSH deploy).
- docker-compose.yml: change `${POSTGRES_PASSWORD:-mangalord}` to
`${POSTGRES_PASSWORD:?...}` so a deploy without an .env fails fast
instead of booting Postgres with a known-default credential.
- .env.example: change the example value to a "change-me" sentinel,
add a banner explaining that production needs HTTPS in front of
the frontend container because COOKIE_SECURE=true makes browsers
refuse cookies over plain HTTP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose deploy was unreachable because frontend code reads its
API base from `import.meta.env.VITE_API_BASE` at build time, but the
shipped image baked in the fallback `/api` and never picked up the
`PUBLIC_API_BASE` env var. The browser then hit
http://localhost:3000/api/...which the Node adapter doesn't serve, so
every request 404'd.
Fix the topology at the right layer: hooks.server.ts proxies /api/*
requests through to the backend container over docker's internal
network. The browser only ever talks to :3000, cookies stay
same-origin, and CORS can stay empty.
- frontend/src/hooks.server.ts: new proxy. Reads BACKEND_URL (defaults
to http://localhost:8080 for ad-hoc node builds). Strips `host` and
`content-length` so the backend sees the real client request and
recomputes the length. Sets `duplex: 'half'` for streamed POST
bodies. GET/HEAD have no body. Non-/api paths fall through to
SvelteKit normally.
- docker-compose.yml: drop the host port mapping on the backend
(browser doesn't reach it directly anymore — use `ports:` instead of
`expose:` if you want curl access). Set BACKEND_URL=http://backend:8080
on the frontend service. Drop PUBLIC_API_BASE which was unused.
- .env.example: replace PUBLIC_API_BASE with BACKEND_URL, with a note
on what it does.
- README: explain the new topology in Quick start, update the bot
curl examples to hit :3000 (since that's the only published port in
the default deploy), and call out that the TLS terminator only needs
one upstream now.
Lockstep version bump to 0.9.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README rewritten end-to-end: stack, quick start, dev workflow, full
/api/v1 endpoint table, error and pagination envelopes, auth
quick-start (browser + bot bearer), configuration table, deployment
notes, backup/restore pointer. Stale "next features" section dropped
now that all eight feat branches are in.
- .env.example now lists every env var the backend reads, with
inline explanations:
- COOKIE_SECURE / COOKIE_DOMAIN / SESSION_TTL_DAYS (auth)
- CORS_ALLOWED_ORIGINS (same-origin by default)
- MAX_REQUEST_BYTES / MAX_FILE_BYTES (upload caps)
- Postgres + storage + log vars carried over.
- docker-compose.yml forwards all of the above into the backend
service with `${VAR:-default}` so an unset value falls back to the
same default the code uses, and any `.env` override flows through
without a compose edit.
- docs/backup.md: step-by-step backup, restore, and smoke-test drill
for both stateful volumes (postgres-data + storage-data), plus a
list of what's deliberately *not* in the backup (e.g., .env).
- playwright.config.ts: pins the e2e dev server to port 5174 with
`--strictPort` so it neither reuses nor silently bumps off
collision with another vite instance on 5173. Drops the flaky
manual-start workflow the earlier branches needed.
- docker-compose syntax (both prod and dev) validates cleanly against
.env.example with no undefined-variable warnings.
No version bump — this is documentation, config, and tooling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
