The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR decompression-bomb guard added the env var to .env.example and
config.rs but not docker-compose.yml, so the container never received it
and the compose-env-coverage test failed once the branches were integrated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.
Make .env.example the single source of truth:
* docker_compose_wires_every_documented_backend_env_var now reads
.env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
(compose-side composed vars, frontend-only vars, sidecar vars),
and requires every remaining key to be wired into compose with a
matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
* every_backend_src_env_read_is_documented_or_allowlisted scans
backend/src/ for env::var / env_bool / env_i64 / ... reads and
requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
with a per-entry rationale. Catches the silent-no-op bug from the
other side.
Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.
Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.1 added 12+ env vars to `.env.example` but the compose backend
`environment:` block forwarded none of them — `.env` interpolation
doesn't pass vars to the container. Operators got
anonymous-serving sites, silent ADMIN_* no-ops, and missing
ANALYSIS_* configuration.
Wire 36 backend-consumed vars; also substitute the hardcoded
`BACKEND_URL` on the frontend; emit a startup warning on half-set
`ADMIN_USERNAME`/`ADMIN_PASSWORD`. Regression-test parses
docker-compose.yml and asserts the wiring.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three operational gaps tripped a recent review:
1. **Postgres had no `restart:` policy.** tor / docker-socket-proxy /
vision-manager all set `unless-stopped`; pg was the lone exception.
A transient OOM or panic left backend's `depends_on: service_healthy`
blocking every future restart and took the stack offline. Add
`unless-stopped` on postgres and on backend/frontend in the base
compose so the prod overlay is the same shape.
2. **`BACKEND_PROXY_TIMEOUT_MS` was documented but silently a no-op.**
`.env.example` carried it but `docker-compose.yml` never forwarded it
into the frontend service env, so the value never reached
`hooks.server.ts`. Wire it through.
3. **`.env.example` omitted required env vars.** Operators copying the
template got no admin bootstrap, no boot-seed crawler URL, no
analysis configuration. Add: `ADMIN_USERNAME`, `ADMIN_PASSWORD`,
`PRIVATE_MODE`, `ALLOW_SELF_REGISTER`, `CRAWLER_START_URL`,
`CRAWLER_CDN_HOST`, `ANALYSIS_ENABLED`, `ANALYSIS_VISION_URL`,
`ANALYSIS_VISION_MODEL`, `ANALYSIS_WORKERS`,
`ANALYSIS_JOB_TIMEOUT_SECS`, `ANALYSIS_API_KEY` — each with the
comment block that explains seed-vs-runtime semantics.
Also adds `frontend/vite.config.ts.timestamp-*.mjs` to `.gitignore` — a
crashed dev server leaves these transient files behind otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Log on readiness state transitions (ready<->not-ready) so a misconfigured
ANALYSIS_VISION_HEALTH_URL, which otherwise parks the worker silently
forever, is diagnosable.
- Add unit tests for HttpVisionReadiness: 2xx->ready, non-2xx->not-ready,
connection error->not-ready (the production status mapping had no test).
vision-manager:
- Guard the backlog count against non-numeric output before `-gt`, so a stray
value can't exit-2 and kill the loop under `set -e`.
- Throttle the crawl-mutex "deferring start" log to once per episode.
- Only reset the idle/uptime timers when `docker stop` actually succeeds, so a
failed stop retries next tick instead of waiting a full debounce window.
- Decouple the per-probe curl timeout (HEALTH_TIMEOUT) from the warm-up poll
cadence.
Docs:
- Correct the docker-socket-proxy comment: CONTAINERS+POST permits the full
container lifecycle (not just start/stop); state the real trust boundary.
- Document that the externally-defined mangalord-vision container must share
the compose network for name resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a tiny privileged sidecar (vision-manager/) that polls the analysis
backlog in Postgres and starts/stops the mangalord-vision container by name:
start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle.
It is the single owner of the vision lifecycle and the only component with
Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST
only) on an internal-only network, so the internet-facing backend never
touches the socket.
Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is
unaffected). The manager debounces the stop, gates the first request on
GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB
box, and owns its own idle timer (leak-safe if the backend dies). Backlog
query uses the real schema (state + payload->>'kind'); a read-only DB role
(readonly-role.sql) keeps it off the backend creds.
Bump 0.80.0 -> 0.81.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dockurr/tor's stock entrypoint binds the control port to localhost
(unreachable from a sibling container), refuses to run as a
non-default user (its setup chowns dirs and su-execs down to its
`tor` user, both requiring root), and skips its own
HashedControlPassword injection whenever the user's torrc declares
a ControlPort. The combination meant the original cookie-via-shared-
volume design couldn't work without fighting the image.
This commit:
- Adds tor/entrypoint.sh, a small wrapper that hashes $PASSWORD
with `tor --hash-password`, appends the hash to a writable copy
of /etc/tor/torrc, then execs tor. Container runs as root only
for that bring-up; the torrc's `User tor` directive drops privs
after port binding.
- Adds a healthcheck on the tor service that gates downstream
containers on both 9050 + 9051 actually listening (was
service_started, which fires before tor finishes bootstrap).
- Loosens MaxCircuitDirtiness 60 → 600. The 60s value would have
rotated mid-chapter for any chapter with > ~50 images, which is
exactly the kind of fingerprint we're trying to avoid.
- Wires TOR_CONTROL_PASSWORD as a REQUIRED .env var on both sides
(PASSWORD on tor, CRAWLER_TOR_CONTROL_PASSWORD on backend).
docker-compose.yml fails fast if unset.
- Removes the tor-data shared volume on backend (cookie auth is no
longer the default; operators wanting cookie can mount it back).
- Documents the pivot + the cookie-vs-password tradeoff in
.env.example.
End-to-end validated: `docker compose up -d tor`, then
`printf 'AUTHENTICATE "test"\r\nSIGNAL NEWNYM\r\nQUIT\r\n' | nc tor 9051`
returns three `250 OK` lines.
Audit ref: #2, #3, #6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `tor` service to the compose stack (dockurr/tor) with a torrc
tuned for the crawler — SOCKS5 on 9050 with IsolateDestAddr +
IsolateDestPort so NEWNYM picks up promptly, control port on 9051
with cookie auth, MaxCircuitDirtiness 60.
Backend defaults CRAWLER_PROXY → socks5h://tor:9050 and
CRAWLER_TOR_CONTROL_URL → tcp://tor:9051 so TOR + recircuit are on
out-of-the-box. Operators can override both to empty in .env to opt
out without removing the service.
The tor-data named volume is mounted ro on the backend so it can read
/var/lib/tor/control_auth_cookie; CookieAuthFileGroupReadable handles
the permissions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- .gitea/workflows/deploy.yml: trigger on pull_request to main so PRs
get test feedback; gate build-and-push + deploy on push events so
PRs only run the test jobs (no registry push, no SSH deploy).
- docker-compose.yml: change `${POSTGRES_PASSWORD:-mangalord}` to
`${POSTGRES_PASSWORD:?...}` so a deploy without an .env fails fast
instead of booting Postgres with a known-default credential.
- .env.example: change the example value to a "change-me" sentinel,
add a banner explaining that production needs HTTPS in front of
the frontend container because COOKIE_SECURE=true makes browsers
refuse cookies over plain HTTP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose deploy was unreachable because frontend code reads its
API base from `import.meta.env.VITE_API_BASE` at build time, but the
shipped image baked in the fallback `/api` and never picked up the
`PUBLIC_API_BASE` env var. The browser then hit
http://localhost:3000/api/...which the Node adapter doesn't serve, so
every request 404'd.
Fix the topology at the right layer: hooks.server.ts proxies /api/*
requests through to the backend container over docker's internal
network. The browser only ever talks to :3000, cookies stay
same-origin, and CORS can stay empty.
- frontend/src/hooks.server.ts: new proxy. Reads BACKEND_URL (defaults
to http://localhost:8080 for ad-hoc node builds). Strips `host` and
`content-length` so the backend sees the real client request and
recomputes the length. Sets `duplex: 'half'` for streamed POST
bodies. GET/HEAD have no body. Non-/api paths fall through to
SvelteKit normally.
- docker-compose.yml: drop the host port mapping on the backend
(browser doesn't reach it directly anymore — use `ports:` instead of
`expose:` if you want curl access). Set BACKEND_URL=http://backend:8080
on the frontend service. Drop PUBLIC_API_BASE which was unused.
- .env.example: replace PUBLIC_API_BASE with BACKEND_URL, with a note
on what it does.
- README: explain the new topology in Quick start, update the bot
curl examples to hit :3000 (since that's the only published port in
the default deploy), and call out that the TLS terminator only needs
one upstream now.
Lockstep version bump to 0.9.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README rewritten end-to-end: stack, quick start, dev workflow, full
/api/v1 endpoint table, error and pagination envelopes, auth
quick-start (browser + bot bearer), configuration table, deployment
notes, backup/restore pointer. Stale "next features" section dropped
now that all eight feat branches are in.
- .env.example now lists every env var the backend reads, with
inline explanations:
- COOKIE_SECURE / COOKIE_DOMAIN / SESSION_TTL_DAYS (auth)
- CORS_ALLOWED_ORIGINS (same-origin by default)
- MAX_REQUEST_BYTES / MAX_FILE_BYTES (upload caps)
- Postgres + storage + log vars carried over.
- docker-compose.yml forwards all of the above into the backend
service with `${VAR:-default}` so an unset value falls back to the
same default the code uses, and any `.env` override flows through
without a compose edit.
- docs/backup.md: step-by-step backup, restore, and smoke-test drill
for both stateful volumes (postgres-data + storage-data), plus a
list of what's deliberately *not* in the backup (e.g., .env).
- playwright.config.ts: pins the e2e dev server to port 5174 with
`--strictPort` so it neither reuses nor silently bumps off
collision with another vite instance on 5173. Drops the flaky
manual-start workflow the earlier branches needed.
- docker-compose syntax (both prod and dev) validates cleanly against
.env.example with no undefined-variable warnings.
No version bump — this is documentation, config, and tooling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
