Add a CPU-only, English-first OCR engine (the `ocrs` crate on the `rten`
runtime) as an alternative to the local-LLM "vision" backend, which ran
too slowly on the target Raspberry Pi 5.
- `ANALYSIS_BACKEND` (default `ocr`) selects the engine; `vision` keeps the
existing LLM path. Deploy-time, env-only — not admin-tunable.
- `analysis::ocr` adds an `OcrEngine` trait (testable seam), the `OcrsEngine`
impl (models loaded once, inference on the blocking pool), and an
`OcrAnalyzeDispatcher` that plugs into the existing `AnalyzeDispatcher`
seam and reuses `persist_analysis`, so OCR text lands in `page_ocr_text`
and the `search_doc` tsvector exactly as the vision path produces them.
- The backend image bakes the two `.rten` models into /models, where
`OCRS_DETECTION_MODEL` / `OCRS_RECOGNITION_MODEL` default.
Because `/v1/me/page-search` ranks on `search_doc`, OCR text search works
the moment pages are processed. Auto-tagging, scene description and NSFW
flags remain the vision backend's job (deferred).
Bumps to 0.90.0 (minor) in both manifests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.
Make .env.example the single source of truth:
* docker_compose_wires_every_documented_backend_env_var now reads
.env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
(compose-side composed vars, frontend-only vars, sidecar vars),
and requires every remaining key to be wired into compose with a
matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
* every_backend_src_env_read_is_documented_or_allowlisted scans
backend/src/ for env::var / env_bool / env_i64 / ... reads and
requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
with a per-entry rationale. Catches the silent-no-op bug from the
other side.
Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.
Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.1 added 12+ env vars to `.env.example` but the compose backend
`environment:` block forwarded none of them — `.env` interpolation
doesn't pass vars to the container. Operators got
anonymous-serving sites, silent ADMIN_* no-ops, and missing
ANALYSIS_* configuration.
Wire 36 backend-consumed vars; also substitute the hardcoded
`BACKEND_URL` on the frontend; emit a startup warning on half-set
`ADMIN_USERNAME`/`ADMIN_PASSWORD`. Regression-test parses
docker-compose.yml and asserts the wiring.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three operational gaps tripped a recent review:
1. **Postgres had no `restart:` policy.** tor / docker-socket-proxy /
vision-manager all set `unless-stopped`; pg was the lone exception.
A transient OOM or panic left backend's `depends_on: service_healthy`
blocking every future restart and took the stack offline. Add
`unless-stopped` on postgres and on backend/frontend in the base
compose so the prod overlay is the same shape.
2. **`BACKEND_PROXY_TIMEOUT_MS` was documented but silently a no-op.**
`.env.example` carried it but `docker-compose.yml` never forwarded it
into the frontend service env, so the value never reached
`hooks.server.ts`. Wire it through.
3. **`.env.example` omitted required env vars.** Operators copying the
template got no admin bootstrap, no boot-seed crawler URL, no
analysis configuration. Add: `ADMIN_USERNAME`, `ADMIN_PASSWORD`,
`PRIVATE_MODE`, `ALLOW_SELF_REGISTER`, `CRAWLER_START_URL`,
`CRAWLER_CDN_HOST`, `ANALYSIS_ENABLED`, `ANALYSIS_VISION_URL`,
`ANALYSIS_VISION_MODEL`, `ANALYSIS_WORKERS`,
`ANALYSIS_JOB_TIMEOUT_SECS`, `ANALYSIS_API_KEY` — each with the
comment block that explains seed-vs-runtime semantics.
Also adds `frontend/vite.config.ts.timestamp-*.mjs` to `.gitignore` — a
crashed dev server leaves these transient files behind otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Log on readiness state transitions (ready<->not-ready) so a misconfigured
ANALYSIS_VISION_HEALTH_URL, which otherwise parks the worker silently
forever, is diagnosable.
- Add unit tests for HttpVisionReadiness: 2xx->ready, non-2xx->not-ready,
connection error->not-ready (the production status mapping had no test).
vision-manager:
- Guard the backlog count against non-numeric output before `-gt`, so a stray
value can't exit-2 and kill the loop under `set -e`.
- Throttle the crawl-mutex "deferring start" log to once per episode.
- Only reset the idle/uptime timers when `docker stop` actually succeeds, so a
failed stop retries next tick instead of waiting a full debounce window.
- Decouple the per-probe curl timeout (HEALTH_TIMEOUT) from the warm-up poll
cadence.
Docs:
- Correct the docker-socket-proxy comment: CONTAINERS+POST permits the full
container lifecycle (not just start/stop); state the real trust boundary.
- Document that the externally-defined mangalord-vision container must share
the compose network for name resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a tiny privileged sidecar (vision-manager/) that polls the analysis
backlog in Postgres and starts/stops the mangalord-vision container by name:
start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle.
It is the single owner of the vision lifecycle and the only component with
Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST
only) on an internal-only network, so the internet-facing backend never
touches the socket.
Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is
unaffected). The manager debounces the stop, gates the first request on
GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB
box, and owns its own idle timer (leak-safe if the backend dies). Backlog
query uses the real schema (state + payload->>'kind'); a read-only DB role
(readonly-role.sql) keeps it off the backend creds.
Bump 0.80.0 -> 0.81.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dockurr/tor's stock entrypoint binds the control port to localhost
(unreachable from a sibling container), refuses to run as a
non-default user (its setup chowns dirs and su-execs down to its
`tor` user, both requiring root), and skips its own
HashedControlPassword injection whenever the user's torrc declares
a ControlPort. The combination meant the original cookie-via-shared-
volume design couldn't work without fighting the image.
This commit:
- Adds tor/entrypoint.sh, a small wrapper that hashes $PASSWORD
with `tor --hash-password`, appends the hash to a writable copy
of /etc/tor/torrc, then execs tor. Container runs as root only
for that bring-up; the torrc's `User tor` directive drops privs
after port binding.
- Adds a healthcheck on the tor service that gates downstream
containers on both 9050 + 9051 actually listening (was
service_started, which fires before tor finishes bootstrap).
- Loosens MaxCircuitDirtiness 60 → 600. The 60s value would have
rotated mid-chapter for any chapter with > ~50 images, which is
exactly the kind of fingerprint we're trying to avoid.
- Wires TOR_CONTROL_PASSWORD as a REQUIRED .env var on both sides
(PASSWORD on tor, CRAWLER_TOR_CONTROL_PASSWORD on backend).
docker-compose.yml fails fast if unset.
- Removes the tor-data shared volume on backend (cookie auth is no
longer the default; operators wanting cookie can mount it back).
- Documents the pivot + the cookie-vs-password tradeoff in
.env.example.
End-to-end validated: `docker compose up -d tor`, then
`printf 'AUTHENTICATE "test"\r\nSIGNAL NEWNYM\r\nQUIT\r\n' | nc tor 9051`
returns three `250 OK` lines.
Audit ref: #2, #3, #6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a `tor` service to the compose stack (dockurr/tor) with a torrc
tuned for the crawler — SOCKS5 on 9050 with IsolateDestAddr +
IsolateDestPort so NEWNYM picks up promptly, control port on 9051
with cookie auth, MaxCircuitDirtiness 60.
Backend defaults CRAWLER_PROXY → socks5h://tor:9050 and
CRAWLER_TOR_CONTROL_URL → tcp://tor:9051 so TOR + recircuit are on
out-of-the-box. Operators can override both to empty in .env to opt
out without removing the service.
The tor-data named volume is mounted ro on the backend so it can read
/var/lib/tor/control_auth_cookie; CookieAuthFileGroupReadable handles
the permissions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- .gitea/workflows/deploy.yml: trigger on pull_request to main so PRs
get test feedback; gate build-and-push + deploy on push events so
PRs only run the test jobs (no registry push, no SSH deploy).
- docker-compose.yml: change `${POSTGRES_PASSWORD:-mangalord}` to
`${POSTGRES_PASSWORD:?...}` so a deploy without an .env fails fast
instead of booting Postgres with a known-default credential.
- .env.example: change the example value to a "change-me" sentinel,
add a banner explaining that production needs HTTPS in front of
the frontend container because COOKIE_SECURE=true makes browsers
refuse cookies over plain HTTP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose deploy was unreachable because frontend code reads its
API base from `import.meta.env.VITE_API_BASE` at build time, but the
shipped image baked in the fallback `/api` and never picked up the
`PUBLIC_API_BASE` env var. The browser then hit
http://localhost:3000/api/...which the Node adapter doesn't serve, so
every request 404'd.
Fix the topology at the right layer: hooks.server.ts proxies /api/*
requests through to the backend container over docker's internal
network. The browser only ever talks to :3000, cookies stay
same-origin, and CORS can stay empty.
- frontend/src/hooks.server.ts: new proxy. Reads BACKEND_URL (defaults
to http://localhost:8080 for ad-hoc node builds). Strips `host` and
`content-length` so the backend sees the real client request and
recomputes the length. Sets `duplex: 'half'` for streamed POST
bodies. GET/HEAD have no body. Non-/api paths fall through to
SvelteKit normally.
- docker-compose.yml: drop the host port mapping on the backend
(browser doesn't reach it directly anymore — use `ports:` instead of
`expose:` if you want curl access). Set BACKEND_URL=http://backend:8080
on the frontend service. Drop PUBLIC_API_BASE which was unused.
- .env.example: replace PUBLIC_API_BASE with BACKEND_URL, with a note
on what it does.
- README: explain the new topology in Quick start, update the bot
curl examples to hit :3000 (since that's the only published port in
the default deploy), and call out that the TLS terminator only needs
one upstream now.
Lockstep version bump to 0.9.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README rewritten end-to-end: stack, quick start, dev workflow, full
/api/v1 endpoint table, error and pagination envelopes, auth
quick-start (browser + bot bearer), configuration table, deployment
notes, backup/restore pointer. Stale "next features" section dropped
now that all eight feat branches are in.
- .env.example now lists every env var the backend reads, with
inline explanations:
- COOKIE_SECURE / COOKIE_DOMAIN / SESSION_TTL_DAYS (auth)
- CORS_ALLOWED_ORIGINS (same-origin by default)
- MAX_REQUEST_BYTES / MAX_FILE_BYTES (upload caps)
- Postgres + storage + log vars carried over.
- docker-compose.yml forwards all of the above into the backend
service with `${VAR:-default}` so an unset value falls back to the
same default the code uses, and any `.env` override flows through
without a compose edit.
- docs/backup.md: step-by-step backup, restore, and smoke-test drill
for both stateful volumes (postgres-data + storage-data), plus a
list of what's deliberately *not* in the backup (e.g., .env).
- playwright.config.ts: pins the e2e dev server to port 5174 with
`--strictPort` so it neither reuses nor silently bumps off
collision with another vite instance on 5173. Drops the flaky
manual-start workflow the earlier branches needed.
- docker-compose syntax (both prod and dev) validates cleanly against
.env.example with no undefined-variable warnings.
No version bump — this is documentation, config, and tooling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Set up Mangalord with a Rust/axum backend, SvelteKit frontend, Postgres,
and Docker Compose deployment. Establishes the architecture and TDD
patterns the project will extend:
- Hexagonal-ish backend layering (domain / repo / storage / api) with
a pluggable Storage trait (LocalStorage today, S3 as a future impl).
- Initial migration: users, mangas, chapters, bookmarks.
- Vertical slice for mangas (list, search, create, get) with
#[sqlx::test] integration coverage and storage unit tests.
- SvelteKit frontend using Svelte 5 runes, typed API client, Vitest
unit tests and Playwright e2e with route mocking.
- CLAUDE.md documenting layering, TDD/git/SemVer workflow rules, and
extension points (tags, fulltext search, OCR, S3, auth).
- Project-scoped .claude/settings.json with permission allowlist for
the toolchain (git, cargo, npm/vite, docker, psql, gh, doc fetches).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
