Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The desktop /profile/history page and the mobile Library "History" tab
each hand-rendered reading history with separate markup, which had drifted:
the Library tab was a read-only 2-column list with no date and no
removed-chapter / whole-manga fallbacks, while /profile/history was a
3-column list with a per-row clear button. Extract a shared HistoryList
component so the two can't diverge again.
- New HistoryList.svelte owns row rendering, the optimistic-removal UX
(instant remove, rollback + inline error on failure), and the empty
state. Clear is opt-in via an `onClear` prop so a caller can render
read-only, but both surfaces now pass it.
- The mobile Library History tab gains the clear action, the "Read {date}"
line, and the (chapter removed) / whole-manga fallbacks it was missing.
- Continue label is built in script (was inline) so the " — page N" suffix
keeps its spaces — Svelte trimmed them at the {#if} edge, rendering
"Chapter 3— page 7". Both surfaces now read "Continue Chapter N".
Net -204 lines across the two pages. Uploads parity on the mobile tab is
left as a follow-up (it needs the library loader to fetch uploads).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The desktop/mobile consistency review found primary mobile controls
rendering below the ~44px comfortable touch floor. Address the genuinely
shared primitives in one pass:
- Add a --tap-min: 44px token (documented anchor for the floor).
- tokens.css: text inputs, selects, and submit buttons grow to --tap-min
under the 640px breakpoint (fixes the 36px auth/login form controls).
Scoped to form controls + type=submit so dense icon buttons aren't
inflated.
- SegmentedControl: .seg options reach the floor on mobile (it doubles as
the catalog sort toggle and the Library tab switcher).
- Chip: expand .chip-remove's tappable area to --tap-min via a centered
pseudo-element so the 16px glyph stays compact and tag rows don't grow.
Pure-CSS responsive change — jsdom can't evaluate @media or layout, so the
test pins the stylesheet contract (each shared primitive bumps to --tap-min
inside the mobile breakpoint), guarding against the bump being dropped.
The per-route .icon-btn (copy-pasted across seven files) is left for a
separate refactor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sort the manga catalog by created / updated / title / author with an
independent asc/desc direction control. An omitted `order` defaults per field
(dates newest-first, text A→Z), matching the UI, so a bare `?sort=<field>`
means the same thing in the browser and over the API; `sort=recent` remains a
back-compat alias for `created`.
- Backend: SortField/SortOrder parsed with validation (structured 422 on bad
input), per-field default_order, NULLS LAST only on the nullable author key,
and migration 0033 indexing mangas(updated_at DESC, id) to back the default
sort and its id tie-break.
- Frontend: catalog sort field + direction (SegmentedControl) on desktop and a
mobile bottom sheet; pure helpers in $lib/mangaSort; keyboard-accessible
direction control; visible Direction labels and a sheet "Done" button.
- Tests: backend integration coverage (defaults, alias, invalid input,
ascending-id tie-break), frontend unit + e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.
Make .env.example the single source of truth:
* docker_compose_wires_every_documented_backend_env_var now reads
.env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
(compose-side composed vars, frontend-only vars, sidecar vars),
and requires every remaining key to be wired into compose with a
matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
* every_backend_src_env_read_is_documented_or_allowlisted scans
backend/src/ for env::var / env_bool / env_i64 / ... reads and
requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
with a per-entry rationale. Catches the silent-no-op bug from the
other side.
Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.
Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CancellableLoader.run() previously swallowed AbortError but propagated
every other rejection. A predecessor whose fetch survived its abort
(timing race, or fn that ignores signal) and then 5xx'd would bubble
the error into the caller's catch, clearing the successor's spinner
and writing its `error` field — exactly the flicker the loader was
supposed to prevent.
Strengthen the contract: a call that no longer owns this.current
returns null regardless of how it died. Symmetric with the existing
late-resolve suppression at the success path. The loadCoverage catch
in admin/analysis/+page.svelte now only fires when this call genuinely
failed AND still owns the loader — so clearing the spinner there is
sound.
Three new unit tests pin all three contract corners: a superseded
predecessor's non-Abort rejection resolves to null; a standalone call
with no successor still throws normally; the AbortError swallow on a
still-owning call (synthetic AbortError from fn with no successor or
cancel()) remains protected, since the new supersede check would
otherwise have eaten every existing AbortError-path test and left
that branch as silently-removable dead code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.16 followup. Extract the four debounce-race invariants
(predecessor abort, late-result suppression, AbortError swallow,
ownership-aware finally) from `loadCoverage` into a reusable
`CancellableLoader` helper at $lib/util/cancellable, with 9 unit
tests covering each. Page rewrite uses the helper.
Also corrects: hooks.server.test.ts (added Node-fetch rejection
shape for already-aborted entry; removed redundant tick); admin.test.ts
(AbortError-propagation now uses pre-aborted signal + mockRejectedValueOnce
mirroring real Node fetch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.14 followup. The 0.87.14 test used MissedTickBehavior::Burst —
any post-prep yield replayed missed ticks and could fake the >= 2
threshold. Switch to Skip and assert on `ticks_post - ticks_pre >= 50`,
which sits well above the inlined-prep ceiling (~5) and far below
the spawn_blocking floor (~700+). Mutation-confirmed.
Also corrects: prepare_analysis doc count ("five fallback paths" → 4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.
Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.11 followup. The UPDATE matched pending+running but the worker
holds the pre-update `force=false` in an in-memory local — so on the
running branch the row flipped but the worker's skip-if-done still
acked done without re-analyzing. Now `jobs::release` any running row
the UPDATE matched, so the worker's ack_done no-ops (state guard) and
a fresh lease picks up the updated payload.
New test mints the running-state shape, asserts state=pending +
force=true + attempts=0 post-click. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.10 followup from the rereview. The fix lacked a test for the
specific axis it changed (admin authority via cookie only). New test
mints a bearer for a promoted admin and asserts PATCH/PUT-cover/DELETE-
cover all 403 on NULL-uploader rows. Mutation-confirmed.
Also: `MultipartBuilder::finalize` now `pub` for bearer-multipart use;
`admin_csrf_guard` rustdoc updated to describe the post-0.87.10
cookie-precedence rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.15 ship-blockers:
1. mem_check_and_stop test now ALSO greps for the "deferring to next
tick" warning — distinguishes fix from bug. Mutation-confirmed.
2. Extract `refresh_running_after_mem_check` (sets global instead of
echoing so loop-scope flags persist across calls) and drive it
directly from the harness, closing the structurally-untestable
post-mem refresh ERR guard.
Adjacent: log-on-flip gating on both ERR warnings; `log` to stderr;
dangling-TempDir in app::tests reclaim test fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two ship-blockers from the rereview:
1. Env var name mismatch: 0.87.12 renamed every operator surface to
ANALYSIS_VISION_MODEL but the reader still called env::var on
ANALYSIS_MODEL — every documented deploy ran analysis with an
empty model. Rename the reader.
2. Compose regression test was substring-only and missed exactly the
ANALYSIS_VISION_MODEL vs ANALYSIS_MODEL RHS mismatch from 0.87.12.
Tightened to require each backend-consumed var either be in a
small hardcoded allowlist (BIND_ADDRESS, STORAGE_DIR) or have its
RHS interpolate `${KEY...}` with the same name. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three 0.87.9 follow-ups:
1. hooks.server.ts: two new tests covering the
already-aborted-at-entry and mid-flight client-disconnect branches
of the abort chain. Mutation-confirmed.
2. lib/api/admin.ts: two new tests pinning the
getAnalysisMangaCoverage(init?.signal) pass-through and the
AbortError-rejection propagation the dashboard's debounce-race
handler relies on. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.7 + 0.87.8 follow-ups:
1. mem_check_and_stop now defers on docker ERR instead of silently
no-op'ing the SIGTERM under pressure.
2. Post-mem_check_and_stop loop refresh now also handles the ERR
sentinel (continues the loop) rather than letting it reset up_for
and defeat MAX_UPTIME.
3. New `vision-manager/test_manager.sh` — 7 smoke tests covering
vision_running's three exit shapes and mem_check_and_stop's ERR
symmetry, via stubbed docker/psql/curl on PATH. Production loop
guarded by MANAGER_TEST_NO_MAIN.
4. New `app::tests::spawn_analysis_daemon_reclaims_orphaned_analyze_leases`
pins the 0.87.7 wire-through: seeds an expired analyze_page lease,
spawns + shuts down, asserts pending + attempts refunded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.5 follow-ups:
1. Pin spawn_blocking dispatch: new current_thread runtime test runs
analyze() on a 6 MP image while a counter task ticks every 5 ms.
Without spawn_blocking the counter is starved (0 ticks); with it
the runtime stays responsive (≥2 ticks). Mutation-confirmed.
2. Warn on Undecodable fallback: five fallback paths in prepare_analysis
now emit a tracing::warn! with dimensions so a JPEG encoder
regression isn't indistinguishable from "the page was just garbage."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.1 added 12+ env vars to `.env.example` but the compose backend
`environment:` block forwarded none of them — `.env` interpolation
doesn't pass vars to the container. Operators got
anonymous-serving sites, silent ADMIN_* no-ops, and missing
ANALYSIS_* configuration.
Wire 36 backend-consumed vars; also substitute the hardcoded
`BACKEND_URL` on the frontend; emit a startup warning on half-set
`ADMIN_USERNAME`/`ADMIN_PASSWORD`. Regression-test parses
docker-compose.yml and asserts the wiring.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four 0.87.6 follow-ups from the adversarial review:
1. **Migration 0031 pre-dedup.** Demote all-but-lowest-id duplicate
`analyze_page` rows in `(pending|running)` to `dead` before
creating the unique index, with a curator-recoverable last_error
marker. Without this, `sqlx::migrate!` would refuse to boot on
any dirty production DB.
2. **`enqueue_for_page(force=true)` collision.** The partial unique
index used to silently swallow force requests when a
`force=false` job was already pending. Repo function now upgrades
the pending row's `force` flag in place (or falls through to
re-INSERT if the sibling drained mid-call), and reports an
`EnqueueForPageOutcome` for accurate auditing.
3. **`record_duration` gate test.** New test seeds a `done` row with
known duration, force-re-analyzes with failing dispatcher +
max_attempts=3 (non-terminal), asserts duration_ms wasn't
overwritten.
4. **`bookmark.rs` PK comment correction.** Use `b.id DESC` instead
of the wrong `manga_id`; PK is actually `id`, and migration 0004
allows both chapter-level and manga-level bookmarks on the same
manga.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to 0.87.2 + 0.87.3 from the adversarial review:
1. **CSRF cookie-ride.** Bearer-bypass branch checked Authorization
header presence only — an attacker page could mint `Authorization:
Bearer junk` and ride the still-attached session cookie. Bypass now
requires bearer AND no cookie. Drive the cookie-name comparison off
`SESSION_COOKIE_NAME` and split on `=` (not prefix).
2. **`require_can_edit` admin path open to bearer.** Now reads admin
authority off an `Option<CurrentSessionUser>` extractor — None on
bearer-only requests. Owner-match path still accepts bearer; admin
path is cookie-gated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three frontend findings (medium):
1. **`hooks.server.ts` didn't chain client-disconnect into the upstream
fetch.** Closing an admin tab left the backend's `broadcast::Receiver`
+ spawned tokio task running until next backend restart. Listen for
`event.request.signal.abort` and forward to the timeout controller.
2. **`admin/analysis/+page.svelte` missing visibility/pagehide/pageshow
handlers.** Mobile Safari throttles SSE in background tabs; bfcache
leaves a dead connection on back-nav. Apply the close/open-stream
pattern from `admin/crawler/+page.svelte`.
3. **`loadCoverage` had no AbortController.** Debounced search
keystrokes raced. Per-loader controller plumbed through
`getAnalysisMangaCoverage`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three medium vision-manager findings:
1. **`vision_running()` conflated "container absent" with "inspect
failed".** A transient docker-socket-proxy hiccup made the manager
think vision had stopped, which reset `up_for`/`idle_for` and
defeated MAX_UPTIME. Distinguishes: exit 0 → echo true/false;
"No such container" → false; else → ERR (loop skips tick).
2. **`crawl_running()` fail-closed where `analysis_enabled()` fail-opens.**
A psql `ERR` logged a misleading "RAM mutex" line every tick.
Numeric-guard at the call site with a distinct log.
3. **`start_vision` left wedged containers running** on
`START_HEALTH_TIMEOUT` expiry — pinned forever in not-ready state.
Now `stop_vision` so next tick retries from a clean state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`reclaim_orphaned` previously ran only inside `spawn_crawler_daemon` at
startup. A deploy with the crawler disabled and analysis enabled
(analysis-only mode) never refunded orphaned `analyze_page` leases —
recovery deferred to the lazy lease-expiry path. Wire it into
`spawn_analysis_daemon` too. Safe under multi-replica + idempotent
against the crawler-side call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three medium correctness fixes:
1. **Pagination ORDER BY missing tiebreakers** — four admin list queries
sorted by a timestamp or `chapters.number` with no stable secondary
sort. Add `id` tiebreakers across `repo::admin_view`,
`repo::admin_audit`, `repo::bookmark`.
2. **`enqueue_pages` NOT EXISTS read-then-insert race** — concurrent
admin clicks could land duplicate analyze_page jobs. Migration 0031
adds a partial unique index mirroring 0014; query relies on
`ON CONFLICT DO NOTHING`.
3. **`record_duration` overwrote a prior done row's duration on a
non-terminal failed retry of a force-re-analyze.** Gate the call
on "did this attempt actually write a row?".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`VisionClient::analyze` ran `image::load_from_memory`, `DynamicImage::resize_exact(Triangle)`,
and per-band JPEG encodes directly on the tokio task. Each call is
hundreds of ms of pure CPU per page. With multiple workers, the runtime
threads got starved — axum handlers, SSE streams, the crawler daemon's
async timers, and other tasks sharing the runtime all stalled while
analysis was active.
Extract a `prepare_analysis` helper that does ALL the CPU work
(decode + plan + width-reduce + slice + JPEG encode) and returns a
`PreparedAnalysis { Single | Sliced | Undecodable }` of already-encoded
byte payloads. `analyze` calls it inside `tokio::task::spawn_blocking`,
then the async HTTP loop only iterates over the prepared bytes — no
image-crate operations happen on the runtime any more.
Single page → one spawn_blocking → one HTTP call.
Tall page → one spawn_blocking → N OCR calls + 1 grounding call.
Memory peak rises slightly for the Sliced path (all bands held
encoded at the same time instead of one-at-a-time) — for a typical
manga page split into 4 slices at ~200 KB each, that's ~600 KB of
extra peak. Negligible vs. the runtime-starvation cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`require_can_edit` matched `Some(owner) != caller → Forbidden` but let
`None` through. Every crawler row (`repo::crawler::upsert_manga`) has
NULL `uploaded_by`, so any signed-in user could PATCH /api/v1/mangas/<id>
and rewrite the catalog — `update`, `put_cover`, and `delete_cover`
were all in scope. With self-registration on by default, the path was:
register → mass-edit catalog + delete cover blobs from storage.
The original carve-out at the comment said "Once an admin role lands the
NULL case can flip to admin-only." The admin role landed in migration
0018; flipping it now. New rule:
* `Some(owner)` and owner == caller → ok (unchanged)
* `Some(_)` and caller is admin → ok (admin moderation)
* `Some(_)` otherwise → 403 (unchanged)
* `None` and caller is admin → ok (NEW — operator can curate)
* `None` otherwise → 403 (was: ok — the bug)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four high/medium findings from the audit, plus their tests:
1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
`AnalysisSettings::endpoint` validated only with `Url::parse`. A
hostile or CSRF-able admin could repoint at `169.254.169.254`
(cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
— and the vision worker bearer-attaches an env-managed API key to
every call. Extract `ensure_public_target` from `safety::is_safe_url`
(allowlist-free public-host check: scheme + private-IP literal +
localhost) and wire it into both fields. Docker DNS names
(`mangalord-vision`) keep passing because they're hostnames, not
IP literals. The analysis check is gated on `enabled=true` so the
dev-default `localhost:8000` stays usable until the worker is
actually turned on (toggling enabled=true later re-runs the gate).
2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
silently skipped the entire CSRF check, so an operator forgetting
the env var shipped an unguarded admin surface. Cookie-auth POSTs
now fail-closed when the allowlist is empty AND when neither
`Origin` nor `Referer` accompanies the request. Two carve-outs:
`Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
and requests with NO session cookie at all bypass to let the auth
extractor return a clean 401 instead of a confusing 403.
3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
would exfiltrate page-image bytes + the bearer key through an
upstream proxy. Same for the readiness probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three operational gaps tripped a recent review:
1. **Postgres had no `restart:` policy.** tor / docker-socket-proxy /
vision-manager all set `unless-stopped`; pg was the lone exception.
A transient OOM or panic left backend's `depends_on: service_healthy`
blocking every future restart and took the stack offline. Add
`unless-stopped` on postgres and on backend/frontend in the base
compose so the prod overlay is the same shape.
2. **`BACKEND_PROXY_TIMEOUT_MS` was documented but silently a no-op.**
`.env.example` carried it but `docker-compose.yml` never forwarded it
into the frontend service env, so the value never reached
`hooks.server.ts`. Wire it through.
3. **`.env.example` omitted required env vars.** Operators copying the
template got no admin bootstrap, no boot-seed crawler URL, no
analysis configuration. Add: `ADMIN_USERNAME`, `ADMIN_PASSWORD`,
`PRIVATE_MODE`, `ALLOW_SELF_REGISTER`, `CRAWLER_START_URL`,
`CRAWLER_CDN_HOST`, `ANALYSIS_ENABLED`, `ANALYSIS_VISION_URL`,
`ANALYSIS_VISION_MODEL`, `ANALYSIS_WORKERS`,
`ANALYSIS_JOB_TIMEOUT_SECS`, `ANALYSIS_API_KEY` — each with the
comment block that explains seed-vs-runtime semantics.
Also adds `frontend/vite.config.ts.timestamp-*.mjs` to `.gitignore` — a
crashed dev server leaves these transient files behind otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.
Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.
Closes the Phase-1 observability set (audit log · health checks · trends).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The admin overview dashboard polls /admin/overview every 30s. Its
`manga_stats` aggregate evaluated MANGA_SYNC_STATE_CASE over the WHOLE
library (14.5k mangas) with no LIMIT, and that CASE runs a correlated
EXISTS over crawler_jobs. With no index on the `sync_chapter_list`
manga_id path, Postgres seqscanned all in-flight jobs *per manga* —
O(mangas x jobs). The disabled-analysis backlog (9.5k pending
`analyze_page` rows that can never match the sync-kind filter) inflated
every scan, so each call took 6-7 min. The 30s poll stacked ~10 of them
concurrently → load 11, Postgres at ~100%.
EXPLAIN ANALYZE on the live DB: the rewrite drops the query from 6-7 min
to ~1.5s (per-manga crawler_jobs seqscan → single index-backed pass).
Fixes:
- Rewrite `manga_stats` to collect the in-flight manga-id set in ONE pass
over the sync jobs (index-backed), then classify via a hash semi-join —
O(mangas + jobs), immune to the analyze_page backlog size.
- Add partial index crawler_jobs_sync_chapter_list_manga_idx (migration
0029) covering the sync_chapter_list manga_id path; mirrors the existing
sync_manga index (0020). Also speeds the per-row Mangas tab listing.
- statement_timeout backstop (5s) on the aggregate so a pathological plan
can never pin a backend for minutes again.
- Single-flight + 10s TTL cache on the /admin/overview handler so
concurrent pollers coalesce instead of stacking.
- Slow the dashboard poll 30s -> 60s (server-cached now anyway).
The in_progress rule in the rewrite is kept in lockstep with the first
arm of MANGA_SYNC_STATE_CASE (documented in both places).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The crawler's headless Chromium advertised the default HeadlessChrome UA,
which Cloudflare's bot detection challenges on sight — over Tor that meant an
unsolvable 'Just a moment' page on every fetch, so session verification kept
classifying pages as transient and the metadata pass failed at
'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal
Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome
UA returns the challenge.) The existing user_agent setting only fed reqwest,
never the browser. Override the launcher's UA with a realistic non-headless
default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike
CRAWLER_BROWSER_ARGS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three failure-mode fixes surfaced by a crawler recovery audit:
- Graceful shutdown mid-dispatch now releases the in-flight job back to
pending without burning a retry attempt. process_lease wraps the
dispatch in a biased tokio::select! cancel arm that aborts the
heartbeat and calls jobs::release; previously a mid-job SIGTERM left
the row 'running' until lease expiry and cost one of max_attempts.
- Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired
lease back to pending (attempt refunded), run once at daemon startup
before workers start. Crash recovery is now immediate instead of
waiting a full lease window for the lazy lease-expiry path. Safe under
multi-replica: only already-expired leases are touched, which a
healthy heartbeating peer never has.
- Manga-list parsing now warns when listing anchors are dropped for a
missing/empty href or title (split into parse_manga_list_anchors so
the drop count is testable), turning silent source markup drift into
an observable signal. Returned refs are byte-identical to before.
Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt,
reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit
test. Patch bump 0.81.0 -> 0.81.1 (both manifests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a tiny privileged sidecar (vision-manager/) that polls the analysis
backlog in Postgres and starts/stops the mangalord-vision container by name:
start when analyze_page jobs are pending, stop after STOP_DEBOUNCE idle.
It is the single owner of the vision lifecycle and the only component with
Docker access — scoped through tecnativa/docker-socket-proxy (CONTAINERS+POST
only) on an internal-only network, so the internet-facing backend never
touches the socket.
Both helpers sit behind `profiles: [ai]` (vanilla `compose up` is
unaffected). The manager debounces the stop, gates the first request on
GET /health==200 after a cold start, honours a crawl RAM-mutex on the 8 GiB
box, and owns its own idle timer (leak-safe if the backend dies). Backlog
query uses the real schema (state + payload->>'kind'); a read-only DB role
(readonly-role.sql) keeps it off the backend creds.
Bump 0.80.0 -> 0.81.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Small vision models (qwen3-vl-4b) sometimes loop the OCR array, running
the response into the token ceiling (finish_reason: length) and failing to
parse. Layered guardrails:
- Prompts: explicit "transcribe each element once, never repeat/loop, at
most N, STOP when done" in the OCR, grounding and combined prompts.
- Schema: hard maxItems on ocr_results/tagging_results/content_type and
maxLength on text/scene — LM Studio's grammar enforces these, so a loop
is grammar-bounded rather than relying on max_tokens.
- Sampling: a configurable frequency_penalty (ANALYSIS_FREQUENCY_PENALTY,
default 0.3) sent with each request — the decode-time lever that actually
breaks loops.
- Code: sanitize() collapses runaway consecutive duplicates (same
normalized text + kind) so a loop that slips through still can't flood
the row.
Tests: schema caps present, frequency_penalty included only when nonzero,
sanitize collapses consecutive repeats; config default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Boundary text duplicated across slices (often with slightly different,
cropped transcriptions) survived the text-only merge. The OCR pass now
asks for a per-piece vertical position and the merge uses it:
- OcrResult gains optional `y` (fraction 0..1 of the slice); the Pass-A
OCR schema/prompt request it (combined path leaves it None). Not
persisted.
- merge_ocr takes each slice's working-image y-band and maps `y` to a
page-global position. A seam pair is a duplicate when position-close
(same kind) OR text-similar, so a mis-OCR'd boundary line is caught even
when the text differs. The kept copy is the one more central in its slice
(less cropped); falls back to keep-longer when positions are missing.
- Self-calibrates the model's y values (pixels vs. fraction) and ignores
degenerate columns, so a bad localizer can't over-merge.
Tests: position pairs differing texts and keeps the less-cropped copy;
text-only fallback (dedup/keep-longer/non-adjacent/order) still holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Very tall webtoon pages were squashed by the fixed longest-edge downscale,
so OCR was garbage. The vision client now slices to the model's pixel
budget at native resolution and assembles one result:
- plan_slices: native-resolution bands sized to max_pixels (portrait OR
landscape), only when height/width > tall_aspect_threshold; min_slice_
height guards pathologically wide pages (reduce width); max_slices caps
the count (coarser fallback). Normal pages still take one combined call.
- Two-pass, OCR-first: Pass A OCRs each band (near-native), merge_ocr
stitches them with seam-scoped fuzzy dedup (keep the longer transcript,
preserve order/kind, don't collapse non-adjacent repeats); Pass B feeds
the whole downscaled image + the merged OCR text to get tags/scene/safety
grounded in the real dialogue. Only OCR is merged.
- Config: replace max_image_dim with max_pixels / min_slice_height /
slice_overlap / tall_aspect_threshold / max_slices (+ env_f64); bump
job_timeout default 180->600 (N sequential slice calls per page); raise
MAX_OCR_PIECES 60->200. New prompts/schemas: OCR_PROMPT/ocr_json_schema,
GROUNDING_PROMPT/grounding_json_schema.
Tests: plan_slices (single/portrait/landscape/pathological-wide/cap +
coverage/overlap), merge_ocr (seam dedup, keep-longer, non-adjacent
repeats, order), render_whole/render_slice, the new request builders, and
updated config defaults/env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A text-dense page's full analysis JSON exceeded the 900-token output cap,
so the model stopped mid-object (finish_reason: length) and the truncated
response failed to parse ("EOF while parsing a list"). The page image +
prompt are only a few hundred tokens, so a larger output budget still fits
comfortably in an 8k context window. Still overridable via
ANALYSIS_MAX_TOKENS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadcasts analysis progress so the dashboard updates live:
- analysis::events: AnalysisEvents broadcaster + AnalysisEvent
(Enqueued / Started / Completed / Failed), carrying the
manga/chapter/page breadcrumb.
- The worker daemon resolves each page's breadcrumb (repo::page::locate)
and publishes Started before dispatch and Completed/Failed after.
- The admin reenqueue publishes Enqueued (scoped by manga/chapter).
- GET /v1/admin/analysis/status/stream — SSE (RequireAdmin) forwarding
each event as a named `analysis` frame; broadcast lag emits a `lagged`
frame. AppState carries the always-present events bus.
Tests: worker publishes started+completed (with breadcrumb) and failed;
SSE route is admin-gated (403 non-admin) and returns text/event-stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks /admin/analysis from a blind enqueue form into a coverage browser:
- Loads a paginated overview of mangas with a CoverageBadge
(Full/Partial/None, analyzed/total) per row; debounced search filters it.
- Drill manga → chapters (coverage badge each) → page grid (chips colored
done/queued/failed/none) with a legend.
- Click a page chip → Modal showing the full result: status, model, time,
NSFW + content-warning chips, scene description, tags, and OCR lines
(labeled by kind). Unanalyzed/failed pages get a "Queue this page"
action (force re-analyze).
- Keeps "Queue all" + per-manga/chapter enqueue and the include-analyzed
toggle; enqueue refreshes the open chapter's grid so queued pages show.
API client: getAnalysisMangaCoverage / ChapterCoverage / ChapterPages /
PageDetail + types. New CoverageBadge component.
Tests: vitest for the four clients; Playwright for overview+badges+queue-all,
drill+inspect, and queue-from-detail. svelte-check + build clean; 266 vitest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
