Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sort the manga catalog by created / updated / title / author with an
independent asc/desc direction control. An omitted `order` defaults per field
(dates newest-first, text A→Z), matching the UI, so a bare `?sort=<field>`
means the same thing in the browser and over the API; `sort=recent` remains a
back-compat alias for `created`.
- Backend: SortField/SortOrder parsed with validation (structured 422 on bad
input), per-field default_order, NULLS LAST only on the nullable author key,
and migration 0033 indexing mangas(updated_at DESC, id) to back the default
sort and its id tie-break.
- Frontend: catalog sort field + direction (SegmentedControl) on desktop and a
mobile bottom sheet; pure helpers in $lib/mangaSort; keyboard-accessible
direction control; visible Direction labels and a sheet "Done" button.
- Tests: backend integration coverage (defaults, alias, invalid input,
ascending-id tie-break), frontend unit + e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.12 added a regression test for backend env wiring, but its
BACKEND_CONSUMED list was hand-maintained and drifted: ~22 vars read
by backend/src/ never made it into compose's backend.environment.
An operator setting ANALYSIS_TEMPERATURE=0.3 in .env got a silent
no-op — exactly the bug class the test was supposed to catch.
Make .env.example the single source of truth:
* docker_compose_wires_every_documented_backend_env_var now reads
.env.example at test time, filters NOT_BACKEND_CONSUMED_IN_ENV_EXAMPLE
(compose-side composed vars, frontend-only vars, sidecar vars),
and requires every remaining key to be wired into compose with a
matching ${KEY...} interpolation OR allowlisted via COMPOSE_RHS_EXEMPT.
* every_backend_src_env_read_is_documented_or_allowlisted scans
backend/src/ for env::var / env_bool / env_i64 / ... reads and
requires each key to be in .env.example OR in INTERNAL_NOT_CONFIGURABLE
with a per-entry rationale. Catches the silent-no-op bug from the
other side.
Wire the previously-missing env vars into docker-compose.yml's
backend.environment (CRAWLER_DAEMON, CRAWLER_DAILY_AT, CRAWLER_TZ,
CRAWLER_IDLE_TIMEOUT_S, CRAWLER_CHAPTER_WORKERS, CRAWLER_JOB_RETENTION_DAYS,
CRAWL_METRICS_RETENTION_DAYS, CRAWLER_RATE_MS, CRAWLER_CDN_RATE_MS,
CRAWLER_USER_AGENT, CRAWLER_PHPSESSID, CRAWLER_COOKIE_DOMAIN,
CRAWLER_TOR_CONTROL_COOKIE_PATH, all ANALYSIS_* tuning knobs).
Document each new var in .env.example with the same default value
shown in compose's `${KEY:-default}` form. Allowlist internal/system
vars (HOME = chromium cache dir fallback, CRAWLER_BROWSER_*,
CRAWLER_SKIP_*, the multi-paragraph ANALYSIS_*_PROMPT seeds) with
per-entry rationale.
Mutation-tested both sides: adding `env::var("MANGALORD_FAKE_NEW_KNOB")`
to main.rs fails the scanner; removing the new compose wiring fails
the documented-var test with all 23 missing keys named.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.14 followup. The 0.87.14 test used MissedTickBehavior::Burst —
any post-prep yield replayed missed ticks and could fake the >= 2
threshold. Switch to Skip and assert on `ticks_post - ticks_pre >= 50`,
which sits well above the inlined-prep ceiling (~5) and far below
the spawn_blocking floor (~700+). Mutation-confirmed.
Also corrects: prepare_analysis doc count ("five fallback paths" → 4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.
Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.11 followup. The UPDATE matched pending+running but the worker
holds the pre-update `force=false` in an in-memory local — so on the
running branch the row flipped but the worker's skip-if-done still
acked done without re-analyzing. Now `jobs::release` any running row
the UPDATE matched, so the worker's ack_done no-ops (state guard) and
a fresh lease picks up the updated payload.
New test mints the running-state shape, asserts state=pending +
force=true + attempts=0 post-click. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.10 followup from the rereview. The fix lacked a test for the
specific axis it changed (admin authority via cookie only). New test
mints a bearer for a promoted admin and asserts PATCH/PUT-cover/DELETE-
cover all 403 on NULL-uploader rows. Mutation-confirmed.
Also: `MultipartBuilder::finalize` now `pub` for bearer-multipart use;
`admin_csrf_guard` rustdoc updated to describe the post-0.87.10
cookie-precedence rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.15 ship-blockers:
1. mem_check_and_stop test now ALSO greps for the "deferring to next
tick" warning — distinguishes fix from bug. Mutation-confirmed.
2. Extract `refresh_running_after_mem_check` (sets global instead of
echoing so loop-scope flags persist across calls) and drive it
directly from the harness, closing the structurally-untestable
post-mem refresh ERR guard.
Adjacent: log-on-flip gating on both ERR warnings; `log` to stderr;
dangling-TempDir in app::tests reclaim test fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two ship-blockers from the rereview:
1. Env var name mismatch: 0.87.12 renamed every operator surface to
ANALYSIS_VISION_MODEL but the reader still called env::var on
ANALYSIS_MODEL — every documented deploy ran analysis with an
empty model. Rename the reader.
2. Compose regression test was substring-only and missed exactly the
ANALYSIS_VISION_MODEL vs ANALYSIS_MODEL RHS mismatch from 0.87.12.
Tightened to require each backend-consumed var either be in a
small hardcoded allowlist (BIND_ADDRESS, STORAGE_DIR) or have its
RHS interpolate `${KEY...}` with the same name. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.7 + 0.87.8 follow-ups:
1. mem_check_and_stop now defers on docker ERR instead of silently
no-op'ing the SIGTERM under pressure.
2. Post-mem_check_and_stop loop refresh now also handles the ERR
sentinel (continues the loop) rather than letting it reset up_for
and defeat MAX_UPTIME.
3. New `vision-manager/test_manager.sh` — 7 smoke tests covering
vision_running's three exit shapes and mem_check_and_stop's ERR
symmetry, via stubbed docker/psql/curl on PATH. Production loop
guarded by MANAGER_TEST_NO_MAIN.
4. New `app::tests::spawn_analysis_daemon_reclaims_orphaned_analyze_leases`
pins the 0.87.7 wire-through: seeds an expired analyze_page lease,
spawns + shuts down, asserts pending + attempts refunded.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.5 follow-ups:
1. Pin spawn_blocking dispatch: new current_thread runtime test runs
analyze() on a 6 MP image while a counter task ticks every 5 ms.
Without spawn_blocking the counter is starved (0 ticks); with it
the runtime stays responsive (≥2 ticks). Mutation-confirmed.
2. Warn on Undecodable fallback: five fallback paths in prepare_analysis
now emit a tracing::warn! with dimensions so a JPEG encoder
regression isn't indistinguishable from "the page was just garbage."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.1 added 12+ env vars to `.env.example` but the compose backend
`environment:` block forwarded none of them — `.env` interpolation
doesn't pass vars to the container. Operators got
anonymous-serving sites, silent ADMIN_* no-ops, and missing
ANALYSIS_* configuration.
Wire 36 backend-consumed vars; also substitute the hardcoded
`BACKEND_URL` on the frontend; emit a startup warning on half-set
`ADMIN_USERNAME`/`ADMIN_PASSWORD`. Regression-test parses
docker-compose.yml and asserts the wiring.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four 0.87.6 follow-ups from the adversarial review:
1. **Migration 0031 pre-dedup.** Demote all-but-lowest-id duplicate
`analyze_page` rows in `(pending|running)` to `dead` before
creating the unique index, with a curator-recoverable last_error
marker. Without this, `sqlx::migrate!` would refuse to boot on
any dirty production DB.
2. **`enqueue_for_page(force=true)` collision.** The partial unique
index used to silently swallow force requests when a
`force=false` job was already pending. Repo function now upgrades
the pending row's `force` flag in place (or falls through to
re-INSERT if the sibling drained mid-call), and reports an
`EnqueueForPageOutcome` for accurate auditing.
3. **`record_duration` gate test.** New test seeds a `done` row with
known duration, force-re-analyzes with failing dispatcher +
max_attempts=3 (non-terminal), asserts duration_ms wasn't
overwritten.
4. **`bookmark.rs` PK comment correction.** Use `b.id DESC` instead
of the wrong `manga_id`; PK is actually `id`, and migration 0004
allows both chapter-level and manga-level bookmarks on the same
manga.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to 0.87.2 + 0.87.3 from the adversarial review:
1. **CSRF cookie-ride.** Bearer-bypass branch checked Authorization
header presence only — an attacker page could mint `Authorization:
Bearer junk` and ride the still-attached session cookie. Bypass now
requires bearer AND no cookie. Drive the cookie-name comparison off
`SESSION_COOKIE_NAME` and split on `=` (not prefix).
2. **`require_can_edit` admin path open to bearer.** Now reads admin
authority off an `Option<CurrentSessionUser>` extractor — None on
bearer-only requests. Owner-match path still accepts bearer; admin
path is cookie-gated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`reclaim_orphaned` previously ran only inside `spawn_crawler_daemon` at
startup. A deploy with the crawler disabled and analysis enabled
(analysis-only mode) never refunded orphaned `analyze_page` leases —
recovery deferred to the lazy lease-expiry path. Wire it into
`spawn_analysis_daemon` too. Safe under multi-replica + idempotent
against the crawler-side call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three medium correctness fixes:
1. **Pagination ORDER BY missing tiebreakers** — four admin list queries
sorted by a timestamp or `chapters.number` with no stable secondary
sort. Add `id` tiebreakers across `repo::admin_view`,
`repo::admin_audit`, `repo::bookmark`.
2. **`enqueue_pages` NOT EXISTS read-then-insert race** — concurrent
admin clicks could land duplicate analyze_page jobs. Migration 0031
adds a partial unique index mirroring 0014; query relies on
`ON CONFLICT DO NOTHING`.
3. **`record_duration` overwrote a prior done row's duration on a
non-terminal failed retry of a force-re-analyze.** Gate the call
on "did this attempt actually write a row?".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`VisionClient::analyze` ran `image::load_from_memory`, `DynamicImage::resize_exact(Triangle)`,
and per-band JPEG encodes directly on the tokio task. Each call is
hundreds of ms of pure CPU per page. With multiple workers, the runtime
threads got starved — axum handlers, SSE streams, the crawler daemon's
async timers, and other tasks sharing the runtime all stalled while
analysis was active.
Extract a `prepare_analysis` helper that does ALL the CPU work
(decode + plan + width-reduce + slice + JPEG encode) and returns a
`PreparedAnalysis { Single | Sliced | Undecodable }` of already-encoded
byte payloads. `analyze` calls it inside `tokio::task::spawn_blocking`,
then the async HTTP loop only iterates over the prepared bytes — no
image-crate operations happen on the runtime any more.
Single page → one spawn_blocking → one HTTP call.
Tall page → one spawn_blocking → N OCR calls + 1 grounding call.
Memory peak rises slightly for the Sliced path (all bands held
encoded at the same time instead of one-at-a-time) — for a typical
manga page split into 4 slices at ~200 KB each, that's ~600 KB of
extra peak. Negligible vs. the runtime-starvation cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`require_can_edit` matched `Some(owner) != caller → Forbidden` but let
`None` through. Every crawler row (`repo::crawler::upsert_manga`) has
NULL `uploaded_by`, so any signed-in user could PATCH /api/v1/mangas/<id>
and rewrite the catalog — `update`, `put_cover`, and `delete_cover`
were all in scope. With self-registration on by default, the path was:
register → mass-edit catalog + delete cover blobs from storage.
The original carve-out at the comment said "Once an admin role lands the
NULL case can flip to admin-only." The admin role landed in migration
0018; flipping it now. New rule:
* `Some(owner)` and owner == caller → ok (unchanged)
* `Some(_)` and caller is admin → ok (admin moderation)
* `Some(_)` otherwise → 403 (unchanged)
* `None` and caller is admin → ok (NEW — operator can curate)
* `None` otherwise → 403 (was: ok — the bug)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four high/medium findings from the audit, plus their tests:
1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
`AnalysisSettings::endpoint` validated only with `Url::parse`. A
hostile or CSRF-able admin could repoint at `169.254.169.254`
(cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
— and the vision worker bearer-attaches an env-managed API key to
every call. Extract `ensure_public_target` from `safety::is_safe_url`
(allowlist-free public-host check: scheme + private-IP literal +
localhost) and wire it into both fields. Docker DNS names
(`mangalord-vision`) keep passing because they're hostnames, not
IP literals. The analysis check is gated on `enabled=true` so the
dev-default `localhost:8000` stays usable until the worker is
actually turned on (toggling enabled=true later re-runs the gate).
2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
silently skipped the entire CSRF check, so an operator forgetting
the env var shipped an unguarded admin surface. Cookie-auth POSTs
now fail-closed when the allowlist is empty AND when neither
`Origin` nor `Referer` accompanies the request. Two carve-outs:
`Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
and requests with NO session cookie at all bypass to let the auth
extractor return a clean 401 instead of a confusing 403.
3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
would exfiltrate page-image bytes + the bearer key through an
upstream proxy. Same for the readiness probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.
Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.
Closes the Phase-1 observability set (audit log · health checks · trends).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.
GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).
Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.
Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The admin overview dashboard polls /admin/overview every 30s. Its
`manga_stats` aggregate evaluated MANGA_SYNC_STATE_CASE over the WHOLE
library (14.5k mangas) with no LIMIT, and that CASE runs a correlated
EXISTS over crawler_jobs. With no index on the `sync_chapter_list`
manga_id path, Postgres seqscanned all in-flight jobs *per manga* —
O(mangas x jobs). The disabled-analysis backlog (9.5k pending
`analyze_page` rows that can never match the sync-kind filter) inflated
every scan, so each call took 6-7 min. The 30s poll stacked ~10 of them
concurrently → load 11, Postgres at ~100%.
EXPLAIN ANALYZE on the live DB: the rewrite drops the query from 6-7 min
to ~1.5s (per-manga crawler_jobs seqscan → single index-backed pass).
Fixes:
- Rewrite `manga_stats` to collect the in-flight manga-id set in ONE pass
over the sync jobs (index-backed), then classify via a hash semi-join —
O(mangas + jobs), immune to the analyze_page backlog size.
- Add partial index crawler_jobs_sync_chapter_list_manga_idx (migration
0029) covering the sync_chapter_list manga_id path; mirrors the existing
sync_manga index (0020). Also speeds the per-row Mangas tab listing.
- statement_timeout backstop (5s) on the aggregate so a pathological plan
can never pin a backend for minutes again.
- Single-flight + 10s TTL cache on the /admin/overview handler so
concurrent pollers coalesce instead of stacking.
- Slow the dashboard poll 30s -> 60s (server-cached now anyway).
The in_progress rule in the rewrite is kept in lockstep with the first
arm of MANGA_SYNC_STATE_CASE (documented in both places).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The crawler's headless Chromium advertised the default HeadlessChrome UA,
which Cloudflare's bot detection challenges on sight — over Tor that meant an
unsolvable 'Just a moment' page on every fetch, so session verification kept
classifying pages as transient and the metadata pass failed at
'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal
Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome
UA returns the challenge.) The existing user_agent setting only fed reqwest,
never the browser. Override the launcher's UA with a realistic non-headless
default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike
CRAWLER_BROWSER_ARGS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three failure-mode fixes surfaced by a crawler recovery audit:
- Graceful shutdown mid-dispatch now releases the in-flight job back to
pending without burning a retry attempt. process_lease wraps the
dispatch in a biased tokio::select! cancel arm that aborts the
heartbeat and calls jobs::release; previously a mid-job SIGTERM left
the row 'running' until lease expiry and cost one of max_attempts.
- Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired
lease back to pending (attempt refunded), run once at daemon startup
before workers start. Crash recovery is now immediate instead of
waiting a full lease window for the lazy lease-expiry path. Safe under
multi-replica: only already-expired leases are touched, which a
healthy heartbeating peer never has.
- Manga-list parsing now warns when listing anchors are dropped for a
missing/empty href or title (split into parse_manga_list_anchors so
the drop count is testable), turning silent source markup drift into
an observable signal. Returned refs are byte-identical to before.
Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt,
reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit
test. Patch bump 0.81.0 -> 0.81.1 (both manifests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The private_mode_* config unit tests set then remove_var(DATABASE_URL).
That env is process-global, and the #[sqlx::test] lib tests
(crawler::content, crawler::session_control, ...) read it in parallel,
so on a multi-core runner they intermittently saw it unset and panicked
'DATABASE_URL must be set: NotPresent' (6 failures on the Pi's 4 cores).
Never overwrite/remove it: set only if absent via ensure_database_url(),
leaving CI's real URL stable for the parallel DB tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend:
- Log on readiness state transitions (ready<->not-ready) so a misconfigured
ANALYSIS_VISION_HEALTH_URL, which otherwise parks the worker silently
forever, is diagnosable.
- Add unit tests for HttpVisionReadiness: 2xx->ready, non-2xx->not-ready,
connection error->not-ready (the production status mapping had no test).
vision-manager:
- Guard the backlog count against non-numeric output before `-gt`, so a stray
value can't exit-2 and kill the loop under `set -e`.
- Throttle the crawl-mutex "deferring start" log to once per episode.
- Only reset the idle/uptime timers when `docker stop` actually succeeds, so a
failed stop retries next tick instead of waiting a full debounce window.
- Decouple the per-probe curl timeout (HEALTH_TIMEOUT) from the warm-up poll
cadence.
Docs:
- Correct the docker-socket-proxy comment: CONTAINERS+POST permits the full
container lifecycle (not just start/stop); state the real trust boundary.
- Document that the externally-defined mangalord-vision container must share
the compose network for name resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The analysis worker leases an analyze_page job (incrementing attempts in
SQL) before calling vision, so a stopped/loading vision burns all retries
and writes a permanent failed page_analysis row. With the vision-manager
autoscaler idle-stopping the container, that would poison the queue on
every cold start.
Add an optional VisionReadiness seam (HTTP GET /health in production) wired
via the new env-only ANALYSIS_VISION_HEALTH_URL. When set, the worker parks
without leasing until the probe answers 2xx; jobs stay pending with
attempts untouched and resume the instant vision is ready. None preserves
today's behavior for always-on endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Small vision models (qwen3-vl-4b) sometimes loop the OCR array, running
the response into the token ceiling (finish_reason: length) and failing to
parse. Layered guardrails:
- Prompts: explicit "transcribe each element once, never repeat/loop, at
most N, STOP when done" in the OCR, grounding and combined prompts.
- Schema: hard maxItems on ocr_results/tagging_results/content_type and
maxLength on text/scene — LM Studio's grammar enforces these, so a loop
is grammar-bounded rather than relying on max_tokens.
- Sampling: a configurable frequency_penalty (ANALYSIS_FREQUENCY_PENALTY,
default 0.3) sent with each request — the decode-time lever that actually
breaks loops.
- Code: sanitize() collapses runaway consecutive duplicates (same
normalized text + kind) so a loop that slips through still can't flood
the row.
Tests: schema caps present, frequency_penalty included only when nonzero,
sanitize collapses consecutive repeats; config default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Boundary text duplicated across slices (often with slightly different,
cropped transcriptions) survived the text-only merge. The OCR pass now
asks for a per-piece vertical position and the merge uses it:
- OcrResult gains optional `y` (fraction 0..1 of the slice); the Pass-A
OCR schema/prompt request it (combined path leaves it None). Not
persisted.
- merge_ocr takes each slice's working-image y-band and maps `y` to a
page-global position. A seam pair is a duplicate when position-close
(same kind) OR text-similar, so a mis-OCR'd boundary line is caught even
when the text differs. The kept copy is the one more central in its slice
(less cropped); falls back to keep-longer when positions are missing.
- Self-calibrates the model's y values (pixels vs. fraction) and ignores
degenerate columns, so a bad localizer can't over-merge.
Tests: position pairs differing texts and keeps the less-cropped copy;
text-only fallback (dedup/keep-longer/non-adjacent/order) still holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Very tall webtoon pages were squashed by the fixed longest-edge downscale,
so OCR was garbage. The vision client now slices to the model's pixel
budget at native resolution and assembles one result:
- plan_slices: native-resolution bands sized to max_pixels (portrait OR
landscape), only when height/width > tall_aspect_threshold; min_slice_
height guards pathologically wide pages (reduce width); max_slices caps
the count (coarser fallback). Normal pages still take one combined call.
- Two-pass, OCR-first: Pass A OCRs each band (near-native), merge_ocr
stitches them with seam-scoped fuzzy dedup (keep the longer transcript,
preserve order/kind, don't collapse non-adjacent repeats); Pass B feeds
the whole downscaled image + the merged OCR text to get tags/scene/safety
grounded in the real dialogue. Only OCR is merged.
- Config: replace max_image_dim with max_pixels / min_slice_height /
slice_overlap / tall_aspect_threshold / max_slices (+ env_f64); bump
job_timeout default 180->600 (N sequential slice calls per page); raise
MAX_OCR_PIECES 60->200. New prompts/schemas: OCR_PROMPT/ocr_json_schema,
GROUNDING_PROMPT/grounding_json_schema.
Tests: plan_slices (single/portrait/landscape/pathological-wide/cap +
coverage/overlap), merge_ocr (seam dedup, keep-longer, non-adjacent
repeats, order), render_whole/render_slice, the new request builders, and
updated config defaults/env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
