The OCR and vision dispatchers read the whole page image into a Vec via
storage.get() and only then checked max_image_bytes, so the cap couldn't
bound the read. Stream via get_stream() + safety::accumulate_capped so the
read bails as soon as the running total exceeds the cap.
Bump to 0.124.6.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The vision prep pass decoded pages with bare `image::load_from_memory`,
which applies no allocation bound. `max_pixels` only downscales after a
full decode, so a tiny WebP/JPEG header declaring huge dimensions could
OOM the blocking worker — the same decompression-bomb the OCR backend
already guards against. Manga pages are commonly WebP/JPEG, where the
format's own self-limits are weaker than PNG's.
Route the decode through `decode_within`, applying an `image::Limits`
alloc cap sized from the shared `ocr_max_decode_pixels`
(ANALYSIS_OCR_MAX_DECODE_PIXELS). Add real-WebP bomb coverage asserting
both the helper and the end-to-end Undecodable fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OCR dispatcher acquired a borrowed semaphore permit, then ran the
CPU-bound inference in `spawn_blocking`. On cancellation (graceful
shutdown) the future dropped the permit while the detached blocking task
kept running, briefly oversubscribing the blocking pool past
ANALYSIS_WORKERS.
Extract `run_ocr_blocking`, which acquires an *owned* permit and moves it
into the blocking task so the concurrency slot's lifetime matches the
actual work. Covered by a cancellation-invariant test asserting the
permit stays held after the caller future is aborted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:
- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
literal `Err`).
- test helpers: build configs via struct-update syntax instead of
`Default::default()` + field reassignment; add a type alias for a
complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.
Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.14 followup. The 0.87.14 test used MissedTickBehavior::Burst —
any post-prep yield replayed missed ticks and could fake the >= 2
threshold. Switch to Skip and assert on `ticks_post - ticks_pre >= 50`,
which sits well above the inlined-prep ceiling (~5) and far below
the spawn_blocking floor (~700+). Mutation-confirmed.
Also corrects: prepare_analysis doc count ("five fallback paths" → 4).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.
Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.5 follow-ups:
1. Pin spawn_blocking dispatch: new current_thread runtime test runs
analyze() on a 6 MP image while a counter task ticks every 5 ms.
Without spawn_blocking the counter is starved (0 ticks); with it
the runtime stays responsive (≥2 ticks). Mutation-confirmed.
2. Warn on Undecodable fallback: five fallback paths in prepare_analysis
now emit a tracing::warn! with dimensions so a JPEG encoder
regression isn't indistinguishable from "the page was just garbage."
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three medium correctness fixes:
1. **Pagination ORDER BY missing tiebreakers** — four admin list queries
sorted by a timestamp or `chapters.number` with no stable secondary
sort. Add `id` tiebreakers across `repo::admin_view`,
`repo::admin_audit`, `repo::bookmark`.
2. **`enqueue_pages` NOT EXISTS read-then-insert race** — concurrent
admin clicks could land duplicate analyze_page jobs. Migration 0031
adds a partial unique index mirroring 0014; query relies on
`ON CONFLICT DO NOTHING`.
3. **`record_duration` overwrote a prior done row's duration on a
non-terminal failed retry of a force-re-analyze.** Gate the call
on "did this attempt actually write a row?".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`VisionClient::analyze` ran `image::load_from_memory`, `DynamicImage::resize_exact(Triangle)`,
and per-band JPEG encodes directly on the tokio task. Each call is
hundreds of ms of pure CPU per page. With multiple workers, the runtime
threads got starved — axum handlers, SSE streams, the crawler daemon's
async timers, and other tasks sharing the runtime all stalled while
analysis was active.
Extract a `prepare_analysis` helper that does ALL the CPU work
(decode + plan + width-reduce + slice + JPEG encode) and returns a
`PreparedAnalysis { Single | Sliced | Undecodable }` of already-encoded
byte payloads. `analyze` calls it inside `tokio::task::spawn_blocking`,
then the async HTTP loop only iterates over the prepared bytes — no
image-crate operations happen on the runtime any more.
Single page → one spawn_blocking → one HTTP call.
Tall page → one spawn_blocking → N OCR calls + 1 grounding call.
Memory peak rises slightly for the Sliced path (all bands held
encoded at the same time instead of one-at-a-time) — for a typical
manga page split into 4 slices at ~200 KB each, that's ~600 KB of
extra peak. Negligible vs. the runtime-starvation cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend:
- Log on readiness state transitions (ready<->not-ready) so a misconfigured
ANALYSIS_VISION_HEALTH_URL, which otherwise parks the worker silently
forever, is diagnosable.
- Add unit tests for HttpVisionReadiness: 2xx->ready, non-2xx->not-ready,
connection error->not-ready (the production status mapping had no test).
vision-manager:
- Guard the backlog count against non-numeric output before `-gt`, so a stray
value can't exit-2 and kill the loop under `set -e`.
- Throttle the crawl-mutex "deferring start" log to once per episode.
- Only reset the idle/uptime timers when `docker stop` actually succeeds, so a
failed stop retries next tick instead of waiting a full debounce window.
- Decouple the per-probe curl timeout (HEALTH_TIMEOUT) from the warm-up poll
cadence.
Docs:
- Correct the docker-socket-proxy comment: CONTAINERS+POST permits the full
container lifecycle (not just start/stop); state the real trust boundary.
- Document that the externally-defined mangalord-vision container must share
the compose network for name resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The analysis worker leases an analyze_page job (incrementing attempts in
SQL) before calling vision, so a stopped/loading vision burns all retries
and writes a permanent failed page_analysis row. With the vision-manager
autoscaler idle-stopping the container, that would poison the queue on
every cold start.
Add an optional VisionReadiness seam (HTTP GET /health in production) wired
via the new env-only ANALYSIS_VISION_HEALTH_URL. When set, the worker parks
without leasing until the probe answers 2xx; jobs stay pending with
attempts untouched and resume the instant vision is ready. None preserves
today's behavior for always-on endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Small vision models (qwen3-vl-4b) sometimes loop the OCR array, running
the response into the token ceiling (finish_reason: length) and failing to
parse. Layered guardrails:
- Prompts: explicit "transcribe each element once, never repeat/loop, at
most N, STOP when done" in the OCR, grounding and combined prompts.
- Schema: hard maxItems on ocr_results/tagging_results/content_type and
maxLength on text/scene — LM Studio's grammar enforces these, so a loop
is grammar-bounded rather than relying on max_tokens.
- Sampling: a configurable frequency_penalty (ANALYSIS_FREQUENCY_PENALTY,
default 0.3) sent with each request — the decode-time lever that actually
breaks loops.
- Code: sanitize() collapses runaway consecutive duplicates (same
normalized text + kind) so a loop that slips through still can't flood
the row.
Tests: schema caps present, frequency_penalty included only when nonzero,
sanitize collapses consecutive repeats; config default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Boundary text duplicated across slices (often with slightly different,
cropped transcriptions) survived the text-only merge. The OCR pass now
asks for a per-piece vertical position and the merge uses it:
- OcrResult gains optional `y` (fraction 0..1 of the slice); the Pass-A
OCR schema/prompt request it (combined path leaves it None). Not
persisted.
- merge_ocr takes each slice's working-image y-band and maps `y` to a
page-global position. A seam pair is a duplicate when position-close
(same kind) OR text-similar, so a mis-OCR'd boundary line is caught even
when the text differs. The kept copy is the one more central in its slice
(less cropped); falls back to keep-longer when positions are missing.
- Self-calibrates the model's y values (pixels vs. fraction) and ignores
degenerate columns, so a bad localizer can't over-merge.
Tests: position pairs differing texts and keeps the less-cropped copy;
text-only fallback (dedup/keep-longer/non-adjacent/order) still holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Very tall webtoon pages were squashed by the fixed longest-edge downscale,
so OCR was garbage. The vision client now slices to the model's pixel
budget at native resolution and assembles one result:
- plan_slices: native-resolution bands sized to max_pixels (portrait OR
landscape), only when height/width > tall_aspect_threshold; min_slice_
height guards pathologically wide pages (reduce width); max_slices caps
the count (coarser fallback). Normal pages still take one combined call.
- Two-pass, OCR-first: Pass A OCRs each band (near-native), merge_ocr
stitches them with seam-scoped fuzzy dedup (keep the longer transcript,
preserve order/kind, don't collapse non-adjacent repeats); Pass B feeds
the whole downscaled image + the merged OCR text to get tags/scene/safety
grounded in the real dialogue. Only OCR is merged.
- Config: replace max_image_dim with max_pixels / min_slice_height /
slice_overlap / tall_aspect_threshold / max_slices (+ env_f64); bump
job_timeout default 180->600 (N sequential slice calls per page); raise
MAX_OCR_PIECES 60->200. New prompts/schemas: OCR_PROMPT/ocr_json_schema,
GROUNDING_PROMPT/grounding_json_schema.
Tests: plan_slices (single/portrait/landscape/pathological-wide/cap +
coverage/overlap), merge_ocr (seam dedup, keep-longer, non-adjacent
repeats, order), render_whole/render_slice, the new request builders, and
updated config defaults/env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadcasts analysis progress so the dashboard updates live:
- analysis::events: AnalysisEvents broadcaster + AnalysisEvent
(Enqueued / Started / Completed / Failed), carrying the
manga/chapter/page breadcrumb.
- The worker daemon resolves each page's breadcrumb (repo::page::locate)
and publishes Started before dispatch and Completed/Failed after.
- The admin reenqueue publishes Enqueued (scoped by manga/chapter).
- GET /v1/admin/analysis/status/stream — SSE (RequireAdmin) forwarding
each event as a named `analysis` frame; broadcast lag emits a `lagged`
frame. AppState carries the always-present events bus.
Tests: worker publishes started+completed (with breadcrumb) and failed;
SSE route is admin-gated (403 non-admin) and returns text/event-stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The worker hardcoded response_format={type:json_object}, which LM Studio
rejects with 400 ('must be json_schema or text'); it also leaves
reasoning models (Gemma) with an empty `content`. Fix:
- Default to response_format=json_schema with the real analysis schema
(prompt::output_json_schema), which LM Studio accepts and which forces
clean JSON into message.content. Configurable via ANALYSIS_RESPONSE_FORMAT
= json_schema (default) | json_object | none (config::ResponseFormat).
- build_request_body is now a pure, unit-tested function.
- Vision errors now include the server's response body (status + text)
instead of a bare status, so a 400's reason is visible in logs.
Tests: response-format body shapes (none/json_object/json_schema), schema
shape + round-trip with the DTO, config parsing/defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAI-compatible vision client and the bounded prompt/output handling
for the analysis worker (no DB, fully unit-tested):
- analysis::prompt: terse system prompt carrying the JSON schema, the OCR
kind / content-warning vocabularies, and the output-size caps.
- analysis::vision: VisionClient (downscale via the new `image` dep →
base64 data-URL → chat/completions with an image_url part), plus pure
parse_chat_completion (fence/prose-tolerant) and sanitize (drop empty
OCR, truncate, clamp tags to 10 via the shared page-tag normalizer,
filter unknown warnings).
- config::AnalysisConfig extended with endpoint/model/api_key/timeouts/
max_tokens/max_image_dim/max_image_bytes + from_env.
- Deps: add `image` (jpeg/png/webp), reqwest `json` feature. Expose
api::page_tags::normalize_tag as pub(crate) for reuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
