The retention reaper deleted only `state = 'done'` rows, so terminal
`dead` jobs (retries exhausted) accumulated in crawler_jobs forever —
slow unbounded table growth. Both states are end-of-life and never
leased again.
Rename `reap_done` → `reap_terminal` and broaden the filter to
`state IN ('done','dead')`; `pending`/`running` stay untouched. Test
extended to assert old done + old dead both reap while fresh terminal and
old-but-active rows survive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`cargo clippy --all-targets` was failing on a deny-by-default
`never_loop` in the analysis SSE handler (the `loop` always returned on
the first iteration — the stream `unfold` already re-enters per event),
plus ~28 warnings. All resolved with no behaviour change:
- admin/analysis SSE: drop the dead `loop` wrapper.
- app: match port literals directly instead of `if p == 80` guards.
- repo/user: separate doc list from the following paragraphs.
- repo/upload_history: `sort_by_key(Reverse(..))` over `sort_by`.
- crawler/nav test: construct the error directly (no `unwrap_err` on a
literal `Err`).
- test helpers: build configs via struct-update syntax instead of
`Default::default()` + field reassignment; add a type alias for a
complex audit-row tuple.
- plus the mechanical `deref`/etc. fixes from `cargo clippy --fix`.
Note: not running `cargo fmt` — the backend uses a consistent
hand-formatted style that differs from rustfmt-default across ~110
files, so a blanket reformat would be pure churn.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds an in-process ocrs OCR backend as the active analysis engine
(vision left dormant), enables OCR text search on the page-tag
aggregation endpoints, and reshapes the admin analysis/settings UI to
present the OCR-only surface. Bumps version 0.89.0 -> 0.90.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sort the manga catalog by created / updated / title / author with an
independent asc/desc direction control. An omitted `order` defaults per field
(dates newest-first, text A→Z), matching the UI, so a bare `?sort=<field>`
means the same thing in the browser and over the API; `sort=recent` remains a
back-compat alias for `created`.
- Backend: SortField/SortOrder parsed with validation (structured 422 on bad
input), per-field default_order, NULLS LAST only on the nullable author key,
and migration 0033 indexing mangas(updated_at DESC, id) to back the default
sort and its id tie-break.
- Frontend: catalog sort field + direction (SegmentedControl) on desktop and a
mobile bottom sheet; pure helpers in $lib/mangaSort; keyboard-accessible
direction control; visible Direction labels and a sheet "Done" button.
- Tests: backend integration coverage (defaults, alias, invalid input,
ascending-id tie-break), frontend unit + e2e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.87.20 narrowed but did not close the force-analyze race:
ack_done / ack_failed / renew matched only on (id, state='running'),
so a worker A whose lease was released mid-flight (force-analyze,
reclaim_orphaned) could still come back later — once a successor B
re-leased the same id and put it back to state='running' — and
clobber B's lease. A's stale result became the row's `done` payload;
B's in-flight work was silently lost. The pre-existing test even
documented this hole: "We can't make ack_done's lease_id distinguish
A from B today".
Add `lease_generation BIGINT NOT NULL DEFAULT 0` to crawler_jobs
(migration 0032). It is bumped by every lease, release / release_unowned,
and reclaim_orphaned, so lease identity is `(id, generation)` rather
than just `id`. Each Lease struct carries its generation. ack_done /
ack_failed / renew / release match on `(id, generation, state='running')`
so a stale ack from a prior generation finds zero rows and falls
back to the existing warn-and-skip branch.
Split the release surface in two:
* release(lease_id, lease_generation) — owner-aware path for
workers (graceful shutdown, session-expired), matches the
caller's specific lease.
* release_unowned(lease_id) — id-only path for force-analyze and
ops tools that drop a running lease without holding a Lease
struct; unconditionally bumps generation so any pending ack
from the in-flight original becomes a no-op.
Force-analyze in repo::page_analysis now uses release_unowned; the
test in api_admin_analysis still passes unchanged (it only asserts
the steady-state outcome). The pre-existing `ack_done_no_ops_when_
lease_was_stolen` test is strengthened: it now mints both A and B
leases, asserts their generations differ, and verifies A's stale
ack does NOT clobber B's running row — the assertion the old test
explicitly couldn't make.
Five new tests in tests/crawler_jobs.rs pin every facet:
* lease_assigns_strictly_increasing_generation_per_release
* ack_done_from_dead_lease_after_release_unowned_is_a_no_op
* ack_failed_from_dead_lease_after_release_unowned_is_a_no_op
* renew_from_dead_lease_is_a_no_op
* release_unowned_bumps_generation_even_when_lease_is_held
Mutation-confirmed: relaxing the ack_done guard to
`lease_generation >= $2` (bind 0) makes the race test fail with
state=done — the exact prior-bug shape.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0.87.13 followup. The Cancelled publish had no asserting test —
mechanical revert of the publish block shipped green. New sqlx test
subscribes before spawn, drives SlowDispatcher into flight, cancels,
asserts both Started and Cancelled frames with breadcrumb fields.
Mutation-confirmed.
Also corrects: cron-test rationale comment (inverted unlock-axis),
Cancelled docstring (self-contradicting clear-triggers sentence).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.11 followup. The UPDATE matched pending+running but the worker
holds the pre-update `force=false` in an in-memory local — so on the
running branch the row flipped but the worker's skip-if-done still
acked done without re-analyzing. Now `jobs::release` any running row
the UPDATE matched, so the worker's ack_done no-ops (state guard) and
a fresh lease picks up the updated payload.
New test mints the running-state shape, asserts state=pending +
force=true + attempts=0 post-click. Mutation-confirmed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0.87.10 followup from the rereview. The fix lacked a test for the
specific axis it changed (admin authority via cookie only). New test
mints a bearer for a promoted admin and asserts PATCH/PUT-cover/DELETE-
cover all 403 on NULL-uploader rows. Mutation-confirmed.
Also: `MultipartBuilder::finalize` now `pub` for bearer-multipart use;
`admin_csrf_guard` rustdoc updated to describe the post-0.87.10
cookie-precedence rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two 0.87.4 follow-ups:
1. Cron tick: existing shutdown-on-cancel test now also asserts the
advisory lock is re-acquirable after cancel-during-tick. Without
this, a refactor moving `pg_advisory_unlock` under the cancel arm
would ship green and break multi-replica safety.
2. Analysis worker: cancel-mid-dispatch left the dashboard banner stuck
on the cancelled page until a later page overwrote it. Add a
`Cancelled` AnalysisEvent variant, publish on the cancel-release
branch, wire the analysis and admin overview dashboards to clear
on it. Per-page chip rolls back from `analyzing` to `queued`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four 0.87.6 follow-ups from the adversarial review:
1. **Migration 0031 pre-dedup.** Demote all-but-lowest-id duplicate
`analyze_page` rows in `(pending|running)` to `dead` before
creating the unique index, with a curator-recoverable last_error
marker. Without this, `sqlx::migrate!` would refuse to boot on
any dirty production DB.
2. **`enqueue_for_page(force=true)` collision.** The partial unique
index used to silently swallow force requests when a
`force=false` job was already pending. Repo function now upgrades
the pending row's `force` flag in place (or falls through to
re-INSERT if the sibling drained mid-call), and reports an
`EnqueueForPageOutcome` for accurate auditing.
3. **`record_duration` gate test.** New test seeds a `done` row with
known duration, force-re-analyzes with failing dispatcher +
max_attempts=3 (non-terminal), asserts duration_ms wasn't
overwritten.
4. **`bookmark.rs` PK comment correction.** Use `b.id DESC` instead
of the wrong `manga_id`; PK is actually `id`, and migration 0004
allows both chapter-level and manga-level bookmarks on the same
manga.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to 0.87.2 + 0.87.3 from the adversarial review:
1. **CSRF cookie-ride.** Bearer-bypass branch checked Authorization
header presence only — an attacker page could mint `Authorization:
Bearer junk` and ride the still-attached session cookie. Bypass now
requires bearer AND no cookie. Drive the cookie-name comparison off
`SESSION_COOKIE_NAME` and split on `=` (not prefix).
2. **`require_can_edit` admin path open to bearer.** Now reads admin
authority off an `Option<CurrentSessionUser>` extractor — None on
bearer-only requests. Owner-match path still accepts bearer; admin
path is cookie-gated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`require_can_edit` matched `Some(owner) != caller → Forbidden` but let
`None` through. Every crawler row (`repo::crawler::upsert_manga`) has
NULL `uploaded_by`, so any signed-in user could PATCH /api/v1/mangas/<id>
and rewrite the catalog — `update`, `put_cover`, and `delete_cover`
were all in scope. With self-registration on by default, the path was:
register → mass-edit catalog + delete cover blobs from storage.
The original carve-out at the comment said "Once an admin role lands the
NULL case can flip to admin-only." The admin role landed in migration
0018; flipping it now. New rule:
* `Some(owner)` and owner == caller → ok (unchanged)
* `Some(_)` and caller is admin → ok (admin moderation)
* `Some(_)` otherwise → 403 (unchanged)
* `None` and caller is admin → ok (NEW — operator can curate)
* `None` otherwise → 403 (was: ok — the bug)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four high/medium findings from the audit, plus their tests:
1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
`AnalysisSettings::endpoint` validated only with `Url::parse`. A
hostile or CSRF-able admin could repoint at `169.254.169.254`
(cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
— and the vision worker bearer-attaches an env-managed API key to
every call. Extract `ensure_public_target` from `safety::is_safe_url`
(allowlist-free public-host check: scheme + private-IP literal +
localhost) and wire it into both fields. Docker DNS names
(`mangalord-vision`) keep passing because they're hostnames, not
IP literals. The analysis check is gated on `enabled=true` so the
dev-default `localhost:8000` stays usable until the worker is
actually turned on (toggling enabled=true later re-runs the gate).
2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
silently skipped the entire CSRF check, so an operator forgetting
the env var shipped an unguarded admin surface. Cookie-auth POSTs
now fail-closed when the allowlist is empty AND when neither
`Origin` nor `Referer` accompanies the request. Two carve-outs:
`Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
and requests with NO session cookie at all bypass to let the auth
extractor return a clean 401 instead of a confusing 403.
3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
would exfiltrate page-image bytes + the bearer key through an
upstream proxy. Same for the readiness probe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Turn the dormant timing data in crawl_metrics / page_analysis into "is it
healthy over time" views. New bucketed series queries (GROUP BY date_trunc,
hour|day via a closed Bucket enum so the unit can't be attacker-controlled)
behind GET /v1/admin/{crawler,analysis}/metrics/series, with a shared
SeriesParams/resolve_bucket helper (bad bucket → 400) and migration 0030
indexing page_analysis(analyzed_at) for the analysis scan.
Frontend: a dependency-free SVG TrendChart (line+area, null buckets render as
gaps, empty-state, role=img) embedded above the per-op tables in Crawler and
Analysis → Metrics, driven by each panel's existing window selector with
AbortController-cancelled fetches. A buildSeries() util fills the continuous
bucket axis (throughput 0 for empty buckets, success/duration null) — unit
tested alongside the chart and the series API client.
Closes the Phase-1 observability set (audit log · health checks · trends).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New /admin/health rolls up signals already collected — disk/memory sensors,
dead-job + missing-cover backlogs, 24h crawl/analysis failure rates, metadata
cron freshness, and the live crawler session/browser flags — into a flat
checks list (ok/warn/critical) with an overall status and remediation links.
The overview gains a compact Health summary banner linking in.
GET /v1/admin/health evaluates the checks (DB sub-queries run concurrently via
try_join; all hit existing indexes). PUT /v1/admin/health/thresholds persists
the thresholds to app_settings (merged over compiled defaults, forward-compat
via serde(default)), validates ranges (400 on bad input), and audit-logs the
change in one transaction. New repo::crawler::last_metadata_tick_at getter and
a lightweight system::memory_percent_used (no CPU settle delay).
Pure status helpers (over_limit for gauges, exceeds for backlog counts so a
0-limit doesn't false-alarm at 0) are unit-tested; endpoint tests cover shape,
gating, persistence+audit, and validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface the admin_audit table (written by every mutating admin action but
previously unreadable) as a new /admin/audit tab. New repo::admin_audit::list
(LEFT JOIN users for the actor's username, filter by action/target_kind/actor/
since, at DESC via admin_audit_at_idx) behind GET /v1/admin/audit, mirroring the
crawler history endpoint's paged/clamped idiom.
Frontend: a self-contained AuditTable (target-kind + window + action filters,
expandable JSON payload, AbortController-cancelled fetches, loading/error/empty
states) and shared fmtAgo() in format.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interleaved metadata pass misses mangas to list drift (a title slips a
pagination slot during the slow detail walk). Add a reconcile pass: a cheap,
full, list-only walk (refs only, no detail visit, no early stop) that
set-diffs the walked keys against manga_sources and enqueues the strictly-
missing ones as SyncManga jobs. A set-diff is immune to drift — a manga only
has to appear somewhere in the list.
- Build the previously-dead SyncManga worker by extending RealChapterDispatcher
to run the shared pipeline::process_manga_ref (fetch → upsert → cover →
chapters), refactored out of run_metadata_pass so both paths stay in lockstep.
The crawl worker now leases both sync_chapter_content and sync_manga
(jobs::lease_kinds); both serialize on the single exclusive browser.
- SyncManga payload carries url + title so the worker can rebuild the ref and
the dead-jobs/history UI can label a missing manga that has no manga row yet.
- "Missing" = strict NOT EXISTS (dropped rows count as present). Re-enqueue is
skipped when a pending/running/dead SyncManga job already exists, so a gone
manga (detail 404 → retries → dead) is left dead and not retried.
- New POST /v1/admin/crawler/reconcile (fire-and-forget, shares
manual_pass_lock) + "Reconcile missing" admin button + Reconciling status
phase streamed over SSE.
- dead-jobs/history queries surface payload title/url/key; tables fall back to
them for sync_manga rows; both searches match the payload title.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up to the UA fix — the ignored browser smoke test builds LaunchOptions
as a struct literal and needs the new user_agent field to compile.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three failure-mode fixes surfaced by a crawler recovery audit:
- Graceful shutdown mid-dispatch now releases the in-flight job back to
pending without burning a retry attempt. process_lease wraps the
dispatch in a biased tokio::select! cancel arm that aborts the
heartbeat and calls jobs::release; previously a mid-job SIGTERM left
the row 'running' until lease expiry and cost one of max_attempts.
- Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired
lease back to pending (attempt refunded), run once at daemon startup
before workers start. Crash recovery is now immediate instead of
waiting a full lease window for the lazy lease-expiry path. Safe under
multi-replica: only already-expired leases are touched, which a
healthy heartbeating peer never has.
- Manga-list parsing now warns when listing anchors are dropped for a
missing/empty href or title (split into parse_manga_list_anchors so
the drop count is testable), turning silent source markup drift into
an observable signal. Returned refs are byte-identical to before.
Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt,
reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit
test. Patch bump 0.81.0 -> 0.81.1 (both manifests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two requeue-audit tests in api_admin_crawler ordered by created_at, but
the admin_audit table's timestamp column is 'at' (migration 0019), so the
queries failed at runtime with 42703 (column does not exist). This was the
last pre-0.81 test breakage gating deploy; full suite now green locally
(cargo test --no-fail-fast).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The admin-settings refactor replaced AppState's resync/crawler/
analysis_enabled fields with runtime/reloader/crawler_base/
analysis_base, but tests/api_admin_role.rs still built the old shape,
so `cargo test` failed to compile (E0560) and blocked every deploy
since 0.80.0. Mirror the no-daemon harness in tests/common/mod.rs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The analysis worker leases an analyze_page job (incrementing attempts in
SQL) before calling vision, so a stopped/loading vision burns all retries
and writes a permanent failed page_analysis row. With the vision-manager
autoscaler idle-stopping the container, that would poison the queue on
every cold start.
Add an optional VisionReadiness seam (HTTP GET /health in production) wired
via the new env-only ANALYSIS_VISION_HEALTH_URL. When set, the worker parks
without leasing until the probe answers 2xx; jobs stay pending with
attempts untouched and resume the instant vision is ready. None preserves
today's behavior for always-on endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Boundary text duplicated across slices (often with slightly different,
cropped transcriptions) survived the text-only merge. The OCR pass now
asks for a per-piece vertical position and the merge uses it:
- OcrResult gains optional `y` (fraction 0..1 of the slice); the Pass-A
OCR schema/prompt request it (combined path leaves it None). Not
persisted.
- merge_ocr takes each slice's working-image y-band and maps `y` to a
page-global position. A seam pair is a duplicate when position-close
(same kind) OR text-similar, so a mis-OCR'd boundary line is caught even
when the text differs. The kept copy is the one more central in its slice
(less cropped); falls back to keep-longer when positions are missing.
- Self-calibrates the model's y values (pixels vs. fraction) and ignores
degenerate columns, so a bad localizer can't over-merge.
Tests: position pairs differing texts and keeps the less-cropped copy;
text-only fallback (dedup/keep-longer/non-adjacent/order) still holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadcasts analysis progress so the dashboard updates live:
- analysis::events: AnalysisEvents broadcaster + AnalysisEvent
(Enqueued / Started / Completed / Failed), carrying the
manga/chapter/page breadcrumb.
- The worker daemon resolves each page's breadcrumb (repo::page::locate)
and publishes Started before dispatch and Completed/Failed after.
- The admin reenqueue publishes Enqueued (scoped by manga/chapter).
- GET /v1/admin/analysis/status/stream — SSE (RequireAdmin) forwarding
each event as a named `analysis` frame; broadcast lag emits a `lagged`
frame. AppState carries the always-present events bus.
Tests: worker publishes started+completed (with breadcrumb) and failed;
SSE route is admin-gated (403 non-admin) and returns text/event-stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes the admin re-enqueue beyond global backfill:
- repo::page_analysis::enqueue_pages(scope, only_unanalyzed) with
ReenqueueScope::{All,Manga,Chapter}. Fixes include-analyzed semantics:
only_unanalyzed=false now enqueues jobs with force=true so the worker
actually re-analyzes already-done pages instead of skipping them.
- POST /v1/admin/analysis/reenqueue accepts optional manga_id / chapter_id
(mutually exclusive, 404 on unknown target) alongside only_unanalyzed.
Tests: manga/chapter scope counts, include-analyzed sets force=true on the
done page, mutual-exclusivity 422, unknown-target 404. Existing global
backfill tests still green (13 total).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the analysis worker's NSFW moderation at the manga level:
- repo::manga FILTER_WHERE gains cw_include (AND, page-content-warning
join to mangas) and cw_exclude (NONE) clauses; ListQuery + binds
renumbered ($6/$7, LIMIT/OFFSET $8/$9).
- repo::page_analysis::warnings_for_manga: deduped, alphabetical union
across all the manga's pages.
- domain::MangaDetail gains content_warnings; get_detail populates it.
- api::mangas list accepts cw_include/cw_exclude (reusing the shared
parse_warnings_csv validator).
Tests: detail union (deduped/sorted) + empty case; list include/exclude
filter; unknown-warning 422. Existing manga tests still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New GET /v1/me/page-search surfacing the analysis worker's output:
- repo::page_analysis::page_search + PageSearchQuery: multi-tag AND across
(user page_tags ∪ global page_auto_tags) via the unnest double-negative
idiom, weighted OCR/scene text ranking (ts_rank over search_doc), and
content-warning include/exclude. One row per page with is_nsfw +
deduped content_warnings + rank.
- domain::PageSearchItem.
- api::page_tags: /me/page-search handler with CSV tag/warning parsing
(parse_tags_csv reuses normalize_tag; parse_warnings_csv validates the
closed vocabulary), requiring at least one positive filter (422 else).
This is where the reserved OCR text search lands for pages.
Tests: multi-tag AND user∪auto, speech>sfx ranking, cw include/exclude
(+ row flags), text-only (no tags), missing-filter 422, unknown-warning
422, auth required.
Note: the /me/page-tags/chapters|mangas aggregations keep single-tag
behavior + the reserved text=501 for now; page-level search is the
primary text surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the analyze_page job kind into the page-create paths and adds admin
controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled,
config::AnalysisConfig):
- Chapter upload (api::chapters) enqueues one analyze_page job per page
after the tx commits (best-effort; failures logged, never fatal).
- Crawler content sync (persist_pages → RETURNING id) enqueues per
persisted page when the daemon dispatcher's analysis_enabled flag is set;
threaded through sync_chapter_content (CLI/resync pass false).
- repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT,
skipping done pages (only_unanalyzed) and existing pending jobs.
- New admin endpoints (RequireAdmin, 503 when disabled, audited):
POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force).
Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages
enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503,
non-admin 403) and force-analyze (force flag, 404).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the persistence foundation for the AI content-analysis worker:
- Migration 0025: page_analysis (status + scene + is_nsfw + weighted
search_doc tsvector), page_ocr_text (kind-tagged pieces), global
page_auto_tags (shared tags vocabulary, separate from per-user
page_tags), and page_content_warnings (closed vocabulary).
- domain::page_analysis: AnalysisStatus/OcrKind/ContentWarning enums with
kind->tsvector-weight mapping and lenient model-string parsing, plus the
VisionAnalysis response DTOs.
- JobPayload::AnalyzePage { page_id, force } + KIND_ANALYZE_PAGE, reusing
the existing crawler_jobs queue.
- repo::page_analysis: enqueue_for_page, load, mark_failed, and the single
transactional persist_analysis (delete+reinsert; idempotent; never
touches user page_tags; computes the A/B/C/D-weighted search_doc).
Tests: repo integration (persist/idempotency/user-tags-untouched/enqueue/
mark_failed/speech>sfx ranking) + unit (weights, parsing, DTO, job serde).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
page_tags previously stored the tag inline as free-form text — the lone
un-normalized tag concept, while authors/genres/manga-tags all went
through a lookup table + FK. Fold page tags into the SAME shared `tags`
table that manga_tags uses: `page_tags.tag` becomes `page_tags.tag_id`
referencing `tags(id)`. Manga tags and page tags now share one global
vocabulary.
The HTTP contract is unchanged — the API still speaks tag *names*; the
repo resolves name<->id via `repo::tag::upsert_by_name` (the manga-tag
helper) and keeps the stricter page-tag `normalize_tag` at the boundary.
Frontend, DTOs, and response shapes are untouched. User-visible effect:
page-tag names now appear in the manga-tag autocomplete and vice versa.
Migration 0024 backfills `tags` from existing inline values (deduping
case-insensitively), repoints rows, swaps the unique/secondary indexes
to tag_id, and drops the inline column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /v1/mangas/:id/similar returning the top 5 mangas ranked by
Jaccard tag overlap (shared / union), excluding self, as MangaCard items
in a plain { items } object. Surface them in a hidden-when-empty
"Similar" section on the manga detail page, reusing MangaCard.
Backend: new repo::manga::list_similar with a manga_tags self-join; a
shared cards_from_rows helper (also used by list_cards) that loads
authors+genres concurrently; single-sourced column list via MANGA_COLS +
manga_cols(alias) to replace the fragile SELECT_COLS string-splitting.
Frontend: getSimilarMangas client fn (guards a malformed body), loader
fetch with non-critical .catch fallback, and the catalog grid hoisted to
a shared global .manga-grid (lib/styles/tokens.css), de-duplicating the
per-route copies.
Bump version 0.62.0 -> 0.63.0 (backend + frontend in lockstep).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the /search surface (Pages / Chapters / Mangas tabs) backed by
per-user page tags and per-page collections: schema (migration 0023),
backend endpoints for page tags/collections and tagged-page aggregations
(with the OCR text-search param reserved at 501), plus the frontend API
clients, library Page-tags tab, collection page sections, page context
menu / AddTagsSheet, and reader long-press wiring. Includes the
continuous-reader navigation fixes (?page=N handling, chapter-reset
timing, back-button pops history) and tag-normalization hardening
accumulated on the branch.
Bump version 0.60.2 -> 0.62.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The user-facing chapter list ordered by (number ASC, created_at ASC),
which broke the source site's order in two ways: non-numeric entries
("notice. : Officials") parsed to number=0 and clustered at the top,
even though the site placed them mid-list, and variants sharing a
number ("Ch.14 : PH" / "Ch.14 : Official") were torn apart by the
created_at tiebreak.
Capture each chapter's position in the source DOM as `source_index`
(0 = first = newest on this site) on every crawler sync, including the
UPDATE branch so a new chapter prepended on the source shifts every
existing row down by one on the next tick. The list query reverses
this with `ORDER BY source_index DESC NULLS LAST, number ASC,
created_at ASC` so the oldest chapter appears first, variants stay
adjacent in the order the site shows them, and non-numeric entries
land where the site placed them. User-uploaded chapters and pre-
migration rows keep their NULL source_index and fall through to the
prior number/created_at tiebreak via NULLS LAST.
The reader's client-side `[...chapters].sort((a,b) => a.number - b.number)`
is dropped; prev/next now walks the server-ordered array positionally
so it traverses variants and non-numeric entries in display order.
Existing data populates on the next cron tick or via admin force-resync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both enqueue paths now order by chapters.number so the cron tick and the
bookmark hook insert jobs from chapter 1 upward instead of source-discovery
or random-UUID order. The lease query tiebreaks on created_at so jobs
sharing a batch's scheduled_at come off the queue in insertion order,
propagating the enqueue intent through to dequeue. Concurrent workers
and per-CDN latency can still drift actual completion order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tick cover-backfill pass to the crawler daemon so mangas whose
cover download failed on first attempt get retried — the metadata pass's
early-stop optimisation otherwise prevents the walk from revisiting them.
Adds admin-only POST /admin/mangas/:id/resync and POST /admin/chapters/:id/resync
that refetch metadata + cover (or chapter content with force_refetch) from the
crawler source synchronously and return the refreshed row. Surfaced in the
UI as "Force resync" buttons on the manga detail and reader pages,
admin-only via session.user.is_admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
