Persist blob sizes (covers + chapter pages) and surface upload storage
usage to admins in two places:
- Admin System tab: total/covers/chapters totals, ratio of disk, average
image sizes, and top-5 largest mangas/chapters leaderboards, behind a
new GET /api/v1/admin/storage endpoint (separate from /admin/system so
the cheap DB aggregates aren't coupled to its 250ms CPU sample).
- Manga detail page: total chapter-content size and per-chapter size,
with an em-dash for uncrawled or not-yet-measured chapters.
Sizes are captured at write time (crawler put_stream return, uploaded
page/cover byte length) into nullable pages.size_bytes /
mangas.cover_size_bytes columns; NULL means "not yet measured", distinct
from a real 0. A one-shot, idempotent admin "Backfill sizes" action
(POST /api/v1/admin/storage/backfill) stats pre-existing blobs in
keyset-paginated, capped batches and reports more_remaining so a large
legacy library can be drained over multiple runs.
Aggregates and leaderboards treat NULL honestly (only fully-measured
entities are ranked); a dashboard banner flags unmeasured rows so partial
figures aren't read as complete. persist_pages now prunes stale page rows
on a shrinking re-crawl so totals and page_count stay consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The crawler's headless Chromium advertised the default HeadlessChrome UA,
which Cloudflare's bot detection challenges on sight — over Tor that meant an
unsolvable 'Just a moment' page on every fetch, so session verification kept
classifying pages as transient and the metadata pass failed at
'acquire browser lease'. (Confirmed empirically: same Tor exit + a normal
Chrome UA returns the real page with the #logo sentinel; the HeadlessChrome
UA returns the challenge.) The existing user_agent setting only fed reqwest,
never the browser. Override the launcher's UA with a realistic non-headless
default (CRAWLER_USER_AGENT to customize); the UA may contain spaces, unlike
CRAWLER_BROWSER_ARGS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three failure-mode fixes surfaced by a crawler recovery audit:
- Graceful shutdown mid-dispatch now releases the in-flight job back to
pending without burning a retry attempt. process_lease wraps the
dispatch in a biased tokio::select! cancel arm that aborts the
heartbeat and calls jobs::release; previously a mid-job SIGTERM left
the row 'running' until lease expiry and cost one of max_attempts.
- Boot-time jobs::reclaim_orphaned resets 'running' jobs with an expired
lease back to pending (attempt refunded), run once at daemon startup
before workers start. Crash recovery is now immediate instead of
waiting a full lease window for the lazy lease-expiry path. Safe under
multi-replica: only already-expired leases are touched, which a
healthy heartbeating peer never has.
- Manga-list parsing now warns when listing anchors are dropped for a
missing/empty href or title (split into parse_manga_list_anchors so
the drop count is testable), turning silent source markup drift into
an observable signal. Returned refs are byte-identical to before.
Tests added: shutdown_mid_dispatch_releases_lease_without_burning_attempt,
reclaim_orphaned_resets_only_expired_running_jobs, and a drop-count unit
test. Patch bump 0.81.0 -> 0.81.1 (both manifests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The private_mode_* config unit tests set then remove_var(DATABASE_URL).
That env is process-global, and the #[sqlx::test] lib tests
(crawler::content, crawler::session_control, ...) read it in parallel,
so on a multi-core runner they intermittently saw it unset and panicked
'DATABASE_URL must be set: NotPresent' (6 failures on the Pi's 4 cores).
Never overwrite/remove it: set only if absent via ensure_database_url(),
leaving CI's real URL stable for the parallel DB tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend:
- Log on readiness state transitions (ready<->not-ready) so a misconfigured
ANALYSIS_VISION_HEALTH_URL, which otherwise parks the worker silently
forever, is diagnosable.
- Add unit tests for HttpVisionReadiness: 2xx->ready, non-2xx->not-ready,
connection error->not-ready (the production status mapping had no test).
vision-manager:
- Guard the backlog count against non-numeric output before `-gt`, so a stray
value can't exit-2 and kill the loop under `set -e`.
- Throttle the crawl-mutex "deferring start" log to once per episode.
- Only reset the idle/uptime timers when `docker stop` actually succeeds, so a
failed stop retries next tick instead of waiting a full debounce window.
- Decouple the per-probe curl timeout (HEALTH_TIMEOUT) from the warm-up poll
cadence.
Docs:
- Correct the docker-socket-proxy comment: CONTAINERS+POST permits the full
container lifecycle (not just start/stop); state the real trust boundary.
- Document that the externally-defined mangalord-vision container must share
the compose network for name resolution.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The analysis worker leases an analyze_page job (incrementing attempts in
SQL) before calling vision, so a stopped/loading vision burns all retries
and writes a permanent failed page_analysis row. With the vision-manager
autoscaler idle-stopping the container, that would poison the queue on
every cold start.
Add an optional VisionReadiness seam (HTTP GET /health in production) wired
via the new env-only ANALYSIS_VISION_HEALTH_URL. When set, the worker parks
without leasing until the probe answers 2xx; jobs stay pending with
attempts untouched and resume the instant vision is ready. None preserves
today's behavior for always-on endpoints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move crawler and analysis configuration out of boot-only env vars and into
a DB-backed, admin-editable surface applied live without a restart.
- New `app_settings` table (migration 0026) holds one JSONB row per group;
env vars seed it on first boot, then the DB is the source of truth.
- `settings.rs` adds serializable Crawler/Analysis DTOs with env-base +
DB-overlay conversion and field-level validation; env-only/secret fields
(browser, proxy, TOR, vision API key, PHPSESSID) are never persisted.
- `GET/PUT /api/v1/admin/settings/{crawler,analysis}` (RequireAdmin,
cookie-only, CSRF-guarded): GET returns the editable DTO + a read-only
env-managed view + prompt defaults; PUT validates (422 + per-field
details), persists + audits in one tx, then gracefully respawns the
affected daemon via a Supervisor.
- AppState swaps the crawler control/resync handles and analysis enable
gate behind shared runtime cells so reloads take effect with no restart;
a boot-time spawn failure is logged rather than aborting startup.
- Make the three vision prompts and sampling temperature configurable
(defaults preserved); cookie_domain is admin-editable too.
- Frontend: tabbed /admin/settings page with grouped fieldsets, prompt
reset-to-default, env-managed read-only panel, save-&-apply confirm, and
per-field validation; typed client + ApiError.details.
- Bump version 0.79.1 -> 0.80.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Small vision models (qwen3-vl-4b) sometimes loop the OCR array, running
the response into the token ceiling (finish_reason: length) and failing to
parse. Layered guardrails:
- Prompts: explicit "transcribe each element once, never repeat/loop, at
most N, STOP when done" in the OCR, grounding and combined prompts.
- Schema: hard maxItems on ocr_results/tagging_results/content_type and
maxLength on text/scene — LM Studio's grammar enforces these, so a loop
is grammar-bounded rather than relying on max_tokens.
- Sampling: a configurable frequency_penalty (ANALYSIS_FREQUENCY_PENALTY,
default 0.3) sent with each request — the decode-time lever that actually
breaks loops.
- Code: sanitize() collapses runaway consecutive duplicates (same
normalized text + kind) so a loop that slips through still can't flood
the row.
Tests: schema caps present, frequency_penalty included only when nonzero,
sanitize collapses consecutive repeats; config default.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Boundary text duplicated across slices (often with slightly different,
cropped transcriptions) survived the text-only merge. The OCR pass now
asks for a per-piece vertical position and the merge uses it:
- OcrResult gains optional `y` (fraction 0..1 of the slice); the Pass-A
OCR schema/prompt request it (combined path leaves it None). Not
persisted.
- merge_ocr takes each slice's working-image y-band and maps `y` to a
page-global position. A seam pair is a duplicate when position-close
(same kind) OR text-similar, so a mis-OCR'd boundary line is caught even
when the text differs. The kept copy is the one more central in its slice
(less cropped); falls back to keep-longer when positions are missing.
- Self-calibrates the model's y values (pixels vs. fraction) and ignores
degenerate columns, so a bad localizer can't over-merge.
Tests: position pairs differing texts and keeps the less-cropped copy;
text-only fallback (dedup/keep-longer/non-adjacent/order) still holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Very tall webtoon pages were squashed by the fixed longest-edge downscale,
so OCR was garbage. The vision client now slices to the model's pixel
budget at native resolution and assembles one result:
- plan_slices: native-resolution bands sized to max_pixels (portrait OR
landscape), only when height/width > tall_aspect_threshold; min_slice_
height guards pathologically wide pages (reduce width); max_slices caps
the count (coarser fallback). Normal pages still take one combined call.
- Two-pass, OCR-first: Pass A OCRs each band (near-native), merge_ocr
stitches them with seam-scoped fuzzy dedup (keep the longer transcript,
preserve order/kind, don't collapse non-adjacent repeats); Pass B feeds
the whole downscaled image + the merged OCR text to get tags/scene/safety
grounded in the real dialogue. Only OCR is merged.
- Config: replace max_image_dim with max_pixels / min_slice_height /
slice_overlap / tall_aspect_threshold / max_slices (+ env_f64); bump
job_timeout default 180->600 (N sequential slice calls per page); raise
MAX_OCR_PIECES 60->200. New prompts/schemas: OCR_PROMPT/ocr_json_schema,
GROUNDING_PROMPT/grounding_json_schema.
Tests: plan_slices (single/portrait/landscape/pathological-wide/cap +
coverage/overlap), merge_ocr (seam dedup, keep-longer, non-adjacent
repeats, order), render_whole/render_slice, the new request builders, and
updated config defaults/env.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A text-dense page's full analysis JSON exceeded the 900-token output cap,
so the model stopped mid-object (finish_reason: length) and the truncated
response failed to parse ("EOF while parsing a list"). The page image +
prompt are only a few hundred tokens, so a larger output budget still fits
comfortably in an 8k context window. Still overridable via
ANALYSIS_MAX_TOKENS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Broadcasts analysis progress so the dashboard updates live:
- analysis::events: AnalysisEvents broadcaster + AnalysisEvent
(Enqueued / Started / Completed / Failed), carrying the
manga/chapter/page breadcrumb.
- The worker daemon resolves each page's breadcrumb (repo::page::locate)
and publishes Started before dispatch and Completed/Failed after.
- The admin reenqueue publishes Enqueued (scoped by manga/chapter).
- GET /v1/admin/analysis/status/stream — SSE (RequireAdmin) forwarding
each event as a named `analysis` frame; broadcast lag emits a `lagged`
frame. AppState carries the always-present events bus.
Tests: worker publishes started+completed (with breadcrumb) and failed;
SSE route is admin-gated (403 non-admin) and returns text/event-stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The worker hardcoded response_format={type:json_object}, which LM Studio
rejects with 400 ('must be json_schema or text'); it also leaves
reasoning models (Gemma) with an empty `content`. Fix:
- Default to response_format=json_schema with the real analysis schema
(prompt::output_json_schema), which LM Studio accepts and which forces
clean JSON into message.content. Configurable via ANALYSIS_RESPONSE_FORMAT
= json_schema (default) | json_object | none (config::ResponseFormat).
- build_request_body is now a pure, unit-tested function.
- Vision errors now include the server's response body (status + text)
instead of a bare status, so a 400's reason is visible in logs.
Tests: response-format body shapes (none/json_object/json_schema), schema
shape + round-trip with the DTO, config parsing/defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes the admin re-enqueue beyond global backfill:
- repo::page_analysis::enqueue_pages(scope, only_unanalyzed) with
ReenqueueScope::{All,Manga,Chapter}. Fixes include-analyzed semantics:
only_unanalyzed=false now enqueues jobs with force=true so the worker
actually re-analyzes already-done pages instead of skipping them.
- POST /v1/admin/analysis/reenqueue accepts optional manga_id / chapter_id
(mutually exclusive, 404 on unknown target) alongside only_unanalyzed.
Tests: manga/chapter scope counts, include-analyzed sets force=true on the
done page, mutual-exclusivity 422, unknown-target 404. Existing global
backfill tests still green (13 total).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the analysis worker's NSFW moderation at the manga level:
- repo::manga FILTER_WHERE gains cw_include (AND, page-content-warning
join to mangas) and cw_exclude (NONE) clauses; ListQuery + binds
renumbered ($6/$7, LIMIT/OFFSET $8/$9).
- repo::page_analysis::warnings_for_manga: deduped, alphabetical union
across all the manga's pages.
- domain::MangaDetail gains content_warnings; get_detail populates it.
- api::mangas list accepts cw_include/cw_exclude (reusing the shared
parse_warnings_csv validator).
Tests: detail union (deduped/sorted) + empty case; list include/exclude
filter; unknown-warning 422. Existing manga tests still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New GET /v1/me/page-search surfacing the analysis worker's output:
- repo::page_analysis::page_search + PageSearchQuery: multi-tag AND across
(user page_tags ∪ global page_auto_tags) via the unnest double-negative
idiom, weighted OCR/scene text ranking (ts_rank over search_doc), and
content-warning include/exclude. One row per page with is_nsfw +
deduped content_warnings + rank.
- domain::PageSearchItem.
- api::page_tags: /me/page-search handler with CSV tag/warning parsing
(parse_tags_csv reuses normalize_tag; parse_warnings_csv validates the
closed vocabulary), requiring at least one positive filter (422 else).
This is where the reserved OCR text search lands for pages.
Tests: multi-tag AND user∪auto, speech>sfx ranking, cw include/exclude
(+ row flags), text-only (no tags), missing-filter 422, unknown-warning
422, auth required.
Note: the /me/page-tags/chapters|mangas aggregations keep single-tag
behavior + the reserved text=501 for now; page-level search is the
primary text surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAI-compatible vision client and the bounded prompt/output handling
for the analysis worker (no DB, fully unit-tested):
- analysis::prompt: terse system prompt carrying the JSON schema, the OCR
kind / content-warning vocabularies, and the output-size caps.
- analysis::vision: VisionClient (downscale via the new `image` dep →
base64 data-URL → chat/completions with an image_url part), plus pure
parse_chat_completion (fence/prose-tolerant) and sanitize (drop empty
OCR, truncate, clamp tags to 10 via the shared page-tag normalizer,
filter unknown warnings).
- config::AnalysisConfig extended with endpoint/model/api_key/timeouts/
max_tokens/max_image_dim/max_image_bytes + from_env.
- Deps: add `image` (jpeg/png/webp), reqwest `json` feature. Expose
api::page_tags::normalize_tag as pub(crate) for reuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the analyze_page job kind into the page-create paths and adds admin
controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled,
config::AnalysisConfig):
- Chapter upload (api::chapters) enqueues one analyze_page job per page
after the tx commits (best-effort; failures logged, never fatal).
- Crawler content sync (persist_pages → RETURNING id) enqueues per
persisted page when the daemon dispatcher's analysis_enabled flag is set;
threaded through sync_chapter_content (CLI/resync pass false).
- repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT,
skipping done pages (only_unanalyzed) and existing pending jobs.
- New admin endpoints (RequireAdmin, 503 when disabled, audited):
POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force).
Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages
enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503,
non-admin 403) and force-analyze (force flag, 404).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the persistence foundation for the AI content-analysis worker:
- Migration 0025: page_analysis (status + scene + is_nsfw + weighted
search_doc tsvector), page_ocr_text (kind-tagged pieces), global
page_auto_tags (shared tags vocabulary, separate from per-user
page_tags), and page_content_warnings (closed vocabulary).
- domain::page_analysis: AnalysisStatus/OcrKind/ContentWarning enums with
kind->tsvector-weight mapping and lenient model-string parsing, plus the
VisionAnalysis response DTOs.
- JobPayload::AnalyzePage { page_id, force } + KIND_ANALYZE_PAGE, reusing
the existing crawler_jobs queue.
- repo::page_analysis: enqueue_for_page, load, mark_failed, and the single
transactional persist_analysis (delete+reinsert; idempotent; never
touches user page_tags; computes the A/B/C/D-weighted search_doc).
Tests: repo integration (persist/idempotency/user-tags-untouched/enqueue/
mark_failed/speech>sfx ranking) + unit (weights, parsing, DTO, job serde).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
page_tags previously stored the tag inline as free-form text — the lone
un-normalized tag concept, while authors/genres/manga-tags all went
through a lookup table + FK. Fold page tags into the SAME shared `tags`
table that manga_tags uses: `page_tags.tag` becomes `page_tags.tag_id`
referencing `tags(id)`. Manga tags and page tags now share one global
vocabulary.
The HTTP contract is unchanged — the API still speaks tag *names*; the
repo resolves name<->id via `repo::tag::upsert_by_name` (the manga-tag
helper) and keeps the stricter page-tag `normalize_tag` at the boundary.
Frontend, DTOs, and response shapes are untouched. User-visible effect:
page-tag names now appear in the manga-tag autocomplete and vice versa.
Migration 0024 backfills `tags` from existing inline values (deduping
case-insensitively), repoints rows, swaps the unique/secondary indexes
to tag_id, and drops the inline column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /v1/mangas/:id/similar returning the top 5 mangas ranked by
Jaccard tag overlap (shared / union), excluding self, as MangaCard items
in a plain { items } object. Surface them in a hidden-when-empty
"Similar" section on the manga detail page, reusing MangaCard.
Backend: new repo::manga::list_similar with a manga_tags self-join; a
shared cards_from_rows helper (also used by list_cards) that loads
authors+genres concurrently; single-sourced column list via MANGA_COLS +
manga_cols(alias) to replace the fragile SELECT_COLS string-splitting.
Frontend: getSimilarMangas client fn (guards a malformed body), loader
fetch with non-critical .catch fallback, and the catalog grid hoisted to
a shared global .manga-grid (lib/styles/tokens.css), de-duplicating the
per-route copies.
Bump version 0.62.0 -> 0.63.0 (backend + frontend in lockstep).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the /search surface (Pages / Chapters / Mangas tabs) backed by
per-user page tags and per-page collections: schema (migration 0023),
backend endpoints for page tags/collections and tagged-page aggregations
(with the OCR text-search param reserved at 501), plus the frontend API
clients, library Page-tags tab, collection page sections, page context
menu / AddTagsSheet, and reader long-press wiring. Includes the
continuous-reader navigation fixes (?page=N handling, chapter-reset
timing, back-button pops history) and tag-normalization hardening
accumulated on the branch.
Bump version 0.60.2 -> 0.62.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The user-facing chapter list ordered by (number ASC, created_at ASC),
which broke the source site's order in two ways: non-numeric entries
("notice. : Officials") parsed to number=0 and clustered at the top,
even though the site placed them mid-list, and variants sharing a
number ("Ch.14 : PH" / "Ch.14 : Official") were torn apart by the
created_at tiebreak.
Capture each chapter's position in the source DOM as `source_index`
(0 = first = newest on this site) on every crawler sync, including the
UPDATE branch so a new chapter prepended on the source shifts every
existing row down by one on the next tick. The list query reverses
this with `ORDER BY source_index DESC NULLS LAST, number ASC,
created_at ASC` so the oldest chapter appears first, variants stay
adjacent in the order the site shows them, and non-numeric entries
land where the site placed them. User-uploaded chapters and pre-
migration rows keep their NULL source_index and fall through to the
prior number/created_at tiebreak via NULLS LAST.
The reader's client-side `[...chapters].sort((a,b) => a.number - b.number)`
is dropped; prev/next now walks the server-ordered array positionally
so it traverses variants and non-numeric entries in display order.
Existing data populates on the next cron tick or via admin force-resync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both enqueue paths now order by chapters.number so the cron tick and the
bookmark hook insert jobs from chapter 1 upward instead of source-discovery
or random-UUID order. The lease query tiebreaks on created_at so jobs
sharing a batch's scheduled_at come off the queue in insertion order,
propagating the enqueue intent through to dequeue. Concurrent workers
and per-CDN latency can still drift actual completion order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tick cover-backfill pass to the crawler daemon so mangas whose
cover download failed on first attempt get retried — the metadata pass's
early-stop optimisation otherwise prevents the walk from revisiting them.
Adds admin-only POST /admin/mangas/:id/resync and POST /admin/chapters/:id/resync
that refetch metadata + cover (or chapter content with force_refetch) from the
crawler source synchronously and return the refreshed row. Surfaced in the
UI as "Force resync" buttons on the manga detail and reader pages,
admin-only via session.user.is_admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When `PRIVATE_MODE=true`, every API path except a small allowlist
(`/health`, `/auth/{config,login,logout,register}`) requires a valid
session cookie or bearer token — anonymous reads are rejected with
401. Self-registration is force-disabled in private mode regardless
of `ALLOW_SELF_REGISTER`, so a locked-down instance flips with a
single switch (admins still mint accounts via `POST /admin/users`).
The backend gate is a tower middleware that reuses the existing
`CurrentUser` extractor, so the cookie + bearer paths cannot drift
from per-handler auth. `/auth/config` now exposes the flag plus the
effective `self_register_enabled` value so the frontend can render
the navbar correctly on the first paint.
On the frontend, a new universal root `+layout.ts` fetches the
config and redirects anonymous visitors to `/login?next=<path>`
before page-specific loads fire. The redirect is UX only — the
backend middleware is the source of truth, so crafted requests
still 401.
Defaults stay public (`PRIVATE_MODE=false`); existing deployments
need no env change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CLI binary already capped runs at CRAWLER_LIMIT mangas, but the
daemon's RealMetadataPass passed a hardcoded `0` (no cap) to
`pipeline::run_metadata_pass`, so the env var was silently ignored once
the daemon took over the metadata pass.
Adds `manga_limit` to `CrawlerConfig`, reads it from `CRAWLER_LIMIT`
(default 0 = no cap), and threads it through `RealMetadataPass::run`
so a daemon-driven sweep stops at the same boundary as a CLI run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chromium doesn't know the socks5h scheme (curl/reqwest convention)
and bails navigations with ERR_NO_SUPPORTED_PROXIES. It does, however,
send destination hostnames over SOCKS5 by default, so stripping the
`h` is a pure scheme rename — remote-DNS behaviour is preserved.
reqwest keeps the user's original CRAWLER_PROXY string (`socks5h://...`
remains valid and meaningful for it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The startup log line in app.rs and bin/crawler.rs `?t`-debug-formats
the TorController, which through the derived Debug on TorAuth would
expand TorAuth::Password(p) and leak the plaintext password to logs.
Implement Debug manually on TorAuth — None / Password(<redacted>) /
Cookie(<path>) — and lock the redaction with a regression test.
Drop the per-NEWNYM success log from info to debug: a busy crawl
rotates circuits many times per minute. Failed NEWNYMs already log
at warn — those stay loud.
Tightens the closed_connection_mid_reply_is_an_error assertion which
was tautological (`closed connection` OR `AUTHENTICATE`) by driving
the mock to read the AUTH line then drop, exercising only the
EOF-mid-reply path.
Audit ref: #7, #9, nit on tautological test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three retry-with-recircuit sites disagreed: detect.rs's
retry_on_transient_with_hook used "N = total attempts" (3 → 3
fetches), but session.rs's unauth branch and content.rs's chapter
loop used "N = recircuits" (3 → 4 fetches). At the same wall-clock
"max=3", different sites hit the upstream a different number of times.
Unify on N = total attempts (matching the existing
retry_on_transient convention). The CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS
env var now means exactly what its name suggests. Disabling the
recircuit feature collapses to max_attempts=1 (single attempt, no
retry) — bit-for-bit pre-TOR behavior preserved.
Adds a debug_assert!(max >= 1) on both helpers and a new
content.rs test exercising the mixed Transient → Unauth → Ok
sequence to lock in the shared-counter invariant.
Audit ref: #5.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- target.rs swaps retry_on_transient → retry_on_transient_with_hook,
signaling NEWNYM via ctx.tor between attempts when configured.
- session.rs gains verify_session_with_recircuit; the bare
verify_session is now a one-line wrapper passing tor=None,
unauth_max_recircuit=0. The inner run_session_probe_loop is
pure-over-IO and unit-tested with closure-based fakes.
- content.rs extracts fetch_chapter_html_once + the closure-driven
fetch_chapter_html_with_recircuit, used by sync_chapter_content to
retry on Transient or Unauthenticated up to a recircuit_budget.
Budget = 0 (no TOR) preserves original behavior bit-for-bit.
- app.rs and bin/crawler.rs construct the controller before on_launch
and pass it into verify_session_with_recircuit, so a transient
hiccup at startup no longer requires PHPSESSID rotation.
Recircuit budget defaults to CRAWLER_TOR_RECIRCUIT_MAX_ATTEMPTS (3).
Errors from NEWNYM are logged and swallowed — failing to recircuit
should not take down the crawl.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds CRAWLER_TOR_CONTROL_URL / _PASSWORD / _COOKIE_PATH /
_RECIRCUIT_MAX_ATTEMPTS to CrawlerConfig and to bin/crawler.rs's
env reads. Constructs an Option<Arc<TorController>> at daemon /
CLI startup and threads it through FetchContext,
pipeline::run_metadata_pass, and content::sync_chapter_content as
Option<&TorController>.
Pure scaffolding — the controller isn't used yet; behavior is
unchanged. Next commit wires the retry hooks and session-probe
recircuit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a sibling fn that fires a caller-supplied async hook between a
transient failure and the next attempt. The existing
retry_on_transient becomes a thin wrapper over it (no-op hook), so
no call sites churn yet.
Hook contract: fires only between attempts (N-1 times for N
attempts), never after a non-transient error or after the final
attempt. Designed for TOR NEWNYM, but the signature is generic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minimal client over tokio::net::TcpStream — AUTHENTICATE then
SIGNAL NEWNYM, one-shot connection. Supports cookie-file and
password auth (cookie preferred when both provided); covers the
multi-line `250-...\r\n250 OK` reply form so future torrc tweaks
won't confuse the parser.
Not yet wired into the crawler — that lands in the next commits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Handle::close aborts its chromiumoxide driver task when another
Arc<Browser> outlives the call, so shutdown returns instead of
hanging on a stream that never terminates. Generic close_or_abort
helper with regression tests covering both Arc paths.
- daemon.shutdown() is wrapped in a 5s timeout in main as defense
in depth.
- Default RUST_LOG silences chromiumoxide::conn / chromiumoxide::handler
WS-deserialize ERROR spam.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with the ALLOW_SELF_REGISTER toggle from 0.42.0: admins can mint
accounts regardless of the toggle state, so a closed-membership
deployment still has a working enrollment path. The endpoint accepts
{ username, password, is_admin? } so admins can mint co-admins in one
call (avoiding a separate promote + extra audit row for the common
"invite a co-admin" flow).
Implementation:
- POST /api/v1/admin/users guarded by RequireAdmin
- Reuses validate_username / validate_password from api::auth (made
pub(crate)) so the admin path can never produce an account self-
register would reject and vice versa
- repo::user::admin_create_user wraps INSERT + admin_audit insert in
a single tx — same "audit reflects what committed" semantics as the
existing admin_safe_* fns
- Audit row: action="create_user", payload={username, is_admin}
Frontend:
- createAdminUser() in lib/api/admin.ts
- /admin/users grows a collapsible "Create user" form above the table
(username, password, "Make admin" checkbox). Errors surface inline;
the list reloads on success.
Backend tests: 7 new, including the headline
`create_user_works_even_when_self_register_disabled` that pins the
admin-create path is NOT gated by the public toggle.
Lets operators run a closed-membership deployment by setting
ALLOW_SELF_REGISTER=false (default true, so existing deploys are
unaffected). When off, POST /auth/register returns 403 forbidden. The
rate-limit token is consumed BEFORE the disabled check so the timing
doesn't distinguish enabled-but-rejected from disabled — closes the
toggle-state probe channel.
New public GET /auth/config returns { self_register_enabled: bool }
so the frontend can render its register affordances correctly
without conflating "disabled" with "rate-limited" (which a probe
attempt would).
Frontend: a lightweight reactive `authConfig` store loads the flag
once on root-layout mount (and again on /register direct navigation,
which bypasses the layout's onMount). Header hides the Register link
when the toggle is off; /register renders a "self-registration is
disabled — ask an administrator" notice instead of the form.
Admin-create endpoint that pairs with this toggle is intentionally
not in this PR — it lands as the next branch (feat/admin-user-create).
The toggle alone is independently useful for deployments that want
to lock down enrollment without yet wiring an admin UI.
Addresses the security-audit findings on top of the admin feature stack:
M1: /admin/mangas/:id/chapters now paginates (default limit 200, max 500).
A long-runner with thousands of chapters would otherwise produce a multi-MB
response with that many scalar subqueries per row — admin-only but a real
stall risk on one expand-click. Adds explicit pagination tests for the cap
and offset; frontend renders a "Showing first N of M" hint when the cap
clips the result.
L1: repo::user::set_is_admin renamed to set_is_admin_unchecked with a
doc-comment pointing at admin_safe_set_is_admin for production use. The
short name was a footgun — a future contributor reaching for it would
silently bypass self-protection, the last-admin invariant, and the audit
log. Used only by integration-test setup; production code goes through
the admin_safe_* paths.
CSRF posture: build_session_cookie carries a comment that the
SameSite=Lax default is the project's CSRF defense for state-changing
mutations and breaks the instant anyone adds a side-effecting GET under
/admin/*. Spells out what to do then (Strict + explicit token check).
Test counts: 43 backend admin tests + 12 vitest admin tests all green;
svelte-check 0/0 across 446 files.
- admin_safe_set_is_admin: short-circuit when target.is_admin == value,
before writing audit. PATCH {is_admin: true} on someone already admin
previously wrote a misleading "promote_user" row even though the UPDATE
was a no-op.
- list_chapters (/admin/mangas/:id/chapters): explicit exists() check on
manga_id, returns 404 instead of 200 [] for a typo'd / deleted manga.
- ChapterSyncState priority: the Failed branch now requires page_count = 0,
so a chapter with pages on disk AND a historical dead job (from a
re-download attempt that crashed) stays Synced. The old order
contradicted Synced's documented "downloaded at some point" contract.
Doc comments updated alongside the SQL.
Three new regression tests pin the behaviour.
Adds GET /api/v1/admin/system returning disk (scoped to storage_dir
via statvfs), memory, CPU, and a server-side alerts array that fires
at >90% disk or memory.
Disk uses nix::sys::statvfs directly rather than sysinfo's Disks API
to avoid mountpoint-matching gymnastics for the storage_dir. A new
`Storage::local_root() -> Option<&Path>` trait method exposes the
root; the default returns None so a future S3Storage gets `disk:
null` in the response instead of fabricated numbers.
CPU is sampled inline (refresh → 250ms sleep → refresh → read) so the
endpoint adds 250ms of latency per call. No background-cache yet —
admin traffic is low-volume and the moving parts aren't worth it
until polling shows up.
Alerts are evaluated server-side so the frontend can render them
without re-implementing the thresholds.
Adds GET /api/v1/admin/mangas and /admin/mangas/:id/chapters guarded by
RequireAdmin. Sync state is computed at query time from the existing
crawler signals (manga_sources / chapter_sources / crawler_jobs) — no
new state column is persisted, so the crawler stays the single writer
of these signals.
Per-manga priority: InProgress (in-flight sync_manga or
sync_chapter_list job) > Dropped (all source rows soft-dropped) >
Synced (default; covers user-uploaded mangas with zero source rows).
Per-chapter priority: Downloading (in-flight sync_chapter_content) >
Dropped (all source rows soft-dropped) > Failed (most-recent terminal
job is dead) > NotDownloaded (page_count = 0) > Synced. The Failed
check sits ABOVE NotDownloaded so the more informative "we tried and
it died" state wins over "we never got around to it" — see the
priority comment in repo/admin_view.rs.
Migration 0020 adds a partial index on
crawler_jobs((payload->>'source_manga_key')) for the one job kind
(sync_manga) whose payload doesn't carry manga_id directly — without it
the in-flight detection for a manga falls back to a seqscan over the
job table.
Adds /api/v1/admin/users list / DELETE / PATCH guarded by RequireAdmin,
plus the audit-log substrate every future destructive admin endpoint
will reuse.
Safety properties:
- Cannot self-delete or self-demote (409 conflict, message calls out
"yourself" so the UI can render an explanation).
- Cannot remove the last admin via either DELETE or demote. The check
takes pg_advisory_xact_lock(ADMIN_INVARIANT_LOCK_KEY) and re-counts
admins inside the same tx, closing the parallel-demote race that a
bare "if count > 1" check would let through. The HTTP-serial path to
this guard is structurally unreachable (the actor would have to be
the lone admin demoting themselves, which the self-guard fires on
first); the parallel race test exercises it via repo calls.
Audit log (admin_audit table) records the action inside the same tx
as the action itself, so a rolled-back action never leaves an orphan
audit row. actor_user_id is ON DELETE SET NULL so the log outlives a
later-deleted admin. target_id is not a FK because future audit kinds
will target non-user rows.
Adds an `is_admin` flag on users plus the substrate every later PR in the
admin feature builds on:
- migration 0018 adds the column with default false
- `repo::user::bootstrap_admin` creates or promotes the user named by
`ADMIN_USERNAME` at startup, hashing `ADMIN_PASSWORD` only when the row
is new — never overwriting an existing hash, so an operator can rotate
the admin password via the UI without env-var conflict
- `CurrentSessionUser` extractor accepts only the session cookie;
`RequireAdmin` composes over it and additionally requires
`user.is_admin`. Bearer tokens are intentionally excluded so an
admin's bot token never inherits admin authority (privilege-escalation
surface that bites every "API keys reuse user perms" auth design)
- demotion is instant: `RequireAdmin` re-reads the user row each request
`/api/v1/auth/me` now exposes `is_admin`; no other response embeds
`User`, so no privacy fanout to audit.
