The worker hardcoded response_format={type:json_object}, which LM Studio
rejects with 400 ('must be json_schema or text'); it also leaves
reasoning models (Gemma) with an empty `content`. Fix:
- Default to response_format=json_schema with the real analysis schema
(prompt::output_json_schema), which LM Studio accepts and which forces
clean JSON into message.content. Configurable via ANALYSIS_RESPONSE_FORMAT
= json_schema (default) | json_object | none (config::ResponseFormat).
- build_request_body is now a pure, unit-tested function.
- Vision errors now include the server's response body (status + text)
instead of a bare status, so a 400's reason is visible in logs.
Tests: response-format body shapes (none/json_object/json_schema), schema
shape + round-trip with the DTO, config parsing/defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the raw manga-id / chapter-id UUID inputs with a real picker:
- Search a manga (debounced, reuses listAdminMangas) → result rows show
title + chapter count; "Queue manga" enqueues the whole manga.
- Expand a result to load its chapters (listAdminChapters) and "Queue
chapter" enqueues a single chapter.
- Keeps the "Queue all (whole library)" action and a single global
"include already-analyzed" toggle applied to every action; per-action
busy state + a result message naming what was queued.
Tests: rewrote the Playwright spec to drive the search → manga/chapter
flow (whole library, search+queue manga, expand+queue chapter with
include-analyzed). svelte-check + build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the scoped re-enqueue and per-page force-analyze in the UI:
- api/admin: reenqueueAnalysis({onlyUnanalyzed, mangaId, chapterId}) and
analyzePage(pageId).
- Reader PageContextMenu gains an admin-only "Queue for analysis" item
(canAnalyze/onAnalyzePage/analyzeState props) wired to the force-analyze
endpoint with inline busy/done/error feedback; reset per page.
- New /admin/analysis dashboard section + nav tab: scope selector
(whole library / one manga / one chapter), id input, and an
"include already-analyzed" toggle (default off), with busy/notice/error.
Tests: vitest for the two clients; Playwright for the context-menu action
(admin vs non-admin) and the admin section (scope + include toggle +
validation). svelte-check + build clean; 262 vitest, 10 new e2e green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalizes the admin re-enqueue beyond global backfill:
- repo::page_analysis::enqueue_pages(scope, only_unanalyzed) with
ReenqueueScope::{All,Manga,Chapter}. Fixes include-analyzed semantics:
only_unanalyzed=false now enqueues jobs with force=true so the worker
actually re-analyzes already-done pages instead of skipping them.
- POST /v1/admin/analysis/reenqueue accepts optional manga_id / chapter_id
(mutually exclusive, 404 on unknown target) alongside only_unanalyzed.
Tests: manga/chapter scope counts, include-analyzed sets force=true on the
done page, mutual-exclusivity 422, unknown-target 404. Existing global
backfill tests still green (13 total).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the analysis pipeline in the UI:
- api clients: searchPages() + PageSearchItem/ContentWarning types;
listMangas cwInclude/cwExclude; MangaDetail.content_warnings.
- /search gains a content-search mode: free-text over OCR+scene and
tri-state content-warning toggles (include/exclude), driven by URL
params (text, cw_include, cw_exclude) and the new /me/page-search
endpoint. Tag browsing is preserved when no content filter is active.
- manga detail renders a deduped content-warning banner.
Tests: vitest param serialization (searchPages CSV/text/cw, listMangas
cw); Playwright text-search + cw-toggle flows (route-mocked). svelte-check
+ build clean; full vitest (258) + search e2e (7) green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the analysis worker's NSFW moderation at the manga level:
- repo::manga FILTER_WHERE gains cw_include (AND, page-content-warning
join to mangas) and cw_exclude (NONE) clauses; ListQuery + binds
renumbered ($6/$7, LIMIT/OFFSET $8/$9).
- repo::page_analysis::warnings_for_manga: deduped, alphabetical union
across all the manga's pages.
- domain::MangaDetail gains content_warnings; get_detail populates it.
- api::mangas list accepts cw_include/cw_exclude (reusing the shared
parse_warnings_csv validator).
Tests: detail union (deduped/sorted) + empty case; list include/exclude
filter; unknown-warning 422. Existing manga tests still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New GET /v1/me/page-search surfacing the analysis worker's output:
- repo::page_analysis::page_search + PageSearchQuery: multi-tag AND across
(user page_tags ∪ global page_auto_tags) via the unnest double-negative
idiom, weighted OCR/scene text ranking (ts_rank over search_doc), and
content-warning include/exclude. One row per page with is_nsfw +
deduped content_warnings + rank.
- domain::PageSearchItem.
- api::page_tags: /me/page-search handler with CSV tag/warning parsing
(parse_tags_csv reuses normalize_tag; parse_warnings_csv validates the
closed vocabulary), requiring at least one positive filter (422 else).
This is where the reserved OCR text search lands for pages.
Tests: multi-tag AND user∪auto, speech>sfx ranking, cw include/exclude
(+ row flags), text-only (no tags), missing-filter 422, unknown-warning
422, auth required.
Note: the /me/page-tags/chapters|mangas aggregations keep single-tag
behavior + the reserved text=501 for now; page-level search is the
primary text surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenAI-compatible vision client and the bounded prompt/output handling
for the analysis worker (no DB, fully unit-tested):
- analysis::prompt: terse system prompt carrying the JSON schema, the OCR
kind / content-warning vocabularies, and the output-size caps.
- analysis::vision: VisionClient (downscale via the new `image` dep →
base64 data-URL → chat/completions with an image_url part), plus pure
parse_chat_completion (fence/prose-tolerant) and sanitize (drop empty
OCR, truncate, clamp tags to 10 via the shared page-tag normalizer,
filter unknown warnings).
- config::AnalysisConfig extended with endpoint/model/api_key/timeouts/
max_tokens/max_image_dim/max_image_bytes + from_env.
- Deps: add `image` (jpeg/png/webp), reqwest `json` feature. Expose
api::page_tags::normalize_tag as pub(crate) for reuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the analyze_page job kind into the page-create paths and adds admin
controls, all gated on a new ANALYSIS_ENABLED flag (AppState.analysis_enabled,
config::AnalysisConfig):
- Chapter upload (api::chapters) enqueues one analyze_page job per page
after the tx commits (best-effort; failures logged, never fatal).
- Crawler content sync (persist_pages → RETURNING id) enqueues per
persisted page when the daemon dispatcher's analysis_enabled flag is set;
threaded through sync_chapter_content (CLI/resync pass false).
- repo::page_analysis::enqueue_all_pages: bulk backfill via INSERT..SELECT,
skipping done pages (only_unanalyzed) and existing pending jobs.
- New admin endpoints (RequireAdmin, 503 when disabled, audited):
POST /admin/analysis/reenqueue and POST /admin/pages/:id/analyze (force).
Tests: upload enqueues N jobs / no-op when disabled; crawler persist_pages
enqueue gate; admin reenqueue (backfill, idempotent, only-unanalyzed, 503,
non-admin 403) and force-analyze (force flag, 404).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the persistence foundation for the AI content-analysis worker:
- Migration 0025: page_analysis (status + scene + is_nsfw + weighted
search_doc tsvector), page_ocr_text (kind-tagged pieces), global
page_auto_tags (shared tags vocabulary, separate from per-user
page_tags), and page_content_warnings (closed vocabulary).
- domain::page_analysis: AnalysisStatus/OcrKind/ContentWarning enums with
kind->tsvector-weight mapping and lenient model-string parsing, plus the
VisionAnalysis response DTOs.
- JobPayload::AnalyzePage { page_id, force } + KIND_ANALYZE_PAGE, reusing
the existing crawler_jobs queue.
- repo::page_analysis: enqueue_for_page, load, mark_failed, and the single
transactional persist_analysis (delete+reinsert; idempotent; never
touches user page_tags; computes the A/B/C/D-weighted search_doc).
Tests: repo integration (persist/idempotency/user-tags-untouched/enqueue/
mark_failed/speech>sfx ranking) + unit (weights, parsing, DTO, job serde).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
page_tags previously stored the tag inline as free-form text — the lone
un-normalized tag concept, while authors/genres/manga-tags all went
through a lookup table + FK. Fold page tags into the SAME shared `tags`
table that manga_tags uses: `page_tags.tag` becomes `page_tags.tag_id`
referencing `tags(id)`. Manga tags and page tags now share one global
vocabulary.
The HTTP contract is unchanged — the API still speaks tag *names*; the
repo resolves name<->id via `repo::tag::upsert_by_name` (the manga-tag
helper) and keeps the stricter page-tag `normalize_tag` at the boundary.
Frontend, DTOs, and response shapes are untouched. User-visible effect:
page-tag names now appear in the manga-tag autocomplete and vice versa.
Migration 0024 backfills `tags` from existing inline values (deduping
case-insensitively), repoints rows, swaps the unique/secondary indexes
to tag_id, and drops the inline column.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add GET /v1/mangas/:id/similar returning the top 5 mangas ranked by
Jaccard tag overlap (shared / union), excluding self, as MangaCard items
in a plain { items } object. Surface them in a hidden-when-empty
"Similar" section on the manga detail page, reusing MangaCard.
Backend: new repo::manga::list_similar with a manga_tags self-join; a
shared cards_from_rows helper (also used by list_cards) that loads
authors+genres concurrently; single-sourced column list via MANGA_COLS +
manga_cols(alias) to replace the fragile SELECT_COLS string-splitting.
Frontend: getSimilarMangas client fn (guards a malformed body), loader
fetch with non-critical .catch fallback, and the catalog grid hoisted to
a shared global .manga-grid (lib/styles/tokens.css), de-duplicating the
per-route copies.
Bump version 0.62.0 -> 0.63.0 (backend + frontend in lockstep).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the /search surface (Pages / Chapters / Mangas tabs) backed by
per-user page tags and per-page collections: schema (migration 0023),
backend endpoints for page tags/collections and tagged-page aggregations
(with the OCR text-search param reserved at 501), plus the frontend API
clients, library Page-tags tab, collection page sections, page context
menu / AddTagsSheet, and reader long-press wiring. Includes the
continuous-reader navigation fixes (?page=N handling, chapter-reset
timing, back-button pops history) and tag-normalization hardening
accumulated on the branch.
Bump version 0.60.2 -> 0.62.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reader's "back to manga" link was a naked `<a href="/manga/{id}">`
that pushed a new history entry on every tap, so browser-back kept
ping-ponging between detail and reader instead of walking out to
home:
home → detail → reader → reader-back (push detail) → detail-back
(pop reader) → reader-back (push detail) → … loop forever
The fix intercepts left-click and calls `window.history.back()`
directly when there's same-tab history to pop. Middle-click /
cmd-click / right-click pass through so "open in new tab" still
works, and a deep-link / fresh-tab visit where `history.length ===
1` falls through to the href so the button still leads somewhere
useful.
An earlier attempt gated the back on
`document.referrer.startsWith(origin)` — but SvelteKit's SPA
`pushState` doesn't update `document.referrer`, so that check was
always false in practice and the default href fired anyway. The
e2e regression added here uses real link clicks (not
`window.location.href`) so it actually exercises the SPA-nav path
the bug lives on.
Verified: home → detail → reader, then reader-back keeps
`history.length` at 4 (popped, not pushed), and a second tap on the
detail back button walks out to /.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Visual cleanup pass after wiring up real data — the placeholder
content used during the five mobile phases hid a few sharp edges
that became obvious as soon as user-supplied tags / author handles
/ descriptions hit the screen.
- Mobile gutter unified at 16px (was 12px) — matches iOS / Material
standard and gives chips, description text, and chapter rows
visible breathing room from the screen edge. Hero negative margin
in /manga/[id] tracks the change so the bleed still hits viewport.
- Catalog grid switched from 2 columns to 4 with
`repeat(4, minmax(0, 1fr))` (per user preference). The `minmax(0,
...)` is load-bearing — without it the implicit `minmax(auto, 1fr)`
lets a long single-word card title push the column past the
viewport edge. MangaCard's `.author` and `.genres` lines hide
below 640px so every card in the 4-col layout has identical height
(cover + 2-line title clamp).
- Same minmax(0, 1fr) trick applied to the detail page's
`.overview` grid track, plus defensive `min-width: 0` /
`max-width: 100%` on `.meta` and `.chip-row` so a long
unbreakable tag or romanized URL can't drag the metadata column
past the article gutter.
- BottomNav swapped from `grid-auto-columns` (= `minmax(auto, 1fr)`)
to an explicit `repeat(4, minmax(0, 1fr))`, plus `min-width: 0` on
`.tab` and an ellipsizing `.label` so a long tab label stays
inside the bar. The Browse placeholder is replaced with Upload —
a real action, no longer a no-op duplicate of Home.
- Hero appbar gets 16px all-around inset (was 8px) so the back / ⋯
circle buttons aren't kissing the screen edge or the hero top.
Hero content gets 20px horizontal padding so the cover thumb +
title sit visibly off the hero scrim. Description gains
`overflow-wrap: anywhere` for long URLs / romanized titles.
- Chip now allows `overflow-wrap: anywhere` on its label and uses
`max-width: 100%` + `min-width: 0` so a long token wraps to its
own line within the chip-row and ellipsizes inside the chip.
- Mobile catalog search-row gains `flex-basis: calc(100% - 36px -
--space-2)` so the search input takes its own line — the Filter
and Sort chips wrap to a second row instead of squeezing the
input down to a few characters.
- Global safety net: `html, body { overflow-x: hidden }` below
640px so any future bug that lets an element overflow the
viewport doesn't expose touch-pan that shifts the whole layout.
Fixed descendants (bottom nav, CTA bar) are unaffected.
- audit.mjs — small dev helper that walks all 12 user-facing
routes at 390px and reports horizontal overflow, internal
scrollers, and edge-bleeding controls. Used to drive these fixes
and useful for future mobile changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 5 (final) of the mobile redesign: /profile/account becomes an
inset-grouped iOS-style hub on mobile and the new /library route hosts
a SegmentedControl over Bookmarks / Collections / History. Profile-
layout horizontal tabs hide below 640px. Desktop layout is unchanged.
- New /library/+page.ts loads bookmarks, collections, and read-history
in parallel so segmented-control tabs swap instantly without a new
round trip. 401 → unauthenticated path with sign-in prompt.
- New /library/+page.svelte renders the SegmentedControl with ?tab=
as the source of truth. Bookmarks reuses the existing BookmarkList,
Collections reuses CollectionsGrid, History inlines a slim cover +
title + "Continue Ch. N" row list. `goto(..., { replaceState: true })`
drives the URL — plain `replaceState` from $app/navigation doesn't
reliably re-trigger $page-derived state.
- BottomNav's Library tab now points at /library (Phase 1 placeholder
was /bookmarks). The Phase 1 mobile-chrome spec is updated to expect
the new target and mocks the additional library data endpoints.
- /profile/+layout.svelte hides the horizontal tabs below 640px — the
bottom-nav Library/Account tabs plus the account hub carry mobile
cross-section navigation.
- /profile/account/+page.svelte gains a mobile hub: a centered avatar
+ username + "Member since" header, a row group with Profile /
Preferences / Change password rows (the last opening a bottom
Sheet hosting the existing password form snippet), and a separate
group with a red "Log out" row that reuses the layout's logout
flow (logout API → session.setUser(null) → preferences.clearForLogout
→ goto('/login')). Desktop keeps the inline card with the form.
- matchMedia gates the hub vs. desktop card so the password form
testids never duplicate on the page — the snippet is rendered in
exactly one mount at a time.
- 9 Playwright tests cover the Library nav handoff, segmented-control
sub-tab swap with URL, sign-in CTAs on both routes, account hub
composition, password sheet open flow, logout flow (POST /auth/
logout + redirect to /login), profile tabs hiding on mobile, and
the desktop regression where the inline card is the only password
surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 4 of the mobile redesign: the reader gets a mobile-native
interaction model — invisible tap zones for prev / next / toggle, a
chapter-jump bottom sheet, a reader settings sheet (mode / gap /
brightness), a fixed bottom page scrubber, a brightness overlay
driven by a CSS variable, and an idle-timer auto-hide for the chrome
after 3s of inactivity. Desktop keyboard shortcuts and chevrons are
preserved above 640px.
- New TapZone primitive: invisible 3-column grid that splits the
viewport into prev / toggle / next thirds (Mihon convention). Wired
to the existing `prev()` / `next()` / new `toggleChrome()` so
chapter-edge behavior, page preload, and read-progress tracking
all keep working. Has its own vitest coverage.
- matchMedia gates every mobile addition so the same DOM never
carries two copies of the chapter selector or two sets of nav
controls — the desktop <select>, mode toggle, and gap field stay
inline above 640px; below 640px they swap to a chapter-jump
button and a settings ⋯ button hosting the Sheets.
- Chapter jump Sheet lists every chapter with the current one
highlighted; tapping a row navigates and dismisses.
- Reader settings Sheet uses the SegmentedControl primitive for mode
and (continuous-only) page gap. Brightness lives next to them as a
range slider 0.3..1; its value publishes `--reader-dim` as a CSS
variable that drives the always-rendered fixed dim overlay
(pointer-events: none so taps fall through). Brightness is
client-side only in localStorage — the Preferences table doesn't
carry it and Phase 4 deferred a backend migration.
- Idle auto-hide: 3s timer scoped to mobile + single mode, reuses
the existing focus-mode CSS for the actual slide-off. The timer
resets on every index/mode/viewport/fullscreen change via a tracked
$effect.
- Bottom page scrubber: a styled <input type="range"> at the
viewport foot, single + multi-page only, sliding off in focus mode
alongside the chrome. Honors env(safe-area-inset-bottom).
- Reader joins the layout's `data-mobile-full-bleed` attribute so the
hero/top reader-nav can sit at viewport top under main's cleared
padding-top on mobile.
- Playwright spec covers tap left/right/center, the 3s auto-hide, the
chapter-jump sheet, the settings sheet swapping to continuous, the
brightness slider driving `--reader-dim`, and a desktop regression
for the unchanged inline controls.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3 of the mobile redesign: the manga detail page gets a mobile-
native chrome — blurred-cover hero with a transparent app bar overlay,
secondary actions in an overflow Sheet, a 3-line description clamp
with Read more, and a sticky bottom CTA whose wording reflects read
progress. Desktop layout is untouched.
- /manga/[id] joins the layout's HIDE_MOBILE_CHROME route set so the
global AppBar and BottomNav step aside for the page-specific hero
chrome. A new `html[data-mobile-full-bleed]` opt-in zeroes main's
mobile top padding without touching horizontal/bottom gutters.
- The hero block (mobile-only via CSS) renders a blurred cover
backdrop, a transparent app bar (back / bookmark / overflow ⋯), and
a sharp cover thumbnail beside the title, authors, and status. The
bookmark icon swaps between Bookmark and BookmarkCheck based on the
current bookmark state — reusing the existing toggleBookmark flow.
- Secondary actions (Add to collection / Edit / Upload chapter /
Force resync) move into an overflow Sheet opened from the ⋯
button. The desktop action-row stays.
- Description gets a `.clamped` modifier (-webkit-line-clamp: 3) and
a Read more toggle on mobile, gated by a >200-char heuristic so
short blurbs don't render a dangling button. Desktop shows the
full description as before.
- Sticky bottom CTA bar reads "Continue {chapterLabel}" when
data.readProgress has a chapter_id, "Read first chapter" when
the manga has chapters but no progress, and hides otherwise. The
bar honors env(safe-area-inset-bottom) so it sits above the iOS
home indicator.
- New Playwright spec covers the CTA wording state machine
(no-progress → Read first chapter, has-progress → Continue),
Read more expand/collapse, overflow sheet contents, and a desktop
regression for the unchanged action-row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 2 of the mobile redesign: the catalog at `/` adapts to phone
viewports without disrupting the desktop layout. The same filter form
renders inline on desktop and inside a bottom sheet on mobile, and a
dedicated sort sheet replaces the inline `<select>` below 640px.
- MangaCard gains optional `unreadCount` and `progress` props that
render a top-right pill badge and a bottom progress overlay on the
cover. Both are no-ops when omitted, so existing callers don't
change. Counts past 99 cap at "99+".
- The filter form body is extracted into a Svelte 5 snippet and
rendered conditionally — inline desktop panel OR mobile sheet, never
both — so no testid is duplicated and the focus trap can't double-
fire. A matchMedia listener tracks the 640px breakpoint and drives
the snippet target.
- Mobile catalog adds: Filter / Sort chip buttons, an always-visible
active-filter chip row with per-facet remove buttons, full-width
search, and a 2-column grid.
- Auto-expanding the filter panel from URL params is now desktop-only
— on mobile the active-filter chips signal applied facets without
hiding the catalog under a scrim on first paint.
- Vitest suite covers MangaCard badge / overlay behavior including
clamp / hide edge cases. Playwright spec covers the mobile filter +
sort sheet flow at 390px viewport, with a hydration gate to keep
click dispatch from racing the SSR'd-but-not-yet-reactive button.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 of the mobile redesign: add the reusable primitives every
later phase plugs into, and switch the layout to a 4-tab bottom nav
+ compact AppBar on phone viewports while leaving the desktop header
untouched above 640px.
- New primitives: BottomNav, Sheet, AppBar, SegmentedControl, all
with vitest unit tests.
- Layout swaps the desktop header for AppBar + BottomNav under the
existing 640px breakpoint. Login / register / reader routes opt
out of the mobile chrome.
- Library tab currently points at /bookmarks; the curated /library
wrapper lands in Phase 5.
- Independent ResizeObservers publish --app-header-h,
--mobile-app-bar-h, --app-bottom-nav-h, with a zero-height skip so
hidden bars don't clobber the visible one's value.
- viewport-fit=cover + env(safe-area-inset-*) tokens for notched
devices and the iOS home indicator.
- Playwright spec covers visibility at 390px / 1280px viewports and
tab navigation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The user-facing chapter list ordered by (number ASC, created_at ASC),
which broke the source site's order in two ways: non-numeric entries
("notice. : Officials") parsed to number=0 and clustered at the top,
even though the site placed them mid-list, and variants sharing a
number ("Ch.14 : PH" / "Ch.14 : Official") were torn apart by the
created_at tiebreak.
Capture each chapter's position in the source DOM as `source_index`
(0 = first = newest on this site) on every crawler sync, including the
UPDATE branch so a new chapter prepended on the source shifts every
existing row down by one on the next tick. The list query reverses
this with `ORDER BY source_index DESC NULLS LAST, number ASC,
created_at ASC` so the oldest chapter appears first, variants stay
adjacent in the order the site shows them, and non-numeric entries
land where the site placed them. User-uploaded chapters and pre-
migration rows keep their NULL source_index and fall through to the
prior number/created_at tiebreak via NULLS LAST.
The reader's client-side `[...chapters].sort((a,b) => a.number - b.number)`
is dropped; prev/next now walks the server-ordered array positionally
so it traverses variants and non-numeric entries in display order.
Existing data populates on the next cron tick or via admin force-resync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chapter list on the manga detail page, the reader's chapter-select
dropdown, the continuous-mode chapter bar, the browser tab title, and
the profile upload-history entries all prepended "Chapter {number}:"
in front of the crawled site title. Source titles already include
"Ch.N" themselves and the manga page renders chapters inside an <ol>,
so the prefix duplicated information the user could already see.
A small chapterLabel(c) helper in $lib/api/chapters returns the site
title as-is, falling back to "Chapter {number}" only when the
crawler captured an empty title (link/option stays non-empty). The
five render sites now call it. The previous-/next-chapter nav
buttons still read "Previous chapter (Ch. N)" / "Next chapter (Ch. N)"
since those are wayfinding labels, not title display.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both enqueue paths now order by chapters.number so the cron tick and the
bookmark hook insert jobs from chapter 1 upward instead of source-discovery
or random-UUID order. The lease query tiebreaks on created_at so jobs
sharing a batch's scheduled_at come off the queue in insertion order,
propagating the enqueue intent through to dequeue. Concurrent workers
and per-CDN latency can still drift actual completion order.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a chapter `<select>` to the reader's top nav listing every chapter
of the current manga, defaulting to the open chapter; picking another
entry navigates straight to it without going back to the manga detail
page. Options use the "Ch. N — Title" form to match the existing
chapter tile and prev/next buttons in the reader bar.
Covered by a new Playwright spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tick cover-backfill pass to the crawler daemon so mangas whose
cover download failed on first attempt get retried — the metadata pass's
early-stop optimisation otherwise prevents the walk from revisiting them.
Adds admin-only POST /admin/mangas/:id/resync and POST /admin/chapters/:id/resync
that refetch metadata + cover (or chapter content with force_refetch) from the
crawler source synchronously and return the refreshed row. Surfaced in the
UI as "Force resync" buttons on the manga detail and reader pages,
admin-only via session.user.is_admin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bundle of small UI/UX fixes plus a build hygiene tweak.
* List pagination — Home (`/`) and `/authors/[id]` silently capped at
the backend default of 50 with no UI to advance. New reusable
`Pager.svelte` (Prev/Next + numbered with ellipsis), URL-synced
`?page=N`, and filter/search/sort reset to page 1 so users aren't
stranded on an out-of-range page. Count label now shows a range
("Showing 51–100 of 237").
* Stale page title — Pages without a `<svelte:head><title>` left the
document title at whatever the last manga / author / collection page
set it to. Move static-route titles into a route-id → title map in
the root layout and invert every dynamic title to brand-first
(`Mangalord | {X}`) for consistency.
* Admin filter bar — `/admin/mangas` search input had `flex: 1` and
ballooned across the row, shoving the sync-state select + Search
button to the far right. Cap at 24rem, vertical-align the row, and
promote the previously aria-only "Sync state" label to visible text.
* Build hygiene — `backend/target` had grown to 68 GiB. Cleaned and
added `[profile.dev] debug = "line-tables-only"` (and `[profile.test]`
too) to cut future dev builds by ~50–70% while keeping line numbers
in backtraces.
Also: configure vitest to resolve Svelte's browser entry so
`@testing-library/svelte` can mount components in jsdom — needed for
the new `Pager.svelte.test.ts`.
Bump 0.48.0 -> 0.49.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When `PRIVATE_MODE=true`, every API path except a small allowlist
(`/health`, `/auth/{config,login,logout,register}`) requires a valid
session cookie or bearer token — anonymous reads are rejected with
401. Self-registration is force-disabled in private mode regardless
of `ALLOW_SELF_REGISTER`, so a locked-down instance flips with a
single switch (admins still mint accounts via `POST /admin/users`).
The backend gate is a tower middleware that reuses the existing
`CurrentUser` extractor, so the cookie + bearer paths cannot drift
from per-handler auth. `/auth/config` now exposes the flag plus the
effective `self_register_enabled` value so the frontend can render
the navbar correctly on the first paint.
On the frontend, a new universal root `+layout.ts` fetches the
config and redirects anonymous visitors to `/login?next=<path>`
before page-specific loads fire. The redirect is UX only — the
backend middleware is the source of truth, so crafted requests
still 401.
Defaults stay public (`PRIVATE_MODE=false`); existing deployments
need no env change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The CLI binary already capped runs at CRAWLER_LIMIT mangas, but the
daemon's RealMetadataPass passed a hardcoded `0` (no cap) to
`pipeline::run_metadata_pass`, so the env var was silently ignored once
the daemon took over the metadata pass.
Adds `manga_limit` to `CrawlerConfig`, reads it from `CRAWLER_LIMIT`
(default 0 = no cap), and threads it through `RealMetadataPass::run`
so a daemon-driven sweep stops at the same boundary as a CLI run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CRAWLER_TOR_CONTROL_URL, _PASSWORD, _COOKIE_PATH,
_RECIRCUIT_MAX_ATTEMPTS are new feature env vars; treat per CLAUDE.md
as a minor bump (feat:).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- backend dep-cache stage stubs only main.rs/lib.rs, but Cargo.toml
declares a second [[bin]] crawler at src/bin/crawler.rs, so
`cargo build --locked` aborts ("can't find bin crawler"). Stub it too.
- runtime was debian:bookworm-slim (glibc 2.36) while rust:1-slim now
tracks trixie (glibc 2.41) -> "GLIBC_2.39 not found" at boot. Pin the
runtime to debian:trixie-slim so it matches the builder's glibc.
- frontend healthcheck probed localhost (-> musl picks IPv6 ::1) but the
Node server binds IPv4 0.0.0.0 only -> false "unhealthy". Probe 127.0.0.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Handle::close aborts its chromiumoxide driver task when another
Arc<Browser> outlives the call, so shutdown returns instead of
hanging on a stream that never terminates. Generic close_or_abort
helper with regression tests covering both Arc paths.
- daemon.shutdown() is wrapped in a 5s timeout in main as defense
in depth.
- Default RUST_LOG silences chromiumoxide::conn / chromiumoxide::handler
WS-deserialize ERROR spam.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pairs with the ALLOW_SELF_REGISTER toggle from 0.42.0: admins can mint
accounts regardless of the toggle state, so a closed-membership
deployment still has a working enrollment path. The endpoint accepts
{ username, password, is_admin? } so admins can mint co-admins in one
call (avoiding a separate promote + extra audit row for the common
"invite a co-admin" flow).
Implementation:
- POST /api/v1/admin/users guarded by RequireAdmin
- Reuses validate_username / validate_password from api::auth (made
pub(crate)) so the admin path can never produce an account self-
register would reject and vice versa
- repo::user::admin_create_user wraps INSERT + admin_audit insert in
a single tx — same "audit reflects what committed" semantics as the
existing admin_safe_* fns
- Audit row: action="create_user", payload={username, is_admin}
Frontend:
- createAdminUser() in lib/api/admin.ts
- /admin/users grows a collapsible "Create user" form above the table
(username, password, "Make admin" checkbox). Errors surface inline;
the list reloads on success.
Backend tests: 7 new, including the headline
`create_user_works_even_when_self_register_disabled` that pins the
admin-create path is NOT gated by the public toggle.
Lets operators run a closed-membership deployment by setting
ALLOW_SELF_REGISTER=false (default true, so existing deploys are
unaffected). When off, POST /auth/register returns 403 forbidden. The
rate-limit token is consumed BEFORE the disabled check so the timing
doesn't distinguish enabled-but-rejected from disabled — closes the
toggle-state probe channel.
New public GET /auth/config returns { self_register_enabled: bool }
so the frontend can render its register affordances correctly
without conflating "disabled" with "rate-limited" (which a probe
attempt would).
Frontend: a lightweight reactive `authConfig` store loads the flag
once on root-layout mount (and again on /register direct navigation,
which bypasses the layout's onMount). Header hides the Register link
when the toggle is off; /register renders a "self-registration is
disabled — ask an administrator" notice instead of the form.
Admin-create endpoint that pairs with this toggle is intentionally
not in this PR — it lands as the next branch (feat/admin-user-create).
The toggle alone is independently useful for deployments that want
to lock down enrollment without yet wiring an admin UI.
Addresses the security-audit findings on top of the admin feature stack:
M1: /admin/mangas/:id/chapters now paginates (default limit 200, max 500).
A long-runner with thousands of chapters would otherwise produce a multi-MB
response with that many scalar subqueries per row — admin-only but a real
stall risk on one expand-click. Adds explicit pagination tests for the cap
and offset; frontend renders a "Showing first N of M" hint when the cap
clips the result.
L1: repo::user::set_is_admin renamed to set_is_admin_unchecked with a
doc-comment pointing at admin_safe_set_is_admin for production use. The
short name was a footgun — a future contributor reaching for it would
silently bypass self-protection, the last-admin invariant, and the audit
log. Used only by integration-test setup; production code goes through
the admin_safe_* paths.
CSRF posture: build_session_cookie carries a comment that the
SameSite=Lax default is the project's CSRF defense for state-changing
mutations and breaks the instant anyone adds a side-effecting GET under
/admin/*. Spells out what to do then (Strict + explicit token check).
Test counts: 43 backend admin tests + 12 vitest admin tests all green;
svelte-check 0/0 across 446 files.
- admin_safe_set_is_admin: short-circuit when target.is_admin == value,
before writing audit. PATCH {is_admin: true} on someone already admin
previously wrote a misleading "promote_user" row even though the UPDATE
was a no-op.
- list_chapters (/admin/mangas/:id/chapters): explicit exists() check on
manga_id, returns 404 instead of 200 [] for a typo'd / deleted manga.
- ChapterSyncState priority: the Failed branch now requires page_count = 0,
so a chapter with pages on disk AND a historical dead job (from a
re-download attempt that crashed) stays Synced. The old order
contradicted Synced's documented "downloaded at some point" contract.
Doc comments updated alongside the SQL.
Three new regression tests pin the behaviour.
Adds the SvelteKit /admin route tree backed by the admin endpoints
landed in PR 1-4. Pages: Overview (alerts + summary cards), Users
(list / promote-demote / delete), Mangas (list with sync state +
expandable per-chapter state), System (live disk/mem/cpu bars,
refreshing every 5s).
Security model: the backend's RequireAdmin extractor is the actual
boundary. /admin/+layout.ts calls getSystemStats() at load and
translates the response — 401 → redirect to /login, 403 → throw
SvelteKit error(403) which renders the framework error page. The
header's "Admin" link is hidden unless `session.user?.is_admin`,
but that's UX only.
Carries `is_admin: boolean` through to the frontend User TS type so
the header check works and so admin tables can show role per row.
Vitest covers lib/api/admin.ts (10 tests: list/delete/PATCH for
users, sync-state filter for mangas, nested chapter route, system
disk-nullable case). Playwright is intentionally deferred until the
routes stabilise — admin UI is operator-only and changes shape often
in v0.
Adds GET /api/v1/admin/system returning disk (scoped to storage_dir
via statvfs), memory, CPU, and a server-side alerts array that fires
at >90% disk or memory.
Disk uses nix::sys::statvfs directly rather than sysinfo's Disks API
to avoid mountpoint-matching gymnastics for the storage_dir. A new
`Storage::local_root() -> Option<&Path>` trait method exposes the
root; the default returns None so a future S3Storage gets `disk:
null` in the response instead of fabricated numbers.
CPU is sampled inline (refresh → 250ms sleep → refresh → read) so the
endpoint adds 250ms of latency per call. No background-cache yet —
admin traffic is low-volume and the moving parts aren't worth it
until polling shows up.
Alerts are evaluated server-side so the frontend can render them
without re-implementing the thresholds.
Adds GET /api/v1/admin/mangas and /admin/mangas/:id/chapters guarded by
RequireAdmin. Sync state is computed at query time from the existing
crawler signals (manga_sources / chapter_sources / crawler_jobs) — no
new state column is persisted, so the crawler stays the single writer
of these signals.
Per-manga priority: InProgress (in-flight sync_manga or
sync_chapter_list job) > Dropped (all source rows soft-dropped) >
Synced (default; covers user-uploaded mangas with zero source rows).
Per-chapter priority: Downloading (in-flight sync_chapter_content) >
Dropped (all source rows soft-dropped) > Failed (most-recent terminal
job is dead) > NotDownloaded (page_count = 0) > Synced. The Failed
check sits ABOVE NotDownloaded so the more informative "we tried and
it died" state wins over "we never got around to it" — see the
priority comment in repo/admin_view.rs.
Migration 0020 adds a partial index on
crawler_jobs((payload->>'source_manga_key')) for the one job kind
(sync_manga) whose payload doesn't carry manga_id directly — without it
the in-flight detection for a manga falls back to a seqscan over the
job table.
Adds /api/v1/admin/users list / DELETE / PATCH guarded by RequireAdmin,
plus the audit-log substrate every future destructive admin endpoint
will reuse.
Safety properties:
- Cannot self-delete or self-demote (409 conflict, message calls out
"yourself" so the UI can render an explanation).
- Cannot remove the last admin via either DELETE or demote. The check
takes pg_advisory_xact_lock(ADMIN_INVARIANT_LOCK_KEY) and re-counts
admins inside the same tx, closing the parallel-demote race that a
bare "if count > 1" check would let through. The HTTP-serial path to
this guard is structurally unreachable (the actor would have to be
the lone admin demoting themselves, which the self-guard fires on
first); the parallel race test exercises it via repo calls.
Audit log (admin_audit table) records the action inside the same tx
as the action itself, so a rolled-back action never leaves an orphan
audit row. actor_user_id is ON DELETE SET NULL so the log outlives a
later-deleted admin. target_id is not a FK because future audit kinds
will target non-user rows.
Adds an `is_admin` flag on users plus the substrate every later PR in the
admin feature builds on:
- migration 0018 adds the column with default false
- `repo::user::bootstrap_admin` creates or promotes the user named by
`ADMIN_USERNAME` at startup, hashing `ADMIN_PASSWORD` only when the row
is new — never overwriting an existing hash, so an operator can rotate
the admin password via the UI without env-var conflict
- `CurrentSessionUser` extractor accepts only the session cookie;
`RequireAdmin` composes over it and additionally requires
`user.is_admin`. Bearer tokens are intentionally excluded so an
admin's bot token never inherits admin authority (privilege-escalation
surface that bites every "API keys reuse user perms" auth design)
- demotion is instant: `RequireAdmin` re-reads the user row each request
`/api/v1/auth/me` now exposes `is_admin`; no other response embeds
`User`, so no privacy fanout to audit.
anyhow_looks_browser_dead substring-matched any chain message
containing channel / connection / websocket / transport / closed /
nav timeout. Real chromium failures hit those words, but so do
reqwest TCP-reset errors during CDN image downloads, sqlx pool-
timeout errors, and any number of non-browser failures — each of
which triggered a wasted chromium relaunch + session-probe re-run
against the catalog's rate-limit budget.
Drop the substring pass. Walk the chain looking only for typed
NavError (flagged via is_likely_browser_dead) or CdpError. Every
place we feed a chromium error into anyhow goes through one of
those types, so the typed downcasts cover the real cases without
the false-positive surface.
NavError::is_likely_browser_dead also drops its own substring
check on Cdp(e); any CdpError surfacing at the navigation layer
means the chromium-facing channel is the failing layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The wait_for_selector wait in 0.36.2 narrows the partial-render race
window but doesn't close it: a render that takes longer than
SELECTOR_TIMEOUT (10s) still hands an empty Vec to sync_manga_chapters,
and the soft-drop branch flips every existing chapter to dropped_at.
The next tick recovers but a manga's reader briefly stops working in
between.
Close it at the pipeline level. Between fetch_manga and the upsert/
sync, if the parsed chapter list is empty and the prior live count
for (source_id, source_manga_key) is > 0, treat the fetch as a
transient failure: log, bump mangas_failed, skip upsert + sync + the
seen.insert so a later batch / tick retries. Brand-new mangas with
genuinely zero chapters (prior == 0) pass through unchanged.
New repo helper repo::crawler::live_chapter_count_for_source_manga
joins chapters → chapter_sources → manga_sources with dropped_at IS
NULL — same lockstep as dispatch_target and the enqueue queries.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Worker dispatch was already wrapped in AssertUnwindSafe(...)
.catch_unwind() — a panicking handler ack's the job failed and the
worker keeps going. The cron tick had no such guard: a panic in
metadata.run, enqueue_bookmarked_pending, reap_done, or
write_last_tick would kill the cron task. The JoinSet would drop it,
workers would keep running, and no future metadata pass would ever
fire until daemon restart.
Wrap the tick body (between advisory-lock acquire and unlock) in the
same AssertUnwindSafe(...).catch_unwind() pattern. The unlock and
connection drop run unconditionally so a panicked tick doesn't leave
the lock held for another replica.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chapter dispatcher's URL resolver had no dropped_at filter and no
ORDER BY — a chapter whose only chapter_sources row had been soft-
dropped was still dispatched against the stale URL, eating retry
budget on guaranteed transients. With multiple live sources the LIMIT
1 winner was nondeterministic.
Add `AND cs.dropped_at IS NULL` and `ORDER BY cs.last_seen_at DESC`
to dispatch_target, bringing it in lockstep with the enqueue queries
in pipeline.rs that already filter on dropped_at. Returns None when
all sources are dropped — callers in daemon.rs already treat None
as "ack the job, skip the work."
Tests in tests/repo_chapter.rs cover the three branches (freshest
live wins, dropped sources skipped, all-dropped returns None).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BrowserManager only re-launched chromium when the cached handle was
None. A crash mid-pass left the handle Some pointing at a dead
process — every subsequent acquire returned the zombie Browser, and
every nav cascaded CDP errors until the idle reaper fired.
Add BrowserManager::invalidate(): take the inner mutex, drop the
handle (closing it if present), and signal the next acquire to
relaunch. Idempotent — invalidating an empty handle is a no-op.
Wire detection via NavError::is_likely_browser_dead and a
chain-walking anyhow_looks_browser_dead helper: substring-match
common channel/connection/transport/WebSocket markers and surface
NavError::Timeout as "presumed dead." Apply at both error
boundaries — RealChapterDispatcher::dispatch and
RealMetadataPass::run.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
