Three frontend findings (medium):
1. **`hooks.server.ts` didn't chain client-disconnect into the upstream
fetch.** Closing an admin tab left the backend's `broadcast::Receiver`
+ spawned tokio task running until next backend restart. Listen for
`event.request.signal.abort` and forward to the timeout controller.
2. **`admin/analysis/+page.svelte` missing visibility/pagehide/pageshow
handlers.** Mobile Safari throttles SSE in background tabs; bfcache
leaves a dead connection on back-nav. Apply the close/open-stream
pattern from `admin/crawler/+page.svelte`.
3. **`loadCoverage` had no AbortController.** Debounced search
keystrokes raced. Per-loader controller plumbed through
`getAnalysisMangaCoverage`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Admin-only crawler dashboard backed by an SSE live-status stream,
coordinated browser restart, runtime PHPSESSID refresh, dead-letter
requeue, and a batch of reliability fixes. Closes everything from
the two-pass audit (10 commits' worth) and bumps 0.52.0 -> 0.55.0.
Backend:
- New /admin/crawler/* surface (cookie-auth, RequireAdmin) split
into status / control / dead_jobs / backlog modules. SSE stream
composes in-memory status with DB-derived queue counts, memoizes
the counts for 1s and debounces watch pokes for 250ms (~10x QPS
reduction per subscriber). One-shot GET /admin/crawler shares the
same compose path.
- POST /admin/crawler/run gated by manual_pass_lock try_lock_owned
(409 Conflict on overlapping click); browser restart goes through
the coordinated_restart gate (drain + relaunch + auto-clear of the
sticky session_expired flag on Ok).
- Runtime PHPSESSID refresh via SessionController (allow-list
validation, never logged, audit row carries SHA-256 fingerprint).
Storage layer is repo::crawler::runtime_session_{load,persist}.
- Dead-letter requeue with four scopes (all/manga/chapter/job);
scope=all requires confirm:true; DISTINCT ON dedup keeps the
partial unique index from rejecting requeues for chapters with
multiple dead rows. SQL is four &'static str constants per scope.
- StatusHandle + ChapterGuard / CoverGuard RAII model survives
panics; last-writer-wins on cover so concurrent dispatches don't
clobber each other's slot. Pure functions (should_stop /
should_mark_clean_exit / should_abort_pass) with named regression
tests.
- Reliability bundle: per-lease heartbeat, jitter on retries,
per-job timeout, circuit breaker on consecutive failures, BrowserManager
coordinated restart gate, request fingerprint changes.
- Streaming page download: Storage::put_stream trait method,
LocalStorage impl atomic via temp + fsync + UUID-suffixed rename.
Pages stream through with peak memory ~one HTTP chunk + 64-byte
sniff prefix instead of one full image per dispatch.
- New partial indexes (migration 0022): mangas_missing_cover_idx
and crawler_jobs_dead_idx, both ordered by updated_at DESC to
match the dashboard's LIMIT/OFFSET reads.
- Security hardening: admin_csrf_guard (Origin/Referer allowlist
on /admin/* mutations, opt-in via ADMIN_ALLOWED_ORIGINS),
admin_no_store_guard (Cache-Control: no-store on admin
responses), audit rows carry per-scope target_id.
Frontend:
- /admin/crawler page decomposed into lib/components/crawler/
(11 components: ProgressBar, SearchBar, CrawlerHero,
CrawlerControls, ActiveChaptersCard, ActiveJobsTable,
MissingCoversTable, DeadJobsTable, RestartConfirmModal,
RequeueAllConfirmModal, SessionModal). Page is 532 LOC of
orchestration; each component 22-148 LOC.
- EventSource lifecycle wired to visibilitychange / pagehide /
pageshow (BFCache); after 5 consecutive errors probes the status
endpoint so a 401 routes through the global on401Hook instead of
infinite silent reconnects.
- Backlog $effect refetches debounced 500ms with per-loader
AbortControllers; refresh after a control action only runs when
the SSE stream is dead.
- Inline requeue button on /admin/mangas patches the affected row's
sync_state locally (no full chapter-list refetch); proper
aria-label. Requeue-all gets its own confirm modal; both confirm
modals autofocus Cancel.
- SvelteKit reverse proxy bypasses its 5-minute AbortController
for Accept: text/event-stream; pure shouldBypassProxyTimeout
helper covered by unit tests.
Config / docs:
- New env vars (.env.example): ADMIN_ALLOWED_ORIGINS,
CRAWLER_JOB_TIMEOUT_SECS, CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES,
CRAWLER_BROWSER_RESTART_THRESHOLD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The SvelteKit proxy was only stripping host + content-length; the rest
of RFC 7230 §6.1 (connection, keep-alive, proxy-authenticate,
proxy-authorization, te, trailer, transfer-encoding, upgrade) leaked
through to axum. Axum doesn't emit them so the impact is theoretical,
but the proxy should be RFC-conformant. Also adds an AbortController
with a configurable 60s timeout (BACKEND_PROXY_TIMEOUT_MS) so a
wedged backend can't hang the browser request indefinitely — failures
surface as the standard 502 upstream_unavailable envelope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four small follow-ups from the second-pass audit:
- N1: `manga_upload_rolls_back_when_cover_storage_fails` covers the
manga-side of the transactional rollback path. The chapter case had
a `FailingStorage` regression test already; this completes the
symmetric pair. With fail-on-put-index=0, the cover put fails on
the first call, the transaction aborts, and `SELECT count(*) FROM
mangas WHERE title = 'Berserk'` is 0.
- N2: The SvelteKit proxy now catches network-layer failures from the
upstream `fetch` (DNS / connection refused / TLS handshake) and
returns a 502 with the standard error envelope
(`code: 'upstream_unavailable'`) instead of letting SvelteKit's
generic 500 HTML page through. `client.ts` can `.json()` the result
cleanly so callers see a real ApiError with a meaningful code. The
underlying cause is logged via `console.error` for the operator.
Test in hooks.server.test.ts asserts the 502, the JSON envelope, and
that `resolve` is not called (the proxy short-circuits).
- N3: `GET /api/v1/files/*key` now sets
`X-Content-Type-Options: nosniff`. The upload-time magic-byte sniff
is authoritative for what we declare as Content-Type; `nosniff`
makes the contract explicit so older user-agents can't try to
re-detect HTML/JS in a polyglot file that survived the sniff. Test
in api_uploads.rs asserts the header.
- N4: The /bookmarks page used `{#if b.page}` to gate the "— page N"
display, which falsy-elided a legitimate `page == 0`. Backend now
rejects `page < 1` for new bookmarks (already shipped in 0.9.4),
but any pre-0.9.4 row with page=0 still rendered without its
number. Strengthened to `{#if b.page != null && b.page > 0}`.
Lockstep version bump to 0.10.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose deploy was unreachable because frontend code reads its
API base from `import.meta.env.VITE_API_BASE` at build time, but the
shipped image baked in the fallback `/api` and never picked up the
`PUBLIC_API_BASE` env var. The browser then hit
http://localhost:3000/api/...which the Node adapter doesn't serve, so
every request 404'd.
Fix the topology at the right layer: hooks.server.ts proxies /api/*
requests through to the backend container over docker's internal
network. The browser only ever talks to :3000, cookies stay
same-origin, and CORS can stay empty.
- frontend/src/hooks.server.ts: new proxy. Reads BACKEND_URL (defaults
to http://localhost:8080 for ad-hoc node builds). Strips `host` and
`content-length` so the backend sees the real client request and
recomputes the length. Sets `duplex: 'half'` for streamed POST
bodies. GET/HEAD have no body. Non-/api paths fall through to
SvelteKit normally.
- docker-compose.yml: drop the host port mapping on the backend
(browser doesn't reach it directly anymore — use `ports:` instead of
`expose:` if you want curl access). Set BACKEND_URL=http://backend:8080
on the frontend service. Drop PUBLIC_API_BASE which was unused.
- .env.example: replace PUBLIC_API_BASE with BACKEND_URL, with a note
on what it does.
- README: explain the new topology in Quick start, update the bot
curl examples to hit :3000 (since that's the only published port in
the default deploy), and call out that the TLS terminator only needs
one upstream now.
Lockstep version bump to 0.9.1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
