9fe0f26d75
The backend now boots an internal crawler daemon that runs a daily metadata pass (CRAWLER_DAILY_AT in CRAWLER_TZ, advisory-lock guarded for multi-replica safety) and drains SyncChapterContent jobs from crawler_jobs through a worker pool. Chromium launches lazily on first job and is torn down after CRAWLER_IDLE_TIMEOUT_S seconds of inactivity. Modules: - crawler::browser_manager — lazy-launch / idle-teardown wrapper around browser::Handle, with an on_launch hook that re-injects PHPSESSID on every fresh Chromium spawn. - crawler::pipeline — run_metadata_pass (the shared discover/upsert /cover/sync-chapters loop) and the enqueue_bookmarked_pending helper used by the cron tick. - crawler::daemon — cron task + worker pool, behind two trait seams (MetadataPass, ChapterDispatcher) so tests can inject stubs without standing up Chromium or a live source. Behavior: - CRAWLER_DAEMON=false skips daemon spawn entirely (default for tests). - Catch-up tick fires on startup if the last persisted slot was missed. - A SyncOutcome::SessionExpired sets a sticky AtomicBool; workers idle until operator restart with a refreshed PHPSESSID. - Worker dispatch wrapped in catch_unwind so a panicking handler marks the job failed instead of taking down the worker. - Migration 0015 adds a small crawler_state k-v table for the last_metadata_tick_at watermark. Dep additions: chrono-tz (IANA TZ parsing). CLI (bin/crawler) reuses pipeline::run_metadata_pass and now holds the browser via BrowserManager so the on_launch session injection flow stays in one place. Inline chapter-content sync semantics are unchanged — the queue is for the daemon, force-refetches and manual backfills still bypass it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
33 lines
862 B
JSON
33 lines
862 B
JSON
{
|
|
"name": "mangalord-frontend",
|
|
"version": "0.28.0",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"dev": "vite dev",
|
|
"build": "vite build",
|
|
"preview": "vite preview",
|
|
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
|
|
"test": "vitest run",
|
|
"test:watch": "vitest",
|
|
"test:e2e": "playwright test"
|
|
},
|
|
"devDependencies": {
|
|
"@lucide/svelte": "^1.16.0",
|
|
"@playwright/test": "^1.48.0",
|
|
"@sveltejs/adapter-node": "^5.2.0",
|
|
"@sveltejs/kit": "^2.7.0",
|
|
"@sveltejs/vite-plugin-svelte": "^4.0.0",
|
|
"@testing-library/jest-dom": "^6.6.0",
|
|
"@testing-library/svelte": "^5.2.0",
|
|
"@types/node": "^22.7.0",
|
|
"jsdom": "^25.0.0",
|
|
"svelte": "^5.0.0",
|
|
"svelte-check": "^4.0.0",
|
|
"tslib": "^2.7.0",
|
|
"typescript": "^5.6.0",
|
|
"vite": "^5.4.0",
|
|
"vitest": "^2.1.0"
|
|
}
|
|
}
|