dea9b1aaa8
The target site orders by update_date DESC, and any new or updated manga pushes everyone down by one slot. The paginated walker was blind to this drift: * Backfill (page last -> 1): shifts push items into pages already finished. The displaced manga was silently missed; with mark_dropped_mangas running on a fully-completed walk, items even got false-dropped because last_seen_at was stale. * Incremental (page 1 -> last): a shift causes the slot-last item of an already-read page to reappear on the next page, leading to a redundant fetch_manga and an inflated consecutive_unchanged streak. Fix is two-pronged: 1. Backfill boundary re-check. After fetching each page P, re-fetch the previously-walked page P+1 and check where its old slot-0 key now sits. If it slid to slot K, the first K entries are items that used to live on P and slid past us; they get appended to the batch. If the anchor is gone entirely (multi-page shift or it was bumped to page 1), the whole re-fetched page is processed conservatively and the pipeline dedup absorbs the noise. The re-check must be the *last* navigation of the iteration to close the within-iteration race. 2. Run-scoped dedup in run_metadata_pass. A HashSet<String> of source_manga_keys avoids double-processing. The set uses a contains-then-insert pattern with insert firing *after* a successful upsert, so a transient fetch/upsert failure leaves the key retryable if it reappears later in the same pass (via the boundary re-check or another batch). Incremental mode does not run the re-check (shifts move in the same direction as the walk); only the dedup helps it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
33 lines
862 B
JSON
33 lines
862 B
JSON
{
|
|
"name": "mangalord-frontend",
|
|
"version": "0.35.1",
|
|
"private": true,
|
|
"type": "module",
|
|
"scripts": {
|
|
"dev": "vite dev",
|
|
"build": "vite build",
|
|
"preview": "vite preview",
|
|
"check": "svelte-kit sync && svelte-check --tsconfig ./tsconfig.json",
|
|
"test": "vitest run",
|
|
"test:watch": "vitest",
|
|
"test:e2e": "playwright test"
|
|
},
|
|
"devDependencies": {
|
|
"@lucide/svelte": "^1.16.0",
|
|
"@playwright/test": "^1.48.0",
|
|
"@sveltejs/adapter-node": "^5.2.0",
|
|
"@sveltejs/kit": "^2.7.0",
|
|
"@sveltejs/vite-plugin-svelte": "^4.0.0",
|
|
"@testing-library/jest-dom": "^6.6.0",
|
|
"@testing-library/svelte": "^5.2.0",
|
|
"@types/node": "^22.7.0",
|
|
"jsdom": "^25.0.0",
|
|
"svelte": "^5.0.0",
|
|
"svelte-check": "^4.0.0",
|
|
"tslib": "^2.7.0",
|
|
"typescript": "^5.6.0",
|
|
"vite": "^5.4.0",
|
|
"vitest": "^2.1.0"
|
|
}
|
|
}
|