MechaCat02 ee4594f679 fix(crawler): walk list pages incrementally; stop on empty page (0.45.1) ...

The pre-built `1..=parse_last_page` queue silently broke whenever the
configured CRAWLER_START_URL lacked a `/N/` path segment: page_url
returned the input unchanged, every "next" page re-fetched page 1, and
the dedup set caught the duplicates as a flood of "skip already-seen
key in this run" debug lines. The walker now increments next_page on
each batch and terminates when parse_manga_list_from yields an empty
list (the `#logo` sentinel still converts unrendered pages into
transient errors, so an Ok(vec![]) is a real end-of-index signal).

parse_last_page and build_page_order are deleted along with their
unit tests; they have no callers under the new model. page_url and
the page-1 HTML cache from discover() are retained as-is.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-31 18:06:22 +02:00

1.7 KiB

Raw Permalink Blame History

View Raw