fix(crawler): guard headless nav against internal SSRF targets on list/detail path

`navigate()` (list/detail/pagination) and the session probe called
`Browser::new_page()` without the SSRF check that already guards the
chapter-content path, so a hostile or compromised scraped source could
serve `<a href="http://169.254.169.254/…">` / `http://postgres:5432/` in a
listing and use the in-container Chromium as a read oracle.

Add `guard_navigate_url` (reusing `ensure_public_target`) at the top of
`navigate`, before rate-limiting or opening a page, and apply the same
check to `fetch_probe_html`. Covers base URL, pagination, and detail links.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-03 20:54:45 +02:00
parent 35c02066fe
commit a0d63ac9fd
5 changed files with 57 additions and 3 deletions

View File

@@ -285,6 +285,12 @@ where
}
async fn fetch_probe_html(browser: &Browser, probe_url: &str) -> anyhow::Result<String> {
// Guard the probe navigation for parity with the list/detail and
// chapter-content paths — the probe URL is operator-controlled, but
// keeping every `new_page` behind the same SSRF check avoids a gap if
// the URL ever becomes attacker-influenced.
crate::crawler::safety::ensure_public_target(probe_url)
.with_context(|| format!("refuse to navigate unsafe probe URL {probe_url}"))?;
let page = browser
.new_page(probe_url)
.await