fix(crawler): guard headless nav against internal SSRF targets on list/detail path
`navigate()` (list/detail/pagination) and the session probe called `Browser::new_page()` without the SSRF check that already guards the chapter-content path, so a hostile or compromised scraped source could serve `<a href="http://169.254.169.254/…">` / `http://postgres:5432/` in a listing and use the in-container Chromium as a read oracle. Add `guard_navigate_url` (reusing `ensure_public_target`) at the top of `navigate`, before rate-limiting or opening a page, and apply the same check to `fetch_probe_html`. Covers base URL, pagination, and detail links. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -285,6 +285,12 @@ where
|
||||
}
|
||||
|
||||
async fn fetch_probe_html(browser: &Browser, probe_url: &str) -> anyhow::Result<String> {
|
||||
// Guard the probe navigation for parity with the list/detail and
|
||||
// chapter-content paths — the probe URL is operator-controlled, but
|
||||
// keeping every `new_page` behind the same SSRF check avoids a gap if
|
||||
// the URL ever becomes attacker-influenced.
|
||||
crate::crawler::safety::ensure_public_target(probe_url)
|
||||
.with_context(|| format!("refuse to navigate unsafe probe URL {probe_url}"))?;
|
||||
let page = browser
|
||||
.new_page(probe_url)
|
||||
.await
|
||||
|
||||
Reference in New Issue
Block a user