fix: opt-in CDP re-validation of headless-browser navigations (SSRF)
The reqwest DNS resolver can't see Chromium navigations, so a scraped page that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP Fetch guard that re-validates every Document navigation/redirect through the same ensure_public_target + resolved-IP check, failing internal targets. Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in CI). When off, open_page is byte-identical to browser.new_page; when on, a guard-install failure falls back to an unguarded navigation rather than wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested; an #[ignore] smoke test covers the CDP path. Validate with a manual crawl before enabling in production. Bump to 0.124.11. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -511,6 +511,13 @@ pub struct CrawlerConfig {
|
||||
/// exhausted) that trigger an automatic coordinated browser restart.
|
||||
/// Defaults to 3. `CRAWLER_BROWSER_RESTART_THRESHOLD`.
|
||||
pub browser_restart_threshold: u32,
|
||||
/// Opt-in CDP `Fetch` interception that re-validates every headless-browser
|
||||
/// navigation/redirect against the SSRF check (blocks a scraped page that
|
||||
/// redirects the browser to an internal target). Default `false`: enabling
|
||||
/// `Fetch` is a fragile hook in the crawler's critical path and the wiring
|
||||
/// is not CI-verifiable (no Chromium in CI) — validate with a manual crawl
|
||||
/// before turning on. `CRAWLER_SSRF_INTERCEPT`.
|
||||
pub ssrf_intercept: bool,
|
||||
}
|
||||
|
||||
impl Default for CrawlerConfig {
|
||||
@@ -543,6 +550,7 @@ impl Default for CrawlerConfig {
|
||||
job_timeout: Duration::from_secs(600),
|
||||
metadata_max_consecutive_failures: 10,
|
||||
browser_restart_threshold: 3,
|
||||
ssrf_intercept: false,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -707,6 +715,7 @@ impl CrawlerConfig {
|
||||
) as u32,
|
||||
browser_restart_threshold: env_u64("CRAWLER_BROWSER_RESTART_THRESHOLD", 3).max(1)
|
||||
as u32,
|
||||
ssrf_intercept: env_bool("CRAWLER_SSRF_INTERCEPT", false),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user