fix: opt-in CDP re-validation of headless-browser navigations (SSRF)
The reqwest DNS resolver can't see Chromium navigations, so a scraped page that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP Fetch guard that re-validates every Document navigation/redirect through the same ensure_public_target + resolved-IP check, failing internal targets. Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in CI). When off, open_page is byte-identical to browser.new_page; when on, a guard-install failure falls back to an unguarded navigation rather than wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested; an #[ignore] smoke test covers the CDP path. Validate with a manual crawl before enabling in production. Bump to 0.124.11. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -564,6 +564,10 @@ async fn spawn_crawler_daemon(
|
||||
cfg: &CrawlerConfig,
|
||||
analysis_enabled: Arc<AtomicBool>,
|
||||
) -> anyhow::Result<SpawnedDaemon> {
|
||||
// Publish the opt-in browser SSRF-interception toggle so every headless
|
||||
// navigation (via `intercept::open_page`) honours it. Off by default.
|
||||
crate::crawler::intercept::set_enabled(cfg.ssrf_intercept);
|
||||
|
||||
// Reqwest client with a shared cookie jar so CDN image fetches include
|
||||
// PHPSESSID. The same `Arc<Jar>` is held by the SessionController, so a
|
||||
// runtime session refresh rewrites it in place. Initial value: a
|
||||
|
||||
Reference in New Issue
Block a user