fix: opt-in CDP re-validation of headless-browser navigations (SSRF)

The reqwest DNS resolver can't see Chromium navigations, so a scraped page
that redirects the browser to an internal target (302 -> 127.0.0.1:5432, or
a rebinding hostname) was still loaded. Add crawler::intercept: an opt-in CDP
Fetch guard that re-validates every Document navigation/redirect through the
same ensure_public_target + resolved-IP check, failing internal targets.

Gated behind CRAWLER_SSRF_INTERCEPT (default OFF): enabling Fetch is a fragile
critical-path hook and the CDP wiring is not CI-verifiable (no Chromium in
CI). When off, open_page is byte-identical to browser.new_page; when on, a
guard-install failure falls back to an unguarded navigation rather than
wedging the crawl. The pure decision logic (verdict/is_blocked) is unit-tested;
an #[ignore] smoke test covers the CDP path. Validate with a manual crawl
before enabling in production.

Bump to 0.124.11.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-08 06:29:17 +02:00
parent ff4ca964f5
commit bf425cf8e6
15 changed files with 329 additions and 10 deletions

View File

@@ -184,6 +184,12 @@ CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
# exhausted) that trigger an automatic coordinated browser restart.
# Default 3.
CRAWLER_BROWSER_RESTART_THRESHOLD=3
# Opt-in CDP Fetch interception that re-validates every headless-browser
# navigation/redirect against the SSRF check (blocks a scraped page that
# redirects the browser to an internal target). Default false — enabling
# Fetch is a fragile hook in the crawler's critical path and is not
# CI-verified; validate with a manual crawl before turning on.
CRAWLER_SSRF_INTERCEPT=false
# Path to a system Chromium binary. When set, the crawler skips the
# bundled-fetcher download. Required on platforms without a usable
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On