fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: enable browser SSRF interception by default (H1)

Browse Source 
Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-13 21:51:05 +02:00
parent adce950049
commit e960aae163
6 changed files with 28 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
.env.example
View File

@@ -191,12 +191,19 @@ CRAWLER_METADATA_MAX_CONSECUTIVE_FAILURES=10
# exhausted) that trigger an automatic coordinated browser restart.
# Default 3.
CRAWLER_BROWSER_RESTART_THRESHOLD=3
# Opt-in CDP Fetch interception that re-validates every headless-browser
# navigation/redirect against the SSRF check (blocks a scraped page that
# redirects the browser to an internal target). Default false — enabling
# Fetch is a fragile hook in the crawler's critical path and is not
# CI-verified; validate with a manual crawl before turning on.
CRAWLER_SSRF_INTERCEPT=false
# CDP Fetch interception that re-validates every headless-browser
# navigation/redirect/subresource against the SSRF check (blocks a scraped page
# that drives the browser — via a redirect OR page JS/subresource — to an
# internal target such as the cloud metadata service or postgres). Default TRUE:
# with it off, only the top-level URL string is checked and page JS/subresources
# reach internal IPs (the reqwest SafeResolver does not cover Chromium's own
# stack). The CDP Fetch hook can't be exercised in CI (no Chromium); before
# relying on a fresh deploy, validate it doesn't wedge navigation:
# CRAWLER_CHROMIUM_BINARY=/usr/bin/chromium \
# cargo test -p mangalord --test crawler_browser_smoke -- --ignored \
# ssrf_interception_does_not_wedge_allowed_navigation
# Set to false only as a break-glass if the hook destabilizes a deployment.
CRAWLER_SSRF_INTERCEPT=true
# Path to a system Chromium binary. When set, the crawler skips the
# bundled-fetcher download. Required on platforms without a usable
# upstream Chromium build (notably Linux_arm64 / Raspberry Pi). On