fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: enable browser SSRF interception by default (H1)

Browse Source 
Off-by-default left headless-browser page JS/subresources/redirects able to reach
internal IPs (Chromium bypasses the reqwest SafeResolver). The crawler runs in the
backend container so it can't be network-isolated without breaking Postgres — the
CDP Fetch interceptor is the control. Flip CRAWLER_SSRF_INTERCEPT default to true
(config + .env.example + compose); off-path unchanged as break-glass. The
#[ignore]'d smoke test validates against real Chromium (command documented).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-13 21:51:05 +02:00
parent adce950049
commit e960aae163
6 changed files with 28 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
backend/src/config.rs
View File

@@ -520,12 +520,15 @@ pub struct CrawlerConfig {
/// exhausted) that trigger an automatic coordinated browser restart.
/// Defaults to 3. `CRAWLER_BROWSER_RESTART_THRESHOLD`.
pub browser_restart_threshold: u32,
/// Opt-in CDP `Fetch` interception that re-validates every headless-browser
/// navigation/redirect against the SSRF check (blocks a scraped page that
/// redirects the browser to an internal target). Default `false`: enabling
/// `Fetch` is a fragile hook in the crawler's critical path and the wiring
/// is not CI-verifiable (no Chromium in CI) — validate with a manual crawl
/// before turning on. `CRAWLER_SSRF_INTERCEPT`.
/// CDP `Fetch` interception that re-validates every headless-browser
/// navigation/redirect/subresource against the SSRF check. Default `true`:
/// with it off, only the top-level URL string is validated, so a scraped
/// page's JS/subresources (which use Chromium's own network stack, not the
/// reqwest `SafeResolver`) can reach internal targets like the cloud
/// metadata service or postgres. The Fetch hook can't be exercised in CI (no
/// Chromium) — the `#[ignore]`d `ssrf_interception_does_not_wedge_allowed_navigation`
/// smoke test validates it against a real binary. `CRAWLER_SSRF_INTERCEPT`;
/// set `false` only as a break-glass if the hook destabilizes a deployment.
pub ssrf_intercept: bool,
}
@@ -559,7 +562,7 @@ impl Default for CrawlerConfig {
job_timeout: Duration::from_secs(600),
metadata_max_consecutive_failures: 10,
browser_restart_threshold: 3,
ssrf_intercept: false,
ssrf_intercept: true,
}
}
}
@@ -725,7 +728,7 @@ impl CrawlerConfig {
) as u32,
browser_restart_threshold: env_u64("CRAWLER_BROWSER_RESTART_THRESHOLD", 3).max(1)
as u32,
ssrf_intercept: env_bool("CRAWLER_SSRF_INTERCEPT", false),
ssrf_intercept: env_bool("CRAWLER_SSRF_INTERCEPT", true),
})
}
}