fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(admin-security): SSRF defence + CSRF fail-closed + analysis .no_proxy() (0.87.2)

Browse Source 
Four high/medium findings from the audit, plus their tests:

1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
   `AnalysisSettings::endpoint` validated only with `Url::parse`. A
   hostile or CSRF-able admin could repoint at `169.254.169.254`
   (cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
   — and the vision worker bearer-attaches an env-managed API key to
   every call. Extract `ensure_public_target` from `safety::is_safe_url`
   (allowlist-free public-host check: scheme + private-IP literal +
   localhost) and wire it into both fields. Docker DNS names
   (`mangalord-vision`) keep passing because they're hostnames, not
   IP literals. The analysis check is gated on `enabled=true` so the
   dev-default `localhost:8000` stays usable until the worker is
   actually turned on (toggling enabled=true later re-runs the gate).

2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
   silently skipped the entire CSRF check, so an operator forgetting
   the env var shipped an unguarded admin surface. Cookie-auth POSTs
   now fail-closed when the allowlist is empty AND when neither
   `Origin` nor `Referer` accompanies the request. Two carve-outs:
   `Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
   and requests with NO session cookie at all bypass to let the auth
   extractor return a clean 401 instead of a confusing 403.

3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
   already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
   would exfiltrate page-image bytes + the bearer key through an
   upstream proxy. Same for the readiness probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-22 21:04:50 +02:00
parent b14ed02670
commit dd25f073cd
10 changed files with 364 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

98
backend/src/crawler/safety.rs
View File

@@ -124,6 +124,34 @@ impl DownloadAllowlist {
/// An empty allowlist rejects everything (the conservative default —
/// callers must explicitly allow the catalog and CDN hosts).
pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSafetyError> {
let url = ensure_public_target_inner(raw_url)?;
let lower_host = url
.host_str()
.expect("host validated by ensure_public_target_inner")
.to_ascii_lowercase();
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
}
/// Validate that an admin-supplied URL points at a publicly-routable target:
/// scheme is http or https, host is present, host isn't `localhost`, and (if
/// the host is an IP literal) it isn't loopback/private/link-local/CGNAT/etc.
///
/// Used to validate `endpoint`-style admin settings (crawler `start_url`,
/// analysis `endpoint`) where there is no per-deployment allowlist to consult,
/// but where a hostile or careless admin value (e.g. `http://169.254.169.254/`,
/// `http://127.0.0.1:5432/`) would let the worker pivot inside the deployment.
///
/// Note: DNS hostnames (`mangalord-vision`, `vision.internal.corp`) pass — the
/// check is on *literal* IP private-range strings only, so the documented
/// docker-internal vision endpoint keeps working.
pub fn ensure_public_target(raw_url: &str) -> Result<(), UrlSafetyError> {
ensure_public_target_inner(raw_url).map(|_| ())
}
fn ensure_public_target_inner(raw_url: &str) -> Result<Url, UrlSafetyError> {
let url = Url::parse(raw_url).map_err(|_| UrlSafetyError::Unparseable)?;
let scheme = url.scheme();
if scheme != "http" && scheme != "https" {
@@ -148,10 +176,7 @@ pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSa
return Err(UrlSafetyError::PrivateIp(ip));
}
}
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
Ok(url)
}
fn is_private_ip(ip: &IpAddr) -> bool {
@@ -383,6 +408,71 @@ mod tests {
));
}
// --- ensure_public_target (allowlist-free admin URL validation) ---
#[test]
fn public_target_allows_dns_hostnames() {
// The whole point of this helper is to validate admin-supplied
// endpoint URLs (analysis vision endpoint, crawler start_url)
// WITHOUT consulting an allowlist. Docker-internal DNS names
// (which resolve at runtime to private IPs) MUST pass.
assert!(ensure_public_target("http://mangalord-vision:8000/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://api.openai.com/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://example.com/").is_ok());
}
#[test]
fn public_target_blocks_ip_literal_attacks() {
// Literal IPs in private/loopback/link-local ranges — the routes
// an attacker would actually use (AWS IMDS, postgres on
// 127.0.0.1, RFC1918 inside a corp network).
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:5432/",
"http://10.0.0.1/",
"http://192.168.1.1/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://0.0.0.0/",
] {
assert!(
matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::PrivateIp(_)
),
"must reject {url}"
);
}
}
#[test]
fn public_target_blocks_localhost_hostname() {
assert!(matches!(
ensure_public_target("http://localhost:5432/").unwrap_err(),
UrlSafetyError::Loopback
));
}
#[test]
fn public_target_blocks_non_http_schemes() {
for url in ["file:///etc/passwd", "gopher://x.example/", "ftp://x/"] {
assert!(matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::BadScheme(_)
));
}
}
#[test]
fn public_target_rejects_unparseable() {
assert!(matches!(
ensure_public_target("not a url").unwrap_err(),
UrlSafetyError::Unparseable
));
}
// --- back to is_safe_url ---
#[test]
fn safe_url_blocks_rfc1918() {
let allow = allow_just("10.0.0.1");