fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(admin-security): SSRF defence + CSRF fail-closed + analysis .no_proxy() (0.87.2)

Browse Source 
Four high/medium findings from the audit, plus their tests:

1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
   `AnalysisSettings::endpoint` validated only with `Url::parse`. A
   hostile or CSRF-able admin could repoint at `169.254.169.254`
   (cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
   — and the vision worker bearer-attaches an env-managed API key to
   every call. Extract `ensure_public_target` from `safety::is_safe_url`
   (allowlist-free public-host check: scheme + private-IP literal +
   localhost) and wire it into both fields. Docker DNS names
   (`mangalord-vision`) keep passing because they're hostnames, not
   IP literals. The analysis check is gated on `enabled=true` so the
   dev-default `localhost:8000` stays usable until the worker is
   actually turned on (toggling enabled=true later re-runs the gate).

2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
   silently skipped the entire CSRF check, so an operator forgetting
   the env var shipped an unguarded admin surface. Cookie-auth POSTs
   now fail-closed when the allowlist is empty AND when neither
   `Origin` nor `Referer` accompanies the request. Two carve-outs:
   `Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
   and requests with NO session cookie at all bypass to let the auth
   extractor return a clean 401 instead of a confusing 403.

3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
   already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
   would exfiltrate page-image bytes + the bearer key through an
   upstream proxy. Same for the readiness probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-22 21:04:50 +02:00
parent b14ed02670
commit dd25f073cd
10 changed files with 364 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

84
backend/src/app.rs
View File

@@ -413,6 +413,11 @@ fn spawn_analysis_daemon(
) -> anyhow::Result<crate::analysis::daemon::AnalysisDaemonHandle> {
let http = reqwest::Client::builder()
.timeout(cfg.request_timeout)
// Refuse to honour ambient HTTP_PROXY / HTTPS_PROXY container env.
// The vision call carries an env-managed bearer token + page image
// bytes; a stray upstream proxy would exfiltrate both. Mirrors the
// crawler client's `.no_proxy()` (see `spawn_crawler_daemon`).
.no_proxy()
.build()
.context("build analysis http client")?;
let vision = crate::analysis::vision::VisionClient::new(http, cfg);
@@ -432,6 +437,11 @@ fn spawn_analysis_daemon(
Some(url) if !url.is_empty() => {
let probe = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(5))
// Same reasoning as the main analysis client: do not
// honour ambient HTTP_PROXY env. The readiness probe is
// unauthenticated but a hostile upstream still gets a
// useful side-channel on backend uptime + vision health.
.no_proxy()
.build()
.context("build vision readiness http client")?;
Some(Arc::new(crate::analysis::daemon::HttpVisionReadiness {
@@ -1028,10 +1038,24 @@ const ADMIN_PATH_PREFIX: &str = "/api/v1/admin/";
/// from a malicious page — this middleware rejects such requests by
/// comparing the request's `Origin` (with `Referer` as fallback) against
/// the configured allowlist. Safe methods (`GET`/`HEAD`/`OPTIONS`) are
/// always allowed. Requests with neither `Origin` nor `Referer` are
/// allowed (non-browser callers like curl can't be a CSRF vector). When
/// the allowlist is empty the check is skipped entirely (operator
/// opt-out — documented in `.env.example`).
/// always allowed.
///
/// **Bearer-token requests** (`Authorization: Bearer …`) are bot API
/// callers — they can't be a CSRF vector because the browser never
/// attaches that header automatically. Skip the check for them.
///
/// **Cookie-auth requests** must:
/// * be from an allowed origin (`Origin` then `Referer`), AND
/// * have at least one of those headers present (a browser always
/// sends one on a cross-site POST; a missing pair on a cookie-auth
/// request is the exact niche an extension / no-referrer-policy
/// CSRF would try to exploit).
///
/// When the allowlist is empty we fail-closed for cookie-auth requests
/// — this used to silently let everything through, so an operator who
/// forgot to set `ADMIN_ALLOWED_ORIGINS` shipped an unguarded admin
/// surface. Operators on a pure-bot-token deploy aren't impacted (their
/// requests carry `Authorization: Bearer …` and skip the gate above).
async fn admin_csrf_guard(
State(state): State<AppState>,
req: Request,
@@ -1046,16 +1070,56 @@ async fn admin_csrf_guard(
) {
return Ok(next.run(req).await);
}
if state.admin_allowed_origins.is_empty() {
let headers = req.headers();
// Bearer-token requests are non-browser bot callers and bypass the
// gate entirely (the Authorization header is never set by an
// attacker page).
let is_bearer = headers
.get(axum::http::header::AUTHORIZATION)
.and_then(|v| v.to_str().ok())
.is_some_and(|v| v.trim_start().to_ascii_lowercase().starts_with("bearer "));
if is_bearer {
return Ok(next.run(req).await);
}
// No session cookie either → let the auth extractor return a clean 401
// ("log in first"). A no-auth request can't be a CSRF vector — there's
// no authority to ride. Without this, an anonymous curl POST to an
// admin endpoint surfaces 403 with our CSRF message instead of the
// expected 401, which is confusing for both operators and tests.
let has_session_cookie = headers
.get(axum::http::header::COOKIE)
.and_then(|v| v.to_str().ok())
.is_some_and(|raw| raw.split(';').any(|p| {
p.trim_start().starts_with("mangalord_session=")
}));
if !has_session_cookie {
return Ok(next.run(req).await);
}
let headers = req.headers();
let origin = headers.get("origin").and_then(|v| v.to_str().ok());
let referer = headers.get("referer").and_then(|v| v.to_str().ok());
// No Origin AND no Referer → server-to-server / curl / extension.
// Browsers always send one or the other on a cross-site POST.
let Some(candidate) = origin.or(referer) else {
return Ok(next.run(req).await);
let candidate = origin.or(referer);
if state.admin_allowed_origins.is_empty() {
// Fail-closed: cookie-auth admin mutations without an allowlist
// configuration are refused. Set `ADMIN_ALLOWED_ORIGINS` for the
// browser deployment, or call with `Authorization: Bearer …`.
tracing::warn!(
path = %req.uri().path(),
"admin CSRF: ADMIN_ALLOWED_ORIGINS is empty — cookie-auth admin mutations are refused (set the env var for browser-exposed deploys)"
);
return Err(AppError::Forbidden);
}
// Cookie-auth path with an allowlist configured: a browser always
// sends Origin or Referer on a cross-site POST. A missing pair is
// either a no-referrer-policy edge case or a tool deliberately
// hiding origin — refuse rather than risk it.
let Some(candidate) = candidate else {
tracing::warn!(
path = %req.uri().path(),
"admin CSRF: cookie-auth request with neither Origin nor Referer — refusing"
);
return Err(AppError::Forbidden);
};
if origin_in_allowlist(candidate, &state.admin_allowed_origins) {
return Ok(next.run(req).await);

98
backend/src/crawler/safety.rs
View File

@@ -124,6 +124,34 @@ impl DownloadAllowlist {
/// An empty allowlist rejects everything (the conservative default —
/// callers must explicitly allow the catalog and CDN hosts).
pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSafetyError> {
let url = ensure_public_target_inner(raw_url)?;
let lower_host = url
.host_str()
.expect("host validated by ensure_public_target_inner")
.to_ascii_lowercase();
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
}
/// Validate that an admin-supplied URL points at a publicly-routable target:
/// scheme is http or https, host is present, host isn't `localhost`, and (if
/// the host is an IP literal) it isn't loopback/private/link-local/CGNAT/etc.
///
/// Used to validate `endpoint`-style admin settings (crawler `start_url`,
/// analysis `endpoint`) where there is no per-deployment allowlist to consult,
/// but where a hostile or careless admin value (e.g. `http://169.254.169.254/`,
/// `http://127.0.0.1:5432/`) would let the worker pivot inside the deployment.
///
/// Note: DNS hostnames (`mangalord-vision`, `vision.internal.corp`) pass — the
/// check is on *literal* IP private-range strings only, so the documented
/// docker-internal vision endpoint keeps working.
pub fn ensure_public_target(raw_url: &str) -> Result<(), UrlSafetyError> {
ensure_public_target_inner(raw_url).map(|_| ())
}
fn ensure_public_target_inner(raw_url: &str) -> Result<Url, UrlSafetyError> {
let url = Url::parse(raw_url).map_err(|_| UrlSafetyError::Unparseable)?;
let scheme = url.scheme();
if scheme != "http" && scheme != "https" {
@@ -148,10 +176,7 @@ pub fn is_safe_url(raw_url: &str, allow: &DownloadAllowlist) -> Result<(), UrlSa
return Err(UrlSafetyError::PrivateIp(ip));
}
}
if !allow.contains(&lower_host) {
return Err(UrlSafetyError::HostNotAllowed(lower_host));
}
Ok(())
Ok(url)
}
fn is_private_ip(ip: &IpAddr) -> bool {
@@ -383,6 +408,71 @@ mod tests {
));
}
// --- ensure_public_target (allowlist-free admin URL validation) ---
#[test]
fn public_target_allows_dns_hostnames() {
// The whole point of this helper is to validate admin-supplied
// endpoint URLs (analysis vision endpoint, crawler start_url)
// WITHOUT consulting an allowlist. Docker-internal DNS names
// (which resolve at runtime to private IPs) MUST pass.
assert!(ensure_public_target("http://mangalord-vision:8000/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://api.openai.com/v1/chat/completions").is_ok());
assert!(ensure_public_target("https://example.com/").is_ok());
}
#[test]
fn public_target_blocks_ip_literal_attacks() {
// Literal IPs in private/loopback/link-local ranges — the routes
// an attacker would actually use (AWS IMDS, postgres on
// 127.0.0.1, RFC1918 inside a corp network).
for url in [
"http://169.254.169.254/latest/meta-data/",
"http://127.0.0.1:5432/",
"http://10.0.0.1/",
"http://192.168.1.1/",
"http://[::1]/",
"http://[::ffff:127.0.0.1]/",
"http://0.0.0.0/",
] {
assert!(
matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::PrivateIp(_)
),
"must reject {url}"
);
}
}
#[test]
fn public_target_blocks_localhost_hostname() {
assert!(matches!(
ensure_public_target("http://localhost:5432/").unwrap_err(),
UrlSafetyError::Loopback
));
}
#[test]
fn public_target_blocks_non_http_schemes() {
for url in ["file:///etc/passwd", "gopher://x.example/", "ftp://x/"] {
assert!(matches!(
ensure_public_target(url).unwrap_err(),
UrlSafetyError::BadScheme(_)
));
}
}
#[test]
fn public_target_rejects_unparseable() {
assert!(matches!(
ensure_public_target("not a url").unwrap_err(),
UrlSafetyError::Unparseable
));
}
// --- back to is_safe_url ---
#[test]
fn safe_url_blocks_rfc1918() {
let allow = allow_just("10.0.0.1");

138
backend/src/settings.rs
View File

@@ -153,8 +153,13 @@ impl CrawlerSettings {
errs.push("chapter_workers", "must be at least 1");
}
if let Some(url) = self.start_url.as_deref().map(str::trim).filter(|s| !s.is_empty()) {
if reqwest::Url::parse(url).is_err() {
errs.push("start_url", "must be a valid absolute URL");
// SSRF defence: Url::parse alone admits http://169.254.169.254
// / http://127.0.0.1:5432 etc., and the browser would happily
// navigate there with the admin's session. ensure_public_target
// refuses literal-IP private ranges + localhost while still
// accepting public hosts.
if let Err(e) = crate::crawler::safety::ensure_public_target(url) {
errs.push("start_url", url_safety_message(&e));
}
}
if self.job_timeout_secs < 1 {
@@ -231,6 +236,22 @@ impl CrawlerSettings {
}
}
/// Render a [`crate::crawler::safety::UrlSafetyError`] as an operator-friendly
/// validation message. The variant detail is preserved so a deliberate
/// `10.0.0.1` value gets "private/loopback IP not allowed" rather than the
/// generic "invalid URL".
fn url_safety_message(e: &crate::crawler::safety::UrlSafetyError) -> &'static str {
use crate::crawler::safety::UrlSafetyError as E;
match e {
E::Unparseable => "must be a valid absolute URL",
E::BadScheme(_) => "must use http:// or https://",
E::NoHost => "must include a host",
E::Loopback => "must not point at localhost",
E::PrivateIp(_) => "must not point at a private/loopback/link-local IP (SSRF risk)",
E::HostNotAllowed(_) => "host is not on the allowlist",
}
}
/// Rebuild a [`DownloadAllowlist`] from the DTO, always seeding the start-url
/// and CDN hosts (mirrors `config::build_download_allowlist`).
fn build_allowlist(
@@ -334,8 +355,29 @@ impl AnalysisSettings {
if self.workers < 1 {
errs.push("workers", "must be at least 1");
}
if self.endpoint.trim().is_empty() || reqwest::Url::parse(self.endpoint.trim()).is_err() {
let trimmed_endpoint = self.endpoint.trim();
if trimmed_endpoint.is_empty() {
errs.push("endpoint", "must be a valid absolute URL");
} else if let Err(e) = reqwest::Url::parse(trimmed_endpoint) {
// Always reject obviously-malformed URLs (matches prior behaviour).
let _ = e;
errs.push("endpoint", "must be a valid absolute URL");
} else if self.enabled {
// SSRF + API-key exfiltration defence — only enforced when the
// worker is actually live. Worker attaches an env-managed bearer
// token to every call; without this check, an admin (or one
// CSRF-able request) could repoint the endpoint at
// http://169.254.169.254 (cloud metadata) or http://postgres:5432
// (an internal service that would receive the bearer header).
// Docker DNS names (`mangalord-vision`) still pass because
// they're hostnames, not IP literals.
//
// A disabled config is allowed to carry the dev-default
// `http://localhost:8000/...` placeholder; the worker won't dial
// it, and toggling `enabled=true` later re-runs this validator.
if let Err(e) = crate::crawler::safety::ensure_public_target(trimmed_endpoint) {
errs.push("endpoint", url_safety_message(&e));
}
}
if self.enabled && self.model.trim().is_empty() {
errs.push("model", "required when analysis is enabled");
@@ -525,6 +567,37 @@ mod tests {
assert!(!out.download_allowlist.is_allow_any());
}
#[test]
fn crawler_start_url_rejects_ip_literal_attacks() {
let base = CrawlerConfig::default();
for url in [
"http://169.254.169.254/",
"http://127.0.0.1/catalog",
"http://localhost:5432/",
"http://10.0.0.1/",
] {
let dto = CrawlerSettings {
start_url: Some(url.to_string()),
..CrawlerSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"start_url"), "must reject {url}");
}
}
#[test]
fn crawler_start_url_accepts_public_targets() {
let base = CrawlerConfig::default();
for url in ["https://catalog.example.com/", "http://catalog.example.com/"] {
let dto = CrawlerSettings {
start_url: Some(url.to_string()),
..CrawlerSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok(), "must accept {url}");
}
}
#[test]
fn crawler_allow_any_host_bypasses_list() {
let base = CrawlerConfig::default();
@@ -617,6 +690,65 @@ mod tests {
}
}
#[test]
fn analysis_endpoint_rejects_ip_literal_attacks_when_enabled() {
// The vision worker bearer-attaches an env secret to every call;
// a hostile/CSRF-able admin must NOT be able to point endpoint at
// cloud metadata, loopback services, or RFC1918 hosts — when the
// worker is enabled (toggling enabled=true later re-runs this gate).
let base = AnalysisConfig::default();
for url in [
"http://169.254.169.254/v1/chat/completions",
"http://127.0.0.1:5432/",
"http://localhost:11434/v1/chat/completions",
"http://10.0.0.5/v1/chat/completions",
] {
let dto = AnalysisSettings {
enabled: true,
model: "test-model".to_string(),
endpoint: url.to_string(),
..AnalysisSettings::from_config(&base)
};
let errs = dto.to_config(&base).unwrap_err();
let fields: Vec<_> = errs.errors.iter().map(|e| e.field.as_str()).collect();
assert!(fields.contains(&"endpoint"), "must reject {url}");
}
}
#[test]
fn analysis_endpoint_accepts_docker_dns_and_public_hosts_when_enabled() {
// The documented default — docker DNS name resolving to a private IP
// at runtime — must still validate, because the bearer recipient
// identity is the operator-chosen hostname, not the underlying IP.
let base = AnalysisConfig::default();
for url in [
"http://mangalord-vision:8000/v1/chat/completions",
"https://api.openai.com/v1/chat/completions",
] {
let dto = AnalysisSettings {
enabled: true,
model: "test-model".to_string(),
endpoint: url.to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok(), "must accept {url}");
}
}
#[test]
fn analysis_endpoint_skips_safety_check_when_disabled() {
// Disabled-but-saved bad URL is harmless (worker won't dial it);
// operators get to keep the dev-localhost placeholder until they
// toggle the worker on, when the validator re-runs.
let base = AnalysisConfig::default();
let dto = AnalysisSettings {
enabled: false,
endpoint: "http://localhost:8000/v1/chat/completions".to_string(),
..AnalysisSettings::from_config(&base)
};
assert!(dto.to_config(&base).is_ok());
}
#[test]
fn analysis_requires_model_only_when_enabled() {
let base = AnalysisConfig::default();