fix(admin-security): SSRF defence + CSRF fail-closed + analysis .no_proxy() (0.87.2)

Four high/medium findings from the audit, plus their tests:

1. **SSRF on admin-editable URLs.** `CrawlerSettings::start_url` and
   `AnalysisSettings::endpoint` validated only with `Url::parse`. A
   hostile or CSRF-able admin could repoint at `169.254.169.254`
   (cloud metadata), `127.0.0.1:5432` (postgres), or any RFC1918 host
   — and the vision worker bearer-attaches an env-managed API key to
   every call. Extract `ensure_public_target` from `safety::is_safe_url`
   (allowlist-free public-host check: scheme + private-IP literal +
   localhost) and wire it into both fields. Docker DNS names
   (`mangalord-vision`) keep passing because they're hostnames, not
   IP literals. The analysis check is gated on `enabled=true` so the
   dev-default `localhost:8000` stays usable until the worker is
   actually turned on (toggling enabled=true later re-runs the gate).

2. **Admin CSRF fail-open default.** `ADMIN_ALLOWED_ORIGINS=` empty
   silently skipped the entire CSRF check, so an operator forgetting
   the env var shipped an unguarded admin surface. Cookie-auth POSTs
   now fail-closed when the allowlist is empty AND when neither
   `Origin` nor `Referer` accompanies the request. Two carve-outs:
   `Authorization: Bearer …` callers bypass (bots can't be CSRF'd),
   and requests with NO session cookie at all bypass to let the auth
   extractor return a clean 401 instead of a confusing 403.

3. **Analysis HTTP clients didn't `.no_proxy()`.** The crawler client
   already did. Ambient `HTTP_PROXY`/`HTTPS_PROXY` in container env
   would exfiltrate page-image bytes + the bearer key through an
   upstream proxy. Same for the readiness probe.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/tests/api_admin_crawler.rs

@@ -357,27 +357,35 @@ async fn csrf_allows_mutation_with_allowed_origin(pool: PgPool) {
 }
 
 #[sqlx::test(migrations = "./migrations")]
-async fn csrf_allows_mutation_without_origin_or_referer(pool: PgPool) {
-    // curl/server-to-server callers send neither — they can't be a CSRF
-    // vector since there's no third-party browser context.
+async fn csrf_rejects_cookie_auth_without_origin_or_referer(pool: PgPool) {
+    // Cookie-auth + neither Origin nor Referer was previously waved through
+    // ("curl bypass"). It's now refused: an extension or no-referrer-policy
+    // page can produce exactly this shape and would otherwise be a CSRF
+    // vector. Real curl/script callers should authenticate with a bearer
+    // token instead — see `csrf_allows_bearer_token_without_origin`.
     let h = harness_with_admin_origins(
         pool.clone(),
         vec!["http://localhost:3000".to_string()],
     );
     let cookie = seed_admin(&pool, &h.app).await;
+    // `post_json_with_cookie_origin(..., None, None)` builds the exact "no
+    // Origin, no Referer" cookie-auth shape this test pins.
     let resp = h
         .app
         .clone()
-        .oneshot(post_json_with_cookie(
+        .oneshot(post_json_with_cookie_origin(
             "/api/v1/admin/crawler/session/clear-expired",
             json!({}),
             &cookie,
+            None,
+            None,
         ))
         .await
         .unwrap();
-    assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
+    assert_eq!(resp.status(), StatusCode::FORBIDDEN);
 }
 
+
 #[sqlx::test(migrations = "./migrations")]
 async fn csrf_falls_back_to_referer_when_origin_missing(pool: PgPool) {
     let h = harness_with_admin_origins(
@@ -422,11 +430,13 @@ async fn csrf_skipped_on_safe_methods(pool: PgPool) {
 }
 
 #[sqlx::test(migrations = "./migrations")]
-async fn csrf_disabled_when_allowlist_empty(pool: PgPool) {
-    // Default harness has admin_allowed_origins = empty → operator
-    // opt-out documented in .env.example. A cross-origin POST passes
-    // through to the handler.
-    let h = harness(pool.clone());
+async fn csrf_rejects_cookie_auth_when_allowlist_empty(pool: PgPool) {
+    // Previously an empty allowlist silently skipped the CSRF check, so an
+    // operator who forgot to set ADMIN_ALLOWED_ORIGINS shipped an unguarded
+    // admin surface. We now fail-closed for cookie-auth admin mutations
+    // when the allowlist is empty — operators must either set the env var
+    // (browser deploy) or authenticate with `Authorization: Bearer …`.
+    let h = harness_with_admin_origins(pool.clone(), vec![]);
     let cookie = seed_admin(&pool, &h.app).await;
     let resp = h
         .app
@@ -440,7 +450,7 @@ async fn csrf_disabled_when_allowlist_empty(pool: PgPool) {
         ))
         .await
         .unwrap();
-    assert_eq!(resp.status(), StatusCode::SERVICE_UNAVAILABLE);
+    assert_eq!(resp.status(), StatusCode::FORBIDDEN);
 }
 
 // ---------------------------------------------------------------------------