feat(crawler): CRAWLER_ALLOW_ANY_HOST bypasses the host allowlist (0.44.0)

Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

backend/src/config.rs

@@ -248,10 +248,17 @@ impl CrawlerConfig {
 /// separated). Empty by default — meaning the crawler refuses to
 /// download anything when no source is configured, which is the safe
 /// fail-closed posture.
+///
+/// `CRAWLER_ALLOW_ANY_HOST=true` short-circuits the host enumeration
+/// for operators whose sources shard across numbered CDN subdomains.
+/// Scheme + private-IP defenses still apply.
 fn build_download_allowlist(
     start_url: Option<&str>,
     cdn_host: Option<&str>,
 ) -> DownloadAllowlist {
+    if env_bool("CRAWLER_ALLOW_ANY_HOST", false) {
+        return DownloadAllowlist::allow_any();
+    }
     let mut allow = DownloadAllowlist::new();
     if let Some(url) = start_url {
         if let Ok(parsed) = reqwest::Url::parse(url) {