fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(crawler): CRAWLER_ALLOW_ANY_HOST bypasses the host allowlist (0.44.0)

Browse Source 
Operators whose sources shard images across numbered CDN subdomains
(cdn1/cdn2/…) can't realistically pre-enumerate every host in
CRAWLER_DOWNLOAD_ALLOWLIST. The new flag short-circuits the host check
in DownloadAllowlist::contains while leaving the scheme, localhost,
and private-IP defenses in is_safe_url untouched, so scraped URLs
pointing at 10.x / 169.254.169.254 / file:// stay refused.

Wired in both the server (config::build_download_allowlist) and the
one-shot bin/crawler.rs entry point. Default is false — fail-closed
posture is preserved unless the operator opts in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-05-31 14:52:37 +02:00
parent 1eebb90e25
commit d7173aa180
7 changed files with 111 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
backend/Cargo.lock generated
View File

@@ -1470,7 +1470,7 @@ checksum = "c41e0c4fef86961ac6d6f8a82609f55f31b05e4fce149ac5710e439df7619ba4"
[[package]]
name = "mangalord"
version = "0.43.1"
version = "0.44.0"
dependencies = [
"anyhow",
"argon2",