feat(crawler): CRAWLER_ALLOW_ANY_HOST bypasses the host allowlist (0.44.0) · a2826d6467 - Mangalord

feat(crawler): CRAWLER_ALLOW_ANY_HOST bypasses the host allowlist (0.44.0)

Some checks failed

deploy / test-backend (push) Failing after 11s

Details

deploy / test-frontend (push) Failing after 36s

Details

deploy / build-and-push (push) Has been skipped

Details

deploy / deploy (push) Has been skipped

Details

Operators whose sources shard images across numbered CDN subdomains
can't pre-enumerate every host in CRAWLER_DOWNLOAD_ALLOWLIST. The new
flag short-circuits the host check in DownloadAllowlist::contains
while leaving scheme, localhost, and private-IP defenses in
is_safe_url untouched — scraped URLs pointing at 10.x /
169.254.169.254 / file:// stay refused. Default is false; fail-closed
posture is preserved unless the operator opts in. Wired into both the
server (config::build_download_allowlist) and the bin/crawler.rs
one-shot.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

This commit is contained in:

MechaCat02

2026-05-31 14:52:49 +02:00

parent 1eebb90e25

commit a2826d6467

7 changed files with 111 additions and 20 deletions

									
										2

backend/Cargo.toml
									
												View File
												
				@@ -1,6 +1,6 @@

				[package]

				name = "mangalord"

				version = "0.43.1"

				version = "0.44.0"

				edition = "2021"

				default-run = "mangalord"

2 backend/Cargo.toml Unescape Escape View File

2

backend/Cargo.toml

View File