fabi/Mangalord
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(crawler): cap the number of page images per chapter

Browse Source 
The per-image byte cap bounds each download but not the count, so a
hostile or compromised reader page listing thousands of <img> tags could
drive an unbounded disk fill. Add `CRAWLER_MAX_IMAGES_PER_CHAPTER`
(CrawlerConfig::max_images_per_chapter, default 2000, 0 = disabled) and
reject an over-cap chapter with a failed ack (exponential backoff) rather
than downloading it.

Threaded through sync_chapter_content and its three call sites (daemon
dispatcher, admin resync, CLI); enforced via `image_count_over_cap` right
after parse. The cap is an env-only safety knob, preserved across settings
reloads.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-03 21:05:28 +02:00
parent 592747f1e0
commit 5dc93bfb84
9 changed files with 72 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
backend/src/config.rs
View File

@@ -441,6 +441,13 @@ pub struct CrawlerConfig {
pub download_allowlist: DownloadAllowlist,
/// Hard upper bound on a single image download. Defaults to 32 MiB.
pub max_image_bytes: usize,
/// Hard upper bound on the number of page images in one chapter. A
/// hostile reader page could otherwise list thousands of `<img>`
/// tags; `max_image_bytes` caps each one but not the count, so the
/// product is an unbounded disk-fill. A chapter exceeding this is
/// acked failed rather than downloaded. `0` disables the cap.
/// Defaults to 2000. `CRAWLER_MAX_IMAGES_PER_CHAPTER`.
pub max_images_per_chapter: usize,
/// Max manga detail fetches per metadata pass. `0` means no cap
/// (full sweep up to the source's own bound). Sourced from
/// `CRAWLER_LIMIT`, mirroring the CLI binary.
@@ -485,6 +492,7 @@ impl Default for CrawlerConfig {
browser: LaunchOptions::headless(),
download_allowlist: DownloadAllowlist::new(),
max_image_bytes: DEFAULT_MAX_IMAGE_BYTES,
max_images_per_chapter: 2000,
manga_limit: 0,
job_timeout: Duration::from_secs(600),
metadata_max_consecutive_failures: 10,
@@ -642,6 +650,7 @@ impl CrawlerConfig {
browser: LaunchOptions::from_env(),
download_allowlist,
max_image_bytes: env_usize("CRAWLER_MAX_IMAGE_BYTES", DEFAULT_MAX_IMAGE_BYTES),
max_images_per_chapter: env_usize("CRAWLER_MAX_IMAGES_PER_CHAPTER", 2000),
manga_limit: env_usize("CRAWLER_LIMIT", 0),
job_timeout: Duration::from_secs(env_u64("CRAWLER_JOB_TIMEOUT_SECS", 600).max(1)),
metadata_max_consecutive_failures: env_u64(