fix(upload): stream chapter pages to storage instead of buffering the whole chapter

The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.

Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.

Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.

Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-03 21:33:47 +02:00
parent 141cd52f7e
commit 4e154434a1
13 changed files with 382 additions and 69 deletions

View File

@@ -55,6 +55,11 @@ pub struct UploadConfig {
/// reject a single oversized cover/page without failing the whole
/// request just because the total happens to fit.
pub max_file_bytes: usize,
/// Max page images accepted in one chapter upload. Bounds how many
/// parts the handler will stage before giving up, so a client can't
/// pin a worker streaming an unbounded page count. `0` disables the
/// cap. Defaults to 2000. `MAX_PAGES_PER_CHAPTER`.
pub max_pages_per_chapter: usize,
}
impl Default for UploadConfig {
@@ -62,6 +67,7 @@ impl Default for UploadConfig {
Self {
max_request_bytes: 200 * 1024 * 1024, // 200 MiB
max_file_bytes: 20 * 1024 * 1024, // 20 MiB
max_pages_per_chapter: 2000,
}
}
}
@@ -533,6 +539,7 @@ impl Config {
upload: UploadConfig {
max_request_bytes: env_usize("MAX_REQUEST_BYTES", 200 * 1024 * 1024),
max_file_bytes: env_usize("MAX_FILE_BYTES", 20 * 1024 * 1024),
max_pages_per_chapter: env_usize("MAX_PAGES_PER_CHAPTER", 2000),
},
cors_allowed_origins: std::env::var("CORS_ALLOWED_ORIGINS")
.ok()