fix(upload): stream chapter pages to storage instead of buffering the whole chapter
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
10
.env.example
10
.env.example
@@ -103,6 +103,11 @@ MAX_REQUEST_BYTES=209715200
|
||||
# oversized image is rejected even when the total request fits.
|
||||
# Default 20 MiB.
|
||||
MAX_FILE_BYTES=20971520
|
||||
# Max page images accepted in one chapter upload. Bounds how many parts
|
||||
# the handler will stage before rejecting the request with 413, so a
|
||||
# client can't pin a worker with an unbounded page count. 0 disables the
|
||||
# cap. Default 2000.
|
||||
MAX_PAGES_PER_CHAPTER=2000
|
||||
|
||||
# ----- Crawler download safety -----
|
||||
# Hosts the crawler is allowed to fetch images/covers from, in addition
|
||||
@@ -117,6 +122,11 @@ CRAWLER_DOWNLOAD_ALLOWLIST=
|
||||
CRAWLER_ALLOW_ANY_HOST=false
|
||||
# Hard cap on a single image body. Default 32 MiB.
|
||||
CRAWLER_MAX_IMAGE_BYTES=33554432
|
||||
# Hard cap on the number of page images in one crawled chapter. The
|
||||
# per-image byte cap doesn't stop a hostile reader page listing thousands
|
||||
# of <img> tags; an over-cap chapter is acked failed instead of downloaded.
|
||||
# 0 disables the cap. Default 2000.
|
||||
CRAWLER_MAX_IMAGES_PER_CHAPTER=2000
|
||||
# Max manga detail fetches per metadata pass (both the in-process daemon
|
||||
# and the `bin/crawler` CLI). 0 means no cap — let the source walker run
|
||||
# to completion. Useful for capped test runs against a new source.
|
||||
|
||||
Reference in New Issue
Block a user