fix(upload): stream chapter pages to storage instead of buffering the whole chapter
The chapter upload handler read every `page` part fully into a Vec before
writing any, so peak memory was the whole chapter (bounded only by the
200 MiB body limit and amplified by concurrent uploads). It also accepted
an unbounded number of pages.
Stream each page part to a `staging/{upload_id}/…` key as it arrives — at
most one page's bytes are held at a time — then, once the chapter row (and
its id) exists, promote each staged blob to its final key via a new
`Storage::rename` (LocalStorage: fs rename; default impl: stream+delete for
future backends). Finalization is all-or-nothing: on any failure the DB
rolls back and both staged and already-finalized blobs are cleaned up.
Add MAX_PAGES_PER_CHAPTER (UploadConfig, default 2000, 0 = disabled),
rejecting an over-cap upload with 413 before any DB write. Also document
the crawler-side CRAWLER_MAX_IMAGES_PER_CHAPTER (added earlier) in
.env.example + docker-compose so the env-coverage test passes.
Tests: LocalStorage rename unit tests; a 413 over-cap upload test; existing
rollback + happy-path upload tests still green (the fault-injecting storage
counts put/put_stream, so mid-upload failure still rolls back).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6,7 +6,12 @@
|
||||
//! whitelist with 415. Filename and extension never reach the storage
|
||||
//! key — we derive both from the sniffed type.
|
||||
|
||||
use axum::extract::multipart::Field;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::api::mangas::map_multipart_error;
|
||||
use crate::error::{AppError, AppResult};
|
||||
use crate::storage::Storage;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct UploadedImage {
|
||||
@@ -15,6 +20,59 @@ pub struct UploadedImage {
|
||||
pub ext: &'static str,
|
||||
}
|
||||
|
||||
/// A page image written to a temporary staging key during a chapter
|
||||
/// upload, awaiting promotion to its final chapter-scoped key once the
|
||||
/// chapter row (and thus its id) exists. Carries only the small metadata
|
||||
/// the caller needs — never the image bytes.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct StagedImage {
|
||||
pub staging_key: String,
|
||||
pub mime: &'static str,
|
||||
pub ext: &'static str,
|
||||
pub size_bytes: i64,
|
||||
}
|
||||
|
||||
/// The staging-key prefix. Blobs left here by a failed or abandoned upload
|
||||
/// are orphans a future reaper can sweep; a successful upload renames every
|
||||
/// staged page out of this prefix.
|
||||
pub const STAGING_PREFIX: &str = "staging";
|
||||
|
||||
/// Read one multipart image part and write it straight to a staging key,
|
||||
/// returning only its metadata. The per-file byte cap is enforced as bytes
|
||||
/// arrive, so an oversized part is rejected (413) without being fully
|
||||
/// buffered, and at most one page's bytes are held in memory at a time —
|
||||
/// the whole chapter is never buffered, unlike the previous
|
||||
/// read-all-then-persist path.
|
||||
pub async fn stage_image_part(
|
||||
storage: &dyn Storage,
|
||||
mut field: Field<'_>,
|
||||
upload_id: Uuid,
|
||||
seq: usize,
|
||||
max_size: usize,
|
||||
field_name: &str,
|
||||
) -> AppResult<StagedImage> {
|
||||
let mut bytes: Vec<u8> = Vec::new();
|
||||
while let Some(chunk) = field.chunk().await.map_err(map_multipart_error)? {
|
||||
if bytes.len().saturating_add(chunk.len()) > max_size {
|
||||
return Err(AppError::PayloadTooLarge(format!(
|
||||
"{field_name} exceeds {max_size}-byte cap"
|
||||
)));
|
||||
}
|
||||
bytes.extend_from_slice(&chunk);
|
||||
}
|
||||
// Reuse the shared sniff + whitelist check; it re-verifies the size cap
|
||||
// (already enforced above) and derives mime/ext from magic bytes.
|
||||
let img = parse_image(bytes, max_size, field_name)?;
|
||||
let staging_key = format!("{STAGING_PREFIX}/{}/{:04}.{}", upload_id.simple(), seq, img.ext);
|
||||
storage.put(&staging_key, &img.bytes).await?;
|
||||
Ok(StagedImage {
|
||||
staging_key,
|
||||
mime: img.mime,
|
||||
ext: img.ext,
|
||||
size_bytes: img.bytes.len() as i64,
|
||||
})
|
||||
}
|
||||
|
||||
pub fn parse_image(bytes: Vec<u8>, max_size: usize, field_name: &str) -> AppResult<UploadedImage> {
|
||||
if bytes.len() > max_size {
|
||||
return Err(AppError::PayloadTooLarge(format!(
|
||||
|
||||
Reference in New Issue
Block a user