fix(files): make file writes transactional and close a quota bypass
Audit #6 and #8 for the last of the three stores, plus a bypass found on the way. **#6.** create/update/delete wrote the metadata row, then emitted best-effort, so an outbox failure left a committed file whose trigger never fired. `atomic_write::FilesWriter` commits the metadata row and the fan-out together. Files are the one store where the ordering is subtle, because the BYTES live on disk and cannot join a transaction: * create/update — blob first, then commit metadata + fan-out. A rollback unlinks the blob. (A crash at that exact point still orphans it; that hazard predates this change — the repo already wrote the blob and then inserted the row in a separate, failable statement — and the orphan is inert, referenced by nothing.) * delete — commit the metadata removal + fan-out FIRST, then unlink. The reverse order would destroy the bytes of a row that a rollback keeps, leaving a file that can never be read. **#8.** `GroupFilesService::create` read `total_bytes` on one connection and wrote on another, so concurrent uploads each saw the same pre-write total and together overshot the ceiling. This is the worst instance of the race in the codebase: the ceiling is DISK (10 GiB by default) and one file may be 100 MB, so a racing fleet overshoots by gigabytes. `PostgresGroupFilesWriter` takes the per-group advisory lock (on its own `files` key) across the check and the write. **The bypass.** `GroupFilesService::update` checked NO quota at all — so a 1-byte file could be updated to a 100 MB one without the ceiling ever being consulted, repeatedly, for unbounded disk. It now checks the projected total (the replaced file's bytes subtracted in SQL, so a same-size-or-smaller update near the cap still goes through). Also drive-by: `queue_e2e` asserted the ack the instant the marker appeared, but the marker is written DURING the handler and the ack happens after it returns — a zero-tolerance race. It polls now. (This does not fix the suite's flakiness, which reproduces on the pre-pass commit too.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -599,7 +599,7 @@ pub(crate) fn decode_cursor(cursor: &str) -> Result<Uuid, FilesRepoError> {
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct FileRow {
|
||||
pub(crate) struct FileRow {
|
||||
id: Uuid,
|
||||
collection: String,
|
||||
name: String,
|
||||
@@ -611,7 +611,7 @@ struct FileRow {
|
||||
}
|
||||
|
||||
impl FileRow {
|
||||
fn into_meta(self) -> FileMeta {
|
||||
pub(crate) fn into_meta(self) -> FileMeta {
|
||||
FileMeta {
|
||||
id: self.id,
|
||||
collection: self.collection,
|
||||
@@ -625,6 +625,166 @@ impl FileRow {
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Connection-scoped metadata operations
|
||||
//
|
||||
// The BYTES are written to disk outside any transaction (they are not
|
||||
// transactional and cannot be). What these give `crate::atomic_write` is a way
|
||||
// to commit the metadata row and the trigger fan-out it produces TOGETHER, so a
|
||||
// committed file row always has its event enqueued.
|
||||
//
|
||||
// A rollback after the bytes are on disk leaves an orphan blob. That hazard is
|
||||
// not new — the pre-existing repo already wrote the blob, then inserted the row
|
||||
// in a separate statement that could fail — and the writer now explicitly
|
||||
// unlinks the blob on the rollback path, so the window is strictly smaller than
|
||||
// it was.
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
/// The four metadata columns a blob write sets. Bundled so the `*_meta_on`
|
||||
/// helpers take a target (owner, collection, id) plus one payload, rather than a
|
||||
/// long positional tail it would be easy to transpose.
|
||||
#[derive(Clone, Copy)]
|
||||
pub(crate) struct MetaFields<'a> {
|
||||
pub name: &'a str,
|
||||
pub content_type: &'a str,
|
||||
pub size: i64,
|
||||
pub checksum: &'a str,
|
||||
}
|
||||
|
||||
/// The same, for an update: `name`/`content_type` are `None` to keep the stored
|
||||
/// value (`COALESCE`), while the bytes always change.
|
||||
#[derive(Clone, Copy)]
|
||||
pub(crate) struct MetaPatch<'a> {
|
||||
pub name: Option<&'a str>,
|
||||
pub content_type: Option<&'a str>,
|
||||
pub size: i64,
|
||||
pub checksum: &'a str,
|
||||
}
|
||||
|
||||
const FILE_COLS: &str = "id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at";
|
||||
|
||||
pub(crate) async fn head_on<'c, E>(
|
||||
exec: E,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError>
|
||||
where
|
||||
E: sqlx::PgExecutor<'c>,
|
||||
{
|
||||
let row: Option<FileRow> = sqlx::query_as(&format!(
|
||||
"SELECT {FILE_COLS} FROM files \
|
||||
WHERE app_id = $1 AND collection = $2 AND id = $3"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.fetch_optional(exec)
|
||||
.await?;
|
||||
Ok(row.map(FileRow::into_meta))
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_meta_on<'c, E>(
|
||||
exec: E,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
meta: MetaFields<'_>,
|
||||
) -> Result<FileMeta, FilesRepoError>
|
||||
where
|
||||
E: sqlx::PgExecutor<'c>,
|
||||
{
|
||||
let row: FileRow = sqlx::query_as(&format!(
|
||||
"INSERT INTO files \
|
||||
(app_id, collection, id, name, content_type, size_bytes, checksum_sha256) \
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7) \
|
||||
RETURNING {FILE_COLS}"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.bind(meta.name)
|
||||
.bind(meta.content_type)
|
||||
.bind(meta.size)
|
||||
.bind(meta.checksum)
|
||||
.fetch_one(exec)
|
||||
.await?;
|
||||
Ok(row.into_meta())
|
||||
}
|
||||
|
||||
/// `None` when the file row is gone (the caller maps that to `NotFound`).
|
||||
pub(crate) async fn update_meta_on<'c, E>(
|
||||
exec: E,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
meta: MetaPatch<'_>,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError>
|
||||
where
|
||||
E: sqlx::PgExecutor<'c>,
|
||||
{
|
||||
let row: Option<FileRow> = sqlx::query_as(&format!(
|
||||
"UPDATE files SET \
|
||||
name = COALESCE($4, name), \
|
||||
content_type = COALESCE($5, content_type), \
|
||||
size_bytes = $6, \
|
||||
checksum_sha256 = $7, \
|
||||
updated_at = NOW() \
|
||||
WHERE app_id = $1 AND collection = $2 AND id = $3 \
|
||||
RETURNING {FILE_COLS}"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.bind(meta.name)
|
||||
.bind(meta.content_type)
|
||||
.bind(meta.size)
|
||||
.bind(meta.checksum)
|
||||
.fetch_optional(exec)
|
||||
.await?;
|
||||
Ok(row.map(FileRow::into_meta))
|
||||
}
|
||||
|
||||
/// `DELETE ... RETURNING` — one statement, so no `SELECT ... FOR UPDATE` dance.
|
||||
/// The caller unlinks the bytes AFTER the transaction commits: unlinking first
|
||||
/// would destroy the blob of a row that a rollback then keeps.
|
||||
pub(crate) async fn delete_meta_on<'c, E>(
|
||||
exec: E,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError>
|
||||
where
|
||||
E: sqlx::PgExecutor<'c>,
|
||||
{
|
||||
let row: Option<FileRow> = sqlx::query_as(&format!(
|
||||
"DELETE FROM files \
|
||||
WHERE app_id = $1 AND collection = $2 AND id = $3 \
|
||||
RETURNING {FILE_COLS}"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.fetch_optional(exec)
|
||||
.await?;
|
||||
Ok(row.map(FileRow::into_meta))
|
||||
}
|
||||
|
||||
/// Best-effort unlink of a blob whose metadata write was rolled back (or whose
|
||||
/// row was just deleted). A failure leaves an orphan — logged, never fatal.
|
||||
pub(crate) fn unlink_blob(root: &Path, owner_rel: &Path, collection: &str, id: Uuid) {
|
||||
let path = final_path_at(root, owner_rel, collection, id);
|
||||
if let Err(e) = std::fs::remove_file(&path) {
|
||||
if e.kind() != std::io::ErrorKind::NotFound {
|
||||
tracing::warn!(
|
||||
path = %path.display(), error = %e,
|
||||
"files: unlink failed (orphan blob left on disk)"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
Reference in New Issue
Block a user