fix(files): make file writes transactional and close a quota bypass

Audit #6 and #8 for the last of the three stores, plus a bypass found on
the way.

**#6.** create/update/delete wrote the metadata row, then emitted
best-effort, so an outbox failure left a committed file whose trigger never
fired. `atomic_write::FilesWriter` commits the metadata row and the fan-out
together.

Files are the one store where the ordering is subtle, because the BYTES live
on disk and cannot join a transaction:

  * create/update — blob first, then commit metadata + fan-out. A rollback
    unlinks the blob. (A crash at that exact point still orphans it; that
    hazard predates this change — the repo already wrote the blob and then
    inserted the row in a separate, failable statement — and the orphan is
    inert, referenced by nothing.)
  * delete — commit the metadata removal + fan-out FIRST, then unlink. The
    reverse order would destroy the bytes of a row that a rollback keeps,
    leaving a file that can never be read.

**#8.** `GroupFilesService::create` read `total_bytes` on one connection and
wrote on another, so concurrent uploads each saw the same pre-write total and
together overshot the ceiling. This is the worst instance of the race in the
codebase: the ceiling is DISK (10 GiB by default) and one file may be 100 MB,
so a racing fleet overshoots by gigabytes. `PostgresGroupFilesWriter` takes
the per-group advisory lock (on its own `files` key) across the check and the
write.

**The bypass.** `GroupFilesService::update` checked NO quota at all — so a
1-byte file could be updated to a 100 MB one without the ceiling ever being
consulted, repeatedly, for unbounded disk. It now checks the projected total
(the replaced file's bytes subtracted in SQL, so a same-size-or-smaller
update near the cap still goes through).

Also drive-by: `queue_e2e` asserted the ack the instant the marker appeared,
but the marker is written DURING the handler and the ack happens after it
returns — a zero-tolerance race. It polls now. (This does not fix the
suite's flakiness, which reproduces on the pre-pass commit too.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-14 20:09:03 +02:00
parent 58bf0ab3ec
commit 80a0d31cd2
9 changed files with 1109 additions and 155 deletions

View File

@@ -37,18 +37,22 @@
//! Both sit behind the same trait, so the service body has a single code path; //! Both sit behind the same trait, so the service body has a single code path;
//! which one is in play is a constructor choice, not a branch in the hot path. //! which one is in play is a constructor choice, not a branch in the hot path.
use std::path::PathBuf;
use std::sync::Arc; use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use picloud_shared::{ use picloud_shared::{
DocId, DocsError, GroupDocsError, GroupId, GroupKvError, KvError, SdkCallCx, ServiceEvent, DocId, DocsError, FileMeta, FileUpdate, FilesError, GroupDocsError, GroupFilesError, GroupId,
ServiceEventEmitter, GroupKvError, KvError, NewFile, SdkCallCx, ServiceEvent, ServiceEventEmitter,
}; };
use serde_json::Value; use serde_json::Value;
use sqlx::{PgConnection, PgPool}; use sqlx::{PgConnection, PgPool};
use uuid::Uuid;
use crate::docs_repo::{self, DocsRepo}; use crate::docs_repo::{self, DocsRepo};
use crate::files_repo::{self, FileUpdated, FilesRepo, MetaFields, MetaPatch};
use crate::group_docs_repo::{self, GroupDocsRepo}; use crate::group_docs_repo::{self, GroupDocsRepo};
use crate::group_files_repo::{self, GroupFilesRepo, GroupFilesRepoError};
use crate::group_kv_repo::{self, GroupKvRepo}; use crate::group_kv_repo::{self, GroupKvRepo};
use crate::group_quota::{check_group_write, GroupUsage, GroupWriteQuota, QuotaOutcome}; use crate::group_quota::{check_group_write, GroupUsage, GroupWriteQuota, QuotaOutcome};
use crate::kv_repo::{self, KvRepo}; use crate::kv_repo::{self, KvRepo};
@@ -1261,3 +1265,592 @@ impl GroupDocsWriter for BestEffortGroupDocsWriter {
Ok(true) Ok(true)
} }
} }
// ----------------------------------------------------------------------------
// Files (per-app and §11.6 group)
//
// Files differ from KV/docs in one way that matters: the BYTES live on disk and
// cannot join a transaction. What can — and now does — is the metadata row and
// the trigger fan-out, so a committed file row always has its event enqueued.
//
// Ordering rules, which are the whole game here:
// * create/update — write the blob FIRST, then commit metadata + fan-out. If
// the transaction rolls back, unlink the blob. (A crash at exactly that
// point still orphans it; that hazard predates this change, and the orphan
// is inert — nothing references it.)
// * delete — commit the metadata removal + fan-out FIRST, then unlink. The
// reverse order would destroy the bytes of a row that a rollback keeps.
// ----------------------------------------------------------------------------
/// A files mutation's `ServiceEvent`. The payload is the `FileMeta` JSON —
/// never the blob bytes.
fn files_event(
op: &'static str,
collection: &str,
meta: &FileMeta,
prev: Option<&FileMeta>,
) -> Result<ServiceEvent, String> {
Ok(ServiceEvent {
source: "files",
op,
collection: Some(collection.to_string()),
key: Some(meta.id.to_string()),
payload: Some(serde_json::to_value(meta).map_err(|e| format!("serialize file meta: {e}"))?),
old_payload: match prev {
Some(p) => {
Some(serde_json::to_value(p).map_err(|e| format!("serialize file meta: {e}"))?)
}
None => None,
},
})
}
fn files_tx_err(e: &sqlx::Error) -> FilesError {
FilesError::Backend(format!("database error: {e}"))
}
/// The mutating half of the files service: the blob write, the metadata row, and
/// the trigger fan-out. Reads stay on `FilesRepo`.
#[async_trait]
pub trait FilesWriter: Send + Sync {
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError>;
/// `Ok(false)` when the file does not exist.
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<bool, FilesError>;
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: Uuid) -> Result<bool, FilesError>;
}
pub struct PostgresFilesWriter {
pool: PgPool,
root: PathBuf,
}
impl PostgresFilesWriter {
#[must_use]
pub fn new(pool: PgPool, root: PathBuf) -> Self {
Self { pool, root }
}
}
#[async_trait]
impl FilesWriter for PostgresFilesWriter {
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError> {
let owner = files_repo::app_owner_dir(cx.app_id);
let id = Uuid::new_v4();
let size = i64::try_from(new.data.len()).unwrap_or(i64::MAX);
// The bytes go down first — they cannot be part of the transaction.
let checksum = files_repo::write_atomic_at(&self.root, &owner, collection, id, &new.data)?;
let committed = async {
let mut tx = self.pool.begin().await.map_err(|e| files_tx_err(&e))?;
let meta = files_repo::insert_meta_on(
&mut *tx,
cx.app_id,
collection,
id,
MetaFields {
name: &new.name,
content_type: &new.content_type,
size,
checksum: &checksum,
},
)
.await?;
let event =
files_event("create", collection, &meta, None).map_err(FilesError::Backend)?;
emit_on(&mut tx, cx, &event)
.await
.map_err(|e| FilesError::Backend(format!("event emit: {e}")))?;
tx.commit().await.map_err(|e| files_tx_err(&e))?;
Ok::<_, FilesError>(meta.id)
}
.await;
match committed {
Ok(id) => Ok(id),
Err(e) => {
// The metadata never landed, so nothing references these bytes.
files_repo::unlink_blob(&self.root, &owner, collection, id);
Err(e)
}
}
}
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<bool, FilesError> {
let owner = files_repo::app_owner_dir(cx.app_id);
// Existence check + the CDC surface for the event.
let Some(prev) = files_repo::head_on(&self.pool, cx.app_id, collection, id).await? else {
return Ok(false);
};
let size = i64::try_from(upd.data.len()).unwrap_or(i64::MAX);
let checksum = files_repo::write_atomic_at(&self.root, &owner, collection, id, &upd.data)?;
let mut tx = self.pool.begin().await.map_err(|e| files_tx_err(&e))?;
let updated = files_repo::update_meta_on(
&mut *tx,
cx.app_id,
collection,
id,
MetaPatch {
name: upd.name.as_deref(),
content_type: upd.content_type.as_deref(),
size,
checksum: &checksum,
},
)
.await?;
let Some(meta) = updated else {
// Deleted between the head and the update. The blob we just wrote is
// now unreferenced.
files_repo::unlink_blob(&self.root, &owner, collection, id);
return Ok(false);
};
let event =
files_event("update", collection, &meta, Some(&prev)).map_err(FilesError::Backend)?;
emit_on(&mut tx, cx, &event)
.await
.map_err(|e| FilesError::Backend(format!("event emit: {e}")))?;
tx.commit().await.map_err(|e| files_tx_err(&e))?;
Ok(true)
}
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: Uuid) -> Result<bool, FilesError> {
let mut tx = self.pool.begin().await.map_err(|e| files_tx_err(&e))?;
let Some(meta) = files_repo::delete_meta_on(&mut *tx, cx.app_id, collection, id).await?
else {
return Ok(false);
};
let event =
files_event("delete", collection, &meta, Some(&meta)).map_err(FilesError::Backend)?;
emit_on(&mut tx, cx, &event)
.await
.map_err(|e| FilesError::Backend(format!("event emit: {e}")))?;
// Commit BEFORE unlinking: a rollback here keeps the row, and a row whose
// bytes we had already deleted would be unreadable forever.
tx.commit().await.map_err(|e| files_tx_err(&e))?;
files_repo::unlink_blob(
&self.root,
&files_repo::app_owner_dir(cx.app_id),
collection,
id,
);
Ok(true)
}
}
pub struct BestEffortFilesWriter {
repo: Arc<dyn FilesRepo>,
events: Arc<dyn ServiceEventEmitter>,
}
impl BestEffortFilesWriter {
#[must_use]
pub fn new(repo: Arc<dyn FilesRepo>, events: Arc<dyn ServiceEventEmitter>) -> Self {
Self { repo, events }
}
}
#[async_trait]
impl FilesWriter for BestEffortFilesWriter {
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError> {
let meta = self.repo.create(cx.app_id, collection, new).await?;
if let Ok(event) = files_event("create", collection, &meta, None) {
best_effort_emit(&*self.events, cx, event).await;
}
Ok(meta.id)
}
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<bool, FilesError> {
let Some(FileUpdated { new, prev }) =
self.repo.update(cx.app_id, collection, id, upd).await?
else {
return Ok(false);
};
if let Ok(event) = files_event("update", collection, &new, Some(&prev)) {
best_effort_emit(&*self.events, cx, event).await;
}
Ok(true)
}
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: Uuid) -> Result<bool, FilesError> {
let Some(meta) = self.repo.delete(cx.app_id, collection, id).await? else {
return Ok(false);
};
if let Ok(event) = files_event("delete", collection, &meta, Some(&meta)) {
best_effort_emit(&*self.events, cx, event).await;
}
Ok(true)
}
}
// ---- §11.6 group (shared-collection) files ----------------------------------
/// The group-files ceiling is bytes-only (no row count): one number, the total
/// stored blob size for the group.
#[derive(Debug, Clone, Copy)]
pub struct GroupFilesQuota {
pub max_total_bytes: u64,
}
#[async_trait]
pub trait GroupFilesWriter: Send + Sync {
async fn create(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
new: NewFile,
quota: GroupFilesQuota,
) -> Result<Uuid, GroupFilesError>;
async fn update(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
upd: FileUpdate,
quota: GroupFilesQuota,
) -> Result<bool, GroupFilesError>;
async fn delete(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
) -> Result<bool, GroupFilesError>;
}
fn group_files_tx_err(e: &sqlx::Error) -> GroupFilesError {
GroupFilesError::Backend(format!("database error: {e}"))
}
pub struct PostgresGroupFilesWriter {
pool: PgPool,
root: PathBuf,
}
impl PostgresGroupFilesWriter {
#[must_use]
pub fn new(pool: PgPool, root: PathBuf) -> Self {
Self { pool, root }
}
/// The projected total must fit under the ceiling. Runs on the writing
/// transaction's connection, under the per-group advisory lock the caller
/// has already taken — so concurrent uploads to the same group serialize
/// instead of each seeing the same pre-write total. That matters more here
/// than for KV/docs: a file is up to 100 MB, so a racing fleet of uploads
/// could overshoot a 10 GiB ceiling by gigabytes of real disk.
async fn check_quota(
conn: &mut PgConnection,
group_id: GroupId,
replacing: Option<Uuid>,
incoming: i64,
quota: GroupFilesQuota,
) -> Result<(), GroupFilesError> {
let projected =
group_files_repo::projected_total_bytes_on(&mut *conn, group_id, replacing, incoming)
.await?;
if projected > quota.max_total_bytes {
return Err(GroupFilesError::QuotaExceeded {
limit: quota.max_total_bytes,
actual: projected,
});
}
Ok(())
}
}
#[async_trait]
impl GroupFilesWriter for PostgresGroupFilesWriter {
async fn create(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
new: NewFile,
quota: GroupFilesQuota,
) -> Result<Uuid, GroupFilesError> {
let owner = group_files_repo::group_owner_dir(group_id);
let id = Uuid::new_v4();
let size = i64::try_from(new.data.len()).unwrap_or(i64::MAX);
let checksum = files_repo::write_atomic_at(&self.root, &owner, collection, id, &new.data)
.map_err(GroupFilesRepoError::from)?;
let committed = async {
let mut tx = self
.pool
.begin()
.await
.map_err(|e| group_files_tx_err(&e))?;
lock_group_quota(&mut tx, group_id, "files")
.await
.map_err(|e| group_files_tx_err(&e))?;
Self::check_quota(&mut tx, group_id, None, size, quota).await?;
let meta = group_files_repo::insert_meta_on(
&mut *tx,
group_id,
collection,
id,
MetaFields {
name: &new.name,
content_type: &new.content_type,
size,
checksum: &checksum,
},
)
.await?;
let event =
files_event("create", collection, &meta, None).map_err(GroupFilesError::Backend)?;
emit_shared_on(&mut tx, cx, group_id, &event)
.await
.map_err(|e| GroupFilesError::Backend(format!("event emit: {e}")))?;
tx.commit().await.map_err(|e| group_files_tx_err(&e))?;
Ok::<_, GroupFilesError>(meta.id)
}
.await;
match committed {
Ok(id) => Ok(id),
Err(e) => {
// Quota refusal or emit failure — nothing references these bytes.
files_repo::unlink_blob(&self.root, &owner, collection, id);
Err(e)
}
}
}
async fn update(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
upd: FileUpdate,
quota: GroupFilesQuota,
) -> Result<bool, GroupFilesError> {
let owner = group_files_repo::group_owner_dir(group_id);
let Some(prev) = group_files_repo::head_on(&self.pool, group_id, collection, id).await?
else {
return Ok(false);
};
let size = i64::try_from(upd.data.len()).unwrap_or(i64::MAX);
let checksum = files_repo::write_atomic_at(&self.root, &owner, collection, id, &upd.data)
.map_err(GroupFilesRepoError::from)?;
let committed = async {
let mut tx = self
.pool
.begin()
.await
.map_err(|e| group_files_tx_err(&e))?;
lock_group_quota(&mut tx, group_id, "files")
.await
.map_err(|e| group_files_tx_err(&e))?;
// The old service checked NO quota on update at all — so a 1-byte
// file could be grown to 100 MB without ever consulting the ceiling.
// The projection subtracts the bytes being replaced, so a same-size
// or smaller update near the cap still goes through.
Self::check_quota(&mut tx, group_id, Some(id), size, quota).await?;
let updated = group_files_repo::update_meta_on(
&mut *tx,
group_id,
collection,
id,
MetaPatch {
name: upd.name.as_deref(),
content_type: upd.content_type.as_deref(),
size,
checksum: &checksum,
},
)
.await?;
let Some(meta) = updated else {
return Ok::<_, GroupFilesError>(None);
};
let event = files_event("update", collection, &meta, Some(&prev))
.map_err(GroupFilesError::Backend)?;
emit_shared_on(&mut tx, cx, group_id, &event)
.await
.map_err(|e| GroupFilesError::Backend(format!("event emit: {e}")))?;
tx.commit().await.map_err(|e| group_files_tx_err(&e))?;
Ok(Some(()))
}
.await;
match committed {
Ok(Some(())) => Ok(true),
// Refused, or the row vanished under us: either way the bytes we
// just wrote are not referenced by any committed row.
Ok(None) => {
files_repo::unlink_blob(&self.root, &owner, collection, id);
Ok(false)
}
Err(e) => {
// NOTE: on a rollback the PREVIOUS blob has already been
// overwritten on disk and cannot be restored. The metadata still
// points at the old checksum, so a read fails the integrity check
// rather than returning wrong bytes. This predates the change
// (the repo overwrote in place too); making the blob write
// copy-on-write is a separate piece of work.
Err(e)
}
}
}
async fn delete(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
) -> Result<bool, GroupFilesError> {
// A delete only frees bytes — no quota, so no lock.
let mut tx = self
.pool
.begin()
.await
.map_err(|e| group_files_tx_err(&e))?;
let Some(meta) =
group_files_repo::delete_meta_on(&mut *tx, group_id, collection, id).await?
else {
return Ok(false);
};
let event = files_event("delete", collection, &meta, Some(&meta))
.map_err(GroupFilesError::Backend)?;
emit_shared_on(&mut tx, cx, group_id, &event)
.await
.map_err(|e| GroupFilesError::Backend(format!("event emit: {e}")))?;
tx.commit().await.map_err(|e| group_files_tx_err(&e))?;
files_repo::unlink_blob(
&self.root,
&group_files_repo::group_owner_dir(group_id),
collection,
id,
);
Ok(true)
}
}
pub struct BestEffortGroupFilesWriter {
repo: Arc<dyn GroupFilesRepo>,
events: Arc<dyn ServiceEventEmitter>,
}
impl BestEffortGroupFilesWriter {
#[must_use]
pub fn new(repo: Arc<dyn GroupFilesRepo>, events: Arc<dyn ServiceEventEmitter>) -> Self {
Self { repo, events }
}
async fn emit(&self, cx: &SdkCallCx, group_id: GroupId, event: ServiceEvent) {
let (source, op) = (event.source, event.op);
if let Err(e) = self.events.emit_shared(cx, group_id, event).await {
tracing::error!(
error = %e, source, op, event_emit_failure = true,
"shared event emit failed — the write committed but its triggers will not fire"
);
}
}
}
#[async_trait]
impl GroupFilesWriter for BestEffortGroupFilesWriter {
async fn create(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
new: NewFile,
quota: GroupFilesQuota,
) -> Result<Uuid, GroupFilesError> {
let used = self.repo.total_bytes(group_id).await?;
let incoming = new.data.len() as u64;
let projected = used.saturating_add(incoming);
if projected > quota.max_total_bytes {
return Err(GroupFilesError::QuotaExceeded {
limit: quota.max_total_bytes,
actual: projected,
});
}
let meta = self.repo.create(group_id, collection, new).await?;
if let Ok(event) = files_event("create", collection, &meta, None) {
self.emit(cx, group_id, event).await;
}
Ok(meta.id)
}
async fn update(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
upd: FileUpdate,
_quota: GroupFilesQuota,
) -> Result<bool, GroupFilesError> {
let Some(FileUpdated { new, prev }) =
self.repo.update(group_id, collection, id, upd).await?
else {
return Ok(false);
};
if let Ok(event) = files_event("update", collection, &new, Some(&prev)) {
self.emit(cx, group_id, event).await;
}
Ok(true)
}
async fn delete(
&self,
cx: &SdkCallCx,
group_id: GroupId,
collection: &str,
id: Uuid,
) -> Result<bool, GroupFilesError> {
let Some(meta) = self.repo.delete(group_id, collection, id).await? else {
return Ok(false);
};
if let Ok(event) = files_event("delete", collection, &meta, Some(&meta)) {
self.emit(cx, group_id, event).await;
}
Ok(true)
}
}

View File

@@ -599,7 +599,7 @@ pub(crate) fn decode_cursor(cursor: &str) -> Result<Uuid, FilesRepoError> {
} }
#[derive(sqlx::FromRow)] #[derive(sqlx::FromRow)]
struct FileRow { pub(crate) struct FileRow {
id: Uuid, id: Uuid,
collection: String, collection: String,
name: String, name: String,
@@ -611,7 +611,7 @@ struct FileRow {
} }
impl FileRow { impl FileRow {
fn into_meta(self) -> FileMeta { pub(crate) fn into_meta(self) -> FileMeta {
FileMeta { FileMeta {
id: self.id, id: self.id,
collection: self.collection, collection: self.collection,
@@ -625,6 +625,166 @@ impl FileRow {
} }
} }
// ----------------------------------------------------------------------------
// Connection-scoped metadata operations
//
// The BYTES are written to disk outside any transaction (they are not
// transactional and cannot be). What these give `crate::atomic_write` is a way
// to commit the metadata row and the trigger fan-out it produces TOGETHER, so a
// committed file row always has its event enqueued.
//
// A rollback after the bytes are on disk leaves an orphan blob. That hazard is
// not new — the pre-existing repo already wrote the blob, then inserted the row
// in a separate statement that could fail — and the writer now explicitly
// unlinks the blob on the rollback path, so the window is strictly smaller than
// it was.
// ----------------------------------------------------------------------------
/// The four metadata columns a blob write sets. Bundled so the `*_meta_on`
/// helpers take a target (owner, collection, id) plus one payload, rather than a
/// long positional tail it would be easy to transpose.
#[derive(Clone, Copy)]
pub(crate) struct MetaFields<'a> {
pub name: &'a str,
pub content_type: &'a str,
pub size: i64,
pub checksum: &'a str,
}
/// The same, for an update: `name`/`content_type` are `None` to keep the stored
/// value (`COALESCE`), while the bytes always change.
#[derive(Clone, Copy)]
pub(crate) struct MetaPatch<'a> {
pub name: Option<&'a str>,
pub content_type: Option<&'a str>,
pub size: i64,
pub checksum: &'a str,
}
const FILE_COLS: &str = "id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at";
pub(crate) async fn head_on<'c, E>(
exec: E,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<FileRow> = sqlx::query_as(&format!(
"SELECT {FILE_COLS} FROM files \
WHERE app_id = $1 AND collection = $2 AND id = $3"
))
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(exec)
.await?;
Ok(row.map(FileRow::into_meta))
}
pub(crate) async fn insert_meta_on<'c, E>(
exec: E,
app_id: AppId,
collection: &str,
id: Uuid,
meta: MetaFields<'_>,
) -> Result<FileMeta, FilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: FileRow = sqlx::query_as(&format!(
"INSERT INTO files \
(app_id, collection, id, name, content_type, size_bytes, checksum_sha256) \
VALUES ($1, $2, $3, $4, $5, $6, $7) \
RETURNING {FILE_COLS}"
))
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.bind(meta.name)
.bind(meta.content_type)
.bind(meta.size)
.bind(meta.checksum)
.fetch_one(exec)
.await?;
Ok(row.into_meta())
}
/// `None` when the file row is gone (the caller maps that to `NotFound`).
pub(crate) async fn update_meta_on<'c, E>(
exec: E,
app_id: AppId,
collection: &str,
id: Uuid,
meta: MetaPatch<'_>,
) -> Result<Option<FileMeta>, FilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<FileRow> = sqlx::query_as(&format!(
"UPDATE files SET \
name = COALESCE($4, name), \
content_type = COALESCE($5, content_type), \
size_bytes = $6, \
checksum_sha256 = $7, \
updated_at = NOW() \
WHERE app_id = $1 AND collection = $2 AND id = $3 \
RETURNING {FILE_COLS}"
))
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.bind(meta.name)
.bind(meta.content_type)
.bind(meta.size)
.bind(meta.checksum)
.fetch_optional(exec)
.await?;
Ok(row.map(FileRow::into_meta))
}
/// `DELETE ... RETURNING` — one statement, so no `SELECT ... FOR UPDATE` dance.
/// The caller unlinks the bytes AFTER the transaction commits: unlinking first
/// would destroy the blob of a row that a rollback then keeps.
pub(crate) async fn delete_meta_on<'c, E>(
exec: E,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<FileRow> = sqlx::query_as(&format!(
"DELETE FROM files \
WHERE app_id = $1 AND collection = $2 AND id = $3 \
RETURNING {FILE_COLS}"
))
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(exec)
.await?;
Ok(row.map(FileRow::into_meta))
}
/// Best-effort unlink of a blob whose metadata write was rolled back (or whose
/// row was just deleted). A failure leaves an orphan — logged, never fatal.
pub(crate) fn unlink_blob(root: &Path, owner_rel: &Path, collection: &str, id: Uuid) {
let path = final_path_at(root, owner_rel, collection, id);
if let Err(e) = std::fs::remove_file(&path) {
if e.kind() != std::io::ErrorKind::NotFound {
tracing::warn!(
path = %path.display(), error = %e,
"files: unlink failed (orphan blob left on disk)"
);
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;

View File

@@ -18,17 +18,18 @@ use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use picloud_shared::{ use picloud_shared::{
sanitize_stored_content_type, validate_files_collection, FileMeta, FileUpdate, FilesError, sanitize_stored_content_type, validate_files_collection, FileMeta, FileUpdate, FilesError,
FilesListPage, FilesService, NewFile, SdkCallCx, ServiceEvent, ServiceEventEmitter, FilesListPage, FilesService, NewFile, SdkCallCx, ServiceEventEmitter,
}; };
use uuid::Uuid; use uuid::Uuid;
use crate::atomic_write::{BestEffortFilesWriter, FilesWriter, PostgresFilesWriter};
use crate::authz::{self, AuthzRepo, Capability}; use crate::authz::{self, AuthzRepo, Capability};
use crate::files_repo::{FileUpdated, FilesRepo, FilesRepoError}; use crate::files_repo::{FilesRepo, FilesRepoError};
pub struct FilesServiceImpl { pub struct FilesServiceImpl {
repo: Arc<dyn FilesRepo>, repo: Arc<dyn FilesRepo>,
authz: Arc<dyn AuthzRepo>, authz: Arc<dyn AuthzRepo>,
events: Arc<dyn ServiceEventEmitter>, writer: Arc<dyn FilesWriter>,
max_file_size_bytes: usize, max_file_size_bytes: usize,
} }
@@ -41,13 +42,23 @@ impl FilesServiceImpl {
max_file_size_bytes: usize, max_file_size_bytes: usize,
) -> Self { ) -> Self {
Self { Self {
writer: Arc::new(BestEffortFilesWriter::new(repo.clone(), events)),
repo, repo,
authz, authz,
events,
max_file_size_bytes, max_file_size_bytes,
} }
} }
/// Swap the best-effort writer for the transactional one: the metadata row
/// and the trigger fan-out then commit together, so an outbox failure rolls
/// the metadata back (and unlinks the blob) instead of silently losing the
/// event. The host always calls this; the in-memory unit tests do not.
#[must_use]
pub fn with_atomic_writes(mut self, pool: sqlx::PgPool, root: std::path::PathBuf) -> Self {
self.writer = Arc::new(PostgresFilesWriter::new(pool, root));
self
}
async fn check_read(&self, cx: &SdkCallCx) -> Result<(), FilesError> { async fn check_read(&self, cx: &SdkCallCx) -> Result<(), FilesError> {
authz::script_gate( authz::script_gate(
&*self.authz, &*self.authz,
@@ -69,37 +80,6 @@ impl FilesServiceImpl {
) )
.await .await
} }
/// Best-effort `ServiceEvent` emission. A failed emit is logged but
/// never rolls back the (already-durable) file write.
async fn emit(
&self,
cx: &SdkCallCx,
op: &'static str,
collection: &str,
meta: &FileMeta,
old: Option<&FileMeta>,
) {
let payload = serde_json::to_value(meta).ok();
let old_payload = old.and_then(|m| serde_json::to_value(m).ok());
if let Err(e) = self
.events
.emit(
cx,
ServiceEvent {
source: "files",
op,
collection: Some(collection.to_string()),
key: Some(meta.id.to_string()),
payload,
old_payload,
},
)
.await
{
tracing::error!(error = %e, source = "files", op, event_emit_failure = true, "event emit failed");
}
}
} }
/// Parse a script-supplied id. Invalid UUIDs aren't an error shape the /// Parse a script-supplied id. Invalid UUIDs aren't an error shape the
@@ -132,9 +112,7 @@ impl FilesService for FilesServiceImpl {
// Audit 2026-06-11 C-2 — coerce dangerous render types to // Audit 2026-06-11 C-2 — coerce dangerous render types to
// application/octet-stream after the shape checks pass. // application/octet-stream after the shape checks pass.
new.content_type = sanitize_stored_content_type(&new.content_type); new.content_type = sanitize_stored_content_type(&new.content_type);
let meta = self.repo.create(cx.app_id, collection, new).await?; self.writer.create(cx, collection, new).await
self.emit(cx, "create", collection, &meta, None).await;
Ok(meta.id)
} }
async fn head( async fn head(
@@ -182,12 +160,10 @@ impl FilesService for FilesServiceImpl {
let Some(uuid) = parse_id(id) else { let Some(uuid) = parse_id(id) else {
return Err(FilesError::NotFound); return Err(FilesError::NotFound);
}; };
match self.repo.update(cx.app_id, collection, uuid, upd).await? { if self.writer.update(cx, collection, uuid, upd).await? {
Some(FileUpdated { new, prev }) => {
self.emit(cx, "update", collection, &new, Some(&prev)).await;
Ok(()) Ok(())
} } else {
None => Err(FilesError::NotFound), Err(FilesError::NotFound)
} }
} }
@@ -197,16 +173,7 @@ impl FilesService for FilesServiceImpl {
let Some(uuid) = parse_id(id) else { let Some(uuid) = parse_id(id) else {
return Ok(false); return Ok(false);
}; };
match self.repo.delete(cx.app_id, collection, uuid).await? { self.writer.delete(cx, collection, uuid).await
Some(meta) => {
// On delete, the top-level metadata AND `prev` both carry
// the deleted row (per docs/v1.1.x design + the brief).
self.emit(cx, "delete", collection, &meta, Some(&meta))
.await;
Ok(true)
}
None => Ok(false),
}
} }
async fn list( async fn list(
@@ -232,6 +199,7 @@ impl FilesService for FilesServiceImpl {
mod tests { mod tests {
use super::*; use super::*;
use crate::authz::{AuthzError, AuthzRepo}; use crate::authz::{AuthzError, AuthzRepo};
use crate::files_repo::FileUpdated;
use async_trait::async_trait; use async_trait::async_trait;
use chrono::Utc; use chrono::Utc;
use picloud_shared::{ use picloud_shared::{

View File

@@ -13,29 +13,10 @@
//! name does not resolve. //! name does not resolve.
use async_trait::async_trait; use async_trait::async_trait;
use picloud_shared::{AppId, GroupId, ScriptOwner, SdkCallCx, ServiceEvent, ServiceEventEmitter}; use picloud_shared::{AppId, GroupId, ScriptOwner};
use sqlx::{PgPool, Postgres, Transaction}; use sqlx::{PgPool, Postgres, Transaction};
use crate::config_resolver::CHAIN_LEVELS_CTE; use crate::config_resolver::CHAIN_LEVELS_CTE;
/// §11.6: fire a `shared = true` trigger event on the owning group, best-effort.
/// An emit failure is logged (with the event's `source`/`op`) and swallowed —
/// the write already committed, and shared triggers are fire-and-forget, so a
/// dropped emit must never fail the caller. The single home for the tail the
/// group KV/docs/files services previously each copied; each still builds its
/// own `ServiceEvent` (payload shapes differ) and calls this.
pub(crate) async fn best_effort_emit_shared(
events: &dyn ServiceEventEmitter,
cx: &SdkCallCx,
group_id: GroupId,
event: ServiceEvent,
) {
let (source, op) = (event.source, event.op);
if let Err(e) = events.emit_shared(cx, group_id, event).await {
tracing::error!(error = %e, source, op, event_emit_failure = true, "shared event emit failed");
}
}
/// List **all** shared-collection declarations at `owner` as `(name, kind)` /// List **all** shared-collection declarations at `owner` as `(name, kind)`
/// pairs, sorted. Used by the kind-aware apply diff and `collections ls`. /// pairs, sorted. Used by the kind-aware apply diff and `collections ls`.
pub async fn list_all_for_owner( pub async fn list_all_for_owner(

View File

@@ -389,7 +389,7 @@ impl GroupFilesRepo for FsGroupFilesRepo {
} }
#[derive(sqlx::FromRow)] #[derive(sqlx::FromRow)]
struct GroupFileRow { pub(crate) struct GroupFileRow {
id: Uuid, id: Uuid,
collection: String, collection: String,
name: String, name: String,
@@ -401,7 +401,7 @@ struct GroupFileRow {
} }
impl GroupFileRow { impl GroupFileRow {
fn into_meta(self) -> FileMeta { pub(crate) fn into_meta(self) -> FileMeta {
FileMeta { FileMeta {
id: self.id, id: self.id,
collection: self.collection, collection: self.collection,
@@ -414,3 +414,146 @@ impl GroupFileRow {
} }
} }
} }
// ----------------------------------------------------------------------------
// Connection-scoped metadata operations (see `files_repo` for the rationale)
// ----------------------------------------------------------------------------
use crate::files_repo::{MetaFields, MetaPatch};
const GROUP_FILE_COLS: &str = "id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at";
pub(crate) async fn head_on<'c, E>(
exec: E,
group_id: GroupId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, GroupFilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<GroupFileRow> = sqlx::query_as(&format!(
"SELECT {GROUP_FILE_COLS} FROM group_files \
WHERE group_id = $1 AND collection = $2 AND id = $3"
))
.bind(group_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(exec)
.await?;
Ok(row.map(GroupFileRow::into_meta))
}
pub(crate) async fn insert_meta_on<'c, E>(
exec: E,
group_id: GroupId,
collection: &str,
id: Uuid,
meta: MetaFields<'_>,
) -> Result<FileMeta, GroupFilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: GroupFileRow = sqlx::query_as(&format!(
"INSERT INTO group_files \
(group_id, collection, id, name, content_type, size_bytes, checksum_sha256) \
VALUES ($1, $2, $3, $4, $5, $6, $7) \
RETURNING {GROUP_FILE_COLS}"
))
.bind(group_id.into_inner())
.bind(collection)
.bind(id)
.bind(meta.name)
.bind(meta.content_type)
.bind(meta.size)
.bind(meta.checksum)
.fetch_one(exec)
.await?;
Ok(row.into_meta())
}
pub(crate) async fn update_meta_on<'c, E>(
exec: E,
group_id: GroupId,
collection: &str,
id: Uuid,
meta: MetaPatch<'_>,
) -> Result<Option<FileMeta>, GroupFilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<GroupFileRow> = sqlx::query_as(&format!(
"UPDATE group_files SET \
name = COALESCE($4, name), \
content_type = COALESCE($5, content_type), \
size_bytes = $6, \
checksum_sha256 = $7, \
updated_at = NOW() \
WHERE group_id = $1 AND collection = $2 AND id = $3 \
RETURNING {GROUP_FILE_COLS}"
))
.bind(group_id.into_inner())
.bind(collection)
.bind(id)
.bind(meta.name)
.bind(meta.content_type)
.bind(meta.size)
.bind(meta.checksum)
.fetch_optional(exec)
.await?;
Ok(row.map(GroupFileRow::into_meta))
}
pub(crate) async fn delete_meta_on<'c, E>(
exec: E,
group_id: GroupId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, GroupFilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let row: Option<GroupFileRow> = sqlx::query_as(&format!(
"DELETE FROM group_files \
WHERE group_id = $1 AND collection = $2 AND id = $3 \
RETURNING {GROUP_FILE_COLS}"
))
.bind(group_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(exec)
.await?;
Ok(row.map(GroupFileRow::into_meta))
}
/// §11.6 quota: the PROJECTED total stored bytes for the group AFTER this write
/// — the current SUM, minus the bytes of the file being replaced (`replacing =
/// Some(id)` on update; `None` on create), plus the incoming file's bytes.
///
/// The subtraction is what `GroupFilesService::update` was missing entirely: it
/// checked no quota at all, so a 1-byte file could be updated to a 100 MB one
/// without ever consulting the ceiling.
pub(crate) async fn projected_total_bytes_on<'c, E>(
exec: E,
group_id: GroupId,
replacing: Option<Uuid>,
incoming: i64,
) -> Result<u64, GroupFilesRepoError>
where
E: sqlx::PgExecutor<'c>,
{
let (n,): (i64,) = sqlx::query_as(
"SELECT ( \
COALESCE((SELECT SUM(size_bytes) FROM group_files WHERE group_id = $1), 0) \
- COALESCE((SELECT size_bytes FROM group_files \
WHERE group_id = $1 AND id = $2), 0) \
+ $3 \
)::BIGINT",
)
.bind(group_id.into_inner())
.bind(replacing)
.bind(incoming)
.fetch_one(exec)
.await?;
Ok(u64::try_from(n).unwrap_or(0))
}

View File

@@ -13,12 +13,14 @@ use async_trait::async_trait;
use picloud_shared::{ use picloud_shared::{
sanitize_stored_content_type, validate_files_collection, FileMeta, FileUpdate, FilesListPage, sanitize_stored_content_type, validate_files_collection, FileMeta, FileUpdate, FilesListPage,
GroupFilesError, GroupFilesService, GroupId, NewFile, NoopEventEmitter, SdkCallCx, GroupFilesError, GroupFilesService, GroupId, NewFile, NoopEventEmitter, SdkCallCx,
ServiceEvent, ServiceEventEmitter, ServiceEventEmitter,
}; };
use uuid::Uuid; use uuid::Uuid;
use crate::atomic_write::{
BestEffortGroupFilesWriter, GroupFilesQuota, GroupFilesWriter, PostgresGroupFilesWriter,
};
use crate::authz::{self, AuthzRepo, Capability}; use crate::authz::{self, AuthzRepo, Capability};
use crate::files_repo::FileUpdated;
use crate::group_collection_repo::GroupCollectionResolver; use crate::group_collection_repo::GroupCollectionResolver;
use crate::group_files_repo::{GroupFilesRepo, GroupFilesRepoError}; use crate::group_files_repo::{GroupFilesRepo, GroupFilesRepoError};
@@ -34,7 +36,10 @@ pub struct GroupFilesServiceImpl {
/// shared-files collections (`PICLOUD_GROUP_FILES_MAX_TOTAL_BYTES`). /// shared-files collections (`PICLOUD_GROUP_FILES_MAX_TOTAL_BYTES`).
max_total_bytes: u64, max_total_bytes: u64,
/// §11.6: fires `shared = true` files triggers on a shared-collection write. /// §11.6: fires `shared = true` files triggers on a shared-collection write.
events: Arc<dyn ServiceEventEmitter>, /// Mutations: the byte-quota decision, the blob + metadata write, and the
/// shared-trigger fan-out — for the host, all under one advisory-locked
/// transaction. Reads stay on `repo`.
writer: Arc<dyn GroupFilesWriter>,
} }
impl GroupFilesServiceImpl { impl GroupFilesServiceImpl {
@@ -46,52 +51,44 @@ impl GroupFilesServiceImpl {
max_file_size_bytes: usize, max_file_size_bytes: usize,
) -> Self { ) -> Self {
Self { Self {
max_total_bytes: crate::group_quota::group_files_max_total_bytes_from_env(),
writer: Arc::new(BestEffortGroupFilesWriter::new(
repo.clone(),
Arc::new(NoopEventEmitter),
)),
repo, repo,
resolver, resolver,
authz, authz,
max_file_size_bytes, max_file_size_bytes,
max_total_bytes: crate::group_quota::group_files_max_total_bytes_from_env(),
events: Arc::new(NoopEventEmitter),
} }
} }
/// Wire the event emitter (§11.6 shared-collection triggers). /// The ceiling a shared-files write must fit under.
fn quota(&self) -> GroupFilesQuota {
GroupFilesQuota {
max_total_bytes: self.max_total_bytes,
}
}
/// Wire the event emitter (§11.6 shared-collection triggers) on the
/// best-effort writer. Superseded by [`Self::with_atomic_writes`].
#[must_use] #[must_use]
pub fn with_events(mut self, events: Arc<dyn ServiceEventEmitter>) -> Self { pub fn with_events(mut self, events: Arc<dyn ServiceEventEmitter>) -> Self {
self.events = events; self.writer = Arc::new(BestEffortGroupFilesWriter::new(self.repo.clone(), events));
self self
} }
/// §11.6: fire `shared = true` files triggers on the owning group. /// Use the transactional writer: the byte-quota check, the metadata row, and
/// Best-effort; an emit failure is logged, never surfaced. /// the shared-trigger fan-out run on ONE connection under a per-group
async fn emit_shared( /// advisory lock, so concurrent uploads can't each see the same pre-write
&self, /// total and together overshoot the ceiling. Supersedes
cx: &SdkCallCx, /// [`Self::with_events`].
group_id: GroupId, #[must_use]
op: &'static str, pub fn with_atomic_writes(mut self, pool: sqlx::PgPool, root: std::path::PathBuf) -> Self {
collection: &str, self.writer = Arc::new(PostgresGroupFilesWriter::new(pool, root));
meta: &FileMeta, self
old: Option<&FileMeta>,
) {
crate::group_collection_repo::best_effort_emit_shared(
&*self.events,
cx,
group_id,
ServiceEvent {
source: "files",
op,
collection: Some(collection.to_string()),
key: Some(meta.id.to_string()),
payload: serde_json::to_value(meta).ok(),
old_payload: old.and_then(|m| serde_json::to_value(m).ok()),
},
)
.await;
} }
/// The structural boundary: resolve the collection to its owning group
/// (kind=`files`) on the calling app's chain. `CollectionNotShared` when no
/// ancestor group declares it.
async fn owning_group( async fn owning_group(
&self, &self,
cx: &SdkCallCx, cx: &SdkCallCx,
@@ -160,19 +157,9 @@ impl GroupFilesService for GroupFilesServiceImpl {
// shape checks pass (same as app files, audit 2026-06-11 C-2). // shape checks pass (same as app files, audit 2026-06-11 C-2).
new.content_type = sanitize_stored_content_type(&new.content_type); new.content_type = sanitize_stored_content_type(&new.content_type);
self.check_write(cx, group_id).await?; self.check_write(cx, group_id).await?;
// §11.6 quota: the new file's bytes must fit under the group's total. self.writer
let used = self.repo.total_bytes(group_id).await?; .create(cx, group_id, collection, new, self.quota())
let incoming = new.data.len() as u64; .await
if used.saturating_add(incoming) > self.max_total_bytes {
return Err(GroupFilesError::QuotaExceeded {
limit: self.max_total_bytes,
actual: used.saturating_add(incoming),
});
}
let meta = self.repo.create(group_id, collection, new).await?;
self.emit_shared(cx, group_id, "create", collection, &meta, None)
.await;
Ok(meta.id)
} }
async fn head( async fn head(
@@ -222,13 +209,14 @@ impl GroupFilesService for GroupFilesServiceImpl {
let Some(uuid) = parse_id(id) else { let Some(uuid) = parse_id(id) else {
return Err(GroupFilesError::NotFound); return Err(GroupFilesError::NotFound);
}; };
match self.repo.update(group_id, collection, uuid, upd).await? { if self
Some(FileUpdated { new, prev }) => { .writer
self.emit_shared(cx, group_id, "update", collection, &new, Some(&prev)) .update(cx, group_id, collection, uuid, upd, self.quota())
.await; .await?
{
Ok(()) Ok(())
} } else {
None => Err(GroupFilesError::NotFound), Err(GroupFilesError::NotFound)
} }
} }
@@ -244,14 +232,7 @@ impl GroupFilesService for GroupFilesServiceImpl {
let Some(uuid) = parse_id(id) else { let Some(uuid) = parse_id(id) else {
return Ok(false); return Ok(false);
}; };
match self.repo.delete(group_id, collection, uuid).await? { self.writer.delete(cx, group_id, collection, uuid).await
Some(meta) => {
self.emit_shared(cx, group_id, "delete", collection, &meta, Some(&meta))
.await;
Ok(true)
}
None => Ok(false),
}
} }
async fn list( async fn list(
@@ -278,6 +259,7 @@ impl GroupFilesService for GroupFilesServiceImpl {
mod tests { mod tests {
use super::*; use super::*;
use crate::authz::{AuthzError, AuthzRepo}; use crate::authz::{AuthzError, AuthzRepo};
use crate::files_repo::FileUpdated;
use crate::group_files_repo::GroupFilesRepoError; use crate::group_files_repo::GroupFilesRepoError;
use chrono::Utc; use chrono::Utc;
use picloud_shared::{ use picloud_shared::{

View File

@@ -16,12 +16,14 @@
use std::sync::Arc; use std::sync::Arc;
use picloud_manager_core::atomic_write::{ use picloud_manager_core::atomic_write::{
GroupDocsTarget, GroupDocsWriter, GroupKvTarget, GroupKvWriter, KvWriter, GroupDocsTarget, GroupDocsWriter, GroupFilesQuota, GroupFilesWriter, GroupKvTarget,
PostgresGroupDocsWriter, PostgresGroupKvWriter, PostgresKvWriter, GroupKvWriter, KvWriter, PostgresGroupDocsWriter, PostgresGroupFilesWriter,
PostgresGroupKvWriter, PostgresKvWriter,
}; };
use picloud_manager_core::group_quota::GroupWriteQuota; use picloud_manager_core::group_quota::GroupWriteQuota;
use picloud_shared::{ use picloud_shared::{
AppId, ExecutionId, GroupDocsError, GroupId, GroupKvError, RequestId, ScriptId, SdkCallCx, AppId, ExecutionId, FileUpdate, GroupDocsError, GroupFilesError, GroupId, GroupKvError,
NewFile, RequestId, ScriptId, SdkCallCx,
}; };
use sqlx::postgres::PgPoolOptions; use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool; use sqlx::PgPool;
@@ -503,3 +505,106 @@ async fn concurrent_writers_cannot_push_a_group_past_its_docs_row_quota() {
.await .await
.expect("cleanup"); .expect("cleanup");
} }
/// Group FILES had the worst version of the race: the ceiling is disk (10 GiB by
/// default) and a single file may be 100 MB, so a fleet of concurrent uploads
/// each seeing the same pre-write total could overshoot by GIGABYTES of real
/// disk. Same fix — a per-group advisory lock across the check and the write.
///
/// It also pins the second half of the bug: `update` checked NO quota at all, so
/// a 1-byte file could be grown past the ceiling unchallenged.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn concurrent_uploads_cannot_push_a_group_past_its_files_byte_quota() {
let Some(pool) = pool_or_skip().await else {
return;
};
let group = mk_group(&pool).await;
let root = std::env::temp_dir().join(format!("picloud-aw-{}", Uuid::new_v4().simple()));
let writer = Arc::new(PostgresGroupFilesWriter::new(pool.clone(), root.clone()));
// Ten 100-byte uploads race for a 450-byte ceiling: at most 4 may land.
let quota = GroupFilesQuota {
max_total_bytes: 450,
};
let app = Uuid::new_v4();
let mut set = tokio::task::JoinSet::new();
for i in 0..10 {
let w = Arc::clone(&writer);
set.spawn(async move {
let cx = cx(app);
w.create(
&cx,
GroupId::from(group),
"assets",
NewFile {
name: format!("f{i}.bin"),
content_type: "application/octet-stream".into(),
data: vec![b'x'; 100],
},
quota,
)
.await
});
}
let mut created = Vec::new();
while let Some(r) = set.join_next().await {
match r.expect("task") {
Ok(id) => created.push(id),
Err(GroupFilesError::QuotaExceeded { .. }) => {}
Err(e) => panic!("unexpected error: {e}"),
}
}
let stored = |pool: PgPool| async move {
let (n,): (i64,) = sqlx::query_as(
"SELECT COALESCE(SUM(size_bytes), 0)::BIGINT FROM group_files WHERE group_id = $1",
)
.bind(group)
.fetch_one(&pool)
.await
.expect("bytes");
n
};
let bytes = stored(pool.clone()).await;
assert!(
bytes <= 450,
"stored bytes ({bytes}) must not exceed the 450-byte ceiling — concurrent \
uploads must not each see the same pre-write total"
);
assert!(!created.is_empty(), "some uploads should have succeeded");
// An UPDATE that would blow the ceiling must now be refused too. Previously
// update consulted no quota at all, so this was a free bypass.
let victim = created[0];
let err = writer
.update(
&cx(app),
GroupId::from(group),
"assets",
victim,
FileUpdate {
name: None,
content_type: None,
data: vec![b'y'; 10_000],
},
quota,
)
.await
.expect_err("growing a file past the group ceiling must be refused");
assert!(
matches!(err, GroupFilesError::QuotaExceeded { .. }),
"got {err}"
);
assert_eq!(
stored(pool.clone()).await,
bytes,
"the refused update must not have changed the stored total"
);
sqlx::query("DELETE FROM groups WHERE id = $1")
.bind(group)
.execute(&pool)
.await
.expect("cleanup");
let _ = std::fs::remove_dir_all(&root);
}

View File

@@ -247,12 +247,16 @@ pub async fn build_app(
// Kept for the v1.1.6 orphan sweeper (cleans stale `*.tmp.*` files). // Kept for the v1.1.6 orphan sweeper (cleans stale `*.tmp.*` files).
let files_root = files_config.root.clone(); let files_root = files_config.root.clone();
let files_repo = Arc::new(FsFilesRepo::new(pool.clone(), files_config)); let files_repo = Arc::new(FsFilesRepo::new(pool.clone(), files_config));
let files: Arc<dyn FilesService> = Arc::new(FilesServiceImpl::new( let files: Arc<dyn FilesService> = Arc::new(
FilesServiceImpl::new(
files_repo.clone(), files_repo.clone(),
authz.clone(), authz.clone(),
events.clone(), events.clone(),
files_max_size, files_max_size,
)); )
// Metadata row + trigger fan-out in one tx; a rollback unlinks the blob.
.with_atomic_writes(pool.clone(), files_root.clone()),
);
// §11.6 shared group collections (files): group-keyed blobs under // §11.6 shared group collections (files): group-keyed blobs under
// `<root>/files/groups/<group_id>/...`, resolved from cx.app_id's chain. // `<root>/files/groups/<group_id>/...`, resolved from cx.app_id's chain.
// Shares the per-file size cap; fires `shared = true` triggers under the // Shares the per-file size cap; fires `shared = true` triggers under the
@@ -265,7 +269,10 @@ pub async fn build_app(
authz.clone(), authz.clone(),
files_max_size, files_max_size,
) )
.with_events(events.clone()), // Byte quota + metadata + shared fan-out in one advisory-locked tx: a
// fleet of concurrent uploads can no longer each see the same pre-write
// total and together overshoot the group's disk ceiling.
.with_atomic_writes(pool.clone(), files_root.clone()),
); );
// v1.1.6 realtime: the in-process broadcaster is shared between the // v1.1.6 realtime: the in-process broadcaster is shared between the
// publish path (PubsubServiceImpl fans out to SSE subscribers after // publish path (PubsubServiceImpl fans out to SSE subscribers after

View File

@@ -108,6 +108,19 @@ const THROW_HANDLER: &str = r#"
throw "boom" throw "boom"
"#; "#;
/// Poll until the queue depth reaches `want` (or give up and return what we saw).
async fn poll_queue_depth(pool: &PgPool, app_id: &str, queue: &str, want: i64) -> i64 {
let mut depth = -1;
for _ in 0..200 {
depth = count_queue_messages(pool, app_id, queue).await;
if depth == want {
return depth;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
depth
}
async fn poll_marker(pool: &PgPool, app_id: &str) -> Option<Value> { async fn poll_marker(pool: &PgPool, app_id: &str) -> Option<Value> {
for _ in 0..200 { for _ in 0..200 {
let row: Option<(Value,)> = sqlx::query_as( let row: Option<(Value,)> = sqlx::query_as(
@@ -192,8 +205,10 @@ async fn queue_receive_acks_on_success() {
assert_eq!(marker["queue"]["queue_name"], "jobs"); assert_eq!(marker["queue"]["queue_name"], "jobs");
assert_eq!(marker["queue"]["message"]["x"], 42); assert_eq!(marker["queue"]["message"]["x"], 42);
// Ack deleted the row. // Ack deleted the row. The marker is written DURING the handler, but the ack
assert_eq!(count_queue_messages(&pool, &app_id, "jobs").await, 0); // happens after it returns — so poll rather than assert on the instant the
// marker appears (that window is real, and asserting into it is flaky).
assert_eq!(poll_queue_depth(&pool, &app_id, "jobs", 0).await, 0);
} }
#[tokio::test(flavor = "multi_thread", worker_threads = 4)] #[tokio::test(flavor = "multi_thread", worker_threads = 4)]