feat(queue): dead-letter store for group shared queues (§11.6 D3 / Track A M2)

An exhausted SHARED durable-queue message was `drop_exhausted()`-ed with a
warning — silent data loss. Per-app queues persist to `dead_letters` (0010);
add the symmetric group store so an exhausted shared-queue message is preserved
and operator-visible.

- Migration 0068: `group_dead_letters`, keyed by (group_id, collection),
  CASCADE on the group (an app delete leaves the data — it belongs to the
  group, not the consuming app).
- `GroupQueueRepo::dead_letter` (replaces `drop_exhausted`): one tx that INSERTs
  the dead-letter + DELETEs the live message, filtered by claim_token so a lost
  lease can't dead-letter a re-claimed message (mirrors queue_repo::dead_letter).
- Dispatcher `q_terminal` shared arm now dead-letters instead of dropping. It
  returns None (not the dl id) so the per-app `fan_out_dead_letter` is SKIPPED —
  firing the consuming app's per-app handlers on a shared message (competing
  consumers → nondeterministic app) would be wrong. Fan-out to a *shared*
  dead_letter trigger is deferred (needs a new trigger kind).
- Read side: `GroupDeadLetterRepo::list_for_group` backs a new read-only
  operator endpoint GET /api/v1/admin/groups/{id}/dead-letters (GroupKvRead,
  mirrors the M4 group-blobs surface). `pic dead-letters ls --group` deferred
  (optional; the HTTP endpoint is the operator surface).

Pinned by group_queue::dead_letter_moves_an_exhausted_message_to_the_group_store
(store move + claim-token-mismatch guard + operator read). Schema golden
reblessed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-11 14:41:04 +02:00
parent c06a9e801e
commit fd4336e883
9 changed files with 536 additions and 28 deletions

View File

@@ -408,19 +408,37 @@ impl Dispatcher {
reason: &str,
) -> Option<DeadLetterId> {
match c.shared_group {
Some(_) => {
if let Err(e) = self
Some(group_id) => {
// §11.6 D3: persist the exhausted message to the group dead-letter
// store instead of dropping it. We return None (not the dl id) so
// the per-app `fan_out_dead_letter` below is SKIPPED — firing the
// consuming app's per-app dead_letter handlers on a shared-queue
// message (competing consumers → nondeterministic app) would be
// wrong. Fan-out to a *shared* dead_letter trigger is deferred; the
// row is operator-visible via the group dead-letters admin API.
match self
.group_queue
.drop_exhausted(claimed.id, claimed.claim_token)
.dead_letter(
claimed.id,
claimed.claim_token,
group_id,
&claimed.queue_name,
trigger_id,
script_id,
claimed.attempt,
claimed.enqueued_at,
reason,
)
.await
{
tracing::warn!(?e, "shared-queue drop failed");
Ok(dl_id) => tracing::warn!(
reason,
queue = %claimed.queue_name,
dead_letter_id = %dl_id.into_inner(),
"shared-queue message dead-lettered"
),
Err(e) => tracing::error!(?e, "shared-queue dead-letter write failed"),
}
tracing::warn!(
reason,
queue = %claimed.queue_name,
"shared-queue message dropped (no group dead-letter store yet)"
);
None
}
None => match self
@@ -1663,12 +1681,21 @@ mod tests {
) -> Result<bool, crate::group_queue_repo::GroupQueueRepoError> {
Ok(false)
}
async fn drop_exhausted(
#[allow(clippy::too_many_arguments)]
async fn dead_letter(
&self,
_id: QueueMessageId,
_token: Uuid,
) -> Result<bool, crate::group_queue_repo::GroupQueueRepoError> {
Ok(false)
_group_id: picloud_shared::GroupId,
_collection: &str,
_trigger_id: Option<picloud_shared::TriggerId>,
_script_id: Option<ScriptId>,
_attempt: u32,
_first_attempt_at: chrono::DateTime<chrono::Utc>,
_last_error: &str,
) -> Result<picloud_shared::DeadLetterId, crate::group_queue_repo::GroupQueueRepoError>
{
unreachable!("shared queue not exercised")
}
async fn reclaim_visibility_timeouts(
&self,