feat(queue): dead-letter store for group shared queues (§11.6 D3 / Track A M2)
An exhausted SHARED durable-queue message was `drop_exhausted()`-ed with a
warning — silent data loss. Per-app queues persist to `dead_letters` (0010);
add the symmetric group store so an exhausted shared-queue message is preserved
and operator-visible.
- Migration 0068: `group_dead_letters`, keyed by (group_id, collection),
CASCADE on the group (an app delete leaves the data — it belongs to the
group, not the consuming app).
- `GroupQueueRepo::dead_letter` (replaces `drop_exhausted`): one tx that INSERTs
the dead-letter + DELETEs the live message, filtered by claim_token so a lost
lease can't dead-letter a re-claimed message (mirrors queue_repo::dead_letter).
- Dispatcher `q_terminal` shared arm now dead-letters instead of dropping. It
returns None (not the dl id) so the per-app `fan_out_dead_letter` is SKIPPED —
firing the consuming app's per-app handlers on a shared message (competing
consumers → nondeterministic app) would be wrong. Fan-out to a *shared*
dead_letter trigger is deferred (needs a new trigger kind).
- Read side: `GroupDeadLetterRepo::list_for_group` backs a new read-only
operator endpoint GET /api/v1/admin/groups/{id}/dead-letters (GroupKvRead,
mirrors the M4 group-blobs surface). `pic dead-letters ls --group` deferred
(optional; the HTTP endpoint is the operator surface).
Pinned by group_queue::dead_letter_moves_an_exhausted_message_to_the_group_store
(store move + claim-token-mismatch guard + operator read). Schema golden
reblessed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -408,19 +408,37 @@ impl Dispatcher {
|
||||
reason: &str,
|
||||
) -> Option<DeadLetterId> {
|
||||
match c.shared_group {
|
||||
Some(_) => {
|
||||
if let Err(e) = self
|
||||
Some(group_id) => {
|
||||
// §11.6 D3: persist the exhausted message to the group dead-letter
|
||||
// store instead of dropping it. We return None (not the dl id) so
|
||||
// the per-app `fan_out_dead_letter` below is SKIPPED — firing the
|
||||
// consuming app's per-app dead_letter handlers on a shared-queue
|
||||
// message (competing consumers → nondeterministic app) would be
|
||||
// wrong. Fan-out to a *shared* dead_letter trigger is deferred; the
|
||||
// row is operator-visible via the group dead-letters admin API.
|
||||
match self
|
||||
.group_queue
|
||||
.drop_exhausted(claimed.id, claimed.claim_token)
|
||||
.dead_letter(
|
||||
claimed.id,
|
||||
claimed.claim_token,
|
||||
group_id,
|
||||
&claimed.queue_name,
|
||||
trigger_id,
|
||||
script_id,
|
||||
claimed.attempt,
|
||||
claimed.enqueued_at,
|
||||
reason,
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(?e, "shared-queue drop failed");
|
||||
Ok(dl_id) => tracing::warn!(
|
||||
reason,
|
||||
queue = %claimed.queue_name,
|
||||
dead_letter_id = %dl_id.into_inner(),
|
||||
"shared-queue message dead-lettered"
|
||||
),
|
||||
Err(e) => tracing::error!(?e, "shared-queue dead-letter write failed"),
|
||||
}
|
||||
tracing::warn!(
|
||||
reason,
|
||||
queue = %claimed.queue_name,
|
||||
"shared-queue message dropped (no group dead-letter store yet)"
|
||||
);
|
||||
None
|
||||
}
|
||||
None => match self
|
||||
@@ -1663,12 +1681,21 @@ mod tests {
|
||||
) -> Result<bool, crate::group_queue_repo::GroupQueueRepoError> {
|
||||
Ok(false)
|
||||
}
|
||||
async fn drop_exhausted(
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
async fn dead_letter(
|
||||
&self,
|
||||
_id: QueueMessageId,
|
||||
_token: Uuid,
|
||||
) -> Result<bool, crate::group_queue_repo::GroupQueueRepoError> {
|
||||
Ok(false)
|
||||
_group_id: picloud_shared::GroupId,
|
||||
_collection: &str,
|
||||
_trigger_id: Option<picloud_shared::TriggerId>,
|
||||
_script_id: Option<ScriptId>,
|
||||
_attempt: u32,
|
||||
_first_attempt_at: chrono::DateTime<chrono::Utc>,
|
||||
_last_error: &str,
|
||||
) -> Result<picloud_shared::DeadLetterId, crate::group_queue_repo::GroupQueueRepoError>
|
||||
{
|
||||
unreachable!("shared queue not exercised")
|
||||
}
|
||||
async fn reclaim_visibility_timeouts(
|
||||
&self,
|
||||
|
||||
Reference in New Issue
Block a user