feat(v1.1.1-dispatcher): dispatcher loop + retry + depth limit + outbox emitter
`OutboxEventEmitter` replaces `NoopEventEmitter` in the picloud binary's `Services` bundle. KV mutations now fan out to the outbox via `TriggerRepo::list_matching_kv` — one row per matching trigger, carrying the serialized `TriggerEvent` payload + the matching trigger's retry policy. `Dispatcher` is the single tokio task that polls the outbox every 100ms, claims due rows via FOR UPDATE SKIP LOCKED (with a batch cap), and routes each to the executor. Shares the `ExecutionGate` with sync HTTP per design notes §2 — gate saturation reschedules the row instead of dropping it. Outcome handling matches design notes §3 and §4: - reply_to.is_some() (sync HTTP): never retry. Deliver via `InboxResolver`; if the receiver was dropped, write an `abandoned_executions` row. - is_dead_letter_handler == true: never retry, never DL. On failure, annotate the original DL row with `resolution = 'handler_failed'`. Stops the recursion that would otherwise re-fire a broken handler script. - Otherwise async: bump attempt_count, reschedule with exponential backoff + ±jitter; once max_attempts is reached, write a `dead_letters` row and drop from outbox. - Trigger-depth limit: `cx.trigger_depth > max_trigger_depth` skips execution entirely (log + future metric), NEVER dead-letters. Loops are not retried via the DL chain — they're terminated. `InboxResolver` trait lands in `picloud-shared` with a `NoopInboxResolver` bootstrap that flags every delivery as `Abandoned`. Commit 6 replaces the noop with the real in-process registry in `orchestrator-core`. `AdminPrincipalResolver` builds a `Principal` from a trigger's `registered_by_principal` user id so the dispatched script executes as the trigger registrant (design notes §4). Unit tests cover backoff math (exponential/linear/constant) + jitter range + ExecError → InboxFailureKind classification + the status-code table mapping. Integration tests for the full dispatcher loop need a real Postgres + executor; reviewer runs them via the manual smoke flow in the plan / HANDBACK. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
103
crates/manager-core/src/outbox_event_emitter.rs
Normal file
103
crates/manager-core/src/outbox_event_emitter.rs
Normal file
@@ -0,0 +1,103 @@
|
||||
//! `OutboxEventEmitter` — the real `ServiceEventEmitter` that replaces
|
||||
//! v1.1.0's `NoopEventEmitter` once the triggers framework lands.
|
||||
//!
|
||||
//! On each `emit` (a KV mutation, future doc/file/pubsub event, etc.):
|
||||
//! 1. Look up matching triggers for the event's (app_id, source, op,
|
||||
//! collection) tuple via `TriggerRepo::list_matching_*`.
|
||||
//! 2. For each match, write one outbox row carrying the event payload
|
||||
//! serialized as a `TriggerEvent`.
|
||||
//!
|
||||
//! Defaults applied at write time so `OutboxRow.payload` carries
|
||||
//! everything the dispatcher needs to reconstruct the executor
|
||||
//! invocation without joining back to the trigger row.
|
||||
//!
|
||||
//! Non-KV `ServiceEvent` sources are silently dropped in v1.1.1 — the
|
||||
//! dispatcher only knows how to fire KV triggers this release. Future
|
||||
//! sources (docs/files/pubsub) add their own dispatch arm.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
EmitError, KvEventOp, SdkCallCx, ServiceEvent, ServiceEventEmitter, TriggerEvent,
|
||||
};
|
||||
|
||||
use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxSourceKind};
|
||||
use crate::trigger_repo::TriggerRepo;
|
||||
|
||||
pub struct OutboxEventEmitter {
|
||||
triggers: Arc<dyn TriggerRepo>,
|
||||
outbox: Arc<dyn OutboxRepo>,
|
||||
}
|
||||
|
||||
impl OutboxEventEmitter {
|
||||
#[must_use]
|
||||
pub fn new(triggers: Arc<dyn TriggerRepo>, outbox: Arc<dyn OutboxRepo>) -> Self {
|
||||
Self { triggers, outbox }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl ServiceEventEmitter for OutboxEventEmitter {
|
||||
async fn emit(&self, cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
|
||||
match event.source {
|
||||
"kv" => self.emit_kv(cx, event).await,
|
||||
// Future sources land here. For now, silently drop — the
|
||||
// SDK calls `events.emit(...)` unconditionally for forward
|
||||
// compat, so swallowing without an error is correct.
|
||||
_ => Ok(()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl OutboxEventEmitter {
|
||||
async fn emit_kv(&self, cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
|
||||
let Some(op) = KvEventOp::from_wire(event.op) else {
|
||||
return Ok(()); // unknown op — drop quietly
|
||||
};
|
||||
let Some(collection) = event.collection.clone() else {
|
||||
return Ok(()); // KV events always carry a collection — defensively skip
|
||||
};
|
||||
let key = event.key.clone().unwrap_or_default();
|
||||
|
||||
let matches = self
|
||||
.triggers
|
||||
.list_matching_kv(cx.app_id, &collection, op)
|
||||
.await
|
||||
.map_err(|e| EmitError::Unavailable(format!("trigger lookup: {e}")))?;
|
||||
|
||||
if matches.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Serialize the originating event as a TriggerEvent so the
|
||||
// dispatcher can hand it to the script as `ctx.event` without
|
||||
// round-tripping back to the trigger row.
|
||||
let trigger_event = TriggerEvent::Kv {
|
||||
op,
|
||||
collection,
|
||||
key,
|
||||
value: event.payload.clone(),
|
||||
};
|
||||
let payload = serde_json::to_value(&trigger_event)
|
||||
.map_err(|e| EmitError::Rejected(format!("event serialize: {e}")))?;
|
||||
|
||||
for m in matches {
|
||||
self.outbox
|
||||
.insert(NewOutboxRow {
|
||||
app_id: cx.app_id,
|
||||
source_kind: OutboxSourceKind::Kv,
|
||||
trigger_id: Some(m.trigger_id),
|
||||
script_id: Some(m.script_id),
|
||||
reply_to: None,
|
||||
payload: payload.clone(),
|
||||
origin_principal: cx.principal.as_ref().map(|p| p.user_id),
|
||||
trigger_depth: cx.trigger_depth.saturating_add(1),
|
||||
root_execution_id: Some(cx.root_execution_id),
|
||||
})
|
||||
.await
|
||||
.map_err(|e| EmitError::Unavailable(format!("outbox insert: {e}")))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user