feat(workflows): M2 — durable orchestrator worker (claim/execute/advance)
The v1.2 Workflows durable DAG engine gains its runtime. A dedicated
background worker (`workflow_orchestrator.rs`, mirroring `cron_scheduler`,
NOT folded into the dispatcher) advances every in-flight run step-by-step,
durably, surviving restarts.
Per tick, two phases:
A. Claim + execute — up to a small batch of `ready`, due steps are claimed
with the same `FOR UPDATE SKIP LOCKED` competing-consumer lease the queue
uses (`claim_ready_step`), one execution-gate permit per step acquired
BEFORE the claim so the shared gate bounds real parallelism. Each step
resolves its function by name in the run's app scope (never a
script-passed arg — the isolation boundary), builds an `ExecRequest`, and
runs through the injected `ExecutorClient`. Claimed steps run concurrently.
B. Advance — the outcome is written and the DAG advanced in one token-gated
transaction (`complete_step_and_advance`): pending steps whose deps are
satisfied flip to `ready`, and the run's terminal status is recomputed.
Fan-in falls out (a join waits until its last dep flips it); a stale worker
matches zero rows and writes nothing.
The graph-advance decision is a pure, DB-free function (`compute_advance`) —
promotions + terminal run status, folding `on_error` fail-vs-continue and
skipped/failed dependency satisfaction — so it is unit-tested in isolation.
Retry uses the step's own policy via `compute_backoff`; a second, slower
cadence reclaims steps leased by a crashed worker (`reclaim_stale_steps`) — the
durability safety net. Steps run with no principal (like invoke_async), and log
under the new `ExecutionSource::Workflow` so `pic logs` surfaces them.
M2 executes function steps only; `when` + input templating land in M3, nested
sub-workflows in M4. Seams present (`StepTarget`, `run_input`, `workflow_depth`).
- migration 0072: widen the `execution_logs.source` CHECK with `workflow`
- shared: `ExecutionSource::Workflow`, `StepStatus::is_terminal`
- workflow_repo: run/step state — `start_run`, `claim_ready_step`,
`complete_step_and_advance`, `reclaim_stale_steps`, `get_run`,
`list_run_steps`, `compute_advance` (+ 7 pure unit tests)
- workflow_orchestrator: the worker + config (`PICLOUD_WORKFLOW_*` knobs),
spawned in `picloud/src/lib.rs` beside the dispatcher/cron scheduler
- tests: 8 DB-gated integration tests (linear, parallel fan-out/fan-in, retry,
on_error fail/continue, double-complete idempotency, stale-lease reclaim,
end-to-end tick with a fake executor)
Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
8 workflow_orchestrator DB tests, schema snapshot reblessed, M1 workflow CLI
journeys still pass (binary boots with the orchestrator wired in).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -62,6 +62,7 @@ pub enum ExecutionSource {
|
||||
Email,
|
||||
Invoke,
|
||||
Queue,
|
||||
Workflow,
|
||||
}
|
||||
|
||||
impl ExecutionSource {
|
||||
@@ -78,6 +79,7 @@ impl ExecutionSource {
|
||||
Self::Email => "email",
|
||||
Self::Invoke => "invoke",
|
||||
Self::Queue => "queue",
|
||||
Self::Workflow => "workflow",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -96,6 +98,7 @@ impl ExecutionSource {
|
||||
"email" => Some(Self::Email),
|
||||
"invoke" => Some(Self::Invoke),
|
||||
"queue" => Some(Self::Queue),
|
||||
"workflow" => Some(Self::Workflow),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -181,11 +181,20 @@ impl StepStatus {
|
||||
|
||||
/// A dependency is "satisfied" (a dependent may proceed) once it has
|
||||
/// succeeded OR been skipped (a false `when` — satisfied-but-empty).
|
||||
/// NB: a `failed` step whose `on_error = continue` also satisfies its
|
||||
/// dependents, but that depends on the step's *definition*, not its
|
||||
/// status alone — the orchestrator's advance logic folds that in.
|
||||
#[must_use]
|
||||
pub fn is_satisfied(self) -> bool {
|
||||
matches!(self, Self::Succeeded | Self::Skipped)
|
||||
}
|
||||
|
||||
/// A step has reached a terminal status (no further transitions).
|
||||
#[must_use]
|
||||
pub fn is_terminal(self) -> bool {
|
||||
matches!(self, Self::Succeeded | Self::Failed | Self::Skipped)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
#[allow(clippy::should_implement_trait)]
|
||||
pub fn from_str(s: &str) -> Option<Self> {
|
||||
|
||||
Reference in New Issue
Block a user