feat(workflows): M2 — durable orchestrator worker (claim/execute/advance)
The v1.2 Workflows durable DAG engine gains its runtime. A dedicated
background worker (`workflow_orchestrator.rs`, mirroring `cron_scheduler`,
NOT folded into the dispatcher) advances every in-flight run step-by-step,
durably, surviving restarts.
Per tick, two phases:
A. Claim + execute — up to a small batch of `ready`, due steps are claimed
with the same `FOR UPDATE SKIP LOCKED` competing-consumer lease the queue
uses (`claim_ready_step`), one execution-gate permit per step acquired
BEFORE the claim so the shared gate bounds real parallelism. Each step
resolves its function by name in the run's app scope (never a
script-passed arg — the isolation boundary), builds an `ExecRequest`, and
runs through the injected `ExecutorClient`. Claimed steps run concurrently.
B. Advance — the outcome is written and the DAG advanced in one token-gated
transaction (`complete_step_and_advance`): pending steps whose deps are
satisfied flip to `ready`, and the run's terminal status is recomputed.
Fan-in falls out (a join waits until its last dep flips it); a stale worker
matches zero rows and writes nothing.
The graph-advance decision is a pure, DB-free function (`compute_advance`) —
promotions + terminal run status, folding `on_error` fail-vs-continue and
skipped/failed dependency satisfaction — so it is unit-tested in isolation.
Retry uses the step's own policy via `compute_backoff`; a second, slower
cadence reclaims steps leased by a crashed worker (`reclaim_stale_steps`) — the
durability safety net. Steps run with no principal (like invoke_async), and log
under the new `ExecutionSource::Workflow` so `pic logs` surfaces them.
M2 executes function steps only; `when` + input templating land in M3, nested
sub-workflows in M4. Seams present (`StepTarget`, `run_input`, `workflow_depth`).
- migration 0072: widen the `execution_logs.source` CHECK with `workflow`
- shared: `ExecutionSource::Workflow`, `StepStatus::is_terminal`
- workflow_repo: run/step state — `start_run`, `claim_ready_step`,
`complete_step_and_advance`, `reclaim_stale_steps`, `get_run`,
`list_run_steps`, `compute_advance` (+ 7 pure unit tests)
- workflow_orchestrator: the worker + config (`PICLOUD_WORKFLOW_*` knobs),
spawned in `picloud/src/lib.rs` beside the dispatcher/cron scheduler
- tests: 8 DB-gated integration tests (linear, parallel fan-out/fan-in, retry,
on_error fail/continue, double-complete idempotency, stale-lease reclaim,
end-to-end tick with a fake executor)
Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
8 workflow_orchestrator DB tests, schema snapshot reblessed, M1 workflow CLI
journeys still pass (binary boots with the orchestrator wired in).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -486,7 +486,7 @@ pub async fn build_app(
|
||||
abandoned: abandoned_repo.clone(),
|
||||
principals,
|
||||
executor: executor.clone(),
|
||||
gate,
|
||||
gate: gate.clone(),
|
||||
log_sink: log_sink.clone(),
|
||||
inbox: inbox_resolver,
|
||||
queue: queue_repo.clone(),
|
||||
@@ -496,6 +496,20 @@ pub async fn build_app(
|
||||
}
|
||||
.spawn();
|
||||
|
||||
// v1.2 Workflows (M2): the durable workflow orchestrator. Claims ready
|
||||
// steps of in-flight runs, executes each through the same executor + gate
|
||||
// the dispatcher uses, and advances the DAG — all durably, surviving
|
||||
// restarts. Spawned here (before `executor`/`gate` are moved into the data
|
||||
// plane) so it shares those Arcs.
|
||||
picloud_manager_core::spawn_workflow_orchestrator(picloud_manager_core::WorkflowOrchestrator {
|
||||
pool: pool.clone(),
|
||||
scripts: script_repo.clone(),
|
||||
executor: executor.clone(),
|
||||
gate: gate.clone(),
|
||||
log_sink: log_sink.clone(),
|
||||
config: picloud_manager_core::workflow_orchestrator::WorkflowOrchestratorConfig::from_env(),
|
||||
});
|
||||
|
||||
let admin = AdminState {
|
||||
repo: script_repo.clone(),
|
||||
logs: log_repo,
|
||||
|
||||
Reference in New Issue
Block a user