fabi/PiCloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(workflows): M2 — durable orchestrator worker (claim/execute/advance)

Browse Source 
The v1.2 Workflows durable DAG engine gains its runtime. A dedicated
background worker (`workflow_orchestrator.rs`, mirroring `cron_scheduler`,
NOT folded into the dispatcher) advances every in-flight run step-by-step,
durably, surviving restarts.

Per tick, two phases:
  A. Claim + execute — up to a small batch of `ready`, due steps are claimed
     with the same `FOR UPDATE SKIP LOCKED` competing-consumer lease the queue
     uses (`claim_ready_step`), one execution-gate permit per step acquired
     BEFORE the claim so the shared gate bounds real parallelism. Each step
     resolves its function by name in the run's app scope (never a
     script-passed arg — the isolation boundary), builds an `ExecRequest`, and
     runs through the injected `ExecutorClient`. Claimed steps run concurrently.
  B. Advance — the outcome is written and the DAG advanced in one token-gated
     transaction (`complete_step_and_advance`): pending steps whose deps are
     satisfied flip to `ready`, and the run's terminal status is recomputed.
     Fan-in falls out (a join waits until its last dep flips it); a stale worker
     matches zero rows and writes nothing.

The graph-advance decision is a pure, DB-free function (`compute_advance`) —
promotions + terminal run status, folding `on_error` fail-vs-continue and
skipped/failed dependency satisfaction — so it is unit-tested in isolation.

Retry uses the step's own policy via `compute_backoff`; a second, slower
cadence reclaims steps leased by a crashed worker (`reclaim_stale_steps`) — the
durability safety net. Steps run with no principal (like invoke_async), and log
under the new `ExecutionSource::Workflow` so `pic logs` surfaces them.

M2 executes function steps only; `when` + input templating land in M3, nested
sub-workflows in M4. Seams present (`StepTarget`, `run_input`, `workflow_depth`).

- migration 0072: widen the `execution_logs.source` CHECK with `workflow`
- shared: `ExecutionSource::Workflow`, `StepStatus::is_terminal`
- workflow_repo: run/step state — `start_run`, `claim_ready_step`,
  `complete_step_and_advance`, `reclaim_stale_steps`, `get_run`,
  `list_run_steps`, `compute_advance` (+ 7 pure unit tests)
- workflow_orchestrator: the worker + config (`PICLOUD_WORKFLOW_*` knobs),
  spawned in `picloud/src/lib.rs` beside the dispatcher/cron scheduler
- tests: 8 DB-gated integration tests (linear, parallel fan-out/fan-in, retry,
  on_error fail/continue, double-complete idempotency, stale-lease reclaim,
  end-to-end tick with a fake executor)

Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
8 workflow_orchestrator DB tests, schema snapshot reblessed, M1 workflow CLI
journeys still pass (binary boots with the orchestrator wired in).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-12 17:12:42 +02:00
parent 44f992cbb0
commit a55c2f112e
9 changed files with 1775 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

798
crates/manager-core/src/workflow_repo.rs
View File

@@ -6,8 +6,14 @@
//! declarative `apply` reconcile engine (Create/Update/Delete by `lower(name)`,
//! all in one transaction) — mirroring `trigger_repo::insert_trigger_tx`.
//!
//! Run/step state (`workflow_runs`, `workflow_run_steps`) is added in M2 with
//! the orchestrator.
//! Run/step state (`workflow_runs`, `workflow_run_steps`) lands in M2: the
//! durable orchestrator seeds a run (`start_run`), claims a `ready` step with
//! the same `FOR UPDATE SKIP LOCKED` lease `queue_repo` uses, and — in one
//! token-gated transaction — writes the step outcome and advances the DAG
//! (`complete_step_and_advance`). The graph-advance decision is a pure,
//! DB-free function ([`compute_advance`]) so it is unit-testable.
use std::collections::BTreeMap;
use async_trait::async_trait;
use chrono::{DateTime, Utc};
@@ -15,8 +21,10 @@ use sqlx::PgPool;
use thiserror::Error;
use uuid::Uuid;
use picloud_shared::workflow::WorkflowDefinition;
use picloud_shared::{AppId, WorkflowId};
use picloud_shared::workflow::{
OnError, RunStatus, StepStatus, WorkflowDefinition, WorkflowStepDef,
};
use picloud_shared::{AppId, WorkflowId, WorkflowRunId, WorkflowRunStepId};
#[derive(Debug, Error)]
pub enum WorkflowRepoError {
@@ -227,3 +235,785 @@ pub async fn delete_workflow_tx(
.await?;
Ok(())
}
// ===========================================================================
// M2 — run/step state + the durable orchestrator's persistence surface.
// ===========================================================================
/// Everything needed to seed a fresh run (mirrors `NewQueueMessage`). The
/// orchestrator's M5 `workflow::start` and the M2 tests both build this.
#[derive(Debug, Clone)]
pub struct NewWorkflowRun {
pub workflow_id: WorkflowId,
pub app_id: AppId,
pub input: serde_json::Value,
/// Correlates every step execution in `execution_logs` under one id.
pub root_execution_id: Uuid,
/// 0 for a top-level run; a nested sub-workflow (M4) is parent + 1.
pub workflow_depth: i32,
pub parent_run_id: Option<WorkflowRunId>,
pub parent_step_id: Option<WorkflowRunStepId>,
}
/// A run row, read for the admin/CLI surface + test assertions.
#[derive(Debug, Clone)]
pub struct WorkflowRun {
pub id: WorkflowRunId,
pub workflow_id: WorkflowId,
pub app_id: AppId,
pub status: RunStatus,
pub input: serde_json::Value,
pub output: Option<serde_json::Value>,
pub error: Option<String>,
pub root_execution_id: Uuid,
pub workflow_depth: i32,
pub parent_run_id: Option<WorkflowRunId>,
pub parent_step_id: Option<WorkflowRunStepId>,
pub started_at: Option<DateTime<Utc>>,
pub finished_at: Option<DateTime<Utc>>,
pub created_at: DateTime<Utc>,
}
/// A step row within a run (read for the admin/CLI surface + assertions).
#[derive(Debug, Clone)]
pub struct WorkflowRunStep {
pub id: WorkflowRunStepId,
pub run_id: WorkflowRunId,
pub step_name: String,
pub status: StepStatus,
pub attempt: i32,
pub max_attempts: i32,
pub output: Option<serde_json::Value>,
pub error: Option<String>,
pub child_run_id: Option<WorkflowRunId>,
}
/// A claimed `ready` step, joined with its run + workflow definition so the
/// orchestrator has everything it needs to resolve, execute, and advance
/// without a second round trip.
#[derive(Debug, Clone)]
pub struct ClaimedStep {
pub step_id: WorkflowRunStepId,
pub run_id: WorkflowRunId,
pub step_name: String,
/// 1-indexed current attempt (the claim increments it).
pub attempt: i32,
pub max_attempts: i32,
pub claim_token: Uuid,
// ---- run context ----
pub app_id: AppId,
pub workflow_id: WorkflowId,
pub root_execution_id: Uuid,
pub workflow_depth: i32,
pub run_input: serde_json::Value,
pub definition: WorkflowDefinition,
}
impl ClaimedStep {
/// The definition of the step being executed.
#[must_use]
pub fn step_def(&self) -> Option<&WorkflowStepDef> {
self.definition
.steps
.iter()
.find(|s| s.name == self.step_name)
}
}
/// The outcome the orchestrator hands back after executing a step. On
/// `Failed`, the repo decides retry-vs-terminal from `attempt < max_attempts`;
/// the orchestrator supplies the delay to use if a retry remains.
#[derive(Debug, Clone)]
pub enum StepOutcome {
Succeeded(serde_json::Value),
Failed {
error: String,
retry_delay: chrono::Duration,
},
}
/// What `complete_step_and_advance` did — surfaced mostly for tests/logging.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AdvanceResult {
/// The claim token no longer matched (a stale worker); nothing written.
Stale,
/// The step failed but had attempts left; it was re-armed as `ready`.
Retried,
/// The step reached a terminal status and the run graph was advanced.
Advanced,
}
// ---- the pure graph-advance decision (DB-free, unit-tested) --------------
/// The write plan `complete_step_and_advance` derives after a step reaches a
/// terminal status: which `pending` steps become `ready`, and the run's new
/// status/output/error.
#[derive(Debug, Clone, PartialEq)]
pub struct AdvancePlan {
pub promote: Vec<String>,
pub run_status: RunStatus,
pub run_error: Option<String>,
pub run_output: Option<serde_json::Value>,
}
/// Is `dep` satisfied for the purpose of unblocking a dependent? Succeeded and
/// skipped always satisfy; a `failed` step satisfies **only** when its
/// definition says `on_error = continue` (the run limps on).
fn dep_satisfied(status: StepStatus, on_error: OnError) -> bool {
match status {
StepStatus::Succeeded | StepStatus::Skipped => true,
StepStatus::Failed => matches!(on_error, OnError::Continue),
_ => false,
}
}
/// Pure DAG advance. Given the definition and the current per-step
/// status/output snapshot, decide the promotions + terminal run state.
///
/// - A `pending` step whose every `depends_on` is satisfied becomes `ready`.
/// - If any step `failed` with `on_error = fail`, the run fails immediately.
/// - Else if every step is terminal, the run succeeds; its output is the
/// object of `{ step_name: output }` over the succeeded steps.
/// - Otherwise the run is still `running`.
#[must_use]
pub fn compute_advance(
def: &WorkflowDefinition,
statuses: &BTreeMap<String, StepStatus>,
outputs: &BTreeMap<String, serde_json::Value>,
) -> AdvancePlan {
let by_name: BTreeMap<&str, &WorkflowStepDef> =
def.steps.iter().map(|s| (s.name.as_str(), s)).collect();
let status_of = |name: &str| statuses.get(name).copied().unwrap_or(StepStatus::Pending);
// Promote every pending step whose deps are all satisfied.
let mut promote = Vec::new();
for step in &def.steps {
if status_of(&step.name) != StepStatus::Pending {
continue;
}
let ready = step.depends_on.iter().all(|dep| {
let d_on_err = by_name
.get(dep.as_str())
.map_or(OnError::Fail, |s| s.on_error);
dep_satisfied(status_of(dep), d_on_err)
});
if ready {
promote.push(step.name.clone());
}
}
// A hard failure (on_error = fail that exhausted) fails the whole run.
if let Some(failed) = def
.steps
.iter()
.find(|s| status_of(&s.name) == StepStatus::Failed && matches!(s.on_error, OnError::Fail))
{
return AdvancePlan {
promote,
run_status: RunStatus::Failed,
run_error: Some(format!("step {:?} failed", failed.name)),
run_output: None,
};
}
// All steps terminal (accounting for the promotions we just queued — a
// promoted step is pending → ready, i.e. not terminal, so the run stays
// running whenever `promote` is non-empty).
let all_terminal =
promote.is_empty() && def.steps.iter().all(|s| status_of(&s.name).is_terminal());
if all_terminal {
let mut out = serde_json::Map::new();
for step in &def.steps {
if status_of(&step.name) == StepStatus::Succeeded {
if let Some(v) = outputs.get(&step.name) {
out.insert(step.name.clone(), v.clone());
}
}
}
return AdvancePlan {
promote,
run_status: RunStatus::Succeeded,
run_error: None,
run_output: Some(serde_json::Value::Object(out)),
};
}
AdvancePlan {
promote,
run_status: RunStatus::Running,
run_error: None,
run_output: None,
}
}
// ---- run seeding ----------------------------------------------------------
/// Seed a run: insert the `workflow_runs` row + one `workflow_run_steps` row
/// per definition step (root steps — no `depends_on` — start `ready`, the rest
/// `pending`). All in one transaction so a partially-seeded run is never
/// visible to the orchestrator's claim scan.
pub async fn start_run(
pool: &PgPool,
new: NewWorkflowRun,
definition: &WorkflowDefinition,
) -> Result<WorkflowRunId, WorkflowRepoError> {
let mut tx = pool.begin().await?;
let (run_id,): (Uuid,) = sqlx::query_as(
"INSERT INTO workflow_runs \
(workflow_id, app_id, status, input, root_execution_id, \
workflow_depth, parent_run_id, parent_step_id) \
VALUES ($1, $2, 'pending', $3, $4, $5, $6, $7) RETURNING id",
)
.bind(new.workflow_id.into_inner())
.bind(new.app_id.into_inner())
.bind(&new.input)
.bind(new.root_execution_id)
.bind(new.workflow_depth)
.bind(new.parent_run_id.map(WorkflowRunId::into_inner))
.bind(new.parent_step_id.map(WorkflowRunStepId::into_inner))
.fetch_one(&mut *tx)
.await?;
for step in &definition.steps {
let status = if step.depends_on.is_empty() {
"ready"
} else {
"pending"
};
let max_attempts = i32::try_from(step.max_attempts()).unwrap_or(1);
sqlx::query(
"INSERT INTO workflow_run_steps \
(run_id, step_name, status, max_attempts) \
VALUES ($1, $2, $3, $4)",
)
.bind(run_id)
.bind(&step.name)
.bind(status)
.bind(max_attempts)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
Ok(run_id.into())
}
// ---- claim / complete / reclaim ------------------------------------------
#[derive(sqlx::FromRow)]
struct ClaimedRow {
id: Uuid,
run_id: Uuid,
step_name: String,
attempt: i32,
max_attempts: i32,
app_id: Uuid,
workflow_id: Uuid,
root_execution_id: Uuid,
workflow_depth: i32,
input: serde_json::Value,
definition: serde_json::Value,
}
/// Atomically claim one `ready`, due step across all active runs — the queue
/// competing-consumer lease (`FOR UPDATE SKIP LOCKED`), so parallel workers
/// (and, cluster mode later, parallel nodes) never grab the same step. The
/// claimed step flips to `running` with a fresh `claim_token`; its run flips
/// `pending → running`. Returns `None` when nothing is claimable.
pub async fn claim_ready_step(pool: &PgPool) -> Result<Option<ClaimedStep>, WorkflowRepoError> {
let token = Uuid::new_v4();
let row: Option<ClaimedRow> = sqlx::query_as(
"WITH claimed AS ( \
UPDATE workflow_run_steps \
SET status = 'running', claim_token = $1, claimed_at = NOW(), \
attempt = attempt + 1, updated_at = NOW() \
WHERE id = ( \
SELECT s.id FROM workflow_run_steps s \
JOIN workflow_runs r ON r.id = s.run_id \
WHERE s.status = 'ready' \
AND (s.next_attempt_at IS NULL OR s.next_attempt_at <= NOW()) \
AND r.status IN ('pending', 'running') \
ORDER BY s.created_at \
FOR UPDATE OF s SKIP LOCKED \
LIMIT 1 \
) \
RETURNING id, run_id, step_name, attempt, max_attempts \
) \
SELECT c.id, c.run_id, c.step_name, c.attempt, c.max_attempts, \
r.app_id, r.workflow_id, r.root_execution_id, r.workflow_depth, \
r.input, w.definition \
FROM claimed c \
JOIN workflow_runs r ON r.id = c.run_id \
JOIN workflows w ON w.id = r.workflow_id",
)
.bind(token)
.fetch_optional(pool)
.await?;
let Some(r) = row else { return Ok(None) };
// Reflect the run as running promptly (advance also does this, but a
// long-running first step would otherwise leave the run 'pending').
sqlx::query(
"UPDATE workflow_runs SET status = 'running', started_at = COALESCE(started_at, NOW()) \
WHERE id = $1 AND status = 'pending'",
)
.bind(r.run_id)
.execute(pool)
.await?;
let definition = serde_json::from_value(r.definition)
.map_err(|e| WorkflowRepoError::Serde(e.to_string()))?;
Ok(Some(ClaimedStep {
step_id: r.id.into(),
run_id: r.run_id.into(),
step_name: r.step_name,
attempt: r.attempt,
max_attempts: r.max_attempts,
claim_token: token,
app_id: r.app_id.into(),
workflow_id: r.workflow_id.into(),
root_execution_id: r.root_execution_id,
workflow_depth: r.workflow_depth,
run_input: r.input,
definition,
}))
}
#[derive(sqlx::FromRow)]
struct StepStateRow {
step_name: String,
status: String,
output: Option<serde_json::Value>,
}
/// Write a claimed step's outcome and advance the run graph — all in one
/// token-gated transaction. A stale worker (claim reclaimed out from under it)
/// matches zero rows and writes nothing (`AdvanceResult::Stale`).
///
/// - `Succeeded` → the step is marked `succeeded` with its output, then the
/// graph advances (promotions + terminal run status).
/// - `Failed` with attempts left → the step is re-armed `ready` with a backoff
/// gate; the run keeps running (`Retried`, no advance).
/// - `Failed` exhausted → the step is marked `failed`, then the graph advances
/// (its `on_error` decides whether the run fails or limps on).
#[allow(clippy::too_many_lines)]
pub async fn complete_step_and_advance(
pool: &PgPool,
claimed: &ClaimedStep,
outcome: StepOutcome,
) -> Result<AdvanceResult, WorkflowRepoError> {
let mut tx = pool.begin().await?;
// 1. Token-gated step write.
match &outcome {
StepOutcome::Succeeded(out) => {
let res = sqlx::query(
"UPDATE workflow_run_steps \
SET status = 'succeeded', output = $3, error = NULL, \
claim_token = NULL, claimed_at = NULL, updated_at = NOW() \
WHERE id = $1 AND claim_token = $2",
)
.bind(claimed.step_id.into_inner())
.bind(claimed.claim_token)
.bind(out)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tx.rollback().await?;
return Ok(AdvanceResult::Stale);
}
}
StepOutcome::Failed { error, retry_delay } => {
if claimed.attempt < claimed.max_attempts {
let next = Utc::now() + *retry_delay;
let res = sqlx::query(
"UPDATE workflow_run_steps \
SET status = 'ready', error = $3, claim_token = NULL, \
claimed_at = NULL, next_attempt_at = $4, updated_at = NOW() \
WHERE id = $1 AND claim_token = $2",
)
.bind(claimed.step_id.into_inner())
.bind(claimed.claim_token)
.bind(error)
.bind(next)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tx.rollback().await?;
return Ok(AdvanceResult::Stale);
}
tx.commit().await?;
return Ok(AdvanceResult::Retried);
}
let res = sqlx::query(
"UPDATE workflow_run_steps \
SET status = 'failed', error = $3, claim_token = NULL, \
claimed_at = NULL, updated_at = NOW() \
WHERE id = $1 AND claim_token = $2",
)
.bind(claimed.step_id.into_inner())
.bind(claimed.claim_token)
.bind(error)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tx.rollback().await?;
return Ok(AdvanceResult::Stale);
}
}
}
// 2. Advance — lock the run row so concurrent advances of the same run
// serialize (each does a full recompute, so this is idempotent).
let run: Option<(String,)> =
sqlx::query_as("SELECT status FROM workflow_runs WHERE id = $1 FOR UPDATE")
.bind(claimed.run_id.into_inner())
.fetch_optional(&mut *tx)
.await?;
let Some((run_status,)) = run else {
tx.commit().await?;
return Ok(AdvanceResult::Advanced);
};
if RunStatus::from_str(&run_status).is_some_and(RunStatus::is_terminal) {
// Another worker already finished the run; nothing to do.
tx.commit().await?;
return Ok(AdvanceResult::Advanced);
}
let step_rows: Vec<StepStateRow> = sqlx::query_as(
"SELECT step_name, status, output FROM workflow_run_steps WHERE run_id = $1",
)
.bind(claimed.run_id.into_inner())
.fetch_all(&mut *tx)
.await?;
let mut statuses = BTreeMap::new();
let mut outputs = BTreeMap::new();
for r in step_rows {
if let Some(st) = StepStatus::from_str(&r.status) {
statuses.insert(r.step_name.clone(), st);
}
if let Some(o) = r.output {
outputs.insert(r.step_name, o);
}
}
let plan = compute_advance(&claimed.definition, &statuses, &outputs);
for name in &plan.promote {
sqlx::query(
"UPDATE workflow_run_steps \
SET status = 'ready', next_attempt_at = NULL, updated_at = NOW() \
WHERE run_id = $1 AND step_name = $2 AND status = 'pending'",
)
.bind(claimed.run_id.into_inner())
.bind(name)
.execute(&mut *tx)
.await?;
}
match plan.run_status {
RunStatus::Succeeded => {
sqlx::query(
"UPDATE workflow_runs \
SET status = 'succeeded', output = $2, finished_at = NOW() WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.bind(plan.run_output)
.execute(&mut *tx)
.await?;
}
RunStatus::Failed => {
sqlx::query(
"UPDATE workflow_runs \
SET status = 'failed', error = $2, finished_at = NOW() WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.bind(plan.run_error)
.execute(&mut *tx)
.await?;
}
_ => {
sqlx::query(
"UPDATE workflow_runs \
SET status = 'running', started_at = COALESCE(started_at, NOW()) WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.execute(&mut *tx)
.await?;
}
}
tx.commit().await?;
Ok(AdvanceResult::Advanced)
}
/// Periodic safety net: a step leased by a crashed worker (still `running`
/// with a `claim_token` past the visibility timeout) is re-armed `ready` so
/// another worker retries it. Only touches steps of still-active runs.
/// Returns the number of steps reclaimed.
pub async fn reclaim_stale_steps(
pool: &PgPool,
visibility_timeout_secs: u32,
) -> Result<u64, WorkflowRepoError> {
let res = sqlx::query(
"UPDATE workflow_run_steps s \
SET status = 'ready', claim_token = NULL, claimed_at = NULL, updated_at = NOW() \
FROM workflow_runs r \
WHERE s.run_id = r.id \
AND s.claim_token IS NOT NULL \
AND s.status = 'running' \
AND s.claimed_at < NOW() - ($1 || ' seconds')::INTERVAL \
AND r.status IN ('pending', 'running')",
)
.bind(i64::from(visibility_timeout_secs))
.execute(pool)
.await?;
Ok(res.rows_affected())
}
// ---- run/step reads (admin surface + test assertions) --------------------
#[derive(sqlx::FromRow)]
struct RunRow {
id: Uuid,
workflow_id: Uuid,
app_id: Uuid,
status: String,
input: serde_json::Value,
output: Option<serde_json::Value>,
error: Option<String>,
root_execution_id: Uuid,
workflow_depth: i32,
parent_run_id: Option<Uuid>,
parent_step_id: Option<Uuid>,
started_at: Option<DateTime<Utc>>,
finished_at: Option<DateTime<Utc>>,
created_at: DateTime<Utc>,
}
impl TryFrom<RunRow> for WorkflowRun {
type Error = WorkflowRepoError;
fn try_from(r: RunRow) -> Result<Self, Self::Error> {
Ok(Self {
id: r.id.into(),
workflow_id: r.workflow_id.into(),
app_id: r.app_id.into(),
status: RunStatus::from_str(&r.status).ok_or_else(|| {
WorkflowRepoError::Serde(format!("bad run status {:?}", r.status))
})?,
input: r.input,
output: r.output,
error: r.error,
root_execution_id: r.root_execution_id,
workflow_depth: r.workflow_depth,
parent_run_id: r.parent_run_id.map(Into::into),
parent_step_id: r.parent_step_id.map(Into::into),
started_at: r.started_at,
finished_at: r.finished_at,
created_at: r.created_at,
})
}
}
const RUN_COLS: &str = "id, workflow_id, app_id, status, input, output, error, \
root_execution_id, workflow_depth, parent_run_id, parent_step_id, \
started_at, finished_at, created_at";
/// Read a single run by id (app-scoped for isolation).
pub async fn get_run(
pool: &PgPool,
app_id: AppId,
run_id: WorkflowRunId,
) -> Result<Option<WorkflowRun>, WorkflowRepoError> {
let row: Option<RunRow> = sqlx::query_as(&format!(
"SELECT {RUN_COLS} FROM workflow_runs WHERE id = $1 AND app_id = $2"
))
.bind(run_id.into_inner())
.bind(app_id.into_inner())
.fetch_optional(pool)
.await?;
row.map(TryInto::try_into).transpose()
}
#[derive(sqlx::FromRow)]
struct StepRow {
id: Uuid,
run_id: Uuid,
step_name: String,
status: String,
attempt: i32,
max_attempts: i32,
output: Option<serde_json::Value>,
error: Option<String>,
child_run_id: Option<Uuid>,
}
impl TryFrom<StepRow> for WorkflowRunStep {
type Error = WorkflowRepoError;
fn try_from(r: StepRow) -> Result<Self, Self::Error> {
Ok(Self {
id: r.id.into(),
run_id: r.run_id.into(),
step_name: r.step_name,
status: StepStatus::from_str(&r.status).ok_or_else(|| {
WorkflowRepoError::Serde(format!("bad step status {:?}", r.status))
})?,
attempt: r.attempt,
max_attempts: r.max_attempts,
output: r.output,
error: r.error,
child_run_id: r.child_run_id.map(Into::into),
})
}
}
/// All steps of a run, ordered by name (for the admin/CLI surface + tests).
pub async fn list_run_steps(
pool: &PgPool,
run_id: WorkflowRunId,
) -> Result<Vec<WorkflowRunStep>, WorkflowRepoError> {
let rows: Vec<StepRow> = sqlx::query_as(
"SELECT id, run_id, step_name, status, attempt, max_attempts, output, error, \
child_run_id \
FROM workflow_run_steps WHERE run_id = $1 ORDER BY step_name",
)
.bind(run_id.into_inner())
.fetch_all(pool)
.await?;
rows.into_iter().map(TryInto::try_into).collect()
}
#[cfg(test)]
mod tests {
use super::*;
use picloud_shared::workflow::WorkflowStepDef;
use serde_json::json;
fn step(name: &str, deps: &[&str]) -> WorkflowStepDef {
WorkflowStepDef {
name: name.into(),
function: Some(format!("fn_{name}")),
workflow: None,
input: serde_json::Value::Null,
depends_on: deps.iter().map(|s| (*s).to_string()).collect(),
when: None,
retry: None,
on_error: OnError::Fail,
}
}
fn def(steps: Vec<WorkflowStepDef>) -> WorkflowDefinition {
WorkflowDefinition { steps }
}
fn statuses(pairs: &[(&str, StepStatus)]) -> BTreeMap<String, StepStatus> {
pairs.iter().map(|(n, s)| ((*n).to_string(), *s)).collect()
}
#[test]
fn promotes_dependent_when_dep_succeeds() {
// a -> b. a succeeded, b still pending → b becomes ready.
let d = def(vec![step("a", &[]), step("b", &["a"])]);
let st = statuses(&[("a", StepStatus::Succeeded), ("b", StepStatus::Pending)]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert_eq!(plan.promote, vec!["b".to_string()]);
assert_eq!(plan.run_status, RunStatus::Running);
}
#[test]
fn fan_in_waits_for_all_deps() {
// c depends on [a, b]. Only a done → c must NOT be promoted.
let d = def(vec![step("a", &[]), step("b", &[]), step("c", &["a", "b"])]);
let st = statuses(&[
("a", StepStatus::Succeeded),
("b", StepStatus::Running),
("c", StepStatus::Pending),
]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert!(plan.promote.is_empty());
assert_eq!(plan.run_status, RunStatus::Running);
// Now both a and b done → c promoted.
let st = statuses(&[
("a", StepStatus::Succeeded),
("b", StepStatus::Succeeded),
("c", StepStatus::Pending),
]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert_eq!(plan.promote, vec!["c".to_string()]);
}
#[test]
fn all_terminal_succeeds_with_output_object() {
let d = def(vec![step("a", &[]), step("b", &["a"])]);
let st = statuses(&[("a", StepStatus::Succeeded), ("b", StepStatus::Succeeded)]);
let mut out = BTreeMap::new();
out.insert("a".to_string(), json!({ "x": 1 }));
out.insert("b".to_string(), json!("done"));
let plan = compute_advance(&d, &st, &out);
assert!(plan.promote.is_empty());
assert_eq!(plan.run_status, RunStatus::Succeeded);
assert_eq!(
plan.run_output,
Some(json!({ "a": { "x": 1 }, "b": "done" }))
);
}
#[test]
fn on_error_fail_exhausted_fails_the_run() {
let d = def(vec![step("a", &[]), step("b", &["a"])]);
// a failed (on_error = fail) → run fails, b never promoted.
let st = statuses(&[("a", StepStatus::Failed), ("b", StepStatus::Pending)]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert!(plan.promote.is_empty());
assert_eq!(plan.run_status, RunStatus::Failed);
assert!(plan.run_error.is_some());
}
#[test]
fn on_error_continue_lets_dependents_proceed_and_run_succeed() {
let mut a = step("a", &[]);
a.on_error = OnError::Continue;
let d = def(vec![a, step("b", &["a"])]);
// a failed but continue → b promoted.
let st = statuses(&[("a", StepStatus::Failed), ("b", StepStatus::Pending)]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert_eq!(plan.promote, vec!["b".to_string()]);
assert_eq!(plan.run_status, RunStatus::Running);
// a failed-continue + b succeeded → run succeeds (a contributes no output).
let st = statuses(&[("a", StepStatus::Failed), ("b", StepStatus::Succeeded)]);
let mut out = BTreeMap::new();
out.insert("b".to_string(), json!(42));
let plan = compute_advance(&d, &st, &out);
assert_eq!(plan.run_status, RunStatus::Succeeded);
assert_eq!(plan.run_output, Some(json!({ "b": 42 })));
}
#[test]
fn skipped_dep_satisfies_dependents() {
let d = def(vec![step("a", &[]), step("b", &["a"])]);
let st = statuses(&[("a", StepStatus::Skipped), ("b", StepStatus::Pending)]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert_eq!(plan.promote, vec!["b".to_string()]);
}
#[test]
fn parallel_fan_out_promotes_all_ready_children() {
// root -> b, root -> c (diamond top). After root succeeds, both promote.
let d = def(vec![
step("root", &[]),
step("b", &["root"]),
step("c", &["root"]),
]);
let st = statuses(&[
("root", StepStatus::Succeeded),
("b", StepStatus::Pending),
("c", StepStatus::Pending),
]);
let plan = compute_advance(&d, &st, &BTreeMap::new());
assert_eq!(plan.promote, vec!["b".to_string(), "c".to_string()]);
assert_eq!(plan.run_status, RunStatus::Running);
}
}