fabi/PiCloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(workflows): M4 — nested sub-workflows (start_child, parent-park, depth ceiling)

Browse Source 
A `workflow`-kind step now starts a nested sub-workflow run instead of a
function. The orchestrator's step processing is restructured into prepare →
dispatch (`Prep`): after the shared context/`when`/input resolution, a
function step runs through the executor as before, while a workflow step:

- checks the nesting depth ceiling (`PICLOUD_WORKFLOW_MAX_DEPTH`; the child
  would run at parent depth + 1) — over-deep → the step fails, not infinite;
- resolves the child workflow by name in the run's app scope;
- `start_child_and_park`: in one token-gated tx, seeds a child run (depth + 1,
  `parent_run_id`/`parent_step_id` linkage, correlated under the same
  `root_execution_id`) and PARKS the parent step (`running`, claim_token
  cleared, `child_run_id` set). A parked step is never re-claimed (only `ready`
  steps are) nor reclaimed (only *leased* running steps). The token-gated park
  runs before the child insert, so a stale claim writes nothing (no orphan).

Each tick first runs `resume_finished_children`: a parent step parked on a
now-terminal child is resolved (child output → parent step output, or child
error → parent step failed) and the parent run advanced — idempotent via a
`status='running'` gate. The child's own steps are claimed by the same global
scan, so nesting is just more runs. A failed sub-workflow honors the parent
step's `on_error`.

Shared plumbing extracted for reuse: `advance_run_tx` (out of
`complete_step_and_advance`) and `seed_run_tx` (out of `start_run`).

Apply-time soft-warn (`workflow_nesting_warnings`, wired into `plan_warnings`):
a workflow that nests into itself — directly or via a mutual cycle within the
bundle — is flagged at plan (bounded by the depth ceiling, but almost always a
bug). Not an error.

Tests: nested output-flows-to-parent + depth-ceiling-fails-the-run (DB-gated,
end-to-end), self/mutual-cycle warning (pure unit). fmt + clippy -D warnings
clean, 449 manager-core lib tests, 13 orchestrator DB tests, 26 CLI journeys
(workflows/apply/plan/prune) green, binary boots.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-12 17:30:16 +02:00
parent 41a21c0551
commit 74d5874384
4 changed files with 674 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

229
crates/manager-core/src/workflow_repo.rs
View File

@@ -461,6 +461,19 @@ pub async fn start_run(
definition: &WorkflowDefinition,
) -> Result<WorkflowRunId, WorkflowRepoError> {
let mut tx = pool.begin().await?;
let run_id = seed_run_tx(&mut tx, &new, definition).await?;
tx.commit().await?;
Ok(run_id)
}
/// Insert the `workflow_runs` row + its `workflow_run_steps` (roots `ready`,
/// the rest `pending`) inside an open transaction. Shared by `start_run` and
/// the nested-child start (`start_child_and_park`).
pub async fn seed_run_tx(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
new: &NewWorkflowRun,
definition: &WorkflowDefinition,
) -> Result<WorkflowRunId, WorkflowRepoError> {
let (run_id,): (Uuid,) = sqlx::query_as(
"INSERT INTO workflow_runs \
(workflow_id, app_id, status, input, root_execution_id, \
@@ -474,7 +487,7 @@ pub async fn start_run(
.bind(new.workflow_depth)
.bind(new.parent_run_id.map(WorkflowRunId::into_inner))
.bind(new.parent_step_id.map(WorkflowRunStepId::into_inner))
.fetch_one(&mut *tx)
.fetch_one(&mut **tx)
.await?;
for step in &definition.steps {
@@ -493,11 +506,10 @@ pub async fn start_run(
.bind(&step.name)
.bind(status)
.bind(max_attempts)
.execute(&mut *tx)
.execute(&mut **tx)
.await?;
}
tx.commit().await?;
Ok(run_id.into())
}
@@ -683,28 +695,43 @@ pub async fn complete_step_and_advance(
}
}
// 2. Advance — lock the run row so concurrent advances of the same run
// serialize (each does a full recompute, so this is idempotent).
// 2. Advance the run graph in the same tx.
advance_run_tx(&mut tx, claimed.run_id, &claimed.definition).await?;
tx.commit().await?;
Ok(AdvanceResult::Advanced)
}
/// Advance one run's graph inside an open transaction: lock the run row (so
/// concurrent advances serialize — each does a full recompute, so this is
/// idempotent), promote pending steps whose deps are now satisfied, and write
/// the run's terminal status. A no-op if the run is already terminal. Does NOT
/// commit — the caller owns the transaction.
///
/// Shared by `complete_step_and_advance` (a function/skip step finished) and
/// `resume_finished_children` (a parked sub-workflow step finished).
pub async fn advance_run_tx(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
run_id: WorkflowRunId,
definition: &WorkflowDefinition,
) -> Result<(), WorkflowRepoError> {
let run: Option<(String,)> =
sqlx::query_as("SELECT status FROM workflow_runs WHERE id = $1 FOR UPDATE")
.bind(claimed.run_id.into_inner())
.fetch_optional(&mut *tx)
.bind(run_id.into_inner())
.fetch_optional(&mut **tx)
.await?;
let Some((run_status,)) = run else {
tx.commit().await?;
return Ok(AdvanceResult::Advanced);
return Ok(());
};
if RunStatus::from_str(&run_status).is_some_and(RunStatus::is_terminal) {
// Another worker already finished the run; nothing to do.
tx.commit().await?;
return Ok(AdvanceResult::Advanced);
return Ok(());
}
let step_rows: Vec<StepStateRow> = sqlx::query_as(
"SELECT step_name, status, output FROM workflow_run_steps WHERE run_id = $1",
)
.bind(claimed.run_id.into_inner())
.fetch_all(&mut *tx)
.bind(run_id.into_inner())
.fetch_all(&mut **tx)
.await?;
let mut statuses = BTreeMap::new();
let mut outputs = BTreeMap::new();
@@ -717,7 +744,7 @@ pub async fn complete_step_and_advance(
}
}
let plan = compute_advance(&claimed.definition, &statuses, &outputs);
let plan = compute_advance(definition, &statuses, &outputs);
for name in &plan.promote {
sqlx::query(
@@ -725,9 +752,9 @@ pub async fn complete_step_and_advance(
SET status = 'ready', next_attempt_at = NULL, updated_at = NOW() \
WHERE run_id = $1 AND step_name = $2 AND status = 'pending'",
)
.bind(claimed.run_id.into_inner())
.bind(run_id.into_inner())
.bind(name)
.execute(&mut *tx)
.execute(&mut **tx)
.await?;
}
@@ -737,9 +764,9 @@ pub async fn complete_step_and_advance(
"UPDATE workflow_runs \
SET status = 'succeeded', output = $2, finished_at = NOW() WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.bind(run_id.into_inner())
.bind(plan.run_output)
.execute(&mut *tx)
.execute(&mut **tx)
.await?;
}
RunStatus::Failed => {
@@ -747,9 +774,9 @@ pub async fn complete_step_and_advance(
"UPDATE workflow_runs \
SET status = 'failed', error = $2, finished_at = NOW() WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.bind(run_id.into_inner())
.bind(plan.run_error)
.execute(&mut *tx)
.execute(&mut **tx)
.await?;
}
_ => {
@@ -757,14 +784,13 @@ pub async fn complete_step_and_advance(
"UPDATE workflow_runs \
SET status = 'running', started_at = COALESCE(started_at, NOW()) WHERE id = $1",
)
.bind(claimed.run_id.into_inner())
.execute(&mut *tx)
.bind(run_id.into_inner())
.execute(&mut **tx)
.await?;
}
}
tx.commit().await?;
Ok(AdvanceResult::Advanced)
Ok(())
}
/// Periodic safety net: a step leased by a crashed worker (still `running`
@@ -924,6 +950,161 @@ pub async fn load_run_step_outputs(
.collect())
}
// ---- nested sub-workflows (M4) --------------------------------------------
/// Resolve an app-owned workflow by case-insensitive name (a `workflow`-kind
/// step's target). Pool free-fn mirror of `WorkflowRepo::get_by_name`.
pub async fn get_workflow_by_name(
pool: &PgPool,
app_id: AppId,
name: &str,
) -> Result<Option<Workflow>, WorkflowRepoError> {
let row: Option<WorkflowRow> = sqlx::query_as(&format!(
"SELECT {SELECT_COLS} FROM workflows WHERE app_id = $1 AND LOWER(name) = LOWER($2)"
))
.bind(app_id.into_inner())
.bind(name)
.fetch_optional(pool)
.await?;
row.map(TryInto::try_into).transpose()
}
/// Start a nested sub-workflow run and **park** the parent step on it. In one
/// token-gated transaction: the parent step (still claimed by the caller) is
/// set `running` with `claim_token` cleared and `child_run_id` pointing at a
/// freshly-seeded child run (depth + 1, parent linkage). The parked step is
/// never re-claimed (only `ready` steps are) nor reclaimed (only leased
/// `running` steps — `claim_token` is NULL here); `resume_finished_children`
/// resolves it once the child terminates. A stale worker matches zero rows and
/// nothing (including the child) is written.
pub async fn start_child_and_park(
pool: &PgPool,
claimed: &ClaimedStep,
child_workflow_id: WorkflowId,
child_definition: &WorkflowDefinition,
child_input: serde_json::Value,
) -> Result<AdvanceResult, WorkflowRepoError> {
let mut tx = pool.begin().await?;
// Token-gated park FIRST — so a stale claim short-circuits before we ever
// insert an orphan child run (child_run_id is linked below).
let res = sqlx::query(
"UPDATE workflow_run_steps \
SET status = 'running', claim_token = NULL, claimed_at = NULL, updated_at = NOW() \
WHERE id = $1 AND claim_token = $2",
)
.bind(claimed.step_id.into_inner())
.bind(claimed.claim_token)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tx.rollback().await?;
return Ok(AdvanceResult::Stale);
}
let new = NewWorkflowRun {
workflow_id: child_workflow_id,
app_id: claimed.app_id,
input: child_input,
// Correlate the whole nested tree under the top-level run's root id.
root_execution_id: claimed.root_execution_id,
workflow_depth: claimed.workflow_depth + 1,
parent_run_id: Some(claimed.run_id),
parent_step_id: Some(claimed.step_id),
};
let child_run = seed_run_tx(&mut tx, &new, child_definition).await?;
sqlx::query(
"UPDATE workflow_run_steps SET child_run_id = $2, updated_at = NOW() WHERE id = $1",
)
.bind(claimed.step_id.into_inner())
.bind(child_run.into_inner())
.execute(&mut *tx)
.await?;
sqlx::query(
"UPDATE workflow_runs SET status = 'running', started_at = COALESCE(started_at, NOW()) \
WHERE id = $1 AND status = 'pending'",
)
.bind(claimed.run_id.into_inner())
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(AdvanceResult::Advanced)
}
#[derive(sqlx::FromRow)]
struct ParkedRow {
parent_step_id: Uuid,
parent_run_id: Uuid,
child_status: String,
child_output: Option<serde_json::Value>,
child_error: Option<String>,
parent_definition: serde_json::Value,
}
/// Resolve parent steps parked on a now-terminal child sub-workflow: write the
/// child's output (or error) onto the parent step and advance the parent run.
/// Idempotent — the conditional `status = 'running'` gate means a second pass
/// (or a second worker) matches zero rows. Returns the number resolved.
pub async fn resume_finished_children(pool: &PgPool) -> Result<u64, WorkflowRepoError> {
let rows: Vec<ParkedRow> = sqlx::query_as(
"SELECT ps.id AS parent_step_id, ps.run_id AS parent_run_id, \
child.status AS child_status, child.output AS child_output, \
child.error AS child_error, w.definition AS parent_definition \
FROM workflow_run_steps ps \
JOIN workflow_runs child ON child.id = ps.child_run_id \
JOIN workflow_runs parent ON parent.id = ps.run_id \
JOIN workflows w ON w.id = parent.workflow_id \
WHERE ps.status = 'running' AND ps.child_run_id IS NOT NULL \
AND ps.claim_token IS NULL \
AND child.status IN ('succeeded', 'failed', 'canceled') \
LIMIT 32",
)
.fetch_all(pool)
.await?;
let mut resolved = 0u64;
for r in rows {
let mut tx = pool.begin().await?;
let (new_status, out, err) = if r.child_status == "succeeded" {
("succeeded", r.child_output.clone(), None)
} else {
(
"failed",
None,
Some(
r.child_error
.clone()
.unwrap_or_else(|| "sub-workflow failed".to_string()),
),
)
};
let res = sqlx::query(
"UPDATE workflow_run_steps \
SET status = $2, output = $3, error = $4, updated_at = NOW() \
WHERE id = $1 AND status = 'running' AND child_run_id IS NOT NULL",
)
.bind(r.parent_step_id)
.bind(new_status)
.bind(&out)
.bind(&err)
.execute(&mut *tx)
.await?;
if res.rows_affected() == 0 {
tx.rollback().await?;
continue; // already resolved by another pass
}
let parent_def: WorkflowDefinition = serde_json::from_value(r.parent_definition)
.map_err(|e| WorkflowRepoError::Serde(e.to_string()))?;
advance_run_tx(&mut tx, r.parent_run_id.into(), &parent_def).await?;
tx.commit().await?;
resolved += 1;
}
Ok(resolved)
}
#[cfg(test)]
mod tests {
use super::*;