feat(workflows): M2 — durable orchestrator worker (claim/execute/advance)
The v1.2 Workflows durable DAG engine gains its runtime. A dedicated
background worker (`workflow_orchestrator.rs`, mirroring `cron_scheduler`,
NOT folded into the dispatcher) advances every in-flight run step-by-step,
durably, surviving restarts.
Per tick, two phases:
A. Claim + execute — up to a small batch of `ready`, due steps are claimed
with the same `FOR UPDATE SKIP LOCKED` competing-consumer lease the queue
uses (`claim_ready_step`), one execution-gate permit per step acquired
BEFORE the claim so the shared gate bounds real parallelism. Each step
resolves its function by name in the run's app scope (never a
script-passed arg — the isolation boundary), builds an `ExecRequest`, and
runs through the injected `ExecutorClient`. Claimed steps run concurrently.
B. Advance — the outcome is written and the DAG advanced in one token-gated
transaction (`complete_step_and_advance`): pending steps whose deps are
satisfied flip to `ready`, and the run's terminal status is recomputed.
Fan-in falls out (a join waits until its last dep flips it); a stale worker
matches zero rows and writes nothing.
The graph-advance decision is a pure, DB-free function (`compute_advance`) —
promotions + terminal run status, folding `on_error` fail-vs-continue and
skipped/failed dependency satisfaction — so it is unit-tested in isolation.
Retry uses the step's own policy via `compute_backoff`; a second, slower
cadence reclaims steps leased by a crashed worker (`reclaim_stale_steps`) — the
durability safety net. Steps run with no principal (like invoke_async), and log
under the new `ExecutionSource::Workflow` so `pic logs` surfaces them.
M2 executes function steps only; `when` + input templating land in M3, nested
sub-workflows in M4. Seams present (`StepTarget`, `run_input`, `workflow_depth`).
- migration 0072: widen the `execution_logs.source` CHECK with `workflow`
- shared: `ExecutionSource::Workflow`, `StepStatus::is_terminal`
- workflow_repo: run/step state — `start_run`, `claim_ready_step`,
`complete_step_and_advance`, `reclaim_stale_steps`, `get_run`,
`list_run_steps`, `compute_advance` (+ 7 pure unit tests)
- workflow_orchestrator: the worker + config (`PICLOUD_WORKFLOW_*` knobs),
spawned in `picloud/src/lib.rs` beside the dispatcher/cron scheduler
- tests: 8 DB-gated integration tests (linear, parallel fan-out/fan-in, retry,
on_error fail/continue, double-complete idempotency, stale-lease reclaim,
end-to-end tick with a fake executor)
Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
8 workflow_orchestrator DB tests, schema snapshot reblessed, M1 workflow CLI
journeys still pass (binary boots with the orchestrator wired in).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
@@ -873,7 +873,7 @@ constraints on email_trigger_details:
|
||||
[PRIMARY KEY] email_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on execution_logs:
|
||||
[CHECK] execution_logs_source_check: CHECK ((source = ANY (ARRAY['http'::text, 'kv'::text, 'docs'::text, 'dead_letter'::text, 'cron'::text, 'files'::text, 'pubsub'::text, 'email'::text, 'invoke'::text, 'queue'::text])))
|
||||
[CHECK] execution_logs_source_check: CHECK ((source = ANY (ARRAY['http'::text, 'kv'::text, 'docs'::text, 'dead_letter'::text, 'cron'::text, 'files'::text, 'pubsub'::text, 'email'::text, 'invoke'::text, 'queue'::text, 'workflow'::text])))
|
||||
[CHECK] execution_logs_status_check: CHECK ((status = ANY (ARRAY['success'::text, 'error'::text, 'timeout'::text, 'budget_exceeded'::text])))
|
||||
[FOREIGN KEY] execution_logs_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] execution_logs_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE SET NULL
|
||||
@@ -1120,3 +1120,4 @@ constraints on workflows:
|
||||
0069: email secret version
|
||||
0070: admin session absolute expiry
|
||||
0071: workflows
|
||||
0072: execution source workflow
|
||||
|
||||
537
crates/manager-core/tests/workflow_orchestrator.rs
Normal file
537
crates/manager-core/tests/workflow_orchestrator.rs
Normal file
@@ -0,0 +1,537 @@
|
||||
//! v1.2 Workflows M2 — durable orchestrator integration tests.
|
||||
//!
|
||||
//! Two layers, both DB-gated (skip cleanly when `DATABASE_URL` is unset):
|
||||
//!
|
||||
//! * **State machine** — drive `claim_ready_step` → `complete_step_and_advance`
|
||||
//! in a deterministic loop (no executor, feeding canned outcomes) to pin the
|
||||
//! durable DAG advance: linear chains, parallel fan-out/fan-in, retry,
|
||||
//! on_error fail vs continue, stale-token idempotency, and stale-lease
|
||||
//! reclaim.
|
||||
//! * **End-to-end** — a real orchestrator `tick()` with a fake `ExecutorClient`
|
||||
//! and real `scripts` rows, asserting steps resolve + execute + advance to a
|
||||
//! terminal run.
|
||||
|
||||
#![allow(clippy::too_many_lines)]
|
||||
|
||||
use std::sync::{Arc, LazyLock};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{ExecError, ExecRequest, ExecResponse, ExecStats};
|
||||
use picloud_manager_core::repo::PostgresScriptRepository;
|
||||
use picloud_manager_core::workflow_orchestrator::{
|
||||
WorkflowOrchestrator, WorkflowOrchestratorConfig,
|
||||
};
|
||||
use picloud_manager_core::workflow_repo::{
|
||||
claim_ready_step, complete_step_and_advance, get_run, list_run_steps, reclaim_stale_steps,
|
||||
start_run, AdvanceResult, ClaimedStep, NewWorkflowRun, StepOutcome,
|
||||
};
|
||||
use picloud_orchestrator_core::{ExecutionGate, ExecutorClient, ScriptIdentity};
|
||||
use picloud_shared::workflow::{
|
||||
OnError, RunStatus, StepStatus, WorkflowBackoff, WorkflowDefinition, WorkflowRetry,
|
||||
WorkflowStepDef,
|
||||
};
|
||||
use picloud_shared::{AppId, ExecutionLogSink, LogSinkError, WorkflowId};
|
||||
use serde_json::{json, Value};
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use sqlx::PgPool;
|
||||
use std::time::Duration as StdDuration;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// `claim_ready_step` is deliberately **global** (one production orchestrator
|
||||
/// claims every app's ready steps). These tests share one DB, so a sibling
|
||||
/// test's concurrent claim would steal steps from the run under assertion. This
|
||||
/// process-wide lock serializes the claim-driving tests — each still creates
|
||||
/// its own app/workflow/run, and each finishes with no `ready` steps, so once
|
||||
/// serialized they don't interfere. (See the shared-DB test-harness note.)
|
||||
static CLAIM_LOCK: LazyLock<tokio::sync::Mutex<()>> = LazyLock::new(|| tokio::sync::Mutex::new(()));
|
||||
|
||||
/// Acquire the serialization lock AND clear any leftover runs/steps so the
|
||||
/// global `claim_ready_step`/`reclaim_stale_steps` see only this test's rows.
|
||||
/// (The shared dev DB accumulates rows across runs; `workflow_runs` CASCADEs to
|
||||
/// its steps.) Returns the held guard.
|
||||
async fn lock_and_reset(pool: &PgPool) -> tokio::sync::MutexGuard<'static, ()> {
|
||||
let guard = CLAIM_LOCK.lock().await;
|
||||
sqlx::query("DELETE FROM workflow_runs")
|
||||
.execute(pool)
|
||||
.await
|
||||
.expect("reset workflow_runs");
|
||||
guard
|
||||
}
|
||||
|
||||
async fn pool_or_skip() -> Option<PgPool> {
|
||||
let Ok(url) = std::env::var("DATABASE_URL") else {
|
||||
eprintln!("workflow_orchestrator: DATABASE_URL unset — skipping");
|
||||
return None;
|
||||
};
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(4)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect to DATABASE_URL");
|
||||
sqlx::migrate!("./migrations")
|
||||
.run(&pool)
|
||||
.await
|
||||
.expect("apply migrations");
|
||||
Some(pool)
|
||||
}
|
||||
|
||||
fn step(name: &str, function: &str, deps: &[&str]) -> WorkflowStepDef {
|
||||
WorkflowStepDef {
|
||||
name: name.into(),
|
||||
function: Some(function.into()),
|
||||
workflow: None,
|
||||
input: Value::Null,
|
||||
depends_on: deps.iter().map(|s| (*s).to_string()).collect(),
|
||||
when: None,
|
||||
retry: None,
|
||||
on_error: OnError::Fail,
|
||||
}
|
||||
}
|
||||
|
||||
/// Insert an app + a workflow definition; return `(app_id, workflow_id)`.
|
||||
async fn seed_workflow(pool: &PgPool, def: &WorkflowDefinition) -> (AppId, WorkflowId) {
|
||||
let sfx = Uuid::new_v4().simple().to_string();
|
||||
let grp: (Uuid,) =
|
||||
sqlx::query_as("INSERT INTO groups (slug, name) VALUES ($1, $1) RETURNING id")
|
||||
.bind(format!("wf-grp-{sfx}"))
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("insert group");
|
||||
let app: (Uuid,) =
|
||||
sqlx::query_as("INSERT INTO apps (slug, name, group_id) VALUES ($1, $1, $2) RETURNING id")
|
||||
.bind(format!("wf-app-{sfx}"))
|
||||
.bind(grp.0)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("insert app");
|
||||
let def_json = serde_json::to_value(def).unwrap();
|
||||
let wf: (Uuid,) = sqlx::query_as(
|
||||
"INSERT INTO workflows (app_id, name, definition) VALUES ($1, $2, $3) RETURNING id",
|
||||
)
|
||||
.bind(app.0)
|
||||
.bind(format!("wf-{sfx}"))
|
||||
.bind(def_json)
|
||||
.fetch_one(pool)
|
||||
.await
|
||||
.expect("insert workflow");
|
||||
(app.0.into(), wf.0.into())
|
||||
}
|
||||
|
||||
fn new_run(app_id: AppId, wf_id: WorkflowId) -> NewWorkflowRun {
|
||||
NewWorkflowRun {
|
||||
workflow_id: wf_id,
|
||||
app_id,
|
||||
input: json!({ "seed": true }),
|
||||
root_execution_id: Uuid::new_v4(),
|
||||
workflow_depth: 0,
|
||||
parent_run_id: None,
|
||||
parent_step_id: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Drive the run to a terminal state by claiming + completing steps, deciding
|
||||
/// each outcome via `decide` (keyed on step name + current attempt).
|
||||
async fn run_to_terminal<F>(pool: &PgPool, mut decide: F)
|
||||
where
|
||||
F: FnMut(&ClaimedStep) -> StepOutcome,
|
||||
{
|
||||
// Bounded so a bug can't spin forever.
|
||||
for _ in 0..200 {
|
||||
let Some(claimed) = claim_ready_step(pool).await.expect("claim") else {
|
||||
break;
|
||||
};
|
||||
let outcome = decide(&claimed);
|
||||
complete_step_and_advance(pool, &claimed, outcome)
|
||||
.await
|
||||
.expect("advance");
|
||||
}
|
||||
}
|
||||
|
||||
fn ok(body: Value) -> StepOutcome {
|
||||
StepOutcome::Succeeded(body)
|
||||
}
|
||||
|
||||
fn fail(msg: &str) -> StepOutcome {
|
||||
StepOutcome::Failed {
|
||||
error: msg.into(),
|
||||
retry_delay: chrono::Duration::zero(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn seeds_roots_ready_and_completes_linear_chain() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
// a -> b -> c
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![
|
||||
step("a", "fn_a", &[]),
|
||||
step("b", "fn_b", &["a"]),
|
||||
step("c", "fn_c", &["b"]),
|
||||
],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
// Root `a` starts ready; b, c pending.
|
||||
let steps = list_run_steps(&pool, run).await.unwrap();
|
||||
let by = |n: &str| steps.iter().find(|s| s.step_name == n).unwrap().status;
|
||||
assert_eq!(by("a"), StepStatus::Ready);
|
||||
assert_eq!(by("b"), StepStatus::Pending);
|
||||
assert_eq!(by("c"), StepStatus::Pending);
|
||||
|
||||
run_to_terminal(&pool, |c| ok(json!({ "step": c.step_name }))).await;
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
assert_eq!(
|
||||
r.output,
|
||||
Some(json!({
|
||||
"a": { "step": "a" },
|
||||
"b": { "step": "b" },
|
||||
"c": { "step": "c" },
|
||||
}))
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn parallel_fan_out_and_fan_in() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
// a -> {b, c} -> d (diamond).
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![
|
||||
step("a", "fn_a", &[]),
|
||||
step("b", "fn_b", &["a"]),
|
||||
step("c", "fn_c", &["a"]),
|
||||
step("d", "fn_d", &["b", "c"]),
|
||||
],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
// After `a` completes, BOTH b and c must be claimable before d.
|
||||
let a = claim_ready_step(&pool).await.unwrap().expect("claim a");
|
||||
assert_eq!(a.step_name, "a");
|
||||
complete_step_and_advance(&pool, &a, ok(json!("a")))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let s1 = claim_ready_step(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("first parallel");
|
||||
let s2 = claim_ready_step(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("second parallel");
|
||||
let mut names = [s1.step_name.clone(), s2.step_name.clone()];
|
||||
names.sort();
|
||||
assert_eq!(names, ["b".to_string(), "c".to_string()]);
|
||||
// d must NOT be claimable yet (fan-in waits).
|
||||
assert!(claim_ready_step(&pool).await.unwrap().is_none());
|
||||
|
||||
complete_step_and_advance(&pool, &s1, ok(json!("x")))
|
||||
.await
|
||||
.unwrap();
|
||||
// Still waiting on the other parallel branch.
|
||||
assert!(claim_ready_step(&pool).await.unwrap().is_none());
|
||||
complete_step_and_advance(&pool, &s2, ok(json!("y")))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let d = claim_ready_step(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("d after fan-in");
|
||||
assert_eq!(d.step_name, "d");
|
||||
complete_step_and_advance(&pool, &d, ok(json!("d")))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn retry_then_succeed() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
let mut a = step("a", "fn_a", &[]);
|
||||
a.retry = Some(WorkflowRetry {
|
||||
max_attempts: 3,
|
||||
backoff: WorkflowBackoff::Constant,
|
||||
base_ms: 0, // no delay → immediately re-claimable
|
||||
});
|
||||
let def = WorkflowDefinition { steps: vec![a] };
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
// First claim fails (attempt 1) → retried.
|
||||
let c1 = claim_ready_step(&pool).await.unwrap().expect("attempt 1");
|
||||
assert_eq!(c1.attempt, 1);
|
||||
let res = complete_step_and_advance(&pool, &c1, fail("boom"))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(res, AdvanceResult::Retried);
|
||||
|
||||
// Second claim (attempt 2) succeeds → run done.
|
||||
let c2 = claim_ready_step(&pool).await.unwrap().expect("attempt 2");
|
||||
assert_eq!(c2.attempt, 2);
|
||||
complete_step_and_advance(&pool, &c2, ok(json!("recovered")))
|
||||
.await
|
||||
.unwrap();
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn on_error_fail_fails_the_run() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
// a (fail) -> b; a has no retry and on_error=fail → run fails, b never runs.
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
run_to_terminal(&pool, |c| {
|
||||
if c.step_name == "a" {
|
||||
fail("kaboom")
|
||||
} else {
|
||||
ok(json!("b"))
|
||||
}
|
||||
})
|
||||
.await;
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Failed);
|
||||
assert!(r.error.unwrap().contains('a'));
|
||||
let steps = list_run_steps(&pool, run).await.unwrap();
|
||||
let b = steps.iter().find(|s| s.step_name == "b").unwrap();
|
||||
assert_eq!(b.status, StepStatus::Pending); // never promoted
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn on_error_continue_run_succeeds() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
let mut a = step("a", "fn_a", &[]);
|
||||
a.on_error = OnError::Continue;
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![a, step("b", "fn_b", &["a"])],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
run_to_terminal(&pool, |c| {
|
||||
if c.step_name == "a" {
|
||||
fail("soft")
|
||||
} else {
|
||||
ok(json!("b-ran"))
|
||||
}
|
||||
})
|
||||
.await;
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(
|
||||
r.status,
|
||||
RunStatus::Succeeded,
|
||||
"continue lets the run finish"
|
||||
);
|
||||
assert_eq!(r.output, Some(json!({ "b": "b-ran" })));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn double_complete_is_idempotent() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![step("a", "fn_a", &[])],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
let c = claim_ready_step(&pool).await.unwrap().expect("claim");
|
||||
let first = complete_step_and_advance(&pool, &c, ok(json!("first")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(first, AdvanceResult::Advanced);
|
||||
|
||||
// Replaying the SAME claim (stale token, lease long gone) writes nothing.
|
||||
let second = complete_step_and_advance(&pool, &c, ok(json!("SECOND")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(second, AdvanceResult::Stale);
|
||||
|
||||
let steps = list_run_steps(&pool, run).await.unwrap();
|
||||
assert_eq!(steps[0].output, Some(json!("first")), "first write wins");
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn stale_lease_is_reclaimed() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![step("a", "fn_a", &[])],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
let c = claim_ready_step(&pool).await.unwrap().expect("claim");
|
||||
// Nothing else claimable while `a` is leased.
|
||||
assert!(claim_ready_step(&pool).await.unwrap().is_none());
|
||||
|
||||
// Backdate the lease so it's past the (0s) visibility window.
|
||||
sqlx::query(
|
||||
"UPDATE workflow_run_steps SET claimed_at = NOW() - INTERVAL '1 hour' WHERE id = $1",
|
||||
)
|
||||
.bind(c.step_id.into_inner())
|
||||
.execute(&pool)
|
||||
.await
|
||||
.unwrap();
|
||||
let reclaimed = reclaim_stale_steps(&pool, 0).await.unwrap();
|
||||
assert_eq!(reclaimed, 1);
|
||||
|
||||
// Now claimable again (a fresh attempt).
|
||||
let c2 = claim_ready_step(&pool)
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("re-claim after reclaim");
|
||||
assert_eq!(c2.step_name, "a");
|
||||
complete_step_and_advance(&pool, &c2, ok(json!("done")))
|
||||
.await
|
||||
.unwrap();
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
}
|
||||
|
||||
// ---- end-to-end: a real orchestrator tick with a fake executor ------------
|
||||
|
||||
struct FakeExecutor {
|
||||
calls: std::sync::Mutex<Vec<String>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl ExecutorClient for FakeExecutor {
|
||||
async fn execute(
|
||||
&self,
|
||||
_source: &str,
|
||||
req: ExecRequest,
|
||||
_timeout: StdDuration,
|
||||
) -> Result<ExecResponse, ExecError> {
|
||||
self.calls.lock().unwrap().push(req.script_name.clone());
|
||||
Ok(ExecResponse {
|
||||
status_code: 200,
|
||||
headers: std::collections::BTreeMap::new(),
|
||||
body: json!({ "ran": req.script_name }),
|
||||
logs: vec![],
|
||||
stats: ExecStats::default(),
|
||||
})
|
||||
}
|
||||
|
||||
async fn execute_with_identity(
|
||||
&self,
|
||||
_identity: ScriptIdentity,
|
||||
source: &str,
|
||||
req: ExecRequest,
|
||||
timeout: StdDuration,
|
||||
) -> Result<ExecResponse, ExecError> {
|
||||
self.execute(source, req, timeout).await
|
||||
}
|
||||
}
|
||||
|
||||
struct NoopLogSink;
|
||||
#[async_trait]
|
||||
impl ExecutionLogSink for NoopLogSink {
|
||||
async fn record(&self, _log: picloud_shared::ExecutionLog) -> Result<(), LogSinkError> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn end_to_end_tick_executes_functions_and_completes() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let _claim_guard = lock_and_reset(&pool).await;
|
||||
// a -> b; both bound to real scripts (fn_a, fn_b) so name resolution works.
|
||||
let def = WorkflowDefinition {
|
||||
steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
|
||||
};
|
||||
let (app, wf) = seed_workflow(&pool, &def).await;
|
||||
for fn_name in ["fn_a", "fn_b"] {
|
||||
sqlx::query("INSERT INTO scripts (name, source, app_id) VALUES ($1, 'x', $2)")
|
||||
.bind(fn_name)
|
||||
.bind(app.into_inner())
|
||||
.execute(&pool)
|
||||
.await
|
||||
.expect("insert script");
|
||||
}
|
||||
let run = start_run(&pool, new_run(app, wf), &def)
|
||||
.await
|
||||
.expect("start");
|
||||
|
||||
let executor = Arc::new(FakeExecutor {
|
||||
calls: std::sync::Mutex::new(vec![]),
|
||||
});
|
||||
let orch = WorkflowOrchestrator {
|
||||
pool: pool.clone(),
|
||||
scripts: Arc::new(PostgresScriptRepository::new(pool.clone())),
|
||||
executor: executor.clone(),
|
||||
gate: Arc::new(ExecutionGate::new(8)),
|
||||
log_sink: Arc::new(NoopLogSink),
|
||||
config: WorkflowOrchestratorConfig::default(),
|
||||
};
|
||||
|
||||
// Drive ticks until the run is terminal (bounded).
|
||||
for _ in 0..50 {
|
||||
orch.tick_once().await;
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
if r.status.is_terminal() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let r = get_run(&pool, app, run).await.unwrap().unwrap();
|
||||
assert_eq!(r.status, RunStatus::Succeeded);
|
||||
let calls = executor.calls.lock().unwrap().clone();
|
||||
assert!(calls.contains(&"fn_a".to_string()));
|
||||
assert!(calls.contains(&"fn_b".to_string()));
|
||||
assert_eq!(
|
||||
r.output,
|
||||
Some(json!({ "a": { "ran": "fn_a" }, "b": { "ran": "fn_b" } }))
|
||||
);
|
||||
}
|
||||
Reference in New Issue
Block a user