test: close the DB-suite hermeticity + vacuous-skip gaps

Three related fixes from the test audit.

**The CLI journey fixture gets its own database.** It spawned a real picloud —
whose dispatcher/orchestrator claim loops are global by design (one instance owns
one database) — against the shared dev DB, so it could claim the manager-core
suites' outbox/workflow rows (the same class of bug already fixed for the e2e
suites, one binary over). It now clones one dedicated database per journey run
from the migrated template. test-support gains `named_test_db_url` (explicit
stable name) + a blocking wrapper for the sync `LazyLock` fixture. The one journey
that talks to Postgres directly (dead-letter injection) now uses the fixture's DB
URL, not the base DATABASE_URL, so it hits the database the server reads.

**workflow_orchestrator moves to per-test databases.** Its `claim_ready_step` is
global, so the old harness serialized every test behind a process-wide CLAIM_LOCK
AND ran `DELETE FROM workflow_runs` (unscoped — it wiped every app's runs) before
each one. A private database per test makes the global claim see only that test's
rows, so both the lock and the unscoped DELETE are deleted.

**DB-backed suites fail loud instead of skipping green.** ~15 manager-core suites
`return None` when DATABASE_URL is unset and report PASS — so in any environment
that lost its database the entire integration surface reports green while running
nothing (why the CI gap went unnoticed for so long). New
`picloud_test_support::abort_if_db_required` panics when `PICLOUD_REQUIRE_DB` is
set (CI now sets it) but DATABASE_URL is not, injected into each suite's skip
path. Local runs without the var still skip cleanly.

Mutation-verified: with PICLOUD_REQUIRE_DB=1 and DATABASE_URL unset, a suite
panics; without the var it skips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/workflows/ci.yml

@@ -10,6 +10,11 @@ env:
   # Matches what docker-compose produces locally; the schema-snapshot
   # guardrail and any other DB-backed tests run against this service.
   DATABASE_URL: postgres://picloud:picloud@localhost:5432/picloud
+  # The DB-backed suites skip themselves when DATABASE_URL is unset. In CI it is
+  # always set, so this makes that skip a hard error: if the database ever goes
+  # missing (a broken service container, a lost env), the build goes RED instead
+  # of green-but-empty. Honored by picloud_test_support::abort_if_db_required.
+  PICLOUD_REQUIRE_DB: "1"
 
 jobs:
   rust:

Cargo.lock

@@ -1965,6 +1965,7 @@ dependencies = [
  "libc",
  "percent-encoding",
  "picloud-shared",
+ "picloud-test-support",
  "postgres",
  "predicates",
  "reqwest",

crates/manager-core/tests/email_secret_aad.rs

@@ -21,6 +21,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("email_secret_aad");
         eprintln!("email_secret_aad: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/group_collection_isolation.rs

@@ -39,6 +39,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("group_collection_isolation");
         eprintln!("group_collection_isolation: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/group_queue.rs

@@ -25,6 +25,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("group_queue");
         eprintln!("group_queue: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/group_route_templates.rs

@@ -24,6 +24,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("group_route_templates");
         eprintln!("group_route_templates: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/group_suppression.rs

@@ -24,6 +24,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("group_suppression");
         eprintln!("group_suppression: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/group_trigger_templates.rs

@@ -20,6 +20,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("group_trigger_templates");
         eprintln!("group_trigger_templates: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/migration_queue_messages.rs

@@ -14,6 +14,7 @@ use sqlx::PgPool;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("migration_queue_messages");
         eprintln!("migration_queue_messages: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/projects_repo.rs

@@ -13,6 +13,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("projects_repo");
         eprintln!("projects_repo: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/queue_release.rs

@@ -16,6 +16,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("queue_release");
         eprintln!("queue_release: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/schema_snapshot.rs

@@ -49,6 +49,7 @@ async fn schema_after_replay_matches_snapshot() {
     // Skip cleanly when DATABASE_URL is unset so `cargo test --workspace`
     // stays green without Postgres. CI sets it (postgres:15 service).
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("schema_snapshot");
         eprintln!(
             "schema_snapshot: DATABASE_URL unset — skipping. Set it (e.g. \
              postgres://picloud:picloud@localhost:5432/picloud) to run this guardrail."

crates/manager-core/tests/sealed_templates.rs

@@ -25,6 +25,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("sealed_templates");
         eprintln!("sealed_templates: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/shared_topics.rs

@@ -20,6 +20,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("shared_topics");
         eprintln!("shared_topics: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/shared_triggers.rs

@@ -18,6 +18,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("shared_triggers");
         eprintln!("shared_triggers: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/stateful_templates.rs

@@ -18,6 +18,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("stateful_templates");
         eprintln!("stateful_templates: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/template_suppression.rs

@@ -25,6 +25,7 @@ use uuid::Uuid;
 
 async fn pool_or_skip() -> Option<PgPool> {
     let Ok(url) = std::env::var("DATABASE_URL") else {
+        picloud_test_support::abort_if_db_required("template_suppression");
         eprintln!("template_suppression: DATABASE_URL unset — skipping");
         return None;
     };

crates/manager-core/tests/workflow_orchestrator.rs

@@ -13,7 +13,7 @@
 
 #![allow(clippy::too_many_lines)]
 
-use std::sync::{Arc, LazyLock};
+use std::sync::Arc;
 
 use async_trait::async_trait;
 use picloud_executor_core::{ExecError, ExecRequest, ExecResponse, ExecStats};
@@ -34,47 +34,19 @@ use picloud_shared::workflow::{
 use picloud_shared::{AppId, ExecutionLogSink, LogSinkError, WorkflowId};
 use picloud_shared::{RequestId, ScriptId, WorkflowService};
 use serde_json::{json, Value};
-use sqlx::postgres::PgPoolOptions;
 use sqlx::PgPool;
 use std::time::Duration as StdDuration;
 use uuid::Uuid;
 
-/// `claim_ready_step` is deliberately **global** (one production orchestrator
+/// A PRIVATE database per test. `claim_ready_step`/`reclaim_stale_steps` are
-/// claims every app's ready steps). These tests share one DB, so a sibling
+/// deliberately global (one production orchestrator claims every app's ready
-/// test's concurrent claim would steal steps from the run under assertion. This
+/// steps), so when these tests shared one DB a sibling's concurrent claim would
-/// process-wide lock serializes the claim-driving tests — each still creates
+/// steal the steps under assertion — the reason the old harness needed a
-/// its own app/workflow/run, and each finishes with no `ready` steps, so once
+/// process-wide `CLAIM_LOCK` AND an unscoped `DELETE FROM workflow_runs` before
-/// serialized they don't interfere. (See the shared-DB test-harness note.)
+/// every test. A private database makes both unnecessary: the global claim only
-static CLAIM_LOCK: LazyLock<tokio::sync::Mutex<()>> = LazyLock::new(|| tokio::sync::Mutex::new(()));
+/// ever sees this test's own rows.
-
-/// Acquire the serialization lock AND clear any leftover runs/steps so the
-/// global `claim_ready_step`/`reclaim_stale_steps` see only this test's rows.
-/// (The shared dev DB accumulates rows across runs; `workflow_runs` CASCADEs to
-/// its steps.) Returns the held guard.
-async fn lock_and_reset(pool: &PgPool) -> tokio::sync::MutexGuard<'static, ()> {
-    let guard = CLAIM_LOCK.lock().await;
-    sqlx::query("DELETE FROM workflow_runs")
-        .execute(pool)
-        .await
-        .expect("reset workflow_runs");
-    guard
-}
-
 async fn pool_or_skip() -> Option<PgPool> {
-    let Ok(url) = std::env::var("DATABASE_URL") else {
+    picloud_test_support::test_pool("workflow_orchestrator").await
-        eprintln!("workflow_orchestrator: DATABASE_URL unset — skipping");
-        return None;
-    };
-    let pool = PgPoolOptions::new()
-        .max_connections(4)
-        .connect(&url)
-        .await
-        .expect("connect to DATABASE_URL");
-    sqlx::migrate!("./migrations")
-        .run(&pool)
-        .await
-        .expect("apply migrations");
-    Some(pool)
 }
 
 fn step(name: &str, function: &str, deps: &[&str]) -> WorkflowStepDef {
@@ -165,7 +137,6 @@ async fn seeds_roots_ready_and_completes_linear_chain() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // a -> b -> c
     let def = WorkflowDefinition {
         steps: vec![
@@ -205,7 +176,6 @@ async fn parallel_fan_out_and_fan_in() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // a -> {b, c} -> d (diamond).
     let def = WorkflowDefinition {
         steps: vec![
@@ -268,7 +238,6 @@ async fn retry_then_succeed() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let mut a = step("a", "fn_a", &[]);
     a.retry = Some(WorkflowRetry {
         max_attempts: 3,
@@ -305,7 +274,6 @@ async fn on_error_fail_fails_the_run() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // a (fail) -> b; a has no retry and on_error=fail → run fails, b never runs.
     let def = WorkflowDefinition {
         steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
@@ -337,7 +305,6 @@ async fn on_error_continue_run_succeeds() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let mut a = step("a", "fn_a", &[]);
     a.on_error = OnError::Continue;
     let def = WorkflowDefinition {
@@ -371,7 +338,6 @@ async fn double_complete_is_idempotent() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let def = WorkflowDefinition {
         steps: vec![step("a", "fn_a", &[])],
     };
@@ -403,7 +369,6 @@ async fn stale_lease_is_reclaimed() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let def = WorkflowDefinition {
         steps: vec![step("a", "fn_a", &[])],
     };
@@ -491,7 +456,6 @@ async fn end_to_end_tick_executes_functions_and_completes() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // a -> b; both bound to real scripts (fn_a, fn_b) so name resolution works.
     let def = WorkflowDefinition {
         steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
@@ -738,7 +702,6 @@ async fn when_false_skips_step() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // a -> b, but b runs only `when input.flag` (false here) → b is skipped.
     let mut b = step("b", "fn_b", &["a"]);
     b.when = Some("input.flag".into());
@@ -778,7 +741,6 @@ async fn step_output_flows_into_downstream_input() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     // b's input pulls a field out of a's output, type-preserved.
     let mut b = step("b", "fn_b", &["a"]);
     b.input = json!({ "v": "{{ steps.a.output.n }}", "label": "n={{ steps.a.output.n }}" });
@@ -810,7 +772,6 @@ async fn missing_input_ref_fails_the_step() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let mut b = step("b", "fn_b", &["a"]);
     b.input = json!({ "v": "{{ steps.a.output.does_not_exist }}" });
     let def = WorkflowDefinition {
@@ -848,7 +809,6 @@ async fn nested_sub_workflow_runs_and_output_flows_to_parent() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let app = seed_app(&pool).await;
 
     // Child workflow: one function step producing { answer: 42 }.
@@ -905,7 +865,6 @@ async fn nesting_depth_ceiling_fails_the_run() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let app = seed_app(&pool).await;
 
     // A workflow that nests into ITSELF — bounded only by the depth ceiling.
@@ -957,7 +916,6 @@ async fn workflow_service_start_seeds_a_run() {
     let Some(pool) = pool_or_skip().await else {
         return;
     };
-    let _claim_guard = lock_and_reset(&pool).await;
     let app = seed_app(&pool).await;
     let def = WorkflowDefinition {
         steps: vec![step("only", "fn_only", &[])],

crates/picloud-cli/Cargo.toml

@@ -39,6 +39,10 @@ anyhow = "1"
 assert_cmd = "2"
 predicates = "3"
 tempfile = "3"
+# One private database per journey run (cloned from a migrated template), so the
+# spawned picloud's global dispatcher/claim loops don't fight the manager-core
+# suites over the shared dev DB. Also carries the require-DB gate.
+picloud-test-support.workspace = true
 reqwest = { workspace = true, features = ["json", "blocking"] }
 libc = "0.2"
 # Synchronous Postgres driver used by the dead-letters integration test

crates/picloud-cli/tests/common/mod.rs

@@ -29,6 +29,11 @@ pub struct Fixture {
     pub url: String,
     pub admin_token: String,
     pub admin_username: String,
+    /// The Postgres URL the spawned picloud is using — a per-journey-run cloned
+    /// database, NOT the base `DATABASE_URL`. A test that talks to the DB directly
+    /// (e.g. injecting a dead-letter row) must use THIS so it hits the same
+    /// database the server does.
+    pub database_url: String,
     // Held in a Mutex so Drop can kill it without UB; we never re-enter.
     child: Mutex<Option<Child>>,
 }
@@ -46,8 +51,13 @@ impl Drop for Fixture {
 static FIXTURE: LazyLock<Fixture> = LazyLock::new(init_fixture);
 
 fn init_fixture() -> Fixture {
-    let database_url =
+    // ONE database for the whole journey binary, cloned fresh from the migrated
-        std::env::var("DATABASE_URL").expect("DATABASE_URL is required to spawn picloud");
+    // template. The spawned picloud runs the real dispatcher/orchestrator, whose
+    // claim loops are global by design (one instance owns one database) — sharing
+    // the dev DB with the manager-core suites let it claim their outbox/workflow
+    // rows. A dedicated DB matches production and ends that interference.
+    let database_url = picloud_test_support::named_test_db_url_blocking("pt_cli_journeys")
+        .expect("DATABASE_URL is required to spawn picloud");
     let username = admin_username();
     let password = admin_password();
     let port = server::pick_free_port();
@@ -62,6 +72,7 @@ fn init_fixture() -> Fixture {
         url,
         admin_token: token,
         admin_username: username,
+        database_url,
         child: Mutex::new(Some(child)),
     }
 }
@@ -72,6 +83,13 @@ fn init_fixture() -> Fixture {
 /// no-op outside the integration environment.
 pub fn fixture_or_skip() -> Option<&'static Fixture> {
     if std::env::var("DATABASE_URL").is_err() {
+        // CI sets PICLOUD_REQUIRE_DB so a lost/misconfigured database fails the
+        // build instead of silently skipping the entire journey suite.
+        assert!(
+            std::env::var("PICLOUD_REQUIRE_DB").is_err(),
+            "DATABASE_URL is unset but PICLOUD_REQUIRE_DB is set — refusing to skip \
+             the CLI journeys silently"
+        );
         eprintln!("skipping: DATABASE_URL not set");
         return None;
     }

crates/picloud-cli/tests/dead_letters.rs

@@ -103,9 +103,10 @@ fn replay_against_real_dl_row_succeeds() {
     // minutes to drive. The replay path doesn't care how the row got
     // there, only that it exists, belongs to this app, and is
     // unresolved.
-    let db_url =
+    // Inject into the SAME database the spawned picloud is using — the fixture's
-        std::env::var("DATABASE_URL").expect("DATABASE_URL required for replay happy-path test");
+    // per-journey-run cloned DB, not the base DATABASE_URL. Using DATABASE_URL
-    let mut pg = PgClient::connect(&db_url, NoTls).expect("postgres connect");
+    // here would write the row into a different database than the server reads.
+    let mut pg = PgClient::connect(&fx.database_url, NoTls).expect("postgres connect");
     let dl_id = Uuid::new_v4();
     let app_uuid = Uuid::parse_str(&app_id).expect("app uuid");
     let script_uuid = Uuid::parse_str(&script_id_str).expect("script uuid");

crates/test-support/src/lib.rs

@@ -113,12 +113,82 @@ fn test_name() -> String {
 /// template, cannot clone it) — a misconfigured integration environment should
 /// fail loudly, not silently skip.
 pub async fn test_db_url(suite: &str) -> Option<String> {
-    let Ok(url) = std::env::var("DATABASE_URL") else {
+    let name = db_name(suite, &test_name());
+    clone_template_into(&name).await
+}
+
+/// A freshly-cloned database with an EXPLICIT, stable name — for a fixture that
+/// wants ONE database for a whole test binary rather than one per test (e.g. the
+/// CLI journey harness, which spawns a single real picloud). Same drop-recreate
+/// semantics, so a crashed run reclaims the name. `None` when `DATABASE_URL` is
+/// unset.
+///
+/// # Panics
+/// If `DATABASE_URL` is set but the database can't be created (see [`test_db_url`]).
+pub async fn named_test_db_url(name: &str) -> Option<String> {
+    clone_template_into(name).await
+}
+
+/// Blocking wrapper over [`named_test_db_url`] for sync fixtures (e.g. a
+/// `LazyLock` init that has no async runtime). Spins a private current-thread
+/// runtime so callers need no tokio dependency of their own.
+///
+/// # Panics
+/// As [`named_test_db_url`], plus if a runtime cannot be built.
+pub fn named_test_db_url_blocking(name: &str) -> Option<String> {
+    if std::env::var("DATABASE_URL").is_err() {
+        // Cheap pre-check so we don't spin a runtime just to return None, and so
+        // the require-DB gate below fires without one either.
+        return skip_or_panic("cli-journeys");
+    }
+    tokio::runtime::Builder::new_current_thread()
+        .enable_all()
+        .build()
+        .expect("build a current-thread runtime for named_test_db_url_blocking")
+        .block_on(named_test_db_url(name))
+}
+
+/// Panic if `PICLOUD_REQUIRE_DB` is set while `DATABASE_URL` is unset. CI sets the
+/// var, so a lost/misconfigured database surfaces as a RED build instead of a
+/// suite that reports green while executing nothing (the vacuous-skip trap).
+///
+/// Call this from a suite that has its own `DATABASE_URL`-unset skip path (the
+/// many `manager-core/tests/*` suites that connect to the shared dev DB directly)
+/// so they honor the same CI gate as the test-support-based suites, without
+/// having to be rewritten onto per-test databases.
+///
+/// ```ignore
+/// let Ok(url) = std::env::var("DATABASE_URL") else {
+///     picloud_test_support::abort_if_db_required("my_suite");
+///     return None;
+/// };
+/// ```
+pub fn abort_if_db_required(suite: &str) {
+    assert!(
+        std::env::var("PICLOUD_REQUIRE_DB").is_err(),
+        "{suite}: DATABASE_URL is unset but PICLOUD_REQUIRE_DB is set — refusing to \
+         skip DB-backed tests silently. Point DATABASE_URL at Postgres, or unset \
+         PICLOUD_REQUIRE_DB to allow skipping."
+    );
+}
+
+/// Decide what to do when `DATABASE_URL` is unset. Returns `None` (skip) in a
+/// normal local run, but PANICS via [`abort_if_db_required`] when
+/// `PICLOUD_REQUIRE_DB` is set.
+fn skip_or_panic(suite: &str) -> Option<String> {
+    abort_if_db_required(suite);
     eprintln!("{suite}: DATABASE_URL unset — skipping");
-        return None;
+    None
+}
+
+/// Shared clone core: ensure the migrated template exists, then drop+recreate
+/// `name` as a clone of it. Returns the connection URL, or `None` when
+/// `DATABASE_URL` is unset (respecting the require-DB gate).
+async fn clone_template_into(name: &str) -> Option<String> {
+    let Ok(url) = std::env::var("DATABASE_URL") else {
+        return skip_or_panic(name);
     };
     let (root, query) = split_url(&url).expect("DATABASE_URL must name a database");
-    let name = db_name(suite, &test_name());
 
     let mut admin = PgConnection::connect(&url)
         .await