fabi/PiCloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: close the DB-suite hermeticity + vacuous-skip gaps

Browse Source 
Three related fixes from the test audit.

**The CLI journey fixture gets its own database.** It spawned a real picloud —
whose dispatcher/orchestrator claim loops are global by design (one instance owns
one database) — against the shared dev DB, so it could claim the manager-core
suites' outbox/workflow rows (the same class of bug already fixed for the e2e
suites, one binary over). It now clones one dedicated database per journey run
from the migrated template. test-support gains `named_test_db_url` (explicit
stable name) + a blocking wrapper for the sync `LazyLock` fixture. The one journey
that talks to Postgres directly (dead-letter injection) now uses the fixture's DB
URL, not the base DATABASE_URL, so it hits the database the server reads.

**workflow_orchestrator moves to per-test databases.** Its `claim_ready_step` is
global, so the old harness serialized every test behind a process-wide CLAIM_LOCK
AND ran `DELETE FROM workflow_runs` (unscoped — it wiped every app's runs) before
each one. A private database per test makes the global claim see only that test's
rows, so both the lock and the unscoped DELETE are deleted.

**DB-backed suites fail loud instead of skipping green.** ~15 manager-core suites
`return None` when DATABASE_URL is unset and report PASS — so in any environment
that lost its database the entire integration surface reports green while running
nothing (why the CI gap went unnoticed for so long). New
`picloud_test_support::abort_if_db_required` panics when `PICLOUD_REQUIRE_DB` is
set (CI now sets it) but DATABASE_URL is not, injected into each suite's skip
path. Local runs without the var still skip cleanly.

Mutation-verified: with PICLOUD_REQUIRE_DB=1 and DATABASE_URL unset, a suite
panics; without the var it skips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-15 19:39:13 +02:00
parent 2445074c87
commit 345db4a076
22 changed files with 131 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
crates/manager-core/tests/email_secret_aad.rs
View File

@@ -21,6 +21,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("email_secret_aad");
eprintln!("email_secret_aad: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/group_collection_isolation.rs
View File

@@ -39,6 +39,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("group_collection_isolation");
eprintln!("group_collection_isolation: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/group_queue.rs
View File

@@ -25,6 +25,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("group_queue");
eprintln!("group_queue: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/group_route_templates.rs
View File

@@ -24,6 +24,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("group_route_templates");
eprintln!("group_route_templates: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/group_suppression.rs
View File

@@ -24,6 +24,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("group_suppression");
eprintln!("group_suppression: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/group_trigger_templates.rs
View File

@@ -20,6 +20,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("group_trigger_templates");
eprintln!("group_trigger_templates: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/migration_queue_messages.rs
View File

@@ -14,6 +14,7 @@ use sqlx::PgPool;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("migration_queue_messages");
eprintln!("migration_queue_messages: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/projects_repo.rs
View File

@@ -13,6 +13,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("projects_repo");
eprintln!("projects_repo: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/queue_release.rs
View File

@@ -16,6 +16,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("queue_release");
eprintln!("queue_release: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/schema_snapshot.rs
View File

@@ -49,6 +49,7 @@ async fn schema_after_replay_matches_snapshot() {
// Skip cleanly when DATABASE_URL is unset so `cargo test --workspace`
// stays green without Postgres. CI sets it (postgres:15 service).
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("schema_snapshot");
eprintln!(
"schema_snapshot: DATABASE_URL unset — skipping. Set it (e.g. \
postgres://picloud:picloud@localhost:5432/picloud) to run this guardrail."

1
crates/manager-core/tests/sealed_templates.rs
View File

@@ -25,6 +25,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("sealed_templates");
eprintln!("sealed_templates: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/shared_topics.rs
View File

@@ -20,6 +20,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("shared_topics");
eprintln!("shared_topics: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/shared_triggers.rs
View File

@@ -18,6 +18,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("shared_triggers");
eprintln!("shared_triggers: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/stateful_templates.rs
View File

@@ -18,6 +18,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("stateful_templates");
eprintln!("stateful_templates: DATABASE_URL unset — skipping");
return None;
};

1
crates/manager-core/tests/template_suppression.rs
View File

@@ -25,6 +25,7 @@ use uuid::Uuid;
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
picloud_test_support::abort_if_db_required("template_suppression");
eprintln!("template_suppression: DATABASE_URL unset — skipping");
return None;
};

60
crates/manager-core/tests/workflow_orchestrator.rs
View File

@@ -13,7 +13,7 @@
#![allow(clippy::too_many_lines)]
use std::sync::{Arc, LazyLock};
use std::sync::Arc;
use async_trait::async_trait;
use picloud_executor_core::{ExecError, ExecRequest, ExecResponse, ExecStats};
@@ -34,47 +34,19 @@ use picloud_shared::workflow::{
use picloud_shared::{AppId, ExecutionLogSink, LogSinkError, WorkflowId};
use picloud_shared::{RequestId, ScriptId, WorkflowService};
use serde_json::{json, Value};
use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool;
use std::time::Duration as StdDuration;
use uuid::Uuid;
/// `claim_ready_step` is deliberately **global** (one production orchestrator
/// claims every app's ready steps). These tests share one DB, so a sibling
/// test's concurrent claim would steal steps from the run under assertion. This
/// process-wide lock serializes the claim-driving tests — each still creates
/// its own app/workflow/run, and each finishes with no `ready` steps, so once
/// serialized they don't interfere. (See the shared-DB test-harness note.)
static CLAIM_LOCK: LazyLock<tokio::sync::Mutex<()>> = LazyLock::new(|| tokio::sync::Mutex::new(()));
/// Acquire the serialization lock AND clear any leftover runs/steps so the
/// global `claim_ready_step`/`reclaim_stale_steps` see only this test's rows.
/// (The shared dev DB accumulates rows across runs; `workflow_runs` CASCADEs to
/// its steps.) Returns the held guard.
async fn lock_and_reset(pool: &PgPool) -> tokio::sync::MutexGuard<'static, ()> {
let guard = CLAIM_LOCK.lock().await;
sqlx::query("DELETE FROM workflow_runs")
.execute(pool)
.await
.expect("reset workflow_runs");
guard
}
/// A PRIVATE database per test. `claim_ready_step`/`reclaim_stale_steps` are
/// deliberately global (one production orchestrator claims every app's ready
/// steps), so when these tests shared one DB a sibling's concurrent claim would
/// steal the steps under assertion — the reason the old harness needed a
/// process-wide `CLAIM_LOCK` AND an unscoped `DELETE FROM workflow_runs` before
/// every test. A private database makes both unnecessary: the global claim only
/// ever sees this test's own rows.
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
eprintln!("workflow_orchestrator: DATABASE_URL unset — skipping");
return None;
};
let pool = PgPoolOptions::new()
.max_connections(4)
.connect(&url)
.await
.expect("connect to DATABASE_URL");
sqlx::migrate!("./migrations")
.run(&pool)
.await
.expect("apply migrations");
Some(pool)
picloud_test_support::test_pool("workflow_orchestrator").await
}
fn step(name: &str, function: &str, deps: &[&str]) -> WorkflowStepDef {
@@ -165,7 +137,6 @@ async fn seeds_roots_ready_and_completes_linear_chain() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// a -> b -> c
let def = WorkflowDefinition {
steps: vec![
@@ -205,7 +176,6 @@ async fn parallel_fan_out_and_fan_in() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// a -> {b, c} -> d (diamond).
let def = WorkflowDefinition {
steps: vec![
@@ -268,7 +238,6 @@ async fn retry_then_succeed() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let mut a = step("a", "fn_a", &[]);
a.retry = Some(WorkflowRetry {
max_attempts: 3,
@@ -305,7 +274,6 @@ async fn on_error_fail_fails_the_run() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// a (fail) -> b; a has no retry and on_error=fail → run fails, b never runs.
let def = WorkflowDefinition {
steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
@@ -337,7 +305,6 @@ async fn on_error_continue_run_succeeds() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let mut a = step("a", "fn_a", &[]);
a.on_error = OnError::Continue;
let def = WorkflowDefinition {
@@ -371,7 +338,6 @@ async fn double_complete_is_idempotent() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let def = WorkflowDefinition {
steps: vec![step("a", "fn_a", &[])],
};
@@ -403,7 +369,6 @@ async fn stale_lease_is_reclaimed() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let def = WorkflowDefinition {
steps: vec![step("a", "fn_a", &[])],
};
@@ -491,7 +456,6 @@ async fn end_to_end_tick_executes_functions_and_completes() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// a -> b; both bound to real scripts (fn_a, fn_b) so name resolution works.
let def = WorkflowDefinition {
steps: vec![step("a", "fn_a", &[]), step("b", "fn_b", &["a"])],
@@ -738,7 +702,6 @@ async fn when_false_skips_step() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// a -> b, but b runs only `when input.flag` (false here) → b is skipped.
let mut b = step("b", "fn_b", &["a"]);
b.when = Some("input.flag".into());
@@ -778,7 +741,6 @@ async fn step_output_flows_into_downstream_input() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
// b's input pulls a field out of a's output, type-preserved.
let mut b = step("b", "fn_b", &["a"]);
b.input = json!({ "v": "{{ steps.a.output.n }}", "label": "n={{ steps.a.output.n }}" });
@@ -810,7 +772,6 @@ async fn missing_input_ref_fails_the_step() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let mut b = step("b", "fn_b", &["a"]);
b.input = json!({ "v": "{{ steps.a.output.does_not_exist }}" });
let def = WorkflowDefinition {
@@ -848,7 +809,6 @@ async fn nested_sub_workflow_runs_and_output_flows_to_parent() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let app = seed_app(&pool).await;
// Child workflow: one function step producing { answer: 42 }.
@@ -905,7 +865,6 @@ async fn nesting_depth_ceiling_fails_the_run() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let app = seed_app(&pool).await;
// A workflow that nests into ITSELF — bounded only by the depth ceiling.
@@ -957,7 +916,6 @@ async fn workflow_service_start_seeds_a_run() {
let Some(pool) = pool_or_skip().await else {
return;
};
let _claim_guard = lock_and_reset(&pool).await;
let app = seed_app(&pool).await;
let def = WorkflowDefinition {
steps: vec![step("only", "fn_only", &[])],

4
crates/picloud-cli/Cargo.toml
View File

@@ -39,6 +39,10 @@ anyhow = "1"
assert_cmd = "2"
predicates = "3"
tempfile = "3"
# One private database per journey run (cloned from a migrated template), so the
# spawned picloud's global dispatcher/claim loops don't fight the manager-core
# suites over the shared dev DB. Also carries the require-DB gate.
picloud-test-support.workspace = true
reqwest = { workspace = true, features = ["json", "blocking"] }
libc = "0.2"
# Synchronous Postgres driver used by the dead-letters integration test

22
crates/picloud-cli/tests/common/mod.rs
View File

@@ -29,6 +29,11 @@ pub struct Fixture {
pub url: String,
pub admin_token: String,
pub admin_username: String,
/// The Postgres URL the spawned picloud is using — a per-journey-run cloned
/// database, NOT the base `DATABASE_URL`. A test that talks to the DB directly
/// (e.g. injecting a dead-letter row) must use THIS so it hits the same
/// database the server does.
pub database_url: String,
// Held in a Mutex so Drop can kill it without UB; we never re-enter.
child: Mutex<Option<Child>>,
}
@@ -46,8 +51,13 @@ impl Drop for Fixture {
static FIXTURE: LazyLock<Fixture> = LazyLock::new(init_fixture);
fn init_fixture() -> Fixture {
let database_url =
std::env::var("DATABASE_URL").expect("DATABASE_URL is required to spawn picloud");
// ONE database for the whole journey binary, cloned fresh from the migrated
// template. The spawned picloud runs the real dispatcher/orchestrator, whose
// claim loops are global by design (one instance owns one database) — sharing
// the dev DB with the manager-core suites let it claim their outbox/workflow
// rows. A dedicated DB matches production and ends that interference.
let database_url = picloud_test_support::named_test_db_url_blocking("pt_cli_journeys")
.expect("DATABASE_URL is required to spawn picloud");
let username = admin_username();
let password = admin_password();
let port = server::pick_free_port();
@@ -62,6 +72,7 @@ fn init_fixture() -> Fixture {
url,
admin_token: token,
admin_username: username,
database_url,
child: Mutex::new(Some(child)),
}
}
@@ -72,6 +83,13 @@ fn init_fixture() -> Fixture {
/// no-op outside the integration environment.
pub fn fixture_or_skip() -> Option<&'static Fixture> {
if std::env::var("DATABASE_URL").is_err() {
// CI sets PICLOUD_REQUIRE_DB so a lost/misconfigured database fails the
// build instead of silently skipping the entire journey suite.
assert!(
std::env::var("PICLOUD_REQUIRE_DB").is_err(),
"DATABASE_URL is unset but PICLOUD_REQUIRE_DB is set — refusing to skip \
the CLI journeys silently"
);
eprintln!("skipping: DATABASE_URL not set");
return None;
}

7
crates/picloud-cli/tests/dead_letters.rs
View File

@@ -103,9 +103,10 @@ fn replay_against_real_dl_row_succeeds() {
// minutes to drive. The replay path doesn't care how the row got
// there, only that it exists, belongs to this app, and is
// unresolved.
let db_url =
std::env::var("DATABASE_URL").expect("DATABASE_URL required for replay happy-path test");
let mut pg = PgClient::connect(&db_url, NoTls).expect("postgres connect");
// Inject into the SAME database the spawned picloud is using — the fixture's
// per-journey-run cloned DB, not the base DATABASE_URL. Using DATABASE_URL
// here would write the row into a different database than the server reads.
let mut pg = PgClient::connect(&fx.database_url, NoTls).expect("postgres connect");
let dl_id = Uuid::new_v4();
let app_uuid = Uuid::parse_str(&app_id).expect("app uuid");
let script_uuid = Uuid::parse_str(&script_id_str).expect("script uuid");

76
crates/test-support/src/lib.rs
View File

@@ -113,12 +113,82 @@ fn test_name() -> String {
/// template, cannot clone it) — a misconfigured integration environment should
/// fail loudly, not silently skip.
pub async fn test_db_url(suite: &str) -> Option<String> {
let name = db_name(suite, &test_name());
clone_template_into(&name).await
}
/// A freshly-cloned database with an EXPLICIT, stable name — for a fixture that
/// wants ONE database for a whole test binary rather than one per test (e.g. the
/// CLI journey harness, which spawns a single real picloud). Same drop-recreate
/// semantics, so a crashed run reclaims the name. `None` when `DATABASE_URL` is
/// unset.
///
/// # Panics
/// If `DATABASE_URL` is set but the database can't be created (see [`test_db_url`]).
pub async fn named_test_db_url(name: &str) -> Option<String> {
clone_template_into(name).await
}
/// Blocking wrapper over [`named_test_db_url`] for sync fixtures (e.g. a
/// `LazyLock` init that has no async runtime). Spins a private current-thread
/// runtime so callers need no tokio dependency of their own.
///
/// # Panics
/// As [`named_test_db_url`], plus if a runtime cannot be built.
pub fn named_test_db_url_blocking(name: &str) -> Option<String> {
if std::env::var("DATABASE_URL").is_err() {
// Cheap pre-check so we don't spin a runtime just to return None, and so
// the require-DB gate below fires without one either.
return skip_or_panic("cli-journeys");
}
tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("build a current-thread runtime for named_test_db_url_blocking")
.block_on(named_test_db_url(name))
}
/// Panic if `PICLOUD_REQUIRE_DB` is set while `DATABASE_URL` is unset. CI sets the
/// var, so a lost/misconfigured database surfaces as a RED build instead of a
/// suite that reports green while executing nothing (the vacuous-skip trap).
///
/// Call this from a suite that has its own `DATABASE_URL`-unset skip path (the
/// many `manager-core/tests/*` suites that connect to the shared dev DB directly)
/// so they honor the same CI gate as the test-support-based suites, without
/// having to be rewritten onto per-test databases.
///
/// ```ignore
/// let Ok(url) = std::env::var("DATABASE_URL") else {
/// picloud_test_support::abort_if_db_required("my_suite");
/// return None;
/// };
/// ```
pub fn abort_if_db_required(suite: &str) {
assert!(
std::env::var("PICLOUD_REQUIRE_DB").is_err(),
"{suite}: DATABASE_URL is unset but PICLOUD_REQUIRE_DB is set — refusing to \
skip DB-backed tests silently. Point DATABASE_URL at Postgres, or unset \
PICLOUD_REQUIRE_DB to allow skipping."
);
}
/// Decide what to do when `DATABASE_URL` is unset. Returns `None` (skip) in a
/// normal local run, but PANICS via [`abort_if_db_required`] when
/// `PICLOUD_REQUIRE_DB` is set.
fn skip_or_panic(suite: &str) -> Option<String> {
abort_if_db_required(suite);
eprintln!("{suite}: DATABASE_URL unset — skipping");
None
}
/// Shared clone core: ensure the migrated template exists, then drop+recreate
/// `name` as a clone of it. Returns the connection URL, or `None` when
/// `DATABASE_URL` is unset (respecting the require-DB gate).
async fn clone_template_into(name: &str) -> Option<String> {
let Ok(url) = std::env::var("DATABASE_URL") else {
eprintln!("{suite}: DATABASE_URL unset — skipping");
return None;
return skip_or_panic(name);
};
let (root, query) = split_url(&url).expect("DATABASE_URL must name a database");
let name = db_name(suite, &test_name());
let mut admin = PgConnection::connect(&url)
.await