`dispatcher_e2e` failed about one run in three, on a different test each time,
and never under `--test-threads=1`. Root cause, not timing:
Each test calls `build_app`, which spawns a REAL dispatcher, and they all shared
one database — so they shared one `outbox`. `OutboxRepo::claim_due` is
deliberately not app-scoped (in production one dispatcher serves the whole
instance, so claiming any due row is correct). Test A's dispatcher would
therefore claim test B's row, test A would finish, its server would be dropped
mid-dispatch, and the claim was stranded. Test B polled for its handler's side
effect until it timed out.
The harness was modelling something production never does: N independent
instances sharing one database. So the fix belongs in the harness. Weakening the
dispatcher (app-scoping `claim_due`) would be wrong, and shortening
`PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` to fit a 10s test window would make production
abandon the live claims of scripts that may legitimately run 300s.
`tests/common` now hands each test its own database. Replaying 73 migrations per
test is too slow, so the first caller builds a migrated TEMPLATE database once
and every test clones it (`CREATE DATABASE ... TEMPLATE` is a file copy), guarded
by a cluster-wide advisory lock because Postgres will not clone a template that
has an open connection. Names are derived from (suite, test), so a test reclaims
its own database on entry: a crashed run leaves nothing to clean up.
Only the five local `pool_or_skip` wrappers change — all 33 call sites are
untouched, and the skip-when-`DATABASE_URL`-is-unset contract is preserved.
(`#[sqlx::test]` already does this and is what `api.rs`/`authz.rs` use, but it
requires `DATABASE_URL` and would force `#[ignore]`, silently dropping these
tests from the default `cargo test --workspace` gate.)
This also stops the e2e tests leaving stranded claims behind in the shared dev
database, which is the likely source of the occasional lone journey failure.
Before: dispatcher_e2e red ~1 run in 3. After: workspace green twice over,
journeys 157/157.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dead_letter triggers have been registerable since v1.1.1 but their
handlers never fired: dispatcher::handle_failure wrote the dead_letters
row and stopped — list_matching_dead_letter had no production caller.
Any deploy v1.1.1–v1.1.6 with dead_letter triggers had silently
non-functional handlers.
The fix: after the dead-letter row is inserted on retry exhaustion, fan
out to matching dead_letter triggers (filtered by source / originating
trigger_id / script_id) and enqueue one outbox row per match carrying a
real-shape TriggerEvent::DeadLetter (the §6 brief field names were stale
— used the actual variant: dead_letter_id, original: Box<TriggerEvent>,
attempts, last_error, trigger_id, script_id, first/last_attempt_at).
The recursion-stop (a handler's own failure isn't re-dead-lettered)
is upheld by the existing is_dead_letter_handler short-circuit.
Tests (DB-gated): handler actually fires with the nested original event;
existing row-create test now also asserts handler-fire; source_filter
excludes non-matching; failing handler does not recurse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Encrypted per-app secrets, reachable from scripts as
secrets::{get,set,delete,list}(name) and managed from the dashboard
Secrets tab. Values are AES-256-GCM-sealed with the process master key
(picloud_shared::crypto) before they touch Postgres; the repo only ever
sees ciphertext + nonce. JSON round-trip preserves Rhai types.
- migration 0023_secrets.sql (PRIMARY KEY (app_id, name)).
- SecretsService trait (picloud-shared) + SecretsServiceImpl + repo
(manager-core), wired into the Services bundle and Rhai engine.
- Capability::AppSecretsRead/Write (→ script:read / script:write); no
new Scope variants (seven-scope commitment).
- Admin API GET/POST/DELETE /apps/{id}/secrets (list returns names +
updated_at, never values).
- build_app now takes a MasterKey, sourced from PICLOUD_SECRET_KEY in
main.rs; test callers pass a fixed test key.
- 64 KB value cap (PICLOUD_SECRET_MAX_VALUE_BYTES); no ServiceEvent
emission (secret writes don't fire triggers, by design).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
