`dispatcher_e2e` failed about one run in three, on a different test each time,
and never under `--test-threads=1`. Root cause, not timing:
Each test calls `build_app`, which spawns a REAL dispatcher, and they all shared
one database — so they shared one `outbox`. `OutboxRepo::claim_due` is
deliberately not app-scoped (in production one dispatcher serves the whole
instance, so claiming any due row is correct). Test A's dispatcher would
therefore claim test B's row, test A would finish, its server would be dropped
mid-dispatch, and the claim was stranded. Test B polled for its handler's side
effect until it timed out.
The harness was modelling something production never does: N independent
instances sharing one database. So the fix belongs in the harness. Weakening the
dispatcher (app-scoping `claim_due`) would be wrong, and shortening
`PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` to fit a 10s test window would make production
abandon the live claims of scripts that may legitimately run 300s.
`tests/common` now hands each test its own database. Replaying 73 migrations per
test is too slow, so the first caller builds a migrated TEMPLATE database once
and every test clones it (`CREATE DATABASE ... TEMPLATE` is a file copy), guarded
by a cluster-wide advisory lock because Postgres will not clone a template that
has an open connection. Names are derived from (suite, test), so a test reclaims
its own database on entry: a crashed run leaves nothing to clean up.
Only the five local `pool_or_skip` wrappers change — all 33 call sites are
untouched, and the skip-when-`DATABASE_URL`-is-unset contract is preserved.
(`#[sqlx::test]` already does this and is what `api.rs`/`authz.rs` use, but it
requires `DATABASE_URL` and would force `#[ignore]`, silently dropping these
tests from the default `cargo test --workspace` gate.)
This also stops the e2e tests leaving stranded claims behind in the shared dev
database, which is the likely source of the occasional lone journey failure.
Before: dispatcher_e2e red ~1 run in 3. After: workspace green twice over,
journeys 157/157.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Postgres-backed integration tests surfaced two assertion fixes while
verifying the audit branch:
* email_inbound::create_without_inbound_secret_is_rejected (added in the
H-B2 commit) asserted 400; TriggersApiError::Invalid maps to 422
(UNPROCESSABLE_ENTITY). Corrected.
* api::version_includes_public_base_url had two rotted assertions —
`schema` pinned at 6 (it's migrations::latest_version(), which had
climbed to 41 and is now 42 after 0042_secrets_envelope_version.sql)
and `sdk` pinned at "1.1" (the crate is at "1.10"). The test is
#[ignore]-gated so the rot went unnoticed in normal runs. Both pinned
to current reality. These were pre-existing failures, not caused by
the security changes.
Verified against a throwaway Postgres: schema_snapshot, api (53),
authz (28 of 29 — see below), dispatcher_e2e, email_inbound, invoke_e2e,
queue_e2e all green.
Known pre-existing failure (NOT fixed here, out of Tier 1+2 scope):
authz::deactivating_user_revokes_their_api_keys fails identically on the
pre-branch merge-base. It's the PrincipalCache revocation-lag Medium the
audit flagged as unfixed (resolve_principal caches by token-hash with no
eviction on deactivation; lag bounded by the 60s TTL). Deferred to a
Tier 3/4 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two gaps:
1. /api/v1/admin/apps/{id}/triggers/email accepted `inbound_secret: null`
and treated it as "this trigger is unsigned". The runtime then
skipped HMAC verification entirely. URL discovery (logs, ops
dashboards, bruteforce on the 122-bit TriggerId space) then fan-out-
amplified into the outbox for free — anonymous DoS via the
dispatcher.
2. Even signed triggers had no per-trigger rate limit on signature-
verify failures, so an attacker who knew the URL could pump unsigned
POSTs to force Argon2-equivalent work (HMAC verify + nonce-dedup +
stored-secret decrypt) at line rate.
Fix:
* triggers_api::create_email_trigger now requires a non-empty
inbound_secret. 400 on missing / empty / whitespace-only.
* email_inbound_api::receive_inbound_email returns 401 immediately when
the resolved target has no inbound_secret_encrypted column (pre-fix
rows). The previous `if let Some(ct), Some(nonce)` branch is gone.
* New `BadSignatureLimiter`: per-`(app_id, trigger_id)` sliding-window
bucket (10 fails / 60 s). On lockout the receiver returns 429 with
Retry-After instead of 401. record_failure runs on both the
"no-secret" 401 path and any verify_signature failure.
Test fallout:
* email_inbound integration tests that relied on None secret: removed
the now-impossible `unsigned_trigger_accepts_without_signature` test,
replaced with `create_without_inbound_secret_is_rejected` covering
null / "" / " "; updated `malformed_body_is_422` and
`cross_app_path_is_404` to pass + sign with a secret.
* Two new unit tests pin the limiter behavior (burst lockout + per-
trigger independence).
Audit refs: security_audit/08_dos_resource.md#h-3,
security_audit/09_external_integrations.md#f-ext-m-001.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes 4 High-severity audit findings:
- describe_event broken for HTTP-async payloads: was reading source/op
JSON fields that HttpDispatchPayload doesn't carry, producing empty
source on DL rows. from_wire("") then fell back to Kv on replay, the
dispatcher tried to resolve a non-existent trigger, and replay no-op'd
silently. Now branches on OutboxSourceKind: HTTP rows emit
"<method> <path>" + source="http"; Invoke rows emit "invoke_async" +
source="invoke"; TriggerEvent payloads keep top-level field extraction.
- HTTP outbound Authorization leaked across cross-origin redirects.
Manual redirect loop (Policy::none) reused header_map on every hop
and only scrubbed Content-Type for POST->GET. Now compares
url::Url::origin() across hops and strips Authorization,
Proxy-Authorization, Cookie when origin changes — matching reqwest's
default policy.
- F-S-010 inbound email HMAC had no replay protection. Signature was
computed over body only with no timestamp or nonce, so captured POSTs
were replayable indefinitely. Adds X-Picloud-Timestamp header bound
into the HMAC input (signed string is ts || "." || body), 300s
tolerance window, and a process-local LRU keyed on
(ts, sha256(body)) with 600s TTL to catch within-window replays.
Breaking change: webhook senders must include the timestamp header.
- expected_schema.txt re-blessed for migrations 0036-0039 (the previous
snapshot stopped at 0035 so CI would have gone red on first run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inbound email: a provider POSTs a normalized JSON message to
POST /api/v1/email-inbound/{app_id}/{trigger_id}; the public receiver
verifies the optional HMAC signature, builds a TriggerEvent::Email, and
enqueues an outbox row the dispatcher delivers like any async trigger.
Handlers see ctx.event.email = #{from,to,cc,subject,text,html,
received_at,message_id}.
- migration 0024: widen triggers.kind + outbox.source_kind CHECKs to
'email'; new email_trigger_details table.
- TriggerKind::Email, TriggerDetails::Email{has_inbound_secret},
OutboxSourceKind::Email, TriggerEvent::Email; dispatcher routes the
email row via the generic resolve_trigger path.
- Admin POST /apps/{id}/triggers/email (validate_trigger_target; module
+ cross-app rejection). inbound_secret is stored ENCRYPTED via the
master key (deviation from the brief's plaintext default; decrypted
per inbound request — see HANDBACK §7).
- Dashboard: email trigger form on the Triggers tab + webhook URL +
expected-payload help.
- 8 DB-gated e2e tests (202/401/404/422/cross-app/handler-fire) +
receiver unit tests (HMAC verify, secret round-trip, payload parse).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
