Turning CI on surfaced `atomic_write` as intermittently failing under load —
self-inflicted. Its fault injection is `CREATE TRIGGER ... ON outbox`, which takes
an ACCESS EXCLUSIVE lock on a table every other suite is concurrently inserting
into. The header even claimed it "cannot affect any test running in parallel";
that was wrong — scoping the trigger by app_id bounds which ROWS it rejects, not
the table lock installing it takes. Shipping that alongside "run the DB tests in
CI" would have poisoned the signal.
The fix is the one already used for the e2e suites: a private database per test.
That logic (build a migrated template once, clone it per test via
`CREATE DATABASE ... TEMPLATE`) was duplicated in `picloud/tests/common`, and this
would have been a third copy — so it moves into a shared `picloud-test-support`
dev-dependency crate, and `picloud/tests/common` now re-exports it. The invariant
it encodes: manager-core's claim loops are global by design, so a test must not
share a database with anything that runs them OR with anything that does DDL.
Moved onto it:
- `atomic_write` — the DDL fault injection above; its per-test cleanup is now
deleted (the database is thrown away).
- `outbox_reclaim` — asserts `reclaim_stale_claims` returns exactly 0, a count
over the WHOLE outbox table. On the shared DB any stale row from any other suite
(or a killed picloud that died holding a claim) would fail it permanently, until
someone cleared the table by hand.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI ran `cargo test --workspace` with no `--include-ignored`, so it executed 927
tests and skipped 237 — every DB-backed integration test is
`#[ignore = "needs DATABASE_URL..."]`, which covers ALL of api.rs, ALL of
authz.rs, and the entire CLI journey suite. The isolation and RBAC tests existed
but never ran (AUDIT.md F-Q-014, logged and never remediated). CI provides
Postgres, so it can run them.
Three things had to be right for that to go green:
- **`--all-targets`, not a bare workspace run.** `-- --include-ignored` un-ignores
not just `#[ignore]` tests but also ` ```ignore ` DOCTESTS, which are
illustrative pseudocode that does not compile. `--all-targets` runs lib/bins/
integration tests but excludes doctests (the same reason clippy uses it); a
separate `--doc` step runs the doctests without the flag. Structural, so a
future pseudocode doctest can't silently break CI either.
- **The CLI journeys are their own step.** They spawn a real picloud whose
dispatcher/orchestrator claim loops are global by design; run concurrently with
the manager-core suites on the shared database they would claim those suites'
outbox and workflow rows. Sequential steps keep the live server off the DB
while the other suites use it. The step also rebuilds `-p picloud` first (the
harness execs the prebuilt binary) and sets the dev-mode env the server needs.
- **A higher `max_connections`.** `#[sqlx::test]` pools are lazy, but mass-parallel
test startup briefly opens many at once (each test creates its own throwaway
database); on a many-core box that transient spike exceeded the default 100 and
Postgres answered "sorry, too many clients already". Steady-state peak is only
~26; 500 absorbs the spike with room to spare. Serving the app needs nothing
like this many. (This is the local compose ceiling; a small CI runner's default
100 has ample headroom for its lower parallelism.)
Also fixes the test the CI gap had let rot: api.rs asserted `v["schema"] == 66`
with a hand-bump comment, and since nothing ran it, it sat broken from migration
0066 to 0073. It now asserts `/version` surfaces the live constants
(`migrations::latest_version()`, `SDK_VERSION`) — the wiring — while value drift
stays caught by schema_snapshot + check-versioning. A constant hand-synced to
another constant is a chore, not a test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`dispatcher_e2e` failed about one run in three, on a different test each time,
and never under `--test-threads=1`. Root cause, not timing:
Each test calls `build_app`, which spawns a REAL dispatcher, and they all shared
one database — so they shared one `outbox`. `OutboxRepo::claim_due` is
deliberately not app-scoped (in production one dispatcher serves the whole
instance, so claiming any due row is correct). Test A's dispatcher would
therefore claim test B's row, test A would finish, its server would be dropped
mid-dispatch, and the claim was stranded. Test B polled for its handler's side
effect until it timed out.
The harness was modelling something production never does: N independent
instances sharing one database. So the fix belongs in the harness. Weakening the
dispatcher (app-scoping `claim_due`) would be wrong, and shortening
`PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` to fit a 10s test window would make production
abandon the live claims of scripts that may legitimately run 300s.
`tests/common` now hands each test its own database. Replaying 73 migrations per
test is too slow, so the first caller builds a migrated TEMPLATE database once
and every test clones it (`CREATE DATABASE ... TEMPLATE` is a file copy), guarded
by a cluster-wide advisory lock because Postgres will not clone a template that
has an open connection. Names are derived from (suite, test), so a test reclaims
its own database on entry: a crashed run leaves nothing to clean up.
Only the five local `pool_or_skip` wrappers change — all 33 call sites are
untouched, and the skip-when-`DATABASE_URL`-is-unset contract is preserved.
(`#[sqlx::test]` already does this and is what `api.rs`/`authz.rs` use, but it
requires `DATABASE_URL` and would force `#[ignore]`, silently dropping these
tests from the default `cargo test --workspace` gate.)
This also stops the e2e tests leaving stranded claims behind in the shared dev
database, which is the likely source of the occasional lone journey failure.
Before: dispatcher_e2e red ~1 run in 3. After: workspace green twice over,
journeys 157/157.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit #6 and #8 for the last of the three stores, plus a bypass found on
the way.
**#6.** create/update/delete wrote the metadata row, then emitted
best-effort, so an outbox failure left a committed file whose trigger never
fired. `atomic_write::FilesWriter` commits the metadata row and the fan-out
together.
Files are the one store where the ordering is subtle, because the BYTES live
on disk and cannot join a transaction:
* create/update — blob first, then commit metadata + fan-out. A rollback
unlinks the blob. (A crash at that exact point still orphans it; that
hazard predates this change — the repo already wrote the blob and then
inserted the row in a separate, failable statement — and the orphan is
inert, referenced by nothing.)
* delete — commit the metadata removal + fan-out FIRST, then unlink. The
reverse order would destroy the bytes of a row that a rollback keeps,
leaving a file that can never be read.
**#8.** `GroupFilesService::create` read `total_bytes` on one connection and
wrote on another, so concurrent uploads each saw the same pre-write total and
together overshot the ceiling. This is the worst instance of the race in the
codebase: the ceiling is DISK (10 GiB by default) and one file may be 100 MB,
so a racing fleet overshoots by gigabytes. `PostgresGroupFilesWriter` takes
the per-group advisory lock (on its own `files` key) across the check and the
write.
**The bypass.** `GroupFilesService::update` checked NO quota at all — so a
1-byte file could be updated to a 100 MB one without the ceiling ever being
consulted, repeatedly, for unbounded disk. It now checks the projected total
(the replaced file's bytes subtracted in SQL, so a same-size-or-smaller
update near the cap still goes through).
Also drive-by: `queue_e2e` asserted the ack the instant the marker appeared,
but the marker is written DURING the handler and the ack happens after it
returns — a zero-tolerance race. It polls now. (This does not fix the
suite's flakiness, which reproduces on the pre-pass commit too.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First commit of §7 multi-repo ownership. A 'project' is a repo-root that
declaratively manages a slice of the shared group tree; each group node is
owned by exactly one project (single-owner-per-node). This lands the registry
+ shared types and wires the inert 0047 `owner_project` seam — no behavior
change yet (claim logic is the next commit).
- migration 0066: `projects` table (UUID pk, unique slug, name, created_by →
admin_users ON DELETE SET NULL); `groups.owner_project` gains its FK →
projects(id) ON DELETE SET NULL (un-claim, never cascade-destroy a tree) +
an index.
- shared: `ProjectId` (id_type! macro) + `Project` struct; `Group` gains
`owner_project: Option<ProjectId>`.
- group_repo: GROUP_COLS + the ancestors recursive-CTE term now carry
`owner_project`, so `ancestors()` surfaces ownership for the nearest-claimed
fold; GroupRow + From updated.
- schema snapshot re-blessed; /version schema assertion 65 → 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #[ignore]-gated version_includes_public_base_url pinned schema=42 but
migrations accreted to 63 across the v1.2 template track (…0060–0063), so
it failed under `--include-ignored`. Re-pin to current reality per the
test's own intent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ExecError::Runtime strings (filesystem paths, "blocked by SSRF policy:
link-local" cloud-metadata reconnaissance, pool-exhaustion timing,
panic fragments, and attacker-thrown strings) were returned verbatim in
the public /api/v1/execute + user-route 502 bodies. This handler only
serves anonymous data-plane callers (the authenticated script-test path
is in manager-core and keeps verbose errors), so log the full text
server-side under a correlation id and return a stable generic message
plus the id for operator grep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PrincipalCache (token-hash -> resolved Principal, 60s TTL) had no
eviction hook, so deactivation / password change / API-key revocation
flipped the DB rows but the cache kept serving the stale principal for
up to the TTL window. Closes the audit's PrincipalCache revocation-lag
Medium and greens authz::deactivating_user_revokes_their_api_keys.
- PrincipalCache::evict_user(user_id) + evict_token(hash).
- One shared cache Arc threaded into AuthState / AdminsState /
ApiKeysState (a per-state cache would let the middleware's own copy
keep authenticating a revoked principal).
- Evict on: deactivation, password change (both evict_user), logout
(evict_token, precise), single API-key delete (evict_user).
- H-E1 note: DB-write invalidation stays best-effort by design (a blip
must not undo the rotation); the cache evict is what makes it take
effect on the next request.
- Renamed bearer_and_cookie_produce_same_principal ->
session_token_and_api_key_produce_same_principal (cookie auth was
removed in C-1; the test never tested cookies).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Postgres-backed integration tests surfaced two assertion fixes while
verifying the audit branch:
* email_inbound::create_without_inbound_secret_is_rejected (added in the
H-B2 commit) asserted 400; TriggersApiError::Invalid maps to 422
(UNPROCESSABLE_ENTITY). Corrected.
* api::version_includes_public_base_url had two rotted assertions —
`schema` pinned at 6 (it's migrations::latest_version(), which had
climbed to 41 and is now 42 after 0042_secrets_envelope_version.sql)
and `sdk` pinned at "1.1" (the crate is at "1.10"). The test is
#[ignore]-gated so the rot went unnoticed in normal runs. Both pinned
to current reality. These were pre-existing failures, not caused by
the security changes.
Verified against a throwaway Postgres: schema_snapshot, api (53),
authz (28 of 29 — see below), dispatcher_e2e, email_inbound, invoke_e2e,
queue_e2e all green.
Known pre-existing failure (NOT fixed here, out of Tier 1+2 scope):
authz::deactivating_user_revokes_their_api_keys fails identically on the
pre-branch merge-base. It's the PrincipalCache revocation-lag Medium the
audit flagged as unfixed (resolve_principal caches by token-hash with no
eviction on deactivation; lag bounded by the 60s TTL). Deferred to a
Tier 3/4 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two gaps:
1. /api/v1/admin/apps/{id}/triggers/email accepted `inbound_secret: null`
and treated it as "this trigger is unsigned". The runtime then
skipped HMAC verification entirely. URL discovery (logs, ops
dashboards, bruteforce on the 122-bit TriggerId space) then fan-out-
amplified into the outbox for free — anonymous DoS via the
dispatcher.
2. Even signed triggers had no per-trigger rate limit on signature-
verify failures, so an attacker who knew the URL could pump unsigned
POSTs to force Argon2-equivalent work (HMAC verify + nonce-dedup +
stored-secret decrypt) at line rate.
Fix:
* triggers_api::create_email_trigger now requires a non-empty
inbound_secret. 400 on missing / empty / whitespace-only.
* email_inbound_api::receive_inbound_email returns 401 immediately when
the resolved target has no inbound_secret_encrypted column (pre-fix
rows). The previous `if let Some(ct), Some(nonce)` branch is gone.
* New `BadSignatureLimiter`: per-`(app_id, trigger_id)` sliding-window
bucket (10 fails / 60 s). On lockout the receiver returns 429 with
Retry-After instead of 401. record_failure runs on both the
"no-secret" 401 path and any verify_signature failure.
Test fallout:
* email_inbound integration tests that relied on None secret: removed
the now-impossible `unsigned_trigger_accepts_without_signature` test,
replaced with `create_without_inbound_secret_is_rejected` covering
null / "" / " "; updated `malformed_body_is_422` and
`cross_app_path_is_404` to pass + sign with a secret.
* Two new unit tests pin the limiter behavior (burst lockout + per-
trigger independence).
Audit refs: security_audit/08_dos_resource.md#h-3,
security_audit/09_external_integrations.md#f-ext-m-001.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes 4 dashboard hardening findings and 5 of the Lows from the audit.
Dashboard hardening:
- Subtabs no longer re-fetch the app via api.apps.get on every page
load. users/files/dead-letters drop the fetch outright (the variable
was set but never read); queues + queues/[name] now consume the
layout's AppContext via getContext for the page title. The layout's
reloadApp() owns the historical-slug redirect — subtab-local redirect
blocks are removed so there's no race.
- The global :global(details > summary::before) chevron is now scoped
to details.chevron. The script editor's "Advanced sandbox" details
and the inbound-email-shape help-text both opt in; the script
exec-list logs no longer inherit a spurious chevron.
- deriveTab now matches the path segment by anchored ===, so a future
/apps/<slug>/queues-archived route wouldn't activate the queues tab.
Lows cherry-pick:
- ExecError gains Serialize/Deserialize derives + a snake_case tag so
RemoteExecutorClient (cluster mode v1.3+) can round-trip the variant.
- triggers_api rejects queue triggers whose visibility_timeout_secs is
below the dispatcher's per-message executor budget; with no minimum
the reclaim task races the handler and the queue silently
double-delivers. Existing test using 5s updated to 30s.
- New migration 0040: execution_logs.script_id cascade switched from
ON DELETE CASCADE to ON DELETE SET NULL so deleting a script no
longer wipes the forensic history that motivated the delete.
- New migration 0041: dead_letters composite index on
(app_id, created_at DESC) so the "list all" dashboard view stops
falling back to seqscan + sort when unresolved=false.
- Schema snapshot re-blessed.
Deferred to v1.2: the ExecRequest principal serde(skip) marker
(documented in-place; the cluster-mode PR will introduce the wire-safe
snapshot at that point) and the `pic --help` mention of
`picloud admin reset-password` (one-line follow-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the queue DL fan-out gap (audit Medium) and surfaces the
KV/docs/files non-transactional emit gap to operators.
- handle_queue_failure previously called queue.dead_letter to write the
DL row but never invoked fan_out_dead_letter. The comment claimed
"the outbox arm fires registered dead_letter handlers off the new row"
— but queue DL rows are written via a separate path that bypasses the
outbox entirely, so handlers filtered on source="queue" sat idle
forever. Refactor fan_out_dead_letter to take a DeadLetterFanOutCtx
struct so both the outbox arm and the queue arm can call it; the
queue arm constructs a TriggerEvent::Queue from the claimed message
and passes it through.
- New integration test queue_dead_letter_fans_out_to_dead_letter_handler
registers a dead_letter trigger filtered on "queue", forces queue
exhaustion, asserts the handler fires with the correctly-shaped event.
- KV/docs/files services committed the data write then ran events.emit
as a separate operation, logging-and-swallowing on failure. Bump the
six call sites from tracing::warn to tracing::error with an
event_emit_failure=true marker so operators can grep them. The full
single-tx repo refactor (extending ServiceEventEmitter with
emit_in_tx + tx-aware *_repo methods) is documented in kv_service.rs
as a v1.2 follow-up — it's a meaningful redesign that deserves its
own pass (pubsub_service::fan_out_publish is the reference shape).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes 4 High-severity audit findings:
- describe_event broken for HTTP-async payloads: was reading source/op
JSON fields that HttpDispatchPayload doesn't carry, producing empty
source on DL rows. from_wire("") then fell back to Kv on replay, the
dispatcher tried to resolve a non-existent trigger, and replay no-op'd
silently. Now branches on OutboxSourceKind: HTTP rows emit
"<method> <path>" + source="http"; Invoke rows emit "invoke_async" +
source="invoke"; TriggerEvent payloads keep top-level field extraction.
- HTTP outbound Authorization leaked across cross-origin redirects.
Manual redirect loop (Policy::none) reused header_map on every hop
and only scrubbed Content-Type for POST->GET. Now compares
url::Url::origin() across hops and strips Authorization,
Proxy-Authorization, Cookie when origin changes — matching reqwest's
default policy.
- F-S-010 inbound email HMAC had no replay protection. Signature was
computed over body only with no timestamp or nonce, so captured POSTs
were replayable indefinitely. Adds X-Picloud-Timestamp header bound
into the HMAC input (signed string is ts || "." || body), 300s
tolerance window, and a process-local LRU keyed on
(ts, sha256(body)) with 600s TTL to catch within-window replays.
Breaking change: webhook senders must include the timestamp header.
- expected_schema.txt re-blessed for migrations 0036-0039 (the previous
snapshot stopped at 0035 so CI would have gone red on first run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Running the E2E suites against real Postgres surfaced three shape bugs
in the test scripts that caused false failures:
invoke_e2e:
- invoke_cross_app_rejects used two TestServer instances (one per app),
but the second server's Owner admin isn't a member of the first
server's app. Replaced with a single server that creates both apps
via the same Owner admin (which has implicit access to every app).
- invoke_depth_limit_exceeds_cleanly: the recurser script had its own
try/catch, so when the depth limit fired inside the deepest call the
caught error became the BODY of a 200 response (which invoke()
returns to the caller). The outer caller's try/catch never saw a
throw → assertion failed. Rewrote so the recurser propagates throws
(no inner try-catch); the outer caller's try-catch surfaces the
depth error all the way up.
retry_e2e:
- All three tests used HTTP routes which need a domain claim the test
apps don't have (`no app claims host ""` 404s). Switched to the
admin bypass POST /api/v1/execute/{id} — same pattern dispatcher_e2e
uses. Sidesteps the per-app domain matcher entirely.
- retry_run_surfaces_last_error_after_max_attempts: try-catch is a
statement in Rhai, not an expression, so the block didn't evaluate
to the catch arm's map. Refactored to bind to `let out` inside the
catch arm, then return `#{ statusCode: 200, body: out }` as the
final expression.
All 11 v1.1.9 E2E tests now pass against Postgres:
queue_e2e: 4 passed (33s — exercises retry + dead-letter)
invoke_e2e: 4 passed (2s)
retry_e2e: 3 passed (2.5s)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four new test binaries, DB-gated via the established pool_or_skip()
pattern (returns Some(pool) when DATABASE_URL is set; prints a skip
notice and returns None otherwise — so plain `cargo test` stays green
locally while CI runs them).
crates/picloud/tests/queue_e2e.rs (4 tests):
- queue_receive_acks_on_success: enqueue → consumer fires → KV
marker observable → queue row deleted (ack worked)
- queue_receive_dead_letters_after_max_attempts: throwing handler
retries 3× via the dispatcher's compute_backoff path → dead_letters
row written, queue row deleted
- queue_one_consumer_per_queue_rejected: second trigger create for the
same (app_id, queue_name) returns 4xx with the documented "already
has a consumer trigger" message (advisory-lock guard works)
- queue_visibility_timeout_reclaim: insert message with stale claim
(claimed_at = NOW() - 60s, visibility_timeout = 5s); either the
dispatcher re-claims after reclaim runs, or the claim clears
crates/picloud/tests/invoke_e2e.rs (4 tests):
- invoke_by_name_same_app_returns_value: callee returns body.x+1;
caller invokes by name and writes the result to KV (42 round-trips)
- invoke_cross_app_rejects: callee in app_B, caller in app_A; error
string contains "different app" or "CrossApp"
- invoke_depth_limit_exceeds_cleanly: recursive script throws with
"depth" in the message at the bound (Limits::trigger_depth_max=8)
- invoke_async_enqueues_outbox_row: invoke_async returns execution_id,
dispatcher fires the OutboxSourceKind::Invoke arm, callee marker
appears
crates/picloud/tests/retry_e2e.rs (3 tests):
- retry_run_eventually_succeeds_inside_http_handler: counter mutates
across 3 retries through a real HTTP route, returns 3
- retry_run_surfaces_last_error_after_max_attempts: max_attempts=2,
always throws — caller's try/catch surfaces "boom"
- retry_on_codes_filters_unmatched_errors: counter == 1 after a
throw NOT in the codes list (immediate surface, no retry)
crates/manager-core/tests/migration_queue_messages.rs (4 tests):
- queue_messages_table_exists_with_expected_columns: shape +
nullability of every column
- queue_trigger_details_table_exists: shape
- queue_widens_trigger_kind_and_outbox_source_kind: triggers.kind
CHECK admits 'queue'; outbox.source_kind admits 'invoke'
- queue_messages_dispatch_index_is_partial: idx_queue_messages_dispatch
has WHERE claim_token IS NULL (guards against future migrations
accidentally dropping the partial)
All 15 tests pass clean skip when DATABASE_URL is unset. Workspace unit
suite remains at 432 passing (306 manager-core + 74 executor-core + 34
shared + 18 orchestrator-core).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
dead_letter triggers have been registerable since v1.1.1 but their
handlers never fired: dispatcher::handle_failure wrote the dead_letters
row and stopped — list_matching_dead_letter had no production caller.
Any deploy v1.1.1–v1.1.6 with dead_letter triggers had silently
non-functional handlers.
The fix: after the dead-letter row is inserted on retry exhaustion, fan
out to matching dead_letter triggers (filtered by source / originating
trigger_id / script_id) and enqueue one outbox row per match carrying a
real-shape TriggerEvent::DeadLetter (the §6 brief field names were stale
— used the actual variant: dead_letter_id, original: Box<TriggerEvent>,
attempts, last_error, trigger_id, script_id, first/last_attempt_at).
The recursion-stop (a handler's own failure isn't re-dead-lettered)
is upheld by the existing is_dead_letter_handler short-circuit.
Tests (DB-gated): handler actually fires with the nested original event;
existing row-create test now also asserts handler-fire; source_filter
excludes non-matching; failing handler does not recurse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inbound email: a provider POSTs a normalized JSON message to
POST /api/v1/email-inbound/{app_id}/{trigger_id}; the public receiver
verifies the optional HMAC signature, builds a TriggerEvent::Email, and
enqueues an outbox row the dispatcher delivers like any async trigger.
Handlers see ctx.event.email = #{from,to,cc,subject,text,html,
received_at,message_id}.
- migration 0024: widen triggers.kind + outbox.source_kind CHECKs to
'email'; new email_trigger_details table.
- TriggerKind::Email, TriggerDetails::Email{has_inbound_secret},
OutboxSourceKind::Email, TriggerEvent::Email; dispatcher routes the
email row via the generic resolve_trigger path.
- Admin POST /apps/{id}/triggers/email (validate_trigger_target; module
+ cross-app rejection). inbound_secret is stored ENCRYPTED via the
master key (deviation from the brief's plaintext default; decrypted
per inbound request — see HANDBACK §7).
- Dashboard: email trigger form on the Triggers tab + webhook URL +
expected-payload help.
- 8 DB-gated e2e tests (202/401/404/422/cross-app/handler-fire) +
receiver unit tests (HMAC verify, secret round-trip, payload parse).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Encrypted per-app secrets, reachable from scripts as
secrets::{get,set,delete,list}(name) and managed from the dashboard
Secrets tab. Values are AES-256-GCM-sealed with the process master key
(picloud_shared::crypto) before they touch Postgres; the repo only ever
sees ciphertext + nonce. JSON round-trip preserves Rhai types.
- migration 0023_secrets.sql (PRIMARY KEY (app_id, name)).
- SecretsService trait (picloud-shared) + SecretsServiceImpl + repo
(manager-core), wired into the Services bundle and Rhai engine.
- Capability::AppSecretsRead/Write (→ script:read / script:write); no
new Scope variants (seven-scope commitment).
- Admin API GET/POST/DELETE /apps/{id}/secrets (list returns names +
updated_at, never values).
- build_app now takes a MasterKey, sourced from PICLOUD_SECRET_KEY in
main.rs; test callers pass a fixed test key.
- 64 KB value cap (PICLOUD_SECRET_MAX_VALUE_BYTES); no ServiceEvent
emission (secret writes don't fire triggers, by design).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test creates a script in the default app earlier in the body, so a
plain DELETE /apps/default hits the soft no-cascade guard and 409s
before the capability check runs. The intent is to validate that admin
holds AppAdmin everywhere, not to exercise the cascade contract — pass
?force=true so we reach the gate we're trying to test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Aligns the canonical capability rules with how the dashboard now shadows
its UI. Instance admins become implicit app_admin on every app (only
InstanceManageSettings stays owner-only), and the script-delete handler
moves from AppWriteScript to AppAdmin so editors can save but not delete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `membership_makes_app_appear_in_members_app_list` previously seeded
the membership via the repo helper; switch to the public POST
endpoint so the test actually exercises the full HTTP round-trip
the dashboard depends on.
- Add `add_member_with_missing_user_id_is_rejected` to pin the
Axum-JsonRejection 4xx contract on malformed POST bodies.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 16 integration tests against a real Postgres covering the new
/api/v1/admin/apps/{id_or_slug}/members surface:
- list / add / patch / remove against an explicit member row
- 409 on duplicate, 422 on inactive target, 422 on owner/admin target
- 404 on PATCH without an existing row; 204 idempotent DELETE
- viewer-as-bob receives 403 on every mutating verb
- both slug and UUID paths resolve to the same body
- bob-with-app_admin can manage the member list, including removing
himself (load-bearing for the no-last-app-admin-guard decision)
- granting a `member` user a viewer membership makes the app appear
in their `GET /admin/apps` list (was empty before)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
GET /api/v1/admin/apps/{id_or_slug} now returns an `AppRole`-typed
`my_role` alongside the existing app fields, computed server-side from
the Principal: `Owner → app_admin` and `Admin → editor` (both
implicit per blueprint §11.6), `Member → app_members.role` (looked up
via the existing `AuthzRepo::membership` already in `AppsState`).
The dashboard uses this single field to decide whether to render
admin-only surfaces (Members tab, etc.) instead of duplicating the
implicit-grant rules on the client side — keeps API and UI gate logic
identical with one round-trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /admins create/patch endpoints now plumb email through to the
repo so the dashboard's invite + edit forms aren't silently dropping
it on the floor. Discovered during smoke testing — the database
column existed and was exposed in the response DTO, but neither
the request DTO nor the repo's create() accepted it.
CreateAdminRequest gains optional email; PatchAdminRequest gains
email with JSON Merge Patch semantics:
absent → don't change
null → clear (write NULL)
"<string>" → set to that value
The tri-state needs Option<Option<String>> with a tiny custom
deserializer; serde collapses absent and null otherwise.
normalize_email() trims, treats blanks as None, and rejects
obviously bogus values (no '@', >254 chars) with a 422. Real
email verification is a future concern.
Repo trait gains an email parameter on create() and a new
update_email() method. The unique-violation branch in create now
inspects constraint() to distinguish duplicate username from
duplicate email.
Integration test exercises create-with-email, PATCH null clears,
PATCH value sets, PATCH without email key no-ops on email.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
clippy::map_unwrap_or — drop the map().unwrap_or(false) for the
flatter is_some_and(Value::is_null).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Login and /auth/me now return the same shape — id, username,
instance_role, email — so the dashboard can gate UI on role from
either the login response or the layout's me() refetch without an
extra round-trip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure formatting pass — no behavior changes. Catches the line-wrapping
drift across the new authz / api_keys / middleware / handler edits
that piled up during the implementation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Covers the matrix laid out in the plan:
* bootstrap admin lands as Owner
* owner / admin / member access matrices on the default app
* bearer pic_ key and cookie session resolve to the same Principal
* read-only key cannot write (scope intersection)
* bound key cannot escape its app
* member listing isolation at SQL for /admin/apps + /admin/scripts
* deactivating a user expires every API key for them
* mint rejects bound key carrying instance:* scopes (422)
* list_active_owners returns the right set for the startup warning
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3.5 ships → product minor bump under pre-1.0 rules (any surface
bump triggers minor). Schema is now 6 (0006_users_authz.sql); API
remains 1 (additive endpoints + new credential type, no breaking
shape changes). docs/versioning.md updated.
main.rs gets warn_on_multi_owner_install() which fires once after
bootstrap when more than one active owner exists — points the
operator at PATCH /admin/admins/{id} for cleanup. Soft-fail on DB
error (does not block startup).
The api-test schema assertion was updated to expect 6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* admin_user_repo: surface instance_role + email on AdminUserRow /
Credentials; create() now takes instance_role; add
update_instance_role, list_active_owners, count_other_active_owners.
* admin_users_api: DTO + create/patch accept instance_role (defaults
to Admin on create — only env-var bootstrap defaults to Owner).
PATCH and DELETE enforce the last-owner guard alongside the
existing last-active-admin guard.
* app_members_repo: new — implements AuthzRepo::membership via the
app_members table plus upsert/remove/list_for_user/list_for_app.
* api_key_repo: new — create / find_active_by_prefix / touch_last_used
/ list_for_user / get / delete_by_id_and_user / expire_all_for_user.
Separates ApiKeyRow (no hash) from ApiKeyVerification (hash, for
the middleware verifier) so handlers can't leak the hash.
* auth_bootstrap + picloud tests: pass Owner on the bootstrap seed
and on the test admin seed respectively; in-memory test repo
implements the new trait methods.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apps become the isolation boundary for scripts, routes, domains, and
later data. Doing this now — while the surface is small — avoids
several migrations on populated tables once v1.1 data-plane services
ship.
Schema (migration 0005_apps.sql):
- New tables: apps, app_domains (with shape_key UNIQUE for collision
detection), app_slug_history (for permanent slug-rename redirects).
- app_id added to scripts, routes, execution_logs (non-null, cascading
rules per row).
- Script-name uniqueness becomes per-app; the route unique index is
swapped for an app-scoped version.
- The "default" app is seeded unconditionally with a localhost claim;
existing scripts/routes backfill into it. Fresh installs additionally
get the Hello World seed via seed_hello_world_if_fresh after
migrations run (idempotent — only fires when the default app has no
scripts).
Orchestrator dispatch is two-phase: AppDomainTable resolves Host →
app_id (most-specific match wins, exact beats wildcard), then the
existing route matcher runs against that app's partitioned slice via
RouteTable. Unknown hosts return 404 at the app layer with a clear
message; /api/v1/execute/{id} still works as the implicit
__internal__ claim, decoupled from any public domain.
Manager API: full CRUD for /api/v1/admin/apps/* and
/api/v1/admin/apps/{id_or_slug}/domains/*, with slug:check + force
takeover semantics implementing the rename-history flow (two-step
check → confirm, never a single endpoint). Script create requires
app_id; list accepts ?app= filter. Route create validates host
against the parent app's claims; conflict detection stays strictly
intra-app.
Dashboard: /admin/apps and /admin/apps/{slug} (overview + scripts +
domains + settings tabs, with slug-history-aware redirects). Root
path redirects to the apps list. Script detail page gains an app
breadcrumb and threads app_id into the route preview.
Deferred per design: per-app admin roles. The require_admin middleware
remains the seam where role checks will slot in later.
Blueprint §11.5 and roadmap updated to reflect what shipped; docs/
versioning.md notes the schema 3 → 5 bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the regression risk of the admin API and dashboard being open
to anyone reaching the bound port. Required foundation before v1.1
data-plane services land.
Per-user accounts (admin_users), Argon2id passwords, env-var bootstrap
of the first admin that becomes inert once any admin exists, opaque
32-byte session token doubling as bearer credential, 24h sliding TTL
configurable via PICLOUD_SESSION_TTL_HOURS. is_active column lets
admins be deactivated without losing audit history; last-active-admin
guard on DELETE and on PATCH that flips is_active to false (sessions
also wiped on deactivation).
require_admin middleware fronts every /api/v1/admin/* route. The data
plane (/api/v1/execute/{id}), /healthz, /version, and user routes
stay open. picloud admin reset-password <username> subcommand handles
recovery without going through HTTP.
Dashboard gains /admin/login and /admin/admins surfaces, a top-bar
user menu, and a token store with a localStorage echo so refreshes
don't sign you out. Cookie-based auth works in parallel for non-SPA
clients.
Forward compatibility: future RBAC tables (admin_roles,
admin_user_roles) join on admin_users.id; the auth middleware is the
seam where role checks slot in. Email, 2FA, passkeys, and personal
API tokens are all additive without touching admin_users.
Blueprint §11.4 updated to reflect what actually shipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scripts can now answer at user-chosen paths (e.g. /greet, /greet/:name,
/webhooks/*), on user-chosen hosts (strict or *.example.com wildcards),
on user-chosen methods. The internal /api/v1/execute/{id} endpoint
stays as the always-available ID-based bypass.
Routing rules (decided in design with the user; see chat history):
Path kinds:
exact /greet literal
prefix /greet/* strict-subtree; stored as "/greet/";
does NOT match bare /greet (add an
exact route for that case)
param /users/:id :name captures one whole segment;
mid-segment colons are rejected;
{name} is reserved for a future SDK
Host kinds:
any no Host header constraint
strict sub.example.com literal match (case-insensitive)
wildcard *.example.com suffix match; multi-level subdomains OK
Within-kind uniqueness:
two routes of the same kind that could match the same request
conflict at config time. Algorithm (orchestrator_core::routing::
conflict):
exact: literal equality
prefix: literal equality (longer-prefix coexists; longer wins
at request time)
param: same segment count + same literals at every
literal-vs-literal position (the user's example:
:id vs :userId at same shape is a conflict)
Request-time precedence:
exact > param > prefix
among non-exact: more leading-literal segments wins
tie: param > prefix (more constrained)
within prefix: longest matching prefix wins
host bucket: strict > wildcard (longer suffix) > any; fall through
to less specific buckets when path doesn't match
Reserved path prefixes: /api/, /admin/, /healthz, /version
Routes that look invalid at config time return 422 with the precise
parse error; conflicting routes return 409 with the conflicting route
in the body (so the dashboard can render the conflict inline).
What landed:
* 0003_routes.sql — routes table (host_kind, host, host_param_name,
path_kind, path, method, script_id) with UNIQUE index on the
literal binding tuple. Schema 2 → 3.
* shared::Route / HostKind / PathKind — flat storage shape that
crosses wire boundaries cleanly.
* orchestrator_core::routing — four sub-modules, all unit-tested:
pattern.rs (16 tests) parse + validate + display
conflict.rs (12 tests) within-kind overlap predicate
matcher.rs (12 tests) runtime dispatch (specificity-aware)
table.rs Arc<RwLock<Vec<CompiledRoute>>>
shared by manager (writes) and
orchestrator (reads); atomic replace
after each admin write
* manager-core::route_admin — five new admin endpoints under
/api/v1/admin:
POST /scripts/{id}/routes create
GET /scripts/{id}/routes list per script
DELETE /routes/{route_id} delete (refreshes table)
POST /routes:check pre-flight conflict check
(powers the dashboard's
live conflict warning)
POST /routes:match synthetic URL → matched
route + extracted params
(powers the dashboard's
match-preview tool)
Stored path strings stay raw (user-typed); normalization
happens only in the in-memory CompiledRoute so re-parses are
idempotent.
* orchestrator_core::api::user_routes_router — fallback handler
mounted in picloud after the system routes. Reads Host /
method / path / query from the request, dispatches via the
table, builds an ExecRequest with params/query/rest filled,
calls the executor, writes to the log sink. 10 MiB body cap.
* executor-core::ctx (SDK 1.0 → 1.1) — adds
ctx.request.params (map of named-param captures)
ctx.request.query (parsed query string)
ctx.request.rest (suffix for prefix routes; "" otherwise)
All three are always present (empty when not applicable) so
scripts can read them unconditionally.
* picloud::build_app — now async; loads routes at startup,
populates the shared table, mounts route_admin_router under
/api/v1/admin alongside the script CRUD, and the user-routes
fallback at the app root.
* caddy/Caddyfile + Caddyfile.prod widened: anything not
/healthz, /version, /api/v1/admin/*, /api/v1/execute/*,
/api/* (404 sunset), or /admin/* (dashboard) → picloud.
* Dashboard moves to /admin/* via SvelteKit paths.base. Its
internal Caddy strips the prefix and serves with SPA fallback.
All in-app links use $app/paths. The dashboard URL is now
http://localhost:8000/admin/ — one-time break for the new
URL freedom users gained.
* PICLOUD_PUBLIC_BASE_URL env var, exposed via /version so the
dashboard renders full URLs for routes regardless of the
operator's external port / TLS setup.
* memory_limit_mb stays in the schema, still v1.3+ advisory.
Verified live through Caddy:
/version → schema 3, sdk 1.1, public_base_url
GET /admin/ → 200, dashboard HTML containing "PiCloud"
POST /api/v1/admin/scripts → 201
POST .../scripts/{id}/routes (path=/greet/:name) → 201
GET /greet/alice?lang=en → 200 {"name":"alice","q":"en"}
POST conflicting route → 409 with conflicting_route body
POST /admin/foo route → 422 "reserved"
POST /api/v1/admin/routes:match → matched + params extracted
GET /unbound-path → 404 JSON
Tests:
* 40 routing unit tests (pattern + conflict + matcher tables)
* 14 executor-core unit tests (one new for ctx.request.params/
query/rest exposure)
* 32 integration tests (10 new for routing CRUD + dispatch +
conflict + reserved + specificity tie-break + match preview +
delete invalidation + /version returns public_base_url)
* default cargo test --workspace stays green; opt-in via
DATABASE_URL + --include-ignored for the integration suite
Bumps: schema 2 → 3; SDK 1.0 → 1.1; product 0.3.0 → 0.4.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds optional per-script overrides for the six Rhai sandbox knobs
(max_operations, max_string_size, max_array_size, max_map_size,
max_call_levels, max_expr_depth). The executor merges its defaults
with each script's overrides on every call; the manager validates
overrides against an admin-set ceiling at write time, so the
executor trusts whatever is stored.
Storage chose JSONB on the existing scripts table over six new
columns: lets future knobs land as code-only changes, keeps the
sparse common case (most scripts override nothing) cheap to store
and serialize, and matches how the manager + executor pass the
config across the wire.
* 0002_sandbox.sql — ALTER TABLE scripts ADD COLUMN sandbox
JSONB NOT NULL DEFAULT '{}'
* shared::ScriptSandbox — six Option<u64> fields with
deny_unknown_fields so typos surface as 422
* Script.sandbox + ExecRequest.sandbox_overrides — typed end
to end; cluster mode just serializes the same struct
* executor-core::Limits::with_overrides — field-by-field
replacement; tests cover the override actually tightening
the live engine
* manager-core::SandboxCeiling — built-in conservative
defaults (10M ops, 1 MiB strings, 100k array/map, 128
call/expr depth); env vars override per knob, invalid
values warn-and-skip rather than blocking boot
* manager-core admin API — POST/PUT accept `sandbox`; values
above the ceiling return 422 with the specific field +
requested + ceiling; absent or `{}` keeps platform defaults
* picloud all-in-one — wires SandboxCeiling::from_env() into
AdminState
* memory_limit_mb stays in the schema, marked v1.3+ advisory
(no enforcement until OS-level isolation lands with
cluster-mode executors)
Verified live through Caddy:
* /version reports schema 2, product 0.3.0
* Script with max_operations: 500 → 507 on a 10k-iteration loop
* Same script after PUT raising to 1M → succeeds, returns 10000
* POST with max_operations: 1_000_000_000 → 422 (exceeds ceiling)
Tests:
* 13 executor-core unit tests (added 2 for override semantics)
* 20 integration tests (added 6 for sandbox CRUD + ceiling +
unknown-field rejection + executor honoring overrides)
* default cargo test --workspace stays green (integration tests
remain #[ignore]'d until DATABASE_URL is set)
Bumps:
* schema 1 → 2
* product 0.2.0 → 0.3.0
* SDK unchanged (scripts see nothing new)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Establish how versions are assigned, bumped, and checked across the
five things that actually change for users: the product itself, the
Rhai SDK, the HTTP API, the database schema, and the inter-service
wire (reserved for cluster mode). Crates ship in lockstep — drift
between picloud-shared and picloud-manager-core is fiction since
they always release together — but surfaces are versioned and
checked at their natural boundaries.
* docs/versioning.md is the authoritative reference: what gets a
version, the per-surface compatibility rules, how each surface
bump cascades to the product version (loose pre-1.0, strict
post-1.0), and the five enforcement mechanisms (lockstep at
compile time, /version at runtime, golden SDK contract tests,
migration replay, CI guardrail).
* shared::version exposes four constants — PRODUCT_VERSION (from
CARGO_PKG_VERSION), SDK_VERSION ("1.0"), API_VERSION (1),
WIRE_VERSION (1). Scripts read SDK_VERSION as ctx.sdk_version
and can feature-detect against it.
* Workspace inheritance: `[workspace.package] version = "0.2.0"`
is the single point of truth; every crate uses
`version.workspace = true`. dashboard/package.json mirrors.
* Routes move to /api/v1/* — both control plane
(/api/v1/admin/*) and data plane (/api/v1/execute/{id}).
Picloud composes them via a single `/api/v{API_VERSION}` nest,
so the next major is a copy-paste-and-bump. Caddyfile (dev and
prod) routes /api/v1/* to picloud and 404s any other /api/*
so old clients fail loudly instead of getting the SPA shell.
Dashboard client + integration tests updated.
* /healthz remains a plain "ok" string (k8s probes); /version is
the new JSON endpoint returning every surface version in one
place — product, sdk, api, schema (from
manager-core::migrations::latest_version), wire.
* Reasonable bump rationale: API path changes are breaking by
definition, so 0.1.0 → 0.2.0 (pre-1.0 license to bump minor on
any breaking change). SDK starts at 1.0 because scripts depend
on it more strictly than the product depends on its internals;
we'd rather promise SDK stability early than pull the rug.
Verified live:
* /healthz → "ok" (plain text)
* /version → {product:"0.2.0",sdk:"1.0",api:1,schema:1,wire:1}
* /api/v1/admin/scripts → 200
* /api/admin/scripts → 404 with error JSON (sunset major)
* Script can read ctx.sdk_version → "1.0"
* All 14 integration tests pass against new paths
* 11 executor-core unit tests pass (added one for sdk_version
exposure with the major.minor format invariant)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three threads landing together because they share a public surface
(the new execution_log shape) and verifying any one in isolation
would mean re-doing the work later.
== (A) execution log persistence ==
* shared::ExecutionLog + ExecutionStatus carry the audit-trail
shape that flows from the orchestrator through the sink and
back out via the manager's logs endpoint.
* shared::ExecutionLogSink trait — abstraction the orchestrator
writes through. In single-process MVP mode the manager's
Postgres-backed impl is plugged in directly; in cluster mode
(v1.3+) the orchestrator's impl will post over HTTP to the
manager. Trait lives in `shared` so neither *-core crate has
to know about the other.
* manager-core::PostgresExecutionLogSink writes to the
execution_logs table (already in the initial migration);
PostgresExecutionLogRepository reads them back, paginated.
AdminState now carries both a script repo and a log repo, so
`admin_router` exposes `GET /scripts/{id}/logs?limit=&offset=`
capped at 200 rows per page to keep the dashboard responsive.
* orchestrator-core::DataPlaneState gains `log_sink`. The
execute handler builds an ExecutionLog on every outcome —
success, error, timeout, budget-exceeded — and awaits the
sink. Sink failures are logged at warn and DO NOT mask the
user-facing result, since "we couldn't write the audit row"
is a separate concern from "the script ran".
* picloud binary refactored into a lib (`build_app(pool)` is
the seam) + thin bin shell. Same Postgres pool backs the
script repo, the log repo, and the sink — no double pool.
== (B) dashboard ==
* Typed API client extended with `scripts.logs(id, opts)`,
`scripts.update/remove`, and `execute(id, body, headers)`.
Plain `fetch` wrapper now surfaces server-side error
messages via a typed ApiError so the UI can render them.
* `/` — create-script form now actually creates; on success
the list reloads. List entries link to detail.
* `/scripts/[id]` — new detail route: source editor with save
(calls update, version bumps); Test invoke panel that sends
arbitrary JSON body + headers to /api/execute and shows the
response; Recent executions panel reading from /logs with
expandable per-row request/response/script-log views.
Delete button with confirm. SPA-routed; Caddy serves
`build/` with the same index.html fallback.
== (C) integration tests ==
* crates/picloud/tests/api.rs — 14 sqlx::test cases driving
`build_app` through an axum_test::TestServer against a fresh
Postgres DB per test. Covers: health, full script CRUD,
duplicate-name conflict, invalid-source rejection on both
create and update, execute echoing the body, status+header
passthrough, 404 on missing scripts, error-path executions
landing in the audit log with the right status.
* Tests are `#[ignore]` by default so plain `cargo test
--workspace` stays green without infrastructure. Opt-in via:
`docker compose up -d postgres && \
DATABASE_URL=postgres://picloud:picloud@127.0.0.1:15432/picloud \
cargo test -p picloud --test api -- --include-ignored`
Verified live through Caddy on :8000: three logged invocations
land in the logs endpoint with the right structured `data` on
each `log::info`/`log::warn`, error-path executions are still
captured with status=error, dashboard list + SPA detail route
both reachable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
