A WordPress-inspired CMS whose entire backend is PiCloud Rhai scripts
deployed with pic apply, plus a SvelteKit frontend. Built to dogfood the
project tool, CLI, and SDK; exercises docs/kv/files/users, docs+queue+cron
+pubsub triggers, the transactional-outbox notification chain, a docs
before-interceptor, set_if CAS, SSE, per-app CORS, env overlays, and a
durable Workflow (validate -> enrich || seo -> publish -> finalize) started
with workflow::start and polled with workflow::run_status, visualized live
with Svelte Flow.
FINDINGS.md / SECURITY.md capture every gap, surprise, and security issue
found (each tagged [PiCloud] vs [CMS]); the platform fixes for the
actionable ones ship in the preceding commit.
Also folds in the read-only `pic` allowlist entries added to
.claude/settings.json during the session (fewer-permission-prompts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Additive fixes surfaced by building a full CMS on PiCloud
(examples/cms-poc/). One migration (0077_apps_cors.sql); no destructive
changes.
Security:
- Close the 502 info-leak on user routes: an uncaught script error (incl.
a throw) leaked the app UUID + script fn names + source line/col. The
user-route (inbox) path now shares scrub_runtime_detail with the
execute-by-id path — raw detail is logged under a correlation id and the
client sees only "script execution error (ref: <uuid>)".
Added:
- docs::find $contains operator (array membership via JSONB @>), per-app
and group-shared — the inverse of $in.
- App-user role management from API/CLI: pic users add-role / rm-role,
backed by the apps/{id}/users/{user_id}/roles endpoints (AppUsersAdmin).
- Per-app CORS: pic apps cors set, apps.cors_allowed_origins; the
orchestrator echoes an allowed Origin and answers OPTIONS preflight.
- Non-JSON request bodies: form-urlencoded -> object, text/* -> string,
other -> base64; ctx.request.content_type added. Malformed JSON still 400s.
- Binary responses via #{ body_base64 } with a script-set Content-Type.
- workflow::run_status(run_id) SDK (F-038): the starting script can poll a
run — #{ status, output, error, steps } or () when no such run belongs to
the app (app_id is the isolation boundary); same AppInvoke gate as start.
Fixed:
- pic apply "app not found" is now actionable (points at pic apps create).
- Interceptor- (F-021) and workflow-step-bound (F-037) scripts no longer
warn "no route or trigger" at plan time — both count as reachability.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An opt-in receive-only SMTP listener (PICLOUD_SMTP_BIND) so an MX can point
straight at PiCloud. It speaks minimal SMTP (HELO/EHLO/MAIL/RCPT/DATA/RSET/
NOOP/QUIT), resolves each RCPT TO mailbox to the app-owned email trigger that
claims it, and inserts an Email outbox row the dispatcher fires — the same tail
as the HMAC webhook, unchanged.
- migration 0076: email_trigger_details.inbound_address (case-insensitively
unique among app triggers) + TriggerRepo::email_inbound_target_by_address /
SmtpInboundTarget; the interactive create-email API accepts inbound_address.
- crates/picloud/src/smtp.rs: a small tokio accept loop + session state machine
+ a testable deliver() core + a minimal RFC-5322 header/body split. DATA is
size-capped with dot-unstuffing; no AUTH/STARTTLS (TLS terminates upstream —
the recipient address + per-app isolation are the boundary).
- spawned in run_server alongside axum::serve, sharing the pool, on the same
shutdown signal.
Pinned by a picloud integration test (deliver → one Email outbox row for a
known mailbox, none for an unknown one) + smtp.rs unit tests (address parse,
header/body split). Multipart/MIME decoding is a documented follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the §9.4 track:
- set_if (compare-and-swap), per-app and shared, now runs the same (kv, set)
before/after interceptor as set — otherwise CAS was a silent bypass of a set
policy. The before-hook can transform the new value; the after-hook sees the
swapped bool.
- read-only pic interceptors ls --app|--group: app shows the RESOLVED chain
(every marker guarding its writes, nearest-owner-wins), group its own markers.
New apply_service::interceptor_report + InterceptorInfo, /apps|groups/{id}/
interceptors routes (AppRead/GroupScriptsRead), client + cmd mirroring
extension-points.
CLAUDE.md updated: §9.4 service interceptors are now COMPLETE (M1-M12).
Pinned by a journey: set_if of a guarded key is denied while a free key swaps,
and interceptors ls lists the kv/set guard (12 interceptor journeys green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every non-KV mutating op now runs the same before/after interceptor machinery:
- docs create/update/delete (+ group) — create/update honor the M4 data transform;
- files create/update/delete (+ group) — allow/deny only (the blob is never
surfaced to the hook, value = None);
- queue enqueue (+ shared) — transform-capable; after reports the message id;
- pubsub publish_durable (+ shared topic) — transform-capable;
- http request — all verbs funnel through two svc.request sites; collection =
method, key = url, body not surfaced.
ictx threaded into the free-fn/handle paths that lacked it (http was _ictx;
queue/pubsub per-app publish).
validate_bundle_for generalized to a per-service allowed-ops map (kv set/delete;
docs/files create/update/delete; queue enqueue; pubsub publish; http request) —
unknown service/op still rejected. INVARIANT enforced + verified: every allowed
(service, op) has a matching runtime hook, so no validated-but-unhooked
fail-open. Pinned by a new docs-create deny journey (11 interceptor journeys green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Memoizes the interceptor chain resolution per execution tree, keyed by
(app_id, service, op). The FIRST hooked-eligible op pays the one chain query;
every later op reuses it — so N kv::sets in a script with no interceptors now
issue ONE resolve, not N (the dominant, zero-marker case caches an empty chain).
InterceptorCacheScope is an RAII scope entered in execute_ast next to the
emission budget, same re-entrancy model: a synchronous invoke/interceptor
re-entry shares the cache, and it is cleared at the outermost boundary so a
pooled thread never serves a foreign app's cache. Pinned by an executor-core
test: 25 sets → 1 resolve, and a fresh execution → a new resolve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A [[interceptors]] marker can set timeout_ms (migration 0075, CHECK > 0); a
runaway guard (loop {}) is interrupted and the op DENIED (fail closed) within
budget rather than hanging the write path. The effective deadline is
min(caller-remaining, now + timeout) — a hook can only tighten, never extend,
its caller's deadline. NULL uses PICLOUD_INTERCEPTOR_TIMEOUT_MS (default 5s).
Wiring: run_resolved_blocking splits into a core + a _with_timeout variant that
computes the effective deadline from engine::ambient_deadline() and runs via
execute_ast_with_deadline; run_one_hook passes the marker's timeout_ms (or the
env default). timeout_ms threaded end to end — manifest, plan, BundleInterceptor,
the reconcile diff (part of the mutable body: a timeout change is an Update),
insert_interceptor_tx, resolve_chain/list_for_owner/list_on_app_chain +
SealedInterceptor/InterceptorMarker, and interceptor_service → ResolvedInterceptor.
Schema snapshot re-blessed. Pinned by a journey: a loop{} guard with
timeout_ms=100 is denied and its write does not persist.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M4: a before-interceptor returning #{ allowed: true, data: ... } rewrites the
value actually written (threaded through the chain; each hook sees the prior
transform), size-capped at MAX_JSON_MATERIALIZE_BYTES. Delete never transforms.
M3: an 'after' phase interceptor runs once the write has committed, with the
write's result in its payload. After-hooks observe/deny but CANNOT roll back —
a deny surfaces as an operation error while the write persists (documented at
the call site).
Adds phase authoring end to end: a [[interceptors]] entry gains phase =
before|after (default before), threaded through the plan wire, BundleInterceptor,
validate (phase in {before,after}), the reconcile diff/insert/prune key
(service/op/phase), and the repo (insert/delete/list_for_owner/list_on_app_chain,
+ the marker's phase). run_before now returns the transformed value; run_after is
new; both share one fail-closed per-entry runner. Pinned by two journeys: a
before-hook transforms the stored value, and an after-delete hook sees
result==true, denies, yet the key stays deleted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M1: introduces a clone-cheap InterceptorCtx { interceptors, self_engine,
limits } threaded into every SDK register fn (kv/docs/files/queue/pubsub/http)
so the non-KV services carry the hook seam (unused until M7-M11); KvHandle
collapses its three fields into one ictx.
M2: replaces the nearest-only resolver with ordered before/after chains. The
trait becomes resolve(cx, service, op) -> InterceptorChain { before, after }
(migration 0074 adds a phase column + phase-aware unique indexes). The before
-chain runs ancestor->app (depth DESC) so a group compliance guard can't be
bypassed by a descendant; single-marker behavior is byte-identical to before.
An identity cycle guard (thread-local visited-set keyed by script_id) denies a
detected cycle, alongside the existing binary re-entrancy break. Fail-closed
verdict preserved (allow only on #{ allowed: true }; a Dangling entry or a
missing engine back-ref denies); app_id still derives from cx.app_id only.
after-chains resolve but stay unused until M3. Pinned by two new interceptor
journeys (ancestor->app chain ordering; self-referential no-recurse).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group declares a declaratively-authored [[triggers.dead_letter]] shared=true
handler; when a message in the group's SHARED queue is exhausted, the
dispatcher's q_terminal group branch (after persisting to group_dead_letters)
fans out to it via list_matching_shared_dead_letter(owning_group, "queue", …),
each outbox row stamped the WRITER app_id (the consuming app — the M2 shared
-write model), so the handler runs under the consumer. The per-app
list_matching_dead_letter gained AND t.shared = FALSE (the shared flag is the
namespace boundary); the owning-group filter is the isolation boundary.
Adds BundleTrigger::DeadLetter + a DeadLetterTriggerSpec manifest kind (group
+shared only — validate_bundle_for rejects app-owned or non-shared, and exempts
it from the shared-requires-a-collection rule); insert_trigger_tx now accepts
dead_letter and writes dead_letter_trigger_details; current_trigger_identity
matches a group-shared dead_letter so re-apply is a NoOp (app-owned ones stay
diff-invisible). Pinned by tests/shared_dead_letter.rs (owning group matches,
per-app query does not, foreign group does not).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Aggregates the existing execution_logs table (no hot-path instrumentation):
ExecutionLogRepository::summarize_for_app returns counts, error rate, latency
avg/p50/p95 (percentile_cont), by-status/by-source breakdowns, and an hourly
series over a trailing window (clamped 1h..90d). New metrics_api router at
GET /api/v1/admin/apps/{id}/metrics?window= (AppLogRead), and a dashboard
Metrics tab (summary cards + an inline-SVG per-hour bar chart, no JS dep).
Pinned by a manager-core test asserting the exact rollup (percentiles included)
and the empty-app zero case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exports a group as a [group] manifest: its scripts, [[routes]]/[[triggers.*]]
TEMPLATES (with the group-only sealed/shared flags), a shared collections
entry, [vars]/[secrets], extension_points, and [suppress]. Groups own no
workflows.
The group trigger/route reports were display-only (no ops, dispatch, retry,
host_kind, cron tz, queue timeout) so a pulled manifest could not re-plan
clean. Enriched TriggerTemplateInfo with the full internally-tagged
TriggerDetails JSON + dispatch_mode + retry_max_attempts, and RouteTemplateInfo
with raw manifest-shaped fields (method/host_kind/host/host_param_name/
path_kind/dispatch_mode); pic routes ls --group now applies the ANY/host munge
client-side, consistent with the app pic routes ls. pull.rs factors the
script-writing + trigger-decoding it shares with the app path.
Pinned by a pull journey: a group with a sealed+shared kv trigger, a sealed
route, a shared collection, and a var round-trips and the exported manifest
re-applies clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fills the one per-app admin-read gap kv/files already covered. New
docs_api.rs mirrors kv_api (GET /apps/{id}/docs[/{collection}/{doc_id}],
AppDocsRead capability, DocsRepo::list/get) with the group_blobs_api docs
response shape so the CLI shares one deserialize. DocsCmd gains optional
--app/--group (mutually exclusive, like KvCmd) dispatched via
require_one_owner; new client docs_list/docs_get. Read-only by design,
matching kv/files. Pinned by a collections journey: a script writes the
app's own docs collection, the operator lists + fetches it with no script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per-app (non-shared) email inbound secrets already work and are the
original path: an app-owned email trigger resolves its inbound_secret_ref
against the app's own secret store and seals it AAD-bound to the app
(SecretOwner::App, AAD email:{app_id}); M5.5 generalized that app path to
groups, not the reverse, and the inbound path recovers SecretOwner::App
when materialized_from is NULL. Only the interactive API's inline-vs-named
-ref nuance remains, and it is deliberate. Fixes the CLAUDE.md Track-A
tail and the design-doc M5.5 tail that called it deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- roles: member invoke asserts the script's actual stdout, not just exit 0;
- api: app-slug script filter asserts the returned script's name;
- workflow_orchestrator: unknown-workflow asserts WorkflowError::NotFound,
not just is_err().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extracts the dev-mode acknowledgement decision into a pure
check_dev_acknowledgement(secret, dev_mode, dev_ack) called by from_env
before key resolution, and pins that dev mode without a secret and
without PICLOUD_DEV_INSECURE_KEY is refused. Behavior-preserving.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives PICLOUD_MAX_EMISSIONS_PER_EXECUTION through the live
pubsub::publish_durable surface (not just the emit_budget counter): a
1001-publish loop errors naming the budget and the service sees at most
1000. Mutation-verified (removing charge_emission from publish fails it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An outbox row whose trigger_depth exceeds max_trigger_depth is dropped
without executing the handler. Mirrors the disabled-http drop harness;
mutation-verified (weakening the depth gate fails it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors group_kv_caps_resolve_by_role_up_the_chain for the two other
group write capabilities: a Viewer is denied GroupPubsubPublish and
GroupQueueEnqueue, an Editor is allowed, and an outsider is denied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the atomic_write suite with three write-path gaps:
- an app cannot exceed PICLOUD_APP_DOCS_MAX_ROWS (create-only, update
net-zero, delete frees a slot) via a real PostgresDocsWriter;
- concurrent writers cannot push a group past its docs total-bytes quota
(16 racers against a 450-byte ceiling under pg_advisory_xact_lock);
- a files create with a broken outbox leaves no row AND no blob on disk,
while a files delete with a broken outbox keeps both the row and the
readable blob (the disk-ordering invariant).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three related fixes from the test audit.
**The CLI journey fixture gets its own database.** It spawned a real picloud —
whose dispatcher/orchestrator claim loops are global by design (one instance owns
one database) — against the shared dev DB, so it could claim the manager-core
suites' outbox/workflow rows (the same class of bug already fixed for the e2e
suites, one binary over). It now clones one dedicated database per journey run
from the migrated template. test-support gains `named_test_db_url` (explicit
stable name) + a blocking wrapper for the sync `LazyLock` fixture. The one journey
that talks to Postgres directly (dead-letter injection) now uses the fixture's DB
URL, not the base DATABASE_URL, so it hits the database the server reads.
**workflow_orchestrator moves to per-test databases.** Its `claim_ready_step` is
global, so the old harness serialized every test behind a process-wide CLAIM_LOCK
AND ran `DELETE FROM workflow_runs` (unscoped — it wiped every app's runs) before
each one. A private database per test makes the global claim see only that test's
rows, so both the lock and the unscoped DELETE are deleted.
**DB-backed suites fail loud instead of skipping green.** ~15 manager-core suites
`return None` when DATABASE_URL is unset and report PASS — so in any environment
that lost its database the entire integration surface reports green while running
nothing (why the CI gap went unnoticed for so long). New
`picloud_test_support::abort_if_db_required` panics when `PICLOUD_REQUIRE_DB` is
set (CI now sets it) but DATABASE_URL is not, injected into each suite's skip
path. Local runs without the var still skip cleanly.
Mutation-verified: with PICLOUD_REQUIRE_DB=1 and DATABASE_URL unset, a suite
panics; without the var it skips.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`creating_a_group_requires_group_admin_on_the_parent` asserted only that the
member's apply exited non-zero — so it would pass even with the GroupAdmin check
removed, because the apply would still fail for an unrelated reason (a 404 on the
attach-point lookup, a credential error, a 500). It now asserts the stderr carries
`HTTP 403` — specifically the authz denial — matching the rest of the RBAC suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`PICLOUD_SESSION_ABSOLUTE_TTL_HOURS` (audit C1, migration 0070) is the whole
mitigation for a stolen-but-warm admin token, and nothing tested it — grepping
`absolute_expires_at` across all tests returned zero hits.
New DB-backed suite pins the authoritative enforcement: the lookup filter
`absolute_expires_at > NOW()`. A session with a future sliding `expires_at` but a
past absolute cap must NOT resolve (the stolen-but-warm case), and `touch` cannot
resurrect it even when it pushes `expires_at` a year out — the filter is the
backstop regardless of the middleware clamp.
Mutation-verified: dropping the `absolute_expires_at > NOW()` predicate makes both
tests fail. Runs on a private database via picloud-test-support.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLAUDE.md calls the KV/docs/pubsub/queue value-size caps an anti-DoS rail:
oversized payloads are rejected "before authz so anonymous public scripts can't
DoS Postgres." Only queue had a test, and it used an allow-all authz + anon cx —
which catches a DROPPED cap but not a REORDERED one, because an anon cx passes
script_gate regardless.
Each service now has an ordering-proof test: a DENYING authz repo + an
AUTHENTICATED member cx, so a size-check-first service returns *TooLarge while an
authz-first one would return Forbidden. Each pairs it with an under-cap control
through the same denied cx (returns Forbidden) to prove the cx really is denied,
so the *TooLarge case genuinely bypassed authz. Queue's pre-existing test is
upgraded to the same shape (+ a shared member_cx helper).
Mutation-verified: moving the KV size check after authz flips its result to
Forbidden and the test fails.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three tests that would pass against a broken implementation:
- `kv`/`docs` cross-app isolation asserted only that app B sees nothing — an
execution_id-keyed (or script_id-keyed) bridge would pass that AND every
single-execution round-trip, with app-scoped persistence entirely gone. Added
the positive control: app A re-reads its own value in a LATER execution (a fresh
execution_id/script_id via `baseline_request`), which only holds if storage is
keyed by app_id.
- `retry_policy_rejects_bogus_backoff` asserted a bare `is_err()`, which passes if
`retry::policy` is unregistered (ErrorFunctionNotFound), on a script typo, or if
every policy is rejected. Now it pins the message (`backoff`) and adds a
valid-backoff control that must succeed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`invoke_callee_sees_incremented_depth` asserted NOTHING — it ended `let _ = resp`
with a comment that `ctx.trigger_depth` wasn't exposed, so it would have passed
even if `invoke` forwarded the depth unchanged, i.e. never incrementing the
chain-depth counter that bounds runaway trigger loops.
`build_ctx_map` now surfaces `ctx.trigger_depth` (read-only: 0 for direct ingress,
+1 per synchronous `invoke` re-entry or dispatched handler) — the value the
author intended to read and a legitimate diagnostic for a script. The test now
pins the increment: the caller is depth 0, the invoke callee must see 1.
Mutation-verified: forwarding the depth unchanged makes the callee see 0 and the
test fails.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`GroupQueueServiceImpl` shipped with ZERO tests; `GroupPubsubServiceImpl` had four
that exercised neither boundary — its resolver always resolved and its principal
was always an instance Owner (who bypasses the capability check). So on the two
newest shared services, both the isolation boundary and the anonymous fail-closed
gate were unpinned — and they matter most here: a shared-queue consumer is
materialized into EVERY descendant app, so an unauthorized enqueue is
attacker-controlled input fanned out as script execution across a whole subtree.
Both now get the pair their three older siblings have:
`unrelated_app_gets_collection_not_shared` (with a positive control, so a
reject-everything impl can't pass it) and an anon/non-editor fail-closed test that
also confirms reads (depth) stay open.
Mutation-verified: swapping `script_gate_require_principal` for `script_gate` —
the one-token edit that would let an unauthenticated public route enqueue into any
ancestor group's shared queue — makes the anonymous enqueue succeed, and the test
catches it exactly there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`resolve_owning_group`'s ancestor-chain CTE is the isolation boundary for EVERY
group shared collection — kv, docs, files, topics, queues. Its own doc says so.
Yet no test exercised the actual SQL: the tests that claimed to
(`unrelated_app_gets_collection_not_shared` in the three older group services,
`realtime_authority`'s SSE 404) inject a `FakeResolver` HashMap the test itself
populates — the foreign app resolves to `None` only because the test never
inserted it. They prove the service propagates a `None`; they cannot see the
query that produces it. Dropping `JOIN chain c ON gc.group_id = c.group_owner`
would let ANY app resolve ANY group's shared collection — a cross-tenant breach —
and the whole unit suite would stay green.
This drives the real query over a real tree: a descendant resolves, a
sibling-subtree app gets nothing, the kind filter is part of the boundary (a kv
and a docs collection of one name are distinct stores), nearest-declaration
shadows (security-relevant — a deep app must reach the nearer group's store), an
app-owned marker shares with nobody, and lookup is case-insensitive.
Mutation-verified: with the chain join dropped, the boundary assertion fires — and
under that breach `app_mid` resolves a FOREIGN group's `catalog` left in the DB by
another suite, i.e. the leak reproduces live.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Turning CI on surfaced `atomic_write` as intermittently failing under load —
self-inflicted. Its fault injection is `CREATE TRIGGER ... ON outbox`, which takes
an ACCESS EXCLUSIVE lock on a table every other suite is concurrently inserting
into. The header even claimed it "cannot affect any test running in parallel";
that was wrong — scoping the trigger by app_id bounds which ROWS it rejects, not
the table lock installing it takes. Shipping that alongside "run the DB tests in
CI" would have poisoned the signal.
The fix is the one already used for the e2e suites: a private database per test.
That logic (build a migrated template once, clone it per test via
`CREATE DATABASE ... TEMPLATE`) was duplicated in `picloud/tests/common`, and this
would have been a third copy — so it moves into a shared `picloud-test-support`
dev-dependency crate, and `picloud/tests/common` now re-exports it. The invariant
it encodes: manager-core's claim loops are global by design, so a test must not
share a database with anything that runs them OR with anything that does DDL.
Moved onto it:
- `atomic_write` — the DDL fault injection above; its per-test cleanup is now
deleted (the database is thrown away).
- `outbox_reclaim` — asserts `reclaim_stale_claims` returns exactly 0, a count
over the WHOLE outbox table. On the shared DB any stale row from any other suite
(or a killed picloud that died holding a claim) would fail it permanently, until
someone cleared the table by hand.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI ran `cargo test --workspace` with no `--include-ignored`, so it executed 927
tests and skipped 237 — every DB-backed integration test is
`#[ignore = "needs DATABASE_URL..."]`, which covers ALL of api.rs, ALL of
authz.rs, and the entire CLI journey suite. The isolation and RBAC tests existed
but never ran (AUDIT.md F-Q-014, logged and never remediated). CI provides
Postgres, so it can run them.
Three things had to be right for that to go green:
- **`--all-targets`, not a bare workspace run.** `-- --include-ignored` un-ignores
not just `#[ignore]` tests but also ` ```ignore ` DOCTESTS, which are
illustrative pseudocode that does not compile. `--all-targets` runs lib/bins/
integration tests but excludes doctests (the same reason clippy uses it); a
separate `--doc` step runs the doctests without the flag. Structural, so a
future pseudocode doctest can't silently break CI either.
- **The CLI journeys are their own step.** They spawn a real picloud whose
dispatcher/orchestrator claim loops are global by design; run concurrently with
the manager-core suites on the shared database they would claim those suites'
outbox and workflow rows. Sequential steps keep the live server off the DB
while the other suites use it. The step also rebuilds `-p picloud` first (the
harness execs the prebuilt binary) and sets the dev-mode env the server needs.
- **A higher `max_connections`.** `#[sqlx::test]` pools are lazy, but mass-parallel
test startup briefly opens many at once (each test creates its own throwaway
database); on a many-core box that transient spike exceeded the default 100 and
Postgres answered "sorry, too many clients already". Steady-state peak is only
~26; 500 absorbs the spike with room to spare. Serving the app needs nothing
like this many. (This is the local compose ceiling; a small CI runner's default
100 has ample headroom for its lower parallelism.)
Also fixes the test the CI gap had let rot: api.rs asserted `v["schema"] == 66`
with a hand-bump comment, and since nothing ran it, it sat broken from migration
0066 to 0073. It now asserts `/version` surfaces the live constants
(`migrations::latest_version()`, `SDK_VERSION`) — the wiring — while value drift
stays caught by schema_snapshot + check-versioning. A constant hand-synced to
another constant is a chore, not a test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`dispatcher_e2e` failed about one run in three, on a different test each time,
and never under `--test-threads=1`. Root cause, not timing:
Each test calls `build_app`, which spawns a REAL dispatcher, and they all shared
one database — so they shared one `outbox`. `OutboxRepo::claim_due` is
deliberately not app-scoped (in production one dispatcher serves the whole
instance, so claiming any due row is correct). Test A's dispatcher would
therefore claim test B's row, test A would finish, its server would be dropped
mid-dispatch, and the claim was stranded. Test B polled for its handler's side
effect until it timed out.
The harness was modelling something production never does: N independent
instances sharing one database. So the fix belongs in the harness. Weakening the
dispatcher (app-scoping `claim_due`) would be wrong, and shortening
`PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` to fit a 10s test window would make production
abandon the live claims of scripts that may legitimately run 300s.
`tests/common` now hands each test its own database. Replaying 73 migrations per
test is too slow, so the first caller builds a migrated TEMPLATE database once
and every test clones it (`CREATE DATABASE ... TEMPLATE` is a file copy), guarded
by a cluster-wide advisory lock because Postgres will not clone a template that
has an open connection. Names are derived from (suite, test), so a test reclaims
its own database on entry: a crashed run leaves nothing to clean up.
Only the five local `pool_or_skip` wrappers change — all 33 call sites are
untouched, and the skip-when-`DATABASE_URL`-is-unset contract is preserved.
(`#[sqlx::test]` already does this and is what `api.rs`/`authz.rs` use, but it
requires `DATABASE_URL` and would force `#[ignore]`, silently dropping these
tests from the default `cargo test --workspace` gate.)
This also stops the e2e tests leaving stranded claims behind in the shared dev
database, which is the likely source of the occasional lone journey failure.
Before: dispatcher_e2e red ~1 run in 3. After: workspace green twice over,
journeys 157/157.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`with_events()` and `with_atomic_writes()` both install a writer, last call wins.
Appending them in the wrong order in the host would swap the transactional writer
back out for the best-effort one — silently reverting the write-path invariant AND
dropping the per-group quota enforcement that lives in it, with nothing in the type
system to catch it.
The services now carry an `atomic` flag: once `with_atomic_writes` has run,
`with_events` warns and no-ops instead of clobbering it. Also enforce the group
byte ceiling in `BestEffortGroupFilesWriter::update`, so the non-transactional
writer can't be used as an unmetered bypass and a unit test can catch its removal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stale-claim reclaimer added earlier in this branch un-claimed every timed-out
row, including SYNCHRONOUS ones. `reply_to.is_some()` is the schema's "never
retry" signal (0009_outbox.sql) and the dispatcher honours it: those rows are an
HTTP request someone is blocked on, and re-running one means re-running its side
effects — a charge, an email, an external POST — for a caller that is long gone.
So a dispatcher dying mid-sync-dispatch would have its row resurrected 10 minutes
later and executed a second time, for nobody. Before the reclaimer those rows were
merely stranded; the reclaimer turned inert into harmful.
The reclaim now gates on `reply_to IS NULL`. Stale sync rows are DELETED instead:
they can never be dispatched again and nobody is waiting on them, so leaving them
would just accumulate garbage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-review of this branch caught a data-loss bug I introduced with the
per-app / group files ceilings: the quota check ran AFTER the blob write.
`write_atomic_at` renames the new bytes over the FINAL path, so by the time
the ceiling refused an update the previous bytes were already gone. The
per-app path then unlinked the blob (row survives, file destroyed — every
read 404s); the group path left the new bytes in place under the old row's
checksum (every read fails `Corrupted`). Either way a user permanently lost
a file merely by exceeding a quota — a strictly worse outcome than the
unchecked-update bypass the ceiling was added to close.
The check now precedes the write on create AND update, per-app and group. As
a bonus this stops an over-quota caller driving unbounded write+unlink disk
churn: the ceiling now bounds I/O, not just stored bytes. The blob still goes
down inside the transaction, under the advisory lock, so a rollback unlinks
it and nothing can reference it in between.
The original test passed against the bug: it asserted the refused update did
not change the stored byte TOTAL — true, while the blob was already
destroyed. The regression test now reads the file back through the
checksum-verifying `FsFilesRepo::get`, and was confirmed to fail with
`Corrupted` against the old ordering.
Also in the KV writer (same file): drop the redundant pre-read on the hottest
write path. `set`/`set_if` did a SELECT purely to learn whether the write
would add a row, when the upsert already returns the previous value. Check
after the insert instead (`>` not `>=`) and let the transaction roll back on
refusal — identical outcome, one round-trip fewer, and an update pays nothing
at all. `kv_repo::get_on` and `FsFilesRepo::final_path` are now unused and
deleted; `FilesRepo::delete` delegates to the `delete_meta_on` + `unlink_blob`
helpers rather than re-implementing them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`users_service` still write-then-emits at nine call sites. That is currently
INERT — `triggers.kind` has no `users` value, so the outbox emitter has no arm
for the source and drops the event — but it is the exact shape audit #6
removed from every other stateful service, and it would be reintroduced by
whoever adds a users trigger kind and finds the plumbing already there.
Note that this one does not even log the failure (`let _ = ...`).
Documented at the emit site, pointing at `atomic_write` and the write-path
invariant in CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chasing the e2e flakiness turned up a production durability bug, not a test
bug.
The dispatcher claims an outbox row, executes it, then either deletes it
(success) or reschedules it (failure) — both of which clear the claim. If
the PROCESS DIES in between, neither runs. And `claim_due` only ever selects
`claimed_at IS NULL`. Nothing else in the codebase clears `outbox.claimed_at`
— grep it: there are exactly three writers, and those are two of them.
So a crash or restart mid-dispatch stranded every in-flight row PERMANENTLY.
Its trigger never fired and no retry could notice. The outbox is the
universal trigger path, so the loss covered kv / docs / files / cron /
pubsub / email / invoke_async / dead-letter alike. This is the same
durability class as the audit's #6 (lost trigger event), which was just
fixed at the WRITE end — this is the same hole at the READ end.
Every other claim-based store already had the safety net: `queue_messages`
and `group_queue_messages` have `reclaim_visibility_timeouts`,
`workflow_steps` has its own reclaim. The outbox was the one that didn't.
`OutboxRepo::reclaim_stale_claims(timeout)` returns rows whose claim is older
than `PICLOUD_OUTBOX_CLAIM_TIMEOUT_SEC` (default 600s), run from the
dispatcher's existing reclaim ticker. The default is deliberately generous —
a script may run for up to 300s (the `scripts.timeout_seconds` CHECK), so a
claim held past twice that is abandoned rather than slow; reclaiming a row a
LIVE dispatcher is still working on would double-execute it (survivable —
dispatch is at-least-once — but not worth courting).
A reclaim does NOT bump `attempt_count`: the handler never ran, so it must
not consume the row's retry budget, or repeated restarts would dead-letter an
event that executed zero times. (Same reasoning as the transient queue
`release` fixed earlier in this branch.)
This is also the root cause of the flaky e2e suites: each test drops its
dispatcher, and `claim_due` is not scoped per app or per dispatcher, so a
test's dispatcher could claim another test's row and strand it on teardown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An app's OWN store had no ceiling of any kind — only the group-SHARED
collections did, which is the inversion of what you'd expect. And
`authz::script_gate` returns `Ok(())` when there is no principal, so a
PUBLIC UNAUTHENTICATED route could reach `files::collection(..).create(..)`
and write blobs in a loop until the disk filled. `PICLOUD_FILES_MAX_FILE_SIZE_BYTES`
caps ONE file at 100 MB; nothing capped the count. Same for KV keys and docs
against Postgres.
Ceilings: `PICLOUD_APP_KV_MAX_ROWS` / `PICLOUD_APP_DOCS_MAX_ROWS` (100k) and
`PICLOUD_APP_FILES_MAX_TOTAL_BYTES` (10 GiB).
They are enforced DIFFERENTLY from the group ones, on purpose — porting the
group design as-is would have been a serious regression:
* KV/docs check a ROW count, on INSERT only, with NO lock. `kv::set` is the
hottest write path in the system; taking a per-app advisory lock on every
set would serialize an app's entire data plane, and a `SUM(...)` scan would
make write cost grow with the app's stored size. What the lock buys is
small — unlocked overshoot is bounded by write concurrency (32) against a
ceiling of 100k, ~0.03%. These are anti-DoS rails, not billing. An update
adds no row and is bounded by the per-value cap, so it pays nothing at all.
* No per-app BYTE ceiling for KV/docs: every value is already capped at
`PICLOUD_KV_MAX_VALUE_BYTES`, so `max_rows x max_value_bytes` is ALREADY a
finite bound. A second scan would buy nothing.
* FILES are the exception and DO lock + sum: one blob may be 100 MB, so 32
racing uploads could overshoot by gigabytes of real disk, and a file write
is heavy enough that the lock and the SUM are lost in the noise. Checked on
create AND update — an update that skipped the ceiling would be a free
bypass (the same bug just fixed for group files: grow a 1-byte file to
100 MB, repeat).
`group_quota` is renamed `quota`: it now serves both owners, and the module
doc is where the two enforcement strategies are contrasted.
Pinned by tests/atomic_write.rs: the key ceiling refuses a new key while
still allowing an update at the ceiling (rejecting updates would brick an app
the moment it filled up — worse than the DoS the rail exists to stop), and 10
concurrent uploads cannot exceed the disk ceiling, nor can an update grow past it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three rules a future change to the data-plane write path must not
break, and the two that are counter-intuitive enough to be re-broken:
* everything inside a tx runs on that tx's connection (reaching for a
second pooled connection while holding one can deadlock the pool), and
`shared` must stay sqlx-free — the transactional path is manager-core
internal, the `ServiceEventEmitter` trait stays connection-free;
* a per-group quota needs a LOCK, not just a transaction — under READ
COMMITTED each tx's COUNT/SUM sees a snapshot without the others'
uncommitted rows, so concurrent writers all pass the check anyway
(measured: 19 rows stored against a ceiling of 5);
* files order around the disk write, which cannot join a transaction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit #6 and #8 for the last of the three stores, plus a bypass found on
the way.
**#6.** create/update/delete wrote the metadata row, then emitted
best-effort, so an outbox failure left a committed file whose trigger never
fired. `atomic_write::FilesWriter` commits the metadata row and the fan-out
together.
Files are the one store where the ordering is subtle, because the BYTES live
on disk and cannot join a transaction:
* create/update — blob first, then commit metadata + fan-out. A rollback
unlinks the blob. (A crash at that exact point still orphans it; that
hazard predates this change — the repo already wrote the blob and then
inserted the row in a separate, failable statement — and the orphan is
inert, referenced by nothing.)
* delete — commit the metadata removal + fan-out FIRST, then unlink. The
reverse order would destroy the bytes of a row that a rollback keeps,
leaving a file that can never be read.
**#8.** `GroupFilesService::create` read `total_bytes` on one connection and
wrote on another, so concurrent uploads each saw the same pre-write total and
together overshot the ceiling. This is the worst instance of the race in the
codebase: the ceiling is DISK (10 GiB by default) and one file may be 100 MB,
so a racing fleet overshoots by gigabytes. `PostgresGroupFilesWriter` takes
the per-group advisory lock (on its own `files` key) across the check and the
write.
**The bypass.** `GroupFilesService::update` checked NO quota at all — so a
1-byte file could be updated to a 100 MB one without the ceiling ever being
consulted, repeatedly, for unbounded disk. It now checks the projected total
(the replaced file's bytes subtracted in SQL, so a same-size-or-smaller
update near the cap still goes through).
Also drive-by: `queue_e2e` asserted the ack the instant the marker appeared,
but the marker is written DURING the handler and the ack happens after it
returns — a zero-tolerance race. It polls now. (This does not fix the
suite's flakiness, which reproduces on the pre-pass commit too.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit #6 and #8, applied to the docs store — the same two bugs KV had, in
the same two places.
Per-app docs (#6): create/update/delete wrote the row, then emitted
best-effort. An outbox failure left a committed doc whose trigger never
fired. They now go through `atomic_write::DocsWriter`, whose Postgres impl
writes and fans out on one connection in one transaction.
Group docs (#8): the service read `count_rows`/`projected_total_bytes` on
pooled connections and wrote on another, so concurrent creators each saw the
same pre-write count and together overshot the ceiling.
`PostgresGroupDocsWriter` takes a per-group advisory lock — on its OWN
`docs`-namespaced key, so it serializes against other docs writers to that
group but not against KV writers, which draw on a separate ceiling.
The bespoke `check_total_bytes` (with its own copy of the upper-bound fast
path) is gone; group docs now shares `group_quota::check_group_write` with
group KV, so the row ceiling, the projected-bytes ceiling, and the fast path
that skips the O(n) SUM scan have exactly one implementation between them.
tests/atomic_write.rs gains the docs row-quota race (16 concurrent creators
against a ceiling of 4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit #8. `GroupKvServiceImpl` read the group's usage (`count_rows` /
`projected_total_bytes`) on one pooled connection and wrote on another. N
concurrent writers therefore each observed the same pre-write total, each
concluded it had room, and together sailed past the ceiling. The code
called this out as "best-effort ... quotas are safety rails, not exact
accounting" — but the overshoot is not small. With a ceiling of 5 and 20
concurrent writers, 19 rows land.
Route group-KV mutations through `atomic_write::GroupKvWriter`, whose
Postgres impl runs the quota reads, the row write, and the shared-trigger
fan-out on ONE connection in ONE transaction — and, crucially, takes a
per-group `pg_advisory_xact_lock` first.
The lock is the fix, not the transaction. Putting the check inside the
writing tx is necessary but NOT sufficient: under READ COMMITTED each
transaction's `COUNT(*)`/`SUM(...)` still sees a snapshot without the other
writers' uncommitted rows, so they all still pass. Serializing the
check-then-write per group is what makes a writer see its predecessor's row.
The lock key is namespaced per kind (kv/docs/…) so writes drawing on
different ceilings don't contend. A delete takes no lock — it only frees
space.
The quota POLICY (row ceiling, projected-bytes ceiling, and the
upper-bound fast path that skips the O(n) SUM scan for a group nowhere near
its cap) moves to one place, `group_quota::check_group_write`, over a small
`GroupUsage` trait. Both backends — the transactional one reading through
`&mut *tx`, and the in-memory one the unit tests use — share it, so there is
exactly one copy of the decision.
`set_if` gains a correctness nicety on the way: the row ceiling now keys on
whether the write ADDS a row, so a compare-and-swap against an absent key
with a `Some` precondition (which cannot swap, and so consumes nothing) is
no longer charged for one.
tests/atomic_write.rs pins both ceilings against 20/16 concurrent writers.
Both tests fail without the lock (19 rows stored against a ceiling of 5).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit #6. `KvServiceImpl` wrote the row, waited for it to commit, then
asked the emitter to resolve matching triggers and insert outbox rows —
a second transaction on a second connection. If that second step failed,
the row was permanently in the store with its trigger having never
fired: invisible to the caller (the write "succeeded"), unrecoverable by
any retry, and only ever logged. The code said as much:
// Audit finding (Medium): this is non-transactional with the data
// write — the row above has already committed by the time emit()
// runs, so an emit failure means triggers silently don't fire.
Introduce `atomic_write::KvWriter` — the mutating half of the service (the
write AND the fan-out it produces), so the two can share a transaction:
* `PostgresKvWriter` opens a tx, writes via `kv_repo::*_on(&mut *tx, …)`,
runs the fan-out on the SAME connection via `emit_on(&mut *tx, …)`, and
commits. An emit failure now rolls the write back and surfaces to the
script, which is the honest outcome — the caller learns the write did
not happen instead of silently getting a store that disagrees with its
triggers.
* `BestEffortKvWriter` keeps the old write-then-log-on-emit-failure
semantics for the in-memory unit tests (no Postgres).
Both sit behind one trait, so the service body has a single code path and
the choice is a constructor detail (`with_atomic_writes(pool)` in the host).
Pool-deadlock rule, documented on the module: everything inside the tx runs
on the tx's connection. A writer that held a tx and then reached for a
second pooled connection could starve — the pool is sized to the execution
concurrency cap, so N executions each wanting 2 connections deadlock. The
fan-out takes `&mut *tx` and never touches a repo.
`tests/atomic_write.rs` pins it by injecting an outbox failure (a Postgres
BEFORE-INSERT trigger scoped to the test's own app_id, so parallel tests are
unaffected): the set errors and the key is NOT in the store; likewise a
delete rolls back rather than dropping a key nothing downstream hears about.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`OutboxEventEmitter` resolved matching triggers and inserted outbox rows
through `Arc<dyn TriggerRepo>` / `Arc<dyn OutboxRepo>`, i.e. against the
pool — each query on whatever connection it happened to get. That makes a
transactional outbox impossible: the data write and the outbox rows can
never share a transaction, so a crash (or an outbox error) between them
silently loses the trigger event.
Take the emitter down to a connection instead of a pool:
* `outbox_repo::insert_on(exec, row)` and `trigger_repo::list_matching_on`
/ `list_matching_shared_on(exec, ...)` are generic over `PgExecutor`, so
the same SQL serves a pooled connection or a `&mut *tx`. The repo trait
methods delegate to them — the SQL keeps exactly one home.
* `emit_on` / `emit_shared_on` take a `&mut PgConnection` and run the whole
fan-out on it. The `ServiceEventEmitter` impl acquires one pooled
connection and calls them, so behaviour is unchanged today; a caller
holding a transaction can now pass `&mut *tx` and have the outbox rows
commit with the write.
* `OutboxEventEmitter::new` takes the `PgPool` directly (it was only ever
constructed once, in the host wiring).
Also collapses the three copy-pasted per-app match queries (kv/docs/files
differ only in the `kind` discriminator and detail table) and the three
`emit_*` bodies into one `plan()` + one match fn, so the suppression
anti-join, the chain walk, and the empty-ops-means-any-op semantic each
exist once rather than three times.
No behaviour change — pure refactor. It is the seam the transactional
write lands on next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §11.6 M4 byte quota mixed two incompatible byte measures: `used` came from
Postgres as `octet_length(value::text)` (jsonb CANONICAL text — whitespace after
`:` and `,`, keys normalized), while `old_len`/`new_len` were computed in Rust
with `serde_json::to_vec` (COMPACT, no spaces). The projection
`used - old_len + new_len` therefore subtracted and added a *smaller* measure
than what is actually stored, so a group could be admitted over its ceiling —
the cap drifted permissive. The two forms are not reconcilable in Rust (jsonb
also reorders/dedups keys), so the projection has to be measured by Postgres.
Add `projected_total_bytes` to `GroupKvRepo` (keyed by collection+key) and
`GroupDocsRepo` (keyed by the replaced doc id, `None` on create). Each computes
`SUM(canonical) - existing_row(canonical) + new_value(canonical)` in ONE query,
binding the new value as compact TEXT and re-parsing it through `::jsonb::text`
so PG measures it in exactly the form it stores. All three terms now share one
metric. The services call it instead of the mixed Rust math; the near-cap
fast-path (`rows_after * max_value_bytes <= ceiling` → skip the SUM) is kept, so
the common case still does no extra work. The docs `update` path no longer needs
its old-doc fetch (the subtraction happens in SQL), removing a read.
The in-memory test repos implement the same projection with their own (compact)
metric, so the quota unit tests keep their exact byte arithmetic. No schema
change.
Note: the check-then-write TOCTOU (concurrent writers can each pass and
collectively overshoot) is NOT addressed here — it needs a transaction spanning
the check and the write, the same missing infrastructure as the transactional
outbox. Tracked separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The one-consumer-per-(app_id, queue_name) invariant has no DB unique index (the
parent's app_id and the detail's queue_name live in different tables), so it is
enforced by a pg_advisory_xact_lock on hashtext(app_id||queue_name) around a
SELECT-then-INSERT. But `materialize` guarded its own check-then-insert only
with the GLOBAL rematerialize lock, not that per-(app,queue) key — so a
tree-apply / app-create rematerialize could interleave with an interactive
`create_queue_trigger` for the same (app, queue): each SELECT sees no consumer
(the other's INSERT uncommitted), both INSERT, and the app ends up with TWO
enabled consumers draining the same queue, splitting messages unpredictably.
Fix: `materialize_one` now resolves the template's queue_name and takes the SAME
`advisory_lock_key(app_id, queue_name)` (exposed `pub(crate)`) before its slot
check + insert, so the two paths mutually exclude. No deadlock: the interactive
path never acquires the global rematerialize lock, so there is no lock cycle,
and the rematerialize lock already serializes reconcilers against each other.
No schema change.
stateful_templates DB + journey coverage stays green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`trigger_depth` bounds how DEEP a trigger/invoke chain runs, but nothing bounded
how WIDE one execution could fan out. A single anonymous request running
`for i in 0..1_000_000 { invoke_async("w", #{}) }` costs a few Rhai ops plus one
cheap outbox INSERT per iteration, so within the op / wall-clock budget it could
write ~10^5 durable rows — each dispatched as its own execution — flooding the
outbox and dispatcher (a durable-amplification DoS).
Add a per-execution ceiling on DURABLE emissions (`invoke_async`,
`pubsub::publish_durable`, `queue::enqueue`, and the shared-topic/-queue
variants), env `PICLOUD_MAX_EMISSIONS_PER_EXECUTION` (default 1000). It's a
thread-local counter wrapped by a re-entrancy-aware `EmissionBudgetScope` around
every `execute_ast`: the OUTERMOST scope resets it, so a fresh dispatched
handler (a new pooled-thread task) gets a full budget while a synchronous
`invoke()` / interceptor re-entry nested in the same call stack SHARES it (fan-
out counted across the whole synchronous chain). The outermost Drop re-zeroes
the counter so a pooled thread never leaks a count into the next task.
Pinned by an `emit_budget` unit test (outermost resets, nested shares, ceiling
trips); the invoke/queue/pubsub/workflow journeys (small counts) stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group's template suppression is "coarse by reference" (path for routes,
handler-name for triggers), and both resolution points dropped ANY inherited
row matching that reference across the whole subtree — including one a NEARER
descendant group deliberately re-declared. So an ancestor group G that declines
a far-ancestor's `/x` (or `audit` handler) would also silently kill a child
group H's OWN `/x` / `audit`-bound trigger at the same reference, violating the
documented "an owner can only decline what it inherits, never a descendant's
own rows" invariant.
Fix: gate each decline on the chain DEPTH of the suppressor vs the target's
owner — a suppressor at depth `d_s` may only decline a row whose owner is
strictly ABOVE it (`target_depth > d_s`):
- Routes: `list_route_suppressions` now returns `(app, path, suppressor_depth)`;
the rebuild folds it to the min depth per `(app, path)` and
`compile_effective_routes` skips an inherited route only when
`route.depth > suppressor_depth`. (This also subsumes the old `depth > 0`
inherited-only gate.)
- Triggers: the dispatch anti-join gains `AND sc.depth < c.depth` (the
suppressor's chain depth below the trigger owner's), correlating on the outer
`chain c` that every kv/docs/files/pubsub match query already binds.
An app's own suppression is depth 0 → still declines anything it inherits; a
group's suppression declines only what that group itself inherits. `sealed`
still overrides. No schema change.
Pinned by a new `compile_effective_routes` unit test (a depth-1 descendant
route survives a depth-2 suppression that declines a depth-3 template) and a
new `group_suppression` DB test (same for triggers); all existing suppression /
sealed / template journeys stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`claim` pre-increments `attempt` before the handler runs, so a real execution
failure legitimately counts toward `max_attempts`. But the two TRANSIENT
release paths — gate saturation (server at capacity) and script-disabled-at-
fire-time — re-queue the message WITHOUT executing it, and previously used the
same `nack` that leaves the pre-increment in place. Under sustained overload a
message could therefore be dead-lettered after N gate-rejections having run
zero times (each rejection burning a retry).
Add a non-counting `release` to both `QueueRepo` and `GroupQueueRepo`
(re-queue + `attempt = GREATEST(attempt - 1, 0)`, mirroring `nack` otherwise)
and route the two transient paths through a new `Dispatcher::q_release`. A real
handler failure still uses `nack` and counts. No schema change.
Pinned by a new `queue_release` DB test (release undoes the claim increment;
nack keeps it) and the updated `disabled_queue_consumer_...` dispatcher test
(now asserts a release, not a nack).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-element Rhai sandbox caps (max_string_size 64 KiB, max_array_size /
max_map_size 10 000) do NOT bound the *materialized* size: Rhai shares strings
and arrays by Rc, so a 10 000-element array of one aliased 64 KiB string is
cheap to build (~10 000 ops, far under the 1 M op budget) yet dealiases to
~640 MiB of distinct JSON. Every Dynamic→JSON conversion deep-copies the
aliases; the HTTP response body path had NO size cap at all, and the KV/docs
value caps run only AFTER full materialization — so a handful of anonymous
requests could OOM the node (32 concurrent × ~640 MiB ≈ 20 GiB) on the stated
consumer-hardware target.
Add a byte-budgeted materializer that bails the moment the budget is exceeded,
so the transient allocation is bounded regardless of aliasing:
- bridge.rs: `dynamic_to_json_capped(value, max) -> Result<Json, JsonSizeError>`
(charges a running budget) + `MAX_JSON_MATERIALIZE_BYTES` (16 MiB hard rail,
far above the 256 KiB business caps). `dynamic_to_json` stays infallible for
best-effort paths (logging) but is now internally bounded — it collapses an
over-limit value to a marker string instead of OOMing.
- Route every user-value boundary through the fail-closed capped form: KV
set/set_if (+ the CAS `expected`), docs create/update/find, secrets set,
http request body, workflow input, `json::stringify`, and the HTTP RESPONSE
body (previously the one fully-uncapped exit → now a 500). `invoke` args and
the pubsub/queue message materializers get the same byte budget in place.
- `impl From<JsonSizeError> for Box<EvalAltResult>` so SDK sites protect
themselves with a bare `?`.
Pinned by bridge unit tests: an aliased 10 000×64 KiB array errors on the
budget rather than materializing, and the infallible wrapper yields a marker
instead of OOMing. Workspace 914 + journeys 157/157 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An end-to-end audit of the (unmerged) interceptor slice surfaced seven ways a
KV allow/deny guard could be silently defeated or misbehave. Close all of them:
1. Shared-collection bypass — `kv::shared_collection(...).set/delete` skipped
the hook entirely, so any `(kv, set/delete)` guard was circumvented by
choosing the shared handle. `GroupKvHandle` now runs the same before-op hook.
2. Seal to the declaring owner — the marker resolved nearest-owner-wins but its
script name was then re-resolved on the CALLER's chain, so a descendant app
could shadow a group's mandatory guard with a same-named local script.
`resolve_before` now returns the script sealed to the owner that declared the
marker (one LEFT JOIN in manager-core), so the executor never re-resolves by
name. A descendant can only override with its OWN explicit marker.
3. Re-entrancy — an "allow + audit-log" guard that itself wrote KV re-triggered
itself to the depth cap and then DENIED the original write (plus ~8x
resolve/compile/execute amplification). A thread-local guard makes a write
performed by an interceptor bypass interception.
4. Fail-closed verdict — only `#{ allowed: false }` denied; a bare bool, a
typo'd key, a non-bool, or a bare unit all ALLOWED. Now allow ONLY on an
explicit `#{ allowed: true }`; every other shape denies.
5/6. Fail-closed edges — a dangling (missing/disabled) interceptor script and a
missing engine back-reference now DENY instead of allowing.
7. AppInvoke coupling — resolution no longer routes through `invoke.resolve`, so
installing a guard no longer denies writes to principals who hold KV-write
but not AppInvoke.
The seam (`InterceptorService::resolve_before`) now returns an
`InterceptorResolution { None | Dangling | Run(ResolvedScript) }`; the executor
runs the sealed script straight through the shared `run_resolved_blocking` core
and drops its `InvokeService` dependency. executor-core stays Postgres-free.
No migration change (0073 unchanged).
Pinned by three new journeys in tests/interceptors.rs: the seal (a same-named
app script does NOT shadow the group's guard), shared-collection coverage, and
the fail-closed verdict. Full journey suite 157/157.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Smallest honest vertical slice of §9.4: a `[[interceptors]]` block (app OR
group) binds a script to run BEFORE `kv::set`/`delete`; it reads the operation
context (`ctx.request.body`: service, action, collection, key, value, caller
ids) and returns `#{ allowed, reason }` — `allowed == false` denies the op (the
write never runs, the caller gets a runtime error).
Reuses two existing mechanisms rather than inventing new ones:
- Registration mirrors extension points (§5.5): a marker table
`0073_interceptors.sql` (owner-polymorphic app_id/group_id XOR, keyed
(service, op) → script), `interceptor_repo` (insert/delete/list + the
nearest-owner-wins `resolve_before` chain walk), reconciled through the
declarative apply exactly like `vars` (create/update/delete, prunable).
- Execution reuses the `invoke()` re-entry path: the new `InterceptorService`
(shared trait + Postgres-backed impl) only RESOLVES the script name (keeping
executor-core Postgres-free); the executor's `sdk::interceptor::run_before`
resolves that name and runs it via `run_resolved_blocking` (extracted from
`invoke_blocking` — shared depth bound + AST cache). An un-hooked write pays
one indexed `Ok(None)` resolve; no interceptor ⇒ zero overhead.
Nearest-owner-wins so an app overrides a group's interceptor, and a group
interceptor is inherited by every descendant app — the chain walk is the
isolation boundary (a sibling subtree never matches). `validate_bundle_for`
restricts the MVP to `service = "kv"`, `op ∈ {set, delete}`, one marker per
(service, op).
Deferred (documented in §9.4): the `data` transform return, services other
than kv, `after_*` hooks, chaining + circular-dependency guard, the timeout
policy, and a `pic interceptors ls` read surface (needs a server route).
Pinned by `tests/interceptors.rs` (deny blocks the write; allow passes;
group→app inheritance), schema snapshot re-blessed. 154/154 journeys pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The blueprint, CLAUDE.md, and the groups design doc still described the
Workflows track as "not-yet-started" and listed a per-app `materialized`
column as deferred — both stale. Workflows shipped M1–M6 (DAG execution +
conditional `when`, nested sub-workflows, durable orchestrator,
`workflow::start` SDK + admin API + `pic workflows`, dashboard DAG +
run-history; migrations 0071/0072), and D1's `materialized` column shipped.
Update the status lines to mark Workflows shipped (leaving §9.4 service
interceptors as the one remaining Workflows-track item and cluster mode as
next), bump the "migrations through 0070" note to 0072, and drop the stale
`materialized`-deferred line. Docs-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two read-only operator surfaces existed server-side but the CLI was app-only:
- `pic dead-letters ls --group <slug>` — mirrors the app listing onto a group's
shared-queue dead-letters (§11.6 D3, `GET /groups/{id}/dead-letters`), via the
existing `require_one_owner`/`OwnerRef` dispatch. List-only (show/replay/
resolve stay app-only, matching the server's operator surface); a group DL row
shows its `collection` and carries no trigger, so it renders its own columns.
- `pic docs ls/get --group <slug>` — a new `Docs` command (`cmds/docs.rs`)
wrapping `GET /groups/{id}/docs[/{collection}/{id}]`, mirroring `pic kv`.
Group-only: per-app docs have no admin read route yet (unlike kv/files), so
there is no `--app` variant — noted for a later pass.
Read-only by design (writes go through the SDK). New client methods +
`GroupDocsListDto`/`GroupDeadLetterDto`. Pinned by a new
`operator_reads_shared_docs_and_group_dead_letters_via_cli` journey
(152/152 pass).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`ManifestScript`, `ManifestRoute`, and every trigger spec
(`KvTriggerSpec`…`QueueTriggerSpec`) lacked `#[serde(deny_unknown_fields)]`,
while their parent structs all had it. So a typo inside a `[[triggers.kv]]` /
`[[routes]]` / `[[scripts]]` entry (e.g. `collection` for `collection_glob`,
`methid` for `method`) was silently dropped instead of erroring — the manifest
applied with the mistyped directive missing. Add the attribute to the 9 entry
structs (all fields are exhaustive, so `pull`-emitted TOML still round-trips —
verified against the full journey suite, 151/151). Pinned by a new
`unknown_key_in_an_entry_spec_is_rejected` unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`users_service::require` open-coded the `AuthzDenied → {Forbidden, Backend}`
match that `authz::script_gate` centralizes for every other stateful service —
the one straggler. Extract the mapping into `authz::require_mapped` (used now by
both `script_gate` variants and `users_service::require`), so the verdict→error
mapping lives in exactly one place. Also refresh the stale `to_app_user` comment
(it referenced an unshipped "commit 8" stub; roles are resolved via
`fetch_roles` and passed in). No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two clean, local dedups in the group shared-collection services:
- Extract `GroupKvServiceImpl::check_value_size` — `set` and `set_if` inlined
the identical encode-and-cap block; now one method (mirrors the sibling
`group_docs_service::check_data_size`).
- Add `group_collection_repo::best_effort_emit_shared` — the KV/docs/files
services each copied the same emit-and-log-on-error tail for shared-collection
triggers. Centralize the tail (logging the event's own `source`/`op`); each
service still builds its own `ServiceEvent` (payload shapes differ).
Pure refactor, pinned by the existing group-service unit tests. (The
`owning_group` resolver was evaluated for dedup too but left as-is: the five
error enums diverge — `GroupFilesError::InvalidCollection` carries a `String`,
`GroupPubsubError` lacks the variant, and files/pubsub skip the empty-check kv/
docs/queue do — so a shared conversion would be a behavior change, not a
refactor.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`http.rs` hand-rolled its own `block_on` + `map_http_err`, and `secrets.rs` /
`email.rs` / `users.rs` each carried a byte-identical `runtime_err` — five
`#[allow(clippy::unnecessary_box_returns)]` across four files for one pattern.
Move `runtime_err` to `bridge.rs` beside `block_on` (single home, single allow)
and have the three bridges import it. `http.rs` now calls the shared
`bridge::block_on("http", …)` (same "http:"-prefixed error, so messages are
unchanged) and its `err` validation helper delegates to `bridge::runtime_err`.
Deletes 4 duplicate fns and 3 of the 5 allows; no behavior change (pinned by
the executor-core SDK contract tests).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-review flagged that the previous warn-and-skip (aeebdd2) traded a loud
failure for a SILENT prune hazard: against a server too old to return a
workflow's `definition`, pull dropped it with only a stderr warning, exited 0,
and wrote a workflow-less manifest — which a later `apply --prune` uses to
DELETE the server-side workflow (and `pull --force` would clobber a
hand-authored `[[workflows]]` block first).
A live workflow always has ≥1 step (zero-step is rejected at apply), so an
empty `definition` can only mean an out-of-date server. Pull now bails BEFORE
writing anything — consistent with the existing clobber / unsafe-name fail-fast
gates — so nothing partial hits disk and the operator gets an actionable error.
wire_workflow_to_manifest reverts to infallible (the guard upholds its
precondition).
Verified: fmt + clippy -D warnings clean; 5 workflow CLI journeys pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A re-review of eaf5ace surfaced three follow-ups, all in the fixes themselves:
- pull vs an old server: a returned empty `definition` can only mean a server
predating the definition-in-list field (a real zero-step workflow is rejected
at apply). Emitting a stepless `[[workflows]]` block would just fail the next
apply, so `wire_workflow_to_manifest` now warns and skips it instead.
- base_ms/backoff defaults: the pull mapper keyed its default-omission off a
hardcoded `500` literal. `default_base_ms` is now `pub` + re-exported, and the
mapper omits whatever equals `default_base_ms()` / `WorkflowBackoff::default()`
— no drift if a default ever changes.
- dashboard: a run that succeeds with an `on_error = continue` step failure now
carries a partial-failure note in `run.error` (from the compute_advance
change); the run-detail view rendered it as a red failure, contradicting the
green "succeeded" badge. It now renders as a `.notice` (warning) for a
succeeded run, `.error` only for a failed one.
Verified: fmt + clippy -D warnings clean; 450 lib tests; 5 workflow CLI
journeys (incl. the pull→plan round-trip); dashboard npm run check clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adversarial review of the v1.2 Workflows track surfaced two HIGH footguns and
several correctness/hardening gaps. Fixes:
HIGH
- pull round-trip: `pic pull` dropped `[[workflows]]`, so a later
`apply --prune` silently deleted them. The list endpoint now returns the full
`definition`; pull rebuilds the manifest block (inverse of workflow_to_wire).
- case-colliding names: two workflows differing only by case collided in the
reconcile diff (keyed by lower(name)), silently dropping one. Rejected up
front in validate_bundle_for.
MEDIUM
- reclaim retry budget: a crashed attempt (no outcome) consumed the retry
budget. reclaim_stale_steps now decrements `attempt` (floored) and clears
`next_attempt_at`, so a crash no longer counts as a failed try.
- on_error/backoff: typed the manifest fields against the shared enums so a
bad value fails at TOML parse with a clear message, not an opaque 500.
- dedupe depends_on: a repeated dependency inflated the Kahn in-degree past the
single decrement, reporting a spurious cycle for a valid DAG. Count distinct.
- canceled child: a canceled sub-workflow resolved the parent step with a
message naming the cause instead of a generic "failed"; documented that a
run-level cancel op is not yet supported.
LOW
- list_run_steps is now app-scoped at the query (JOIN workflow_runs) rather
than relying on caller pre-verification.
- partial failure is surfaced: a run that succeeds with on_error=continue
failures records them in the run's `error` field.
- admin-started runs log the root_execution_id against the principal.
- documented the deliberate when(missing→false) vs template(missing→fail)
asymmetry; corrected the claim's atomicity comment.
Tests: unit (dedupe-deps), DB-gated reclaim-budget assertion, and two new CLI
journeys (duplicate-name rejection, pull→plan clean round-trip). fmt + clippy
-D warnings clean; 450 lib + 14 orchestrator DB + 5 workflow journeys pass;
schema snapshot unchanged (no migration).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A per-app "Workflows" tab: list definitions, browse run history, and inspect a
run's per-step progress with a static layered DAG.
- API: the run-detail endpoint now returns each step's `depends_on` (loaded
from the definition via a new `get_workflow_by_id` reader) so the dashboard
can draw the graph edges — the run's step rows don't carry them.
- dashboard `$lib/api`: a `workflows` namespace (list / runs / start / run) +
the matching types.
- `apps/[slug]/workflows/+page.svelte`: workflow table → drill into runs →
drill into a run. The run view renders a step table plus an SVG DAG (nodes =
steps colored by status, laid out by longest-path level; edges = depends_on,
arrowed) and a "Start run" button. Nested/parked steps show their child run.
- AppTabBar + the per-app layout gain the Workflows tab (admin-only).
Tests: `npm run check` clean (0 errors); two Playwright smoke tests
(navigation tabs — workflows loads cleanly + renders its empty state) pass
against the full stack.
With M6 the v1.2 Workflows track (M1 schema/validation → M2 durable
orchestrator → M3 when/templating → M4 nested → M5 SDK/API/CLI → M6 dashboard)
is COMPLETE.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The user-facing surface for the durable DAG engine. A workflow run can now be
started three ways, all resolving the workflow by name in the caller's app
(`cx.app_id` / the resolved app — never a passed-in arg, the isolation
boundary), seeding a run the orchestrator advances.
SDK seam (shared):
- `WorkflowService` trait + `NoopWorkflowService` + `WorkflowError` in
`shared::workflow` (mirrors `InvokeService`); added to `Services` as a
noop-defaulted `with_workflow` builder (all `Services::new` call sites
unchanged).
- `WorkflowServiceImpl` (manager-core): resolves + seeds a run; authenticated
callers gated on `AppInvoke` (anonymous skips, script-as-gate), the same gate
`invoke()` uses.
- `executor-core::sdk::workflow` — the Rhai bridge `workflow::start(name, input)`
/ `workflow::start(name)`, registered in `register_all`. Returns the run id.
Admin API (manager-core `workflows_api`, mounted in the binary):
- `GET /apps/{app}/workflows` — definitions (AppRead)
- `POST /apps/{app}/workflows/{name}/runs` — start a run (AppInvoke)
- `GET /apps/{app}/workflows/{name}/runs` — run history (AppRead)
- `GET /apps/{app}/workflow-runs/{run_id}` — one run + its steps (AppRead)
`app_id` scopes every query; a foreign run 404s.
CLI (`pic workflows`): `ls` · `run <name> [--input JSON]` · `runs <name>` ·
`run-status <run_id>` — all `--app`-scoped; run definitions stay declarative
(`[[workflows]]` → `pic apply`).
Also: `list_runs_for_workflow` repo reader.
Tests: `workflow_service_start_seeds_a_run` (DB-gated — the SDK/API-shared
entry) + `workflow_run_executes_end_to_end` CLI journey (apply → `pic workflows
run` → poll `run-status` → succeeded, through the real binary + orchestrator).
fmt + clippy -D warnings clean, 449 manager-core lib tests, 14 orchestrator DB
tests, 18 executor-core lib tests, 3 workflow CLI journeys green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A `workflow`-kind step now starts a nested sub-workflow run instead of a
function. The orchestrator's step processing is restructured into prepare →
dispatch (`Prep`): after the shared context/`when`/input resolution, a
function step runs through the executor as before, while a workflow step:
- checks the nesting depth ceiling (`PICLOUD_WORKFLOW_MAX_DEPTH`; the child
would run at parent depth + 1) — over-deep → the step fails, not infinite;
- resolves the child workflow by name in the run's app scope;
- `start_child_and_park`: in one token-gated tx, seeds a child run (depth + 1,
`parent_run_id`/`parent_step_id` linkage, correlated under the same
`root_execution_id`) and PARKS the parent step (`running`, claim_token
cleared, `child_run_id` set). A parked step is never re-claimed (only `ready`
steps are) nor reclaimed (only *leased* running steps). The token-gated park
runs before the child insert, so a stale claim writes nothing (no orphan).
Each tick first runs `resume_finished_children`: a parent step parked on a
now-terminal child is resolved (child output → parent step output, or child
error → parent step failed) and the parent run advanced — idempotent via a
`status='running'` gate. The child's own steps are claimed by the same global
scan, so nesting is just more runs. A failed sub-workflow honors the parent
step's `on_error`.
Shared plumbing extracted for reuse: `advance_run_tx` (out of
`complete_step_and_advance`) and `seed_run_tx` (out of `start_run`).
Apply-time soft-warn (`workflow_nesting_warnings`, wired into `plan_warnings`):
a workflow that nests into itself — directly or via a mutual cycle within the
bundle — is flagged at plan (bounded by the depth ceiling, but almost always a
bug). Not an error.
Tests: nested output-flows-to-parent + depth-ceiling-fails-the-run (DB-gated,
end-to-end), self/mutual-cycle warning (pure unit). fmt + clippy -D warnings
clean, 449 manager-core lib tests, 13 orchestrator DB tests, 26 CLI journeys
(workflows/apply/plan/prune) green, binary boots.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the two pure M1 modules into the orchestrator's execute path:
- Before running a step, build a `RunContext` from the run input + every prior
succeeded step's output (`load_run_step_outputs`, read fresh at execution
time so a step sees all upstream results).
- `workflow_expr::eval` the step's `when` condition (if any): false → the step
is `skipped` — never executed — and counts as satisfied-but-empty for its
dependents (a new `StepOutcome::Skipped`, written in `complete_step_and_advance`
then advanced; `compute_advance` already treats skipped as satisfying deps).
- `workflow_template::resolve` the step's `input` against the context: an
exact single `{{ ref }}` preserves the referenced JSON type, an embedded ref
interpolates as text; a missing reference is a hard step failure (a definition
bug surfaced, not hidden), never a silent null.
`when` and templates were already parse-validated at apply time (M1); this is
the runtime half.
Tests (DB-gated, end-to-end via a scripted fake executor):
- `when_false_skips_step` — b skipped, never executed, omitted from run output
- `step_output_flows_into_downstream_input` — a's `{n:7}` feeds b's input,
type-preserved for a bare ref and interpolated in a larger string
- `missing_input_ref_fails_the_step` — an unresolved ref fails b → run fails
Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
11 workflow_orchestrator DB tests (3 new).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The v1.2 Workflows durable DAG engine gains its runtime. A dedicated
background worker (`workflow_orchestrator.rs`, mirroring `cron_scheduler`,
NOT folded into the dispatcher) advances every in-flight run step-by-step,
durably, surviving restarts.
Per tick, two phases:
A. Claim + execute — up to a small batch of `ready`, due steps are claimed
with the same `FOR UPDATE SKIP LOCKED` competing-consumer lease the queue
uses (`claim_ready_step`), one execution-gate permit per step acquired
BEFORE the claim so the shared gate bounds real parallelism. Each step
resolves its function by name in the run's app scope (never a
script-passed arg — the isolation boundary), builds an `ExecRequest`, and
runs through the injected `ExecutorClient`. Claimed steps run concurrently.
B. Advance — the outcome is written and the DAG advanced in one token-gated
transaction (`complete_step_and_advance`): pending steps whose deps are
satisfied flip to `ready`, and the run's terminal status is recomputed.
Fan-in falls out (a join waits until its last dep flips it); a stale worker
matches zero rows and writes nothing.
The graph-advance decision is a pure, DB-free function (`compute_advance`) —
promotions + terminal run status, folding `on_error` fail-vs-continue and
skipped/failed dependency satisfaction — so it is unit-tested in isolation.
Retry uses the step's own policy via `compute_backoff`; a second, slower
cadence reclaims steps leased by a crashed worker (`reclaim_stale_steps`) — the
durability safety net. Steps run with no principal (like invoke_async), and log
under the new `ExecutionSource::Workflow` so `pic logs` surfaces them.
M2 executes function steps only; `when` + input templating land in M3, nested
sub-workflows in M4. Seams present (`StepTarget`, `run_input`, `workflow_depth`).
- migration 0072: widen the `execution_logs.source` CHECK with `workflow`
- shared: `ExecutionSource::Workflow`, `StepStatus::is_terminal`
- workflow_repo: run/step state — `start_run`, `claim_ready_step`,
`complete_step_and_advance`, `reclaim_stale_steps`, `get_run`,
`list_run_steps`, `compute_advance` (+ 7 pure unit tests)
- workflow_orchestrator: the worker + config (`PICLOUD_WORKFLOW_*` knobs),
spawned in `picloud/src/lib.rs` beside the dispatcher/cron scheduler
- tests: 8 DB-gated integration tests (linear, parallel fan-out/fan-in, retry,
on_error fail/continue, double-complete idempotency, stale-lease reclaim,
end-to-end tick with a fake executor)
Verified: cargo fmt, clippy -D warnings clean, 448 manager-core lib tests,
8 workflow_orchestrator DB tests, schema snapshot reblessed, M1 workflow CLI
journeys still pass (binary boots with the orchestrator wired in).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First milestone of the v1.2 Workflows track (blueprint §9.1/§9.2): the durable
DAG engine's foundation — the definition model, its validation, and the
declarative `apply` reconcile path. No execution yet (the orchestrator is M2).
- Migration 0071_workflows: `workflows` (owner-polymorphic like scripts, app-
owned in M1), `workflow_runs`, and `workflow_run_steps` (the per-step
competing-consumer lease table the M2 orchestrator will claim). Schema golden
reblessed.
- `shared::workflow`: the `WorkflowDefinition` / `WorkflowStepDef` DTOs (shared
by the CLI, apply, and the future orchestrator) + run/step status enums.
- Pure, DB-free `workflow_template` (input `{{ input.x }}` / `{{ steps.a.output.y }}`
resolution, type-preserving) and `workflow_expr` (a small safe JSON-predicate
evaluator for `when` — keeps manager-core free of a scripting engine).
- `workflow_repo`: read trait + reconcile tx free-fns (insert/update/delete).
- apply_service: `BundleWorkflow` + `Plan.workflows` + `CurrentState.workflows`
+ `ApplyReport.workflows_*`; server-side `validate_workflow_definition`
(unique/acyclic steps via topological sort, deps exist, function XOR workflow,
`when`/template parse) rejecting on a group node; `diff_workflows` by
lower(name) (Update on a definition change) + in-tx reconcile.
- CLI: `[[workflows]]` + `[[workflows.steps]]` manifest structs → wire bundle;
plan/apply render + report counts.
- Tests: 16 manager-core lib tests (template, expr, definition validation) +
the `workflows` CLI journey (apply → NoOp → prune; cyclic DAG rejected).
Deferred to later milestones: the orchestrator worker (M2), conditional/template
runtime wiring (M3), nested sub-workflows (M4), the SDK/API/CLI run surface
(M5), the dashboard (M6). The `group_id` column + run/step tables ship now so
those slot in without a migration churn.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A five-agent roadmap-verification pass found the v1.2 Hierarchies code fully and
correctly implemented, but the three source-of-truth docs carried stale
"deferred/remaining/v1.3" notes — and one self-contradiction — for features that
have since shipped (§3 M3 server-side approval gate, §6 group create/reparent, the
Track A closeout, and the 2026-07-11 audit remediation). Reconcile all three to
reflect the true code; no behavior change.
CLAUDE.md: retract the inline "groups pre-exist" / "per-env approval deferred" /
"shared-topic SSE deferred" / "group dead-letter store deferred" / "email v0/no-AAD"
notes → mark §6, §3 M3, and Track A M1–M6 shipped; add per-group KV/docs byte
quotas + set_if; note migration 0063; refresh the "Out of MVP" section.
design doc: fix the header ("Phases 4–6 remain" → all shipped); resolve the §11.6
self-contradiction (the "Deferred" list still named shared-topic SSE / byte quotas /
set_if / operator admin API, all shipped — line 1317 already said SSE shipped); mark
the D3 dead-letter store shipped; retract the stale "groups pre-exist / §6 deferred"
forward references.
blueprint (comprehensive sweep): dashboard Alpine.js → SvelteKit + CodeMirror
(diagram, §3.3, tech table); Docker-per-execution → embedded in-process Rhai
(diagram + data flow); §12 Phase 4 "current focus" → shipped through v1.1.9; Phase 5
"in active development" + "Remaining" block → Hierarchies complete; per-app RBAC
"v1.3+" → shipped Phase 3.5; SDK reference → handle-pattern notation note, S3 tag
v1.1 → v1.3+; MVP schema + docker-compose flagged non-authoritative; §9 header noted
Hierarchies-shipped / Workflows-future; top status line refreshed.
Also fix two behavior-neutral stale in-code comments (sdk/kv.rs `kv::shared` →
`kv::shared_collection`; dispatcher.rs `q_terminal` "no group dead-letter store yet"
→ dead-letters to group_dead_letters, Track A M2).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediate the dashboard-coverage and editor findings from the 2026-07-11 audit.
Close the group-surface gap: a GroupTabBar plus read-only surfaces for the v1.2
group data plane the dashboard previously exposed only via the CLI — vars,
secrets (names + timestamps only, never values), shared collections (KV/docs/
files browsers), scripts, triggers, routes, suppressions, extension points,
dead-letters, and ownership. All server data renders through Svelte's escaped
interpolation (no `{@html}`); each tab hits its existing
`/api/v1/admin/groups/{id}/...` read endpoint with independent loading/empty
state.
B7 — the CodeEditor `readOnly` prop is now reactive via a CodeMirror
Compartment, fixing the race where an authorized user opening a script on a warm
SPA nav got a permanently read-only editor (role resolved after mount). The
reconfigure preserves editor content.
A viewer lacking a read cap now sees a calm "you don't have permission" note on
every tab (structured LoadError + classifyError) instead of a red error panel,
generalizing the one-off Secrets 403 handling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediate the CLI/DX findings from the 2026-07-11 audit.
B4 — `pic plan` now previews the apply-time desired-state warnings (disabled
binding, unreachable endpoint, dangling `[suppress]`), so plan and apply agree
for CI review. Wire fields are `#[serde(default)]` (older-server tolerant).
B5 — `pic apply` no longer mutates on a first run (no recorded plan) without a
preview + confirm, and a large blast radius now triggers an extra confirmation.
Both gates check `is_terminal()` before any read — a non-TTY never hangs on
stdin and fails closed without `--yes`; `--yes`/`--force` bypass headlessly.
`--force` help now notes it also skips these prompts.
C3 — `pic files ls`/`get` gain `--group`, mirroring `pic kv --group`, so the
shipped group shared-files admin surface is reachable from the CLI.
C4 — a shared `require_one_owner(app, group)` helper (cmds/mod.rs) gives one
canonical message for the `--app`/`--group` XOR, wired into every such site
(kv, files, vars, secrets, suppress, extension-points, triggers ls, scripts
deploy). routes ls (positional script_id) and scripts ls (lists all) keep their
own shapes deliberately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediate the MEDIUM backend correctness/perf findings from the 2026-07-11 audit.
B1 — a group cron template toggled disabled→enabled no longer re-fires every
descendant's cron. Materialization now KEEPS a disabled cron template's copies
(syncing `enabled` in place) instead of delete+recreate, preserving
`last_fired_at`; the scheduler already skips disabled copies, so keeping them is
inert. Queue/email arms still delete-on-disable (freeing the one-consumer slot).
B2 — `rematerialize_stateful_templates` warnings are now logged at the
app-create/delete and group-reparent chokepoints (were silently dropped); only
the `Err` arm was handled before.
B3 — the group KV/docs byte-quota check skips the O(rows) `SUM(octet_length(..))`
scan when a cheap upper bound (`rows_after * max_value_bytes`) is already under
the cap, so the full aggregate only runs near-cap. Every row is validated
≤ max_value_bytes, so the bound is sound (never lets an over-quota write through).
B6 — the in-process SSE broadcaster caps live channels per map
(`PICLOUD_REALTIME_MAX_CHANNELS`, default 100k); a subscribe that would open a
NEW channel past the cap is refused with 503 + `Retry-After: 1` rather than
growing the (app,topic)/(group,topic) maps unboundedly. Check+insert is atomic
under the map mutex (no TOCTOU); an existing-channel subscribe is exempt.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediate the HIGH and security-relevant findings from the 2026-07-11 audit.
H1 — the per-env approval gate is now server-authoritative. The governing
project is resolved from the target node's nearest-claimed ancestor
(`governing_env_policy`/`_tree` + `governing_project_id` +
`ProjectRepository::get_environments_by_id`), independent of the client-supplied
`[project]`. Omitting or spoofing the project block can no longer skip a gate the
owning project established; a to-create group resolves its declared parent's
chain so a fresh subtree node inherits the gate. Fails closed on any read error.
H2 — the API-key prefix slice (`&rest[..8]`) is now the boundary-safe
`rest.get(..8)`, so an attacker-supplied multibyte bearer can't panic the request
task (unauthenticated per-request DoS). Regression test added.
C1 — admin sessions gain an absolute lifetime cap (migration 0070,
`PICLOUD_SESSION_ABSOLUTE_TTL_HOURS`, default 30d): `lookup` filters it, `touch`
clamps the sliding bump to it, so a continuously-used or stolen-but-warm token
self-expires. Mirrors the data-plane app-user cap.
C2 — `Cache-Control: no-store` on the login and API-key-mint responses (the two
that return a raw credential), so a proxy/CDN/browser cache can't retain it.
B8 — file downloads are header-safe: `sanitize_stored_filename` guarantees a
valid `HeaderValue` (no panic on a control-char name) and BOTH the per-app and
group download paths now set attachment + `X-Content-Type-Options: nosniff` +
a restrictive CSP, closing a group-path stored-XSS gap.
Also folds in the server-side plan-warning plumbing (`plan_warnings`,
`PlanResult::warnings`) and the `app_only_reject` message helper that the CLI
plan-preview change builds on, plus operator security notes (reads-open shared-
topic SSE; the `--env` label is advisory, not a boundary).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shared TOPICS fanned out only to in-cluster trigger handlers; external clients
could not subscribe (per-app topics already can). Add SSE for shared topics.
- RealtimeBroadcaster gains a parallel (group_id, topic) channel map:
subscribe_group / publish_group / drop_group_topic (default no-ops so
NoopRealtimeBroadcaster + test doubles are untouched). InProcessBroadcaster
implements them with a second map; GC + channel_count span both.
- Route GET /realtime/shared/topics/{topic}: Host->app dispatch (as the per-app
route), then RealtimeAuthority::authorize_subscribe_shared resolves the OWNING
GROUP from the app's chain (kind=topic, root segment). Reads-open model — the
resolution IS the authorization, consistent with in-script shared reads; a
foreign-subtree app never resolves (404, the isolation boundary). No principal
machinery needed.
- GroupPubsubServiceImpl::with_realtime bridges a shared-topic publish to the
owning-group channel (best-effort) after the durable trigger fan-out.
- Host wires the broadcaster into the group pubsub service + the collection
resolver into the authority.
Auth-model note: chose reads-open (subtree app's Host is the grant) over
"authenticated principal + GroupKvRead" — it's both simpler and faithful to how
shared-collection reads already work. Pinned by realtime broadcaster group-map
tests, realtime_api shared-route tests (404 + stream), and
group_pubsub_service::publish_bridges_to_the_group_broadcaster. No migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
No atomic CAS existed, so concurrent writers to the same KV key raced. Add
set_if(key, expected, new): writes only when the current value equals expected,
or (expected absent) only when the key is missing — the primitive for lock-free
counters / optimistic concurrency.
- KvService + GroupKvService traits gain set_if with a defaulted "unsupported"
impl (Noop + other impls untouched); the real services override it.
- kv_repo/group_kv_repo: atomic set_if — a single UPDATE ... WHERE value =
$expected (JSONB semantic equality) or INSERT ... ON CONFLICT DO NOTHING.
Atomic without a transaction; returns whether the write happened.
- Services enforce the same value-size cap + (group) writes-authed + row/byte
quotas as set; the event fires only on a successful swap.
- Executor Rhai SDK: kv::collection(c).set_if(key, expected, new) and the
shared_collection variant; Rhai () for expected means "only if absent".
Pinned by kv_service/group_kv_service set_if unit tests (CAS + fail-closed) +
sdk_kv::kv_set_if_compare_and_swap (through a real script). No migration.
docs/sdk-shape.md documents the primitive.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M3 quotas capped a group's shared KV/docs by ROW count only; a group could still
blow past storage with few-but-huge values. Add the symmetric total-bytes ceiling
(files already had one).
- group_quota: PICLOUD_GROUP_{KV,DOCS}_MAX_TOTAL_BYTES env vars (default 256 MiB)
+ accessors.
- GroupKvRepo/GroupDocsRepo: total_bytes(group) = SUM(octet_length(value::text))
(default Ok(0) so non-Postgres impls skip the check).
- GroupKv/DocsServiceImpl: check the PROJECTED total (old value's bytes
subtracted, new added) on every set/create/update, so a same-or-smaller update
near the cap is still allowed (unlike a naive used+new check). New
TotalBytesQuotaExceeded errors; +with_max_total_bytes for tests.
- KV set switched has->get to obtain the old value's size for the delta.
Pinned by group_kv_service + group_docs_service per_group_byte_quota_uses_the_
projected_total unit tests (reject over-cap, allow same/smaller update near cap).
No migration. CLAUDE.md env-var table updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
email_trigger_details.inbound_secret_encrypted was sealed v0 (no AAD, bound only
to the master key) because the table had no version column — a ciphertext could
in principle be relocated between rows (audit 2026-06-11 H-D1, Medium). Bring the
AAD versioning the `secrets` table gained in 0042 to email secrets.
- Migration 0069: `email_trigger_details.inbound_secret_version SMALLINT DEFAULT 0`.
- secrets_service: `seal_email`/`open_email` seal v1 with AAD bound to the
SEALING OWNER (`email:{app}` / `email:group:{group}`) — deliberately NOT the
per-row trigger_id, so a group template's sealed bytes stay valid when the
materializer copies them verbatim to each descendant (all share the group AAD).
v0 rows keep their exact legacy no-AAD read path.
- Both email-trigger create paths (declarative apply resolve_and_seal +
triggers_api create_email_trigger) seal v1 under the trigger's owner; the
version threads through CreateEmailTrigger + insert_email_trigger_tx.
- materialize copies inbound_secret_version verbatim with the bytes.
- email_inbound_target recovers the sealing owner (materialized_from ->
template.group_id, else the app) so receive_inbound_email opens a v1 secret
under the right AAD.
Cross-app/tenant relocation now fails the GCM tag; a same-owner swap is not
distinguished (accepted low-severity residual). Pinned by
email_secret_aad::materialized_email_copy_decrypts_under_the_group_aad (the
novel materialized-copy path) + the extended secret_round_trips_through_seal_open
lib test. Schema golden reblessed; materialization + email e2e regressions green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An exhausted SHARED durable-queue message was `drop_exhausted()`-ed with a
warning — silent data loss. Per-app queues persist to `dead_letters` (0010);
add the symmetric group store so an exhausted shared-queue message is preserved
and operator-visible.
- Migration 0068: `group_dead_letters`, keyed by (group_id, collection),
CASCADE on the group (an app delete leaves the data — it belongs to the
group, not the consuming app).
- `GroupQueueRepo::dead_letter` (replaces `drop_exhausted`): one tx that INSERTs
the dead-letter + DELETEs the live message, filtered by claim_token so a lost
lease can't dead-letter a re-claimed message (mirrors queue_repo::dead_letter).
- Dispatcher `q_terminal` shared arm now dead-letters instead of dropping. It
returns None (not the dl id) so the per-app `fan_out_dead_letter` is SKIPPED —
firing the consuming app's per-app handlers on a shared message (competing
consumers → nondeterministic app) would be wrong. Fan-out to a *shared*
dead_letter trigger is deferred (needs a new trigger kind).
- Read side: `GroupDeadLetterRepo::list_for_group` backs a new read-only
operator endpoint GET /api/v1/admin/groups/{id}/dead-letters (GroupKvRead,
mirrors the M4 group-blobs surface). `pic dead-letters ls --group` deferred
(optional; the HTTP endpoint is the operator surface).
Pinned by group_queue::dead_letter_moves_an_exhausted_message_to_the_group_store
(store move + claim-token-mismatch guard + operator read). Schema golden
reblessed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-env approval policy was applier-supplied — a hand-crafted request
that omitted `project.environments` was ungated, and flipping a gate to
`confirm = false` in the same request un-gated it. Persist the policy
server-side and enforce against `persisted ∪ declared`.
- Migration 0067: `project_environments (project_id, env_name, confirm)`,
CASCADE on the project. Written declaratively (delete-then-insert) inside
`upsert_project_tx`, same tx as the project row + node claim.
- `ProjectRepository::get_environments_by_slug` (read side) +
`ApplyService::effective_env_policy` union the persisted policy with the
request's declared one (confirm if EITHER says so — monotonic).
- `env_gate_check` now evaluates the effective policy; the three handlers load
it before the gate. `plan.approvals_required` is the effective gated set, so
CI sees a persisted gate even when the manifest omits it.
- The union rule closes both bypasses: an omitted policy still trips a
persisted gate, and an ungating apply must itself pass the gate (the
declarative replace then takes effect next time) — TOCTOU-safe.
Pinned by env_approval::{persisted_policy_gates_a_request_that_omits_it,
flipping_a_gate_to_false_in_the_same_request_still_gates} +
projects_repo::get_environments_by_slug_returns_persisted_policy. Schema golden
reblessed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-env approval gate (`[project.environments]`, `confirm = true`) was
client-side only — the policy was `#[serde(skip_serializing)]`, so a patched or
older CLI applying to a confirm-required env bypassed it entirely. This makes
the gate server-enforced, admin-gated, and audited (the deferred §4.2 piece).
- Wire: `ProjectDecl` gains `environments` (the CLI stops skip_serializing it and
sends the selected `env` + the `--approve` set on the apply request). The
server RE-DERIVES the gate from the request (`env_gate_check`): a
confirm-required env not in `approved_envs` → 409 `ApprovalRequired`.
- Admin gate: an approved override additionally requires `AppAdmin`/`GroupAdmin`
on EVERY declared node — a step up from the editor write caps an ordinary
apply needs (reusing the admin caps per §4.2) — and emits a `tracing::info!`
audit line. Enforced on all three handlers (single app, single group, tree);
the check runs before any tx, so a denial (403) precedes any mutation.
- Token: the env policy folds into the TREE bound-plan token (a policy edit
between plan and apply trips StateMoved). The single-node token is left
unchanged (a bare live-state hash), so plan/apply still match without threading
a project into it.
- CLI: single-node `apply`/`plan` now resolve the GOVERNING project (own block
else nearest ancestor's) so a leaf apply carries the root's policy; `plan`
surfaces `approvals_required`. Client-side `require_env_approval` fast-fail
kept as UX.
Adapts the earlier feat/ownership-and-approval M5 (654e387) to main's DTOs; main
addresses tree group nodes by slug only, so the admin loop mirrors authz_tree
(app: resolve_app_id fail-closed; group: get_by_slug, skip to-create).
Threat-model note (honest scope, in the doc + code comments): this closes the
patched/older-CLI bypass and admin-gates + audits the override, but is NOT a
hermetic boundary against a hand-crafted request that OMITS the policy — the
policy is applier-supplied, not persisted server state. Full closure (persist a
`project_environments` source of truth + a server-determined env) is deferred.
Review (adversarial pass): fixed a MEDIUM (single-node `plan` didn't resolve the
governing project, so a leaf plan under-reported the gate `apply` enforces) and a
LOW (duplicate clippy attr); corrected overstated "can't bypass" wording.
Tests: ProjectDecl gate unit test; env_approval journeys
`server_enforces_the_gate_against_a_non_stock_client` (raw request → 409 without
approval, admin-approved → 200) and `approving_a_gated_apply_requires_admin`
(editor + --approve → 403). 417 lib tests + 33 CLI journeys + workspace clippy
-D warnings + fmt clean. No migration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`group_blast_radius`'s `app_chain` up-walk is depth-capped (`ac.depth < 64`,
added by 76926de to stop a should-be-impossible `groups.parent_id` cycle from
spinning the recursive CTE forever), but the sibling `subtree` down-walk in the
same query was left unbounded. A `parent_id` cycle would loop `subtree`
indefinitely and hang the plan — the exact exposure 76926de set out to close,
left half-done on the descendant side.
Add a `depth` column + `WHERE s.depth < 64` to `subtree`, mirroring `app_chain`.
Defense-in-depth: the reparent cycle-guard already prevents `parent_id` cycles,
so this hardens against an unreachable state — but that is precisely the bar
76926de applied. Surfaced by a post-adoption audit of the §7/§6 ownership code.
Verified: 416 manager-core lib tests; 12 ownership/create/divergence/approval
journeys (incl. plan_previews_ownership_and_blast_radius, which drives this
query); clippy -D warnings + fmt clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review #4 (LOW, cleanup). `reconcile_group_structure_tx` carried a
`#[allow(clippy::too_many_lines)]` over a ~145-line body doing create +
reparent + attach/RBAC/lock in one loop. Extract the create branch
(`create_group_node_tx`) and the divergence-resolution branch
(`reparent_diverged_group_tx`) into helper methods, threading the shared
attach-ceiling / principal / mode context through a small `ReconcileCtx` — the
loop is now thin enough to drop the allow. Also improve the child-before-parent
error in `resolve_declared_parent` to hint that group nodes must be ordered
parents-first (only a hand-rolled bundle hits it; the CLI already sorts).
No behavior change. Documents the Tier-1 review follow-ups in the design doc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review #3 (LOW). `divergence_preview` (plan path) compared parent slugs
case-sensitively from its own `get_by_slug`, while the apply path
(`reconcile_group_structure_tx`) compares resolved parent ids. On a mixed-case
parent slug the two could disagree — `pic plan` reporting `diverged` while
`pic apply` hard-errors "parent does not exist".
`resolve_existing_group_ids` becomes `load_existing_groups`, returning the full
`Group` rows it already loads; `plan_tree` derives the id map from them and
passes the map to `divergence_preview`, which now reads each node's own row
without a per-node `get_by_slug` (kills the N+1), resolves an in-tree parent's
slug from the same map, and compares parent slugs case-insensitively so the
preview agrees with the apply path's id-based check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review #2 (MEDIUM). The M3 `[project.environments]` approval gate silently
failed open in three shapes:
- `EnvPolicy.confirm` was `#[serde(default)]`, so `production = {}` parsed as
un-gated. `confirm` is now a REQUIRED field — an env listed with an empty
policy is a load error (`missing field 'confirm'`), not a silent no-gate.
- `pic apply --file <leaf>` where the leaf carried no `[project]` skipped the
gate even when the repo root gated the env. `run` now discovers the governing
`[project]` by walking up from the manifest to the nearest ancestor
`picloud.toml` that declares one (`find_governing_project`, loaded without the
env overlay — only the block matters).
- A `[project].environments` in a non-root manifest under `--dir` was dropped
with a generic "ignored" note; `build_tree` now warns explicitly that its
approval gating is not enforced.
Pinned by `tests/env_approval.rs` (empty-policy load error; leaf `--file` apply
honoring the root gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review #1 (HIGH). M1's `authz_tree` skips the group content-write capability
check (`require_group_node_writes`) for a group node absent at API-request
time — a to-create group whose authorization is the service create-gate. But
that gate only fires when the group is STILL absent at reconcile, and the
content writer performs no authz. So a group created by a racer in the window
between the API check and the in-tx reconcile could have its scripts/vars
written by a principal with zero rights on it (the bound-plan token doesn't
save a hand-rolled request that omits `expected_token`).
Close it with an in-tx content-write re-check in `apply_tree` for every group
NOT structurally created/reparented by this apply: a group WE created is
covered by create-RBAC (`GroupAdmin(parent)` cascades) and its uncommitted row
is invisible to the pool-based authz cascade anyway, so it must not be
re-checked; a pre-existing (incl. racer-created) group is committed, so its
ancestry resolves and a missing cap denies. `require_group_node_writes` moves
from `apply_api` onto `ApplyService` (up the existing dependency edge) so the
API pre-check and the in-tx re-check share one implementation.
Pinned by `tests/group_create.rs::tree_apply_denies_group_content_write_without_the_cap`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A `[project.environments]` block maps an env name to `{ confirm = bool }`
(`ManifestProject.environments`, client-side only — `skip_serializing`).
`pic apply --env <e>` is refused before any request when `<e>` is
confirm-required unless it is explicitly `--approve <e>`d; a blanket `--yes`
does NOT cover a gated env (§4.2 "CI must opt in per environment"); an unlisted
or `confirm = false` env applies freely. `require_env_approval` gates both the
single (`run`) and tree (`run_tree`) apply paths; `--approve <env>` is a
repeatable flag. Per-env value overlays (`picloud.<env>.toml`) already merge
client-side, so this is the confirm-policy gate only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tests/structural_divergence.rs`: a repo that nests a group under a different
parent than the server → `pic plan` shows it `diverged`; a bare apply is
refused (422); `--force-local-structure` reparents to the manifest shape;
`--adopt-server-structure` keeps the server placement. Design doc §6 records M2
shipped, leaving only per-env approval gating deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With the declared parent on the wire (M1), `apply_tree` now compares each
EXISTING group node's server parent to the manifest's (directory-nesting)
parent and resolves a divergence per a request `StructureMode` (default
`Refuse` — a pre-M2 CLI never reshapes the tree):
- `Refuse` → `ApplyError::StructuralDivergence` (422), naming the flags.
- `ForceLocal` → reparent to the declared parent IN the apply tx via the
extracted `group_repo::reparent_group_tx` (cycle guard + structure_version
bump), §5.6-gated (`GroupAdmin` on node + source + destination) and
attach-ceilinged.
- `AdoptServer` → keep the server shape, reconcile content in place.
M1's `create_missing_groups_tx` generalized to `reconcile_group_structure_tx`
(create + reparent share the coarse-lock / attach-ceiling / RBAC path; both are
recorded `structurally_changed` so the pool-based attach re-check skips their
uncommitted rows). `pic plan` previews the drift (`divergence_preview` → a
`structure` row naming server + manifest parents). CLI:
`--force-local-structure` / `--adopt-server-structure` (mutually exclusive) →
the `structure_mode` wire field; the 422 surfaces verbatim. A concurrent
`pic groups reparent` between plan and apply still trips `StateMoved` (the tree
token already folds structure_version).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`tests/group_create.rs`: a nested `[group]` tree with no server groups yet →
`pic apply --dir` creates both (parent from directory nesting), claims them for
the `[project]`, and re-applies as a no-op; a member with only editor (no
group-admin) is refused and leaves nothing behind. Design doc §6 records M1
shipped (parent-by-nesting, Phase-0 create-in-tx, claim, RBAC + attach ceiling)
with reparent/divergence still deferred to M2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lift "groups must pre-exist" for the tree apply: a `[group]` node now carries a
declared parent (inferred from directory nesting) and `pic apply --dir` creates
the ones the server lacks, in the same transaction.
Wire: `TreeNode` gains `parent`/`name`/`description` (`#[serde(default)]` — a
pre-§6 CLI still reconciles existing groups). `discover::build_tree` computes a
group's parent as the nearest ancestor directory holding a `[group]` (topmost →
the repo's `[project] parent_group` attach point, else instance root) and emits
nodes parents-first by directory depth.
Server: `apply_tree` runs a Phase 0 (`create_missing_groups_tx`) that
resolves-or-creates every group node IN the tx (a same-tree parent resolved
before its child via the new `group_repo::{create_group_tx,
read_group_id_by_slug_tx}`), under the coarse `GROUP_STRUCTURAL_LOCK_KEY` (taken
only when creating), enforcing the attach-point ceiling + RBAC
(`InstanceCreateGroup` / `GroupAdmin(parent)`) per create. A created group is
CLAIMED in Phase A alongside existing ones (`decide_group_claim` → `Claim`).
`prepare_tree` now takes a caller-supplied `resolved_ids` map and returns a
`to_create` list the plan path previews (empty current → full-create plan,
ownership `claim`); a to-create group's token part equals its post-create part
so plan/apply tokens match across a create. `validate_bundle_for` takes
`is_group: bool` (a to-create group has no id yet); `authz_tree` skips a
to-create group (its create authz is the service gate, not an existing-group
capability).
Reparent of a diverged existing group + the detect-and-refuse flags are M2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second-review follow-ups on the ownership cleanups:
- The new `group_blast_radius` `app_chain` recursion walked app→root with no
depth cap, unlike the `ancestors()` CTE it replaced (`WHERE c.depth < 64`).
Add the same `ac.depth < 64` guard so a (should-be-impossible) cycle in
`groups.parent_id` can't spin the recursive CTE unbounded and hang a plan.
- Drop the now-false "memoized per group" line from `group_blast_radius`'s doc
(it is a single set-based query now).
- Correct the `PlanRequest` flatten comment: apply is deliberately NOT flattened
(its pre-§7 wire was already `{bundle, …}`), so the two endpoints are
asymmetric by history — the old comment wrongly claimed parity.
Verified: the blast-radius journey + ownership journeys still pass; the INNER
JOIN to `projects` is safe (owner_project FK is ON DELETE SET NULL, so a
non-NULL owner always references a live row — no dangling drop).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four cleanups from the code review of the §7 ownership track, no behavior
change to any passing test (416 manager-core units + 133 CLI journeys green):
- #3 app-ownership read parity: `check_app_owner_single` now resolves the
nearest-claimed ancestor WITHIN the apply tx (`nearest_claimed_in_tx`, the
same mechanism the tree path uses) instead of on a separate pool connection,
so the check and this apply's writes see one consistent snapshot. (Fully
serializing against a concurrent claim on an out-of-tree ancestor stays part
of the documented pool→tx follow-up.)
- #4 blast-radius N+1 → one query: `group_blast_radius` replaced the
per-descendant-group `ancestors()` round-trips with a single recursive
`subtree → app_chain → nearest` CTE that groups + counts server-side. Cost is
now one query regardless of subtree size.
- #5 one ownership policy, two projections: `decide_group_claim` /
`decide_app_owner` are generic over the identity key, and `preview_ownership`
(plan side) now DERIVES its label from them (keyed on slugs, takeover-agnostic)
rather than re-encoding the arms — plan and apply can no longer drift.
- #6 shared slug validator: `validate_project_slug` delegates to a new
`groups_api::validate_slug_format`, which `groups_api::validate_slug` also now
uses — the format rule lives once. Projects intentionally skip the
reserved-word check (a project slug is never routed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
§7 M3 wrapped the three plan request bodies in a `{bundle, project}` struct,
silently breaking version-skew compatibility: a pre-§7 `pic plan` POSTs a BARE
bundle at the JSON top level and now fails deserialization (422 "missing field
bundle"), even though the sibling apply request deliberately kept its added
fields additive (`#[serde(default)]`) so an older CLI still works.
`#[serde(flatten)]` the bundle in `PlanRequest`/`TreePlanRequest` so the bundle
fields sit at the top level again: a bare bundle deserializes (`project`
defaults to `None`) and a newer client sends the same fields plus a top-level
`project` key. The CLI now builds that shape via a `plan_body` helper (bundle
object + optional `project`), which an older server also accepts (Bundle has no
`deny_unknown_fields`, so the extra key is ignored) — compatible in both skew
directions. Apply is left unwrapped (it was already `{bundle, ...}` pre-§7).
Pinned by `plan_request_accepts_a_bare_bundle` / `_a_flattened_project` /
`tree_plan_request_accepts_a_bare_bundle` unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`upsert_project_tx` defaulted a missing `[project].name` to the slug and then
`ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name` wrote that fabricated
value unconditionally — so a re-apply from a clone whose manifest omits `name`
silently clobbered the stored display name back to the slug (visible in
`pic projects ls` / `pic groups ls`).
Bind the raw optional name once and guard both sides on it:
`VALUES ($1, COALESCE($2, $1), $3)` (first apply falls back to the slug for the
NOT NULL column) and `DO UPDATE SET name = COALESCE($2, projects.name)` (a
re-apply updates the name only when the manifest actually declares one). An
omitted optional field now preserves persisted data instead of mutating it.
Pinned by a new `reapply_without_name_preserves_project_name` journey.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
§7 M3 (part 2): planning a change to a GROUP node now reports the descendant
apps owned by OTHER projects that the change fans out to — the cross-repo
signal an operator needs before touching shared config.
- OwnershipPreview gains `blast_radius: Vec<ProjectImpact>`; `group_blast_radius`
walks the group's subtree (recursive CTE) and groups descendant apps by their
nearest-claimed-ancestor project, excluding the planning project. Ancestor
resolution is memoized per group (one walk per distinct descendant group, not
per app).
- `pic plan` renders `blast_radius` rows (mode-agnostic).
- the apply_ownership journey gains a plan-preview case pinning both the
ownership annotation (claim / conflict) and the blast radius (two other
projects' apps surfaced) end to end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
§7 M3 (part 1): `pic plan` now surfaces the ownership outcome BEFORE apply.
- plan / plan_owner / plan_tree take the declaring `[project]`; each result
carries an `ownership` preview (pure `preview_ownership`, unit-tested):
claim (unclaimed + project), owned (already this project), conflict (owned by
X — named), or unclaimed. A group node previews its OWN owner; an app node its
nearest claimed ancestor (read-only, on the pool).
- the attach-point ceiling (M2) is now previewed at plan too, so `pic plan`
outside the subtree fails early — consistent with apply.
- wire: PlanRequest / TreePlanRequest wrap { bundle, project } (project
`#[serde(default)]` → a pre-M3 CLI still plans); the plan DTOs + `pic plan`
gain an `ownership` row (mode-agnostic: shows in tsv + json).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The apply_ownership journey gains an attach-ceiling case: a group node below
the attach point applies; the attach point itself and a sibling subtree are
both refused (422, message names the attach point). Design doc §7 + CLAUDE.md
record M2 shipped and re-point 'Next' at M3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
§6/§7 M2: a project's attach point is its ceiling — applies are refused for
any node not strictly within the declared subtree, so a repo can't reach (or
claim) above its local root even with the RBAC to do so.
- `[project] parent_group = "<slug>"` (ManifestProject + ProjectDecl); absent
= instance root = no ceiling (the default, backward-compatible).
- `check_within_attach`: the attach group must be a PROPER ancestor of a group
node (you can't apply the attach point itself) or an ancestor (inclusive) of
an app node's group; resolved via `groups.ancestors`. Enforced read-only in
apply_owner (prologue) + apply_tree (per node), before the claim.
- `ApplyError::OutsideAttachPoint` → 422 (a scope error, like Invalid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end coverage of the M1 ownership claim through the real CLI + server:
claim-on-first-apply + owner column, idempotent re-apply by the owner, a
second project refused (409, naming the owner), --takeover by an admin,
app-inheritance from the nearest claimed ancestor (owning project ok; foreign
or absent project refused), and the capability gate — a member with only an
editor GROUP role can reconcile but is refused --takeover (needs group-admin),
with ownership unchanged after the failed takeover.
- tests/apply_ownership.rs (registered in cli.rs); a grant_group_membership
helper in tests/common/member.rs (group-level, mirroring the app one).
- design doc §7 gains an M1-shipped status note; CLAUDE.md records §7 M1 and
re-points 'Next' at M2 (attach ceiling) + M3 (blast-radius / pic projects ls).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Makes §7 ownership usable end to end from the CLI.
- manifest: a `[project]` block (slug + optional name), independent of the
[app]/[group] XOR; threaded onto every apply from the repo. --dir reads it
from the tree ROOT's picloud.toml (a [project] elsewhere is ignored with a
note).
- client: apply_node/apply_tree carry project + takeover; the 409 branch now
surfaces the server message verbatim (covers StateMoved AND OwnershipConflict,
both self-contained). New groups_list_with_owner captures the owner slug.
- pic apply --takeover flag; pic groups ls gains an `owner` column (the owning
project's slug, or — when unclaimed).
- server: /api/v1/admin/groups now returns each group flattened with its owner
slug (via list_with_owner) — the shared Group deserialize ignores the extra
field, so pic groups tree / the dashboard are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The heart of multi-repo ownership: an apply now claims/verifies node ownership
before it reconciles, so two repos can't clobber each other's subtree.
- Pure policy (DB-free, unit-tested): `decide_group_claim` (unclaimed→claim,
owner→noop, foreign→conflict unless --takeover, no-project-into-claimed→
conflict) and `decide_app_owner` (an app must match its nearest claimed
ancestor; an unclaimed subtree stays open — backward-compatible).
- Execution under the per-node advisory lock: `upsert_project_tx` (first apply
registers the project), `read/write_group_owner_tx`, `claim_group_owner`
(takeover additionally requires `GroupAdmin` — ownership ⟂ RBAC),
`check_app_owner_single` + tree `nearest_claimed_in_tx` (in-tree Phase-A
claim overlays committed ancestors). No structure_version bump — a claim
isn't a diff change, so it must not churn a pending bound plan.
- `apply`/`apply_owner`/`apply_tree` take an `OwnershipClaim` (project +
takeover + principal); the claim slots after the lock, before load_current,
so a conflict short-circuits before any diff work.
- New `ApplyError::OwnershipConflict` → HTTP 409 (actionable); wired through
`ApplyRequest`/`TreeApplyRequest` (both `#[serde(default)]`, so the existing
CLI stays compatible until it learns [project]).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read side of §7 ownership, ahead of the claim logic.
- project_repo: `ProjectRepository` trait + `PostgresProjectRepository`
(list / get_by_id / get_by_slug). Writes are NOT here — a claim registers
its project via an in-tx upsert (next commit).
- group_repo: `list_with_owner()` LEFT-JOINs projects for each group's owner
slug (backs `pic groups ls`); columns qualified since id/slug/name exist on
both tables.
- ApplyService gains a `projects` handle; constructed in the picloud binary.
- tests/projects_repo: round-trip pinning list_with_owner (claimed→slug,
unclaimed→None) and that ancestors() surfaces owner_project nearest-first —
the fold input the claim path walks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First commit of §7 multi-repo ownership. A 'project' is a repo-root that
declaratively manages a slice of the shared group tree; each group node is
owned by exactly one project (single-owner-per-node). This lands the registry
+ shared types and wires the inert 0047 `owner_project` seam — no behavior
change yet (claim logic is the next commit).
- migration 0066: `projects` table (UUID pk, unique slug, name, created_by →
admin_users ON DELETE SET NULL); `groups.owner_project` gains its FK →
projects(id) ON DELETE SET NULL (un-claim, never cascade-destroy a tree) +
an index.
- shared: `ProjectId` (id_type! macro) + `Project` struct; `Group` gains
`owner_project: Option<ProjectId>`.
- group_repo: GROUP_COLS + the ancestors recursive-CTE term now carry
`owner_project`, so `ancestors()` surfaces ownership for the nearest-claimed
fold; GroupRow + From updated.
- schema snapshot re-blessed; /version schema assertion 65 → 66.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The M5.5 shared-group-secret work made a [group] [[triggers.email]] template
parse successfully (it materializes per descendant app; the inbound secret is
resolved against the group's own store server-side at apply). The unit test
still asserted the old parse-time rejection and had been failing on main under
`cargo test --workspace`. Update it to assert the current behavior: parse
accepts a group email template, like the group cron template above it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLI journey (shared_queues): authoring + `pic triggers ls --group` shared
column + the two validation rejections (undeclared queue collection; shared
on an app), mirroring the shared_topics/shared_triggers norm; the live
competing-consumer + dispatcher behaviour is pinned by the deterministic
manager-core tests. Docs: CLAUDE.md + design doc §11.6 record D3 as shipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consumption side of shared durable queues:
- Thread `shared` through BundleTrigger::Queue + QueueTriggerSpec + the
trigger identity; validate_bundle_for requires a shared queue on a group
to name a declared kind='queue' collection (a shared queue on an app is
rejected by the existing app-owner shared guard).
- materialize: a shared queue template materializes a consumer per
descendant (the M5 one-consumer-slot skip is bypassed for shared —
competing consumers are intended; each descendant gets one copy).
- dispatcher: ActiveQueueConsumer gains shared_group (from the materialized
copy's source template via LEFT JOIN); dispatch_one_queue +
handle_queue_failure route claim/ack/nack/terminal to the group store
when shared_group is Some, via q_claim/q_ack/q_nack/q_terminal helpers. A
group claim is normalized to a ClaimedMessage under the consuming app so
the handler path is unchanged; the reclaim task also drains the group
store. Exhausted shared messages are dropped (no group dead-letter store
yet — documented deferral).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deterministic manager-core test (tests/shared_topics.rs) proves the
namespace boundary both ways: a shared publish fans out only to the
group's shared pubsub trigger (not a per-app or non-shared group
template), and a per-app publish never hits the shared trigger. CLI
journey (shared_topics) covers authoring + `pic triggers ls --group`
shared column + the two validation rejections (undeclared topic; shared
on an app), mirroring the shared_triggers norm. Docs: CLAUDE.md + design
doc §11.6 record D1 + D2 as shipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime for shared topics:
- pubsub_repo: fan_out_publish gains `AND t.shared = FALSE` (a shared
trigger never fires on a per-app publish); new fan_out_shared_publish
matches `shared = true` pubsub triggers on the resolved owning group,
each delivery stamped with the writer app_id (M2 model).
- GroupPubsubService trait (shared) + GroupPubsubServiceImpl: resolve the
owning group (kind='topic') from cx.app_id's chain, require editor+
(GroupPubsubPublish, fails closed for anon), size-cap, fan out.
- SDK: `pubsub::shared_topic("name")` -> GroupTopicHandle with
`.publish(subtopic, msg)` / `.publish(msg)`; wired through Services +
the picloud binary (reuses the one PubsubRepo).
- authz: Capability::GroupPubsubPublish(GroupId), editor+ / script:write.
The owning-group chain walk is the isolation boundary; a sibling-subtree
app never resolves the topic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `shared` to BundleTrigger::Pubsub + PubsubTriggerSpec + the trigger
identity (so toggling re-diffs) + current_trigger_identity. validate_bundle_for
now allows a `shared` pubsub trigger on a group and requires the topic
pattern's ROOT segment (events.* -> events) be a declared kind='topic'
collection; a wildcard root is a runtime match. Apply-side plumbing only —
the shared publish path + dispatch boundary land next.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Widen the group_collections kind allow-list (migration 0064) + the
manifest CollectionKind enum + apply_service COLLECTION_KINDS to include
'topic' (D2, storeless publish namespace) and 'queue' (D3, group-keyed
durable queue — store lands in 0065). Foundation shared by both
milestones; routes through the existing owner-generic reconcile +
resolve_owning_group path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A materialized copy of an M5 group stateful template (cron/queue/email) now
reads as `materialized = true` — inherited, read-only — distinct from a
hand-authored trigger. Threads a derived `materialized` bool (=
`materialized_from IS NOT NULL`) row → domain → API → CLI, mirroring the
`sealed`/`shared` columns; `list_for_app` SELECTs `materialized_from`. No
migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review NIT: the #[cfg(test)] InMemoryTriggerRepo's
email_inbound_target `.expect()`s app_id, which would panic if a future
unit test fetched a group-owned email TEMPLATE. Filter `app_id.is_some()`
like the production query's `AND t.app_id IS NOT NULL` so the mock stays
faithful. Test-only; no production change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #[ignore]-gated version_includes_public_base_url pinned schema=42 but
migrations accreted to 63 across the v1.2 template track (…0060–0063), so
it failed under `--include-ignored`. Re-pin to current reality per the
test's own intent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the design doc §4.5 and CLAUDE.md current-focus: email joins
cron+queue as a materialized stateful template via the shared-group-secret
model; drop the email deferral and the stale "email rejected on a group"
note; retarget "Next" to multi-node / multi-repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CLI journey: set a group secret, apply a group [[triggers.email]] template
referencing it, create an app under the group → `pic triggers ls --app`
shows a materialized email trigger; re-apply is a NoOp.
Required loading a group's own set-secret names into CurrentState (was
hardcoded empty) so the apply plan-time email-secret check resolves the
ref against the GROUP store. Informational only — the secrets diff is
never executed on apply (secrets are set out-of-band via `pic secret
set`); the state token now folds in group secrets, matching apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drive rematerialize directly with a group email template carrying a fake
sealed secret: a descendant app gets one copy whose inbound-secret
ciphertext + nonce byte-equal the template's (verbatim copy, no reseal),
the reconcile is idempotent, and reparenting the app out of the subtree
de-materializes the copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the last M5 gap: a [group] may now declare an email trigger
template, materialized into a per-descendant-app row like cron/queue.
Uses the shared-group-secret model: the template resolves its
inbound_secret_ref against the GROUP's own secret store once at apply and
seals it (resolve_and_seal + insert_email_trigger_tx generalized to a
SecretOwner / ScriptOwner). Email secrets are v0/no-AAD, so the sealed
blob is portable across rows — materialize copies the bytes verbatim into
each descendant (no master key, no reseal, no new schema column, no
threading into the CRUD hooks). Each copy has its own trigger_id / inbound
webhook URL.
- apply_service/manifest: drop the two group-email rejections; a group
email template with an unset group secret still fails apply hard.
- materialize: add "email" to MATERIALIZED_KINDS + a byte-copy detail arm.
- trigger_repo: email_inbound_target gains AND t.app_id IS NOT NULL so a
group TEMPLATE row is never directly invocable via its own webhook URL
(mirrors the cron/queue app_id guards).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The reconciler read `should` and `existing` in separate statements under READ
COMMITTED, and its queue one-consumer check wasn't locked — so two concurrent
reconcilers (e.g. two parallel app-creates) could (a) spuriously DELETE a valid
materialized cron copy when one committed a copy between the other's two reads
(a lost copy self-healing only on the next mutation), or (b) both pass the
queue slot check and double-fill a consumer slot. Take a fixed xact-scoped
advisory lock (REMATERIALIZE_LOCK_KEY) at the top of the reconcile so each pass
is atomic w.r.t. the others. Reconciles are infrequent + fast, so serializing
is cheap; the ON CONFLICT guard stays as defense in depth.
Found by the M5 review. Pinned by the existing stateful_templates test + the
full journey suite (which previously flaked on the race).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- 0063: partial unique index on (app_id, materialized_from) + ON CONFLICT DO
NOTHING in the reconciler, so a materialized copy is idempotent under
concurrent reconciles (two parallel app-creates no longer duplicate a copy).
- stateful_templates journey: a group cron template + a descendant app → a
materialized cron copy in `pic triggers ls --app`; re-apply is a NoOp.
- docs (§4.5 + CLAUDE.md): cron+queue materialization implemented; group EMAIL
templates deferred (per-descendant inbound-secret reseal needs the master key
threaded through the hooks + a secret-ref schema) and cleanly rejected at
apply for now, alongside a `materialized` ls column.
Completes the v1.2 near-term batch (M1-M5, cron+queue); group email is the one
scoped follow-up.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
materialize::rematerialize_stateful_templates reconciles a group cron/queue
template into one app-owned copy per descendant (materialized_from = template),
via the all-apps app_chain CTE ⋈ group-owned stateful templates. A precise
create/delete diff preserves cron last_fired_at on unchanged copies; a queue
copy is skipped-with-warning when the app already fills that queue's consumer
slot (the one-consumer invariant). Called full-live at the route-rebuild
chokepoints: apply (single + tree), app create/delete (apps_api), group
reparent (groups_api) — AppsState/GroupsState gain a pool. Pinned by
tests/stateful_templates.rs (per-app copy, idempotent, new-app materializes,
reparent de-materializes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
0062 adds `materialized_from` on triggers (a managed app-owned copy links back
to its group template; CASCADE). The scheduler + queue-consumer queries gain
`AND t.app_id IS NOT NULL` so a group-owned stateful TEMPLATE is never
dispatched directly — only the per-descendant materialized app rows are.
validate_bundle_for + manifest parse now allow cron/queue on a group (email
stays rejected pending its per-app inbound-secret handling in M5.5). The
materialization reconcile that expands templates into app rows lands in M5.2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- `pic kv ls --group` / `pic kv get --group` (mutually exclusive with --app)
hit the M4 admin API; client group_kv_list/group_kv_get.
- collections journey: a script writes a shared KV value, the operator lists +
fetches it via the admin API with no script.
- docs: §11.6 + CLAUDE.md record M3 quotas + M4 read-only admin API.
Completes M4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
group_blobs_api mirrors the per-app kv_api/files_api for groups: GET
/groups/{id}/kv[/{c}/{k}], /docs[/{c}/{id}], /files[/{c}/{id}] — list +
fetch/download a group's §11.6 shared KV/docs/files. Authz GroupKvRead/
GroupDocsRead/GroupFilesRead (viewer tier, reads-open trust model). Read-only
by design (writes stay script-only). The host hoists the three group repos so
the SDK services and this router share one instance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global env-var ceilings enforced in the group write path (mirrors the
per-value caps): PICLOUD_GROUP_KV_MAX_ROWS / _DOCS_MAX_ROWS (per-group row
count, checked only on a NEW key/doc — updates exempt) and
_FILES_MAX_TOTAL_BYTES (per-group total blob bytes). New group_quota env
helpers; count_rows/total_bytes repo methods (trait-default 0 for non-Postgres);
QuotaExceeded error variants on the three group error enums; a with_max_rows
test builder. Pinned by a group_kv_service unit test (3rd key rejected, update
of an existing key at the cap allowed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- `shared` column in `pic triggers ls --group` (TriggerTemplateInfo + DTO +
report + renderer).
- manager-core/tests/shared_triggers.rs: a shared write matches the group's
shared trigger and NOT a same-named per-app trigger; a per-app write matches
the app trigger and NOT the shared one (the `shared` flag is the boundary).
- shared_triggers journey: declarative authoring applies, ls --group shows
shared, an undeclared-collection shared trigger + an app shared trigger are
rejected.
- docs: §11.6 + CLAUDE.md move shared-collection triggers from Deferred to
implemented.
Completes M2. (Pre-existing async-dispatcher timing flakes pass in isolation.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`shared = true` on a group [[triggers.kv|docs|files]] flows manifest → Bundle
→ insert_trigger_tx (new shared column), mirroring sealed: shared lives on the
Trigger DTO + is part of the apply diff identity (bundle + current sides) so a
toggle re-applies. validate_bundle_for rejects shared on an app owner, on a
non-collection kind (pubsub/cron/etc.), and on a concrete collection the group
does not declare as a shared collection of that kind. Emission (M2.3) now has
authored triggers to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ServiceEventEmitter gains emit_shared(cx, owning_group, event) (default no-op);
OutboxEventEmitter implements it — matches shared triggers on the owning group
and writes outbox rows stamped with the WRITER app_id (dispatch runs under the
writer). GroupKv/Docs/FilesServiceImpl gain an events field (Noop default +
with_events builder) and emit on set/create/update/delete; the host wires the
outbox emitter via with_events. Closes the "group trigger has no app to watch"
gap. Authoring + validation land in M2.4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-app list_matching_kv/docs/files gain `AND t.shared = FALSE` (a shared
trigger never matches a per-app write). New list_matching_shared_{kv,docs,files}
select enabled `shared = TRUE` triggers on the OWNING group (glob+op filtered
in Rust), returning the same match types — no chain, no suppression anti-join.
Trait defaults return empty so non-Postgres impls degrade. Emission wiring in
M2.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group-owned trigger marked `shared = true` watches a §11.6 shared
collection instead of per-app collections — the namespace boundary that lets
a shared-collection write fire a trigger. Mirrors the sealed column; existing
rows default false (per-app, unchanged). Match-query split + emission land in
M2.2/M2.3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- dangling_suppress_warnings takes an ApplyOwner and walks the group's own
chain (GROUP_CHAIN_LEVELS_CTE) for a group node; runs for both owners.
- GET /groups/{id}/suppressions + `pic suppress ls --group` (client + cmd +
main.rs; --app/--group mutually exclusive).
- suppress journey: a child group declines a parent template for its subtree;
suppress ls --group shows the marker. Docs §4.5 + CLAUDE.md move group-level
suppression from Deferred to implemented.
Completes M1. (An unrelated pre-existing email_inbound dispatch-timing flake
passes in isolation.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both suppression filters now match an owner's own OR any ancestor group's
suppression on the firing app's chain:
- trigger anti-join joins the `chain` CTE (ts.app_id = sc.app_owner OR
ts.group_id = sc.group_owner) instead of ts.app_id = $1;
- list_route_suppressions expands group-owned suppressions across descendants
via the all-apps app_chain CTE, yielding (effective_app_id, path) the rebuild
consumes unchanged.
A child group declining a parent template opts out its whole subtree; a sibling
subtree still inherits; sealed still overrides. Pinned by
tests/group_suppression.rs; per-app suppression + sealed regressions green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
suppression_repo takes a ScriptOwner (app or group), binding the matching
owner column. apply_service load_current / create / prune / suppression_report
all thread owner.as_script_owner(); the validate_bundle_for group-suppress
rejection and the manifest parse guard are dropped — a [group] may now declare
[suppress] to decline a template it inherits from a higher ancestor. No
runtime consumption effect yet (the chain filters land in M1.3/M1.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reshape the app-only suppression marker to a group/app polymorphic owner
(mirrors 0056/0057 + the 0051/0052 config markers): nullable group_id
(CASCADE), nullable app_id, exactly-one CHECK, per-owner partial unique
indexes. Lets a [group] decline a template it inherits from a higher
ancestor for its whole subtree; consumption filters land in M1.3/M1.4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- tests/sealed.rs: a group declares a sealed [[routes]] template; a descendant
that suppresses it still serves the path, apply warns "sealed — no effect",
routes ls --group shows sealed=true, and sealing an [app] route is rejected.
- docs: §4.5 trust-model callout + CLAUDE.md move `sealed` from Deferred to
implemented — group templates are advisory-by-default *unless* sealed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- `pic triggers ls --group` / `routes ls --group` gain a `sealed` column
(threaded through TriggerTemplateInfo/RouteTemplateInfo + the two DTOs).
- dangling_suppress_warnings now also warns when a suppress reference matches
only SEALED inherited templates ("... is sealed — the suppression has no
effect") via `bool_or(NOT sealed)` per handler/path — making the mandatory
guarantee observable at plan/apply time, alongside the existing typo guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime effect: a sealed group template ignores a descendant's opt-out.
- Trigger dispatch: `AND t.sealed = FALSE` inside TRIGGER_SUPPRESSION_ANTIJOIN
— a sealed row is never excluded, so it fires through the suppression (one
edit covers list_matching_kv/docs/files + pubsub fan-out).
- Route rebuild: compile_effective_routes gates its suppression `continue` on
`!er.route.sealed`, so a sealed inherited route stays in the app's slice.
Pinned by tests/sealed_templates.rs (live-DB): a sealed trigger + route
survive an app's suppression of both; an unsealed sibling with the same
suppression is still declined — the gate is exactly `sealed`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread `sealed` from the manifest through to the DB column:
- manifest: `sealed` on ManifestRoute + the four event-kind trigger specs
(Kv/Docs/Files/Pubsub), flowing to Bundle via serde.
- persist: NewRoute.sealed + insert_route_tx; insert_trigger_tx `sealed`
param; the reconcile passes br.sealed / bt.sealed().
- validate_bundle_for rejects `sealed` on an app owner (meaningless — an
app route/trigger is never inherited).
- diff: `sealed` joins the route Update comparison and the trigger identity
(both bundle + current sides), so toggling it re-applies rather than NoOp.
`sealed` lives on shared Route + manager-core Trigger (both pure DTOs) so
the diff can see the current value; RouteRow/TriggerRow default it so
RETURNING clauses that omit it still hydrate. No runtime read effect yet —
the two suppression gates land in M3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group template can be marked `sealed = true` so the per-app suppression
filters skip it — closing the advisory-by-default compliance footgun the
suppression review flagged. Only meaningful on a group-owned template;
existing rows default unsealed. Consumption gates land in M3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two review findings.
1. Per-app opt-out changes the group-template guarantee from "runs on every
descendant" to "runs unless the descendant declines" — a footgun for
compliance hooks (an audit/security trigger a tenant can silently opt out
of). There is no non-suppressible flag. Document this in design §4.5 +
CLAUDE.md, and record the deferred mitigation: a `sealed`/`mandatory`
group-template marker the trigger anti-join + route filter skip (cheap —
both filters are centralized).
2. The dangling-suppress warning path had no coverage. The suppress journey
now applies a `[suppress] triggers=["does-not-exist"]`, asserts the apply
still succeeds and the report warns "no effect".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Journey tests/suppress.rs: a group declares a `[[routes]]` template; a
descendant app applies `[suppress] routes=["/ghello"]` → stops serving it
(via `routes match`), a sibling still serves it, `pic suppress ls --app`
shows it, re-apply is a NoOp; pruning the block re-inherits. The
trigger-side filter + isolation are pinned at the repo layer by
template_suppression.rs.
- Docs: design §4.5 (per-app opt-out closing the deferred gap for both
templates — coarse-by-reference, inheritance-only, the two consumption
points; group-level suppress still deferred), CLAUDE.md current-focus.
Full journey suite 121/121; workspace tests 34 suites/0 failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Visibility + typo guard.
- `pic suppress ls --app <a>` — read-only (kind, reference):
ApplyService::suppression_report(App) → GET /apps/{id}/suppressions
(viewer-tier AppRead), client + cmd + main.rs wiring.
- Dangling-suppress warning: at apply, a suppress reference that matches no
inherited template on the app's chain (no ancestor-group trigger bound to
that script name / no ancestor-group route at that path) emits an
ApplyReport warning — the suppression silently does nothing otherwise.
Soft (never fails the apply), reusing the existing warnings channel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime half — suppressions now actually decline inherited templates.
- Triggers (live, per-event): a correlated NOT EXISTS anti-join
(TRIGGER_SUPPRESSION_ANTIJOIN) appended to all four dispatch match
queries (list_matching_kv/docs/files + the pubsub fan-out). It excludes a
group-owned trigger whose handler script name the firing app suppresses;
`$1` is the firing app (already bound), and the `t.group_id IS NOT NULL`
guard keeps an app's OWN trigger unsuppressable.
- Routes (rebuild-time): compile_effective_routes takes a
`suppressed_paths: &HashSet<(AppId, path)>` and drops an inherited
(`depth > 0`) route at a suppressed path — the binding 404s.
rebuild_route_table loads the set via a new
RouteRepository::list_route_suppressions, so every existing rebuild edge
(route CRUD, apply, tree mutations) already applies it. No new
invalidation edges; the marker CASCADEs on app delete.
Pinned by tests/template_suppression.rs (live DB): a suppressing app
matches NEITHER the inherited trigger NOR route; a sibling that did not
suppress still inherits both; the app's OWN trigger on the suppressed
handler still fires (suppression is inheritance-only).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The declarative half — a `[suppress]` block persists as markers; no runtime
effect yet (S3 consumes them).
- manifest: `[suppress]` table on an app with `triggers = [...]` (handler
script names) + `routes = [...]` (paths), `deny_unknown_fields`; rejected
on a `[group]` (a group just wouldn't declare the template).
- suppression_repo.rs: app-keyed `list_for_app` / `insert` / `delete_tx`
over `(app_id, target_kind, reference)`, mirroring extension_point_repo
minus the owner polymorphism.
- apply_service: the extension-point marker-reconcile pattern —
`Bundle.suppress_triggers/_routes`, `Plan.suppressions`,
`CurrentState.suppressions`, `load_current(App)` load, `diff_suppressions`
(key `"{kind}:{reference}"`, split on the first `:` so a route param path
survives), create + prune reconcile blocks, `validate_bundle_for` group
reject, `ApplyReport` counters.
- CLI: `build_bundle` carries the two vecs; `pic plan` + apply report gain a
suppressions row (DTOs + display).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group TRIGGER/ROUTE template inherits to every descendant app; until now
a descendant could shadow one but not decline it. This marker records that
an app opts OUT of a specific inherited template — coarse by REFERENCE (a
handler script name for triggers, a path for routes), not row id (template
ids churn on re-apply, references are stable so re-apply is a NoOp).
App-only (a group would just not declare the template) → app_id NOT NULL,
no polymorphic owner; pure app config → ON DELETE CASCADE. A target_kind
discriminator ('trigger'|'route') keeps it one table, one reconcile loop.
Schema blessed at 58 migrations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two review findings.
1. After R3 every production rebuild path uses `compile_effective_routes`;
`compile_routes` (the old app-only compile) was called only by its own
two unit tests, and its "runs in build_app" doc was stale — so the
lenient-skip + disabled-drop regression guards no longer covered the
live path. Remove `compile_routes` + its re-export, fold the lenient-skip
rationale onto `compile_one`, and port both guards to
`compile_effective_routes` (app-only rows are just depth-0 effective
rows). Add a third guard pinning nearest-owner shadowing at the pure-
compile layer. Fix the stale `compile_routes` mention in apply_service.
2. Document the disabled + inherited semantic: the `enabled` filter runs
before the shadow check, so a disabled own-route does not claim its
binding and an enabled ancestor template at the same path falls through
("disabled = absent", §4.3). Recorded on `compile_effective_routes` and
in design §4.5 — the corollary (can't 404 an inherited route by
disabling a same-path own route) points at the deferred per-app opt-out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Visibility + end-to-end coverage + docs for group route templates.
- ApplyService::route_report(Group) → GET /api/v1/admin/groups/{id}/routes
(viewer-tier GroupScriptsRead), surfacing a group's own route templates
(method, host, path, handler script, dispatch, enabled).
- CLI: `pic routes ls --group <g>` (the script-id positional now conflicts
with --group), client `group_routes_list` + RouteTemplateDto.
- Journey `group_routes.rs`: a group declares a `[[routes]]` template
binding a group handler; a DESCENDANT app serves it (via `routes match`
against the live RouteTable), a SIBLING-subtree app does not, `routes ls
--group` shows it, re-apply is a NoOp. Shadowing + isolation are
additionally pinned at the repo layer by group_route_templates.rs.
- Update the manifest unit test: a [group] now accepts route templates and
rejects only the stateful trigger kinds.
- Docs: design §4.5 (the live route-template model + full-live
invalidation + nearest-wins shadowing + deferrals), CLAUDE.md focus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Open the declarative path so a [group] can declare route TEMPLATES.
- manifest.rs: drop the blanket "[group] cannot declare [[routes]]"
rejection (routes join the event trigger templates a group may own).
- apply_service:
- validate_bundle_for: remove the group routes rejection. A group
route's `script` is validated as binding a group-owned ENDPOINT via
`resolve_inherited_targets_for(Group)`, now extended to scan
bundle.routes as well as bundle.triggers — so a template binding the
group's own (declared or pre-existing) endpoint resolves, and one
pointing at a module or a missing name is a 422.
- insert_bundle_route takes a ScriptOwner (was app-only): a group node
writes a group_id route TEMPLATE, an app node an app-owned route.
- load_current(Group) loads the group's own routes via list_for_group
so re-apply is a NoOp and the template is prune-able.
Host-claim validation stays App-only (already guarded) — a template has
no single app; a descendant serves it on whatever host it claims (so
templates use host_kind = any in practice).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime half. Group route TEMPLATES now expand into every descendant
app's in-memory match slice, live, with no per-app materialization.
- `compile_effective_routes` consumes the `list_effective` expansion and
applies NEAREST-OWNER-WINS shadowing: for one app, identical binding
tuples (method+host+path) owned at multiple chain levels collapse to the
nearest (an app's own route shadows an ancestor-group template); a nearer
group beats a farther one. Non-identical bindings coexist and the
existing matcher precedence resolves each request. `compile_route` now
takes the effective app_id, so the matcher + dispatch are unchanged.
- `rebuild_route_table` is the single refresh chokepoint (list_effective →
compile_effective_routes → replace_all). Route CRUD, the apply reconcile,
and startup all route through it; the apply guards drop the "a group node
touches no routes" assumption (a group apply may now create templates).
- FULL-LIVE INVALIDATION: app create/delete (apps_api) and group reparent
(groups_api) rebuild the snapshot, so inherited routes take effect the
instant the tree changes — matching the live model of triggers/vars.
Best-effort, mirroring apply: a failed rebuild self-heals on the next
write or restart, never failing the committed mutation.
The isolation boundary is the chain expansion: a template lands only in the
slices of apps whose ancestor chain contains the owning group. Pinned by a
deterministic live-DB integration test (descendant inherits, sibling
subtree does not, app shadows by identical binding, non-identical coexists).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the route layer owner-aware ahead of the live rebuild. `Route.app_id`
becomes Option + a `group_id` (mirrors `Trigger`); `NewRoute` carries a
`ScriptOwner` so the interactive API passes App and the reconcile can pass
a group. `insert_route_tx` writes both owner columns; `RouteRow` selects
`group_id` everywhere (the interactive delete/list 404 a group template,
which is apply-managed, not app-addressable).
Adds the two repo methods the live rebuild + apply need:
- `list_for_group` — a group's own route templates (apply diff, ls).
- `list_effective` — the all-apps generalization of CHAIN_LEVELS_CTE: for
every app, walk its ancestor chain and join routes owned at each level,
tagging each with the effective (firing) app + the owner's chain depth.
Runs only on a RouteTable rebuild, never per request; validated with
EXPLAIN against the live schema.
`compile_route` now takes the effective app_id explicitly (reused next
commit for the per-app expansion); `compile_routes` skips group templates
(no single app) — they materialize via the effective path. Runtime
behavior is unchanged this commit: the table still rebuilds from
`list_all`, and no group routes can exist yet (authoring lands in R4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A group-owned route is a TEMPLATE, expanded into every descendant app's
in-memory RouteTable slice at rebuild (live, no materialization). The
schema reshape mirrors 0056_group_triggers exactly: a nullable group_id
FK→groups ON DELETE RESTRICT (a route template is a binding, not data),
app_id made nullable, an exactly-one owner CHECK, and the per-app unique
binding index split into per-owner partials. A group can't declare two
identical templates; an app declaring an identical binding is a
deliberate shadow resolved at rebuild (nearest-owner-wins), not a DB
conflict. Adds routes_group_id_idx for the expansion join.
Schema blessed at 57 migrations.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`get` can fetch any trigger by id, including a group-owned TEMPLATE. It
omitted `group_id` and leaned on `#[sqlx(default)]`, so a template would
hydrate with both owners None — silently breaking the exactly-one-owner
invariant in memory. No caller hits this today (interactive trigger
management is app-scoped; prune deletes by id), but the fetch now
round-trips the owner faithfully and stays honest for future callers.
`list_for_app` is left as-is: its rows are all app-owned, so group_id is
legitimately NULL there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- apply_service: trigger_report(group) → TriggerTemplateInfo
(kind/target/script/enabled); resolve_inherited_targets_for(Group) now
surfaces the group's OWN endpoint scripts so a template's handler
validates (fixes "binds to unknown script" when the handler is a
pre-existing group script, not declared in the same manifest).
- apply_api: GET /groups/{id}/triggers (GroupScriptsRead).
- CLI: `pic triggers ls --group <g>` (--app/--group mutually exclusive)
+ the client method + DTO.
- tests/group_trigger_templates.rs (manager-core, live DB): the chain
union matches a descendant app's kv insert against the group template
and NOT a sibling subtree — the isolation boundary, deterministic.
- tests/group_triggers.rs (journey): apply a kv template, ls --group
shows it, re-apply NoOp, cron-on-group rejected.
- docs: design §4.5 (live-event-kinds decision + deferrals), CLAUDE.md.
Full journey suite 119/119; workspace tests 0 failures; clippy -D clean;
schema unchanged (blessed in T1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime heart: a descendant app's event now fires its ancestor
groups' trigger templates, resolved LIVE (no materialization).
- trigger_repo: list_matching_kv/docs/files prepend CHAIN_LEVELS_CTE and
`JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)`
(bind $1 = app_id), so each returns the app's own triggers PLUS
ancestor-group templates in one statement. NULL-comparison semantics
keep app/group rows from cross-matching; each row matches at exactly one
chain depth (no dup fan-out).
- pubsub_repo: same union on the publish fan-out query.
- The outbox row stamps the firing app_id + the (group-owned) script_id;
the dispatcher's existing script_invocable() (app-owned OR invocable via
chain membership) already runs a group handler under a descendant app —
the Phase-4 inheriting-app boundary, unchanged.
Glob/ops/topic filtering in Rust is untouched. 404 manager-core unit
tests green; clippy -D clean. Cross-subtree firing is journey-proven in T4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A [group] node may now declare EVENT trigger templates (kv/docs/files/
pubsub) that bind a group-owned handler. cron/queue/email stay app-only
(rejected on a group at both the manifest and the reconcile layer).
- Trigger is now owner-polymorphic (app_id: Option, group_id: Option),
mirroring Script's Phase-4 reshape; TriggerRow carries group_id
(#[sqlx(default)] so interactive RETURNING clauses still hydrate).
- insert_trigger_tx takes a ScriptOwner (App XOR Group) and writes the
group_id column; the queue advisory-lock branch is app-only.
- TriggerRepo::list_for_group loads a group's templates; load_current's
Group arm uses it so the diff is NoOp on re-apply and prune-able.
- apply_service reconcile drops the "triggers are app-only" assumption;
validate_bundle_for rejects non-event kinds on a group. The semantic-
identity diff is already per-owner.
- CLI manifest allows event-kind [[triggers]] on [group], rejects
cron/queue/email there.
Templates don't fire yet — the live dispatch union lands in T3. All
trigger/apply/manifest unit tests green; clippy -D clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reshape the triggers table to allow a GROUP owner, mirroring
0050_group_scripts exactly. A group-owned trigger is a TEMPLATE: never
dispatched directly, but unioned into every descendant app's match
queries via the ancestor-chain CTE (live, no per-app materialization).
- 0056_group_triggers.sql: add nullable group_id (FK→groups ON DELETE
RESTRICT), make app_id nullable, add the exactly-one owner CHECK, and
split the name-unique + dispatch-hot indexes into per-owner partials
(idx_triggers_group_kind_enabled serves the chain-union lookups).
- Detail tables unchanged (they hang off trigger_id).
Schema snapshot blessed (56 migrations); existing trigger tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §11.6 files slice relies on the orphan sweeper covering the new
`files/groups/<group_id>/...` subtree "for free" via its recursive walk
of `<root>/files/`, with no sweeper code change. Lock that guarantee
with an explicit test so a future sweeper refactor can't silently drop
group-shared tmp files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- collections.rs: shared_files_collection_is_read_write_across_the_subtree
— a group declares kv+docs+files in one string-or-table manifest; authed
app A creates a blob via files::shared_collection("assets").create(...),
app B lists + gets the SAME bytes back across the subtree, a
sibling-subtree app gets CollectionNotShared, collections ls shows all
three kinds. Live-verified the bytes land under
files/groups/<group_id>/assets/<id[0:2]>/<id>.
- .gitignore: ignore /crates/picloud-cli/data (the CLI journey harness
spawns the server from that dir; the files journey is the first CLI test
to write blobs).
- docs: design §11.6 (KV+DOCS+FILES shipped; topics/queue still deferred),
sdk-shape (files::shared_collection), CLAUDE.md current-focus.
Full journey suite 117/117; schema blessed (55 migrations); workspace
clippy -D clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- manifest: add CollectionKind::Files (+ "files" in as_str()); the
string-or-table `collections` form now accepts { name, kind = "files" }.
- apply_service: add "files" to COLLECTION_KINDS. The rest of the
reconcile (CollectionSpec, diff_collections, validate_bundle,
state_token, the app-owner rejection) is already kind-generic.
- `pic collections ls` shows the files kind with no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clone of the GroupKv*/GroupDocs* arms at all five wiring sites
(Capability variants, app_id()⇒None, required_scope()⇒ScriptRead/Write,
the group_member_grants route, group_role_satisfies: Read=viewer+,
Write=editor+). Unit test mirrors group_docs_caps_resolve_by_role_up_the_chain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the §11.6 shared-collection machinery to the filesystem-backed
`files` blob store (after KV/0053 and docs/0054).
- 0055_group_files.sql: widen the group_collections kind CHECK to admit
'files'; add a group-keyed `group_files` metadata table mirroring
`files` (0018), CASCADE on group delete.
- files_repo: generalize the four path/IO free functions
(shard_dir_at / final_path_at / write_atomic_at / read_verify_at) to
take an owner-relative dir instead of `app_id`, and make them (plus the
cursor helpers) pub(crate). The security-sensitive atomic-write +
checksum-on-read mechanics now have a single source shared by the app
and group repos. App behavior is unchanged (apps shard at `<app_id>/`).
- group_files_repo: FsGroupFilesRepo keyed by group_id, blobs at
`<root>/files/groups/<group_id>/<collection>/<id[0:2]>/<id>` — a
`groups/` infix disjoint from the per-app subtree, so the existing
recursive orphan sweeper covers it with zero change.
Schema snapshot blessed (55 migrations); app-files fs tests still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-kind `list_for_owner` was superseded by `list_all_for_owner`
(the apply diff and `collection_report` both moved to it during the docs
slice). It had no callers but, being `pub`, escaped the dead-code lint —
removing it drops two unexercised SQL queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end docs journey: a group declares a kv AND a docs collection via the
string-or-table manifest; an authenticated app A docs::shared_collection(
"articles").create(#{...}), app B finds it back across the subtree (exercising
the shared find DSL); a sibling-subtree app gets CollectionNotShared;
collections ls shows both kinds; re-apply NoOp. Full journey suite 116/116.
Docs: groups-and-project-tool §11.6 (KV+docs slices shipped; files/topics/queue
deferred), sdk-shape.md (docs::shared_collection + string-or-table), CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the declarative collection surface carry a `kind` so docs (and future
kinds) are declarable. Back-compatible: the shipped `collections = ["catalog"]`
form still means kv.
- manifest: `collections` entries are now `CollectionDecl` — an untagged enum of
a bare string (kv shorthand) or a `{ name, kind }` table (kind ∈ kv/docs).
`Manifest::collections()` normalizes to `(name, kind)` pairs; build_bundle
emits `[{name, kind}]`. Still `[group]`-only. Unit test covers the
string-or-table mix + app rejection.
- apply: `Bundle.collections: Vec<CollectionSpec>` ({name, kind=default "kv"});
`CurrentState.collections: Vec<(name,kind)>` via list_all_for_owner;
`diff_collections` keys each change by name with `detail = kind` and identity
`(LOWER(name), kind)` (so a kv and a docs collection of the same name are
distinct); reconcile reads the kind from `detail`; state_token folds
`coll|{kind}|{name}`; validate_bundle rejects unknown kinds + dups by
(name,kind); `collection_report`/CollectionInfo + `pic collections ls` gain a
kind column.
Existing kv journeys + nearest-wins still green (the bare-string form normalizes
to kind=kv end to end).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared-scope authz refinement for group docs collections — same trust shape
as the GroupKv* caps: GroupDocsRead = viewer+, GroupDocsWrite = editor+,
resolved via the group ancestor walk. Wired into app_id() (None),
required_scope() (ScriptRead/ScriptWrite), the Member→group_member_grants
route, and group_role_satisfies. Unit test mirrors the group-kv cap test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Data layer for extending shared group collections from KV to docs. KV behavior
unchanged (callers pass kind="kv").
- 0054_group_docs.sql: widen the group_collections kind CHECK to ('kv','docs')
and add the group-keyed group_docs store (clone of docs/0013, keyed by
group_id, CASCADE on group delete) + its (group_id,collection) and data GIN
indexes.
- group_collection_repo: thread a `kind` parameter through list_for_owner,
resolve_owning_group, insert/delete_collection_tx; add list_all_for_owner ->
(name,kind) for the kind-aware diff (D4). Relocate the injectable
GroupCollectionResolver trait + Postgres impl here (now shared by kv+docs;
resolve takes kind) — was in group_kv_service.
- group_docs_repo: a near-clone of docs_repo keyed by group_id; find reuses the
generalized build_find_query.
- docs_repo: generalize build_find_query(table, owner_col, owner_id, …) — both
are compile-time literals (no injection); app docs passes ("docs","app_id"),
group docs ("group_docs","group_id"). row_to_doc/build_find_query are now
pub(crate) for reuse.
Schema snapshot re-blessed (55 migrations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two review findings:
- F2 (defense-in-depth): validate_bundle_for now rejects `collections` on an
app node — the inverse of the existing group route/trigger guard. The CLI
already prevents it (ManifestApp has no field + deny_unknown_fields), but a
hand-rolled wire Bundle POSTed to /apps/{id}/apply would otherwise insert an
inert app_id-owned group_collections row that no resolver reads.
- F1 (coverage): a journey for nearest-ancestor-group-wins, the CoW-shadowing
guarantee the resolver's `ORDER BY depth LIMIT 1` provides. A parent and a
child group both declare `catalog`; an app under the child writes, and a
second app directly under the parent reads its (separate, empty) store —
proving the deep write stayed in the nearest (child) store. Previously only
the FakeResolver and flat sibling groups were exercised, so the ordering was
untested.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Finish the §11.6 KV slice: declarative authoring, read-only inspection, the
runtime journey, and docs — plus a forced SDK-name fix.
- SDK name: `shared` is a Rhai RESERVED keyword, so `kv::shared(...)` is a parse
error. Renamed the handle constructor to `kv::shared_collection(...)`
(keeps the user's "shared" intent; unambiguous). Updated everywhere.
- CLI manifest: `collections = [...]` on `[group]` only (ManifestApp has no such
field + deny_unknown_fields → an app manifest carrying it is a hard error);
Manifest::collections() accessor; build_bundle emits it.
- plan/apply: a `collection` row group in `pic plan`; created/deleted counts in
the apply summary; collections threaded through PlanDto/NodePlanDto/
ApplyReportDto.
- read-only `pic collections ls --group` → GET /admin/groups/{id}/collections
(viewer+), backed by ApplyService::collection_report + CollectionInfo.
- journey: a group declares `catalog`; app A writes, app B (same subtree) reads
it back; a sibling-subtree app gets CollectionNotShared; `collections ls` +
re-apply NoOp. Full suite 114/114.
- docs: groups-and-project-tool §11.6 (KV-only MVP shipped + deferrals),
sdk-shape.md (the shared-collection handle + trust model), CLAUDE.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread group-collection markers through the apply engine — a name-only
resource modeled field-for-field on extension_points (§5.5): Bundle.collections,
CurrentState.collection_names, Plan.collections (+ is_noop), diff_collections
(declared→Create / live-undeclared→Delete / else NoOp), load_current loads the
names, reconcile_node_tx inserts on Create + deletes on prune via
group_collection_repo, state_token folds in coll|<name>, validate_bundle
rejects empty/duplicate names (no reserved-name guard — a collection is a data
namespace, not an importable module), ApplyReport gains collections_created/
deleted. Pruning a marker removes only the declaration; the group_kv_entries
data survives until the owning group is deleted.
The apply engine stays owner-generic; restricting declarations to [group] nodes
is enforced at the CLI manifest layer (next commit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The runtime for shared group collections. A script reaches a shared store via
the explicit kv::shared("name") handle (distinct GroupKvHandle Rhai type, so
private-vs-shared scope is a deliberate choice). The service resolves the
owning group from cx.app_id's ancestor chain — the script never names a group,
and that walk is the isolation boundary (a foreign app's chain never contains
the owning group → CollectionNotShared).
- shared: GroupKvService trait + GroupKvError + NoopGroupKvService; threaded
into the Services bundle as a field defaulted to noop, opted into via a new
with_group_kv() (keeps the ~14 Services::new call sites unchanged).
- manager-core: GroupKvServiceImpl with the reads-open/writes-authed split —
reads use script_gate (anonymous public scripts skip), writes use the new
script_gate_require_principal (anonymous fails closed). Owner resolution
behind a GroupCollectionResolver trait (Postgres impl + injectable fake);
unit tests cover off-chain CollectionNotShared, anonymous-read-OK/
anonymous-write-Forbidden, authed-no-role-Forbidden.
- executor-core: GroupKvHandle + kv::shared constructor + get/set/has/delete/
list (Rhai dispatches by receiver type, reusing the block_on bridge).
- picloud: wire PostgresGroupKvRepo + resolver + GroupKvServiceImpl.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared-scope authz refinement for group collections. Two group-scoped caps
resolved via the existing group ancestor walk (effective_group_role):
GroupKvRead = viewer+, GroupKvWrite = editor+. Wired into app_id() (None —
group caps carry no app_id, so a bound API key is denied at the binding layer),
required_scope() (ScriptRead / ScriptWrite), the Member→group_member_grants
route, and group_role_satisfies. Unit test covers viewer-reads-not-writes,
editor-writes, outsider-denied, bound-key-denied up the chain.
The reads-open/writes-authed trust split (anonymous read bypass; anonymous
write fail-closed) is enforced at the service layer in the next commit — these
caps are the authenticated-principal refinement on top of the structural
subtree boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Data layer for shared cross-app group collections (KV-only MVP), nothing wired
yet. Two migrations + two repos, modeled on the extension_points (0051) marker
and the kv_entries (0007) store:
- 0052_group_collections.sql: owner-polymorphic marker table declaring a
collection name group-shared, with a `kind` discriminator (CHECK 'kv' for
now; generalizes to docs/files/topics/queue later) and per-owner
partial-unique (owner, LOWER(name), kind) indexes. CASCADE — a marker is
config, not code.
- 0053_group_kv_entries.sql: the shared store, keyed by (group_id, collection,
key) — NO app_id (a shared row belongs to the group). CASCADE on group delete
(data dies with its group, like vars/secrets config; an app delete leaves it).
- group_collection_repo: list_for_owner, insert/delete_collection_tx, and the
load-bearing resolve_owning_group — walks the reading app's chain
(CHAIN_LEVELS_CTE) for the nearest ancestor group declaring the name
(nearest-wins via ORDER BY depth LIMIT 1). That walk IS the isolation
boundary; the join is on group_owner only.
- group_kv_repo: a near-clone of kv_repo keyed by group_id.
Schema snapshot re-blessed (53 migrations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The single-node path runs `check_imports_resolve` +
`check_extension_points_provided`; the tree path can't (an in-tree,
uncommitted node may legitimately provide a module, so a pool-based check
would false-positive) and leans on the runtime backstop. Emit a debug log
when a tree apply contains app nodes so the deferral isn't a silent gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`extension_points` is an `[app]`/`[group]` node key. Placed at top level
(before any table header) TOML reads it as a root key — and `Manifest`
silently dropped unknown root keys, the same silent-drop class that bit in
C5 (a key absorbed under `[group]` and discarded). Add
`deny_unknown_fields` to `Manifest`, `ManifestApp`, and `ManifestGroup` so a
misplaced or typo'd key is a hard parse error instead of a no-op apply.
Regression test covers top-level placement, an in-`[app]` typo, and the
correct node-key form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Move the manifest `extension_points` from a top-level key to an `[app]`/
`[group]` node key (`ManifestApp`/`ManifestGroup` + a `Manifest::extension_points()`
accessor). A bare top-level array placed after the node header was silently
re-nested by TOML into that table and dropped; as a node key it parses
unambiguously wherever it sits. build_bundle/pull/init updated.
- Journeys (live server + Postgres): a group default body resolves for an
inheriting app; the app provides its own module and OVERRIDES the EP (the
inverse of the Phase 4b sealed import); `extension-points ls` shows the
provider; a body-less EP with no provider fails `pic plan`.
- Re-bless the schema snapshot (new `extension_points` table).
- Docs: §5.5 marked complete (extension points ✅) + CLAUDE.md current-focus.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- apply: `check_extension_points_provided` (app nodes) — every EP visible on
the app's chain (its own + inherited) must have a provider: a same-apply
module, or one resolvable up-chain (override or default body). Else a clean
`pic plan` error instead of a runtime failure. Single-node only (the tree
path leans on the runtime backstop, like the dangling-import check).
- read-only surface: `extension_point_report` + `ExtensionPointInfo`;
`GET /{apps|groups}/{id}/extension-points` (AppRead / GroupScriptsRead);
`pic extension-points ls --app|--group` showing declared-vs-inherited + the
resolved provider (`app override` / `inherited default` / `unset`).
- `pic pull` round-trips an app's OWN declared EPs (filtered `declared_here`),
so pull→plan stays idempotent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thread extension-point markers through the manifest and the apply engine —
a name-only resource modeled on `secrets` (shape) with `vars`-style
create/prune write behavior.
- manifest: top-level `extension_points = ["theme", …]` (allowed on app and
group; overlay stays base-only via deny_unknown_fields); `build_bundle`
emits the names.
- apply_service: `Bundle.extension_points`, `CurrentState.extension_point_names`,
`Plan.extension_points` (+ is_noop), `diff_extension_points`
(declared→Create / live-undeclared→Delete / else NoOp), `load_current`
loads them, `reconcile_node_tx` inserts on Create + deletes on prune,
`state_token_with_names` folds in `ep|<name>`, `validate_bundle` rejects
duplicates + reserved names, `ApplyReport` gains created/deleted counts.
- CLI client + plan/apply rendering: `extension_points` in PlanDto/NodePlanDto/
ApplyReportDto, an `extension_point` row group in `pic plan`, a count in the
apply summary.
pull sets it empty for now (wired to the read endpoint in C4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the runtime heart of extension points: `ModuleSource::resolve_policy`
decides lexical vs dynamic resolution by the NEAREST declaration of a name
walking up the importing origin's chain — a concrete module → lexical (seal
to origin, Phase 4b); an extension-point marker → dynamic, resolved against
the inheriting app (`cx.app_id`) so the app can override, falling back to the
default body up-chain; the EP wins a depth tie.
- shared: `ModuleResolution { Module | NoProvider | NotFound }` + a
`resolve_policy(origin, inheriting_app, name)` trait method with a
lexical-only default impl (existing fakes inherit it unchanged).
- PostgresModuleSource: one-round-trip nearest-declaration query (min EP depth
vs min module depth over the chain CTEs), then the lexical/dynamic branch.
- module_resolver: calls `resolve_policy(origin, cx.app_id, path)`; maps
NoProvider → a clear "extension point has no provider" error, NotFound →
module-not-found. Cache + AST-source sealing unchanged.
- executor-core tests: a policy fake + 3 cases (app override binds dynamically,
no-provider errors, non-EP stays lexical).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An extension point (§5.5) marks a module name a node offers for descendants
to provide/override — resolved dynamically against the inheriting app rather
than lexically sealed. Lay the storage:
- migration 0051: owner-polymorphic `extension_points(id, group_id?, app_id?,
name)` marker table — exactly-one CHECK + per-owner partial-unique LOWER(name)
indexes + lookup indexes (mirrors 0050). ON DELETE CASCADE (a marker is
config, not code). No default-body column — the optional default body is a
co-located kind=module script (Phase 4b stores/resolves/caches those).
- extension_point_repo: `list_for_owner` + idempotent `insert_extension_point_tx`
/ `delete_extension_point_tx`, keyed by the shared `ScriptOwner` (mirrors the
var/secret tx-fn style).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial-review finding: `create_group_script` validated every source
with `validate()` regardless of kind, so an imperatively-created group
module (`pic scripts deploy --group g --name kv --kind module`) skipped the
two gates every other module-write path enforces (app create/update in
api.rs, declarative apply in apply_service): the stricter `validate_module`
shape check and the `RESERVED_MODULE_NAMES` guard. A malformed-shape group
module was accepted at create (only failing later at import-resolve time)
and a reserved name (`kv`, `log`, …) slipped through.
Branch on kind to mirror the app path. Adds a journey asserting a reserved
group-module name is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two journeys against a live server + Postgres:
- `group_module_is_lexically_sealed_under_inheritance`: a group module +
a group endpoint that imports it, inherited by an app — proves the §5.5
trust boundary (a leaf's same-named module does NOT shadow the inherited
endpoint's import) and app-origin CoW (an app endpoint's import resolves
the app's module).
- `dangling_import_is_rejected_by_plan`: a manifest script importing a
non-existent module is a `pic plan` error.
Docs: mark §5.5 residual resolved + §11 Phase 4b ✅ in the design doc;
update CLAUDE.md current-focus (extension points are the remaining §5.5
piece).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lift the Phase-4-lite restrictions now that import resolution is lexical:
- `create_group_script` (group_scripts_api): allow `kind = module` and
group scripts with `import`s; record the import edges. Module-shape and
sandbox-ceiling validation stay.
- The declarative apply path already reconciles group modules generically
(`BundleScript.kind` + `owner.script_owner()` flow through
`reconcile_node_tx`); no change needed beyond stale-comment fixes.
Adds the §5.5 dangling-import plan check (single-node plan/apply):
`check_imports_resolve` rejects an `import "<m>"` that resolves to neither a
bundle-local module nor a module reachable lexically up the node's chain —
caught at plan time instead of as a runtime 404. Roots resolution at the
node's owner (an app node reaches ancestor group modules; a group node walks
its own ancestry). ApplyService gains an injected `ModuleSource`. The tree
path opts out (a node may import a module created by another node in the same
uncommitted tx); the runtime check is the backstop there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `import` resolution lexical (§5.5): an inherited group script's
imports resolve against the module set at its OWN defining node, walking
up from there — a leaf app can't shadow them, and a group script behaves
identically under every inheriting app.
Mechanism — Rhai's `_source` carries the importing AST's origin tag:
- The entry script has no source tag → resolve from `default_origin` (its
defining node, threaded in via C2).
- Each resolved module's AST source is set to `encode(module.owner())`
before `eval_ast_as_new`, so its nested `import`s carry the module's own
node as `_source` and resolve from there — lexical chaining.
- `encode_origin`/`decode_origin` map a `ScriptOwner` to/from `app:<uuid>`
/ `group:<uuid>`.
Cache rekeyed from `(app_id, name)` to the resolved `ScriptId`: a compiled
module is lexically self-contained, so its body is a pure function of
`(script_id, updated_at)` — this also dedupes a shared group module across
inheriting apps and keeps same-named cross-app modules distinct.
Adds two executor-core unit tests (origin-keyed fake) proving the trust
boundary: a group module's import binds the group's module even when an
app-owned module of the same name exists; `ScriptOwner` gains `Hash`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carry the resolved script's owner (its defining node) from dispatch into
the executor so `import` resolution can be lexical (§5.5). The executing
app (`app_id`) stays the SDK isolation boundary; `script_owner` is a
separate axis — the lexical origin for imports.
- `ExecRequest.script_owner: Option<ScriptOwner>` (serde default; `None`
falls back to `App(app_id)` in the engine for old payloads / cluster wire).
- `ScriptOwner` gains `Serialize`/`Deserialize` for the wire.
- The engine computes `default_origin` and hands it to the resolver
(used fully in C3; the resolve() lookup already uses it).
- Every dispatch site sets it from the resolved `Script.owner()`:
the 4 dispatcher arms (queue/trigger/http/invoke_async, via a new
`ResolvedTrigger.script_owner`), the orchestrator id-bypass, and the
`invoke()` SDK path (new `ResolvedScript.owner`, so a group script
invoked by an app resolves imports from the group).
Behaviour-preserving: app scripts still resolve `App(app_id)`; group
scripts can't yet carry imports (lifted in C4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the module-loading contract origin-aware so group-owned modules
become resolvable and imports can resolve lexically (§5.5).
- `ModuleScript` gains a polymorphic owner (`app_id`/`group_id` + `owner()`),
mirroring `Script`.
- `ModuleSource::lookup(&cx, name)` -> `resolve(origin: ScriptOwner, name)`:
resolution walks the group chain rooted at the importing script's
defining node (nearest-owner-wins), not the inheriting app's view.
- `PostgresModuleSource::resolve` branches on origin: app-rooted
(`CHAIN_LEVELS_CTE`) or a new group-rooted `GROUP_CHAIN_LEVELS_CTE`,
joining `kind='module'` rows.
The executor resolver passes a temporary `App(cx.app_id)` origin (behaviour-
preserving for app scripts; true lexical origin lands in C3). Group scripts
still can't carry imports until C4, so no trust-inversion window opens here.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 C5. Three `pic ... --dir` tree journeys:
* a group + a nested app apply atomically as one tree (the app's route binds
the group's same-apply `shared` script); the group node's script is
created; re-plan is all-noop (idempotent),
* one invalid node aborts the whole tree — the valid group node's script is
NOT created (true all-or-nothing),
* a `pic groups reparent` between `pic plan --dir` and `pic apply --dir` trips
the bound-plan (StateMoved) check via the tree's structure version.
Docs: `groups-and-project-tool.md` §11.5 status (✅ shipped — single-repo
nested tree apply; multi-repo single-owner + per-env approval gating deferred
per §11.1) + CLAUDE.md current-focus.
Gates: fmt, clippy -D warnings, cargo test --workspace, check-versioning (50
migrations — Phase 5 added none), schema snapshot unchanged, full journey suite
108/108. (Pre-existing flake `queue_e2e::queue_receive_acks_on_success` — a
racy immediate `count==0` after the async ack; flaky at the pre-Phase-5
baseline too, unrelated to this work.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 C4. `pic plan --dir <root>` / `pic apply --dir <root>` discover every
`picloud.toml` under a directory (groups + apps), assemble the wire tree
bundle, and drive the C3 `/tree/{plan,apply}` endpoints — a whole project
subtree reconciled and reviewed as one unit.
* New `discover` module: recursive walk (skips `.picloud`/`.git`/`scripts`/
`target`), one node per manifest, app/group inferred from the manifest;
rejects a duplicate (kind, slug). On-disk nesting is organizational — the
server resolves ancestry from its own group tree.
* `client`: `plan_tree`/`apply_tree` + `TreePlanDto`/`NodePlanDto`. `pic plan
--dir` renders a per-node diff table and records ONE tree bound-token under
a `<tree>` linkstate marker; `pic apply --dir` replays it (force/`--yes`/
prune apply tree-wide). `--dir` conflicts with `--file`.
Live-validated: a nested tree (root group dir + nested app dir, app route bound
to the group's inherited script) → `pic plan --dir .` shows both nodes' changes
→ `pic apply --dir .` applies all in one tx → re-plan all-noop.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 C3 — the headline. A whole project subtree (group + app nodes) is
reconciled in ONE Postgres transaction, all-or-nothing.
`POST /api/v1/admin/tree/{plan,apply}` take a `TreeBundle { nodes: [{kind,
slug, bundle}] }`. The actor must hold the relevant read/write caps for EVERY
node (per-kind, widened on prune) — `authz_tree` + reusable
`require_{app,group}_node_writes` (now shared with the single-node handlers).
Engine:
* The in-tx reconcile body of `apply_owner` is extracted to `reconcile_node_tx`
(returns the node's post-create `name → script_id`), so single-node and
tree apply share one implementation — behaviour-preserving (391 unit tests
+ app journeys green).
* `prepare_tree` resolves each slug→owner, validates (an app's route/trigger
target may bind to an in-tree ancestor group's script), computes each
node's plan, and folds a combined bound-plan token = every node's
`state_token` + every in-scope group's `structure_version` (so a reparent
or content edit between plan and apply trips StateMoved).
* `apply_tree`: lock every node key (sorted, deadlock-free); reconcile GROUPS
first, recording each group's `name → id` in an in-memory index; then APPS,
resolving inherited route/trigger targets nearest-ancestor-wins across the
in-tree index (incl. scripts created earlier in THIS tx, invisible to the
pool resolver) and pre-existing out-of-tree ancestors. One commit, one
post-commit route refresh.
Live-validated against the dev DB:
* Headline: a group node creates `shared`; an app node in the SAME apply
binds `GET /greet` to it — the route ends up bound to the group-owned
script, and re-plan is all-noop (idempotent).
* Atomicity: one invalid node → 422 and ZERO rows written (the valid group
node's script is not created).
CLI tree discovery + `pic plan/apply` tree mode is C4; journeys + docs are C5.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 C2. A `picloud.toml` can now declare a `[group]` node (instead of
`[app]`): the group's own scripts + `[vars]` (+ secret names). `pic plan` /
`pic apply` reconcile it via the C1 group endpoints — the same plan/apply/
bound-token flow as an app, routed by node kind.
* `Manifest` is now app-XOR-group: `app`/`group` are both optional with an
exactly-one check in `parse`, plus a group-node guard that rejects
`[[routes]]`/`[[triggers]]` (those bind to an app). Accessors `slug()` /
`is_group()` replace the hardcoded `manifest.app.slug`.
* `client`: `plan_node`/`apply_node` take a `NodeKind { App | Group }` that
selects the `apps` vs `groups` API path; the old `plan`/`apply` wrappers
are gone (callers pass the kind).
* `pic plan`/`apply` detect the node kind from the manifest and label output
`group`/`app`; `pic config --effective` cleanly rejects a group manifest
(effective config is an app's inherited view).
Live-validated: a `[group]` manifest with a script + var → `pic plan` (script
+ var creates) → `pic apply` (group-owned) → idempotent re-plan noop →
`pic scripts ls --group` shows it. Manifest unit test for group parse +
app-only-block rejection; init/pull/plan/apply/overlay journeys all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 5 C1. Generalize the reconcile engine from app-only to an `ApplyOwner
{ App | Group }`, so a GROUP's own config + scripts can be declaratively
reconciled — the foundation for the tree-spanning project tool.
A group node is a subset of an app node (scripts + vars only; no routes /
triggers / secrets-values), so the diff, script/route/trigger/var CRUD, prune,
idempotency, and the Phase-4 inherited-target resolution are all reused
unchanged. Only the owner-varying bits switch on `ApplyOwner`:
* `load_current` — `list_for_group` + `VarOwner::Group` for a group (empty
routes/triggers/secrets); `list_for_app` for an app, exactly as before.
* script create owner (`NewScript{app_id|group_id}`), var writes (new
`set_group_var_tx`/`delete_group_var_tx` mirroring the app variants at
scope `*`), and the advisory lock (owner-tagged).
* `validate_bundle_for` rejects routes/triggers on a group bundle (422);
route-host validation, inherited-target resolution, the post-commit route
refresh, and the unreachable-endpoint warning are app-only.
`apply`/`plan` are unchanged thin wrappers over the new `apply_owner`/
`plan_owner`. New endpoints `POST /groups/{id}/{plan,apply}` gated by
`GroupScripts{Read,Write}` / `GroupVarsWrite`. `ApplyService` gains a `groups`
repo (the tree apply in C3 needs it too).
Live-validated: group plan → apply (group-owned script + var) → idempotent
re-plan noop → routes rejected 422 → DB confirms group ownership. App apply
unchanged: 391 manager-core unit tests + the apply/prune/staleness/overlay
journeys all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review of Phase 4-lite found two real gaps, both rooted in the
apply engine resolving script `id → name` from `current.scripts` (app-own)
only — so a route/trigger bound to an inherited GROUP script (whose id is not
app-own) couldn't be named:
* **Prune gap (reachable via pure declarative apply):** a trigger bound to an
inherited group script, once dropped from the manifest, never diffed to a
Delete (the diff couldn't resolve its identity once the bundle stopped
referencing the name), so `apply --prune` silently orphaned it.
* **Bound-plan false-negative:** `state_token` excluded such a trigger from
its fingerprint, so the StateMoved check missed a concurrent edit to that
binding class. (Routes were already safe — they hash the raw `script_id`.)
Fix: `resolve_current_target_names` resolves the names of group scripts the
app's CURRENT routes/triggers are bound to (by id, independent of the bundle),
and that map now feeds `compute_diff_with_names`, `state_token_with_names`, and
the prune trigger-identity map. App-owned bindings are unaffected (their names
are already in `current.scripts`). Also fixes a stale "in this app" conflict
message for group-owned scripts.
New journey `inherited_bound_trigger_is_pruned_when_dropped` proves prune now
removes an inherited-bound cron trigger. Full journey suite 105/105.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mark §11.4 ✅ shipped with the live-resolution decision (no body
materialization / invalidation fan-out, mirroring Phase 3's config), the
shipped surface, and the deliberate Phase-4b deferrals (group modules + lexical
import resolver, invoke-by-id, live auto-rebinding). Note the sharp edge:
routes.script_id ON DELETE CASCADE means deleting a group script drops
descendant routes. Update the §5.1 materialization box and CLAUDE.md
(current-focus + the data-model invariant now covers group-owned scripts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4-lite C5. Two `pic` journeys:
* `group_script_is_inherited_with_cow_and_isolation` — a group endpoint
`shared` is invoked by name from an app under the group; the app's own
`shared` then shadows it (CoW); an app outside the group cannot reach it.
* `manifest_binds_route_to_inherited_group_script_idempotently` — a manifest
that declares no `greet` script binds a route to the inherited group
`greet`; plan shows a route create (no script create), apply succeeds, and
re-plan is a clean no-op.
Adds a `ScriptGuard` (deletes a script by id on drop) since a group-owned
script blocks its group's deletion (ON DELETE RESTRICT) — it must drop before
the GroupGuard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4-lite C4 — the headline declarative path. An app's manifest can now bind
a `[[routes]]` / `[[triggers]]` to a script it does NOT declare, resolving the
name to an inherited group-owned endpoint on the app's chain (nearest-owner
wins; an app's own script of the same name still shadows it — CoW).
* `resolve_inherited_targets(app_id, bundle)` resolves every route/trigger
target name that isn't an app-own manifest script to a group endpoint via
`get_by_name_inherited`, returning `name → script_id`. Run once before the
apply transaction (group scripts pre-exist, committed).
* `validate_bundle` takes the inherited endpoint names so "binds to unknown
script" / "is a module" checks pass for inherited targets.
* The apply transaction merges the inherited targets into `name_to_id`
(app-own keeps precedence), so route/trigger inserts bind to the group
script's id.
* `compute_diff_with_inherited` adds the inherited ids → names to the diff's
name map, so a route bound to a group script resolves its name and
re-apply is idempotent (without this it read as a perpetual rebind).
Live-validated: an app under group G with a manifest binding `GET /greet` to
the group's inherited `greet` plans as `route create → greet` (no script
create), applies (+1 route, +0 scripts), and re-plans as `noop` (idempotent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4-lite C3. A descendant app can now resolve and run a group-owned script
inherited down its ownership chain — the core of group-script inheritance.
Resolver (repo, reusing config_resolver's CHAIN_LEVELS_CTE):
* `get_by_name_inherited(app_id, name)` — walk app → ancestor groups
nearest-first, return the nearest script of that name (an app's own script
shadows an inherited group one: CoW).
* `is_invocable_by_app(script_id, app_id)` — EXISTS over the same chain; the
cross-app isolation predicate generalized to the hierarchy.
invoke(): `invoke("name", …)` resolves via `get_by_name_inherited`, so a script
can call a shared group endpoint by name. A group script always runs under the
*caller's* app context, never its owner.
Dispatch guards (the isolation backstops touched in C1) are now chain-aware via
a `Dispatcher::script_invocable` helper and the chain-aware
`validate_trigger_target`: app-owned scripts keep the in-memory fast path (no
query, behavior byte-identical to the pre-Phase-4 same-app guard); only a
group-owned candidate pays one chain query, and a non-member / query error
fails closed (dead-letter or reject — never run under another app's SdkCallCx).
The orchestrator HTTP route path needs no change: its per-app route trie
already scopes routes to one app, and it dispatches under the route's app
without reading script.app_id — a route bound to a group script just works.
Live-validated against the dev DB: an app under group G invoking the group's
`shared` returns its output; an app NOT under G cannot resolve it (isolation);
and after the app deploys its own `shared`, the same call returns the app's
version (CoW shadowing). Route/trigger binding of inherited scripts via the
declarative engine is C4.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4-lite C2. Lets a group own scripts (templates inherited by descendant
apps), with the create/list/manage surface — but not yet the inherited
resolution (binding + runtime), which is C3.
Capabilities: add `GroupScriptsRead` (viewer+ on the group → script:read) and
`GroupScriptsWrite` (editor+ → script:write), mirroring the group-vars tier and
resolved through the same group-ancestor walk.
API:
* New `group_scripts_api`: `POST/GET /groups/{id}/scripts` — create a
group-owned endpoint script and list a group's own (non-inherited) rows.
Phase 4-lite is endpoint-only and self-contained: `kind=module` and any
`import` are rejected (group modules + the lexical resolver are Phase 4b).
Owner resolved first (slug-or-uuid); capability bound to the resolved id.
* The by-id `/scripts/{id}` get/update/delete/logs handlers are now
owner-polymorphic via a `script_cap` helper: app-owned scripts gate on
`App*` exactly as before; group-owned on `GroupScripts*`. This is what
makes `deploy --group` idempotent (update reuses the by-id PUT) and lets a
group script be deleted by id. (C1 had these fail closed for groups.)
Repo: `list_for_group(group_id)` (the group's own rows).
CLI: `pic scripts ls --group <g>`, `pic scripts deploy --group <g>` (create or
update by name; `--app`/`--group` are mutually exclusive, exactly one
required). `pic scripts delete <id>` already works for group scripts via the
owner-polymorphic by-id route.
Live-validated against the dev DB: create → update (v2 via the by-id PUT) →
list → delete, plus module rejection and the polymorphic row shape
(`app_id NULL`, `group_id` set). Group scripts still can't be routed/triggered
or invoked — that lands in C3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 4-lite C1. Make script ownership polymorphic so a script can be owned
by a GROUP (a template inherited by descendant apps) instead of an app —
mirroring vars/secrets (0048/0049), but ON DELETE RESTRICT (code is not data).
Schema (0050): `scripts.group_id` (nullable FK→groups RESTRICT), `app_id` made
nullable, `scripts_owner_exactly_one` CHECK, and the per-app name index split
into two per-owner partial-unique indexes. Existing app-owned rows keep their
exact `(app_id, lower(name))` uniqueness.
Type: `Script.app_id` becomes `Option<AppId>`; add `Script.group_id` +
`ScriptOwner` / `is_owned_by_app()`. `NewScript` gains the same polymorphic
owner. The execution-context app (what a script runs *under*) is supplied by
the invoking route/trigger/caller, never read off the script — group scripts
have no single app.
Behavior is fully preserved for app-owned scripts (the only kind creatable
today): every isolation backstop and authz site now uses `is_owned_by_app`,
which is byte-identical for app owners and **fails closed** for group owners.
A group script therefore can't yet be run, route/trigger-bound, invoked, or
managed via the app-script API — those land in C2 (group-script creation) and
C3 (chain-membership resolution + binding). The `/execute/{id}` bypass 404s a
group script (no app context to run under).
Re-blesses expected_schema.txt; note the golden was last blessed at migration
0044, so this also captures the already-committed 0045–0049 schema.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`apply_reconciles_app_vars` drives the full cycle through `pic` against a
real server: a manifest `[vars] region = "eu"` is applied (var created +
injected — a script `vars::get("region")` returns "eu"), the value is
changed and re-applied (Update, returns "us"), then the var is dropped and
`apply --prune` removes it (`vars::get` resolves to `()` / JSON null).
Exercises the Bundle.vars wire, diff_vars create/update/delete, the in-tx
writes, and the runtime resolver end to end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the vars round-trip: pull now fetches the app's OWN env-agnostic
vars (`vars_list` filtered to scope `*`, non-tombstone) and writes them
into the manifest `[vars]` block, alongside scripts/routes/secrets.
Inherited group vars stay out (pull exports own rows only, §4.6); a value
TOML can't represent (e.g. JSON null) is skipped with a warning. So
`pic pull` → edit → `pic apply` is now lossless for app config vars.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a `[vars]` block to `picloud.toml` (key → TOML value), reconciled by
`pic plan`/`pic apply` alongside scripts/routes/triggers/secrets. Unlike
`[secrets]` (name-only), var values live inline since they aren't secret.
The overlay (`picloud.<env>.toml`) may carry `[vars]`; overlay keys win
per key (the env-specific value of an env-agnostic default).
`build_bundle` serializes the vars to JSON for the wire (TOML round-trips
via serde); `pic plan` shows a `var` row group and `pic apply` reports
`vars +c ~u -d`. The `PlanDto`/`ApplyReportDto` gain the matching fields.
`pic pull` carries an empty `[vars]` for now (export lands next). Adds a
round-trip + overlay-precedence unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the Phase-1 apply engine to manage app `vars` declaratively,
alongside scripts/routes/triggers/secrets. The `Bundle` gains a `vars`
map (key → JSON value; values are non-secret, so they live inline, unlike
secret names). `diff_vars` is value-sensitive: a changed value is an
Update, a live var the manifest dropped is a Delete (applied only under
`--prune` — vars are prunable config, unlike never-pruned secrets).
Writes go through `set_app_var_tx`/`delete_app_var_tx` inside the apply
transaction (mirroring `PostgresVarsRepo::set` for `VarOwner::App`, scope
`*`), so they commit atomically with the rest of the apply and roll back
together on failure. `CurrentState` loads the app's OWN scope-`*`,
non-tombstone vars (inherited group vars and tombstones are out of scope
for the manifest); `state_token` folds them in so the bound-plan check
catches an out-of-band `pic vars set` between plan and apply. Keys are
validated against the same kebab rule as the vars admin API, and the
apply handler now requires `AppVarsWrite` when vars are touched or
`--prune` is set.
Scope: app-owned vars at scope `*` (an app *is* an environment). Group
vars via manifest and tombstone-via-manifest wait for the nested-manifest
model. Unit tests cover the create/update/noop/delete diff and the
state-token value sensitivity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the net-new env-scoped `vars` table + the §3 resolution engine
(env pre-filter, proximity-first, per-key map merge, explicit tombstones),
group-owned environment-scoped secrets with owner-discriminated AAD
(secret:group:{group_id}:{name}, disjoint from the unchanged app AAD so
existing ciphertexts still decrypt), masked group-secret reads (app devs
see exists, never plaintext; only owning-group principals read values;
runtime injection bypasses the human gate), the vars:: SDK, and
pic vars / pic secrets --group / config --effective. Migrations 0048
(vars) and 0049 (group secrets reshape).
Verified before merge: fmt/clippy(-D warnings) clean; manager-core +
executor-core 523 unit tests pass (incl. AAD cross-owner-swap rejection
and the pure §3 resolver semantics); check-versioning OK (49 migrations);
all 101 CLI journeys pass against a live DB, including masked-read
(group_secret_value_is_masked_from_app_devs), runtime injection +
override, and var inheritance. Commit 79f8c9d also fixes the 6
long-standing stale logs/scripts/roles journey assertions, so the suite
is now fully green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The groups/project-tool work had no home in the blueprint's version
scheme, two "Phase N" numbering schemes overlapped, and CLAUDE.md's
"current focus" was stale (claimed v1.1.0 SDK foundation; we're at v1.1.9
with the SDK line complete). Reconcile all three:
* blueprint §12 Phase 5 (v1.2, already titled "Advanced Workflows &
Hierarchies") — split into a Hierarchies track (the groups + project
tool initiative, with §11 Phases 1–3 marked shipped) and a Workflows
track, with sequencing flagged as an open call. This gives the work an
explicit version home: the Hierarchies half of v1.2.
* design doc header — Status Draft → Active, "scheduled as the Hierarchies
track of v1.2", and an explicit warning that §11's local Phase 1–6
numbering is distinct from the blueprint product-phase numbering.
* design doc §11.1 — a post-Phase-3 re-sequencing review: resolve Phase 4
scripts LIVE (drop the fan-out-invalidation sub-project, per the Phase 3
result); consider a "Phase 5-lite" (declarative apply of vars/secrets +
app scripts) before the hard group-scripts work; and force the two
deferred decisions (trigger/route templates vs tenant cardinality;
multi-repo ownership vs a single-repo start).
* CLAUDE.md — refresh "current focus" to v1.2 Hierarchies, and correct the
now-broken "every v1.1+ table is app_id NOT NULL" invariant: the new
group-inheritable config tables (vars, secrets) use a polymorphic owner
(group_id XOR app_id), not app_id NOT NULL.
Doc-only; no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 (group-inherited config) is implemented, so fold the outcome back
into the design doc (per its own "before it drifts" note) — and flag the
one deliberate divergence from the plan.
§5.1 / §11.3 / §12 prescribed a *materialized* per-app effective view with
a generation counter and fan-out invalidation. We shipped **live,
resolve-on-read inheritance** instead: one recursive CTE per
`vars::get`/`secrets::get`/`config/effective`, no cache. The org tree is
small and there was no pre-existing config cache to fight, so this is
cheap and sidesteps §10's unsolved invalidation SLA.
Records, mirroring the Phase 1/2 status blocks:
* a "Status (Phase 3): ✅ shipped" block in §11.3 with the full surface
(0048/0049 schema incl. `apps.environment`, the §3 resolver, vars SDK +
CRUD, group-secret AAD + inherited read, the two secret gates, masked
`config/effective`, `--explain`) and the live-vs-materialized rationale;
* §5.1 runtime-model bullet annotated as the deferred (Phase-4 script
inheritance) design, not Phase-3 runtime;
* §10 "invalidation SLA" and "inherited-membership revocation lag" marked
moot/resolved — no cache means immediate propagation;
* §12 resolver contract + the header framing updated to live resolution.
Doc-only; no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six journeys asserted against output formats that changed in earlier
commits and had been failing regardless of branch:
* `pic scripts deploy` prints a `name`/`version`/`action` KvBlock, not a
prose "Created X vN" line. The name-override, version-bump, and the
editor-can-deploy role check now parse the KvBlock fields (asserting the
exact version + created/updated action) via a small `kv_field` helper.
* `pic logs` gained a `source` column (now `created_at, source, status,
summary`); the success/error-status and summary-truncation checks now
index the status at col 2 and the summary at col 3.
No production code touched — the deploys/logs always succeeded; only the
assertions were stale. Full `--test cli --include-ignored` suite is now
green (101 passed, 0 failed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`list_meta` orders by `(name, environment_scope)` — a group secret name
can carry several env-scoped rows — but the cursor encoded only `name`
and paginated with `name > $cursor`. When a name's scopes straddled a
page boundary, the remaining scope rows were silently skipped from the
admin listing. Switch to a composite `(name, environment_scope)` keyset
(base64url of `name \x1f scope`) and a Postgres row-comparison predicate.
App secrets (single `*` scope per name) were unaffected; group secrets
with multi-scope names now page completely. Found in final branch review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two end-to-end journeys via `pic` against a real server:
* `group_secret_is_injected_then_app_value_overrides` — a group-owned
secret is injected into a descendant app's script through `secrets::get`
(decrypted under the group AAD), then an app-owned secret of the same
name shadows it (decrypted under the app AAD). Drives the resolver + both
owner-AAD open paths against Postgres.
* `group_secret_value_is_masked_from_app_devs` — the §5.3 boundary: an app
dev (editor on the app, no group role) is denied the secret VALUE (403)
yet still sees it EXISTS, masked, in `config/effective` (status set,
owner group, no value); a group_admin reads the value. Proves an app
runs with config its own devs cannot read.
Also updates the existing app-secrets `ls` journey for the new `env`
column (group secrets are env-scoped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirrors `pic vars` on the secrets surface: `pic secrets ls/set/rm` take an
`--app`/`--group` owner selector (exactly-one) with `--env` for group
secrets. Adds `pic secrets read --group <g> <name> [--env]` — the only
command that reveals a secret value, hitting the group-gated value
endpoint. `pic config --effective` now folds in the resolved vars section
(value + owner + scope) and an `--explain` provenance view, alongside the
existing masked-secrets cross-reference, via `GET /config/effective`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the Phase-3 admin surface on top of the group-secrets storage:
* `secrets_api` gains group routes under `/groups/{id}/secrets`
(set/list/delete, env-scoped) gated `GroupSecretsWrite` (editor+), plus
the ONE plaintext endpoint `GET /groups/{id}/secrets/{name}/value` gated
`GroupSecretsRead` (group_admin only). That is the masked-secret
boundary: a descendant app's dev sees a group secret EXISTS and consumes
it at runtime via `secrets::get`, but only a reader at the OWNING group
gets the value. App secrets stay env-agnostic (a stray `env` is rejected).
The owner is resolved first, then the capability binds to the resolved
id — never a path param.
* `config_api`: `GET /apps/{id}/config/effective` (gated `AppVarsRead`)
returns the resolved view a dev would get — every inherited var with its
value + provenance, and every inherited secret MASKED (name/owner/scope,
never the value). Backed by a new `fetch_effective_secret_meta`
(DISTINCT-ON nearest-wins, same ordering as the per-name resolver).
* authz: `GroupSecretsWrite` moves from `app:admin` to `script:write`
scope so its API-key scope matches its editor role tier (closing the
latent scope/role mismatch the checkpoint review flagged); the value
read `GroupSecretsRead` stays at `app:admin`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migration 0049 reshapes `secrets` to the same polymorphic-owner contract
as `vars` (0048): a secret is owned by an app XOR an ancestor group,
carries an `environment_scope`, and the old PK `(app_id, name)` becomes
two partial-unique indexes `(owner, environment_scope, name)`. Existing
rows backfill to `app_id` + scope `'*'`; the v1 AAD does NOT include the
scope, so every current ciphertext keeps decrypting byte-for-byte.
`SecretsRepo` is generalised to be owner+scope keyed (`SecretOwner` moves
down from `secrets_service` and is re-exported for path stability). The
SDK read path now goes through `SecretsRepo::resolve`, which reuses the
shared `CHAIN_LEVELS_CTE` to walk app→ancestor-group→root, env-filters,
and takes the nearest level (`@E` beating `*` within a level) — returning
the winning owner so the value is decrypted under the AAD it was sealed
with. Runtime injection stays anchored to `cx.app_id`: an app only ever
resolves its own and its ancestors' secrets.
All callers updated to owner+scope (`SecretOwner::App(_)`, scope `'*'`):
the app-secrets admin API, the SDK service, and the apply email-secret
path. The 21 secrets unit tests (incl. the app/group AAD-disjointness and
cross-row swap proofs) stay green; the chain-walk was live-verified
against Postgres (nearest-wins + env-filter). Admin API for group secrets
and the masked-read endpoint land next (Step E).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
§3 step 1 treats an environment scope as *eligibility*, not a merge tier:
within a single level a value scoped `@E` shadows the same level's `*`
fallback — it never layers on top of it. The resolver's map-run loop
collected every leading object row regardless of (depth, scope), so a
level holding both an `@E` map and a `*` map would deep-merge the two
(and report a bogus `*` layer in `merged_from`) rather than letting the
`@E` map win alone.
Dedup the sorted candidates by depth before the map run: each level is a
single owner (single-parent tree) with at most one `@E` and one `*` row
per key after env-filtering, and `@E` sorts first, so keeping the first
row per level yields the level's winner. Scalar/tombstone cases already
went through the boundary path and were unaffected; only same-level
map-vs-map mis-merged.
Adds two regression tests (same-level suppression; suppressed `*` stays
invisible to cross-level merge). Found in checkpoint review.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The crypto foundation for group-owned secrets (the §5.3 'single hardest
correctness detail'), done first and proven before the storage/resolution
layer.
- SecretOwner{App(AppId)|Group(GroupId)}; secret_aad/seal/open take the
owner. The app AAD is byte-identical to the pre-Phase-3
'secret:{app_id}:{name}', so every existing v1 row decrypts unchanged;
group secrets use a disjoint 'secret:group:{group_id}:{name}' namespace
(the 'group:' infix separates app/group AAD even for equal UUIDs).
- All callers (SDK get/set, secrets_api, apply email-secret read) wrapped
in SecretOwner::App — behavior identical for app secrets.
- New audit test aad_distinguishes_app_and_group_owner proves the two
namespaces don't collide (both directions, even reusing the UUID); the
existing cross-app/cross-name swap audit tests stay green.
16 secrets_service tests pass; clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the vars half of Phase 3 end-to-end:
- vars_repo: VarOwner{Group|App} upsert/delete/list (owner-kind-specific
SQL; ON CONFLICT restates the partial-index predicate).
- vars_api: GET/PUT/DELETE under /apps/{id}/vars and /groups/{id}/vars,
resolve-then-require gated on App/GroupVars{Read,Write}, secrets-style
error mapping + key/env-scope validation.
- pic vars ls/set/rm (--group|--app, --env, --json, --tombstone).
- journey test: a group var is inherited by a descendant app's
vars::get(), and an app-level value overrides it (proximity) — green.
386 manager-core lib tests + the vars journey pass; clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scripts can now read group-inherited config via vars::get(key) / vars::all().
- shared: VarsService trait (read-only get/all) + NoopVarsService; added as
the 14th field on the Services bundle.
- manager-core: VarsServiceImpl resolves the calling app's config via
config_resolver (derives app_id from cx.app_id; AppVarsRead-gated for
authed principals, script-as-gate for anon).
- executor-core: vars:: Rhai bridge (get/all), registered in the SDK.
- authz: AppVarsRead/Write, GroupVarsRead/Write, GroupSecretsRead/Write
capabilities (group caps resolve via the Phase-2 ancestor walk; the
GroupSecretsRead human-read gate is group_admin-only, distinct from an
app's runtime injection).
- wired VarsServiceImpl into the picloud binary; updated the executor-core
test Services::new call sites.
386 manager-core lib tests green; clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 3 foundation (docs §3): env-filtered, proximity-first config
inheritance down the group tree.
- Migration 0048: `vars` (polymorphic group|app owner via two nullable FKs
+ CHECK exactly-one, env scope, JSONB value, explicit tombstone) +
`apps.environment` (the env marker the resolver filters on — 'an
environment is an app').
- config_resolver: a shared chain-walk CTE (mirrors effective_app_role) that
walks app → ancestor groups, env-filters in SQL, plus a pure Rust
resolution pass implementing the §3 semantics SQL can't — per-key
proximity-first, @E-over-* within a level, map deep-merge, tombstone
suppression, and provenance for --explain.
Verified: 8 unit tests (incl. the §3.2 proximity-beats-env-specificity call,
deep-merge, tombstone, boundary) + the candidate-fetch CTE against live
Postgres (env-filter drops a non-matching @production value; nearer level
shadows farther).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the GitLab-style single-parent group tree above apps: migration
0047 (groups + group_members + apps.group_id NOT NULL/RESTRICT with
single-root backfill), tree repos with an ancestor-walk cycle guard
under a coarse structural advisory lock, slug-freeze and structure_version,
hierarchy-aware RBAC (effective role = MAX across app membership + every
ancestor group, resolved live so revocation is immediate), new
Group{Read,Write,Admin}/InstanceCreateGroup caps that bound API keys
cannot obtain, plus `pic groups` and a dashboard group tree/detail/members.
Verified before merge: fmt/clippy(-D warnings)/check clean; manager-core
378 unit tests pass (incl. hierarchy authz + group repos); dashboard
npm run check 0 errors; check-versioning OK (47 migrations sequential).
New CLI groups journeys pass against a live DB — cycle guard
(reparent-into-descendant rejected), delete=RESTRICT, and
inherited-group-admin deploy + live revoke. The 6 failing logs/scripts/roles
journeys are the same pre-existing format drift on main, not regressions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SvelteKit UI for Phase-2 groups, mirroring the apps/users patterns:
- api.ts: Group/GroupDetail/GroupMember types + an api.groups client
(list/get/create/update/reparent/delete + nested members CRUD); optional
group selector wired into app create.
- routes/groups: a collapsible tree overview (assembled from parent_id)
with a New-group form, and a detail page with breadcrumb, subgroups/apps
lists, a rename form (no slug field — slug is frozen), a reparent
control, a delete button that surfaces the 409 RESTRICT message, and a
Members tab reusing RoleChip/ConfirmModal/ActionMenu.
- +layout.svelte: a Groups nav entry beside Apps.
npm run check: 0 errors; npm run build: success.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the `pic groups` command family over /api/v1/admin/groups:
ls/tree, create (--parent), show (path + subgroups + apps), rename
(name/description; slug frozen), reparent (--to), rm (--recursive,
leaf-first; never auto-deletes apps), and members add/set/rm/ls. Also
`pic apps create --group <slug>` to place an app under a group.
End-to-end journey tests (tests/groups.rs, DB-gated) cover tree CRUD,
delete=RESTRICT (409 on a non-empty group), reparent cycle rejection, and
the headline invariant: a group_admin on an ancestor group can deploy to
an app it is NOT a direct member of (inherited membership), and loses that
access immediately on revoke — all green against a live server.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Server-side foundation for Phase-2 groups (no group-owned resources yet):
Shared types:
- GroupId, Group; App gains group_id; AppRole::{precedence,max} for
folding the highest effective role across the membership chain.
Repos:
- group_repo: tree CRUD with reparent (ancestor-walk cycle guard under a
coarse instance-wide structural advisory lock; slug frozen; bumps
structure_version) and delete=RESTRICT (refuses non-empty groups).
- group_members_repo: per-(user, group) role grants, mirroring app_members.
Hierarchy-aware authz (§5.3):
- AuthzRepo gains effective_app_role / effective_group_role (default to
direct membership / none, so the ~18 existing test stubs are untouched);
the Postgres impl resolves each via one depth-bounded recursive CTE that
MAXes the app's own row with every ancestor group_members row.
- can(): the Member path now folds inherited group roles, so a group_admin
on any ancestor is implicitly app_admin beneath it. New Capability
variants InstanceCreateGroup / Group{Read,Write,Admin}; group caps carry
no app_id (bound API keys can't manage groups). 8 new unit tests.
Admin API:
- groups_api: group CRUD + reparent (admin at both source and destination
parent, §5.6) + per-group members, all capability-gated.
- apps: POST /apps takes an optional parent group (default root); app
responses carry group_id; my_role now reflects the effective role.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 2 (blueprint §5): groups form a single-parent org tree ABOVE apps.
This migration adds the tree + membership tables and gives every app a
parent (§9 adoption):
- groups: id, parent_id (self-FK ON DELETE RESTRICT), slug (instance-
global UNIQUE, frozen at creation), name, description, structure_version
(bumped on structural mutation), owner_project (inert §7 seam).
- group_members: (group_id, user_id, role) with the SAME three role
literals as app_members so AppRole round-trips and the authz rank table
covers both.
- apps.group_id: nullable add → backfill every app under a seeded 'root'
group → promote to NOT NULL + FK RESTRICT.
No group-owned resources yet (scripts/secrets stay app-owned — Phase 3).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the declarative `pic` project tool (init/pull/plan/apply/prune,
env-scoped overlays, config --effective), the three-state `enabled`
lifecycle on scripts/routes with dispatcher fire-time re-checks, the
trigger `name` column + backfill, and cross-app/disabled-execution
backstops on every async dispatch path. Design captured in
docs/design/groups-and-project-tool.md.
Verified before merge: cargo fmt/clippy(-D warnings)/check clean;
manager-core 370+ unit tests pass (incl. dispatcher enabled-gates and
cross-app isolation); all new CLI journey tests pass against a live DB.
The 6 failing logs/scripts/roles journey tests are pre-existing format
drift on main, not regressions from this branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update §4.3: the fire-time `enabled` re-check is shipped and test-backed on
BOTH async arms (the unified outbox `!resolved.active` gate and the queue
arm's fresh-script nack), with the specific regression tests cited, plus the
same-app backstop now present on every async build path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Incidental `cargo fmt --all` normalization of two pre-existing over-width
`.lock()` chains in DevEmailSink; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Four review fixes on the `pic` surface:
- `config --effective` gains `--env`: it now resolves through the same
overlay as `plan`/`apply`, so the masked report targets the app those
commands act on instead of silently reading the base slug's secrets.
- `pull` gains `--force` and refuses to overwrite an existing
`picloud.toml` (and colliding `scripts/*.rhai`) without it — fail-fast
before any network call or write, mirroring `init`.
- Overlays (`ManifestOverlay`/`OverlayApp`) get `deny_unknown_fields`: a
`[[scripts]]`/`[[routes]]`/`[[triggers]]` table or a typo'd key in an
overlay now errors loudly instead of being silently dropped. +unit test.
- `apply` 409 (StateMoved — the only 409 the endpoint emits) surfaces an
actionable hint (re-run `pic plan`, or `--force`) instead of a bare
`HTTP 409`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The interactive create path rejects a module whose name shadows a built-in
SDK namespace (`kv`, `secrets`, …) so `import "kv"` can't resolve to a user
module, but `pic apply`'s `validate_bundle` skipped the check — a parity gap
that let a manifest create what the dashboard forbids. Gate it on
`ScriptKind::Module` and share the same `RESERVED_MODULE_NAMES` const (now
`pub(crate)`); endpoints remain unrestricted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `enabled` lifecycle's "a disabled script is not executable via ANY
path" guarantee leaked on the queue arm, and the outbox trigger arm lacked
the cross-app backstop its siblings carry.
- Queue arm: `dispatch_one_queue` bypasses the unified `dispatch_one`
fire-time gate, so its only `enabled` guard was the per-tick
`list_active_queue_consumers` snapshot — a TOCTOU window for a message
claimed in a tick where the script was still enabled. Re-check the
freshly-read `script.enabled` before executing and release the claim
(`nack`) when disabled; the message stays queued and resumes on
re-enable. The nack is load-bearing: `reclaim_visibility_timeouts` only
reclaims claims for enabled triggers, so without it the message would
sit claimed indefinitely.
- Trigger arm: `resolve_trigger` built the ExecRequest with
`app_id = row.app_id` while sourcing the body from `trigger.script_id`
with no same-app check — the guard the queue/http/invoke arms already
have. Add it so a hand-edited/restored row can't run one app's script
under another's SdkCallCx.
Both regression-locked by new unit tests (full mock harness, verified to
fail if the gate is removed):
- queue_enabled_gate::disabled_queue_consumer_releases_claim_without_executing
- outbox_enabled_gate::disabled_http_outbox_row_is_dropped_without_executing
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Holistic Phase-1 review found the "a disabled script can't execute via
any path" guarantee (§4.3) held only for the sync user-route, the
execute-by-id bypass, and the trigger outbox arm — three paths still ran
disabled scripts:
- Queue arm: `list_active_queue_consumers` filtered the trigger's
`enabled` but not the bound script's. Add `JOIN scripts … AND
s.enabled = TRUE` so a disabled script's queue trigger stops consuming.
- Async-HTTP (202) + queued invoke() arms: `build_http_request` /
`build_invoke_request` hardcoded `active: true`. Set it to
`script.enabled`, and MOVE the dispatcher's fire-time `active` drop to
after the source-kind match so it covers all three arms uniformly
(previously trigger-only).
- `invoke()` (script-to-script): `resolve_id`/`resolve_name` checked
cross-app isolation but not `enabled`. A disabled target now resolves
to NotFound (indistinguishable from absent), matching the data plane.
Also (review LOW/§4.7):
- The route-on/script-off 404 returned `NotFound(script_id)`, leaking the
internal id and distinguishable from absent. Return the same flat "no
route matches" 404 as the unmatched case.
- Add the §4.7 "enabled endpoint with no route and no trigger" reachability
warning (was unimplemented; only disabled-target shipped).
Tested: manager-core lib 368 + orchestrator 75 + 22 project-tool journeys
(incl. a new enabled-route→disabled-script flat-404 e2e) green; clippy
-D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold the implementation outcome back into the project-tool spec (§11) as
the design asks. Records what landed (init/pull/config/plan/apply/prune,
env overlays, enabled lifecycle, trigger name+backfill, content-fingerprint
bound-plan, apply lock) and the deliberate deferrals: tree-structure
version → groups; vars/--explain → Phase 3; trigger upsert-by-name + the
dashboard enabled toggle → tracked follow-ups beyond the §11 bullets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Read the new `name` column through the data model: `Trigger`/`TriggerRow`
carry it, every trigger SELECT/RETURNING includes it, and the row→Trigger
assembly + the interactive create paths populate it (the create INSERTs
still omit it, so they take the migration's default). Behavior unchanged
— the apply diff still keys on the semantic tuple; B-2 switches it to
name-keying (enabling Update) and wires the manifest.
Tested: manager-core lib 367 + trigger journeys green; clippy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Schema groundwork for the trigger `name` identity (the manifest merge key
that will let `apply` Update a trigger in place rather than only
Create/Delete). A `gen_random_uuid()` default keeps the existing write
paths valid and unique without code changes; existing rows are backfilled
to the readable `{kind}-{n}` form; `UNIQUE(app_id, name)` is enforced.
No behavior change yet — the manager-core write paths and the apply diff
start using the name in the follow-up. Verified the migration applies and
the trigger journeys (which create triggers via the unnamed path → default
name) stay green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §4.1 base+overlay model for the single-app world: a base `picloud.toml`
holds the shared scripts/routes/triggers; a sparse `picloud.<env>.toml`
carries the per-environment slug (+ extra secret names). `pic plan/apply
--env <E>` merges the overlay onto the base before diffing/applying, so an
environment deploys to its own app ("an environment is an app", §2)
without duplicating the shared definitions.
Merge is sparse: overlay `app.slug`/`name` replace the base's; secret
names union. Scripts/routes/triggers always come from the base. Rich
per-key `vars` resolution stays Phase 3.
Tested: unit tests for the merge + overlay-path derivation; a journey
proving `apply --env staging` deploys to the staging app and leaves the
base app untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §11 Phase-1 `config --effective` surface. Single-app today, "config"
= the app's secrets, so it cross-references the manifest's declared
secret names against what's set on the server and renders each masked:
`<set>` (managed / on-server-not-in-manifest) or `<unset>` (declared but
not pushed). Values are never fetched or shown (§4.6). Shaped to grow an
`--explain` mode and inherited `vars` when groups/vars arrive (Phase 3).
Tested: a journey verifying declared-but-unset → `<unset>`, post-`secret
set` → `<set>`/managed, and the value never appears in output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the coverage asymmetry flagged in review: script-disable had a full
e2e journey but route-disable was only unit-tested (compile_routes) plus
the apply-refresh path. Add an HTTP-level journey — claim a Host for the
app (two-phase dispatch needs it), then serve `/hello` (200), flip the
route's `enabled = false` and re-apply (404, indistinguishable from
absent), and re-enable (200).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- §4.7 warning: `apply` now reports when an enabled route or trigger is
bound to a disabled script — deployed but unreachable (route 404s,
trigger won't fire). A warning, not an error: it's valid desired state,
the operator just shouldn't be surprised by the silent 404.
- E2E journey (`tests/enabled.rs`): apply an active script and confirm
`/api/v1/execute/{id}` 200s; flip `enabled = false` in the manifest and
re-apply → 404; re-enable → 200. Exercises the whole path: manifest →
diff → apply → runtime honoring.
Tested: manager-core lib 367 + cli bins 31 + 15 project-tool journeys
(incl. the new enabled e2e) green; clippy -D warnings clean; versioning
gate passes (migration 0045).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the §4.3 outbox gap: the match-time `enabled` check can't see a
trigger (or its target script) disabled *after* an outbox row was
enqueued, so a pending row would still fire. `resolve_trigger` now folds
`trigger.enabled && script.enabled` into the resolved row, and
`dispatch_one` drops the row at fire time when inactive instead of firing
a stale event.
The queue arm needs no change here: `list_active_queue_consumers` already
re-lists only enabled queue triggers each tick, so a disable is honored
promptly without a stale-row window.
Tested: manager-core lib 366 green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the `enabled` flag bite at the data plane (§4.3).
- Routes: `compile_routes` drops disabled rows from the match table, so a
request to a disabled route 404s indistinguishably from an absent one
(no info leak). Rebuilt on every route write + at startup, as today.
- Scripts: the orchestrator's two execute paths — `/execute/{id}` bypass
and the user-route handler — 404 when the resolved script is disabled.
The route-handler check also covers an enabled-route-bound-to-disabled-
script (§4.7): the binding is unreachable rather than 500/executing.
Still pending: the dispatcher fire-time re-check for already-enqueued
outbox rows (next commit).
Tested: route_admin disabled-route-dropped unit test + orchestrator suite
(75) green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase-1 `enabled` three-state lifecycle (§4.3), data-model half. Triggers
already carried `enabled`; this adds it to scripts and routes and threads
it through the declarative project tool. Runtime honoring (disabled route
404 / script non-invocable / dispatcher fire-time re-check) is the next
commit — this change only stores and reconciles the flag.
- Migration 0045: `enabled BOOLEAN NOT NULL DEFAULT TRUE` on scripts +
routes. Default true ⇒ no behavior change on migrate.
- `Script`/`Route` (shared) gain `enabled` (serde default true via the new
`picloud_shared::default_true`); repos' SELECT/INSERT/UPDATE SQL, row
structs, `NewScript`/`NewRoute`/`ScriptPatch` all carry it.
- Apply diff treats `enabled` as a declarative field (omitted ⇒ active):
`script_update_reason` + `diff_routes` detect a toggle as an Update, and
the create/update/insert paths persist it. The bound-plan `state_token`
re-includes script/route `enabled` (removed in the earlier review fix
precisely because the diff didn't key on it yet — now it does).
- CLI manifest model + `build_bundle` + `pull` round-trip `enabled`
(serialized only when false; omitted ⇒ active). `pic init` scaffold and
the interactive script/route create paths default active.
Tested: manager-core lib 365 (incl. enabled diff + token sensitivity) +
cli bins 31 (incl. manifest skip-serialize) green; clippy -D warnings
clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediation after an independent review of the two feature commits.
pic init:
- Fix the headline `pic init --dir <new-dir>` flow: slug derivation used
`canonicalize()`, which fails on a not-yet-created directory → empty
slug → bail. Derive from the path's own final component first, falling
back to canonicalize only for `.`. Add a test that exercised the gap
(the existing tests all passed an explicit slug).
- `ensure_gitignored` no longer overwrites an existing-but-unreadable
`.gitignore` (only a genuinely-absent file is treated as empty).
Bound-plan staleness:
- Token trigger coverage now mirrors the diff exactly via
`current_trigger_identity`: dead-letter triggers (which the diff
ignores and the manifest can't represent) and the currently-inert
`enabled` flag are no longer hashed, so an out-of-band dead-letter
change can't force a spurious `--force`. Add a test.
- Correct two overclaiming comments: the check shares the diff's
pool-read apply-vs-interactive-write window (it is not tx-scoped), and
the token deliberately mirrors the diff's inputs.
- `.picloud/` self-ignores via a `*` `.gitignore` written alongside the
plan token, so projects created with `pic pull`/`plan` (not just
`init`) keep it out of git.
- `clear_plan` is now app-scoped: it won't discard a token recorded for a
different app sharing the directory.
Tested: manager-core lib 364 + cli bins 31 + 14 project-tool journeys
green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`pic plan` now records a fingerprint of the live state it diffed against;
`pic apply` replays it and the server refuses (HTTP 409) if the app
changed underneath the reviewed plan — the §4.2 "apply exactly what you
reviewed" guarantee, in its content-addressed form (no migration, no
changes to interactive write paths).
Server (manager-core):
- `state_token(CurrentState)`: deterministic FNV-1a fingerprint over what
the diff keys on — script name+version (version bumps on any edit),
route identity+binding/attrs, trigger membership+enabled, secret names.
Order-independent; a collision can only yield a false "unchanged", never
a false refusal.
- `plan` returns it (flattened onto the plan JSON, so the wire stays
additive); `apply` takes an optional `expected_token` and, under the
apply lock before any write, returns `StateMoved` (409) on mismatch.
CLI:
- `.picloud/` link state (`linkstate`): `pic plan` writes the token scoped
to the app slug; `pic apply` replays it, then clears it on success (the
token is single-use — the next apply re-plans). `--force` skips the
check; apply with no recorded plan still works standalone (today's
behavior). `.picloud/` is already gitignored by `pic init`.
The tree-structure version (the other half of §4.2's counter) stays a
deliberate no-op until groups exist — it only guards reparent/structural
moves, which don't exist single-app.
Tested: state_token unit test (stable/order-independent/sensitive) + a
staleness journey (plan → out-of-band deploy → apply refuses → --force
applies); manager-core lib 363 + cli bins 31 + 13 journeys green; clippy
-D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Offline project scaffolding — the on-ramp to pull/plan/apply. Writes a
`picloud.toml` describing a minimal *working* app (a `hello` endpoint
bound to GET /hello) plus commented examples for the other blocks,
`scripts/hello.rhai`, and a `.gitignore` that pre-ignores the project
tool's `.picloud/` link-state directory (the home for the bound-plan
state token, landing next). Refuses to overwrite an existing manifest
without `--force`; derives the app slug from the directory name when not
given (validated against the canonical slug rule).
The active manifest is rendered through the same `to_toml` model the rest
of the tool uses, so a freshly-init'd project is guaranteed valid and
immediately `pic plan`-able. Offline — no server contact, so the journey
tests need no DATABASE_URL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remediation of the single-app reconcile foundation after two independent
review passes. No new feature surface — closes correctness, parity, and
safety gaps in pull/plan/apply/prune.
apply engine (manager-core):
- validate_bundle reached parity with the interactive trigger API: reject
an empty kv/docs/files collection_glob and a malformed pubsub
topic_pattern (previously written and silently never matched), and lift
the queue visibility floor to MIN_QUEUE_VISIBILITY_TIMEOUT_SECS (30) —
apply had accepted a [5,29] value the dashboard refuses. Per-kind checks
extracted into a pure, unit-tested validate_trigger_shape.
- An omitted script `description` now means "leave as-is" (matching the
other optional fields and the function's own documented contract)
instead of clearing the stored value.
- Doc fixes: drop stale "next milestone" notes; record the one deliberate
plan/apply divergence (set-but-empty secret); document delete_route_tx
as intentionally idempotent for reconcile.
CLI:
- Gate destructive `apply --prune` behind confirmation: interactive y/N,
or `--yes` for CI; a non-interactive prune without `--yes` refuses
rather than silently deleting. Journey tests pass `--yes`; a new test
proves the gate refuses and deletes nothing.
- Harden pull's filename safety: reject all control characters and the
Unicode bidi-override / zero-width chars used for terminal/filename
spoofing, and cap length at 200 bytes (NAME_MAX headroom).
Tested: manager-core lib (incl. queue-floor + sparse-description parity)
+ CLI bins + 9 project-tool journeys green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a server-side, atomic, declarative reconcile loop for a single app —
the foundation of the project-tool design. Developers describe an app's
scripts, routes, triggers, and secret-names in `picloud.toml`, then
`pic pull / plan / apply [--prune]` to converge live state to the manifest.
Server (manager-core):
- apply_service: a pure diff engine (compute_diff) shared by plan and
apply, plus an ApplyService that composes the existing per-repo writes
into ONE Postgres transaction. Identity keys mirror the DB UNIQUE
constraints (script=lower(name); route=(method,host_kind,host,
path_kind,path); trigger=per-kind semantic tuple; secret=name).
Apply takes a per-app advisory lock, recomputes the diff in-tx, applies
scripts -> routes -> triggers, prunes dependents-first, commits, then
refreshes the route table once post-commit.
- apply_api: POST /apps/{id}/plan (AppRead) and /apps/{id}/apply.
Apply requires the per-kind write caps the bundle exercises (all three
when --prune), plus AppSecretsRead when it binds an email trigger.
- tx-accepting repo siblings (insert/update/delete *_tx) so the existing
create/update/delete delegate to one SQL definition each.
- email triggers reference an inbound secret by NAME; the value is
resolved, decrypted (AAD-bound), and re-sealed server-side at apply —
it never travels in the manifest.
CLI (picloud-cli):
- manifest.rs (picloud.toml model), client plan/apply, and the pull/plan/
apply commands. pull rejects filesystem-unsafe script names up front.
Safety properties enforced and tested:
- idempotent: a freshly-pulled manifest re-applies as all-NoOp.
- atomic: a mid-bundle failure rolls back with nothing written.
- routes delete-before-insert so a freed binding is reusable in one apply.
- queue one-consumer invariant held inside the shared tx.
- email triggers are never pruned, and a script that still owns an
email/dead-letter trigger can't be pruned (the FK cascade would destroy
the sealed secret) — refused with a pointer to `pic triggers rm`.
- plan and apply agree on unset email-secret references.
No migration: the existing schema's UNIQUE constraints serve as identity
keys. Groups, env-scoping, and the `enabled` toggle are later milestones.
Tested: manager-core lib (360) + CLI bins (27) + 8 project-tool journeys
(pull/plan/apply/prune/email+queue), all green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The architecture spec for the full vision — a declarative project tool
(picloud.toml + pull/plan/apply/prune) and a server-side nested-groups
hierarchy with live-resolve inheritance. Reviewed across several passes
for consistency, gaps, and contradictions; resolutions are folded in
beside the topics they settle.
This commit lands the spec only. The first slice of implementation (the
single-app reconcile foundation) follows; groups, env-scoping, and the
`enabled` toggle are later milestones called out in the doc.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the gaps and the one security finding from the second end-to-end
CLI test (E2E_STASH_REPORT.md), plus the H1 boot-regression found while
re-reviewing those fixes.
Security
- S6: reserved-path validation (`check_reserved`) now case-folds before
comparing, so `/API/v2/x`, `/HEALTHZ`, `/Admin/x` are rejected like
their lowercase forms. Request-time matching stays case-sensitive.
- S10: "public route != public data" callout in sdk-shape.md (script_gate
skips authz when the principal is anonymous).
Observability / features
- G1: trigger executions now write `execution_logs`. Migration 0043 adds
a `source` column (CHECK mirrors ExecutionSource/OutboxSourceKind,
DEFAULT 'http' backfills history); a shared `build_execution_log` helper
in executor-core; dispatcher logging for outbox triggers + queue
consumers (skips sync-HTTP rows the orchestrator already logs). `pic
logs` gains a source column + `--source` filter.
- G5: dev-only in-memory email capture under PICLOUD_DEV_MODE with no SMTP
(email::send succeeds locally), readable at GET /api/v1/admin/dev/emails
(Owner/Admin only; route mounted only in capture mode).
- G6: generalized the Rhai in-place-mutation footgun note (trim/replace/
make_upper/make_lower/crop/truncate/pad return ()).
- G2/G3/G4 (CLI): `pic members`, `pic files`, `pic queues`, read-only
`pic kv` (+ new kv_api.rs); `pic deploy --timeout/--memory/--kind/
--sandbox`; first-class `pic triggers create-{docs,files,pubsub,queue,
email}` wrappers. All new client path segments percent-encoded via seg().
H1 regression fix (found in re-review)
- The S6 change also runs in `compile_routes`, which compiles every stored
route at boot and on each route CRUD. A single stored route the new
validation rejects (creatable while the S6 gap existed) made the whole
compile Err and aborted startup. `compile_routes` is now lenient: it
skips an un-compilable row with a warning instead of bricking boot
(route creation still validates separately). Migration 0044 sweeps
pre-existing reserved-path routes on upgrade (WHERE mirrors
check_reserved exactly). Added regression tests for both.
Verified: cargo fmt, clippy --all-targets --all-features -D warnings, the
schema_snapshot test, and the new S6/lenient-compile unit tests all pass;
boot-resilience and G1/G5 confirmed live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 15:01:04 +02:00
402 changed files with 71915 additions and 2280 deletions
End-to-end stats after the run: `{total_pastes: 2, total_words: 13}` — proving queue → worker →
invoke → kv all fired on real Postgres rows + on-disk file bytes.
---
## Security matrix (10 probes, all reproduced live)
| # | Probe | Verdict | Evidence |
|---|---|---|---|
| **S1** | Cross-app isolation | ✅ HOLDS | A script in app `evil` doing `kv…list()` / `secrets::get("admin-token")` / `files…list()` returned `{paste_keys:[], attachment_count:0, stolen_secret:null, my_secret_names:[]}`. `app_id` derives from `cx.app_id`; no SDK call takes a script-passed app_id. |
| **S2** | Secret confidentiality | ✅ HOLDS | `pic secrets ls` shows names only; the plaintext `sup3r-s3cret-admin` does **not** appear anywhere in the server log even though `/stats` reads it each request. |
| **S3** | API-key scope + app-binding | ✅ HOLDS | `script:read` key → **403** on `POST /apps` and `GET /admins`, **200** on in-scope `GET /scripts`. `app:admin` key bound to `stash` → **403** on app `evil`. `instance:admin` + `--app` → **422**. Unknown scope `bogus:scope` → rejected by CLI. |
| **S4** | Sandbox op-budget | ✅ HOLDS | `loop {}` → **HTTP 507 "execution exceeded operation budget"** in **0.30 s**; `/healthz` still `ok` (no hang). |
| **S5** | Value/size caps | ✅ HOLDS | Oversized kv write rejected (HTTP 502). Notably the detail (`"Length of string too large"`) was **scrubbed from the response** and logged with `correlation_id` — good error hygiene. Caps are checked before authz, so anonymous callers can't DoS. |
| **S6** | Reserved-path prefixes | ⚠️ **GAP (Low)** | `/api/x`, `/admin/x`, `/healthz`, `/version` → correctly **422**. But case variants `/API/v2/x`, `/Admin/x`, `/HEALTHZ` were **accepted**. See finding below. |
| **S7** | Files path traversal | ✅ HOLDS | `files::collection("../../../etc")` → `"invalid collection name: must not contain '/', '\\', '..', or NUL"`; a non-UUID file id → not found. No FS escape. |
| **S8** | docs filter injection | ✅ HOLDS | `docs::find(#{ "x'); DROP TABLE docs;--": 1 })` → ran safely, `count=0`; the `docs` table still exists (`to_regclass` = true). Values/paths are bound as `$N` params. |
| **S9** | SSRF via `http::` | ✅ HOLDS | `http::get("http://127.0.0.1:8099/healthz")` → `"blocked by SSRF policy: loopback"`; `http://169.254.169.254/…` (cloud metadata) → `"blocked by SSRF policy: link-local"`. |
| **S10** | Anonymous data-plane access | ℹ️ By design (caveat) | Every Stash route is public (no auth) yet freely uses kv/secrets/files/queue. `script_gate` returns `Ok` when `principal.is_none()` — the *script* is the gate. Correct per design, but a sharp threat-model edge (see note). |
### S6 — reserved-path validation is case-sensitive (Low)
**What:** route-creation rejects the exact reserved prefixes but accepts case variants:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.