diff --git a/CLAUDE.md b/CLAUDE.md index 63edcac..b721f34 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Authoritative design: [serverless_cloud_blueprint.md](serverless_cloud_blueprint **Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. The single-owner ownership **claim** shipped as §7 M1, the attach-point ceiling + blast-radius as §7 M2/M3, and per-env approval gating as §3 M3 — all server-authoritative (see the tail); declarative group **create/reparent** + structural-divergence detection shipped as §6 (`reconcile_group_structure_tx`/`reparent_group_tx`/`StructureMode`), so groups no longer need to pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `/files/groups//...`, a `groups/` infix disjoint from the per-app `files//` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Topic shared collections shipped as D2 (storeless), queue as D3 — see below), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email need materialization — see M5 below; per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey), and **§11.6 M2 shared-collection triggers** (a write to a group SHARED collection now fires a trigger — closes the "group trigger has no app to watch" gap. A group-owned trigger marked `shared = true` (`shared BOOLEAN` on `triggers`, `0061_shared_triggers.sql`) watches the group's shared collection; the per-app `list_matching_kv/docs/files` add `AND t.shared = FALSE`, new `list_matching_shared_{kv,docs,files}(owning_group,…)` select `shared = TRUE` triggers on the owning group — the `shared` flag is the namespace boundary. `ServiceEventEmitter` gained `emit_shared(cx, owning_group, event)`; `GroupKv/Docs/FilesServiceImpl` (via a `with_events` builder) emit on write, and the outbox emitter stamps the WRITER app_id so the handler runs under the writer. `shared` is authored on a group `[[triggers.kv|docs|files]]`, is part of the diff identity, and `validate_bundle_for` rejects it on an app owner / a non-collection kind / an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by `tests/shared_triggers.rs` + the `shared_triggers` journey. Shared pubsub triggers shipped as D2 — see below), and **§11.6 M3 per-group quotas** (global env-var ceilings enforced in the group write path: `PICLOUD_GROUP_KV_MAX_ROWS`/`_DOCS_MAX_ROWS` (per-group row count, new-key only), `PICLOUD_GROUP_{KV,DOCS}_MAX_TOTAL_BYTES` (per-group total stored bytes, projected-total check — Track A M4) + `_FILES_MAX_TOTAL_BYTES` (per-group total bytes); `group_quota` helpers + `count_rows`/`total_bytes` repo methods + `QuotaExceeded`/`TotalBytesQuotaExceeded` errors), and **§11.6 M4 read-only operator admin API** (`group_blobs_api` mirrors the per-app kv/files admin surface for a group's shared collections: `GET /groups/{id}/{kv,docs,files}[/{collection}/{key|id}]`, authz `GroupKvRead`/`GroupDocsRead`/`GroupFilesRead`; the host hoists the group repos to share one instance; `pic kv ls/get --group`; reads-only, writes stay script-only), and **§4.5 M5 stateful group trigger templates via materialization** (cron + queue + email — the kinds that can't resolve live because each needs a per-app row: cron `last_fired_at`, the queue one-consumer advisory lock, the email sealed inbound secret. A group `[[triggers.cron|queue|email]]` template is **materialized** into an app-owned copy per descendant (`materialized_from` col, `0062_materialized_triggers.sql` + the `0063_materialized_unique.sql` partial-unique index that makes rematerialization idempotent under concurrency) by `materialize::rematerialize_stateful_templates` — all-apps `app_chain` CTE ⋈ group-owned stateful templates, a precise create/delete diff preserving cron state — run full-live at the route-rebuild chokepoints (apply single+tree, app create/delete in `apps_api`, group reparent in `groups_api`; `AppsState`/`GroupsState` gained a `pool`). The scheduler + `list_active_queue_consumers` + `email_inbound_target` gained `AND t.app_id IS NOT NULL` so a group TEMPLATE is never dispatched directly (nor invocable via its own webhook URL) — only its per-app copies are; a queue copy is skipped-with-warning when the app already fills that queue's slot. **M5.5 email uses the shared-group-secret model:** the template resolves its `inbound_secret_ref` against the **group's own** secret store once at apply and seals it (`resolve_and_seal`/`insert_email_trigger_tx` generalized to a `SecretOwner`/`ScriptOwner`); email secrets are now **v1 AAD-bound to the SEALING OWNER** (Track A M3, migration `0069_email_secret_version` + `seal_email`/`open_email`; the group AAD is stable across rows, so materialization still copies the sealed bytes **verbatim** — the inbound path recovers the sealing group via `materialized_from` and opens under its AAD; a legacy v0 read path remains for pre-M3 rows). All descendant webhooks share the one group HMAC secret; an unset group secret fails apply hard. Loading a group's own set-secret names into `CurrentState` (was hardcoded empty) lets the plan-time email-secret check resolve against the group. Pinned by `tests/stateful_templates.rs` + the `stateful_templates` journey. **Per-app (non-shared) email secrets already work** — an app-owned email trigger resolves its `inbound_secret_ref` against the **app's own** secret store and seals it AAD-bound to the app (`SecretOwner::App`, AAD `email:{app_id}` in `secrets_service::email_secret_aad`); M5.5 generalized *that* original app path to groups, not the reverse, and the inbound path recovers `SecretOwner::App` when `materialized_from` is NULL. The only per-app nuance still open is deliberate: the *interactive* `POST .../triggers/email` API takes the secret **inline**, whereas the apply path resolves a **named** app secret — the sealing/AAD machinery is fully app-aware either way), and **D1 the `materialized` column** (`pic triggers ls --app` now shows a read-only `materialized` column — a copy of an M5 group stateful template reads `true`, distinct from a hand-authored trigger; a derived `materialized` bool = `materialized_from IS NOT NULL` threaded row→domain→API→CLI mirroring `sealed`/`shared`), and **§11.6 D2 shared TOPICS + shared pub/sub triggers** (a group declares a storeless `topic` shared collection (`group_collections.kind` widened to `topic`+`queue` in `0064`); a `[[triggers.pubsub]] shared = true` handler watches it. `BundleTrigger::Pubsub` gained `shared` (part of the identity); `validate_bundle_for` requires the topic pattern's ROOT segment (`events.*` → `events`) be a declared `kind='topic'` collection, group-only. Scripts publish via the explicit `pubsub::shared_topic("events").publish("created", msg)` handle → `GroupPubsubServiceImpl` resolves the owning group (kind `topic`) from `cx.app_id`'s chain, requires editor+ (`GroupPubsubPublish`, fails closed on anon), and fans out via `PubsubRepo::fan_out_shared_publish` to `shared = true` pubsub triggers on that group, each outbox row stamped the WRITER app_id (M2 model). The per-app `fan_out_publish` gained `AND t.shared = FALSE` — a shared trigger never fires on a per-app publish and vice-versa (the `shared` flag is the namespace boundary); the owning-group chain walk is the isolation boundary. Pinned by `tests/shared_topics.rs` + the `shared_topics` journey. **External SSE subscription shipped** (Track A M6): `GET /realtime/shared/topics/{topic}` streams a shared topic to external clients; `RealtimeAuthority::authorize_subscribe_shared` resolves the owning group from the subscriber app's chain (reads-open — the resolution IS the authorization, a foreign subtree 404s), and `GroupPubsubServiceImpl::with_realtime` bridges publish→broadcast after the durable fan-out. **Deferred:** multi-node broadcast propagation (cluster mode)), and **§11.6 D3 shared durable QUEUES** (a group declares a `queue` shared collection; any subtree app enqueues into ONE group-keyed store (`group_queue_messages`, `0065`, mirrors `queue_messages` keyed by `(group_id, collection)`; CASCADE on group delete) via `queue::shared_collection("name").enqueue(...)` → `GroupQueueServiceImpl` resolves the owning group (kind `queue`), requires editor+ (`GroupQueueEnqueue`, fails closed on anon). **Consumption is by COMPETING CONSUMERS:** a group `[[triggers.queue]] shared = true` consumer (shared threaded through `BundleTrigger::Queue` + identity) **materializes** a consumer copy per descendant app (`materialize` skips the M5 one-consumer-slot check for shared — each descendant intentionally gets a consumer), and all copies claim the SHARED store with `FOR UPDATE SKIP LOCKED` — each message delivered at-most-once across the subtree, scaling horizontally, each handler under its own `cx.app_id`. The dispatcher's queue arm gained a `shared_group` on `ActiveQueueConsumer` (recovered via a LEFT JOIN to the materialized copy's source template) and routes claim/ack/nack/terminal to the group store when set (`q_claim/q_ack/q_nack/q_terminal` helpers; a group claim normalizes to a `ClaimedMessage` under the consuming app); the reclaim task drains both stores. `validate_bundle_for` requires a shared queue on a group to name a declared `kind='queue'` collection; a shared queue on an app is rejected by the app-owner shared guard. Pinned by `tests/group_queue.rs` (competing-consumer at-most-once) + `stateful_templates.rs` + the `shared_queues` journey. **Group dead-letter store shipped** (Track A M2): an exhausted shared-queue message is preserved in `group_dead_letters` (`0068_group_dead_letters`) via `GroupQueueRepo::dead_letter` (atomic INSERT+DELETE) and is operator-visible at read-only `GET /api/v1/admin/groups/{id}/dead-letters` (`GroupKvRead`). **Shared dead-letter fan-out shipped (B2):** a group declares a declaratively-authored `[[triggers.dead_letter]] shared = true` handler; when a shared-queue message exhausts, the dispatcher's `q_terminal` group branch (after persisting to `group_dead_letters`) fans out to `list_matching_shared_dead_letter(owning_group, "queue", …)`, each outbox row stamped the WRITER `app_id` (the consuming app — M2 model), so the handler runs under the consumer. The per-app `list_matching_dead_letter` gained `AND t.shared = FALSE` (the `shared` flag is the namespace boundary); `insert_trigger_tx` now accepts `dead_letter` (`BundleTrigger::DeadLetter`), and `validate_bundle_for` requires it group-owned + shared. Pinned by `tests/shared_dead_letter.rs`), and **§7 multi-repo ownership M1 — the single-owner claim** (the `owner_project` seam (0047) is now live behind a first-class `projects` table (`0066_projects.sql`, UUID pk + unique slug; `owner_project` FKs it `ON DELETE SET NULL` — un-claim, never cascade-destroy a tree). A `[project]` block (slug + optional name) in the repo's ROOT manifest declares identity (independent of the `[app]`/`[group]` XOR); the first apply with a new slug registers the project and **claims** each group node it touches. The claim runs inside the apply tx under the per-node advisory lock **before the diff**, so a conflict short-circuits with a **409** before any write. Pure policy `decide_group_claim`/`decide_app_owner` in `apply_service` (unit-tested, DB-free): unclaimed→claim, owner→no-op, foreign→conflict unless `--takeover` (which additionally requires `GroupAdmin` — ownership ⟂ RBAC, mapped to 403 vs the 409 conflict), no-project-into-a-claimed-subtree→conflict. **Apps carry no owner** — an app inherits ownership from its **nearest claimed ancestor group** (the ancestor walk, via `groups.ancestors` now carrying `owner_project` through its recursive CTE, is the isolation boundary); an unclaimed subtree stays open, so nothing changes until a repo first declares `[project]` (backward-compatible). `ProjectRepository` (read side) + tx free-fns `upsert_project_tx`/`read_group_owner_tx`/`write_group_owner_tx`; the claim deliberately does **not** bump `structure_version` (not a diff change → won't churn a pending bound plan). Wire: `project`/`takeover` on the apply request (both `#[serde(default)]` → the pre-M1 CLI stays compatible); the CLI surfaces the server's 409 message verbatim (covers `StateMoved` + `OwnershipConflict`). Visibility: `pic groups ls` `owner` column (server `list_with_owner` LEFT JOIN; the shared `Group` deserialize ignores the extra field so `pic groups tree`/dashboard are unaffected) + `pic apply --takeover`. Pinned by `apply_service` unit tests, `tests/projects_repo.rs`, and the `apply_ownership` journey. **M2 shipped — the attach-point ceiling:** a `[project] parent_group = ""` binds the repo under a pre-existing group; `check_within_attach` refuses (422 `OutsideAttachPoint`) any node not strictly within that subtree (a group node must be a *proper* descendant — you can't apply the attach point itself; an app node's group must be at-or-below it), resolved via `groups.ancestors` and enforced read-only before the claim in `apply_owner`/`apply_tree`; absent = instance root = no ceiling. **M3 shipped — plan preview + `pic projects ls`:** `pic plan` now carries the `[project]` and returns an `ownership` preview per node (`claim`/`owned`/`conflict`-owner-named/`unclaimed`, pure `preview_ownership`) plus, for a group node, the cross-repo **blast radius** (descendant apps owned by OTHER projects the change reaches, via `group_blast_radius` — a subtree CTE with per-group memoized nearest-claimed resolution); the attach ceiling is previewed at plan too. Read-only `pic projects ls` (`GET /api/v1/admin/projects`, `list_with_counts`) lists projects + owned-group counts. Pinned by the `apply_ownership` plan-preview case. **With M3 the §7 multi-repo ownership track (M1 claim · M2 attach ceiling · M3 preview + `pic projects ls`) is COMPLETE.** Also shipped: **§6 group-tree Tier 1** (declarative group create via dir-nesting · structural-divergence detection · declarative reparent — `reconcile_group_structure_tx`/`StructureMode`, 422 `StructuralDivergence`) and **§3 M3 the per-env approval gate**, now **server-authoritative** (migration `0067_project_environments`; the gate resolves the governing project from the target node's nearest-claimed ancestor — `governing_env_policy`/`_tree` — so omitting/spoofing `[project]` can't bypass it; an approved gated apply requires AppAdmin/GroupAdmin step-up + audit). -**Track A (v1.2 deferred-gap closeout, migrations 0067–0069) shipped to local main:** M1 hermetic approval gate · M2 shared-queue dead-letter store · M3 email-secret AAD v0→v1 · M4 per-group KV/docs byte quotas · M5 `set_if` compare-and-swap for KV (per-app + shared + Rhai SDK) · M6 shared-topic external SSE. **Audit 2026-07-11 remediation** (migration 0070, admin-session absolute cap) also shipped. **With that, v1.2 _Hierarchies_ is complete.** The **Workflows** track then shipped too (M1–M6: DAG execution + conditional `when`, nested sub-workflows, durable orchestrator, `workflow::start` SDK + admin run API + `pic workflows`, dashboard DAG + run-history; migrations `0071`/`0072`). A **§9.4 service-interceptor** thin slice then shipped (migration `0073_interceptors.sql`): a `[[interceptors]]` block (app or group) binds a script to run before `kv::set`/`delete` and allow/deny it, resolved nearest-owner-wins on the app chain and run via the `invoke()` re-entry path; the rest of §9.4 (data transform, non-kv services, after-hooks, chaining) is deferred. Next: the rest of §9.4 and multi-node cluster mode (the deferred multi-node route/broadcast propagation lives there). +**Track A (v1.2 deferred-gap closeout, migrations 0067–0069) shipped to local main:** M1 hermetic approval gate · M2 shared-queue dead-letter store · M3 email-secret AAD v0→v1 · M4 per-group KV/docs byte quotas · M5 `set_if` compare-and-swap for KV (per-app + shared + Rhai SDK) · M6 shared-topic external SSE. **Audit 2026-07-11 remediation** (migration 0070, admin-session absolute cap) also shipped. **With that, v1.2 _Hierarchies_ is complete.** The **Workflows** track then shipped too (M1–M6: DAG execution + conditional `when`, nested sub-workflows, durable orchestrator, `workflow::start` SDK + admin run API + `pic workflows`, dashboard DAG + run-history; migrations `0071`/`0072`). **§9.4 service interceptors are now COMPLETE** (migrations `0073`–`0075`). A `[[interceptors]]` block (app or group) binds a script to a `(service, op, phase)` marker resolved on the app chain and run via the `invoke()` re-entry path. Shipped across M1–M12: **all six data-plane services** — `kv` (set/delete/`set_if`), `docs`/`files` (create/update/delete), `queue` (enqueue), `pubsub` (publish), `http` (request), each per-app AND group-shared; **before + after phases** (`phase = "before"|"after"`, migration `0074`) — before = allow/deny + a `data` **transform** (rewrites the written value, size-capped), after = observe/audit with the write `result` (cannot roll back); **ordered chaining** ancestor→app (a group guard runs first; before ancestor→app, after app→ancestor) with an **identity cycle guard** + the existing re-entrancy bypass; a **per-interceptor timeout** (`timeout_ms`, migration `0075`, env default `PICLOUD_INTERCEPTOR_TIMEOUT_MS`, tightened to at most the caller's remaining deadline); a **per-execution resolve cache** (N un-hooked writes → 1 chain query); the **seal** (a group's interceptor script resolves at the group — a descendant can't shadow it) and **fail-closed** verdict (only `#{ allowed: true }` allows). `validate_bundle_for` enforces a per-service allowed-ops map (every validated `(service, op)` has a matching runtime hook — no fail-open). Read-only `pic interceptors ls --app` (resolved chain view) / `--group` (own markers). Pinned by `tests/interceptors.rs` (12 journeys) + `executor-core` unit tests. **Deferred:** interceptor coverage of non-listed services/ops. Next: multi-node cluster mode (the deferred multi-node route/broadcast propagation lives there). **Write-path invariant — the transactional outbox (`manager-core::atomic_write`).** A data-plane mutation (KV / docs / files, per-app and group-shared) and the trigger fan-out it produces commit in **ONE transaction on ONE connection**, via a `*Writer` injected into each service (`with_atomic_writes(pool)` in the host). The services previously wrote the row, waited for it to commit, and *then* asked the emitter to resolve triggers and insert outbox rows — a second transaction on a second connection, so an outbox failure left a committed row whose trigger never fired, invisible to the caller and unrecoverable by any retry. Now an emit failure **rolls the write back** and surfaces as an error. Three rules make it work: - **Everything inside a transaction runs on that transaction's connection.** A writer that held a `tx` and then reached for a *second* pooled connection could deadlock — the pool is sized to the execution-concurrency cap, so N executions each wanting 2 connections starve. Hence `outbox_event_emitter::emit_on(&mut *tx, …)` and the `*_on(exec, …)` repo free fns (the trait methods delegate to them, so each query has one home). **`shared` has no sqlx and must not gain it** — the `ServiceEventEmitter` trait stays connection-free; the transactional path is entirely manager-core-internal. @@ -156,6 +156,6 @@ Environment variables consumed by the `picloud` binary: ## Out of MVP -This section captured the original MVP cut. Most of it has since **shipped in v1.1.x**: queue triggers, cron triggers, inbound email (`email:receive`, HMAC-webhook model), KV / docs / email / users / HTTP SDKs, function-to-function `invoke()`, and secrets are all live (blueprint §12 Phase 4 table). The **Workflows** track (DAG + nested workflows) has since **shipped** (migrations `0071`/`0072`). A **§9.4 service-interceptor** KV allow/deny slice has since shipped (migration `0073`). A read-only **metrics/observability dashboard** has since shipped (A2): `GET /api/v1/admin/apps/{id}/metrics?window=` aggregates the existing `execution_logs` table (counts, error rate, latency avg/p50/p95, an hourly series) via `ExecutionLogRepository::summarize_for_app` + `metrics_api`, surfaced as the dashboard's per-app **Metrics** tab — no hot-path instrumentation. **Still deferred:** the rest of §9.4 (data transform, non-kv services, after-hooks, chaining), a raw SMTP-listener ingress, and **multi-node cluster mode**. Don't pre-build for them — but don't make decisions that close the door on them either. +This section captured the original MVP cut. Most of it has since **shipped in v1.1.x**: queue triggers, cron triggers, inbound email (`email:receive`, HMAC-webhook model), KV / docs / email / users / HTTP SDKs, function-to-function `invoke()`, and secrets are all live (blueprint §12 Phase 4 table). The **Workflows** track (DAG + nested workflows) has since **shipped** (migrations `0071`/`0072`). The **§9.4 service-interceptor** track has since shipped in full (migrations `0073`–`0075`; before/after phases, data-transform, all six services, per-interceptor timeout, resolve cache, `pic interceptors ls`). A read-only **metrics/observability dashboard** has since shipped (A2): `GET /api/v1/admin/apps/{id}/metrics?window=` aggregates the existing `execution_logs` table (counts, error rate, latency avg/p50/p95, an hourly series) via `ExecutionLogRepository::summarize_for_app` + `metrics_api`, surfaced as the dashboard's per-app **Metrics** tab — no hot-path instrumentation. **Still deferred:** a raw SMTP-listener ingress, and **multi-node cluster mode**. Don't pre-build for them — but don't make decisions that close the door on them either. **Pulled forward to Phase 3 (pre-v1.1):** admin auth, multi-app scoping. The general cross-app **export/import** sharing model stays at v1.3+; note that v1.2 §11.6 shipped a narrower form — **group-owned shared collections** (KV/docs/files/topics/queues) let apps in one subtree share data through the owning group, with the ancestor-chain walk as the isolation boundary. See blueprint §11.5 + design-doc §11.6. diff --git a/crates/executor-core/src/sdk/kv.rs b/crates/executor-core/src/sdk/kv.rs index 7601cd2..a59b5a1 100644 --- a/crates/executor-core/src/sdk/kv.rs +++ b/crates/executor-core/src/sdk/kv.rs @@ -215,12 +215,38 @@ fn register_set_if(engine: &mut RhaiEngine) { expected: Dynamic, new: Dynamic| -> Result> { - let h = handle.clone(); let exp = expected_from_dynamic(&expected)?; - let new = dynamic_to_json_capped(&new, MAX_JSON_MATERIALIZE_BYTES)?; - block_on("kv", async move { - h.service.set_if(&h.cx, &h.collection, key, exp, new).await - }) + let json = dynamic_to_json_capped(&new, MAX_JSON_MATERIALIZE_BYTES)?; + // §9.4 M12: CAS is a WRITE, so it runs the `(kv, set)` guard too — + // otherwise `set_if` would be a silent bypass of a `set` policy. + let write_val = super::interceptor::run_before( + &handle.ictx, + &handle.cx, + "kv", + "set", + &handle.collection, + key, + Some(&json), + )? + .unwrap_or(json); + let written = write_val.clone(); + let h = handle.clone(); + let swapped = block_on("kv", async move { + h.service + .set_if(&h.cx, &h.collection, key, exp, write_val) + .await + })?; + super::interceptor::run_after( + &handle.ictx, + &handle.cx, + "kv", + "set", + &handle.collection, + key, + Some(&written), + serde_json::Value::Bool(swapped), + )?; + Ok(swapped) }, ); } @@ -395,12 +421,25 @@ fn register_group_set_if(engine: &mut RhaiEngine) { expected: Dynamic, new: Dynamic| -> Result> { - let h = handle.clone(); let exp = expected_from_dynamic(&expected)?; - let new = dynamic_to_json_capped(&new, MAX_JSON_MATERIALIZE_BYTES)?; - block_on("kv", async move { - h.service.set_if(&h.cx, &h.collection, key, exp, new).await - }) + let json = dynamic_to_json_capped(&new, MAX_JSON_MATERIALIZE_BYTES)?; + // §9.4 M12: CAS on a shared collection runs the `(kv, set)` guard too. + let write_val = group_run_before(handle, "set", key, Some(&json))?.unwrap_or(json); + let written = write_val.clone(); + let h = handle.clone(); + let swapped = block_on("kv", async move { + h.service + .set_if(&h.cx, &h.collection, key, exp, write_val) + .await + })?; + group_run_after( + handle, + "set", + key, + Some(&written), + serde_json::Value::Bool(swapped), + )?; + Ok(swapped) }, ); } diff --git a/crates/manager-core/src/apply_api.rs b/crates/manager-core/src/apply_api.rs index dc677fc..d343453 100644 --- a/crates/manager-core/src/apply_api.rs +++ b/crates/manager-core/src/apply_api.rs @@ -18,8 +18,9 @@ use serde_json::json; use crate::app_repo::AppRepository; use crate::apply_service::{ ApplyError, ApplyOwner, ApplyReport, ApplyService, Bundle, BundleTrigger, CollectionInfo, - ExtensionPointInfo, NodeKind, OwnershipClaim, PlanResult, ProjectDecl, RouteTemplateInfo, - StructureMode, SuppressionInfo, TreeBundle, TreePlanResult, TriggerTemplateInfo, + ExtensionPointInfo, InterceptorInfo, NodeKind, OwnershipClaim, PlanResult, ProjectDecl, + RouteTemplateInfo, StructureMode, SuppressionInfo, TreeBundle, TreePlanResult, + TriggerTemplateInfo, }; use crate::authz::{require, AuthzDenied, Capability}; use crate::group_repo::GroupRepository; @@ -46,9 +47,45 @@ pub fn apply_router(service: ApplyService) -> Router { .route("/groups/{id}/routes", get(group_routes_handler)) .route("/apps/{id}/suppressions", get(app_suppressions_handler)) .route("/groups/{id}/suppressions", get(group_suppressions_handler)) + .route("/apps/{id}/interceptors", get(app_interceptors_handler)) + .route("/groups/{id}/interceptors", get(group_interceptors_handler)) .with_state(service) } +/// Read-only §9.4 interceptor report for an app: the RESOLVED chain view (every +/// marker guarding its writes, nearest-owner-wins). Viewer-tier `AppRead`. Backs +/// `pic interceptors ls --app`. +async fn app_interceptors_handler( + State(svc): State, + Extension(principal): Extension, + Path(id_or_slug): Path, +) -> Result>, ApplyError> { + let app_id = resolve_app_id(svc.apps.as_ref(), &id_or_slug).await?; + require(svc.authz.as_ref(), &principal, Capability::AppRead(app_id)) + .await + .map_err(map_authz)?; + let report = svc.interceptor_report(ApplyOwner::App(app_id)).await?; + Ok(Json(report)) +} + +/// Read-only §9.4 interceptor report for a group: its OWN declared markers. +async fn group_interceptors_handler( + State(svc): State, + Extension(principal): Extension, + Path(id_or_slug): Path, +) -> Result>, ApplyError> { + let group_id = resolve_group_id(svc.groups.as_ref(), &id_or_slug).await?; + require( + svc.authz.as_ref(), + &principal, + Capability::GroupScriptsRead(group_id), + ) + .await + .map_err(map_authz)?; + let report = svc.interceptor_report(ApplyOwner::Group(group_id)).await?; + Ok(Json(report)) +} + /// Read-only §11 tail suppression report for an app: the inherited templates it /// declines (`target_kind`, `reference`). Viewer-tier `AppRead`. Backs /// `pic suppress ls --app`. diff --git a/crates/manager-core/src/apply_service.rs b/crates/manager-core/src/apply_service.rs index 13f783e..4d1e4f8 100644 --- a/crates/manager-core/src/apply_service.rs +++ b/crates/manager-core/src/apply_service.rs @@ -854,6 +854,19 @@ pub struct CollectionInfo { pub kind: String, } +/// One row of the read-only §9.4 interceptor report (`pic interceptors ls`). +/// For an app it is the RESOLVED chain view (markers visible on its ancestor +/// chain, nearest-owner-wins); for a group it is the group's OWN declarations. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct InterceptorInfo { + pub service: String, + pub op: String, + pub phase: String, + pub script: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub timeout_ms: Option, +} + /// One row of the read-only §11 tail suppression report (`pic suppress ls /// --app`). `target_kind` is `trigger` | `route`; `reference` is the handler /// script name (trigger) or path (route) the app declines. @@ -3671,6 +3684,32 @@ impl ApplyService { /// `inherited default` / `null` = unset). For a **group**: its own declared /// names (no single app to resolve a provider against). Backs the read-only /// `extension-points ls` and the `pull` round-trip. + /// §9.4 read-only interceptor report. For an app: the RESOLVED chain view + /// (every marker visible on its ancestor chain, nearest-owner-wins) — what + /// actually guards its writes. For a group: its OWN declared markers. + pub async fn interceptor_report( + &self, + owner: ApplyOwner, + ) -> Result, ApplyError> { + let markers = match owner { + ApplyOwner::App(a) => crate::interceptor_repo::list_on_app_chain(&self.pool, a).await, + ApplyOwner::Group(g) => { + crate::interceptor_repo::list_for_owner(&self.pool, ScriptOwner::Group(g)).await + } + } + .map_err(|e| ApplyError::Backend(e.to_string()))?; + Ok(markers + .into_iter() + .map(|m| InterceptorInfo { + service: m.service, + op: m.op, + phase: m.phase, + script: m.script, + timeout_ms: m.timeout_ms, + }) + .collect()) + } + pub async fn extension_point_report( &self, owner: ApplyOwner, diff --git a/crates/picloud-cli/src/client.rs b/crates/picloud-cli/src/client.rs index a07b3e3..45f1a9f 100644 --- a/crates/picloud-cli/src/client.rs +++ b/crates/picloud-cli/src/client.rs @@ -1608,6 +1608,18 @@ pub struct ExtensionPointInfoDto { pub provider: Option, } +/// One row of the §9.4 interceptor report. +#[derive(Debug, Deserialize)] +pub struct InterceptorInfoDto { + pub service: String, + pub op: String, + #[serde(default)] + pub phase: String, + pub script: String, + #[serde(default)] + pub timeout_ms: Option, +} + impl Client { /// `GET /api/v1/admin/{apps|groups}/{ident}/extension-points`. pub async fn extension_points_list( @@ -1626,6 +1638,24 @@ impl Client { decode(resp).await } + /// §9.4: `GET /api/v1/admin/{apps|groups}/{ident}/interceptors`. For an app, + /// the resolved chain view (what guards its writes); for a group, its own. + pub async fn interceptors_list( + &self, + kind: NodeKind, + ident: &str, + ) -> Result> { + let ident = seg(ident); + let resp = self + .request( + Method::GET, + &format!("/api/v1/admin/{}/{ident}/interceptors", kind.path()), + ) + .send() + .await?; + decode(resp).await + } + /// `GET /api/v1/admin/groups/{ident}/collections` (§11.6). Group-only — /// shared collections are declared on groups. pub async fn collections_list(&self, group_ident: &str) -> Result> { diff --git a/crates/picloud-cli/src/cmds/interceptors.rs b/crates/picloud-cli/src/cmds/interceptors.rs new file mode 100644 index 0000000..568b6e1 --- /dev/null +++ b/crates/picloud-cli/src/cmds/interceptors.rs @@ -0,0 +1,36 @@ +//! `pic interceptors ls --app|--group` — read-only view of a node's §9.4 +//! service interceptors. For an app, the RESOLVED chain view (every marker +//! guarding its writes, nearest-owner-wins — what actually runs); for a group, +//! its own declared markers. Authoring is declarative only (the manifest +//! `[[interceptors]]`). + +use anyhow::Result; + +use crate::client::{Client, NodeKind}; +use crate::cmds::OwnerRef; +use crate::config; +use crate::output::{OutputMode, Table}; + +pub async fn ls(app: Option<&str>, group: Option<&str>, mode: OutputMode) -> Result<()> { + let (kind, ident) = match crate::cmds::require_one_owner(app, group)? { + OwnerRef::App(a) => (NodeKind::App, a), + OwnerRef::Group(g) => (NodeKind::Group, g), + }; + let creds = config::resolve()?; + let client = Client::from_creds(&creds)?; + let items = client.interceptors_list(kind, ident).await?; + + let mut table = Table::new(["service", "op", "phase", "script", "timeout_ms"]); + for i in &items { + table.row([ + i.service.clone(), + i.op.clone(), + i.phase.clone(), + i.script.clone(), + i.timeout_ms + .map_or_else(|| "-".to_string(), |t| t.to_string()), + ]); + } + table.print(mode); + Ok(()) +} diff --git a/crates/picloud-cli/src/cmds/mod.rs b/crates/picloud-cli/src/cmds/mod.rs index 34db67a..dfead43 100644 --- a/crates/picloud-cli/src/cmds/mod.rs +++ b/crates/picloud-cli/src/cmds/mod.rs @@ -13,6 +13,7 @@ pub mod extension_points; pub mod files; pub mod groups; pub mod init; +pub mod interceptors; pub mod kv; pub mod login; pub mod logout; diff --git a/crates/picloud-cli/src/main.rs b/crates/picloud-cli/src/main.rs index 0aace3b..76d3b89 100644 --- a/crates/picloud-cli/src/main.rs +++ b/crates/picloud-cli/src/main.rs @@ -176,6 +176,15 @@ enum Cmd { cmd: ExtensionPointsCmd, }, + /// Service interceptors (§9.4) — read-only view of the before/after hooks + /// guarding a node's data-plane writes. Authored declaratively via the + /// manifest `[[interceptors]]`; `--app` shows the resolved chain (what + /// runs), `--group` the group's own declarations. + Interceptors { + #[command(subcommand)] + cmd: InterceptorsCmd, + }, + /// Shared group collections (§11.6) — read-only view of the KV collection /// names a group offers as cross-app-shared. Authored declaratively via the /// `[group]` manifest `collections = [...]`; scripts read/write them with @@ -661,6 +670,19 @@ enum ExtensionPointsCmd { }, } +#[derive(Subcommand)] +enum InterceptorsCmd { + /// List a node's service interceptors. `--app` shows the resolved chain + /// (every marker guarding its writes, nearest-owner-wins); `--group` shows + /// the group's own declarations. + Ls { + #[arg(long)] + app: Option, + #[arg(long, conflicts_with = "app")] + group: Option, + }, +} + #[derive(Subcommand)] enum CollectionsCmd { /// List the shared collections a group declares (§11.6). Group-only — @@ -2204,6 +2226,9 @@ async fn main() -> ExitCode { Cmd::ExtensionPoints { cmd: ExtensionPointsCmd::Ls { app, group }, } => cmds::extension_points::ls(app.as_deref(), group.as_deref(), mode).await, + Cmd::Interceptors { + cmd: InterceptorsCmd::Ls { app, group }, + } => cmds::interceptors::ls(app.as_deref(), group.as_deref(), mode).await, Cmd::Collections { cmd: CollectionsCmd::Ls { group }, } => cmds::collections::ls(&group, mode).await, diff --git a/crates/picloud-cli/tests/interceptors.rs b/crates/picloud-cli/tests/interceptors.rs index 6313e35..66110c4 100644 --- a/crates/picloud-cli/tests/interceptors.rs +++ b/crates/picloud-cli/tests/interceptors.rs @@ -872,3 +872,71 @@ fn a_runaway_interceptor_is_denied_by_its_timeout() { "the timed-out write must NOT persist, got: {k}" ); } + +// --- §9.4 M12: set_if (CAS) is guarded + `pic interceptors ls` ------------ + +/// A writer that CAS-writes the guarded key (must be denied) and a free key +/// (must succeed) — proving `set_if` runs the `(kv, set)` guard, not a bypass. +const CAS_WRITER: &str = r#" +let denied = false; +try { kv::collection("c").set_if("secret", (), 1); } catch(e) { denied = true; } +let ok = kv::collection("c").set_if("free", (), 2); +#{ denied: denied, ok: ok } +"#; + +/// M12: `set_if` (compare-and-swap) is a write, so it runs the same `(kv, set)` +/// interceptor as `set` — otherwise CAS would be a silent bypass of a set policy. +#[ignore = "needs DATABASE_URL pointing at a running Postgres"] +#[test] +fn set_if_runs_the_set_interceptor() { + let Some(fx) = common::fixture_or_skip() else { + return; + }; + let env = common::admin_env(fx); + let app = common::unique_slug("cas-app"); + let _a = AppGuard::new(&env.url, &env.token, &app); + common::pic_as(&env) + .args(["apps", "create", &app]) + .assert() + .success(); + + let dir = manifest_dir(); + fs::write(dir.path().join("scripts/guard.rhai"), GUARD).unwrap(); + fs::write(dir.path().join("scripts/writer.rhai"), CAS_WRITER).unwrap(); + fs::write( + dir.path().join("picloud.toml"), + format!( + "[app]\nslug = \"{app}\"\nname = \"CAS\"\n\n\ + [[scripts]]\nname = \"guard\"\nfile = \"scripts/guard.rhai\"\n\n\ + [[scripts]]\nname = \"writer\"\nfile = \"scripts/writer.rhai\"\n\n\ + [[interceptors]]\nscript = \"guard\"\nops = [\"set\"]\n" + ), + ) + .unwrap(); + common::pic_as(&env) + .args(["apply", "--file"]) + .arg(dir.path().join("picloud.toml")) + .assert() + .success(); + + let body = invoke_body(&env, &app_script_id(&env, &app, "writer")); + assert_eq!( + body, + serde_json::json!({ "denied": true, "ok": true }), + "set_if of the guarded key must be denied; a free key must swap" + ); + + // `pic interceptors ls --app` shows the resolved marker. + let ls = String::from_utf8( + common::pic_as(&env) + .args(["interceptors", "ls", "--app", &app]) + .output() + .unwrap() + .stdout, + ) + .unwrap(); + assert!( + ls.contains("kv") && ls.contains("set") && ls.contains("guard"), + "interceptors ls --app must list the kv/set guard:\n{ls}" + ); +}