diff --git a/CLAUDE.md b/CLAUDE.md index 2197a54..1b43674 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,7 @@ Authoritative design: [serverless_cloud_blueprint.md](serverless_cloud_blueprint **v1.1.x — SDK foundation + services — is complete.** The SDK shape (handle pattern, `::` namespaces, `Services`/`SdkCallCx`; see [docs/sdk-shape.md](docs/sdk-shape.md), stdlib at [docs/stdlib-reference.md](docs/stdlib-reference.md)) fixed in v1.1.0, then KV, docs, modules, HTTP, cron, files, pub/sub, email, users, and durable queues + `invoke()` filled it in through **v1.1.9** — blueprint §12 has the table. Earlier groundwork: blueprint Phase 3 (admin auth, multi-app scoping, Phase 3.5 capability gating — `manager-core::authz::{can, require, Capability}`, migration `0006_users_authz.sql`). -**Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. The single-owner ownership **claim** shipped as §7 M1, the attach-point ceiling + blast-radius as §7 M2/M3, and per-env approval gating as §3 M3 — all server-authoritative (see the tail); declarative group **create/reparent** + structural-divergence detection shipped as §6 (`reconcile_group_structure_tx`/`reparent_group_tx`/`StructureMode`), so groups no longer need to pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `/files/groups//...`, a `groups/` infix disjoint from the per-app `files//` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Topic shared collections shipped as D2 (storeless), queue as D3 — see below), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email need materialization — see M5 below; per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey), and **§11.6 M2 shared-collection triggers** (a write to a group SHARED collection now fires a trigger — closes the "group trigger has no app to watch" gap. A group-owned trigger marked `shared = true` (`shared BOOLEAN` on `triggers`, `0061_shared_triggers.sql`) watches the group's shared collection; the per-app `list_matching_kv/docs/files` add `AND t.shared = FALSE`, new `list_matching_shared_{kv,docs,files}(owning_group,…)` select `shared = TRUE` triggers on the owning group — the `shared` flag is the namespace boundary. `ServiceEventEmitter` gained `emit_shared(cx, owning_group, event)`; `GroupKv/Docs/FilesServiceImpl` (via a `with_events` builder) emit on write, and the outbox emitter stamps the WRITER app_id so the handler runs under the writer. `shared` is authored on a group `[[triggers.kv|docs|files]]`, is part of the diff identity, and `validate_bundle_for` rejects it on an app owner / a non-collection kind / an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by `tests/shared_triggers.rs` + the `shared_triggers` journey. Shared pubsub triggers shipped as D2 — see below), and **§11.6 M3 per-group quotas** (global env-var ceilings enforced in the group write path: `PICLOUD_GROUP_KV_MAX_ROWS`/`_DOCS_MAX_ROWS` (per-group row count, new-key only), `PICLOUD_GROUP_{KV,DOCS}_MAX_TOTAL_BYTES` (per-group total stored bytes, projected-total check — Track A M4) + `_FILES_MAX_TOTAL_BYTES` (per-group total bytes); `group_quota` helpers + `count_rows`/`total_bytes` repo methods + `QuotaExceeded`/`TotalBytesQuotaExceeded` errors), and **§11.6 M4 read-only operator admin API** (`group_blobs_api` mirrors the per-app kv/files admin surface for a group's shared collections: `GET /groups/{id}/{kv,docs,files}[/{collection}/{key|id}]`, authz `GroupKvRead`/`GroupDocsRead`/`GroupFilesRead`; the host hoists the group repos to share one instance; `pic kv ls/get --group`; reads-only, writes stay script-only), and **§4.5 M5 stateful group trigger templates via materialization** (cron + queue + email — the kinds that can't resolve live because each needs a per-app row: cron `last_fired_at`, the queue one-consumer advisory lock, the email sealed inbound secret. A group `[[triggers.cron|queue|email]]` template is **materialized** into an app-owned copy per descendant (`materialized_from` col, `0062_materialized_triggers.sql` + the `0063_materialized_unique.sql` partial-unique index that makes rematerialization idempotent under concurrency) by `materialize::rematerialize_stateful_templates` — all-apps `app_chain` CTE ⋈ group-owned stateful templates, a precise create/delete diff preserving cron state — run full-live at the route-rebuild chokepoints (apply single+tree, app create/delete in `apps_api`, group reparent in `groups_api`; `AppsState`/`GroupsState` gained a `pool`). The scheduler + `list_active_queue_consumers` + `email_inbound_target` gained `AND t.app_id IS NOT NULL` so a group TEMPLATE is never dispatched directly (nor invocable via its own webhook URL) — only its per-app copies are; a queue copy is skipped-with-warning when the app already fills that queue's slot. **M5.5 email uses the shared-group-secret model:** the template resolves its `inbound_secret_ref` against the **group's own** secret store once at apply and seals it (`resolve_and_seal`/`insert_email_trigger_tx` generalized to a `SecretOwner`/`ScriptOwner`); email secrets are now **v1 AAD-bound to the SEALING OWNER** (Track A M3, migration `0069_email_secret_version` + `seal_email`/`open_email`; the group AAD is stable across rows, so materialization still copies the sealed bytes **verbatim** — the inbound path recovers the sealing group via `materialized_from` and opens under its AAD; a legacy v0 read path remains for pre-M3 rows). All descendant webhooks share the one group HMAC secret; an unset group secret fails apply hard. Loading a group's own set-secret names into `CurrentState` (was hardcoded empty) lets the plan-time email-secret check resolve against the group. Pinned by `tests/stateful_templates.rs` + the `stateful_templates` journey. **Per-app (non-shared) email secrets already work** — an app-owned email trigger resolves its `inbound_secret_ref` against the **app's own** secret store and seals it AAD-bound to the app (`SecretOwner::App`, AAD `email:{app_id}` in `secrets_service::email_secret_aad`); M5.5 generalized *that* original app path to groups, not the reverse, and the inbound path recovers `SecretOwner::App` when `materialized_from` is NULL. The only per-app nuance still open is deliberate: the *interactive* `POST .../triggers/email` API takes the secret **inline**, whereas the apply path resolves a **named** app secret — the sealing/AAD machinery is fully app-aware either way), and **D1 the `materialized` column** (`pic triggers ls --app` now shows a read-only `materialized` column — a copy of an M5 group stateful template reads `true`, distinct from a hand-authored trigger; a derived `materialized` bool = `materialized_from IS NOT NULL` threaded row→domain→API→CLI mirroring `sealed`/`shared`), and **§11.6 D2 shared TOPICS + shared pub/sub triggers** (a group declares a storeless `topic` shared collection (`group_collections.kind` widened to `topic`+`queue` in `0064`); a `[[triggers.pubsub]] shared = true` handler watches it. `BundleTrigger::Pubsub` gained `shared` (part of the identity); `validate_bundle_for` requires the topic pattern's ROOT segment (`events.*` → `events`) be a declared `kind='topic'` collection, group-only. Scripts publish via the explicit `pubsub::shared_topic("events").publish("created", msg)` handle → `GroupPubsubServiceImpl` resolves the owning group (kind `topic`) from `cx.app_id`'s chain, requires editor+ (`GroupPubsubPublish`, fails closed on anon), and fans out via `PubsubRepo::fan_out_shared_publish` to `shared = true` pubsub triggers on that group, each outbox row stamped the WRITER app_id (M2 model). The per-app `fan_out_publish` gained `AND t.shared = FALSE` — a shared trigger never fires on a per-app publish and vice-versa (the `shared` flag is the namespace boundary); the owning-group chain walk is the isolation boundary. Pinned by `tests/shared_topics.rs` + the `shared_topics` journey. **External SSE subscription shipped** (Track A M6): `GET /realtime/shared/topics/{topic}` streams a shared topic to external clients; `RealtimeAuthority::authorize_subscribe_shared` resolves the owning group from the subscriber app's chain (reads-open — the resolution IS the authorization, a foreign subtree 404s), and `GroupPubsubServiceImpl::with_realtime` bridges publish→broadcast after the durable fan-out. **Deferred:** multi-node broadcast propagation (cluster mode)), and **§11.6 D3 shared durable QUEUES** (a group declares a `queue` shared collection; any subtree app enqueues into ONE group-keyed store (`group_queue_messages`, `0065`, mirrors `queue_messages` keyed by `(group_id, collection)`; CASCADE on group delete) via `queue::shared_collection("name").enqueue(...)` → `GroupQueueServiceImpl` resolves the owning group (kind `queue`), requires editor+ (`GroupQueueEnqueue`, fails closed on anon). **Consumption is by COMPETING CONSUMERS:** a group `[[triggers.queue]] shared = true` consumer (shared threaded through `BundleTrigger::Queue` + identity) **materializes** a consumer copy per descendant app (`materialize` skips the M5 one-consumer-slot check for shared — each descendant intentionally gets a consumer), and all copies claim the SHARED store with `FOR UPDATE SKIP LOCKED` — each message delivered at-most-once across the subtree, scaling horizontally, each handler under its own `cx.app_id`. The dispatcher's queue arm gained a `shared_group` on `ActiveQueueConsumer` (recovered via a LEFT JOIN to the materialized copy's source template) and routes claim/ack/nack/terminal to the group store when set (`q_claim/q_ack/q_nack/q_terminal` helpers; a group claim normalizes to a `ClaimedMessage` under the consuming app); the reclaim task drains both stores. `validate_bundle_for` requires a shared queue on a group to name a declared `kind='queue'` collection; a shared queue on an app is rejected by the app-owner shared guard. Pinned by `tests/group_queue.rs` (competing-consumer at-most-once) + `stateful_templates.rs` + the `shared_queues` journey. **Group dead-letter store shipped** (Track A M2): an exhausted shared-queue message is preserved in `group_dead_letters` (`0068_group_dead_letters`) via `GroupQueueRepo::dead_letter` (atomic INSERT+DELETE) and is operator-visible at read-only `GET /api/v1/admin/groups/{id}/dead-letters` (`GroupKvRead`). **Deferred:** fan-out to a *shared* dead-letter trigger), and **§7 multi-repo ownership M1 — the single-owner claim** (the `owner_project` seam (0047) is now live behind a first-class `projects` table (`0066_projects.sql`, UUID pk + unique slug; `owner_project` FKs it `ON DELETE SET NULL` — un-claim, never cascade-destroy a tree). A `[project]` block (slug + optional name) in the repo's ROOT manifest declares identity (independent of the `[app]`/`[group]` XOR); the first apply with a new slug registers the project and **claims** each group node it touches. The claim runs inside the apply tx under the per-node advisory lock **before the diff**, so a conflict short-circuits with a **409** before any write. Pure policy `decide_group_claim`/`decide_app_owner` in `apply_service` (unit-tested, DB-free): unclaimed→claim, owner→no-op, foreign→conflict unless `--takeover` (which additionally requires `GroupAdmin` — ownership ⟂ RBAC, mapped to 403 vs the 409 conflict), no-project-into-a-claimed-subtree→conflict. **Apps carry no owner** — an app inherits ownership from its **nearest claimed ancestor group** (the ancestor walk, via `groups.ancestors` now carrying `owner_project` through its recursive CTE, is the isolation boundary); an unclaimed subtree stays open, so nothing changes until a repo first declares `[project]` (backward-compatible). `ProjectRepository` (read side) + tx free-fns `upsert_project_tx`/`read_group_owner_tx`/`write_group_owner_tx`; the claim deliberately does **not** bump `structure_version` (not a diff change → won't churn a pending bound plan). Wire: `project`/`takeover` on the apply request (both `#[serde(default)]` → the pre-M1 CLI stays compatible); the CLI surfaces the server's 409 message verbatim (covers `StateMoved` + `OwnershipConflict`). Visibility: `pic groups ls` `owner` column (server `list_with_owner` LEFT JOIN; the shared `Group` deserialize ignores the extra field so `pic groups tree`/dashboard are unaffected) + `pic apply --takeover`. Pinned by `apply_service` unit tests, `tests/projects_repo.rs`, and the `apply_ownership` journey. **M2 shipped — the attach-point ceiling:** a `[project] parent_group = ""` binds the repo under a pre-existing group; `check_within_attach` refuses (422 `OutsideAttachPoint`) any node not strictly within that subtree (a group node must be a *proper* descendant — you can't apply the attach point itself; an app node's group must be at-or-below it), resolved via `groups.ancestors` and enforced read-only before the claim in `apply_owner`/`apply_tree`; absent = instance root = no ceiling. **M3 shipped — plan preview + `pic projects ls`:** `pic plan` now carries the `[project]` and returns an `ownership` preview per node (`claim`/`owned`/`conflict`-owner-named/`unclaimed`, pure `preview_ownership`) plus, for a group node, the cross-repo **blast radius** (descendant apps owned by OTHER projects the change reaches, via `group_blast_radius` — a subtree CTE with per-group memoized nearest-claimed resolution); the attach ceiling is previewed at plan too. Read-only `pic projects ls` (`GET /api/v1/admin/projects`, `list_with_counts`) lists projects + owned-group counts. Pinned by the `apply_ownership` plan-preview case. **With M3 the §7 multi-repo ownership track (M1 claim · M2 attach ceiling · M3 preview + `pic projects ls`) is COMPLETE.** Also shipped: **§6 group-tree Tier 1** (declarative group create via dir-nesting · structural-divergence detection · declarative reparent — `reconcile_group_structure_tx`/`StructureMode`, 422 `StructuralDivergence`) and **§3 M3 the per-env approval gate**, now **server-authoritative** (migration `0067_project_environments`; the gate resolves the governing project from the target node's nearest-claimed ancestor — `governing_env_policy`/`_tree` — so omitting/spoofing `[project]` can't bypass it; an approved gated apply requires AppAdmin/GroupAdmin step-up + audit). +**Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. The single-owner ownership **claim** shipped as §7 M1, the attach-point ceiling + blast-radius as §7 M2/M3, and per-env approval gating as §3 M3 — all server-authoritative (see the tail); declarative group **create/reparent** + structural-divergence detection shipped as §6 (`reconcile_group_structure_tx`/`reparent_group_tx`/`StructureMode`), so groups no longer need to pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `/files/groups//...`, a `groups/` infix disjoint from the per-app `files//` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Topic shared collections shipped as D2 (storeless), queue as D3 — see below), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email need materialization — see M5 below; per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey), and **§11.6 M2 shared-collection triggers** (a write to a group SHARED collection now fires a trigger — closes the "group trigger has no app to watch" gap. A group-owned trigger marked `shared = true` (`shared BOOLEAN` on `triggers`, `0061_shared_triggers.sql`) watches the group's shared collection; the per-app `list_matching_kv/docs/files` add `AND t.shared = FALSE`, new `list_matching_shared_{kv,docs,files}(owning_group,…)` select `shared = TRUE` triggers on the owning group — the `shared` flag is the namespace boundary. `ServiceEventEmitter` gained `emit_shared(cx, owning_group, event)`; `GroupKv/Docs/FilesServiceImpl` (via a `with_events` builder) emit on write, and the outbox emitter stamps the WRITER app_id so the handler runs under the writer. `shared` is authored on a group `[[triggers.kv|docs|files]]`, is part of the diff identity, and `validate_bundle_for` rejects it on an app owner / a non-collection kind / an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by `tests/shared_triggers.rs` + the `shared_triggers` journey. Shared pubsub triggers shipped as D2 — see below), and **§11.6 M3 per-group quotas** (global env-var ceilings enforced in the group write path: `PICLOUD_GROUP_KV_MAX_ROWS`/`_DOCS_MAX_ROWS` (per-group row count, new-key only), `PICLOUD_GROUP_{KV,DOCS}_MAX_TOTAL_BYTES` (per-group total stored bytes, projected-total check — Track A M4) + `_FILES_MAX_TOTAL_BYTES` (per-group total bytes); `group_quota` helpers + `count_rows`/`total_bytes` repo methods + `QuotaExceeded`/`TotalBytesQuotaExceeded` errors), and **§11.6 M4 read-only operator admin API** (`group_blobs_api` mirrors the per-app kv/files admin surface for a group's shared collections: `GET /groups/{id}/{kv,docs,files}[/{collection}/{key|id}]`, authz `GroupKvRead`/`GroupDocsRead`/`GroupFilesRead`; the host hoists the group repos to share one instance; `pic kv ls/get --group`; reads-only, writes stay script-only), and **§4.5 M5 stateful group trigger templates via materialization** (cron + queue + email — the kinds that can't resolve live because each needs a per-app row: cron `last_fired_at`, the queue one-consumer advisory lock, the email sealed inbound secret. A group `[[triggers.cron|queue|email]]` template is **materialized** into an app-owned copy per descendant (`materialized_from` col, `0062_materialized_triggers.sql` + the `0063_materialized_unique.sql` partial-unique index that makes rematerialization idempotent under concurrency) by `materialize::rematerialize_stateful_templates` — all-apps `app_chain` CTE ⋈ group-owned stateful templates, a precise create/delete diff preserving cron state — run full-live at the route-rebuild chokepoints (apply single+tree, app create/delete in `apps_api`, group reparent in `groups_api`; `AppsState`/`GroupsState` gained a `pool`). The scheduler + `list_active_queue_consumers` + `email_inbound_target` gained `AND t.app_id IS NOT NULL` so a group TEMPLATE is never dispatched directly (nor invocable via its own webhook URL) — only its per-app copies are; a queue copy is skipped-with-warning when the app already fills that queue's slot. **M5.5 email uses the shared-group-secret model:** the template resolves its `inbound_secret_ref` against the **group's own** secret store once at apply and seals it (`resolve_and_seal`/`insert_email_trigger_tx` generalized to a `SecretOwner`/`ScriptOwner`); email secrets are now **v1 AAD-bound to the SEALING OWNER** (Track A M3, migration `0069_email_secret_version` + `seal_email`/`open_email`; the group AAD is stable across rows, so materialization still copies the sealed bytes **verbatim** — the inbound path recovers the sealing group via `materialized_from` and opens under its AAD; a legacy v0 read path remains for pre-M3 rows). All descendant webhooks share the one group HMAC secret; an unset group secret fails apply hard. Loading a group's own set-secret names into `CurrentState` (was hardcoded empty) lets the plan-time email-secret check resolve against the group. Pinned by `tests/stateful_templates.rs` + the `stateful_templates` journey. **Per-app (non-shared) email secrets already work** — an app-owned email trigger resolves its `inbound_secret_ref` against the **app's own** secret store and seals it AAD-bound to the app (`SecretOwner::App`, AAD `email:{app_id}` in `secrets_service::email_secret_aad`); M5.5 generalized *that* original app path to groups, not the reverse, and the inbound path recovers `SecretOwner::App` when `materialized_from` is NULL. The only per-app nuance still open is deliberate: the *interactive* `POST .../triggers/email` API takes the secret **inline**, whereas the apply path resolves a **named** app secret — the sealing/AAD machinery is fully app-aware either way), and **D1 the `materialized` column** (`pic triggers ls --app` now shows a read-only `materialized` column — a copy of an M5 group stateful template reads `true`, distinct from a hand-authored trigger; a derived `materialized` bool = `materialized_from IS NOT NULL` threaded row→domain→API→CLI mirroring `sealed`/`shared`), and **§11.6 D2 shared TOPICS + shared pub/sub triggers** (a group declares a storeless `topic` shared collection (`group_collections.kind` widened to `topic`+`queue` in `0064`); a `[[triggers.pubsub]] shared = true` handler watches it. `BundleTrigger::Pubsub` gained `shared` (part of the identity); `validate_bundle_for` requires the topic pattern's ROOT segment (`events.*` → `events`) be a declared `kind='topic'` collection, group-only. Scripts publish via the explicit `pubsub::shared_topic("events").publish("created", msg)` handle → `GroupPubsubServiceImpl` resolves the owning group (kind `topic`) from `cx.app_id`'s chain, requires editor+ (`GroupPubsubPublish`, fails closed on anon), and fans out via `PubsubRepo::fan_out_shared_publish` to `shared = true` pubsub triggers on that group, each outbox row stamped the WRITER app_id (M2 model). The per-app `fan_out_publish` gained `AND t.shared = FALSE` — a shared trigger never fires on a per-app publish and vice-versa (the `shared` flag is the namespace boundary); the owning-group chain walk is the isolation boundary. Pinned by `tests/shared_topics.rs` + the `shared_topics` journey. **External SSE subscription shipped** (Track A M6): `GET /realtime/shared/topics/{topic}` streams a shared topic to external clients; `RealtimeAuthority::authorize_subscribe_shared` resolves the owning group from the subscriber app's chain (reads-open — the resolution IS the authorization, a foreign subtree 404s), and `GroupPubsubServiceImpl::with_realtime` bridges publish→broadcast after the durable fan-out. **Deferred:** multi-node broadcast propagation (cluster mode)), and **§11.6 D3 shared durable QUEUES** (a group declares a `queue` shared collection; any subtree app enqueues into ONE group-keyed store (`group_queue_messages`, `0065`, mirrors `queue_messages` keyed by `(group_id, collection)`; CASCADE on group delete) via `queue::shared_collection("name").enqueue(...)` → `GroupQueueServiceImpl` resolves the owning group (kind `queue`), requires editor+ (`GroupQueueEnqueue`, fails closed on anon). **Consumption is by COMPETING CONSUMERS:** a group `[[triggers.queue]] shared = true` consumer (shared threaded through `BundleTrigger::Queue` + identity) **materializes** a consumer copy per descendant app (`materialize` skips the M5 one-consumer-slot check for shared — each descendant intentionally gets a consumer), and all copies claim the SHARED store with `FOR UPDATE SKIP LOCKED` — each message delivered at-most-once across the subtree, scaling horizontally, each handler under its own `cx.app_id`. The dispatcher's queue arm gained a `shared_group` on `ActiveQueueConsumer` (recovered via a LEFT JOIN to the materialized copy's source template) and routes claim/ack/nack/terminal to the group store when set (`q_claim/q_ack/q_nack/q_terminal` helpers; a group claim normalizes to a `ClaimedMessage` under the consuming app); the reclaim task drains both stores. `validate_bundle_for` requires a shared queue on a group to name a declared `kind='queue'` collection; a shared queue on an app is rejected by the app-owner shared guard. Pinned by `tests/group_queue.rs` (competing-consumer at-most-once) + `stateful_templates.rs` + the `shared_queues` journey. **Group dead-letter store shipped** (Track A M2): an exhausted shared-queue message is preserved in `group_dead_letters` (`0068_group_dead_letters`) via `GroupQueueRepo::dead_letter` (atomic INSERT+DELETE) and is operator-visible at read-only `GET /api/v1/admin/groups/{id}/dead-letters` (`GroupKvRead`). **Shared dead-letter fan-out shipped (B2):** a group declares a declaratively-authored `[[triggers.dead_letter]] shared = true` handler; when a shared-queue message exhausts, the dispatcher's `q_terminal` group branch (after persisting to `group_dead_letters`) fans out to `list_matching_shared_dead_letter(owning_group, "queue", …)`, each outbox row stamped the WRITER `app_id` (the consuming app — M2 model), so the handler runs under the consumer. The per-app `list_matching_dead_letter` gained `AND t.shared = FALSE` (the `shared` flag is the namespace boundary); `insert_trigger_tx` now accepts `dead_letter` (`BundleTrigger::DeadLetter`), and `validate_bundle_for` requires it group-owned + shared. Pinned by `tests/shared_dead_letter.rs`), and **§7 multi-repo ownership M1 — the single-owner claim** (the `owner_project` seam (0047) is now live behind a first-class `projects` table (`0066_projects.sql`, UUID pk + unique slug; `owner_project` FKs it `ON DELETE SET NULL` — un-claim, never cascade-destroy a tree). A `[project]` block (slug + optional name) in the repo's ROOT manifest declares identity (independent of the `[app]`/`[group]` XOR); the first apply with a new slug registers the project and **claims** each group node it touches. The claim runs inside the apply tx under the per-node advisory lock **before the diff**, so a conflict short-circuits with a **409** before any write. Pure policy `decide_group_claim`/`decide_app_owner` in `apply_service` (unit-tested, DB-free): unclaimed→claim, owner→no-op, foreign→conflict unless `--takeover` (which additionally requires `GroupAdmin` — ownership ⟂ RBAC, mapped to 403 vs the 409 conflict), no-project-into-a-claimed-subtree→conflict. **Apps carry no owner** — an app inherits ownership from its **nearest claimed ancestor group** (the ancestor walk, via `groups.ancestors` now carrying `owner_project` through its recursive CTE, is the isolation boundary); an unclaimed subtree stays open, so nothing changes until a repo first declares `[project]` (backward-compatible). `ProjectRepository` (read side) + tx free-fns `upsert_project_tx`/`read_group_owner_tx`/`write_group_owner_tx`; the claim deliberately does **not** bump `structure_version` (not a diff change → won't churn a pending bound plan). Wire: `project`/`takeover` on the apply request (both `#[serde(default)]` → the pre-M1 CLI stays compatible); the CLI surfaces the server's 409 message verbatim (covers `StateMoved` + `OwnershipConflict`). Visibility: `pic groups ls` `owner` column (server `list_with_owner` LEFT JOIN; the shared `Group` deserialize ignores the extra field so `pic groups tree`/dashboard are unaffected) + `pic apply --takeover`. Pinned by `apply_service` unit tests, `tests/projects_repo.rs`, and the `apply_ownership` journey. **M2 shipped — the attach-point ceiling:** a `[project] parent_group = ""` binds the repo under a pre-existing group; `check_within_attach` refuses (422 `OutsideAttachPoint`) any node not strictly within that subtree (a group node must be a *proper* descendant — you can't apply the attach point itself; an app node's group must be at-or-below it), resolved via `groups.ancestors` and enforced read-only before the claim in `apply_owner`/`apply_tree`; absent = instance root = no ceiling. **M3 shipped — plan preview + `pic projects ls`:** `pic plan` now carries the `[project]` and returns an `ownership` preview per node (`claim`/`owned`/`conflict`-owner-named/`unclaimed`, pure `preview_ownership`) plus, for a group node, the cross-repo **blast radius** (descendant apps owned by OTHER projects the change reaches, via `group_blast_radius` — a subtree CTE with per-group memoized nearest-claimed resolution); the attach ceiling is previewed at plan too. Read-only `pic projects ls` (`GET /api/v1/admin/projects`, `list_with_counts`) lists projects + owned-group counts. Pinned by the `apply_ownership` plan-preview case. **With M3 the §7 multi-repo ownership track (M1 claim · M2 attach ceiling · M3 preview + `pic projects ls`) is COMPLETE.** Also shipped: **§6 group-tree Tier 1** (declarative group create via dir-nesting · structural-divergence detection · declarative reparent — `reconcile_group_structure_tx`/`StructureMode`, 422 `StructuralDivergence`) and **§3 M3 the per-env approval gate**, now **server-authoritative** (migration `0067_project_environments`; the gate resolves the governing project from the target node's nearest-claimed ancestor — `governing_env_policy`/`_tree` — so omitting/spoofing `[project]` can't bypass it; an approved gated apply requires AppAdmin/GroupAdmin step-up + audit). **Track A (v1.2 deferred-gap closeout, migrations 0067–0069) shipped to local main:** M1 hermetic approval gate · M2 shared-queue dead-letter store · M3 email-secret AAD v0→v1 · M4 per-group KV/docs byte quotas · M5 `set_if` compare-and-swap for KV (per-app + shared + Rhai SDK) · M6 shared-topic external SSE. **Audit 2026-07-11 remediation** (migration 0070, admin-session absolute cap) also shipped. **With that, v1.2 _Hierarchies_ is complete.** The **Workflows** track then shipped too (M1–M6: DAG execution + conditional `when`, nested sub-workflows, durable orchestrator, `workflow::start` SDK + admin run API + `pic workflows`, dashboard DAG + run-history; migrations `0071`/`0072`). A **§9.4 service-interceptor** thin slice then shipped (migration `0073_interceptors.sql`): a `[[interceptors]]` block (app or group) binds a script to run before `kv::set`/`delete` and allow/deny it, resolved nearest-owner-wins on the app chain and run via the `invoke()` re-entry path; the rest of §9.4 (data transform, non-kv services, after-hooks, chaining) is deferred. Next: the rest of §9.4 and multi-node cluster mode (the deferred multi-node route/broadcast propagation lives there). diff --git a/crates/manager-core/src/apply_service.rs b/crates/manager-core/src/apply_service.rs index f91482f..adb49e1 100644 --- a/crates/manager-core/src/apply_service.rs +++ b/crates/manager-core/src/apply_service.rs @@ -294,6 +294,23 @@ pub enum BundleTrigger { #[serde(default)] shared: bool, }, + /// §11.6 B2: a dead-letter handler. Declarative authoring is restricted to + /// GROUP-owned + `shared = true` — it fires when a message in the group's + /// SHARED queue is exhausted (dead-lettered). An app-owned or non-shared + /// `dead_letter` bundle trigger is rejected in `validate_bundle_for`. + DeadLetter { + script: String, + /// Match only dead-letters filed under this source (`"queue"` for a + /// shared-queue exhaustion). `None` matches any source. + #[serde(default)] + source_filter: Option, + #[serde(default)] + dispatch_mode: Option, + #[serde(default)] + retry_max_attempts: Option, + #[serde(default)] + shared: bool, + }, } fn default_timezone() -> String { @@ -311,7 +328,8 @@ impl BundleTrigger { | Self::Cron { script, .. } | Self::Pubsub { script, .. } | Self::Email { script, .. } - | Self::Queue { script, .. } => script, + | Self::Queue { script, .. } + | Self::DeadLetter { script, .. } => script, } } @@ -332,7 +350,10 @@ impl BundleTrigger { | Self::Docs { sealed, .. } | Self::Files { sealed, .. } | Self::Pubsub { sealed, .. } => *sealed, - Self::Cron { .. } | Self::Email { .. } | Self::Queue { .. } => false, + Self::Cron { .. } + | Self::Email { .. } + | Self::Queue { .. } + | Self::DeadLetter { .. } => false, } } @@ -347,7 +368,8 @@ impl BundleTrigger { | Self::Docs { shared, .. } | Self::Files { shared, .. } | Self::Pubsub { shared, .. } - | Self::Queue { shared, .. } => *shared, + | Self::Queue { shared, .. } + | Self::DeadLetter { shared, .. } => *shared, Self::Cron { .. } | Self::Email { .. } => false, } } @@ -427,6 +449,17 @@ impl BundleTrigger { Self::Queue { queue_name, shared, .. } => format!("queue|{queue_name}|{shared}"), + // §11.6 B2: mirrored by `current_trigger_identity` for a group-owned + // shared dead_letter, so a re-apply diffs as NoOp. + Self::DeadLetter { + script, + source_filter, + shared, + .. + } => format!( + "dead_letter|{script}|{}|{shared}", + source_filter.as_deref().unwrap_or("") + ), } } @@ -440,6 +473,7 @@ impl BundleTrigger { Self::Pubsub { .. } => "pubsub", Self::Email { .. } => "email", Self::Queue { .. } => "queue", + Self::DeadLetter { .. } => "dead_letter", } } } @@ -1117,6 +1151,11 @@ impl ApplyService { (topic_pattern.split('.').next().unwrap_or(""), "topic") } BundleTrigger::Queue { queue_name, .. } => (queue_name.as_str(), "queue"), + // §11.6 B2: a shared dead_letter watches the group's + // shared queue exhaustion, not a named collection store — + // no declared-collection requirement. The group/shared + // gate for it lives below. + BundleTrigger::DeadLetter { .. } => continue, _ => { return Err(ApplyError::Invalid(format!( "a `shared` trigger must be a kv/docs/files/pubsub/queue kind; \ @@ -1138,6 +1177,17 @@ impl ApplyService { ))); } } + // §11.6 B2: a group `dead_letter` template is only meaningful as + // `shared = true` — it fires on the group's SHARED queue + // exhaustion. A non-shared group dead_letter would never fire (no + // per-app queue to watch at the group level), so reject it. + if matches!(t, BundleTrigger::DeadLetter { .. }) && !t.shared() { + return Err(ApplyError::Invalid( + "a group `dead_letter` trigger must be `shared = true` — it \ + fires on the group's shared-queue exhaustion" + .into(), + )); + } } } // §11.6: shared collections are owned by GROUPS. Reject them on an app @@ -1182,6 +1232,21 @@ impl ApplyService { declares the collection", )); } + // §11.6 B2: a `dead_letter` trigger is authored only as a group-owned + // shared template (it watches the group's shared-queue exhaustion). + // An app-owned dead_letter handler is created via `pic triggers`, not + // the declarative manifest. + if bundle + .triggers + .iter() + .any(|t| matches!(t, BundleTrigger::DeadLetter { .. })) + { + return Err(app_only_reject( + "trigger cannot be a `dead_letter` kind", + "a declarative dead_letter is a group-owned shared template; \ + create an app dead-letter handler with `pic triggers`", + )); + } } // §11 tail M1: both an app and a group may declare suppressions — an // app declines an inherited template for itself, a group for its whole @@ -3313,6 +3378,11 @@ impl ApplyService { dispatch_mode, retry_max_attempts, .. + } + | BundleTrigger::DeadLetter { + dispatch_mode, + retry_max_attempts, + .. } => ( dispatch_mode.unwrap_or(TriggerDispatchMode::Async), retry_max_attempts.unwrap_or(self.trigger_config.retry_max_attempts), @@ -5347,7 +5417,22 @@ fn current_trigger_identity(t: &Trigger, name_by_id: &HashMap) } TriggerDetails::Email { .. } => Some(format!("email|{script}")), TriggerDetails::Queue { queue_name, .. } => Some(format!("queue|{queue_name}|{shared}")), - TriggerDetails::DeadLetter { .. } => None, + // §11.6 B2: a GROUP-owned shared dead_letter is declarative (a `[group]` + // template), so it participates in the diff like the other group + // templates — its identity mirrors `BundleTrigger::identity`. An + // APP-owned dead_letter (interactive API, not manifest-representable) + // stays `None`, so the diff neither matches nor prunes it (same as + // email). + TriggerDetails::DeadLetter { source_filter, .. } => { + if t.group_id.is_some() && shared { + Some(format!( + "dead_letter|{script}|{}|{shared}", + source_filter.as_deref().unwrap_or("") + )) + } else { + None + } + } } } @@ -5794,6 +5879,11 @@ fn bundle_trigger_details(bt: &BundleTrigger, default_visibility: u32) -> Trigge visibility_timeout_secs: visibility_timeout_secs.unwrap_or(default_visibility), last_fired_at: None, }, + BundleTrigger::DeadLetter { source_filter, .. } => TriggerDetails::DeadLetter { + source_filter: source_filter.clone(), + trigger_id_filter: None, + script_id_filter: None, + }, BundleTrigger::Email { .. } => unreachable!("email handled separately"), } } diff --git a/crates/manager-core/src/dispatcher.rs b/crates/manager-core/src/dispatcher.rs index 07af521..8e9cfbf 100644 --- a/crates/manager-core/src/dispatcher.rs +++ b/crates/manager-core/src/dispatcher.rs @@ -456,10 +456,14 @@ impl Dispatcher { // §11.6 D3: persist the exhausted message to the group dead-letter // store instead of dropping it. We return None (not the dl id) so // the per-app `fan_out_dead_letter` below is SKIPPED — firing the - // consuming app's per-app dead_letter handlers on a shared-queue + // consuming app's *per-app* dead_letter handlers on a shared-queue // message (competing consumers → nondeterministic app) would be - // wrong. Fan-out to a *shared* dead_letter trigger is deferred; the - // row is operator-visible via the group dead-letters admin API. + // wrong. + // + // §11.6 B2: instead, fan out to the group's *shared* dead_letter + // handlers (`shared = true` on the owning group). Each runs under + // the WRITER app (`claimed.app_id`, the consumer that exhausted + // the message) — the M2 shared-write model. match self .group_queue .dead_letter( @@ -475,12 +479,42 @@ impl Dispatcher { ) .await { - Ok(dl_id) => tracing::warn!( - reason, - queue = %claimed.queue_name, - dead_letter_id = %dl_id.into_inner(), - "shared-queue message dead-lettered" - ), + Ok(dl_id) => { + tracing::warn!( + reason, + queue = %claimed.queue_name, + dead_letter_id = %dl_id.into_inner(), + "shared-queue message dead-lettered" + ); + let original = TriggerEvent::Queue { + queue_name: claimed.queue_name.clone(), + message: claimed.payload.clone(), + enqueued_at: claimed.enqueued_at, + attempt: claimed.attempt, + message_id: claimed.id.to_string(), + }; + self.fan_out_shared_dead_letter( + group_id, + DeadLetterFanOutCtx { + // Writer app: the consumer that exhausted the msg. + app_id: claimed.app_id, + original, + source: "queue".to_string(), + dead_letter_id: dl_id, + attempts: claimed.attempt, + last_error: reason.to_string(), + trigger_id, + script_id, + first_attempt_at: claimed.enqueued_at, + last_attempt_at: Utc::now(), + // Shared-queue messages root a depth-1 chain (the + // queue is depth 0; a DL handler ticks up). + trigger_depth: 1, + root_execution_id: None, + }, + ) + .await; + } Err(e) => tracing::error!(?e, "shared-queue dead-letter write failed"), } None @@ -1499,6 +1533,83 @@ impl Dispatcher { } } + /// §11.6 B2: the shared analogue of `fan_out_dead_letter`. When a message in + /// a group's SHARED queue is exhausted, fire the group's `shared = true` + /// `dead_letter` handlers. Each outbox row is stamped `ctx.app_id` (the + /// writer/consumer that exhausted the message) so the handler runs under + /// that app's `SdkCallCx` — the M2 shared-write model. Best-effort, mirroring + /// the per-app path (the group dead-letter row is already durably written). + async fn fan_out_shared_dead_letter( + &self, + owning_group: picloud_shared::GroupId, + ctx: DeadLetterFanOutCtx, + ) { + let DeadLetterFanOutCtx { + app_id, + original, + source, + dead_letter_id, + attempts, + last_error, + trigger_id, + script_id, + first_attempt_at, + last_attempt_at, + trigger_depth, + root_execution_id, + } = ctx; + + let matches = match self + .triggers + .list_matching_shared_dead_letter(owning_group, &source, trigger_id, script_id) + .await + { + Ok(m) => m, + Err(e) => { + tracing::error!(?e, "shared dead-letter trigger lookup failed"); + return; + } + }; + + for m in matches { + let event = TriggerEvent::DeadLetter { + dead_letter_id, + original: Box::new(original.clone()), + attempts, + last_error: last_error.clone(), + trigger_id, + script_id, + first_attempt_at, + last_attempt_at, + }; + let payload = match serde_json::to_value(&event) { + Ok(p) => p, + Err(e) => { + tracing::error!(?e, "failed to serialize shared dead-letter event"); + continue; + } + }; + if let Err(e) = self + .outbox + .insert(NewOutboxRow { + // Writer app — the consumer that exhausted the message. + app_id, + source_kind: OutboxSourceKind::DeadLetter, + trigger_id: Some(m.trigger_id), + script_id: Some(m.script_id), + reply_to: None, + payload, + origin_principal: Some(m.registered_by_principal), + trigger_depth: trigger_depth.saturating_add(1), + root_execution_id, + }) + .await + { + tracing::error!(?e, "failed to enqueue shared dead-letter handler delivery"); + } + } + } + async fn deliver_inbox(&self, row: &OutboxRow, inbox_id: Uuid, result: InboxResult) { match self.inbox.deliver(inbox_id, result.clone()).await { InboxDeliveryOutcome::Delivered => {} diff --git a/crates/manager-core/src/trigger_repo.rs b/crates/manager-core/src/trigger_repo.rs index cc59e21..3358b0b 100644 --- a/crates/manager-core/src/trigger_repo.rs +++ b/crates/manager-core/src/trigger_repo.rs @@ -563,6 +563,23 @@ pub trait TriggerRepo: Send + Sync { script_id: Option, ) -> Result, TriggerRepoError>; + /// §11.6 B2 shared dead-letter fan-out: enabled `shared = true` + /// `dead_letter` triggers on the OWNING group — fired when a message in + /// that group's SHARED queue is dead-lettered. Same source/trigger/script + /// filter logic as `list_matching_dead_letter`, but keyed on the owning + /// group instead of a chain walk. Default empty so non-Postgres impls + /// degrade to "no shared dead-letter triggers". + async fn list_matching_shared_dead_letter( + &self, + owning_group: GroupId, + source: &str, + trigger_id: Option, + script_id: Option, + ) -> Result, TriggerRepoError> { + let _ = (owning_group, source, trigger_id, script_id); + Ok(Vec::new()) + } + /// v1.1.9. Create a queue:receive trigger. Enforces exactly one /// consumer per `(app_id, queue_name)` via `pg_advisory_xact_lock` /// + SELECT-then-INSERT (a partial unique index across the parent @@ -725,7 +742,9 @@ pub(crate) async fn insert_trigger_tx( TriggerDetails::Cron { .. } => "cron", TriggerDetails::Pubsub { .. } => "pubsub", TriggerDetails::Queue { .. } => "queue", - TriggerDetails::DeadLetter { .. } | TriggerDetails::Email { .. } => { + // §11.6 B2: a group-owned, shared dead_letter template is declarative. + TriggerDetails::DeadLetter { .. } => "dead_letter", + TriggerDetails::Email { .. } => { return Err(TriggerRepoError::Invalid( "trigger kind not supported by declarative apply".into(), )); @@ -870,7 +889,27 @@ pub(crate) async fn insert_trigger_tx( .execute(&mut **tx) .await?; } - TriggerDetails::DeadLetter { .. } | TriggerDetails::Email { .. } => { + // §11.6 B2: mirror `create_dead_letter_trigger`'s detail insert. For the + // declarative (bundle) path the trigger_id/script_id filters are always + // None — a group shared dead_letter matches by source only. + TriggerDetails::DeadLetter { + source_filter, + trigger_id_filter, + script_id_filter, + } => { + sqlx::query( + "INSERT INTO dead_letter_trigger_details \ + (trigger_id, source_filter, trigger_id_filter, script_id_filter) \ + VALUES ($1, $2, $3, $4)", + ) + .bind(tid) + .bind(source_filter.as_deref()) + .bind(trigger_id_filter.map(TriggerId::into_inner)) + .bind(script_id_filter.map(ScriptId::into_inner)) + .execute(&mut **tx) + .await?; + } + TriggerDetails::Email { .. } => { unreachable!("guarded above") } } @@ -1654,6 +1693,7 @@ impl TriggerRepo for PostgresTriggerRepo { FROM triggers t \ JOIN dead_letter_trigger_details d ON d.trigger_id = t.id \ WHERE t.app_id = $1 AND t.kind = 'dead_letter' AND t.enabled = TRUE \ + AND t.shared = FALSE \ AND (d.source_filter IS NULL OR d.source_filter = $2) \ AND (d.trigger_id_filter IS NULL OR d.trigger_id_filter = $3) \ AND (d.script_id_filter IS NULL OR d.script_id_filter = $4)", @@ -1676,6 +1716,42 @@ impl TriggerRepo for PostgresTriggerRepo { .collect()) } + async fn list_matching_shared_dead_letter( + &self, + owning_group: GroupId, + source: &str, + trigger_id: Option, + script_id: Option, + ) -> Result, TriggerRepoError> { + let rows: Vec = sqlx::query_as( + "SELECT t.id, t.script_id, t.dispatch_mode, t.registered_by_principal, \ + d.source_filter, d.trigger_id_filter, d.script_id_filter \ + FROM triggers t \ + JOIN dead_letter_trigger_details d ON d.trigger_id = t.id \ + WHERE t.group_id = $1 AND t.kind = 'dead_letter' AND t.enabled = TRUE \ + AND t.shared = TRUE \ + AND (d.source_filter IS NULL OR d.source_filter = $2) \ + AND (d.trigger_id_filter IS NULL OR d.trigger_id_filter = $3) \ + AND (d.script_id_filter IS NULL OR d.script_id_filter = $4)", + ) + .bind(owning_group.into_inner()) + .bind(source) + .bind(trigger_id.map(TriggerId::into_inner)) + .bind(script_id.map(ScriptId::into_inner)) + .fetch_all(&self.pool) + .await?; + + Ok(rows + .into_iter() + .map(|r| DeadLetterTriggerMatch { + trigger_id: r.id.into(), + script_id: r.script_id.into(), + dispatch_mode: dispatch_from_str(&r.dispatch_mode), + registered_by_principal: r.registered_by_principal.into(), + }) + .collect()) + } + async fn create_queue_trigger( &self, app_id: AppId, diff --git a/crates/manager-core/tests/shared_dead_letter.rs b/crates/manager-core/tests/shared_dead_letter.rs new file mode 100644 index 0000000..6dc2c65 --- /dev/null +++ b/crates/manager-core/tests/shared_dead_letter.rs @@ -0,0 +1,177 @@ +//! §11.6 B2 integration test: SHARED dead-letter triggers. +//! A group-owned `dead_letter` trigger marked `shared = true` fires when a +//! message in that group's SHARED queue is exhausted. It matches via the +//! OWNING-group query (`list_matching_shared_dead_letter`); a descendant app's +//! per-app dead-letter query (`list_matching_dead_letter`) does NOT match it +//! (the `shared` flag is the namespace boundary), and a foreign sibling group +//! does NOT match it (the owning-group filter is the isolation boundary). +//! +//! Deterministic: drives the repo match queries directly (no async dispatcher). +//! Skips when `DATABASE_URL` is unset. + +#![allow(clippy::needless_pass_by_value, clippy::too_many_lines)] + +use picloud_manager_core::trigger_repo::{PostgresTriggerRepo, TriggerRepo}; +use picloud_shared::{AppId, GroupId}; +use sqlx::postgres::PgPoolOptions; +use sqlx::PgPool; +use uuid::Uuid; + +async fn pool_or_skip() -> Option { + let Ok(url) = std::env::var("DATABASE_URL") else { + picloud_test_support::abort_if_db_required("shared_dead_letter"); + eprintln!("shared_dead_letter: DATABASE_URL unset — skipping"); + return None; + }; + let pool = PgPoolOptions::new() + .max_connections(2) + .connect(&url) + .await + .expect("connect"); + sqlx::migrate!("./migrations") + .run(&pool) + .await + .expect("migrate"); + Some(pool) +} + +/// Insert a group-owned dead_letter trigger (`shared` flag) + its details. +async fn dead_letter_trigger( + pool: &PgPool, + group_id: Uuid, + script: Uuid, + admin: Uuid, + shared: bool, +) -> Uuid { + let row: (Uuid,) = sqlx::query_as( + "INSERT INTO triggers \ + (app_id, group_id, script_id, kind, enabled, dispatch_mode, \ + retry_max_attempts, retry_backoff, retry_base_ms, \ + registered_by_principal, name, shared) \ + VALUES (NULL, $1, $2, 'dead_letter', TRUE, 'async', 1, 'constant', 0, $3, $4, $5) \ + RETURNING id", + ) + .bind(group_id) + .bind(script) + .bind(admin) + .bind(Uuid::new_v4().simple().to_string()) + .bind(shared) + .fetch_one(pool) + .await + .expect("trigger"); + sqlx::query( + "INSERT INTO dead_letter_trigger_details \ + (trigger_id, source_filter, trigger_id_filter, script_id_filter) \ + VALUES ($1, NULL, NULL, NULL)", + ) + .bind(row.0) + .execute(pool) + .await + .expect("details"); + row.0 +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn shared_dead_letter_matches_only_owning_group() { + let Some(pool) = pool_or_skip().await else { + return; + }; + let sfx = Uuid::new_v4().simple().to_string(); + let admin = { + let r: (Uuid,) = sqlx::query_as( + "INSERT INTO admin_users (username, password_hash) VALUES ($1, 'x') RETURNING id", + ) + .bind(format!("dl-{sfx}")) + .fetch_one(&pool) + .await + .unwrap(); + r.0 + }; + + // Group G with a group-owned handler + a SHARED dead_letter trigger. + let g: (Uuid,) = sqlx::query_as("INSERT INTO groups (slug, name) VALUES ($1, $1) RETURNING id") + .bind(format!("dl-g-{sfx}")) + .fetch_one(&pool) + .await + .unwrap(); + let handler: (Uuid,) = sqlx::query_as( + "INSERT INTO scripts (name, source, group_id) VALUES ($1, 'x', $2) RETURNING id", + ) + .bind(format!("on-dl-{sfx}")) + .bind(g.0) + .fetch_one(&pool) + .await + .unwrap(); + let shared_trig = dead_letter_trigger(&pool, g.0, handler.0, admin, true).await; + + // Descendant app A under G. + let a: (Uuid,) = + sqlx::query_as("INSERT INTO apps (slug, name, group_id) VALUES ($1, $1, $2) RETURNING id") + .bind(format!("dl-a-{sfx}")) + .bind(g.0) + .fetch_one(&pool) + .await + .unwrap(); + + // Sibling group S (foreign — not an ancestor of A). + let s: (Uuid,) = sqlx::query_as("INSERT INTO groups (slug, name) VALUES ($1, $1) RETURNING id") + .bind(format!("dl-s-{sfx}")) + .fetch_one(&pool) + .await + .unwrap(); + + let trig = PostgresTriggerRepo::new(pool.clone()); + + // 1. The owning group G's shared query returns the trigger. + let g_matches = trig + .list_matching_shared_dead_letter(GroupId::from(g.0), "queue", None, None) + .await + .expect("shared match"); + assert!( + g_matches.iter().any(|m| m.trigger_id == shared_trig.into()), + "the owning group's shared dead-letter query must match its shared trigger" + ); + + // 2. The per-app query on descendant A returns NOTHING — a shared group + // template must not match the per-app (`shared = FALSE`) path. + let a_matches = trig + .list_matching_dead_letter(AppId::from(a.0), "queue", None, None) + .await + .expect("app match"); + assert!( + !a_matches.iter().any(|m| m.trigger_id == shared_trig.into()), + "a per-app dead-letter query must NOT match the group's shared trigger" + ); + + // 3. A foreign sibling group S returns NOTHING (owning-group isolation). + let s_matches = trig + .list_matching_shared_dead_letter(GroupId::from(s.0), "queue", None, None) + .await + .expect("foreign match"); + assert!( + !s_matches.iter().any(|m| m.trigger_id == shared_trig.into()), + "a foreign group's shared dead-letter query must NOT match another group's trigger" + ); + + // Cleanup. + let _ = sqlx::query("DELETE FROM triggers WHERE id = $1") + .bind(shared_trig) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM apps WHERE id = $1") + .bind(a.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM scripts WHERE id = $1") + .bind(handler.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM groups WHERE id = ANY($1)") + .bind(vec![g.0, s.0]) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM admin_users WHERE id = $1") + .bind(admin) + .execute(&pool) + .await; +} diff --git a/crates/picloud-cli/src/cmds/plan.rs b/crates/picloud-cli/src/cmds/plan.rs index f42bddf..944e267 100644 --- a/crates/picloud-cli/src/cmds/plan.rs +++ b/crates/picloud-cli/src/cmds/plan.rs @@ -223,6 +223,9 @@ pub fn build_bundle(manifest: &Manifest, base_dir: &Path) -> Result { for s in &t.queue { triggers.push(tagged("queue", s)?); } + for s in &t.dead_letter { + triggers.push(tagged("dead_letter", s)?); + } // Vars: key → JSON value. TOML values round-trip to JSON via serde so the // wire shape matches the server's `serde_json::Value`. diff --git a/crates/picloud-cli/src/manifest.rs b/crates/picloud-cli/src/manifest.rs index aff9b28..f69fc07 100644 --- a/crates/picloud-cli/src/manifest.rs +++ b/crates/picloud-cli/src/manifest.rs @@ -468,6 +468,8 @@ pub struct ManifestTriggers { pub email: Vec, #[serde(default, skip_serializing_if = "Vec::is_empty")] pub queue: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub dead_letter: Vec, } impl ManifestTriggers { @@ -480,6 +482,7 @@ impl ManifestTriggers { && self.pubsub.is_empty() && self.email.is_empty() && self.queue.is_empty() + && self.dead_letter.is_empty() } } @@ -606,6 +609,26 @@ pub struct QueueTriggerSpec { pub shared: bool, } +/// §11.6 B2: a `[[triggers.dead_letter]]` handler. Declarative authoring is +/// restricted to GROUP-owned + `shared = true` — it fires when a message in the +/// group's SHARED queue is exhausted. `source_filter` narrows to a dead-letter +/// source (`"queue"`); omitted matches any. +#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct DeadLetterTriggerSpec { + pub script: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source_filter: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub dispatch_mode: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub retry_max_attempts: Option, + /// Must be `true` — a declarative dead_letter is a group-owned shared + /// template. The server rejects a non-shared or app-owned one. + #[serde(default, skip_serializing_if = "is_false")] + pub shared: bool, +} + /// `[secrets] names = [...]` — declares which secrets the app expects. /// Values are never in the manifest; `pic secret set` pushes them. #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]