From 19ba6dbfb4077ee91f4f74c578b42174438ccb02 Mon Sep 17 00:00:00 2001 From: MechaCat02 Date: Thu, 2 Jul 2026 21:54:32 +0200 Subject: [PATCH] test/docs(shared-topics): dispatch test + journey + docs (D2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deterministic manager-core test (tests/shared_topics.rs) proves the namespace boundary both ways: a shared publish fans out only to the group's shared pubsub trigger (not a per-app or non-shared group template), and a per-app publish never hits the shared trigger. CLI journey (shared_topics) covers authoring + `pic triggers ls --group` shared column + the two validation rejections (undeclared topic; shared on an app), mirroring the shared_triggers norm. Docs: CLAUDE.md + design doc §11.6 record D1 + D2 as shipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 2 +- crates/manager-core/tests/shared_topics.rs | 225 +++++++++++++++++++++ crates/picloud-cli/tests/cli.rs | 1 + crates/picloud-cli/tests/shared_topics.rs | 159 +++++++++++++++ docs/design/groups-and-project-tool.md | 17 +- 5 files changed, 401 insertions(+), 3 deletions(-) create mode 100644 crates/manager-core/tests/shared_topics.rs create mode 100644 crates/picloud-cli/tests/shared_topics.rs diff --git a/CLAUDE.md b/CLAUDE.md index d382a3a..50da14d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,7 +10,7 @@ Authoritative design: [serverless_cloud_blueprint.md](serverless_cloud_blueprint **v1.1.x — SDK foundation + services — is complete.** The SDK shape (handle pattern, `::` namespaces, `Services`/`SdkCallCx`; see [docs/sdk-shape.md](docs/sdk-shape.md), stdlib at [docs/stdlib-reference.md](docs/stdlib-reference.md)) fixed in v1.1.0, then KV, docs, modules, HTTP, cron, files, pub/sub, email, users, and durable queues + `invoke()` filled it in through **v1.1.9** — blueprint §12 has the table. Earlier groundwork: blueprint Phase 3 (admin auth, multi-app scoping, Phase 3.5 capability gating — `manager-core::authz::{can, require, Capability}`, migration `0006_users_authz.sql`). -**Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. Multi-repo single-owner/attach-point and per-env approval gating are deferred; groups pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `/files/groups//...`, a `groups/` infix disjoint from the per-app `files//` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Deferred: topics/queue shared collections (same trigger-centric gap)), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email need materialization — see M5 below; per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey), and **§11.6 M2 shared-collection triggers** (a write to a group SHARED collection now fires a trigger — closes the "group trigger has no app to watch" gap. A group-owned trigger marked `shared = true` (`shared BOOLEAN` on `triggers`, `0061_shared_triggers.sql`) watches the group's shared collection; the per-app `list_matching_kv/docs/files` add `AND t.shared = FALSE`, new `list_matching_shared_{kv,docs,files}(owning_group,…)` select `shared = TRUE` triggers on the owning group — the `shared` flag is the namespace boundary. `ServiceEventEmitter` gained `emit_shared(cx, owning_group, event)`; `GroupKv/Docs/FilesServiceImpl` (via a `with_events` builder) emit on write, and the outbox emitter stamps the WRITER app_id so the handler runs under the writer. `shared` is authored on a group `[[triggers.kv|docs|files]]`, is part of the diff identity, and `validate_bundle_for` rejects it on an app owner / a non-collection kind / an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by `tests/shared_triggers.rs` + the `shared_triggers` journey. Deferred: shared pubsub triggers), and **§11.6 M3 per-group quotas** (global env-var ceilings enforced in the group write path: `PICLOUD_GROUP_KV_MAX_ROWS`/`_DOCS_MAX_ROWS` (per-group row count, new-key only) + `_FILES_MAX_TOTAL_BYTES` (per-group total bytes); `group_quota` helpers + `count_rows`/`total_bytes` repo methods + `QuotaExceeded` errors), and **§11.6 M4 read-only operator admin API** (`group_blobs_api` mirrors the per-app kv/files admin surface for a group's shared collections: `GET /groups/{id}/{kv,docs,files}[/{collection}/{key|id}]`, authz `GroupKvRead`/`GroupDocsRead`/`GroupFilesRead`; the host hoists the group repos to share one instance; `pic kv ls/get --group`; reads-only, writes stay script-only), and **§4.5 M5 stateful group trigger templates via materialization** (cron + queue + email — the kinds that can't resolve live because each needs a per-app row: cron `last_fired_at`, the queue one-consumer advisory lock, the email sealed inbound secret. A group `[[triggers.cron|queue|email]]` template is **materialized** into an app-owned copy per descendant (`materialized_from` col, `0062_materialized_triggers.sql`) by `materialize::rematerialize_stateful_templates` — all-apps `app_chain` CTE ⋈ group-owned stateful templates, a precise create/delete diff preserving cron state — run full-live at the route-rebuild chokepoints (apply single+tree, app create/delete in `apps_api`, group reparent in `groups_api`; `AppsState`/`GroupsState` gained a `pool`). The scheduler + `list_active_queue_consumers` + `email_inbound_target` gained `AND t.app_id IS NOT NULL` so a group TEMPLATE is never dispatched directly (nor invocable via its own webhook URL) — only its per-app copies are; a queue copy is skipped-with-warning when the app already fills that queue's slot. **M5.5 email uses the shared-group-secret model:** the template resolves its `inbound_secret_ref` against the **group's own** secret store once at apply and seals it (`resolve_and_seal`/`insert_email_trigger_tx` generalized to a `SecretOwner`/`ScriptOwner`); email secrets are v0/no-AAD so the ciphertext is portable across rows — materialization copies the sealed bytes **verbatim** (no master key, no reseal, no new schema column, no threading into the CRUD hooks). All descendant webhooks share the one group HMAC secret; an unset group secret fails apply hard. Loading a group's own set-secret names into `CurrentState` (was hardcoded empty) lets the plan-time email-secret check resolve against the group. Pinned by `tests/stateful_templates.rs` + the `stateful_templates` journey. **Deferred:** a `materialized` column in `pic triggers ls --app`; the per-app (non-shared) email secret model). Next: multi-node cluster mode, or multi-repo ownership. +**Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. Multi-repo single-owner/attach-point and per-env approval gating are deferred; groups pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `/files/groups//...`, a `groups/` infix disjoint from the per-app `files//` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Topic shared collections shipped as D2 (storeless), queue as D3 — see below), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email need materialization — see M5 below; per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey), and **§11.6 M2 shared-collection triggers** (a write to a group SHARED collection now fires a trigger — closes the "group trigger has no app to watch" gap. A group-owned trigger marked `shared = true` (`shared BOOLEAN` on `triggers`, `0061_shared_triggers.sql`) watches the group's shared collection; the per-app `list_matching_kv/docs/files` add `AND t.shared = FALSE`, new `list_matching_shared_{kv,docs,files}(owning_group,…)` select `shared = TRUE` triggers on the owning group — the `shared` flag is the namespace boundary. `ServiceEventEmitter` gained `emit_shared(cx, owning_group, event)`; `GroupKv/Docs/FilesServiceImpl` (via a `with_events` builder) emit on write, and the outbox emitter stamps the WRITER app_id so the handler runs under the writer. `shared` is authored on a group `[[triggers.kv|docs|files]]`, is part of the diff identity, and `validate_bundle_for` rejects it on an app owner / a non-collection kind / an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by `tests/shared_triggers.rs` + the `shared_triggers` journey. Shared pubsub triggers shipped as D2 — see below), and **§11.6 M3 per-group quotas** (global env-var ceilings enforced in the group write path: `PICLOUD_GROUP_KV_MAX_ROWS`/`_DOCS_MAX_ROWS` (per-group row count, new-key only) + `_FILES_MAX_TOTAL_BYTES` (per-group total bytes); `group_quota` helpers + `count_rows`/`total_bytes` repo methods + `QuotaExceeded` errors), and **§11.6 M4 read-only operator admin API** (`group_blobs_api` mirrors the per-app kv/files admin surface for a group's shared collections: `GET /groups/{id}/{kv,docs,files}[/{collection}/{key|id}]`, authz `GroupKvRead`/`GroupDocsRead`/`GroupFilesRead`; the host hoists the group repos to share one instance; `pic kv ls/get --group`; reads-only, writes stay script-only), and **§4.5 M5 stateful group trigger templates via materialization** (cron + queue + email — the kinds that can't resolve live because each needs a per-app row: cron `last_fired_at`, the queue one-consumer advisory lock, the email sealed inbound secret. A group `[[triggers.cron|queue|email]]` template is **materialized** into an app-owned copy per descendant (`materialized_from` col, `0062_materialized_triggers.sql`) by `materialize::rematerialize_stateful_templates` — all-apps `app_chain` CTE ⋈ group-owned stateful templates, a precise create/delete diff preserving cron state — run full-live at the route-rebuild chokepoints (apply single+tree, app create/delete in `apps_api`, group reparent in `groups_api`; `AppsState`/`GroupsState` gained a `pool`). The scheduler + `list_active_queue_consumers` + `email_inbound_target` gained `AND t.app_id IS NOT NULL` so a group TEMPLATE is never dispatched directly (nor invocable via its own webhook URL) — only its per-app copies are; a queue copy is skipped-with-warning when the app already fills that queue's slot. **M5.5 email uses the shared-group-secret model:** the template resolves its `inbound_secret_ref` against the **group's own** secret store once at apply and seals it (`resolve_and_seal`/`insert_email_trigger_tx` generalized to a `SecretOwner`/`ScriptOwner`); email secrets are v0/no-AAD so the ciphertext is portable across rows — materialization copies the sealed bytes **verbatim** (no master key, no reseal, no new schema column, no threading into the CRUD hooks). All descendant webhooks share the one group HMAC secret; an unset group secret fails apply hard. Loading a group's own set-secret names into `CurrentState` (was hardcoded empty) lets the plan-time email-secret check resolve against the group. Pinned by `tests/stateful_templates.rs` + the `stateful_templates` journey. **Deferred:** the per-app (non-shared) email secret model), and **D1 the `materialized` column** (`pic triggers ls --app` now shows a read-only `materialized` column — a copy of an M5 group stateful template reads `true`, distinct from a hand-authored trigger; a derived `materialized` bool = `materialized_from IS NOT NULL` threaded row→domain→API→CLI mirroring `sealed`/`shared`), and **§11.6 D2 shared TOPICS + shared pub/sub triggers** (a group declares a storeless `topic` shared collection (`group_collections.kind` widened to `topic`+`queue` in `0064`); a `[[triggers.pubsub]] shared = true` handler watches it. `BundleTrigger::Pubsub` gained `shared` (part of the identity); `validate_bundle_for` requires the topic pattern's ROOT segment (`events.*` → `events`) be a declared `kind='topic'` collection, group-only. Scripts publish via the explicit `pubsub::shared_topic("events").publish("created", msg)` handle → `GroupPubsubServiceImpl` resolves the owning group (kind `topic`) from `cx.app_id`'s chain, requires editor+ (`GroupPubsubPublish`, fails closed on anon), and fans out via `PubsubRepo::fan_out_shared_publish` to `shared = true` pubsub triggers on that group, each outbox row stamped the WRITER app_id (M2 model). The per-app `fan_out_publish` gained `AND t.shared = FALSE` — a shared trigger never fires on a per-app publish and vice-versa (the `shared` flag is the namespace boundary); the owning-group chain walk is the isolation boundary. Pinned by `tests/shared_topics.rs` + the `shared_topics` journey. **Deferred:** shared-topic external SSE subscription). Next: **D3 shared durable queues** (competing per-descendant consumers over a group-keyed store), then multi-node cluster mode / multi-repo ownership. **Data-model invariant:** app-owned data-plane tables (KV, docs, files, …) start with `app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE`; the group-inheritable tables — _config_ (`vars`, `secrets`) and now group-owned _code_ (`scripts`, `0050`) — instead carry a **polymorphic owner**: nullable `group_id` and `app_id` with an exactly-one CHECK and per-owner partial-unique indexes (config is `ON DELETE CASCADE`, scripts `RESTRICT` — code is not data). Inheritance resolves **live** down `apps.group_id → groups.parent_id` via `CHAIN_LEVELS_CTE` (no materialized view); nearest-owner-wins with an app's own row shadowing the inherited one (CoW). Every Rhai SDK call resolves its app from `cx.app_id`, never a script-passed arg, and a group script always runs under the *inheriting* app's `cx.app_id` (the cross-app isolation boundary). diff --git a/crates/manager-core/tests/shared_topics.rs b/crates/manager-core/tests/shared_topics.rs new file mode 100644 index 0000000..489ba4c --- /dev/null +++ b/crates/manager-core/tests/shared_topics.rs @@ -0,0 +1,225 @@ +//! §11.6 D2 integration test: SHARED-topic pubsub fan-out. +//! +//! A group-owned `shared = true` pubsub trigger watches the group's shared +//! topic. A SHARED publish (`fan_out_shared_publish` on the owning group) fans +//! out to it — but NOT to a per-app pubsub trigger or a non-shared group +//! template of the same pattern. Conversely a per-app publish +//! (`fan_out_publish`) fans out to the app's own + inherited non-shared group +//! templates but NEVER the shared trigger (`shared` is the namespace boundary). +//! +//! Deterministic: drives the repo fan-out directly (no async dispatcher). Skips +//! when `DATABASE_URL` is unset. + +#![allow(clippy::too_many_lines)] + +use picloud_manager_core::pubsub_repo::{PostgresPubsubRepo, PublishCtx, PubsubRepo}; +use picloud_shared::{AppId, ExecutionId, GroupId}; +use sqlx::postgres::PgPoolOptions; +use sqlx::PgPool; +use uuid::Uuid; + +async fn pool_or_skip() -> Option { + let Ok(url) = std::env::var("DATABASE_URL") else { + eprintln!("shared_topics: DATABASE_URL unset — skipping"); + return None; + }; + let pool = PgPoolOptions::new() + .max_connections(2) + .connect(&url) + .await + .expect("connect"); + sqlx::migrate!("./migrations") + .run(&pool) + .await + .expect("migrate"); + Some(pool) +} + +/// Insert a pubsub trigger (owner + `shared` flag) + its details; returns id. +async fn pubsub_trigger( + pool: &PgPool, + app_id: Option, + group_id: Option, + script: Uuid, + admin: Uuid, + shared: bool, + pattern: &str, +) -> Uuid { + let row: (Uuid,) = sqlx::query_as( + "INSERT INTO triggers \ + (app_id, group_id, script_id, kind, enabled, dispatch_mode, \ + retry_max_attempts, retry_backoff, retry_base_ms, \ + registered_by_principal, name, shared) \ + VALUES ($1, $2, $3, 'pubsub', TRUE, 'async', 3, 'exponential', 1000, $4, $5, $6) \ + RETURNING id", + ) + .bind(app_id) + .bind(group_id) + .bind(script) + .bind(admin) + .bind(Uuid::new_v4().simple().to_string()) + .bind(shared) + .fetch_one(pool) + .await + .unwrap(); + sqlx::query("INSERT INTO pubsub_trigger_details (trigger_id, topic_pattern) VALUES ($1, $2)") + .bind(row.0) + .bind(pattern) + .execute(pool) + .await + .unwrap(); + row.0 +} + +async fn outbox_count(pool: &PgPool, trigger_id: Uuid) -> i64 { + let (n,): (i64,) = sqlx::query_as("SELECT COUNT(*) FROM outbox WHERE trigger_id = $1") + .bind(trigger_id) + .fetch_one(pool) + .await + .unwrap(); + n +} + +fn ctx(app: Uuid) -> PublishCtx { + PublishCtx { + app_id: AppId::from(app), + origin_principal: None, + trigger_depth: 0, + root_execution_id: ExecutionId::new(), + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn shared_topic_fan_out_respects_the_namespace_boundary() { + let Some(pool) = pool_or_skip().await else { + return; + }; + let sfx = Uuid::new_v4().simple().to_string(); + let admin: (Uuid,) = sqlx::query_as( + "INSERT INTO admin_users (username, password_hash) VALUES ($1, 'x') RETURNING id", + ) + .bind(format!("st-{sfx}")) + .fetch_one(&pool) + .await + .unwrap(); + let g: (Uuid,) = sqlx::query_as("INSERT INTO groups (slug, name) VALUES ($1, $1) RETURNING id") + .bind(format!("st-g-{sfx}")) + .fetch_one(&pool) + .await + .unwrap(); + let handler: (Uuid,) = sqlx::query_as( + "INSERT INTO scripts (name, source, group_id) VALUES ($1, 'x', $2) RETURNING id", + ) + .bind(format!("onmsg-{sfx}")) + .bind(g.0) + .fetch_one(&pool) + .await + .unwrap(); + let app: (Uuid,) = + sqlx::query_as("INSERT INTO apps (slug, name, group_id) VALUES ($1, $1, $2) RETURNING id") + .bind(format!("st-a-{sfx}")) + .bind(g.0) + .fetch_one(&pool) + .await + .unwrap(); + let app_script: (Uuid,) = sqlx::query_as( + "INSERT INTO scripts (name, source, app_id) VALUES ($1, 'x', $2) RETURNING id", + ) + .bind(format!("appmsg-{sfx}")) + .bind(app.0) + .fetch_one(&pool) + .await + .unwrap(); + + // Three triggers, all watching `events.*`: + let shared = pubsub_trigger(&pool, None, Some(g.0), handler.0, admin.0, true, "events.*").await; + let group_tmpl = pubsub_trigger( + &pool, + None, + Some(g.0), + handler.0, + admin.0, + false, + "events.*", + ) + .await; + let per_app = pubsub_trigger( + &pool, + Some(app.0), + None, + app_script.0, + admin.0, + false, + "events.*", + ) + .await; + + let repo = PostgresPubsubRepo::new(pool.clone()); + let payload = serde_json::json!({"topic": "events.created", "message": 1}); + + // SHARED publish → only the shared trigger fires. + let n = repo + .fan_out_shared_publish( + ctx(app.0), + GroupId::from(g.0), + "events.created", + payload.clone(), + ) + .await + .unwrap(); + assert_eq!(n, 1, "shared publish matched exactly the shared trigger"); + assert_eq!(outbox_count(&pool, shared).await, 1, "shared trigger fired"); + assert_eq!( + outbox_count(&pool, group_tmpl).await, + 0, + "a non-shared group template must NOT fire on a shared publish" + ); + assert_eq!( + outbox_count(&pool, per_app).await, + 0, + "a per-app trigger must NOT fire on a shared publish" + ); + + // PER-APP publish → the app's own + inherited non-shared template fire, but + // NEVER the shared trigger. + repo.fan_out_publish(ctx(app.0), "events.created", payload) + .await + .unwrap(); + assert_eq!( + outbox_count(&pool, shared).await, + 1, + "the shared trigger must NOT fire on a per-app publish (still 1 from before)" + ); + assert_eq!( + outbox_count(&pool, group_tmpl).await, + 1, + "the inherited non-shared group template fires on a per-app publish" + ); + assert_eq!( + outbox_count(&pool, per_app).await, + 1, + "the per-app trigger fires on a per-app publish" + ); + + // Cleanup. + let _ = sqlx::query("DELETE FROM triggers WHERE registered_by_principal = $1") + .bind(admin.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM apps WHERE group_id = $1") + .bind(g.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM scripts WHERE group_id = $1") + .bind(g.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM groups WHERE id = $1") + .bind(g.0) + .execute(&pool) + .await; + let _ = sqlx::query("DELETE FROM admin_users WHERE id = $1") + .bind(admin.0) + .execute(&pool) + .await; +} diff --git a/crates/picloud-cli/tests/cli.rs b/crates/picloud-cli/tests/cli.rs index b45c26a..190d815 100644 --- a/crates/picloud-cli/tests/cli.rs +++ b/crates/picloud-cli/tests/cli.rs @@ -43,6 +43,7 @@ mod routes; mod scripts; mod sealed; mod secrets; +mod shared_topics; mod shared_triggers; mod staleness; mod stateful_templates; diff --git a/crates/picloud-cli/tests/shared_topics.rs b/crates/picloud-cli/tests/shared_topics.rs new file mode 100644 index 0000000..a9b83a5 --- /dev/null +++ b/crates/picloud-cli/tests/shared_topics.rs @@ -0,0 +1,159 @@ +//! §11.6 D2 — shared TOPICS + shared pub/sub triggers, declarative authoring +//! end to end via `pic`. A group declares a shared `topic` collection + a +//! `[[triggers.pubsub]] shared = true` handler watching it; the template +//! applies, `pic triggers ls --group` shows `shared = true`; a shared pubsub +//! trigger on an UNDECLARED topic is rejected, and `shared = true` on an app +//! pubsub trigger is rejected. +//! +//! The live FIRING (a `pubsub::shared_topic(...).publish(...)` matches the +//! shared trigger, a per-app publish does not) is pinned deterministically by +//! `manager-core/tests/shared_topics.rs` — driving the async dispatcher from a +//! journey is deliberately avoided (codebase norm, mirroring `shared_triggers`). + +use std::fs; + +use tempfile::TempDir; + +use crate::common; +use crate::common::cleanup::{AppGuard, GroupGuard, ScriptGuard}; + +fn manifest_dir() -> TempDir { + let dir = TempDir::new().expect("tempdir"); + fs::create_dir_all(dir.path().join("scripts")).expect("scripts dir"); + dir +} + +fn group_script_id(env: &common::TestEnv, group: &str, name: &str) -> String { + let ls = common::pic_as(env) + .args(["scripts", "ls", "--group", group]) + .output() + .expect("scripts ls"); + let table = String::from_utf8(ls.stdout).unwrap(); + table + .lines() + .map(common::cells) + .find(|c| c.get(2) == Some(&name)) + .and_then(|c| c.first().map(|s| (*s).to_string())) + .unwrap_or_else(|| panic!("group script `{name}` not found:\n{table}")) +} + +#[ignore = "needs DATABASE_URL pointing at a running Postgres"] +#[test] +fn shared_topic_trigger_applies_lists_and_validates() { + let Some(fx) = common::fixture_or_skip() else { + return; + }; + let env = common::admin_env(fx); + let group = common::unique_slug("shtop-grp"); + + let _g = GroupGuard::new(&env.url, &env.token, &group); + common::pic_as(&env) + .args(["groups", "create", &group]) + .assert() + .success(); + + // Group handler for the shared topic. + let dir = manifest_dir(); + fs::write( + dir.path().join("scripts/on-evt.rhai"), + r#"log::info("shared topic event"); "ok""#, + ) + .unwrap(); + common::pic_as(&env) + .args(["scripts", "deploy"]) + .arg(dir.path().join("scripts/on-evt.rhai")) + .args(["--group", &group, "--name", "on-evt"]) + .assert() + .success(); + let _gs = ScriptGuard::new( + &env.url, + &env.token, + &group_script_id(&env, &group, "on-evt"), + ); + + // Group declares a shared `topic` collection `events` + a shared pubsub + // trigger watching `events.*`. + let gmanifest = format!( + "[group]\nslug = \"{group}\"\nname = \"ShTopG\"\n\ + collections = [{{ name = \"events\", kind = \"topic\" }}]\n\n\ + [[triggers.pubsub]]\nscript = \"on-evt\"\ntopic_pattern = \"events.*\"\nshared = true\n" + ); + let gpath = dir.path().join("group.toml"); + fs::write(&gpath, &gmanifest).unwrap(); + common::pic_as(&env) + .args(["apply", "--file"]) + .arg(&gpath) + .assert() + .success(); + + // `pic triggers ls --group` shows shared = true for the pubsub trigger. + let ls = String::from_utf8( + common::pic_as(&env) + .args(["triggers", "ls", "--group", &group]) + .output() + .unwrap() + .stdout, + ) + .unwrap(); + let row = ls + .lines() + .map(common::cells) + .find(|c| c.contains(&"on-evt")) + .unwrap_or_else(|| panic!("no on-evt trigger row:\n{ls}")); + assert!( + row.contains(&"true"), + "the shared column must read true for the shared pubsub trigger:\n{ls}" + ); + + // A shared pubsub trigger whose topic root is NOT a declared shared topic is + // rejected at apply. + let bad_group = format!( + "[group]\nslug = \"{group}\"\nname = \"ShTopG\"\n\ + collections = [{{ name = \"events\", kind = \"topic\" }}]\n\n\ + [[triggers.pubsub]]\nscript = \"on-evt\"\ntopic_pattern = \"other.created\"\nshared = true\n" + ); + fs::write(&gpath, &bad_group).unwrap(); + let out = common::pic_as(&env) + .args(["apply", "--file"]) + .arg(&gpath) + .output() + .expect("apply bad"); + assert!( + !out.status.success(), + "a shared pubsub trigger on an undeclared topic must be rejected" + ); + + // `shared = true` on an APP pubsub trigger is rejected. + let app = common::unique_slug("shtop-app"); + let _a = AppGuard::new(&env.url, &env.token, &app); + common::pic_as(&env) + .args(["apps", "create", &app]) + .assert() + .success(); + fs::write( + dir.path().join("scripts/app-h.rhai"), + r#"log::info("app"); "ok""#, + ) + .unwrap(); + let amanifest = format!( + "[app]\nslug = \"{app}\"\nname = \"App\"\n\n\ + [[scripts]]\nname = \"app-h\"\nfile = \"scripts/app-h.rhai\"\n\n\ + [[triggers.pubsub]]\nscript = \"app-h\"\ntopic_pattern = \"events.*\"\nshared = true\n" + ); + let apath = dir.path().join("app.toml"); + fs::write(&apath, &amanifest).unwrap(); + let out = common::pic_as(&env) + .args(["apply", "--file"]) + .arg(&apath) + .output() + .expect("apply app"); + assert!( + !out.status.success(), + "a shared pubsub trigger on an app is rejected (apps don't own shared topics)" + ); + let err = String::from_utf8_lossy(&out.stderr); + assert!( + err.to_lowercase().contains("shared"), + "the rejection must mention shared:\n{err}" + ); +} diff --git a/docs/design/groups-and-project-tool.md b/docs/design/groups-and-project-tool.md index 14efff3..97e483a 100644 --- a/docs/design/groups-and-project-tool.md +++ b/docs/design/groups-and-project-tool.md @@ -1193,8 +1193,21 @@ Resolved items now live inline next to their topic. What genuinely remains: > kind, or an undeclared collection. Read-only `shared` column in `pic triggers ls --group`; pinned by > `tests/shared_triggers.rs` + the `shared_triggers` journey. > - > **Deferred (documented gaps):** shared **pubsub** triggers (no shared topic store yet); **topics/queue** - > shared collections (trigger-centric, same gap); per-group total-size quotas + write-rate limits; + > **Shipped — D2 shared TOPICS + shared pub/sub triggers.** A group declares a **storeless** `topic` + > shared collection (`group_collections.kind` widened to `topic`+`queue`, `0064`); a + > `[[triggers.pubsub]] shared = true` handler watches it. Scripts publish via the explicit + > `pubsub::shared_topic("events").publish("created", msg)` handle → `GroupPubsubServiceImpl` resolves the + > owning group (kind `topic`) from `cx.app_id`'s chain, requires editor+ (`GroupPubsubPublish`, fails + > closed on anon — a publish is a shared write), and fans out via `PubsubRepo::fan_out_shared_publish` to + > `shared = true` pubsub triggers on that group, each outbox row stamped the WRITER `app_id` (M2 model). + > The per-app `fan_out_publish` gained `AND t.shared = FALSE`, and `validate_bundle_for` requires the + > topic pattern's ROOT segment (`events.*` → `events`) be a declared `kind='topic'` collection + > (group-only) — the `shared` flag + owning-group chain walk are the isolation boundary. No message store + > (topics are events, not persistence). Pinned by `tests/shared_topics.rs` + the `shared_topics` journey. + > + > **Deferred (documented gaps):** shared-topic external SSE subscription; **queue** shared collections + > (D3, competing per-descendant consumers over a group-keyed store); per-group total-size quotas + + > write-rate limits; > CAS/`set_if`; an operator admin API for shared blobs (scripts use the SDK; `pic collections ls` shows > the marker — matches KV/docs); app-declared collections. Multi-node tree-apply leans on the runtime > backstop for no-op edges, as elsewhere.