PiCloud/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project

**PiCloud** is a self-hosted, event-driven serverless compute platform. Users upload Rhai scripts, get HTTP endpoints. Optimized for solo-dev / consumer hardware (single node MVP, multi-node cluster in v1.3+).

Authoritative design: [serverless_cloud_blueprint.md](serverless_cloud_blueprint.md). The blueprint is a living document — when architecture decisions are made in conversation that contradict it, treat the latest decision as truth and update the blueprint.

**v1.1.x — SDK foundation + services — is complete.** The SDK shape (handle pattern, `::` namespaces, `Services`/`SdkCallCx`; see [docs/sdk-shape.md](docs/sdk-shape.md), stdlib at [docs/stdlib-reference.md](docs/stdlib-reference.md)) fixed in v1.1.0, then KV, docs, modules, HTTP, cron, files, pub/sub, email, users, and durable queues + `invoke()` filled it in through **v1.1.9** — blueprint §12 has the table. Earlier groundwork: blueprint Phase 3 (admin auth, multi-app scoping, Phase 3.5 capability gating — `manager-core::authz::{can, require, Capability}`, migration `0006_users_authz.sql`).

**Current focus: v1.2 _Hierarchies_ — groups + the declarative project tool** ([docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md)). That doc's §11 uses its own **Phase 1–6 numbering, distinct from the blueprint product-phase numbering above — do not conflate them** (its "Phase 3" = group-inherited config, not admin auth). Implemented on `feat/groups-*` branches: §11 Phase 1 (declarative `pic plan`/`apply`/`prune` + env overlays), Phase 2 (single-parent groups tree + hierarchy-aware RBAC), Phase 3 (group-inherited, env-scoped `vars` + secrets resolved **live** via a recursive CTE — no materialized cache), Phase 4-lite (group-owned **endpoint** scripts: `scripts` polymorphic owner in `0050_group_scripts.sql`, `get_by_name_inherited`/`is_invocable_by_app` chain resolution, inherited `invoke()` + declarative route/trigger binding — all **live**, no body materialization), Phase 5 (the **declarative project tool maps onto the group tree**: the reconcile engine generalized to `ApplyOwner{App|Group}`, a `[group]` manifest kind, and a single atomic **tree apply** — `pic plan/apply --dir` reconciles a whole directory tree of `picloud.toml` nodes in one Postgres transaction, groups-before-apps so an app route can bind a group script created in the same tx; the bound token folds in each group's `structure_version`. Multi-repo single-owner/attach-point and per-env approval gating are deferred; groups pre-exist), Phase 4b (group **modules** + the **lexical (sealed-by-default) import resolver**, §5.5: owner-polymorphic `ModuleScript`, origin-rooted `ModuleSource::resolve` walking the importing node's chain, `ExecRequest.script_owner` threaded from every dispatch + `invoke()` site, `_source`-driven lexical chaining in `PicloudModuleResolver` with the compiled-module cache re-keyed by `ScriptId`, group modules/imports allowed, single-node dangling-import `plan` check — an inherited group script's imports **seal to the group**, a leaf can't shadow them), §5.5 **extension points** (opt-in polymorphism — **§5.5 now complete**: marker table `0051_extension_points.sql` (owner-polymorphic, CASCADE — structurally a `secrets` name; default body = a co-located `kind=module` script), `ModuleSource::resolve_policy` with **nearest-declaration-kind-wins** — a concrete module resolves lexically, an EP marker resolves **dynamically against the inheriting app** (its override else the default body up-chain), `NoProvider` is a hard error; declarative-only authoring via the `[app]`/`[group]` manifest key `extension_points = [...]`, reconcile mirrors `secrets`, single-node no-provider `plan` check, read-only `pic extension-points ls` + `pull` round-trip — the app can **override** a group default, the deliberate inverse of the Phase 4b sealed import), §11.6 **group-level collections — KV + DOCS + FILES slices** (full cross-app shared read/write: a group declares a collection shared via the `[group]` manifest `collections = [...]` → owner-polymorphic marker `0052_group_collections.sql` with a `kind` discriminator + a per-kind group-keyed store: `0053_group_kv_entries.sql` (`kind='kv'`), `0054_group_docs.sql` (`kind='docs'`, the queryable-JSON store), and `0055_group_files.sql` (`kind='files'`, blob metadata in Postgres + bytes on disk under `<root>/files/groups/<group_id>/...`, a `groups/` infix disjoint from the per-app `files/<app_id>/` subtree so the existing recursive orphan sweeper covers both with zero change) — no `app_id`, a shared row belongs to the group; CASCADE on group delete, an app delete leaves the data. Scripts use the **explicit** `kv::shared_collection("name")` / `docs::shared_collection("name")` / `files::shared_collection("name")` handles (`shared` alone is a Rhai reserved word); `GroupKv`/`GroupDocs`/`GroupFilesServiceImpl` resolve the owning group from `cx.app_id`'s ancestor chain **filtered by kind** (nearest-wins) — **that walk is the isolation boundary**, a foreign app gets `CollectionNotShared`; a `kv`, a `docs`, and a `files` collection of the same name are distinct stores. The docs slice reuses the `docs_filter` DSL — `build_find_query` generalized on its owner column (`docs`/`app_id` vs `group_docs`/`group_id`, both literals); the files slice likewise generalized the atomic-write + checksum-on-read path helpers on an owner-relative dir (one source for the security-sensitive disk mechanics). **Reads open** to any subtree script (anonymous incl. — the declaration is the grant), **writes require an authenticated editor+** on the owning group (`GroupKvRead/Write`, `GroupDocsRead/Write`, `GroupFilesRead/Write`, `script_gate_require_principal` fails closed on anon). Declarative authoring is the **string-or-table** form `collections = ["catalog", { name = "articles", kind = "docs" }, { name = "assets", kind = "files" }]` (bare string = kv); reconcile keys markers by `(name, kind)`; read-only `pic collections ls --group` shows a kind column. Deferred: shared-collection triggers (the "group trigger has no app to watch" problem), topics/queue shared collections (same trigger-centric gap), per-group quotas, an operator admin API for shared blobs (scripts use the SDK; `pic collections ls` shows the marker)), and **§4.5 group TRIGGER templates** (live, event kinds — a `[group]` declares a `[[triggers.kv|docs|files|pubsub]]` template binding a group-owned handler; `triggers` gained a polymorphic owner `0056_group_triggers.sql` mirroring `0050`; the dispatcher's `list_matching_kv/docs/files` + the pubsub publish fan-out prepend `CHAIN_LEVELS_CTE` + `JOIN chain c ON (t.app_id = c.app_owner OR t.group_id = c.group_owner)` so a descendant app's event matches its own triggers **plus** ancestor-group templates in one query, the handler running under the firing `app_id` — **the chain walk is the isolation boundary**, a sibling-subtree app never matches; stateful kinds cron/queue/email rejected on a group (need materialization), per-app opt-out deferred; read-only `pic triggers ls --group`), and **§4.5 group ROUTE templates** (live, inherited — a `[group]` declares a `[[routes]]` template binding a group-owned endpoint; `routes` gained a polymorphic owner `0057_group_routes.sql` mirroring `0056`. Unlike triggers (per-event SQL), routes serve from the in-memory `RouteTable`, so the HTTP hot path can't resolve inheritance per request — instead the table **rebuild** expands templates into each descendant app's slice via `RouteRepository::list_effective` (all-apps generalization of `CHAIN_LEVELS_CTE`: every app × its ancestor chain ⋈ routes), and `compile_effective_routes` applies **nearest-owner-wins shadowing** (an app's own identical binding shadows the inherited template; non-identical bindings coexist under the matcher's existing precedence — a route picks one winner, unlike a fanning trigger). Because the table is a cache, inheritance is rebuilt **full-live** through the single `rebuild_route_table` chokepoint on every edge that changes it: route CRUD, apply, **and tree mutations** — app create/delete (`apps_api`) + group reparent (`groups_api`) — so a new app under a group serves its templates instantly. Host-claim validation is skipped for a group template (descendants serve it on their own host claim; templates use `host_kind = any`). **The chain expansion is the isolation boundary** — a sibling-subtree app never inherits (pinned by `tests/group_route_templates.rs` + the `group_routes` journey); read-only `pic routes ls --group`. Deferred: multi-node snapshot propagation), and **§11 tail per-app opt-out (template suppression)** (a descendant declines an inherited group template: an `[app]` declares `[suppress]` with `triggers = [...]` (handler script names) + `routes = [...]` (paths) — **coarse by reference**, not a full definition, since template row-ids churn on re-apply but a reference is stable (re-apply NoOp, may decline several templates bound to the same script/path). App-only marker `0058_template_suppressions.sql` (`app_id NOT NULL` CASCADE, a `target_kind` discriminator), reconciled with the extension-point marker pattern (prunable → re-inherits). Consumed at the two resolution points: the trigger dispatch queries gain a correlated `NOT EXISTS` anti-join (gated to `t.group_id IS NOT NULL`), and `compile_effective_routes` drops an inherited (`depth > 0`) route at a suppressed path (loaded via `RouteRepository::list_route_suppressions`). **Inheritance-only** — the `group_id IS NOT NULL` / `depth > 0` gates mean an app can only decline what it inherits, never its own or a sibling's (pinned by `tests/template_suppression.rs` + the `suppress` journey); a dangling suppress is an apply-time warning; read-only `pic suppress ls --app`. **Trust-model consequence:** group templates are advisory-by-default (run *unless* a descendant declines) — a footgun for compliance hooks (audit/security triggers a tenant can opt out of)), and **§11 tail `sealed` (mandatory) group templates** (closes that footgun: a `[group]` marks a route/event-trigger template `sealed = true` and the two suppression filters skip it, so a descendant's `[suppress]` is ignored — it fires/serves on every descendant. Column `sealed BOOLEAN` on `triggers` + `routes` (`0059_sealed_templates.sql`); the trigger anti-join gains `AND t.sealed = FALSE` (a sealed row is never excluded → fires through), and `compile_effective_routes` gates its suppression `continue` on `!er.route.sealed`. `sealed` lives on the shared `Route` DTO + manager-core `Trigger` (both pure data) so the apply diff sees the current value — part of the route Update comparison + the trigger identity, so toggling it re-applies. Authored per-template (`sealed = true` on a `[[routes]]`/`[[triggers.kv|docs|files|pubsub]]`), **group-only** — `validate_bundle_for` rejects it on an app owner (an app resource is never inherited). Sealing only *strengthens* the guarantee (a sealed template can't be declined; it never grants new reach — the chain walk is still the isolation boundary). The dangling-suppress warning also flags a suppression matching only sealed templates ("… is sealed — the suppression has no effect"), and `pic triggers/routes ls --group` show a `sealed` column; pinned by `tests/sealed_templates.rs` + the `sealed` journey. Deferred: multi-node snapshot propagation), and **§11 tail M1 group-level suppression** (`template_suppressions` gained a polymorphic owner `0060_group_suppressions.sql` — a `[group]` declares `[suppress]` to decline a template it inherits from a higher ancestor for its **whole subtree**. Both filters generalized to the chain: the trigger anti-join joins the `chain` CTE (`ts.app_id = sc.app_owner OR ts.group_id = sc.group_owner`), and `list_route_suppressions` expands group suppressions across descendants via the all-apps `app_chain` CTE (`compile_effective_routes` unchanged). Still inheritance-only + `sealed` overrides; owner-polymorphic `suppression_repo::list_for_owner`/`insert`/`delete`; read-only `pic suppress ls --group`; the ineffective-suppress warning walks `GROUP_CHAIN_LEVELS_CTE` for a group node. Pinned by `tests/group_suppression.rs` + the `suppress` journey). Next: multi-node cluster mode, or extend templates to the stateful kinds, or multi-repo ownership.

**Data-model invariant:** app-owned data-plane tables (KV, docs, files, …) start with `app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE`; the group-inheritable tables — _config_ (`vars`, `secrets`) and now group-owned _code_ (`scripts`, `0050`) — instead carry a **polymorphic owner**: nullable `group_id` and `app_id` with an exactly-one CHECK and per-owner partial-unique indexes (config is `ON DELETE CASCADE`, scripts `RESTRICT` — code is not data). Inheritance resolves **live** down `apps.group_id → groups.parent_id` via `CHAIN_LEVELS_CTE` (no materialized view); nearest-owner-wins with an app's own row shadowing the inherited one (CoW). Every Rhai SDK call resolves its app from `cx.app_id`, never a script-passed arg, and a group script always runs under the *inheriting* app's `cx.app_id` (the cross-app isolation boundary).

## Three-Service Architecture

The platform splits into three logical services, each backed by a `*-core` library crate so the same logic runs in single-process MVP mode and split-process cluster mode:

| Service | Role | Library crate |
|---------|------|---------------|
| **Manager** | Control plane: script CRUD, scheduling/cron, dashboard backend, config. Single-writer to Postgres. | `manager-core` |
| **Orchestrator** | Per-node ingress: receives HTTP (later SMTP, queue) events, resolves script, dispatches to local executor. Stateless. | `orchestrator-core` |
| **Executor** | Per-node compute: runs Rhai scripts in a sandboxed engine. Stateless. | `executor-core` |

In MVP, all three run in one process (`picloud` binary). In cluster mode, each runs as its own binary on each node, with one manager total and one orchestrator + executor per node.

**Key boundary:** the orchestrator never imports `executor-core` directly — it depends on an `ExecutorClient` trait. The local impl calls `executor-core` in-process; the remote impl is an HTTP client. Same pattern keeps cluster mode a swap, not a rewrite.

## Path Scheme

Versioned API surfaces live under `/api/v{N}/...`. See [docs/versioning.md](docs/versioning.md) for the full scheme.

- `/api/v1/admin/*` — manager (control plane: script CRUD, routes CRUD + check + match, logs, config; apps CRUD once Phase 3b lands)
- `/api/v1/execute/{id}` — orchestrator (data plane: invoke a script by ID, always-available bypass)
- `/admin/*` — dashboard SPA (SvelteKit, `paths.base = '/admin'`)
- `/healthz` — liveness (string `"ok"`)
- `/version` — every compatibility-surface version + `public_base_url` (JSON)
- **everything else** — orchestrator's user-route matcher: user scripts bind to arbitrary paths via `POST /api/v1/admin/scripts/{id}/routes`; if no route matches, picloud returns 404 with a JSON error.

Reserved path prefixes (rejected at route creation): `/api/`, `/admin/`, `/healthz`, `/version`.

Caddy fronts everything. Same Caddyfile shape works for single-node and cluster — only upstream targets change.

**Param syntax convention:** route paths use `:name` (e.g., `/users/:id`); domains (once apps land) use `{name}` (e.g., `{tenant}.example.com`). These are deliberately distinct — never use `:` in a domain context or `{}` in a route-path context.

**Two-phase dispatch (Phase 3b onward):** the orchestrator first resolves `Host` → app (most-specific domain claim wins), then runs that app's route trie. The route matcher itself is unchanged and never sees other apps' routes.

## Tech Stack

- **Rust 1.92+** workspace, pinned via `rust-toolchain.toml`
- **Axum** for HTTP, **Tokio** async, **sqlx** for Postgres
- **Rhai** embedded scripting (in `executor-core`)
- **PostgreSQL 15+** with `pgcrypto`. v1.1+ data-plane tables use JSONB for value columns (hstore was considered for KV and rejected — see blueprint §8.1).
- **SvelteKit** dashboard, static adapter, CodeMirror 6 for the script editor
- **Caddy 2** reverse proxy (auto-HTTPS in prod)
- **Docker Compose** for dev and single-node prod

## Common Commands

```sh
# Rust workspace
cargo check --workspace
cargo test --workspace
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings

# Run the all-in-one binary (MVP entry point)
cargo run -p picloud

# Run a single test
cargo test -p executor-core sandbox::tests::respects_operation_budget

# Dashboard (from dashboard/)
npm run dev
npm run build
npm run check

# Full stack (once docker-compose.yml exists)
docker compose up
docker compose down -v   # reset Postgres data
```

## Workspace Layout

```
crates/
  shared/                 # cross-cutting types (Script, IDs, error enum, db pool)
  executor-core/          # Rhai engine, sandbox, ctx, log, SDK
  orchestrator-core/      # event ingress + ExecutorClient trait + dispatch
  manager-core/           # control plane: repos, scheduler, config
  picloud/                # ★ MVP all-in-one binary
  picloud-manager/        # cluster mode binary (skeleton)
  picloud-orchestrator/   # cluster mode binary (skeleton)
  picloud-executor/       # cluster mode binary (skeleton)
dashboard/                # SvelteKit
caddy/                    # Caddyfile, Caddyfile.prod
docker/                   # Dockerfiles
docs/
  git-workflow.md         # trunk-based workflow
  architecture.md         # (TBD)
```

## Working Rules

- **Honor the three-service boundary.** Don't reach across `*-core` crates *for behavior*. If `orchestrator-core` needs to invoke logic from `manager-core`, define a trait in `shared` and inject the impl — keep implementations decoupled. **Transport DTOs are not behavior**: types like `ExecRequest` / `ExecResponse` / `ExecError` represent values produced or consumed across the wire, and depending on the originating crate's type definitions is fine. The bright line is "don't call across crates," not "don't import types." When in doubt: if the imported item is a `struct`/`enum`/`type alias` with no methods (or only data-shape methods), it's a DTO and crossing is fine; if it's a trait, function, or service, define the abstraction in `shared` and inject.
- **`executor-core` has no Postgres dependency.** Data-plane services (kv, docs, users — v1.1+) come in via injected `ServiceProvider` traits.
- **Database writes only from `manager-core`.** `orchestrator-core` reads scripts (cached); `executor-core` doesn't touch the DB.
- **Stateful SDK services use the handle pattern + `SdkCallCx`.** Collection-scoped surfaces look like `kv::collection("x").get(k)`, not `kv::get("x", k)`. Every service trait method takes `&SdkCallCx` and **MUST** derive `app_id` from `cx.app_id` — never trust a script-passed `app_id`. That is the cross-app isolation boundary. See [docs/sdk-shape.md](docs/sdk-shape.md).
- **MVP builds only the `picloud` all-in-one binary.** The three split binaries exist as skeletons so the crate boundaries stay honest; flesh them out only when cluster mode is being implemented.
- **Trunk-based dev.** See [docs/git-workflow.md](docs/git-workflow.md). No long-lived branches. Feature flags for incomplete work.

## Runtime configuration

Environment variables consumed by the `picloud` binary:

| Variable | Default | Purpose |
|---|---|---|
| `PICLOUD_BIND` | `0.0.0.0:8080` | HTTP listen address. Port 8080 is owned by another process on this host — override locally. |
| `PICLOUD_MAX_CONCURRENT_EXECUTIONS` | `32` | Global concurrency cap on data-plane script executions. Overflow returns HTTP 503 with `Retry-After: 1` immediately (no queue). |
| `DATABASE_URL` | — | Required. Postgres connection string. |
| `PICLOUD_SECRET_KEY` | — | Master encryption key (base64). Required at startup unless dev mode is acknowledged (below). |
| `PICLOUD_DEV_MODE` | `false` | `true` enables local-dev conveniences. Without `PICLOUD_SECRET_KEY` it ALSO requires the acknowledgement var below — `PICLOUD_DEV_MODE=true` alone aborts at startup. Also: when no SMTP relay is configured, `email::send` switches from disabled (`NotConfigured`) to an **in-memory dev sink** — sends succeed and the last 100 messages are readable at `GET /api/v1/admin/dev/emails` (instance Owner/Admin only; route exists only in this mode). Never in production. |
| `PICLOUD_DEV_INSECURE_KEY` | — | Set to the literal `i-understand-this-is-insecure` to let dev mode boot without `PICLOUD_SECRET_KEY`, using a deterministic, world-known dev master key. Never set in production — it would encrypt everything with a public value. |
| `PICLOUD_DB_MAX_CONNECTIONS` | `32` | Postgres pool size. Matched to `PICLOUD_MAX_CONCURRENT_EXECUTIONS` so the data plane can't starve background workers. |
| `PICLOUD_SESSION_TTL_HOURS` | `24` | Sliding-window session lifetime. |
| `PICLOUD_SANDBOX_MAX_*` | conservative defaults | Per-knob admin ceilings on Rhai sandbox overrides. See `manager-core::sandbox::SandboxCeiling`. |
| `PICLOUD_FILES_ROOT` | `./data` | Filesystem root for `files::*` blob storage (v1.1.5). Bytes live at `<root>/files/<app_id>/<collection>/<id[0:2]>/<id>`; metadata in Postgres. |
| `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` | `104857600` (100 MB) | Per-file hard size cap for `files::*` (v1.1.5). Per-app quotas deferred to v1.2. |
| `PICLOUD_KV_MAX_VALUE_BYTES` | `262144` (256 KB) | Per-key JSON-encoded value cap for `kv::set`. Rejects oversized payloads before authz so anonymous public scripts can't DoS Postgres JSONB columns. |
| `PICLOUD_DOCS_MAX_VALUE_BYTES` | `262144` (256 KB) | Per-document JSON-encoded data cap for `docs::create`/`update`. |
| `PICLOUD_PUBSUB_MAX_MESSAGE_BYTES` | `262144` (256 KB) | Per-message JSON-encoded payload cap for `pubsub::publish_durable`. Prevents one publish from amplifying into N outbox rows × M MB. |
| `PICLOUD_QUEUE_MAX_PAYLOAD_BYTES` | `262144` (256 KB) | Per-message JSON-encoded payload cap for `queue::enqueue`. |

## Out of MVP

Queue triggers, cron triggers, SMTP ingress, KV / docs / email / users / HTTP SDKs in scripts, interceptors, workflows, function-to-function `invoke()`, secrets, metrics dashboard. All deferred to v1.1+ per the blueprint. Don't pre-build for them — but don't make decisions that close the door on them either.

**Pulled forward to Phase 3 (pre-v1.1):** admin auth, multi-app scoping. Cross-app data sharing (export/import) stays at v1.3+; the initial cut enforces strict isolation. See blueprint §11.5.