fabi/PiCloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reconcile blueprint/design-doc/CLAUDE.md to shipped v1.2 state

Browse Source 
A five-agent roadmap-verification pass found the v1.2 Hierarchies code fully and
correctly implemented, but the three source-of-truth docs carried stale
"deferred/remaining/v1.3" notes — and one self-contradiction — for features that
have since shipped (§3 M3 server-side approval gate, §6 group create/reparent, the
Track A closeout, and the 2026-07-11 audit remediation). Reconcile all three to
reflect the true code; no behavior change.

CLAUDE.md: retract the inline "groups pre-exist" / "per-env approval deferred" /
"shared-topic SSE deferred" / "group dead-letter store deferred" / "email v0/no-AAD"
notes → mark §6, §3 M3, and Track A M1–M6 shipped; add per-group KV/docs byte
quotas + set_if; note migration 0063; refresh the "Out of MVP" section.

design doc: fix the header ("Phases 4–6 remain" → all shipped); resolve the §11.6
self-contradiction (the "Deferred" list still named shared-topic SSE / byte quotas /
set_if / operator admin API, all shipped — line 1317 already said SSE shipped); mark
the D3 dead-letter store shipped; retract the stale "groups pre-exist / §6 deferred"
forward references.

blueprint (comprehensive sweep): dashboard Alpine.js → SvelteKit + CodeMirror
(diagram, §3.3, tech table); Docker-per-execution → embedded in-process Rhai
(diagram + data flow); §12 Phase 4 "current focus" → shipped through v1.1.9; Phase 5
"in active development" + "Remaining" block → Hierarchies complete; per-app RBAC
"v1.3+" → shipped Phase 3.5; SDK reference → handle-pattern notation note, S3 tag
v1.1 → v1.3+; MVP schema + docker-compose flagged non-authoritative; §9 header noted
Hierarchies-shipped / Workflows-future; top status line refreshed.

Also fix two behavior-neutral stale in-code comments (sdk/kv.rs `kv::shared` →
`kv::shared_collection`; dispatcher.rs `q_terminal` "no group dead-letter store yet"
→ dead-letters to group_dead_letters, Track A M2).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-12 14:53:34 +02:00
parent c03c0c83c5
commit c8c4f012ff
5 changed files with 150 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files

180
serverless_cloud_blueprint.md
View File

@@ -1,7 +1,7 @@
# Project Blueprint: Lightweight Event-Based Serverless Cloud
**Status**: Phase 4 — Blueprint Complete
**Last Updated**: 2026-05-27
**Status**: v1.1 shipped (SDK + services) · v1.2 *Hierarchies* track **complete**; v1.2 *Workflows* track + v1.3 cluster mode are next
**Last Updated**: 2026-07-12 (reconciled to shipped code — CLAUDE.md is the live source of truth)
**Audience**: Solo developer (DIY self-hosted)
---
@@ -48,55 +48,57 @@ A lightweight, self-hosted, event-driven compute platform that allows developers
### High-Level System Diagram
> **Note (shipped reality):** the original design spawned a Docker container per
> execution. That was replaced by an **embedded, in-process Rhai executor** (better
> latency, less infra — see §12 Phase 5 note). The MVP ships as ONE `picloud` binary
> that runs the Manager, Orchestrator, and Executor as in-process crates
> (`manager-core` / `orchestrator-core` / `executor-core`); the same crates split into
> separate per-node binaries in cluster mode (v1.3+). Caddy fronts everything; the
> dashboard is a SvelteKit static SPA under `/admin`.
```
┌─────────────────────────────────────────────────────────────────┐
│ Self-Hosted Server │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ Web Dashboard │ │ Orchestrator API │ │
│ │ (Alpine.js SPA) │ │ (Rust + Axum) │ │
│ │ Port 3000 │ │ Port 8080 │ │
│ └──────┬───────────────┘ └──────────┬───────────┘ │
│ │ │ │
│ │ Upload script │ HTTP requests │
│ │ Manage scripts │ Script metadata │
│ │ │ │
│ └────────────────┬────────────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ PostgreSQL │ │
│ │ (scripts, MD) │ │
│ └────────────────┘ │
│ │ │
│ ┌────────────────┼────────────────┐ │
│ │ │ │ │
│ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ │
│ │Container │ │Container │ │Container │ │
│ │ Instance │ │ Instance │ │ Instance │ (on-demand) │
│ │(Rhai Ex.)│ │(Rhai Ex.)│ │(Rhai Ex.)│ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │
│ └─────────────────┼────────────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ Docker Daemon │ │
│ │ (container mgmt) │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ Self-Hosted Server (single node — MVP) │
├──────────────────────────────────────────────────────────────────┤
│ │
│ HTTP/HTTPS ──▶ ┌────────────────────────┐ │
│ │ Caddy 2 (reverse proxy, │ │
│ │ auto-HTTPS) │ │
│ └───────────┬────────────┘ │
│ │ │
│ ┌─────────────────▼───────────────────┐ │
│ │ picloud — all-in-one binary │ │
│ │ (Rust + Axum + Tokio) │ │
│ │ │ │
│ │ Manager Orchestrator Executor │ │
│ │ (control (ingress + (embedded│ │
│ │ plane) dispatch) Rhai + │ │
│ │ sandbox)│ │
│ │ in-process manager-core / │ │
│ │ orchestrator-core / executor-core │ │
│ │ crates (split per node in cluster │ │
│ │ mode, v1.3+) │ │
│ └──────┬────────────────────┬──────────┘ │
│ │ │ │
│ ┌───────▼────────┐ ┌───────▼─────────────┐ │
│ │ PostgreSQL 15+ │ │ SvelteKit dashboard │ │
│ │ control + data │ │ (/admin static SPA, │ │
│ │ plane (JSONB) │ │ CodeMirror editor) │ │
│ └────────────────┘ └─────────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────────┘
```
### Data Flow: HTTP Request → Response
1. **HTTP Request** arrives at Orchestrator (`POST /api/execute/{script_id}`)
2. **Orchestrator** fetches script from PostgreSQL
3. **Docker daemon** spawns container from pre-built executor image
4. **Container startup** loads script into Rhai runtime + passes request context
5. **Rhai script** executes, processes request, returns JSON object
6. **Orchestrator** extracts `statusCode`, `headers`, `body` from response
1. **HTTP Request** arrives (via Caddy) at the Orchestrator (a user route, or `POST /api/v1/execute/{id}`)
2. **Orchestrator** resolves Host → app → route, fetches the script (cached) from PostgreSQL
3. **Orchestrator** dispatches to the local Executor through the `ExecutorClient` trait (in-process call in MVP; an HTTP client in cluster mode)
4. **Executor** runs the script in the **embedded Rhai engine** with sandbox limits, passing the request context
5. **Rhai script** executes, processes the request, returns a JSON object
6. **Orchestrator** extracts `statusCode`, `headers`, `body` from the response
7. **HTTP Response** sent to client
8. **Container** is destroyed (scale to zero)
8. No per-call container — the engine is embedded, so there is nothing to spin up or tear down
---
@@ -150,7 +152,7 @@ rhai_executor --script $SCRIPT_PATH --request "$REQUEST_JSON"
---
### 3.3 Dashboard (Web UI)
**Framework**: Alpine.js (MVP), Svelte (v1.0+)
**Framework**: SvelteKit (static adapter) with a CodeMirror 6 script editor, served under `/admin`. *(The original MVP sketch below named Alpine.js; the dashboard shipped directly on SvelteKit — there was no Alpine phase.)*
**Port**: 3000 (default)
**Features (MVP):**
@@ -160,15 +162,19 @@ rhai_executor --script $SCRIPT_PATH --request "$REQUEST_JSON"
- List of deployed scripts
- Simple "Deploy" / "Delete" actions
**Technology Stack:**
- HTML + CSS + Alpine.js
- Fetch API to call Orchestrator
- No build step (initially), just serve static files
**Technology Stack (shipped):**
- SvelteKit (static adapter) + CodeMirror 6 editor, `paths.base = '/admin'`
- Fetch API to call the Manager/Orchestrator; session/bearer-token auth
- Built to static assets (`npm run build`) and served behind Caddy
---
### 3.4 PostgreSQL Database
**Schema (MVP):**
**Schema (MVP sketch — NOT authoritative):** the block below is the original MVP shape. The **authoritative
schema is the migration set** in `crates/manager-core/migrations/` (through `0070` as of v1.2), which has
since added apps/domains, RBAC (`admin_users`/`app_members`/`api_keys`), the v1.1 data-plane services
(KV/docs/files/queues/…), and the v1.2 groups/collections/templates/projects tables. Treat this as
illustration only.
```sql
CREATE TABLE scripts (
@@ -660,12 +666,12 @@ users.set_permissions(user_id, {
| Layer | Technology | Rationale |
|-------|-----------|-----------|
| **Orchestrator** | Rust + Axum | Performance, safety, async-first; minimal overhead |
| **Dashboard** | Alpine.js + vanilla HTML/CSS | Zero dependencies, simple to deploy, fast enough for MVP |
| **Dashboard** | SvelteKit (static adapter) + CodeMirror 6 | Reactive SPA, static build served under `/admin` behind Caddy |
| **Database** | PostgreSQL 15+ (`pgcrypto`) | Robust ACID database; JSONB carries data-plane values (v1.1+). See §8.1. |
| **Container Runtime** | Docker (Docker daemon) | Industry standard, simple CLI |
| **Executor Image** | Alpine Linux + Rhai | Minimal image size (~50-100MB), fast startup |
| **Reverse proxy** | Caddy 2 | Auto-HTTPS in prod; same Caddyfile shape for single-node and cluster |
| **Execution model** | Embedded, in-process Rhai engine (no per-call containers) | Lower latency + less infra than the original Docker-per-execution model |
| **Scripting** | Rhai | Lightweight, embedded-friendly, safe by default |
| **Deployment** | Docker Compose (local) / systemd (production) | Simple multi-service orchestration |
| **Deployment** | Docker Compose (dev + single-node prod) | One `picloud` all-in-one binary + Postgres + Caddy |
---
@@ -694,7 +700,14 @@ docker-compose up
docker-compose -f docker-compose.prod.yml up -d
```
### docker-compose.yml (MVP Template)
### docker-compose.yml (MVP Template — illustrative)
> **Outdated shape.** This template predates two decisions that shipped: the executor is now **embedded
> in-process** (no Docker daemon / `docker.sock` mount, no per-call executor containers), and **Caddy** is
> the reverse proxy (not nginx). The current dev/single-node stack is: `postgres` + the all-in-one
> `picloud` binary + `caddy`. Kept below for historical shape; see the repo's `docker-compose.yml` and
> `caddy/` for the real thing.
```yaml
version: '3.8'
services:
@@ -738,7 +751,7 @@ volumes:
**Purpose**: gate the admin API (`/api/v1/admin/*`) and dashboard (`/admin/*`) behind per-user authentication. Before this phase the surface was open — anyone reaching the bound port could create, edit, and delete scripts.
**Why per-user, not a shared secret**: shared admin passwords get shared between humans, leave no audit trail, and can't be revoked per-person. Per-user accounts solve all three. The initial cut deliberately stops there — no roles, no per-app permissions — because that scope is small enough to ship in a single phase without blocking Phase 3b. Roles + per-app permissions are queued for v1.3+.
**Why per-user, not a shared secret**: shared admin passwords get shared between humans, leave no audit trail, and can't be revoked per-person. Per-user accounts solve all three. The initial Phase-3a cut deliberately stopped there — no roles, no per-app permissions. Those **shipped shortly after in Phase 3.5** (`instance_role` on `admin_users`; `app_members` for per-app `app_admin`/`editor`/`viewer`; a unified `can(principal, capability)` gate), and **hierarchy-aware group RBAC** followed in v1.2 §11 Phase 2 (`effective_app_role`/`effective_group_role`).
### Naming: `admin_users` vs `users`
@@ -863,7 +876,7 @@ Permission checks land in middleware that initially only enforces "authenticated
**Deviations from the design below**: none of substance. Two operational notes:
- The Hello-World seed lives in `crates/manager-core/seeds/hello.rhai` and is inserted by a Rust bootstrap step (`seed_hello_world_if_fresh`) rather than from the migration — keeps it testable and gives the dashboard editor real source to render. The migration always inserts the `default` app + `localhost` claim; the seed only fires when that app is otherwise empty.
- Per-app admin roles/permissions are deferred — every authenticated admin can act on every app. The middleware seam (`auth_middleware::require_admin`) is the place where role checks slot in later.
- Per-app admin roles/permissions **shipped in Phase 3.5** — `app_members` grants (`app_admin`/`editor`/`viewer`) enforced through the `can(principal, capability)` gate; hierarchy-aware group RBAC extended this in v1.2 §11 Phase 2. (Originally this slot deferred them; the `auth_middleware` seam is where the checks landed.)
**Purpose**: PiCloud hosts multiple independent applications on one platform. Each app is the isolation boundary for scripts, routes, domains, and (later) data — App A cannot see or modify App B's resources except through HTTP calls between them.
@@ -1229,7 +1242,7 @@ Three foundation pieces that must land before the v1.1 service expansion, becaus
**3a. Admin auth** — ✓ shipped. See section 11.4. Per-user `admin_users` (not a shared secret), Argon2id passwords, env-var bootstrap of the first admin, session-token doubling as bearer token for API. No roles in this cut; schema is forward-compatible with later RBAC.
**3b. Multi-app scoping** — ✓ shipped. See section 11.5. `apps`, `app_domains`, `app_slug_history` tables; `app_id` columns on `scripts`, `routes`, `execution_logs`. Migration assigns existing data to a `default` app and always claims `localhost`; a Rust-side bootstrap inserts a `Hello World` script + `/hello` route when the default app is empty. Orchestrator dispatch is two-phase (Host → app → route trie). `/api/v1/execute/{id}/*` continues to work without a public domain claim. Dashboard is app-hierarchical (`/admin/apps`, `/admin/apps/{slug}/...`); API stays flat with new endpoints under `/api/v1/admin/apps/*` and a `?app=` filter on script listing. Per-app admin roles deferred.
**3b. Multi-app scoping** — ✓ shipped. See section 11.5. `apps`, `app_domains`, `app_slug_history` tables; `app_id` columns on `scripts`, `routes`, `execution_logs`. Migration assigns existing data to a `default` app and always claims `localhost`; a Rust-side bootstrap inserts a `Hello World` script + `/hello` route when the default app is empty. Orchestrator dispatch is two-phase (Host → app → route trie). `/api/v1/execute/{id}/*` continues to work without a public domain claim. Dashboard is app-hierarchical (`/admin/apps`, `/admin/apps/{slug}/...`); API stays flat with new endpoints under `/api/v1/admin/apps/*` and a `?app=` filter on script listing. Per-app admin roles shipped in Phase 3.5 (§11.6).
**3c. Users, roles, and bearer-token auth (Phase 3.5)** — ✓ shipped. See section 11.6. Adds `instance_role` to `admin_users` (`owner`/`admin`/`member`), `app_members` for per-app `app_admin`/`editor`/`viewer` grants, and `api_keys` for `Authorization: Bearer pic_…` credentials. Unifies cookie-session and API-key paths behind a single `can(principal, capability)` gate; list endpoints filter by membership at SQL for `member` users. Dashboard surfaces, invites, MFA, service accounts, and the `picloud` CLI binary are deferred — schema room only.
@@ -1237,7 +1250,7 @@ Three foundation pieces that must land before the v1.1 service expansion, becaus
---
### Phase 4: v1.1 (Expand Capabilities & Services) — Current focus
### Phase 4: v1.1 (Expand Capabilities & Services) — ✓ Shipped (through v1.1.9)
Released in patch steps (v1.1.0 → v1.1.8), each landing one focused capability. The split lets each release ship behind tests + docs without long-lived branches. SDK shape (handle pattern, `::` namespace, error convention, `ExecutionGate`, `SdkCallCx`, `ServiceEventEmitter` — see §7.5 and [docs/sdk-shape.md](../docs/sdk-shape.md)) is fixed in v1.1.0; every subsequent release fills in the contents without re-litigating the shape.
@@ -1257,9 +1270,10 @@ Released in patch steps (v1.1.0 → v1.1.8), each landing one focused capability
### Phase 5: v1.2 (Advanced Workflows & Hierarchies)
Two tracks under the same minor line. The **Hierarchies** track is in active development; the
**Workflows** track follows it (or runs in parallel — sequencing is an open call, see the design doc's
§11 re-sequencing note).
Two tracks under the same minor line. The **Hierarchies** track is **complete** (all §11 Phases 1–6
shipped, plus the Track A closeout + the 2026-07-11 audit remediation); the **Workflows** track is the
remaining, not-yet-started half (sequencing was an open call — see the design doc's §11 re-sequencing
note).
**Hierarchies — groups + the declarative project tool** (the detailed design and 6-phase breakdown
live in [docs/design/groups-and-project-tool.md](../docs/design/groups-and-project-tool.md) §11; this
@@ -1272,10 +1286,18 @@ absorbs the original one-line "Hierarchies" bullet into a full initiative):
`0048_vars.sql`, `0049_group_secrets.sql`.)*
- A **declarative, file-based project tool** (`pic plan`/`apply`/`prune`, env overlays, the three-state
`enabled` lifecycle, bound-plan staleness check). *(§11 Phase 1 — shipped.)*
- Remaining: group-owned **scripts/modules** with the scope-aware import resolver (§11 Phase 4); the
project tool **mapping onto the group tree** (nested manifests, attach points, ownership — §11 Phase
5). Group-level shared *data* (collections/topics) stays in v1.3 (the cross-app-sharing problem
below).
- **Group-owned scripts/modules** with the scope-aware, sealed-by-default import resolver + opt-in
extension points. *(§11 Phase 4/4b — shipped: `0050_group_scripts.sql`, `0051_extension_points.sql`.)*
- The **project tool mapping onto the group tree** — nested manifests, single atomic tree apply, §7
multi-repo ownership (claim/attach-ceiling/takeover), §3 M3 server-authoritative per-env approval
gate, and §6 declarative group create/reparent. *(§11 Phase 5 + §6/§7 — shipped: `0066_projects.sql`,
`0067_project_environments.sql`.)*
- **Group-level shared *data*** — shared collections (KV/docs/files/topics/queues), §4.5 group
trigger/route templates, suppression + sealed templates, per-group quotas. Pulled forward from v1.3
into v1.2 §11.6 (the ancestor-chain walk is the isolation boundary; the general cross-app
export/import model remains v1.3+). *(shipped: migrations `0052`–`0065`, `0068`.)*
- **Only remaining in this track:** multi-node cluster propagation (route-table + broadcast snapshot),
which lives in the Phase 6 / v1.3+ scaling work below.
**Workflows:**
- Function workflows (DAG execution, conditional branching, error handling)
@@ -1291,7 +1313,7 @@ absorbs the original one-line "Hierarchies" bullet into a full initiative):
- Cross-app data sharing (explicit export/import model — see section 11.5)
- Script versioning + rollback (keep N historical versions in a side table; rollback endpoint)
- Rate limiting on endpoints
- Auth (richer model: API keys, OAuth, etc.)
- Auth (richer model: OAuth / SSO — `Authorization: Bearer pic_…` **API keys already shipped in Phase 3.5**)
- Metrics + monitoring dashboard
- Distributed tracing (OpenTelemetry)
- Webhooks for execution events
@@ -1301,12 +1323,21 @@ absorbs the original one-line "Hierarchies" bullet into a full initiative):
## 7. Complete Rhai SDK Reference (MVP → v1.1+)
> **Notation note (shipped surface):** the tables below list capabilities with an
> illustrative `service.method(...)` shorthand. The **actual shipped SDK uses the
> collection-scoped handle pattern with `::` namespaces** — e.g. `kv::collection("x").get(key)`
> and `docs::collection("x").create(data)`, not `kv.get("x", key)`. Group-shared variants
> use the explicit `kv::shared_collection("x")` / `docs::shared_collection("x")` /
> `pubsub::shared_topic("x")` / `queue::shared_collection("x")` handles. See
> [docs/sdk-shape.md](docs/sdk-shape.md) and [docs/stdlib-reference.md](docs/stdlib-reference.md)
> for the authoritative signatures.
### Storage & Data
| Component | Methods | Availability |
|-----------|---------|--------------|
| **KV Store** | `kv.get(collection, key)`, `kv.set(collection, key, value, ttl?)`, `kv.delete(collection, key)`, `kv.has(collection, key)` | v1.1 |
| **Documents** | `docs.create(collection, data, schema?)`, `docs.find(collection, id)`, `docs.update(collection, id, data, schema?)`, `docs.delete(collection, id)`, `docs.list(collection, opts?)`, `docs.query(collection, filter?)` | v1.1 |
| **S3** | `s3.get(key)`, `s3.put(key, data)`, `s3.delete(key)`, `s3.list(prefix?)` | v1.1 |
| **KV Store** | `kv::collection("x").get/set/delete/has(...)`, `set_if(...)` (CAS) | v1.1 |
| **Documents** | `docs::collection("x").create/find/update/delete/list/query(...)` | v1.1 |
| **S3** | `s3.get(key)`, `s3.put(key, data)`, `s3.delete(key)`, `s3.list(prefix?)` | v1.3+ (not shipped — see Phase 6) |
| **Users** | `users.create(data)`, `users.get(id)`, `users.find_by_email(email)`, `users.search(query, limit, offset)`, `users.list(filters)`, `users.update(id, data)`, `users.authenticate(email, password)`, `users.update_password(id, old, new)`, `users.lock/unlock(id)`, `users.delete(id)`, `users.send_invite(email)`, `users.send_password_reset(email)`, `users.send_login_link(email)`, `users.has_role/permission(id, role/perm)`, `users.add/remove_role(id, role)` | v1.1 |
### Communication
@@ -1524,6 +1555,13 @@ email.send({
## 9. v1.2+ Future Vision: Workflows & Hierarchies
> **Status note:** the **Hierarchies** half of this vision has **shipped** in v1.2 (groups, inherited
> config/scripts/modules, shared collections, templates, multi-repo ownership, approval gates — see
> [docs/design/groups-and-project-tool.md](docs/design/groups-and-project-tool.md) §11 and §12 Phase 5
> above). What remains genuinely future here is the **Workflows** track below — DAG workflows, nested
> workflows, and **service interceptors** (§9.4) — plus advanced `docs.query()` filters. Those are
> unbuilt and the designs below are still speculative.
### 9.1 Function Workflows (DAG Execution)
**Concept**: Chain multiple functions together in a directed acyclic graph (DAG).