Phase 5 C5. Three `pic ... --dir` tree journeys:
* a group + a nested app apply atomically as one tree (the app's route binds
the group's same-apply `shared` script); the group node's script is
created; re-plan is all-noop (idempotent),
* one invalid node aborts the whole tree — the valid group node's script is
NOT created (true all-or-nothing),
* a `pic groups reparent` between `pic plan --dir` and `pic apply --dir` trips
the bound-plan (StateMoved) check via the tree's structure version.
Docs: `groups-and-project-tool.md` §11.5 status (✅ shipped — single-repo
nested tree apply; multi-repo single-owner + per-env approval gating deferred
per §11.1) + CLAUDE.md current-focus.
Gates: fmt, clippy -D warnings, cargo test --workspace, check-versioning (50
migrations — Phase 5 added none), schema snapshot unchanged, full journey suite
108/108. (Pre-existing flake `queue_e2e::queue_receive_acks_on_success` — a
racy immediate `count==0` after the async ack; flaky at the pre-Phase-5
baseline too, unrelated to this work.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mark §11.4 ✅ shipped with the live-resolution decision (no body
materialization / invalidation fan-out, mirroring Phase 3's config), the
shipped surface, and the deliberate Phase-4b deferrals (group modules + lexical
import resolver, invoke-by-id, live auto-rebinding). Note the sharp edge:
routes.script_id ON DELETE CASCADE means deleting a group script drops
descendant routes. Update the §5.1 materialization box and CLAUDE.md
(current-focus + the data-model invariant now covers group-owned scripts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The groups/project-tool work had no home in the blueprint's version
scheme, two "Phase N" numbering schemes overlapped, and CLAUDE.md's
"current focus" was stale (claimed v1.1.0 SDK foundation; we're at v1.1.9
with the SDK line complete). Reconcile all three:
* blueprint §12 Phase 5 (v1.2, already titled "Advanced Workflows &
Hierarchies") — split into a Hierarchies track (the groups + project
tool initiative, with §11 Phases 1–3 marked shipped) and a Workflows
track, with sequencing flagged as an open call. This gives the work an
explicit version home: the Hierarchies half of v1.2.
* design doc header — Status Draft → Active, "scheduled as the Hierarchies
track of v1.2", and an explicit warning that §11's local Phase 1–6
numbering is distinct from the blueprint product-phase numbering.
* design doc §11.1 — a post-Phase-3 re-sequencing review: resolve Phase 4
scripts LIVE (drop the fan-out-invalidation sub-project, per the Phase 3
result); consider a "Phase 5-lite" (declarative apply of vars/secrets +
app scripts) before the hard group-scripts work; and force the two
deferred decisions (trigger/route templates vs tenant cardinality;
multi-repo ownership vs a single-repo start).
* CLAUDE.md — refresh "current focus" to v1.2 Hierarchies, and correct the
now-broken "every v1.1+ table is app_id NOT NULL" invariant: the new
group-inheritable config tables (vars, secrets) use a polymorphic owner
(group_id XOR app_id), not app_id NOT NULL.
Doc-only; no code change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the gaps and the one security finding from the second end-to-end
CLI test (E2E_STASH_REPORT.md), plus the H1 boot-regression found while
re-reviewing those fixes.
Security
- S6: reserved-path validation (`check_reserved`) now case-folds before
comparing, so `/API/v2/x`, `/HEALTHZ`, `/Admin/x` are rejected like
their lowercase forms. Request-time matching stays case-sensitive.
- S10: "public route != public data" callout in sdk-shape.md (script_gate
skips authz when the principal is anonymous).
Observability / features
- G1: trigger executions now write `execution_logs`. Migration 0043 adds
a `source` column (CHECK mirrors ExecutionSource/OutboxSourceKind,
DEFAULT 'http' backfills history); a shared `build_execution_log` helper
in executor-core; dispatcher logging for outbox triggers + queue
consumers (skips sync-HTTP rows the orchestrator already logs). `pic
logs` gains a source column + `--source` filter.
- G5: dev-only in-memory email capture under PICLOUD_DEV_MODE with no SMTP
(email::send succeeds locally), readable at GET /api/v1/admin/dev/emails
(Owner/Admin only; route mounted only in capture mode).
- G6: generalized the Rhai in-place-mutation footgun note (trim/replace/
make_upper/make_lower/crop/truncate/pad return ()).
- G2/G3/G4 (CLI): `pic members`, `pic files`, `pic queues`, read-only
`pic kv` (+ new kv_api.rs); `pic deploy --timeout/--memory/--kind/
--sandbox`; first-class `pic triggers create-{docs,files,pubsub,queue,
email}` wrappers. All new client path segments percent-encoded via seg().
H1 regression fix (found in re-review)
- The S6 change also runs in `compile_routes`, which compiles every stored
route at boot and on each route CRUD. A single stored route the new
validation rejects (creatable while the S6 gap existed) made the whole
compile Err and aborted startup. `compile_routes` is now lenient: it
skips an un-compilable row with a warning instead of bricking boot
(route creation still validates separately). Migration 0044 sweeps
pre-existing reserved-path routes on upgrade (WHERE mirrors
check_reserved exactly). Added regression tests for both.
Verified: cargo fmt, clippy --all-targets --all-features -D warnings, the
schema_snapshot test, and the new S6/lenient-compile unit tests all pass;
boot-resilience and G1/G5 confirmed live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- S2: document that the response envelope is statusCode-gated — a returned
map is only unwrapped into {statusCode, headers, body} when it contains
statusCode, else the whole map silently becomes the 200 body.
- S3: note that Rhai String.replace() mutates in place and returns (),
with the sub_string bearer-parsing idiom.
- Document ctx.request fields including the new `method`.
- O1: document PICLOUD_DEV_INSECURE_KEY next to PICLOUD_DEV_MODE in
CLAUDE.md (the acknowledgement var was previously only in the startup
error).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
init_db hard-coded max_connections(10). The execution gate
(ExecutionGate, PICLOUD_MAX_CONCURRENT_EXECUTIONS default 32) lets 32
script executions run concurrently, each doing multiple sequential
DB calls — plus the dispatcher tick every 100ms, the cron tick, three
GC sweeps, and the auth middleware running per request all draw from
the same pool. With max=10 vs gate=32, pool starvation surfaces as
acquire_timeout errors and tail-latency spikes under load.
- DEFAULT_DB_MAX_CONNECTIONS = 32 (matches gate default).
- Env knob: PICLOUD_DB_MAX_CONNECTIONS overrides; invalid/zero → default.
- Documented in CLAUDE.md runtime config table.
AUDIT.md anchor: F-P-003.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files (per-file cap), secrets (64 KB default), and email (25 MB default)
already enforce limits; kv::set, docs::create/update, pubsub::publish_durable
and queue::enqueue accepted any JSON value straight to a JSONB column with
no size validation. An anonymous public-HTTP script could fill disk via
queue::enqueue or amplify a single publish into N outbox rows × payload bytes.
Adds four new error variants:
- KvError::ValueTooLarge { limit, actual }
- DocsError::ValueTooLarge { limit, actual }
- PubsubError::MessageTooLarge { limit, actual }
- QueueError::PayloadTooLarge { limit, actual }
Each stateful service grows a `max_value_bytes` field with:
- Conservative 256 KB default (DEFAULT_KV_MAX_VALUE_BYTES etc.).
- New `with_max_*` constructor preserving the old `new()` signature.
- Env-knob reader (`*_max_*_from_env()`) — mirrors SecretsConfig::from_env.
Validation runs at the entry point BEFORE authz so an anonymous DoS doesn't
pay a membership lookup per attempt.
Wired via env knobs:
- PICLOUD_KV_MAX_VALUE_BYTES
- PICLOUD_DOCS_MAX_VALUE_BYTES
- PICLOUD_PUBSUB_MAX_MESSAGE_BYTES
- PICLOUD_QUEUE_MAX_PAYLOAD_BYTES
Documented in CLAUDE.md runtime config table.
New unit test in queue_service verifying the cap fires before authz.
AUDIT.md anchor: F-S-001. Depends on F-Q-004 (Backend variant).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "don't reach across *-core crates" rule was being read as
prohibiting any cross-crate import, but the load-bearing intent is
to keep *behavior* decoupled (so cluster-mode can swap implementations
behind traits in shared). Importing transport DTOs across crates is
fine — ExecRequest/ExecResponse/ExecError live in executor-core
because that's where they're produced, and the v1.1.1 dispatcher in
manager-core legitimately consumes them.
Bright line: structs/enums/type-aliases crossing is fine; traits,
functions, and service handles crossing is not.
Surfaced during the v1.1.1 audit (see REVIEW.md §4).
A script author opening docs/stdlib-reference.md should see every
function they can call without imports: the Rhai built-in stdlib (math,
string, array, map, blob) plus the seven new PiCloud namespaces. Tight
tables over prose — scannable rather than exhaustive.
CLAUDE.md current-focus paragraph picks up a pointer to the new doc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Current focus moves to v1.1.0 (SDK foundation + stdlib) with a
pointer to docs/sdk-shape.md. Notes Phase 3.5 capability gating is
shipped end-to-end.
- Tech-stack line drops the misleading "v1.1+ hstore" mention; v1.1+
data-plane tables now use JSONB (see blueprint §8.1).
- New Working Rules bullet for the handle pattern + SdkCallCx rule:
services derive app_id from cx.app_id, never from a script-passed
arg. That is the cross-app isolation boundary.
- New "Runtime configuration" table documenting every env var the
picloud binary consumes — including the new
PICLOUD_MAX_CONCURRENT_EXECUTIONS alongside the existing
PICLOUD_BIND, DATABASE_URL, session TTL, and sandbox knobs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apps become the isolation boundary for scripts, routes, domains, and
later data. Doing this now — while the surface is small — avoids
several migrations on populated tables once v1.1 data-plane services
ship.
Schema (migration 0005_apps.sql):
- New tables: apps, app_domains (with shape_key UNIQUE for collision
detection), app_slug_history (for permanent slug-rename redirects).
- app_id added to scripts, routes, execution_logs (non-null, cascading
rules per row).
- Script-name uniqueness becomes per-app; the route unique index is
swapped for an app-scoped version.
- The "default" app is seeded unconditionally with a localhost claim;
existing scripts/routes backfill into it. Fresh installs additionally
get the Hello World seed via seed_hello_world_if_fresh after
migrations run (idempotent — only fires when the default app has no
scripts).
Orchestrator dispatch is two-phase: AppDomainTable resolves Host →
app_id (most-specific match wins, exact beats wildcard), then the
existing route matcher runs against that app's partitioned slice via
RouteTable. Unknown hosts return 404 at the app layer with a clear
message; /api/v1/execute/{id} still works as the implicit
__internal__ claim, decoupled from any public domain.
Manager API: full CRUD for /api/v1/admin/apps/* and
/api/v1/admin/apps/{id_or_slug}/domains/*, with slug:check + force
takeover semantics implementing the rename-history flow (two-step
check → confirm, never a single endpoint). Script create requires
app_id; list accepts ?app= filter. Route create validates host
against the parent app's claims; conflict detection stays strictly
intra-app.
Dashboard: /admin/apps and /admin/apps/{slug} (overview + scripts +
domains + settings tabs, with slug-history-aware redirects). Root
path redirects to the apps list. Script detail page gains an app
breadcrumb and threads app_id into the route preview.
Deferred per design: per-app admin roles. The require_admin middleware
remains the seam where role checks will slot in later.
Blueprint §11.5 and roadmap updated to reflect what shipped; docs/
versioning.md notes the schema 3 → 5 bump.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds blueprint sections 11.4 (admin auth) and 11.5 (app scoping) and
restructures the section 12 roadmap to put both ahead of v1.1, since
retrofitting app_id into KV/docs/users schemas after they ship is far
more expensive than adding it now.
Admin auth: per-user admin_users (not a shared secret), Argon2id,
env-var bootstrap that becomes inert after first admin exists, session
token doubling as bearer token, 24h sliding TTL. Schema designed
forward-compatible with later RBAC.
App scoping: apps own scripts/routes/domains. Domain claims at app
level (exact / wildcard / {param} parameterized) with collision check
at claim time, so route-conflict errors stay strictly intra-app.
Two-phase orchestrator dispatch (Host → app → route trie). Slug rename
keeps the old slug as a permanent redirect until another app claims
it. Fresh-install migration seeds a Hello World app; upgrades go into
a default app instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scripts can now answer at user-chosen paths (e.g. /greet, /greet/:name,
/webhooks/*), on user-chosen hosts (strict or *.example.com wildcards),
on user-chosen methods. The internal /api/v1/execute/{id} endpoint
stays as the always-available ID-based bypass.
Routing rules (decided in design with the user; see chat history):
Path kinds:
exact /greet literal
prefix /greet/* strict-subtree; stored as "/greet/";
does NOT match bare /greet (add an
exact route for that case)
param /users/:id :name captures one whole segment;
mid-segment colons are rejected;
{name} is reserved for a future SDK
Host kinds:
any no Host header constraint
strict sub.example.com literal match (case-insensitive)
wildcard *.example.com suffix match; multi-level subdomains OK
Within-kind uniqueness:
two routes of the same kind that could match the same request
conflict at config time. Algorithm (orchestrator_core::routing::
conflict):
exact: literal equality
prefix: literal equality (longer-prefix coexists; longer wins
at request time)
param: same segment count + same literals at every
literal-vs-literal position (the user's example:
:id vs :userId at same shape is a conflict)
Request-time precedence:
exact > param > prefix
among non-exact: more leading-literal segments wins
tie: param > prefix (more constrained)
within prefix: longest matching prefix wins
host bucket: strict > wildcard (longer suffix) > any; fall through
to less specific buckets when path doesn't match
Reserved path prefixes: /api/, /admin/, /healthz, /version
Routes that look invalid at config time return 422 with the precise
parse error; conflicting routes return 409 with the conflicting route
in the body (so the dashboard can render the conflict inline).
What landed:
* 0003_routes.sql — routes table (host_kind, host, host_param_name,
path_kind, path, method, script_id) with UNIQUE index on the
literal binding tuple. Schema 2 → 3.
* shared::Route / HostKind / PathKind — flat storage shape that
crosses wire boundaries cleanly.
* orchestrator_core::routing — four sub-modules, all unit-tested:
pattern.rs (16 tests) parse + validate + display
conflict.rs (12 tests) within-kind overlap predicate
matcher.rs (12 tests) runtime dispatch (specificity-aware)
table.rs Arc<RwLock<Vec<CompiledRoute>>>
shared by manager (writes) and
orchestrator (reads); atomic replace
after each admin write
* manager-core::route_admin — five new admin endpoints under
/api/v1/admin:
POST /scripts/{id}/routes create
GET /scripts/{id}/routes list per script
DELETE /routes/{route_id} delete (refreshes table)
POST /routes:check pre-flight conflict check
(powers the dashboard's
live conflict warning)
POST /routes:match synthetic URL → matched
route + extracted params
(powers the dashboard's
match-preview tool)
Stored path strings stay raw (user-typed); normalization
happens only in the in-memory CompiledRoute so re-parses are
idempotent.
* orchestrator_core::api::user_routes_router — fallback handler
mounted in picloud after the system routes. Reads Host /
method / path / query from the request, dispatches via the
table, builds an ExecRequest with params/query/rest filled,
calls the executor, writes to the log sink. 10 MiB body cap.
* executor-core::ctx (SDK 1.0 → 1.1) — adds
ctx.request.params (map of named-param captures)
ctx.request.query (parsed query string)
ctx.request.rest (suffix for prefix routes; "" otherwise)
All three are always present (empty when not applicable) so
scripts can read them unconditionally.
* picloud::build_app — now async; loads routes at startup,
populates the shared table, mounts route_admin_router under
/api/v1/admin alongside the script CRUD, and the user-routes
fallback at the app root.
* caddy/Caddyfile + Caddyfile.prod widened: anything not
/healthz, /version, /api/v1/admin/*, /api/v1/execute/*,
/api/* (404 sunset), or /admin/* (dashboard) → picloud.
* Dashboard moves to /admin/* via SvelteKit paths.base. Its
internal Caddy strips the prefix and serves with SPA fallback.
All in-app links use $app/paths. The dashboard URL is now
http://localhost:8000/admin/ — one-time break for the new
URL freedom users gained.
* PICLOUD_PUBLIC_BASE_URL env var, exposed via /version so the
dashboard renders full URLs for routes regardless of the
operator's external port / TLS setup.
* memory_limit_mb stays in the schema, still v1.3+ advisory.
Verified live through Caddy:
/version → schema 3, sdk 1.1, public_base_url
GET /admin/ → 200, dashboard HTML containing "PiCloud"
POST /api/v1/admin/scripts → 201
POST .../scripts/{id}/routes (path=/greet/:name) → 201
GET /greet/alice?lang=en → 200 {"name":"alice","q":"en"}
POST conflicting route → 409 with conflicting_route body
POST /admin/foo route → 422 "reserved"
POST /api/v1/admin/routes:match → matched + params extracted
GET /unbound-path → 404 JSON
Tests:
* 40 routing unit tests (pattern + conflict + matcher tables)
* 14 executor-core unit tests (one new for ctx.request.params/
query/rest exposure)
* 32 integration tests (10 new for routing CRUD + dispatch +
conflict + reserved + specificity tie-break + match preview +
delete invalidation + /version returns public_base_url)
* default cargo test --workspace stays green; opt-in via
DATABASE_URL + --include-ignored for the integration suite
Bumps: schema 2 → 3; SDK 1.0 → 1.1; product 0.3.0 → 0.4.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Establish how versions are assigned, bumped, and checked across the
five things that actually change for users: the product itself, the
Rhai SDK, the HTTP API, the database schema, and the inter-service
wire (reserved for cluster mode). Crates ship in lockstep — drift
between picloud-shared and picloud-manager-core is fiction since
they always release together — but surfaces are versioned and
checked at their natural boundaries.
* docs/versioning.md is the authoritative reference: what gets a
version, the per-surface compatibility rules, how each surface
bump cascades to the product version (loose pre-1.0, strict
post-1.0), and the five enforcement mechanisms (lockstep at
compile time, /version at runtime, golden SDK contract tests,
migration replay, CI guardrail).
* shared::version exposes four constants — PRODUCT_VERSION (from
CARGO_PKG_VERSION), SDK_VERSION ("1.0"), API_VERSION (1),
WIRE_VERSION (1). Scripts read SDK_VERSION as ctx.sdk_version
and can feature-detect against it.
* Workspace inheritance: `[workspace.package] version = "0.2.0"`
is the single point of truth; every crate uses
`version.workspace = true`. dashboard/package.json mirrors.
* Routes move to /api/v1/* — both control plane
(/api/v1/admin/*) and data plane (/api/v1/execute/{id}).
Picloud composes them via a single `/api/v{API_VERSION}` nest,
so the next major is a copy-paste-and-bump. Caddyfile (dev and
prod) routes /api/v1/* to picloud and 404s any other /api/*
so old clients fail loudly instead of getting the SPA shell.
Dashboard client + integration tests updated.
* /healthz remains a plain "ok" string (k8s probes); /version is
the new JSON endpoint returning every surface version in one
place — product, sdk, api, schema (from
manager-core::migrations::latest_version), wire.
* Reasonable bump rationale: API path changes are breaking by
definition, so 0.1.0 → 0.2.0 (pre-1.0 license to bump minor on
any breaking change). SDK starts at 1.0 because scripts depend
on it more strictly than the product depends on its internals;
we'd rather promise SDK stability early than pull the rug.
Verified live:
* /healthz → "ok" (plain text)
* /version → {product:"0.2.0",sdk:"1.0",api:1,schema:1,wire:1}
* /api/v1/admin/scripts → 200
* /api/admin/scripts → 404 with error JSON (sunset major)
* Script can read ctx.sdk_version → "1.0"
* All 14 integration tests pass against new paths
* 11 executor-core unit tests pass (added one for sdk_version
exposure with the major.minor format invariant)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sets up the PiCloud monorepo as a Cargo workspace organised around the
three-service architecture (manager / orchestrator / executor), each
backed by a *-core library crate so the same logic powers both the MVP
all-in-one `picloud` binary and the future split-process cluster mode.
* crates/shared, executor-core, orchestrator-core, manager-core
define the library surface and trait seams between the three
services (`ExecutorClient`, `ScriptResolver`, `ScriptRepository`).
* crates/picloud is the MVP entrypoint; serves /healthz on 8080
(override via PICLOUD_BIND).
* crates/picloud-{manager,orchestrator,executor} are skeleton
binaries that keep the crate boundaries honest until cluster
mode is built out in v1.3+.
* docs/git-workflow.md defines the trunk-based workflow:
short-lived branches, Conventional Commits, separate hotfix
flow with mandatory reproduction tests.
* CLAUDE.md captures the working rules for future Claude sessions.
Workspace passes `cargo fmt`, `cargo clippy -D warnings` (with
pedantic enabled), and `cargo test --workspace`. The all-in-one
binary responds on `/healthz` and `/`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
