977abc25d034541b21164a2664d7a9ab87b06acd
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
MechaCat02
|
612074af04 |
test: give the DDL and global-count suites their own database
Turning CI on surfaced `atomic_write` as intermittently failing under load — self-inflicted. Its fault injection is `CREATE TRIGGER ... ON outbox`, which takes an ACCESS EXCLUSIVE lock on a table every other suite is concurrently inserting into. The header even claimed it "cannot affect any test running in parallel"; that was wrong — scoping the trigger by app_id bounds which ROWS it rejects, not the table lock installing it takes. Shipping that alongside "run the DB tests in CI" would have poisoned the signal. The fix is the one already used for the e2e suites: a private database per test. That logic (build a migrated template once, clone it per test via `CREATE DATABASE ... TEMPLATE`) was duplicated in `picloud/tests/common`, and this would have been a third copy — so it moves into a shared `picloud-test-support` dev-dependency crate, and `picloud/tests/common` now re-exports it. The invariant it encodes: manager-core's claim loops are global by design, so a test must not share a database with anything that runs them OR with anything that does DDL. Moved onto it: - `atomic_write` — the DDL fault injection above; its per-test cleanup is now deleted (the database is thrown away). - `outbox_reclaim` — asserts `reclaim_stale_claims` returns exactly 0, a count over the WHOLE outbox table. On the shared DB any stale row from any other suite (or a killed picloud that died holding a claim) would fail it permanently, until someone cleared the table by hand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
MechaCat02
|
8ceb1352dd |
fix(manager-core): F-P-007 concurrent queue dispatch per tick
tick_queue_arm called list_active_queue_consumers() and then iterated serially, awaiting one queue.claim(app, queue) per consumer. With N consumers and a 100ms tick the worst-case throughput was N × (claim + executor) / tick — one slow handler blocked every other queue's progress on the dispatcher's task. Replace the for-loop with `futures::stream::iter(consumers) .for_each_concurrent(QUEUE_DISPATCH_PARALLELISM, …)`. The execution gate already caps real script-concurrency to its permits, so this just removes the dispatcher-side serialization. QUEUE_DISPATCH_PARALLELISM = 32 (matches the default gate). Workspace gains `futures = 0.3` so `for_each_concurrent` is available. AUDIT.md anchor: F-P-007. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
MechaCat02
|
1f78937dd2 |
feat(v1.1.7-email-inbound): webhook receiver + email:receive trigger
Inbound email: a provider POSTs a normalized JSON message to
POST /api/v1/email-inbound/{app_id}/{trigger_id}; the public receiver
verifies the optional HMAC signature, builds a TriggerEvent::Email, and
enqueues an outbox row the dispatcher delivers like any async trigger.
Handlers see ctx.event.email = #{from,to,cc,subject,text,html,
received_at,message_id}.
- migration 0024: widen triggers.kind + outbox.source_kind CHECKs to
'email'; new email_trigger_details table.
- TriggerKind::Email, TriggerDetails::Email{has_inbound_secret},
OutboxSourceKind::Email, TriggerEvent::Email; dispatcher routes the
email row via the generic resolve_trigger path.
- Admin POST /apps/{id}/triggers/email (validate_trigger_target; module
+ cross-app rejection). inbound_secret is stored ENCRYPTED via the
master key (deviation from the brief's plaintext default; decrypted
per inbound request — see HANDBACK §7).
- Dashboard: email trigger form on the Triggers tab + webhook URL +
expected-payload help.
- 8 DB-gated e2e tests (202/401/404/422/cross-app/handler-fire) +
receiver unit tests (HMAC verify, secret round-trip, payload parse).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
8f2d2bc721 |
feat(v1.1.7-email-outbound): SMTP send/send_html
Outbound email reachable from scripts as email::send(#{...}) (plain
text) and email::send_html(#{...}) (multipart text + HTML). Backed by a
lettre SMTP relay configured from PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/
TLS/TIMEOUT_SECS; if HOST/USER/PASSWORD aren't all set the service runs
in disabled mode (every send throws NotConfigured, warned at startup).
- EmailService trait + OutboundEmail DTO (picloud-shared);
EmailServiceImpl + EmailTransport seam + lettre transport
(manager-core), wired into the Services bundle and Rhai engine.
- Capability::AppEmailSend (→ script:write); seven-scope commitment held.
- Required-field + RFC5322-ish address validation; 25 MB per-message cap
(PICLOUD_EMAIL_MAX_MESSAGE_BYTES). reply_to defaults to from.
- Per-call connection (pooling deferred to v1.2); no per-app from
validation (operator's SMTP/SPF/DKIM concern).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
10b5f655d5 |
feat(v1.1.4): outbound HTTP SDK + cron triggers
HTTP (`http::*`):
- `HttpService` trait (picloud-shared) + reqwest-backed `HttpServiceImpl`
(manager-core), wired into the `Services` bundle.
- SSRF deny-list applied to the resolved IP via a custom reqwest
`dns_resolver` (covers every redirect hop + defeats DNS rebinding) plus
a literal-IP check at URL-parse time. Scheme/port restrictions, request
+ response body caps (stream-with-cap), layered timeout. Error reason is
a CIDR category, never the IP. `PICLOUD_HTTP_ALLOW_PRIVATE` dev override
(logs a startup warning).
- Rhai bridge with three-arg split `verb(url, body, opts)` (resolves the
brief's body-vs-opts contradiction; unknown opt keys throw). Body
dispatch by type; response `#{status,headers,body,body_raw}` with JSON
auto-parse; non-2xx does not throw.
- `Capability::AppHttpRequest` → existing `script:write` scope (no new
Scope variant). `SdkCallCx` gains `script_id` (attribution + User-Agent).
Cron triggers (4th trigger kind):
- Migration 0017 widens the kind/source_kind CHECKs and adds
`cron_trigger_details`. `cron`/`chrono-tz` parse + validate 6-field
schedules and IANA timezones.
- `spawn_cron_scheduler` polls due triggers and enqueues to the universal
outbox; the dispatcher delivers them (one-line match-arm extension).
Catch-up fires exactly once per trigger per tick, not once per missed
window. `ctx.event.cron` for handlers.
- `POST /api/v1/admin/apps/{id}/triggers/cron` reuses the v1.1.3
cross-app + kind!=module target check.
- Dashboard: admin-gated Triggers tab (cron create form + list).
Follow-ups: redact module backend errors at the resolver boundary (log
original at error level); pin `rhai = "=1.24"`; CHANGELOG incl. retroactive
v1.1.3 cross-app-trigger security note. Version bumps: workspace 1.1.4,
SDK 1.5, dashboard 0.10.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
6a2971ac70 |
feat(v1.1.1-dispatcher): dispatcher loop + retry + depth limit + outbox emitter
`OutboxEventEmitter` replaces `NoopEventEmitter` in the picloud binary's `Services` bundle. KV mutations now fan out to the outbox via `TriggerRepo::list_matching_kv` — one row per matching trigger, carrying the serialized `TriggerEvent` payload + the matching trigger's retry policy. `Dispatcher` is the single tokio task that polls the outbox every 100ms, claims due rows via FOR UPDATE SKIP LOCKED (with a batch cap), and routes each to the executor. Shares the `ExecutionGate` with sync HTTP per design notes §2 — gate saturation reschedules the row instead of dropping it. Outcome handling matches design notes §3 and §4: - reply_to.is_some() (sync HTTP): never retry. Deliver via `InboxResolver`; if the receiver was dropped, write an `abandoned_executions` row. - is_dead_letter_handler == true: never retry, never DL. On failure, annotate the original DL row with `resolution = 'handler_failed'`. Stops the recursion that would otherwise re-fire a broken handler script. - Otherwise async: bump attempt_count, reschedule with exponential backoff + ±jitter; once max_attempts is reached, write a `dead_letters` row and drop from outbox. - Trigger-depth limit: `cx.trigger_depth > max_trigger_depth` skips execution entirely (log + future metric), NEVER dead-letters. Loops are not retried via the DL chain — they're terminated. `InboxResolver` trait lands in `picloud-shared` with a `NoopInboxResolver` bootstrap that flags every delivery as `Abandoned`. Commit 6 replaces the noop with the real in-process registry in `orchestrator-core`. `AdminPrincipalResolver` builds a `Principal` from a trigger's `registered_by_principal` user id so the dispatched script executes as the trigger registrant (design notes §4). Unit tests cover backoff math (exponential/linear/constant) + jitter range + ExecError → InboxFailureKind classification + the status-code table mapping. Integration tests for the full dispatcher loop need a real Postgres + executor; reviewer runs them via the manual smoke flow in the plan / HANDBACK. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
MechaCat02
|
8659a58eb2 |
feat(manager-core,picloud): api_keys_api + deactivation cascade
* auth: generate_api_key() mints pic_<base32(32 bytes)>, splits the
indexed 8-char prefix, and Argon2-hashes the body. Adds the
data-encoding workspace dep for unpadded base32.
* api_keys_api: POST /api/v1/admin/api-keys (mint, returns raw_token
exactly once), GET (caller's own, no raw), DELETE {id} (caller's
own; 404 deliberately covers both 'missing' and 'not yours').
Mint validation rejects bound keys carrying instance:* scopes (422).
* AdminsState gains the api keys repo; PATCH set_active(false) now
expires every active key for that user alongside session wipe —
Phase 3.5 deactivation symmetry.
* picloud lib wires PostgresApiKeyRepository through AuthDeps into
AdminsState + ApiKeysState; api_keys_router merges into the
guarded_admin layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
6891496589 |
feat(manager-core): admin auth gate (Phase 3a)
Closes the regression risk of the admin API and dashboard being open
to anyone reaching the bound port. Required foundation before v1.1
data-plane services land.
Per-user accounts (admin_users), Argon2id passwords, env-var bootstrap
of the first admin that becomes inert once any admin exists, opaque
32-byte session token doubling as bearer credential, 24h sliding TTL
configurable via PICLOUD_SESSION_TTL_HOURS. is_active column lets
admins be deactivated without losing audit history; last-active-admin
guard on DELETE and on PATCH that flips is_active to false (sessions
also wiped on deactivation).
require_admin middleware fronts every /api/v1/admin/* route. The data
plane (/api/v1/execute/{id}), /healthz, /version, and user routes
stay open. picloud admin reset-password <username> subcommand handles
recovery without going through HTTP.
Dashboard gains /admin/login and /admin/admins surfaces, a top-bar
user menu, and a token store with a localStorage echo so refreshes
don't sign you out. Cookie-based auth works in parallel for non-SPA
clients.
Forward compatibility: future RBAC tables (admin_roles,
admin_user_roles) join on admin_users.id; the auth middleware is the
seam where role checks slot in. Email, 2FA, passkeys, and personal
API tokens are all additive without touching admin_users.
Blueprint §11.4 updated to reflect what actually shipped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
07e2a62d98 |
feat: custom routing — bind scripts to your own URLs
Scripts can now answer at user-chosen paths (e.g. /greet, /greet/:name,
/webhooks/*), on user-chosen hosts (strict or *.example.com wildcards),
on user-chosen methods. The internal /api/v1/execute/{id} endpoint
stays as the always-available ID-based bypass.
Routing rules (decided in design with the user; see chat history):
Path kinds:
exact /greet literal
prefix /greet/* strict-subtree; stored as "/greet/";
does NOT match bare /greet (add an
exact route for that case)
param /users/:id :name captures one whole segment;
mid-segment colons are rejected;
{name} is reserved for a future SDK
Host kinds:
any no Host header constraint
strict sub.example.com literal match (case-insensitive)
wildcard *.example.com suffix match; multi-level subdomains OK
Within-kind uniqueness:
two routes of the same kind that could match the same request
conflict at config time. Algorithm (orchestrator_core::routing::
conflict):
exact: literal equality
prefix: literal equality (longer-prefix coexists; longer wins
at request time)
param: same segment count + same literals at every
literal-vs-literal position (the user's example:
:id vs :userId at same shape is a conflict)
Request-time precedence:
exact > param > prefix
among non-exact: more leading-literal segments wins
tie: param > prefix (more constrained)
within prefix: longest matching prefix wins
host bucket: strict > wildcard (longer suffix) > any; fall through
to less specific buckets when path doesn't match
Reserved path prefixes: /api/, /admin/, /healthz, /version
Routes that look invalid at config time return 422 with the precise
parse error; conflicting routes return 409 with the conflicting route
in the body (so the dashboard can render the conflict inline).
What landed:
* 0003_routes.sql — routes table (host_kind, host, host_param_name,
path_kind, path, method, script_id) with UNIQUE index on the
literal binding tuple. Schema 2 → 3.
* shared::Route / HostKind / PathKind — flat storage shape that
crosses wire boundaries cleanly.
* orchestrator_core::routing — four sub-modules, all unit-tested:
pattern.rs (16 tests) parse + validate + display
conflict.rs (12 tests) within-kind overlap predicate
matcher.rs (12 tests) runtime dispatch (specificity-aware)
table.rs Arc<RwLock<Vec<CompiledRoute>>>
shared by manager (writes) and
orchestrator (reads); atomic replace
after each admin write
* manager-core::route_admin — five new admin endpoints under
/api/v1/admin:
POST /scripts/{id}/routes create
GET /scripts/{id}/routes list per script
DELETE /routes/{route_id} delete (refreshes table)
POST /routes:check pre-flight conflict check
(powers the dashboard's
live conflict warning)
POST /routes:match synthetic URL → matched
route + extracted params
(powers the dashboard's
match-preview tool)
Stored path strings stay raw (user-typed); normalization
happens only in the in-memory CompiledRoute so re-parses are
idempotent.
* orchestrator_core::api::user_routes_router — fallback handler
mounted in picloud after the system routes. Reads Host /
method / path / query from the request, dispatches via the
table, builds an ExecRequest with params/query/rest filled,
calls the executor, writes to the log sink. 10 MiB body cap.
* executor-core::ctx (SDK 1.0 → 1.1) — adds
ctx.request.params (map of named-param captures)
ctx.request.query (parsed query string)
ctx.request.rest (suffix for prefix routes; "" otherwise)
All three are always present (empty when not applicable) so
scripts can read them unconditionally.
* picloud::build_app — now async; loads routes at startup,
populates the shared table, mounts route_admin_router under
/api/v1/admin alongside the script CRUD, and the user-routes
fallback at the app root.
* caddy/Caddyfile + Caddyfile.prod widened: anything not
/healthz, /version, /api/v1/admin/*, /api/v1/execute/*,
/api/* (404 sunset), or /admin/* (dashboard) → picloud.
* Dashboard moves to /admin/* via SvelteKit paths.base. Its
internal Caddy strips the prefix and serves with SPA fallback.
All in-app links use $app/paths. The dashboard URL is now
http://localhost:8000/admin/ — one-time break for the new
URL freedom users gained.
* PICLOUD_PUBLIC_BASE_URL env var, exposed via /version so the
dashboard renders full URLs for routes regardless of the
operator's external port / TLS setup.
* memory_limit_mb stays in the schema, still v1.3+ advisory.
Verified live through Caddy:
/version → schema 3, sdk 1.1, public_base_url
GET /admin/ → 200, dashboard HTML containing "PiCloud"
POST /api/v1/admin/scripts → 201
POST .../scripts/{id}/routes (path=/greet/:name) → 201
GET /greet/alice?lang=en → 200 {"name":"alice","q":"en"}
POST conflicting route → 409 with conflicting_route body
POST /admin/foo route → 422 "reserved"
POST /api/v1/admin/routes:match → matched + params extracted
GET /unbound-path → 404 JSON
Tests:
* 40 routing unit tests (pattern + conflict + matcher tables)
* 14 executor-core unit tests (one new for ctx.request.params/
query/rest exposure)
* 32 integration tests (10 new for routing CRUD + dispatch +
conflict + reserved + specificity tie-break + match preview +
delete invalidation + /version returns public_base_url)
* default cargo test --workspace stays green; opt-in via
DATABASE_URL + --include-ignored for the integration suite
Bumps: schema 2 → 3; SDK 1.0 → 1.1; product 0.3.0 → 0.4.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
0473d295af |
feat: versioning scheme — lockstep crates + four independent surfaces
Establish how versions are assigned, bumped, and checked across the
five things that actually change for users: the product itself, the
Rhai SDK, the HTTP API, the database schema, and the inter-service
wire (reserved for cluster mode). Crates ship in lockstep — drift
between picloud-shared and picloud-manager-core is fiction since
they always release together — but surfaces are versioned and
checked at their natural boundaries.
* docs/versioning.md is the authoritative reference: what gets a
version, the per-surface compatibility rules, how each surface
bump cascades to the product version (loose pre-1.0, strict
post-1.0), and the five enforcement mechanisms (lockstep at
compile time, /version at runtime, golden SDK contract tests,
migration replay, CI guardrail).
* shared::version exposes four constants — PRODUCT_VERSION (from
CARGO_PKG_VERSION), SDK_VERSION ("1.0"), API_VERSION (1),
WIRE_VERSION (1). Scripts read SDK_VERSION as ctx.sdk_version
and can feature-detect against it.
* Workspace inheritance: `[workspace.package] version = "0.2.0"`
is the single point of truth; every crate uses
`version.workspace = true`. dashboard/package.json mirrors.
* Routes move to /api/v1/* — both control plane
(/api/v1/admin/*) and data plane (/api/v1/execute/{id}).
Picloud composes them via a single `/api/v{API_VERSION}` nest,
so the next major is a copy-paste-and-bump. Caddyfile (dev and
prod) routes /api/v1/* to picloud and 404s any other /api/*
so old clients fail loudly instead of getting the SPA shell.
Dashboard client + integration tests updated.
* /healthz remains a plain "ok" string (k8s probes); /version is
the new JSON endpoint returning every surface version in one
place — product, sdk, api, schema (from
manager-core::migrations::latest_version), wire.
* Reasonable bump rationale: API path changes are breaking by
definition, so 0.1.0 → 0.2.0 (pre-1.0 license to bump minor on
any breaking change). SDK starts at 1.0 because scripts depend
on it more strictly than the product depends on its internals;
we'd rather promise SDK stability early than pull the rug.
Verified live:
* /healthz → "ok" (plain text)
* /version → {product:"0.2.0",sdk:"1.0",api:1,schema:1,wire:1}
* /api/v1/admin/scripts → 200
* /api/admin/scripts → 404 with error JSON (sunset major)
* Script can read ctx.sdk_version → "1.0"
* All 14 integration tests pass against new paths
* 11 executor-core unit tests pass (added one for sdk_version
exposure with the major.minor format invariant)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
4f044e7b81 |
feat: end-to-end script CRUD + Rhai execution
Brings the MVP feature set online: upload a Rhai script, get an HTTP
endpoint that runs it sandboxed in-process, list/update/delete it, and
have invalid sources rejected at upload time. Verified live through
Caddy with a full lifecycle (`create → list → get → execute → update
→ delete`) plus error paths (syntax error, duplicate name, deleted).
Layout — every concern lands behind the trait seam its layer owns, so
cluster-mode in v1.3+ is a swap of two impls, not a rewrite:
* shared::ScriptValidator — manager calls into validation without
a hard dep on executor-core; executor-core impls the trait on
`Engine`. Pinned in shared so neither crate has to know about
the other.
* executor-core::Engine — real Rhai engine: sandbox limits (max
operations / string size / map size / call depth), disabled
`print`, blocked `import` (DummyModuleResolver), `log::trace
/info/warn/error` registered as a static module with shared
log-capture buffer (no `log::debug` because `debug` is a Rhai
reserved keyword — `log::trace` covers the same need).
- `ctx` is pushed as a Scope constant exposing
execution_id, script_id, script_name, request_id,
invocation_type, request.{path,headers,body}.
- Response convention: a Map with `statusCode` is the
structured shape (`{statusCode, headers?, body}`); any
other return value is a 200 with the value as the body.
- Engine::execute is now synchronous (pure compute); the
async wrapper + wall-clock timeout live in
LocalExecutorClient, which spawns_blocking and applies a
300s hard ceiling regardless of per-script config.
- 10 unit tests cover validate, exec, structured response,
ctx exposure, log capture, op-budget enforcement, runtime
errors, blocked imports, JSON round-tripping.
* manager-core::repo — full sqlx CRUD over the `scripts` table,
with proper unique-violation handling for duplicate names.
Embedded migrations via `sqlx::migrate!` (one initial
`0001_init.sql` for pgcrypto + scripts + execution_logs).
* manager-core::api — `admin_router` mounts `/scripts` and
`/scripts/{id}`. Create + Update validate source through the
injected `ScriptValidator` before persistence. Returns proper
422/409/404 status codes via `ApiError::IntoResponse`.
* orchestrator-core::api — `data_plane_router` mounts
`/execute/{id}`: resolves the script through `ScriptResolver`,
constructs the `ExecRequest` from headers+body, awaits
`ExecutorClient::execute(..., timeout)`, translates the
`ExecResponse` to an axum `Response` with header passthrough.
Maps `ExecError` variants to 422/504/502/507.
* picloud all-in-one — opens the pool, runs migrations, builds
one engine, nests both routers under `/api/admin` and `/api`,
enables structured JSON tracing and graceful shutdown on
SIGTERM. Single `PostgresScriptRepository` Arc is shared by
the admin router (writes) and the resolver (reads).
Other changes:
* Workspace axum bump 0.7 → 0.8 for the `{id}` path syntax
matching the route definitions.
* Workspace clippy: allow `needless_pass_by_value` and
`boxed_local` to keep API ergonomics over pedantic noise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
MechaCat02
|
b8b544816d |
chore: initial scaffold — workspace, docs, blueprint
Sets up the PiCloud monorepo as a Cargo workspace organised around the
three-service architecture (manager / orchestrator / executor), each
backed by a *-core library crate so the same logic powers both the MVP
all-in-one `picloud` binary and the future split-process cluster mode.
* crates/shared, executor-core, orchestrator-core, manager-core
define the library surface and trait seams between the three
services (`ExecutorClient`, `ScriptResolver`, `ScriptRepository`).
* crates/picloud is the MVP entrypoint; serves /healthz on 8080
(override via PICLOUD_BIND).
* crates/picloud-{manager,orchestrator,executor} are skeleton
binaries that keep the crate boundaries honest until cluster
mode is built out in v1.3+.
* docs/git-workflow.md defines the trunk-based workflow:
short-lived branches, Conventional Commits, separate hotfix
flow with mandatory reproduction tests.
* CLAUDE.md captures the working rules for future Claude sessions.
Workspace passes `cargo fmt`, `cargo clippy -D warnings` (with
pedantic enabled), and `cargo test --workspace`. The all-in-one
binary responds on `/healthz` and `/`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|