Add a server-side, atomic, declarative reconcile loop for a single app —
the foundation of the project-tool design. Developers describe an app's
scripts, routes, triggers, and secret-names in `picloud.toml`, then
`pic pull / plan / apply [--prune]` to converge live state to the manifest.
Server (manager-core):
- apply_service: a pure diff engine (compute_diff) shared by plan and
apply, plus an ApplyService that composes the existing per-repo writes
into ONE Postgres transaction. Identity keys mirror the DB UNIQUE
constraints (script=lower(name); route=(method,host_kind,host,
path_kind,path); trigger=per-kind semantic tuple; secret=name).
Apply takes a per-app advisory lock, recomputes the diff in-tx, applies
scripts -> routes -> triggers, prunes dependents-first, commits, then
refreshes the route table once post-commit.
- apply_api: POST /apps/{id}/plan (AppRead) and /apps/{id}/apply.
Apply requires the per-kind write caps the bundle exercises (all three
when --prune), plus AppSecretsRead when it binds an email trigger.
- tx-accepting repo siblings (insert/update/delete *_tx) so the existing
create/update/delete delegate to one SQL definition each.
- email triggers reference an inbound secret by NAME; the value is
resolved, decrypted (AAD-bound), and re-sealed server-side at apply —
it never travels in the manifest.
CLI (picloud-cli):
- manifest.rs (picloud.toml model), client plan/apply, and the pull/plan/
apply commands. pull rejects filesystem-unsafe script names up front.
Safety properties enforced and tested:
- idempotent: a freshly-pulled manifest re-applies as all-NoOp.
- atomic: a mid-bundle failure rolls back with nothing written.
- routes delete-before-insert so a freed binding is reusable in one apply.
- queue one-consumer invariant held inside the shared tx.
- email triggers are never pruned, and a script that still owns an
email/dead-letter trigger can't be pruned (the FK cascade would destroy
the sealed secret) — refused with a pointer to `pic triggers rm`.
- plan and apply agree on unset email-secret references.
No migration: the existing schema's UNIQUE constraints serve as identity
keys. Groups, env-scoping, and the `enabled` toggle are later milestones.
Tested: manager-core lib (360) + CLI bins (27) + 8 project-tool journeys
(pull/plan/apply/prune/email+queue), all green; clippy -D warnings clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cargo fmt regroups, plus three Phase-B clippy fixes:
- F-P-004: #[allow(clippy::type_complexity)] on the invoke_ast_cache Mutex.
- F-P-008: hoist `futures::stream::{self, TryStreamExt}` to the top-of-
file imports to satisfy clippy::items_after_statements.
- F-P-005: rename `_legacy_offset` to `legacy_offset` + #[allow(dead_code)]
to satisfy clippy's "field is pub but `_`-prefixed" check.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TriggerRepo::list_for_app selected parent rows then called
hydrate_one(...) per row serially — one SELECT against the kind-
specific details table per trigger. For N triggers on an app, that's
N+1 queries serialized on every dashboard `GET /apps/{id}/triggers`.
This commit shrinks the wall-clock cost via buffered concurrency
(`try_buffered(8)`): each row's details fetch still runs but they run
in parallel up to a small fan-out cap. Cuts dashboard load time from
sum(N latencies) to max(N latencies) / 8.
Collapsing to a single per-kind LEFT JOIN (the audit's preferred shape)
would cut the query count to ~7 — large enough to defer to v1.2 as
documented in the inline comment.
AUDIT.md anchor: F-P-008 (partial: wall-clock only; round-trip count
deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends Dispatcher with the v1.1.9 queue path. The queue table IS the
outbox for queue semantics, so the dispatcher polls queue_messages
directly via FOR UPDATE SKIP LOCKED — no outbox indirection.
Per tick (every 100ms):
1) outbox arm — unchanged
2) queue arm: list_active_queue_consumers() → per (app_id, queue_name)
attempt one claim. Bounded by registered-consumer count so one busy
queue can't starve others.
Per claimed message:
- Build TriggerEvent::Queue + ExecRequest (executes as the trigger's
registering principal, matching design notes §4)
- dispatch through executor.execute_with_identity (reuses AST cache)
- success → queue.ack(id, claim_token) — DELETE WHERE id AND token
- throw + attempt < max_attempts → queue.nack(...) — clear claim, set
deliver_after = NOW() + compute_backoff(attempt, backoff, base_ms, jitter)
- throw + exhausted → queue.dead_letter(...) atomic move to dead_letters
+ DELETE in one transaction. fan_out_dead_letter on the outbox arm
fires registered dead_letter handlers off the new row without changes.
Visibility-timeout reclaim task: separate tokio::spawn ticking on
queue_reclaim_interval_ms (default 30000). UPDATE clears claim_token /
claimed_at on rows whose claimed_at exceeds the per-queue
visibility_timeout_secs (joined from queue_trigger_details). A crashed
consumer thus loses its lease and the message becomes claimable again.
Dispatcher gains queue: Arc<dyn QueueRepo>; picloud/lib.rs threads
queue_repo.clone() into the construction.
ActiveQueueConsumer extended with app_id so the queue claim can be
performed without a follow-up trigger lookup; list_active_queue_consumers
SQL extended to SELECT t.app_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inbound email: a provider POSTs a normalized JSON message to
POST /api/v1/email-inbound/{app_id}/{trigger_id}; the public receiver
verifies the optional HMAC signature, builds a TriggerEvent::Email, and
enqueues an outbox row the dispatcher delivers like any async trigger.
Handlers see ctx.event.email = #{from,to,cc,subject,text,html,
received_at,message_id}.
- migration 0024: widen triggers.kind + outbox.source_kind CHECKs to
'email'; new email_trigger_details table.
- TriggerKind::Email, TriggerDetails::Email{has_inbound_secret},
OutboxSourceKind::Email, TriggerEvent::Email; dispatcher routes the
email row via the generic resolve_trigger path.
- Admin POST /apps/{id}/triggers/email (validate_trigger_target; module
+ cross-app rejection). inbound_secret is stored ENCRYPTED via the
master key (deviation from the brief's plaintext default; decrypted
per inbound request — see HANDBACK §7).
- Dashboard: email trigger form on the Triggers tab + webhook URL +
expected-payload help.
- 8 DB-gated e2e tests (202/401/404/422/cross-app/handler-fire) +
receiver unit tests (HMAC verify, secret round-trip, payload parse).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Durable pub/sub through the universal outbox — the sixth trigger kind.
- `pubsub::publish_durable(topic, message)` Rhai SDK (no handle; topics
ARE the grouping unit). Message JSON-encoded; Blobs base64 at any
depth.
- `PubsubService` trait in picloud-shared with the topic matcher +
validator (exact / `<prefix>.*` / `*`; mid-pattern wildcards
rejected). `PostgresPubsubRepo` + `PubsubServiceImpl` in manager-core.
- Publish-time fan-out: one outbox row per matching enabled pubsub
trigger, all in ONE transaction (no half-fan-out on crash). No
matching trigger → publish succeeds silently, zero rows.
- `pubsub:*` trigger kind via Layout-E (0020: widen both CHECKs +
pubsub_trigger_details + partial index), TriggerEvent::Pubsub +
ctx.event.pubsub, dispatcher arm, admin endpoint POST /triggers/pubsub
(validates topic pattern + reuses validate_trigger_target).
- AppPubsubPublish capability → script:write (seven-scope held).
- Dashboard Pub/Sub trigger form on the Triggers tab + list rendering.
publish_ephemeral stays deferred to v1.2. ~18 new tests (service
in-memory incl. transactional-rollback, shared matcher, bridge
encoding). No DB required for the suite.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Filesystem-backed blob storage as the fifth concrete trigger kind.
- `files::collection(c).{create,head,get,update,delete,list}` Rhai SDK
(blob in/out; metadata maps; missing-field throws naming the field).
- `FilesService` trait in picloud-shared; `FsFilesRepo` (atomic
write: temp→fsync→rename→fsync-dir→DB; single-pass SHA-256;
checksum-verified reads → Corrupted) + `FilesServiceImpl` in
manager-core. Metadata in Postgres (0018), bytes on disk under
PICLOUD_FILES_ROOT with 0o700 shard dirs.
- `files:*` trigger kind via the Layout-E pattern (0019: widen both
CHECKs + files_trigger_details), TriggerEvent::Files (metadata only,
no bytes), emit_files fan-out, dispatcher arm, admin endpoint
POST /triggers/files (reuses validate_trigger_target).
- AppFilesRead/AppFilesWrite capabilities → script:read/script:write
(seven-scope commitment held). AppPubsubPublish reserved for v1.1.6.
- Admin files API (list + delete) + dashboard Files view per app.
Cross-app isolation keyed on cx.app_id at every layer. ~45 new tests
(service in-memory, fs tempdir, bridge integration). No DB required
for the suite. publish_ephemeral and the orphan sweep stay deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
HTTP (`http::*`):
- `HttpService` trait (picloud-shared) + reqwest-backed `HttpServiceImpl`
(manager-core), wired into the `Services` bundle.
- SSRF deny-list applied to the resolved IP via a custom reqwest
`dns_resolver` (covers every redirect hop + defeats DNS rebinding) plus
a literal-IP check at URL-parse time. Scheme/port restrictions, request
+ response body caps (stream-with-cap), layered timeout. Error reason is
a CIDR category, never the IP. `PICLOUD_HTTP_ALLOW_PRIVATE` dev override
(logs a startup warning).
- Rhai bridge with three-arg split `verb(url, body, opts)` (resolves the
brief's body-vs-opts contradiction; unknown opt keys throw). Body
dispatch by type; response `#{status,headers,body,body_raw}` with JSON
auto-parse; non-2xx does not throw.
- `Capability::AppHttpRequest` → existing `script:write` scope (no new
Scope variant). `SdkCallCx` gains `script_id` (attribution + User-Agent).
Cron triggers (4th trigger kind):
- Migration 0017 widens the kind/source_kind CHECKs and adds
`cron_trigger_details`. `cron`/`chrono-tz` parse + validate 6-field
schedules and IANA timezones.
- `spawn_cron_scheduler` polls due triggers and enqueues to the universal
outbox; the dispatcher delivers them (one-line match-arm extension).
Catch-up fires exactly once per trigger per tick, not once per missed
window. `ctx.event.cron` for handlers.
- `POST /api/v1/admin/apps/{id}/triggers/cron` reuses the v1.1.3
cross-app + kind!=module target check.
- Dashboard: admin-gated Triggers tab (cron create form + list).
Follow-ups: redact module backend errors at the resolver boundary (log
original at error level); pin `rhai = "=1.24"`; CHANGELOG incl. retroactive
v1.1.3 cross-app-trigger security note. Version bumps: workspace 1.1.4,
SDK 1.5, dashboard 0.10.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docs trigger kind hangs off the same Layout-E shape that v1.1.1
established for KV: a parent triggers row + a docs_trigger_details
row (collection_glob TEXT + ops TEXT[]) with the empty-array =
any-op semantic preserved.
- trigger_repo.rs adds TriggerKind::Docs + TriggerDetails::Docs +
CreateDocsTrigger + DocsTriggerMatch + PostgresTriggerRepo
implementations of create_docs_trigger and list_matching_docs.
list_matching_docs mirrors KV's Rust-side filter (does NOT push
ops membership into SQL — that would exclude empty-ops rows).
- outbox_repo.rs adds OutboxSourceKind::Docs to the enum + wire form.
- dispatcher.rs's generic Kv | DeadLetter routing arm extends to
Kv | DeadLetter | Docs. No kind-specific logic needed — the
resolve_trigger + build_exec_request path is already abstract.
- outbox_event_emitter.rs gains a "docs" arm in the emit match plus
emit_docs which builds TriggerEvent::Docs (carrying data +
prev_data) and fans out across matching triggers.
- triggers_api.rs adds CreateDocsTriggerRequest + create_docs_trigger
+ the POST /api/v1/admin/apps/{id}/triggers/docs route, all
guarded by Capability::AppManageTriggers (same as KV).
3 new triggers_api unit tests covering happy path, empty-glob
rejection, and capability denial. All existing trigger-related
tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrations 0008-0011 lay down the triggers framework's storage:
- `triggers` + `kv_trigger_details` + `dead_letter_trigger_details`
(Layout E, design notes §2). Parent table carries common columns
including `registered_by_principal` — the dispatcher uses this to
run the trigger as the user that registered it (design notes §4).
- `outbox`: universal async dispatch substrate. KV/cron/pubsub/queue/
email/dead-letter all write rows in the same shape; the dispatcher
claims due rows via FOR UPDATE SKIP LOCKED. `reply_to` is the
NATS-style inbox id for sync HTTP (commit 6) — its presence flags
"don't retry" per the design.
- `dead_letters`: exact schema from design notes §4 with the four-
value `resolution` CHECK constraint (`replayed | ignored |
handled_by_script | handler_failed`) and partial index on
unresolved rows for the dashboard badge.
- `abandoned_executions`: forensic table for the dispatcher's
"tried to resolve a dropped inbox" edge case (design notes §3 #9).
Repo surfaces with Postgres impls behind traits so unit tests can
swap in-memory backings:
- `TriggerRepo` — CRUD + the `list_matching_kv` /
`list_matching_dead_letter` hot paths the dispatcher uses.
Includes a `collection_matches` helper that handles `*`, `prefix:*`,
and exact-name globs.
- `OutboxRepo` — insert + claim-due + delete + reschedule.
- `DeadLetterRepo` — insert + get + list + unresolved-count +
resolve + GC.
- `AbandonedRepo` — insert + GC.
`TriggerConfig::from_env` (new module) follows the existing
`SandboxCeiling` env-loading pattern for `PICLOUD_MAX_TRIGGER_DEPTH`,
`PICLOUD_TRIGGER_RETRY_*`, `PICLOUD_DEAD_LETTER_RETENTION_DAYS`, and
`PICLOUD_ABANDONED_EXECUTIONS_RETENTION_DAYS`.
`Capability::AppManageTriggers(AppId)` and `AppDeadLetterManage(AppId)`
join the enum. Both map onto the existing `Scope::AppAdmin` per the
seven-scope commitment; `role_satisfies` grants them at the
`AppAdmin` per-app role.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
