21 Commits

Author SHA1 Message Date
MechaCat02
7040f0df83 docs(handoff): machine-switch handoff report 2026-06-05
Session summary, branch inventory, push instructions, v1.1.8
follow-ups, and pickup-on-new-machine smoke commands. Main is at
v1.1.7 (5cbb6ca); seven minor releases shipped this session via
the dispatch-and-review workflow.

Read this first on the new machine.
2026-06-05 07:12:06 +02:00
MechaCat02
5cbb6ca427 docs(v1.1.7): reviewer audit report — APPROVE verdict
Some checks failed
CI / Dashboard — check (push) Successful in 9m53s
CI / Rust — fmt, clippy, test (push) Failing after 13m11s
Audit of feat/v1.1.7-secrets-email against the v1.1.7 dispatch
prompt. All gates green; awk-summed 617 tests pass (matches HANDBACK
§8 exactly — the v1.1.6 retro discipline lesson landed).

Three flagged items reviewed and resolved:

- Brief-internal contradiction on TriggerEvent::DeadLetter field
  names: agent built from the real variant, flagged not reinterpreted
  (the v1.1.6 retro discipline working again).

- inbound_secret stored encrypted (user-approved deviation): correct
  call. Encryption-at-rest of credentials is the right default; the
  brief's plaintext recommendation was a premature optimization. The
  microsecond decrypt is negligible vs the HMAC + DB round-trip
  already on the path.

- Latent finding: clippy --all-targets didn't pass at v1.1.6 HEAD.
  Four pre-existing warnings the v1.1.6 audit missed (likely due to
  cargo incremental cache interaction). Agent fixed in dedicated
  commit. Real audit oversight in my v1.1.6 review; discipline fix
  folded into v1.1.8 prompt recommendations.

The v1.1.1 dead-letter handler bug (silently broken across six
releases) is finally wired. Two-phase realtime key migration ships
with phase-2 (plaintext column drop) deferred to v1.1.8.
2026-06-04 22:50:09 +02:00
MechaCat02
3cfb795206 docs(v1.1.7): handback report
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:46:33 +02:00
MechaCat02
a7d3dad129 chore(v1.1.7): re-bless schema snapshot for secrets + email migrations
Captures migrations 0023 (secrets), 0024 (email_trigger_details +
widened kind/source CHECKs), 0025 (app_secrets encrypted columns +
NULL-able plaintext). Diff is exactly the new tables/columns/constraints.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:39:24 +02:00
MechaCat02
2ea47eb05a chore(v1.1.7): fix clippy --all-targets warnings
Clears the workspace under `clippy --all-targets --all-features
-D warnings`. Four were pre-existing at v1.1.6 HEAD (latent finding,
see HANDBACK): double_must_use on realtime_router, map_unwrap_or in
pubsub_service, redundant_closure in topic_repo, needless_raw_string in
a subscriber-token test. The rest are v1.1.7 nits (needless_borrow +
semicolon in the dead-letter / realtime-migration code).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:38:11 +02:00
MechaCat02
b35585195b chore(v1.1.7): version bumps + CHANGELOG
- workspace 1.1.6 → 1.1.7
- SDK schema 1.7 → 1.8 (SecretsService, EmailService, TriggerEvent::Email)
- dashboard 0.12.0 → 0.13.0
- CHANGELOG entry: secrets, outbound email, inbound email, retroactive
  dead_letter fix note, realtime-key encryption migration (+ v1.1.8 drop)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:35:07 +02:00
MechaCat02
fffcdf6169 feat(v1.1.7-realtime-migration): encrypt signing keys at rest
Two-phase encryption of app_secrets.realtime_signing_key:
- migration 0025 adds NULL-able realtime_signing_key_encrypted +
  _nonce columns and drops NOT NULL on the plaintext column.
- PostgresAppSecretsRepo now holds the master key: new keys are written
  encrypted-only; reads prefer the encrypted columns and fall back to
  plaintext during the compat window.
- Startup task migrate_plaintext_keys() encrypts any pre-existing
  plaintext rows (plaintext left in place for rollback safety).
- v1.1.8 will drop the plaintext column.

The RealtimeAuthority read path is unchanged (it calls signing_key),
so SSE keeps working throughout. Unit tests cover the
encrypted-wins / plaintext-fallback / post-drop precedence.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:33:23 +02:00
MechaCat02
02335a8132 fix(v1.1.7-dead-letter): wire dispatcher → list_matching_dead_letter
dead_letter triggers have been registerable since v1.1.1 but their
handlers never fired: dispatcher::handle_failure wrote the dead_letters
row and stopped — list_matching_dead_letter had no production caller.
Any deploy v1.1.1–v1.1.6 with dead_letter triggers had silently
non-functional handlers.

The fix: after the dead-letter row is inserted on retry exhaustion, fan
out to matching dead_letter triggers (filtered by source / originating
trigger_id / script_id) and enqueue one outbox row per match carrying a
real-shape TriggerEvent::DeadLetter (the §6 brief field names were stale
— used the actual variant: dead_letter_id, original: Box<TriggerEvent>,
attempts, last_error, trigger_id, script_id, first/last_attempt_at).
The recursion-stop (a handler's own failure isn't re-dead-lettered)
is upheld by the existing is_dead_letter_handler short-circuit.

Tests (DB-gated): handler actually fires with the nested original event;
existing row-create test now also asserts handler-fire; source_filter
excludes non-matching; failing handler does not recurse.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:30:25 +02:00
MechaCat02
1f78937dd2 feat(v1.1.7-email-inbound): webhook receiver + email:receive trigger
Inbound email: a provider POSTs a normalized JSON message to
POST /api/v1/email-inbound/{app_id}/{trigger_id}; the public receiver
verifies the optional HMAC signature, builds a TriggerEvent::Email, and
enqueues an outbox row the dispatcher delivers like any async trigger.
Handlers see ctx.event.email = #{from,to,cc,subject,text,html,
received_at,message_id}.

- migration 0024: widen triggers.kind + outbox.source_kind CHECKs to
  'email'; new email_trigger_details table.
- TriggerKind::Email, TriggerDetails::Email{has_inbound_secret},
  OutboxSourceKind::Email, TriggerEvent::Email; dispatcher routes the
  email row via the generic resolve_trigger path.
- Admin POST /apps/{id}/triggers/email (validate_trigger_target; module
  + cross-app rejection). inbound_secret is stored ENCRYPTED via the
  master key (deviation from the brief's plaintext default; decrypted
  per inbound request — see HANDBACK §7).
- Dashboard: email trigger form on the Triggers tab + webhook URL +
  expected-payload help.
- 8 DB-gated e2e tests (202/401/404/422/cross-app/handler-fire) +
  receiver unit tests (HMAC verify, secret round-trip, payload parse).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 22:24:35 +02:00
MechaCat02
8f2d2bc721 feat(v1.1.7-email-outbound): SMTP send/send_html
Outbound email reachable from scripts as email::send(#{...}) (plain
text) and email::send_html(#{...}) (multipart text + HTML). Backed by a
lettre SMTP relay configured from PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/
TLS/TIMEOUT_SECS; if HOST/USER/PASSWORD aren't all set the service runs
in disabled mode (every send throws NotConfigured, warned at startup).

- EmailService trait + OutboundEmail DTO (picloud-shared);
  EmailServiceImpl + EmailTransport seam + lettre transport
  (manager-core), wired into the Services bundle and Rhai engine.
- Capability::AppEmailSend (→ script:write); seven-scope commitment held.
- Required-field + RFC5322-ish address validation; 25 MB per-message cap
  (PICLOUD_EMAIL_MAX_MESSAGE_BYTES). reply_to defaults to from.
- Per-call connection (pooling deferred to v1.2); no per-app from
  validation (operator's SMTP/SPF/DKIM concern).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 21:47:46 +02:00
MechaCat02
2d11090d1a feat(v1.1.7-secrets): secrets SDK + table + admin API + dashboard
Encrypted per-app secrets, reachable from scripts as
secrets::{get,set,delete,list}(name) and managed from the dashboard
Secrets tab. Values are AES-256-GCM-sealed with the process master key
(picloud_shared::crypto) before they touch Postgres; the repo only ever
sees ciphertext + nonce. JSON round-trip preserves Rhai types.

- migration 0023_secrets.sql (PRIMARY KEY (app_id, name)).
- SecretsService trait (picloud-shared) + SecretsServiceImpl + repo
  (manager-core), wired into the Services bundle and Rhai engine.
- Capability::AppSecretsRead/Write (→ script:read / script:write); no
  new Scope variants (seven-scope commitment).
- Admin API GET/POST/DELETE /apps/{id}/secrets (list returns names +
  updated_at, never values).
- build_app now takes a MasterKey, sourced from PICLOUD_SECRET_KEY in
  main.rs; test callers pass a fixed test key.
- 64 KB value cap (PICLOUD_SECRET_MAX_VALUE_BYTES); no ServiceEvent
  emission (secret writes don't fire triggers, by design).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 21:37:17 +02:00
MechaCat02
dc2e4fa01f feat(v1.1.7-crypto): master-key infra + encryption helpers
Add picloud_shared::crypto: AES-256-GCM encrypt/decrypt envelope
(12-byte CSPRNG nonce, 128-bit tag appended to ciphertext) plus a
MasterKey sourced from PICLOUD_SECRET_KEY (base64 of 32 bytes), with
a deterministic dev-key fallback gated on PICLOUD_DEV_MODE=true. Unset
key without dev mode is fatal. Key rotation is out of v1.1.7 scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 20:50:22 +02:00
MechaCat02
64ad978a89 docs(v1.1.6): reviewer audit report — APPROVE verdict
Audit of feat/v1.1.6-realtime-client against the v1.1.6 dispatch
prompt. All gates green; clippy clean; ~550 tests pass (HANDBACK §8
claimed 482 — minor count-discrepancy flagged for retro, not a
blocker).

Both flagged items verified and resolved:

- §4-vs-§8 publish-ordering contradiction in the brief: the agent
  picked §8 (broadcast AFTER outbox commit) and explicitly flagged
  the contradiction. Confirmed correct — §8's ordering protects
  against subscribers being told an event happened that subsequently
  failed to durably commit. §4's broadcast-first phrasing was a
  latency-optimization aside; §8 is the dedicated numbered spec.
  The v1.1.4 retro discipline lesson (flag-don't-reinterpret) worked.

- Latent finding: dead_letter trigger handlers never fire — verified
  via grep. list_matching_dead_letter has no production caller; the
  bug predates v1.1.6 (shipped silent since v1.1.1). Correctly
  out-of-scope for v1.1.6. The dispatcher e2e test for dead_letter
  asserts the wired behavior (row created) with inline docs explaining
  why it's not asserting handler-fire. Fix folded into v1.1.7 prompt
  recommendations along with a retroactive CHANGELOG note.

Three v1.1.5 follow-ups landed: six dispatcher e2e tests gated on
DATABASE_URL, empty-blob relaxed, orphan tmp-sweeper. HMAC signing
key persisted to app_secrets table (recommended path); streaming-
fetch SSE in the client lib unlocks bearer-header auth + 401
detection + Last-Event-ID resume.
2026-06-04 20:25:04 +02:00
MechaCat02
f5a3f92484 docs(v1.1.6): handback report
Scope coverage, realtime + client-lib notes, §8 attestation (482
workspace tests; e2e 6/6 + schema snapshot verified on a real DB),
every prompt-default deviation, and the latent dead_letter-trigger
fan-out finding + §4-vs-§8 ordering question for the reviewer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 20:19:14 +02:00
MechaCat02
b1dddb9cb9 feat(v1.1.6-client): @picloud/client TypeScript package
First frontend library (v1.0.0), co-shipped with realtime. Hybrid
model — no direct service access from the browser.

- endpoint<Req,Res>(path).get()/.post() — typed HTTP, auth-token
  injection, structured errors, optional zod/valibot validate adapter.
- subscribe(topic, cb, {token, onTokenExpired}) — streaming-fetch SSE
  with exponential-backoff reconnect, 401 token refresh, Last-Event-ID
  resume.
- auth.login/logout/token over dev-defined endpoints.
- React (useTopic/useEndpoint + PicloudProvider) and Svelte
  (topicStore/endpointStore) subpath exports.

Build: tsup (ESM+CJS+.d.ts); tests: vitest (15); lint: tsc --noEmit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 20:19:07 +02:00
MechaCat02
fcbcc576a2 feat(v1.1.6): realtime channels + v1.1.5 follow-ups + version bumps
Server-side realtime SSE on per-app pub/sub topics, plus the three
v1.1.5 follow-ups and the version bumps.

Realtime:
- topics registry (0021) + admin endpoints + Capability::AppTopicManage
  (-> app:admin; no new scope).
- GET /realtime/topics/{topic} SSE endpoint (orchestrator-core data
  plane): Host -> app, RealtimeAuthority gate (404 missing/internal,
  401 bad/absent token), broadcast::Receiver stream + heartbeat.
- RealtimeBroadcaster / RealtimeEvent / RealtimeAuthority traits
  (picloud-shared); InProcessBroadcaster + GC (orchestrator-core);
  DB-backed RealtimeAuthorityImpl (manager-core). Publish path fans out
  to in-process subscribers after the durable outbox commit (best-effort,
  panic-isolated).
- HMAC subscriber tokens (subscriber_token.rs) + app_secrets table (0022)
  + pubsub::subscriber_token SDK (schema 1.6 -> 1.7). TTL clamp + env
  overrides.
- Dashboard Topics tab (register/list/edit/delete, prominent external
  badge, flip confirmation).

v1.1.5 follow-ups:
- Empty blobs accepted (NewFile/FileUpdate::validate) + round-trip test.
- Orphan *.tmp.* sweeper (spawn_files_orphan_sweep).
- Dispatcher e2e tests, one per trigger kind (DATABASE_URL-gated).

Versions: workspace 1.1.6, SDK 1.7, dashboard 0.12.0. Schema-snapshot
golden re-blessed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 20:18:50 +02:00
MechaCat02
d064681c49 docs(v1.1.5): reviewer audit report — APPROVE verdict
Audit of feat/v1.1.5-files-pubsub against the v1.1.5 dispatch prompt.
All gates green on HEAD; 491 tests pass (+64 new), 139 ignored.

Atomic write protocol audited line-by-line: single-pass SHA-256,
temp→fsync→rename→fsync-dir→DB sequence as specified, unique pid+
counter temp suffix, path-traversal defense at SDK boundary and repo.
Pub/sub fan-out is correctly transactional (single tx begin+commit;
one outbox row per matching subscriber; trigger_depth saturating-
bumped). Topic pattern matcher rejects every shape the brief called
out (*.created, **, a.*.b, user.*x, *user, empty).

Three flagged open questions resolved: orphan-sweep deferred (matches
planning decision), test count 63 vs 70 (defensible — gap is the
dispatcher e2e test, which is already covered for kv/docs/cron via
the shared dispatcher path), empty-blob = missing-data (defensible
interpretation, relaxable later).

First CI workflow added; schema_snapshot un-ignored with DATABASE_URL-
absent skip path.
2026-06-03 21:52:34 +02:00
MechaCat02
9492c18d0e docs(v1.1.5): handback report
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 21:47:55 +02:00
MechaCat02
4595db7a7a chore(v1.1.5): version bumps, CI workflow, schema-snapshot un-ignore
- Workspace 1.1.4 → 1.1.5; SDK 1.5 → 1.6; dashboard 0.10.0 → 0.11.0.
- CHANGELOG v1.1.5 entry; CLAUDE.md runtime-config table gains
  PICLOUD_FILES_ROOT + PICLOUD_FILES_MAX_FILE_SIZE_BYTES.
- schema_snapshot test: drop #[ignore] + #[sqlx::test]; run against
  DATABASE_URL when set, skip cleanly when absent. Re-blessed golden
  picks up files / files_trigger_details / pubsub_trigger_details, the
  two widened CHECKs, and the pubsub partial index.
- First CI workflow (.github/workflows/ci.yml): postgres:15 service +
  fmt + clippy + cargo test --workspace; separate dashboard check job.
- Add files/pubsub admin-trigger reject-coverage tests (module +
  cross-app + bad-pattern), mirroring the v1.1.3 regression set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 21:44:12 +02:00
MechaCat02
834c787ee1 feat(v1.1.5): pubsub::publish_durable SDK + pubsub:* triggers
Durable pub/sub through the universal outbox — the sixth trigger kind.

- `pubsub::publish_durable(topic, message)` Rhai SDK (no handle; topics
  ARE the grouping unit). Message JSON-encoded; Blobs base64 at any
  depth.
- `PubsubService` trait in picloud-shared with the topic matcher +
  validator (exact / `<prefix>.*` / `*`; mid-pattern wildcards
  rejected). `PostgresPubsubRepo` + `PubsubServiceImpl` in manager-core.
- Publish-time fan-out: one outbox row per matching enabled pubsub
  trigger, all in ONE transaction (no half-fan-out on crash). No
  matching trigger → publish succeeds silently, zero rows.
- `pubsub:*` trigger kind via Layout-E (0020: widen both CHECKs +
  pubsub_trigger_details + partial index), TriggerEvent::Pubsub +
  ctx.event.pubsub, dispatcher arm, admin endpoint POST /triggers/pubsub
  (validates topic pattern + reuses validate_trigger_target).
- AppPubsubPublish capability → script:write (seven-scope held).
- Dashboard Pub/Sub trigger form on the Triggers tab + list rendering.

publish_ephemeral stays deferred to v1.2. ~18 new tests (service
in-memory incl. transactional-rollback, shared matcher, bridge
encoding). No DB required for the suite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 21:37:06 +02:00
MechaCat02
6e132b6ee0 feat(v1.1.5): files SDK + files:* triggers
Filesystem-backed blob storage as the fifth concrete trigger kind.

- `files::collection(c).{create,head,get,update,delete,list}` Rhai SDK
  (blob in/out; metadata maps; missing-field throws naming the field).
- `FilesService` trait in picloud-shared; `FsFilesRepo` (atomic
  write: temp→fsync→rename→fsync-dir→DB; single-pass SHA-256;
  checksum-verified reads → Corrupted) + `FilesServiceImpl` in
  manager-core. Metadata in Postgres (0018), bytes on disk under
  PICLOUD_FILES_ROOT with 0o700 shard dirs.
- `files:*` trigger kind via the Layout-E pattern (0019: widen both
  CHECKs + files_trigger_details), TriggerEvent::Files (metadata only,
  no bytes), emit_files fan-out, dispatcher arm, admin endpoint
  POST /triggers/files (reuses validate_trigger_target).
- AppFilesRead/AppFilesWrite capabilities → script:read/script:write
  (seven-scope commitment held). AppPubsubPublish reserved for v1.1.6.
- Admin files API (list + delete) + dashboard Files view per app.

Cross-app isolation keyed on cx.app_id at every layer. ~45 new tests
(service in-memory, fs tempdir, bridge integration). No DB required
for the suite. publish_ephemeral and the orphan sweep stay deferred.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-03 21:18:17 +02:00
107 changed files with 20503 additions and 416 deletions

72
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,72 @@
name: CI
on:
push:
branches: [main]
pull_request:
env:
CARGO_TERM_COLOR: always
# Matches what docker-compose produces locally; the schema-snapshot
# guardrail and any other DB-backed tests run against this service.
DATABASE_URL: postgres://picloud:picloud@localhost:5432/picloud
jobs:
rust:
name: Rust — fmt, clippy, test
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_USER: picloud
POSTGRES_PASSWORD: picloud
POSTGRES_DB: picloud
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U picloud"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@v4
# rust-toolchain.toml pins the channel; this action honors it.
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
components: rustfmt, clippy
- name: Cache cargo
uses: Swatinem/rust-cache@v2
- name: Format check
run: cargo fmt --all -- --check
- name: Clippy
run: cargo clippy --all-targets --all-features -- -D warnings
# Runs the whole workspace, including the schema-snapshot guardrail
# (it picks up DATABASE_URL from the env above and the postgres
# service; without a DB it would skip cleanly).
- name: Test
run: cargo test --workspace
dashboard:
name: Dashboard — check
runs-on: ubuntu-latest
defaults:
run:
working-directory: dashboard
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
cache-dependency-path: dashboard/package-lock.json
- name: Install deps
run: npm ci
- name: Svelte check
run: npm run check

3
.gitignore vendored
View File

@@ -22,6 +22,9 @@ Cargo.lock.bak
# Local config overrides # Local config overrides
config.local.toml config.local.toml
/data /data
# Files-root blob storage created when integration tests run build_app
# from the picloud crate dir (PICLOUD_FILES_ROOT default ./data).
/crates/picloud/data
/postgres-data /postgres-data
# Dashboard # Dashboard

View File

@@ -1,5 +1,259 @@
# PiCloud Changelog # PiCloud Changelog
## v1.1.7 — Configuration & Email (unreleased)
The operational-config layer: **encrypted per-app secrets**, **outbound
email**, and an **inbound email trigger** — plus the long-missing
**dead-letter handler wiring** and **at-rest encryption of the realtime
signing key**. All at-rest encryption uses a single process master key
(AES-256-GCM); key rotation is deferred to v1.2.
### Added — Encryption infrastructure
- **Process master key** from `PICLOUD_SECRET_KEY` (base64 of exactly 32
bytes). REQUIRED at startup — an unset or malformed key is fatal.
Generate one with `openssl rand -base64 32`. A deterministic in-memory
dev key is used ONLY when `PICLOUD_SECRET_KEY` is unset AND
`PICLOUD_DEV_MODE=true` (with a prominent startup warning); there is no
quiet unencrypted mode.
- **`picloud_shared::crypto`** — `encrypt`/`decrypt` envelope:
`Aes256Gcm`, 96-bit CSPRNG nonce, 128-bit auth tag appended to the
ciphertext (RustCrypto `Aead` layout). Both ciphertext and nonce are
stored.
- **Key rotation is out of scope.** Changing `PICLOUD_SECRET_KEY` between
deploys renders all existing ciphertext undecryptable. v1.2+ adds
key-version columns + a re-encryption pass.
### Added — Encrypted per-app secrets
- **`secrets::{get,set,delete,list}(name)`** SDK — collection-less,
per-app. `set` accepts a String/Map/Array (JSON-encoded then encrypted);
`get` returns the same Rhai type back; missing → `()`. 64 KB plaintext
cap (`PICLOUD_SECRET_MAX_VALUE_BYTES`). `migrations/0023_secrets.sql`.
- **Admin API** `GET/POST/DELETE /api/v1/admin/apps/{id}/secrets` — list
returns names + `updated_at` only, **never values**.
- **Dashboard Secrets tab** — list names + last-modified, create/update
(masked value with a confirm-gated reveal), delete with confirm.
- `Capability::AppSecretsRead`/`Write` (→ `script:read` / `script:write`).
No new Scope variants (seven-scope commitment). Secret writes
deliberately do **not** emit trigger events.
### Added — Outbound email
- **`email::send` / `email::send_html`** SDK over an SMTP relay
(`lettre`). Config from `PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/TLS/
TIMEOUT_SECS`; if HOST/USER/PASSWORD aren't all set the service runs in
**disabled mode** (every send throws `NotConfigured`, warned at
startup). Required `to`/`from`/`subject` + one of `text`/`html`;
RFC 5322-ish address validation; 25 MB per-message cap
(`PICLOUD_EMAIL_MAX_MESSAGE_BYTES`); `reply_to` defaults to `from`.
Per-call connection (pooling deferred to v1.2); per-app `from`
validation / SPF / DKIM are the operator's SMTP-relay concern.
- `Capability::AppEmailSend` (→ `script:write`).
### Added — Inbound email (`email:receive` trigger)
- **Webhook receiver** `POST /api/v1/email-inbound/{app_id}/{trigger_id}`
— a provider (Mailgun / Postmark / SendGrid / SES) POSTs the generic
JSON shape `{from,to[],cc[],subject,text,html,message_id}`; the
receiver verifies the optional HMAC signature, normalizes to
`TriggerEvent::Email`, and enqueues an outbox row. 202 accepted, 401
bad/missing signature, 404 missing/wrong-kind/cross-app, 422 malformed.
Handlers see `ctx.event.email`. `migrations/0024_email_triggers.sql`.
- **Admin** `POST /api/v1/admin/apps/{id}/triggers/email` +
dashboard form (with the webhook URL + expected payload). The HMAC
`inbound_secret` is stored **encrypted** via the master key (deviation
from the original plaintext design — see HANDBACK §7).
- Provider-specific payload unmarshallers + inbound attachments → v1.2.
Native SMTP listener → v1.3+.
### Security/correctness fix (retroactive) — dead_letter handlers
The `dead_letter` trigger kind has been registerable since v1.1.1 but,
due to missing dispatcher wiring (`list_matching_dead_letter` had no
production caller), handlers have **never fired**. Any deploy running
v1.1.1 through v1.1.6 with `dead_letter` triggers configured has had
silently non-functional handlers. v1.1.7 fixes the wiring; existing
`dead_letters` rows remain (no migration needed) but only NEW
dead-letter events (post-v1.1.7) trigger handlers. To process older
rows, use the existing admin replay surface to re-enqueue them.
### Changed — Realtime signing key encrypted at rest (two-phase)
`app_secrets.realtime_signing_key` was stored as 32 plaintext bytes. It
is now encrypted with the master key. `migrations/0025_encrypt_realtime_keys.sql`
adds NULL-able encrypted columns and drops `NOT NULL` on the plaintext
column; a startup task encrypts pre-existing rows; the read path prefers
the encrypted columns and falls back to plaintext during the compat
window. **v1.1.8 will drop the plaintext `realtime_signing_key`
column** — operators should upgrade through v1.1.7 (which performs the
encryption) before v1.1.8.
### Notes
- **New deps:** `aes-gcm` (RustCrypto AEAD), `lettre` (SMTP).
- **New env vars:** `PICLOUD_SECRET_KEY` (required), `PICLOUD_DEV_MODE`,
`PICLOUD_SECRET_MAX_VALUE_BYTES`, `PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/
TLS/TIMEOUT_SECS`, `PICLOUD_EMAIL_MAX_MESSAGE_BYTES`.
- **SDK schema** 1.7 → 1.8; **dashboard** 0.12.0 → 0.13.0.
## v1.1.6 — Realtime Channels & Client Library (unreleased)
The first **external realtime surface** and the first **frontend
library**, co-shipped per the §5/§6 design-notes decisions. Browser
clients can subscribe over SSE to per-app pub/sub topics that have been
explicitly externalized; everything else stays internal-only. The
`@picloud/client` TypeScript package wraps typed HTTP, SSE, auth, and
React/Svelte hooks. Plus three v1.1.5 follow-ups.
### Added — Realtime
- **`topics` registry** (`migrations/0021_topics.sql`) — pub/sub topics
are internal-only by default; a `topics` row with
`external_subscribable = true` opts one into external SSE subscription.
`auth_mode` is `'public'` or `'token'`.
- **Topic admin endpoints** under `/api/v1/admin/apps/{id}/topics`
`POST` (register), `GET` (list), `PATCH /{name}` (flip
external/auth_mode — its own audited surface), `DELETE /{name}`
(unregister + disconnect live subscribers). Gated by the new
`Capability::AppTopicManage``app:admin` scope (no new scope; the
seven-scope commitment holds).
- **SSE endpoint `GET /realtime/topics/{topic}`** — data-plane surface
(deliberately not under `/api/`). Resolves `Host` → app, authorizes
via the `RealtimeAuthority` (404 for missing/internal topics, 401 for
bad/absent tokens), then streams `data: {topic,message,published_at}`
events with a configurable heartbeat (`PICLOUD_REALTIME_HEARTBEAT_SEC`,
default 30). Token via `Authorization: Bearer` or `?token=`.
- **`RealtimeBroadcaster` + `RealtimeEvent` + `RealtimeAuthority`**
traits (`picloud-shared`); in-process `InProcessBroadcaster`
(`tokio::sync::broadcast`, per-channel capacity
`PICLOUD_REALTIME_BROADCAST_CAPACITY` default 64, periodic empty-channel
GC) and the DB-backed `RealtimeAuthorityImpl` (orchestrator-core /
manager-core respectively). The publish path now also fans out to
in-process SSE subscribers, best-effort, after the durable outbox
fan-out commits — a broadcast failure never fails the publish.
- **`pubsub::subscriber_token(topics, ttl)`** Rhai SDK (SDK schema
1.6 → 1.7) — mints an HMAC-SHA256 subscriber token (URL-safe
`payload.signature`) scoped to externally-subscribable topics.
Requires an authenticated principal + the pub/sub publish capability.
TTL clamped to `[10s, 24h]` (default 1h), env-overridable via
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`. Per-app signing
keys persist in the new `app_secrets` table
(`migrations/0022_app_secrets.sql`), created lazily on first mint. No
per-token revocation (rotation invalidates wholesale; short TTL is the
safety mechanism).
- **Dashboard Topics tab** — register/list/edit/delete topics with a
prominent external/internal badge, auth-mode radio (conditional on
external), and a confirmation when flipping a topic external.
### Added — `@picloud/client` (TypeScript, v1.0.0)
- New top-level package `clients/typescript/` (tsup dual ESM+CJS +
`.d.ts`, vitest). Typed HTTP via `endpoint<Req,Res>(path).get()/.post()`
with auth-token injection and structured errors; SSE `subscribe(topic,
cb, {token, onTokenExpired})` with exponential-backoff reconnect,
401 token-refresh, and `Last-Event-ID` resume; `auth.login/logout/token`
over dev-defined endpoints; React (`useTopic`/`useEndpoint` +
`PicloudProvider`) and Svelte (`topicStore`/`endpointStore`) subpath
exports. Optional zod/valibot runtime validation via a `{ parse }`
adapter (no hard dep). Hybrid model: no direct service access from the
browser.
### Changed / Fixed — v1.1.5 follow-ups
- **Empty blobs accepted** — `NewFile::validate` / `FileUpdate::validate`
no longer reject zero-length `data`; empty files are a valid stored
state (sentinels, placeholders). Non-breaking.
- **Orphan `*.tmp.*` sweeper** — a startup tokio task
(`spawn_files_orphan_sweep`) walks the files root every
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC` (default 6h) and unlinks temp
blobs older than `PICLOUD_FILES_ORPHAN_TMP_TTL_SEC` (default 1h). No DB
cross-check (that full reconciler is v1.3+).
- **Dispatcher end-to-end tests** — `crates/picloud/tests/dispatcher_e2e.rs`,
one per trigger kind (kv/docs/cron/files/pubsub/dead_letter),
DATABASE_URL-gated (skip cleanly when unset).
### Notes
- New deps: `hmac` (token signing, picloud-shared), `tokio-stream` (SSE
body stream, orchestrator-core).
- New env vars: `PICLOUD_REALTIME_HEARTBEAT_SEC`,
`PICLOUD_REALTIME_BROADCAST_CAPACITY`,
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`,
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC`,
`PICLOUD_FILES_ORPHAN_TMP_TTL_SEC`.
## v1.1.5 — Files & Pub/Sub (unreleased)
Two stateful services + two trigger kinds. **`files::*`** is
filesystem-backed blob storage (atomic writes, path-sharded layout,
single-pass SHA-256 with checksum-verified reads); the metadata row
lives in Postgres, the bytes on disk. **`pubsub::publish_durable`** is
durable pub/sub through the universal outbox, fanning out one delivery
row per matching subscriber **at publish time** inside a single
transaction. Both ride the v1.1.1 trigger framework as the fifth and
sixth concrete kinds via the established Layout-E extension pattern.
### Added
- **`files::collection(name).{create,head,get,update,delete,list}`** —
blob storage SDK. `create`/`update` take a Rhai `Blob`; `get` returns
a `Blob` (or `()` if missing); `head`/`list` return metadata maps
(`id, name, content_type, size, checksum, created_at, updated_at`).
`create`/`update`/`delete` throw on failure; `get`/`head` return `()`
for a missing file; `delete` returns a was-present bool. Missing
required field on `create` throws naming the field.
- **Atomic writes** — temp file → fsync → rename → fsync parent dir →
DB row, so a crash never leaves a readable half-written file. SHA-256
is computed in a single pass during the write; `get` re-verifies it
and surfaces `FilesError::Corrupted` (logged with the path, never
auto-deleted) on a mismatch. Shard dirs are created `0o700`.
- **`files:*` trigger kind** — `ctx.event.files` carries the metadata
only (never the bytes; a handler that wants them calls
`files::collection(c).get(id)`). `prev` is `()` on create, the prior
metadata on update, the deleted metadata on delete.
- **`pubsub::publish_durable(topic, message)`** — durable publish.
Message is any JSON-serializable Rhai value; Blobs encode as base64
(at any nesting depth). No matching subscriber → the publish succeeds
silently with zero outbox rows.
- **`pubsub:*` trigger kind** — topic patterns are exact, `<prefix>.*`,
or `*`; mid-pattern wildcards are rejected at trigger creation.
`ctx.event.pubsub` carries `topic`, `message`, `published_at`.
- **`FilesService` + `PubsubService` traits** (`picloud-shared`) +
`FsFilesRepo`/`FilesServiceImpl` and `PostgresPubsubRepo`/
`PubsubServiceImpl` (manager-core). Wired into the `Services` bundle
as `files` and `pubsub`.
- **Capabilities** `AppFilesRead`/`AppFilesWrite``script:read`/
`script:write`, `AppPubsubPublish``script:write`. No new `Scope`
variant — the seven-scope commitment holds. Script-as-gate: skipped
when the script runs unauthenticated.
- **Admin files API** (`GET`/`DELETE /apps/{id}/files`) + dashboard
Files view per app; **Pub/Sub trigger form** on the Triggers tab.
- **CI** — first `.github/workflows/ci.yml` (Postgres service, fmt +
clippy + `cargo test --workspace`); the schema-snapshot guardrail now
runs instead of being `#[ignore]`'d.
### Changed
- Workspace version: 1.1.4 → 1.1.5
- Rhai SDK version: 1.5 → 1.6
- Dashboard version: 0.10.0 → 0.11.0
- `schema_snapshot` test: no longer `#[ignore]`'d — runs against
`DATABASE_URL` when set, skips cleanly when absent.
### Migrations
- 0018_files.sql — `files` metadata table (bytes live on disk).
- 0019_files_triggers.sql — widen kind/source_kind CHECKs + add
`files_trigger_details`.
- 0020_pubsub_triggers.sql — widen kind/source_kind CHECKs + add
`pubsub_trigger_details` + partial index.
### New environment variables
- `PICLOUD_FILES_ROOT` (default `./data`)
- `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` (default 100 MB)
## v1.1.4 — Outbound HTTP & Cron triggers (unreleased) ## v1.1.4 — Outbound HTTP & Cron triggers (unreleased)
Two surfaces. **`http::*`** lets Rhai scripts make outbound HTTP Two surfaces. **`http::*`** lets Rhai scripts make outbound HTTP

View File

@@ -118,6 +118,8 @@ Environment variables consumed by the `picloud` binary:
| `DATABASE_URL` | — | Required. Postgres connection string. | | `DATABASE_URL` | — | Required. Postgres connection string. |
| `PICLOUD_SESSION_TTL_HOURS` | `24` | Sliding-window session lifetime. | | `PICLOUD_SESSION_TTL_HOURS` | `24` | Sliding-window session lifetime. |
| `PICLOUD_SANDBOX_MAX_*` | conservative defaults | Per-knob admin ceilings on Rhai sandbox overrides. See `manager-core::sandbox::SandboxCeiling`. | | `PICLOUD_SANDBOX_MAX_*` | conservative defaults | Per-knob admin ceilings on Rhai sandbox overrides. See `manager-core::sandbox::SandboxCeiling`. |
| `PICLOUD_FILES_ROOT` | `./data` | Filesystem root for `files::*` blob storage (v1.1.5). Bytes live at `<root>/files/<app_id>/<collection>/<id[0:2]>/<id>`; metadata in Postgres. |
| `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` | `104857600` (100 MB) | Per-file hard size cap for `files::*` (v1.1.5). Per-app quotas deferred to v1.2. |
## Out of MVP ## Out of MVP

222
Cargo.lock generated
View File

@@ -2,6 +2,41 @@
# It is not intended for manual editing. # It is not intended for manual editing.
version = 4 version = 4
[[package]]
name = "aead"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0"
dependencies = [
"crypto-common",
"generic-array",
]
[[package]]
name = "aes"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures",
]
[[package]]
name = "aes-gcm"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "831010a0f742e1209b3bcea8fab6a8e149051ba6099432c8cb2cc117dec3ead1"
dependencies = [
"aead",
"aes",
"cipher",
"ctr",
"ghash",
"subtle",
]
[[package]] [[package]]
name = "ahash" name = "ahash"
version = "0.8.12" version = "0.8.12"
@@ -400,6 +435,16 @@ dependencies = [
"phf_codegen", "phf_codegen",
] ]
[[package]]
name = "cipher"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common",
"inout",
]
[[package]] [[package]]
name = "clap" name = "clap"
version = "4.6.1" version = "4.6.1"
@@ -528,7 +573,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f8c3e73077b4b4a6ab1ea5047c37c57aee77657bc8ecd6f29b0af082d0b0c07" checksum = "6f8c3e73077b4b4a6ab1ea5047c37c57aee77657bc8ecd6f29b0af082d0b0c07"
dependencies = [ dependencies = [
"chrono", "chrono",
"nom", "nom 7.1.3",
"once_cell", "once_cell",
] ]
@@ -560,9 +605,19 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [ dependencies = [
"generic-array", "generic-array",
"rand_core 0.6.4",
"typenum", "typenum",
] ]
[[package]]
name = "ctr"
version = "0.9.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0369ee1ad671834580515889b80f2ea915f23b8be8d0daa4bbaf2ac5c7590835"
dependencies = [
"cipher",
]
[[package]] [[package]]
name = "data-encoding" name = "data-encoding"
version = "2.11.0" version = "2.11.0"
@@ -660,6 +715,22 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "email-encoding"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9298e6504d9b9e780ed3f7dfd43a61be8cd0e09eb07f7706a945b0072b6670b6"
dependencies = [
"base64",
"memchr",
]
[[package]]
name = "email_address"
version = "0.2.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e079f19b08ca6239f47f8ba8509c11cf3ea30095831f7fed61441475edd8c449"
[[package]] [[package]]
name = "equivalent" name = "equivalent"
version = "1.0.2" version = "1.0.2"
@@ -880,6 +951,16 @@ dependencies = [
"wasip3", "wasip3",
] ]
[[package]]
name = "ghash"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0d8a4362ccb29cb0b265253fb0a2728f592895ee6854fd9bc13f2ffda266ff1"
dependencies = [
"opaque-debug",
"polyval",
]
[[package]] [[package]]
name = "hashbrown" name = "hashbrown"
version = "0.15.5" version = "0.15.5"
@@ -945,6 +1026,17 @@ dependencies = [
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
[[package]]
name = "hostname"
version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "617aaa3557aef3810a6369d0a99fac8a080891b68bd9f9812a1eeda0c0730cbd"
dependencies = [
"cfg-if",
"libc",
"windows-link",
]
[[package]] [[package]]
name = "http" name = "http"
version = "1.4.0" version = "1.4.0"
@@ -1201,6 +1293,15 @@ version = "0.1.15"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8fae54786f62fb2918dcfae3d568594e50eb9b5c25bf04371af6fe7516452fb" checksum = "c8fae54786f62fb2918dcfae3d568594e50eb9b5c25bf04371af6fe7516452fb"
[[package]]
name = "inout"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"generic-array",
]
[[package]] [[package]]
name = "ipnet" name = "ipnet"
version = "2.12.0" version = "2.12.0"
@@ -1246,6 +1347,34 @@ version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
[[package]]
name = "lettre"
version = "0.11.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0da65617f6cb926332d039cb578aad56178da86e128db6a1b09f4c94fa5b3349"
dependencies = [
"async-trait",
"base64",
"email-encoding",
"email_address",
"fastrand",
"futures-io",
"futures-util",
"hostname",
"httpdate",
"idna",
"mime",
"nom 8.0.0",
"percent-encoding",
"quoted_printable",
"rustls",
"socket2",
"tokio",
"tokio-rustls",
"url",
"webpki-roots 1.0.7",
]
[[package]] [[package]]
name = "libc" name = "libc"
version = "0.2.186" version = "0.2.186"
@@ -1395,6 +1524,15 @@ dependencies = [
"minimal-lexical", "minimal-lexical",
] ]
[[package]]
name = "nom"
version = "8.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405"
dependencies = [
"memchr",
]
[[package]] [[package]]
name = "normalize-line-endings" name = "normalize-line-endings"
version = "0.3.0" version = "0.3.0"
@@ -1477,6 +1615,12 @@ version = "1.70.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
[[package]]
name = "opaque-debug"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381"
[[package]] [[package]]
name = "option-ext" name = "option-ext"
version = "0.2.0" version = "0.2.0"
@@ -1610,7 +1754,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud" name = "picloud"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"async-trait", "async-trait",
@@ -1618,12 +1762,15 @@ dependencies = [
"axum-test", "axum-test",
"chrono", "chrono",
"figment", "figment",
"hex",
"hmac",
"picloud-executor-core", "picloud-executor-core",
"picloud-manager-core", "picloud-manager-core",
"picloud-orchestrator-core", "picloud-orchestrator-core",
"picloud-shared", "picloud-shared",
"serde", "serde",
"serde_json", "serde_json",
"sha2",
"sqlx", "sqlx",
"thiserror 1.0.69", "thiserror 1.0.69",
"tokio", "tokio",
@@ -1636,7 +1783,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-cli" name = "picloud-cli"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"assert_cmd", "assert_cmd",
@@ -1657,7 +1804,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-executor" name = "picloud-executor"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"picloud-executor-core", "picloud-executor-core",
@@ -1669,7 +1816,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-executor-core" name = "picloud-executor-core"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"base64", "base64",
@@ -1693,7 +1840,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-manager" name = "picloud-manager"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"picloud-manager-core", "picloud-manager-core",
@@ -1705,7 +1852,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-manager-core" name = "picloud-manager-core"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"argon2", "argon2",
"async-trait", "async-trait",
@@ -1715,6 +1862,9 @@ dependencies = [
"chrono-tz", "chrono-tz",
"cron", "cron",
"data-encoding", "data-encoding",
"hex",
"hmac",
"lettre",
"picloud-executor-core", "picloud-executor-core",
"picloud-orchestrator-core", "picloud-orchestrator-core",
"picloud-shared", "picloud-shared",
@@ -1733,7 +1883,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-orchestrator" name = "picloud-orchestrator"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"picloud-orchestrator-core", "picloud-orchestrator-core",
@@ -1745,7 +1895,7 @@ dependencies = [
[[package]] [[package]]
name = "picloud-orchestrator-core" name = "picloud-orchestrator-core"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"axum", "axum",
@@ -1759,6 +1909,8 @@ dependencies = [
"serde_json", "serde_json",
"thiserror 1.0.69", "thiserror 1.0.69",
"tokio", "tokio",
"tokio-stream",
"tower",
"tracing", "tracing",
"urlencoding", "urlencoding",
"uuid", "uuid",
@@ -1766,13 +1918,20 @@ dependencies = [
[[package]] [[package]]
name = "picloud-shared" name = "picloud-shared"
version = "1.1.4" version = "1.1.7"
dependencies = [ dependencies = [
"aes-gcm",
"async-trait", "async-trait",
"base64",
"chrono", "chrono",
"hmac",
"rand 0.8.6",
"serde", "serde",
"serde_json", "serde_json",
"sha2",
"thiserror 1.0.69", "thiserror 1.0.69",
"tokio",
"tracing",
"uuid", "uuid",
] ]
@@ -1815,6 +1974,18 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6" checksum = "b4596b6d070b27117e987119b4dac604f3c58cfb0b191112e24771b2faeac1a6"
[[package]]
name = "polyval"
version = "0.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25"
dependencies = [
"cfg-if",
"cpufeatures",
"opaque-debug",
"universal-hash",
]
[[package]] [[package]]
name = "portable-atomic" name = "portable-atomic"
version = "1.13.1" version = "1.13.1"
@@ -1981,6 +2152,12 @@ dependencies = [
"proc-macro2", "proc-macro2",
] ]
[[package]]
name = "quoted_printable"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "478e0585659a122aa407eb7e3c0e1fa51b1d8a870038bd29f0cf4a8551eea972"
[[package]] [[package]]
name = "r-efi" name = "r-efi"
version = "5.3.0" version = "5.3.0"
@@ -2284,6 +2461,7 @@ version = "0.23.40"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b"
dependencies = [ dependencies = [
"log",
"once_cell", "once_cell",
"ring", "ring",
"rustls-pki-types", "rustls-pki-types",
@@ -2990,6 +3168,20 @@ dependencies = [
"futures-core", "futures-core",
"pin-project-lite", "pin-project-lite",
"tokio", "tokio",
"tokio-util",
]
[[package]]
name = "tokio-util"
version = "0.7.18"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098"
dependencies = [
"bytes",
"futures-core",
"futures-sink",
"pin-project-lite",
"tokio",
] ]
[[package]] [[package]]
@@ -3209,6 +3401,16 @@ version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853"
[[package]]
name = "universal-hash"
version = "0.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea"
dependencies = [
"crypto-common",
"subtle",
]
[[package]] [[package]]
name = "untrusted" name = "untrusted"
version = "0.9.0" version = "0.9.0"

View File

@@ -13,7 +13,7 @@ members = [
] ]
[workspace.package] [workspace.package]
version = "1.1.4" version = "1.1.7"
edition = "2021" edition = "2021"
rust-version = "1.92" rust-version = "1.92"
license = "MIT OR Apache-2.0" license = "MIT OR Apache-2.0"
@@ -29,6 +29,8 @@ picloud-manager-core = { path = "crates/manager-core" }
# Async + HTTP # Async + HTTP
tokio = { version = "1.40", features = ["full"] } tokio = { version = "1.40", features = ["full"] }
# Wraps a broadcast::Receiver into a Stream for the SSE endpoint (v1.1.6).
tokio-stream = { version = "0.1", features = ["sync"] }
axum = "0.8" axum = "0.8"
tower = "0.5" tower = "0.5"
tower-http = { version = "0.6", features = ["trace", "cors"] } tower-http = { version = "0.6", features = ["trace", "cors"] }
@@ -75,8 +77,17 @@ urlencoding = "2"
argon2 = "0.5" argon2 = "0.5"
rand = { version = "0.8", features = ["getrandom"] } rand = { version = "0.8", features = ["getrandom"] }
sha2 = "0.10" sha2 = "0.10"
# HMAC-SHA256 for realtime subscriber tokens (v1.1.6).
hmac = "0.12"
base64 = "0.22" base64 = "0.22"
data-encoding = "2.6" data-encoding = "2.6"
# AES-256-GCM at-rest encryption for per-app secrets + the realtime
# signing key (v1.1.7). Audited, pure-Rust RustCrypto AEAD.
aes-gcm = { version = "0.10", features = ["aes", "alloc"] }
# Outbound SMTP email (v1.1.7). Async transport over the Tokio runtime
# with rustls TLS; built messages for text + multipart-alternative.
lettre = { version = "0.11", default-features = false, features = ["smtp-transport", "tokio1-rustls-tls", "builder", "hostname"] }
# Stdlib utility crates (v1.1.0 stdlib PR — registered into the # Stdlib utility crates (v1.1.0 stdlib PR — registered into the
# Rhai engine as the regex::/random::/etc. namespaces) # Rhai engine as the regex::/random::/etc. namespaces)

View File

@@ -1,252 +1,330 @@
# v1.1.4 Handback — Outbound HTTP SDK & Cron Triggers # v1.1.7 — Configuration & Email — HANDBACK
**Branch:** `feat/v1.1.4-http-cron` (off `main`) **Branch:** `feat/v1.1.7-secrets-email` (9 commits off `main`, not pushed)
**Commits:** 1 implementation commit (`feat(v1.1.4): outbound HTTP SDK + cron triggers`) + this HANDBACK commit. **Status:** ready for review. NOT merged, NOT pushed, no PR opened.
> **Note on commit granularity:** the brief suggested split
> `feat(v1.1.4-http)` / `feat(v1.1.4-cron)` commits. The two features are
> interleaved across shared files (`Cargo.toml`, `crates/picloud/src/lib.rs`,
> `crates/manager-core/src/lib.rs`, `version.rs`, `services.rs`), so
> cleanly-*compiling* per-theme commits aren't separable without interactive
> hunk staging (unavailable in this environment). I chose one coherent,
> green commit over shipping broken intermediates. Squash/relabel as you see fit.
---
## Scope coverage
| # | Item | Status |
|---|------|--------|
| 1 | `http::*` SDK surface (get/post/put/patch/delete/head/post_form/request) | **Done** |
| 2 | SSRF deny-list (resolved-IP, DNS-rebinding defense, scheme/port, body caps, UA, timeouts, `PICLOUD_HTTP_ALLOW_PRIVATE`) | **Done** |
| 3 | http authz (`Capability::AppHttpRequest``script:write`, script-as-gate, no new Scope) | **Done** |
| 4 | `HttpService` trait + `HttpServiceImpl` + Services wiring | **Done** |
| 5 | Cron migration `0017` (Layout-E extension) | **Done** |
| 6 | Cron scheduler tokio task (catch-up = fire-once) | **Done** |
| 7 | `ctx.event.cron` shape + `TriggerEvent::Cron` | **Done** |
| 8 | Dispatcher routing extension (`… \| Cron`) | **Done** |
| 9 | Dashboard cron trigger UI (minimal) | **Done** |
| 10a | Redact `ModuleSourceError::Backend` at resolver boundary | **Done** |
| 10b | Pin `rhai = "=1.24"` | **Done** |
| 10c | CHANGELOG retroactive v1.1.3 cross-app-trigger security note | **Done** |
| 11 | Version bumps (workspace 1.1.4, SDK 1.5, dashboard 0.10.0) | **Done** |
| 12 | Tests (~50-70) | **Done** — 70 new |
---
## SSRF policy implementation notes
- **reqwest hook.** `crates/manager-core/src/ssrf.rs` defines `SsrfResolver`
implementing `reqwest::dns::Resolve`, plugged in via
`ClientBuilder::dns_resolver`. It delegates to the system resolver
(injectable for tests — see DNS-rebinding test), then filters each `IpAddr`
through `SsrfPolicy::check`. Because reqwest re-resolves at every connection
(including each redirect hop), the policy applies post-redirect too.
- **`dns_resolver` is generic over a concrete `R: Resolve`** (stores `Arc<R>`),
so the resolver is passed as `Arc<SsrfResolver>`, not `Arc<dyn Resolve>`.
- **Literal-IP gap closed.** reqwest only routes *hostnames* through the custom
resolver — a URL with a literal-IP host (`http://127.0.0.1/`) bypasses it
entirely. The impl therefore *also* runs `SsrfPolicy::check` on literal-IP
hosts at URL-parse time (`validate_url`), on every hop. Both paths are tested.
- **IPv4-mapped IPv6 re-check.** `check_v6` calls `Ipv6Addr::to_ipv4_mapped()`;
if `Some`, it re-runs the v4 deny-list against the embedded address
(`::ffff:127.0.0.1` → denied as "loopback").
- **Applied before AND after redirects.** Redirects are followed *manually*
(client built with `redirect(Policy::none())`) so per-request
`follow_redirects`/`max_redirects` are honored; each hop re-validates
scheme/port + literal-IP and re-resolves hostnames through the SSRF resolver.
- **Script-visible error format.** `"http: blocked by SSRF policy: <reason>"`
where `<reason>` is a CIDR category (`loopback`, `private`, `link-local`,
`carrier-grade-nat`, `multicast`, `reserved`, `unique-local`, `unspecified`).
**The resolved IP is never included.** The all-addresses-denied case surfaces
as `Ssrf` (not a generic DNS error) via a marker error the resolver emits and
the impl detects by walking the reqwest error source chain.
## Cron scheduler implementation notes
- **Catch-up = fire-once.** Matches the brief; no deviation. `next_due` returns a
single canonical scheduled-at (first slot after `last_fired_at`, or
`created_at` if never fired); after firing, `last_fired_at = now`, so the next
tick sees only future slots. **Verified live** against Postgres: an
every-second (`* * * * * *`) trigger with a 2s tick advanced `last_fired_at`
~once per 2s, not once per second.
- **No ExecutionGate contention.** The scheduler only enqueues to the outbox
(one row per due trigger per tick, in a `FOR UPDATE OF d SKIP LOCKED`
transaction that also bumps `last_fired_at`). The existing dispatcher acquires
the gate and delivers it identically to kv/docs/dead_letter — verified live
(the cron outbox row was consumed, the script executed, the row deleted).
- **Timezone handling.** `chrono-tz`. Invalid IANA names are rejected at the
admin endpoint with a 422 (`TriggersApiError::Invalid`, message contains
"timezone"); the repo re-validates defensively before insert.
- **Schema beyond the brief:** none. Followed the brief exactly — `schedule`,
`timezone DEFAULT 'UTC'`, `last_fired_at`, `idx_cron_triggers_due`. **No**
stored `next_scheduled_at` column (an exploration agent suggested one; the
brief computes next-fire in-process, which I followed).
---
## Tests added (70 new)
- **SSRF policy + resolver (`ssrf.rs`, 20):** one per deny CIDR (127/8, 0/8,
10/8, 172.16/12, 192.168/16, 169.254/16 incl. metadata, 100.64/10, 224/4,
240/4, ::1, ::, fe80::/10, fc00::/7, ff00::/8); 172.x outside-range allowed;
public v4/v6 allowed; IPv4-mapped re-check; `allow_private` disables all;
resolver returns only allowed addrs; all-denied → SSRF marker; **DNS rebinding**
(mock resolver: public then private — second denied); empty resolution ≠ SSRF.
- **HTTP client (`http_service.rs`, 16):** GET/POST round-trips vs a hand-rolled
`TcpListener`; body dispatch + default UA; custom UA override; empty body;
non-2xx no-error; response cap via Content-Length; response cap mid-stream
(no Content-Length); request body cap pre-send; redirect-to-max-then-throw;
scheme rejection (file/ftp/gopher); port rejection (22/25/465/587); SSRF
literal-loopback; SSRF hostname-resolves-to-loopback; timeout; authz
(anon skips / member forbidden / member-with-role allowed).
- **Bridge integration (`sdk_http.rs`, 15):** real Rhai engine under
`spawn_blocking` vs a recording fake — status+JSON body, non-JSON string,
empty→`()`, Map→JSON, String→text, `()`→no body, headers+timeout forwarded,
unknown opt key throws, timeout>max throws, non-2xx no-throw, network error
throws `http:`, `post_form` url-encoding, `request` arbitrary method,
default-UA carries `script_id`, `cx.app_id` forwarded for attribution.
- **Cron scheduler (`cron_scheduler.rs`, 11):** 6-field schedule accept / 5-field
+ malformed reject; IANA tz accept / reject; due/not-due; never-fired uses
created_at; **catch-up fires exactly once after 5 missed windows**; timezone
affects fire time; bad schedule/tz → None.
- **Cron admin (`triggers_api.rs`, 6):** create succeeds; invalid schedule;
unknown timezone; **module target rejected** (v1.1.3 regression); **cross-app
target rejected** (v1.1.3 regression); member-without-role forbidden.
- **Module redaction (2):** `modules.rs` — backend error redacted from the
script-visible error (no leak); `module_redaction_logging.rs` — original error
**is** logged at ERROR level (captured via a global tracing subscriber).
---
## Decisions beyond the brief (every prompt-default deviation, flagged)
1. **Three-arg split `verb(url, body, opts)`** (user-approved during planning).
Diverges from the brief's documented two-arg `(url, opts)` shape and
generalizes the escape hatch to `request(method, url, body, opts)`. Resolves
the brief's internal contradiction (its Slack example `http::post(url, #{text:...})`
passed a bare body map, which would be an "unknown opt key" under the
two-arg rule). The `opts` vocabulary is now exactly
`{headers, timeout_ms, follow_redirects, max_redirects}`**`body_raw` was
dropped** (raw strings go through the positional body as a String). The
Slack example works unchanged (`#{text:...}` is the body).
2. **Cron crate = `cron` (0.12), not `croner`.** The brief allowed either; `cron`
handles the 6-field-with-seconds format and named weekdays (`MON-FRI`) used in
the brief's example, and integrates with chrono `Schedule::after`.
3. **Catch-up = fire-once** — matches the brief; called out explicitly as
requested. No deviation.
4. **`SdkCallCx` gained a `script_id` field.** The brief's default User-Agent is
`picloud/<v> (script:<script_id>)`, but `SdkCallCx` didn't carry the script
id. Adding it (sourced from `ExecRequest.script_id` in the engine) is the
clean home and doubles as the audit-attribution key the brief emphasizes. All
19 construction sites updated. The dead-letters admin cx uses a fresh sentinel
id (no script executes there).
5. **SSRF also blocks IPv6 unspecified `::` and IPv4 `0.0.0.0`** with reason
"unspecified". `0.0.0.0/8` is in the brief's list; `::` is not explicitly but
is an obvious sibling hole, so I blocked it too (defensible superset).
6. **No reqwest feature additions needed**`dns_resolver` and `Response::chunk()`
compile under the existing `default-features = false, features = ["json","rustls-tls"]`.
No cookie jar (cookies feature is off, so there's no jar to disable). Added
`url` as an executor-core dep (for `form_urlencoded` in `post_form`).
---
## How to verify locally (§8 attestation — run on this exact HEAD)
All four gates were run on the handback HEAD (the `feat(v1.1.4)` commit, before
this markdown commit):
``` ```
cargo fmt --all -- --check → exit 0 a7d3dad chore(v1.1.7): re-bless schema snapshot for secrets + email migrations
cargo clippy --all-targets --all-features -- -D warnings → exit 0 2ea47eb chore(v1.1.7): fix clippy --all-targets warnings
cargo test --workspace → 427 passed, 0 failed b355851 chore(v1.1.7): version bumps + CHANGELOG
(cd dashboard && npm run check) → 0 errors, 0 warnings (369 files) fffcdf6 feat(v1.1.7-realtime-migration): encrypt signing keys at rest
``` 02335a8 fix(v1.1.7-dead-letter): wire dispatcher → list_matching_dead_letter
1f78937 feat(v1.1.7-email-inbound): webhook receiver + email:receive trigger
This HANDBACK commit is pure markdown (no gate-relevant files), so the numbers 8f2d2bc feat(v1.1.7-email-outbound): SMTP send/send_html
above hold for the final HEAD. 2d11090 feat(v1.1.7-secrets): secrets SDK + table + admin API + dashboard
dc2e4fa feat(v1.1.7-crypto): master-key infra + encryption helpers
**Migrations — verified against a real Postgres (dev stack, port 15432):**
- Fresh-DB replay: the `#[sqlx::test]` schema-snapshot test applies all
migrations on a fresh ephemeral DB and matches the (re-blessed) golden — passes.
- On-top-of-prior-state: booting `picloud` against a dev DB pinned at migration
`0006` applied `0007…0017` cleanly (`"migrations applied"`); `_sqlx_migrations`
max is now `17`; `cron_trigger_details` + widened CHECKs present.
**Live smoke performed:**
- Boot logged the `PICLOUD_HTTP_ALLOW_PRIVATE` warning and started the cron
scheduler + HTTP service without panic.
- Seeded an every-second cron trigger → scheduler set `last_fired_at`, dispatcher
consumed the outbox row and ran the script (row deleted on success), and
`last_fired_at` advanced at the tick cadence (fire-once confirmed). Smoke data
cleaned up afterward.
- HTTP GET / SSRF-block / body-dispatch behaviors are covered by the automated
integration tests (real `TcpListener` round-trips + loopback/hostname SSRF
blocks) rather than a manual curl flow, since a live SSRF-block smoke
conflicts with the `PICLOUD_HTTP_ALLOW_PRIVATE` a local-server smoke requires.
To re-run the schema snapshot:
```
docker compose up -d postgres
DATABASE_URL=postgres://picloud:picloud@127.0.0.1:15432/picloud \
cargo test -p picloud-manager-core --test schema_snapshot -- --include-ignored
``` ```
--- ---
## ⚠️ Latent issue found: stale schema-snapshot golden ## 1. Scope coverage
`crates/manager-core/tests/expected_schema.txt` was **significantly stale** — the | Item | Status |
committed golden was missing many tables from prior releases |---|---|
(`abandoned_executions`, `dead_letters`, `dead_letter_trigger_details`, | Encryption infrastructure (master key + AES-256-GCM envelope) | **Done** |
`docs_*`, etc.). The `schema_snapshot` test is `#[ignore]` (needs a DB), so it | `secrets::*` SDK + `0023_secrets.sql` + admin API + dashboard tab | **Done** |
was apparently never re-blessed across v1.1.1v1.1.3 and silently drifted. | Outbound email `email::send` / `email::send_html` (lettre SMTP) | **Done** |
| Inbound email webhook receiver + `email:receive` trigger + `0024` | **Done** (full scope, per user decision) |
| Dispatcher routing for email | **Done** |
| dead_letter handler wiring fix | **Done** |
| Realtime signing-key encryption (two-phase) + `0025` | **Done** |
| Dashboard (Secrets tab, email trigger form, `npm run check`) | **Done** |
| Version bumps (1.1.7 / SDK 1.8 / dashboard 0.13.0) + CHANGELOG | **Done** |
| Tests (match v1.1.5/v1.1.6 density) | **Done** |
I re-blessed it, so the diff is large (+217 lines) but **only `cron_trigger_details` Nothing deferred from scope-in. Inbound email (the deferrable-if-scope-
+ the two widened CHECK constraints are v1.1.4-new** — the rest is pre-existing blew-up piece) was implemented in full.
drift correction. The blessed golden now matches a clean replay (verified).
Recommend the reviewer skim the diff to confirm, and consider whether the
`#[ignore]` should be lifted in CI (with a DB service) so the golden can't drift
again.
--- ---
## Latent security findings ## 2. Encryption infrastructure notes
None new beyond the (already-known, already-closed-in-v1.1.3) cross-app trigger - **Module:** `crates/shared/src/crypto.rs` (`picloud_shared::crypto`).
gap, which §10c now documents in the CHANGELOG. The SSRF surface is the main - **Master-key sourcing** (`MasterKey::from_env``resolve`):
security mechanism in this release; see the SSRF notes above for the - `PICLOUD_SECRET_KEY` = base64 of exactly 32 bytes. Missing →
defense-in-depth layering (resolver hook + literal-IP check + per-hop `MasterKeyError::Missing` (fatal); non-base64 → `Malformed`; wrong
re-validation + IP-never-leaked errors). length → `WrongLength`. **Sourced in `main.rs::run_server` before any
DB work** — `build_app` takes the `MasterKey` as a parameter (so
tests pass a fixed key and don't mutate process env).
- Dev fallback: deterministic key (`SHA-256("picloud-dev-master-key-v1.1.7")`)
used ONLY when `PICLOUD_SECRET_KEY` is unset **AND**
`PICLOUD_DEV_MODE=true`, with a prominent `warn!`. No quiet
unencrypted mode.
- **aes-gcm version:** `0.10` (features `aes`, `alloc`). `Aes256Gcm`.
- **Nonce generation:** 12 bytes from `rand::thread_rng().fill_bytes`
(OS-CSPRNG-seeded), per-encryption.
- **Storage layout:** ciphertext **with the 16-byte GCM auth tag
appended** (RustCrypto `Aead`-trait layout — `encrypt` returns
`ciphertext || tag`, `decrypt` consumes the same). The 12-byte nonce is
stored in a separate column. `MasterKey`'s `Debug` is redacted.
- **Plaintext cap (secrets):** 64 KB default, enforced in
`secrets_service::seal` (the SDK boundary) → `SecretsError::TooLarge`
with limit + actual size. Override: `PICLOUD_SECRET_MAX_VALUE_BYTES`.
- **Key rotation:** out of scope. Documented in CHANGELOG + the module
docs that changing `PICLOUD_SECRET_KEY` orphans all ciphertext.
One thing for the reviewer to weigh: the SSRF policy is a hardcoded deny-list ---
with no per-app allow-list (deferred to v1.2 per the brief). An operator who
needs a script to reach a private service has only the all-or-nothing
`PICLOUD_HTTP_ALLOW_PRIVATE` global escape hatch today.
## Open questions for the reviewer ## 3. Secrets notes
1. **Three-arg HTTP shape** (decision #1) — confirm you're happy with - `SecretsService` (trait, `picloud-shared`) → `SecretsServiceImpl` +
`verb(url, body, opts)` + dropping `body_raw`, vs the brief's documented `PostgresSecretsRepo` (`manager-core`) → Rhai bridge
two-arg form. This is the one user-facing API-shape divergence. (`executor-core/src/sdk/secrets.rs`). Collection-less; `app_id` from
2. **Stale schema golden** — OK to land the full re-bless in this PR, or would `cx.app_id`.
you prefer the drift correction split out? - **JSON round-trip:** `set` serializes the value to JSON bytes, caps,
encrypts; `get` decrypts + deserializes — a String returns a String
(not a JSON-quoted `"\"…\""`). Verified by unit + bridge tests.
- **No ServiceEvent emission** (secret writes don't fire triggers).
- Admin API: `GET/POST/DELETE /api/v1/admin/apps/{id}/secrets`; list
returns names + `updated_at` only.
- Authz: `Capability::AppSecretsRead/Write``script:read`/`script:write`.
No new Scope variants (seven-scope commitment held).
## Deferred items (per the brief's OUT list — not built) ---
WebSocket/SSE, streaming responses, HTTP/3, per-app outbound allow/deny lists, ## 4. Email implementation notes
per-app rate limits, mTLS, request signing, cookie jar, cron backfill replay,
cron next-fire preview, cron schedule history, drift compensation,
module-import-over-HTTP, files/pubsub/secrets/email/users/queue.
## Known limitations / rough edges - **SMTP transport:** `lettre 0.11` (`smtp-transport`,
`tokio1-rustls-tls`, `builder`, `hostname`). **Connection model:** one
connection per call (lettre default); pooling deferred to v1.2. The
transport sits behind an internal `EmailTransport` trait so the service
is unit-tested with a recording fake (no live SMTP).
- **Disabled mode:** if HOST/USER/PASSWORD aren't all set,
`EmailServiceImpl::from_env` builds no transport and every `send`
returns `NotConfigured` (warned at startup). A malformed relay
descriptor is also logged and yields disabled mode (email is
non-critical; never blocks startup).
- **Address validation:** hand-rolled RFC 5322-ish pre-check (single `@`,
non-empty local part, domain contains a dot, ≤320 bytes) followed by a
`lettre::Mailbox` parse (the authoritative validator). No deliverability
check.
- **Size cap:** 25 MB on `message.formatted()`,
`PICLOUD_EMAIL_MAX_MESSAGE_BYTES`.
- `email::send` forces text-only (ignores any `html`); `email::send_html`
requires `html` and builds `MultiPart::alternative_plain_html`.
`reply_to` defaults to `from`. `to`/`cc`/`bcc` accept a String or an
Array of Strings.
- **Inbound normalization:** only the generic provider-agnostic JSON
shape `{from,to[],cc[],subject,text,html,message_id}` is accepted in
v1.1.7 — `from` required, rest default. Provider-specific unmarshallers
→ v1.2. The expected shape is documented on the dashboard email-trigger
form.
- The dashboard Triggers tab lists all trigger kinds but only *creates* cron ---
triggers (kv/docs creation remains API-only, unchanged from before). No
next-fire-at preview (deferred to v1.2). ## 5. Dead-letter handler fix notes
- `post_form` / body field order follows Rhai map iteration order
(`BTreeMap`-backed, so sorted/deterministic; not insertion order). - **Call site:** `dispatcher::handle_failure`, the retry-exhaustion
- The cron scheduler tick is floored at 1s; sub-second schedules effectively branch. After `DeadLetterRepo::insert` (which returns the new
fire at the tick cadence (by design — see the fire-once policy). `DeadLetterId`), a new helper `fan_out_dead_letter` runs.
- The stale REVIEW.md at repo root is the v1.1.3 reviewer's artifact; the - **What it does:** calls `TriggerRepo::list_matching_dead_letter(app_id,
v1.1.4 reviewer should overwrite it. source, row.trigger_id, Some(resolved.script_id))` (the method that had
no production caller) and inserts one outbox row per match
(`source_kind = DeadLetter`, the DL trigger's id + handler script id,
`trigger_depth + 1`, `origin_principal = the DL trigger's registered
principal`).
- **Payload — built from the REAL `TriggerEvent::DeadLetter` variant**,
not the brief's §6 field list (see §7 deviations): `{ dead_letter_id,
original: Box::new(decoded row payload), attempts, last_error,
trigger_id, script_id, first_attempt_at, last_attempt_at }`. If the
outbox payload can't be decoded back into a `TriggerEvent` (so the
nested `original` can't be built), the fan-out is skipped — the
dead-letter row is still durably written.
- **Recursion-stop:** unchanged. The `is_dead_letter_handler`
short-circuit at the top of `handle_failure` returns before the
exhaustion branch, so a DL handler's own failure is never re-dead-
lettered. No new guard needed.
- **Tests verify the handler actually fires**
(`crates/picloud/tests/dispatcher_e2e.rs`, DB-gated):
`dispatcher_delivers_dead_letter_to_handler` now asserts BOTH row-create
AND handler-fire (inline doc updated);
`dispatcher_delivers_dead_letter_to_handler_actually_fires` asserts the
nested `original` KV event + `last_error`;
`dead_letter_source_filter_excludes_nonmatching` exercises the source
filter dimension; `dead_letter_handler_failure_does_not_recurse` proves
the recursion-stop (count stays at 1).
---
## 6. Realtime signing-key migration notes
- **Two-phase**, as recommended. `0025_encrypt_realtime_keys.sql` adds
NULL-able `realtime_signing_key_encrypted` + `realtime_signing_key_nonce`
and `DROP NOT NULL` on the plaintext column (so new keys can be stored
encrypted-only).
- **Repo:** `PostgresAppSecretsRepo` now holds the `MasterKey`. New keys
are written encrypted-only; the read path (`signing_key` /
`get_or_create_signing_key`) prefers the encrypted columns and falls
back to plaintext during the compat window (pure `decode_signing_key`
helper, unit-tested for all four precedence states).
- **Startup task:** `migrate_plaintext_keys()` runs once in `build_app`
(after the master key is loaded), encrypting any rows that still have
plaintext but no encrypted value. Plaintext is **left in place** for
rollback safety. Idempotent.
- **Plaintext column drop:** deferred to **v1.1.8** (documented in
CHANGELOG + the migration). Operators must upgrade through v1.1.7
(which performs the encryption) before v1.1.8.
- SSE keeps working: `RealtimeAuthorityImpl` is unchanged (it calls
`signing_key`). Verified by the pubsub e2e + unit tests; the dev DB
applied 0025 + the startup encryption cleanly during the test run.
---
## 7. Decisions beyond the brief / deviations flagged
1. **`inbound_secret` stored ENCRYPTED (user-approved deviation).** The
brief defaulted to a plaintext `inbound_secret` column on
`email_trigger_details`; the user chose to encrypt it via the master
key. Implemented: `0024` stores `inbound_secret_encrypted` +
`inbound_secret_nonce`; the admin endpoint seals the secret (as a JSON
string, via the secrets `seal` helper); the receiver `open`s it per
inbound POST to verify the HMAC. **Trade-off:** one AES-GCM decrypt per
inbound request on the hot path — negligible vs. the HMAC + DB
round-trip already there. The decrypted secret is never logged.
2. **Brief-internal contradiction flagged, not reinterpreted — §6
`TriggerEvent::DeadLetter` field names.** The brief's §6 sketches the
payload as `{source, op, original_event_id, original_payload,
attempt_count, last_error, …}`. The actual variant
(`crates/shared/src/trigger_event.rs`) is `{dead_letter_id, original:
Box<TriggerEvent>, attempts, last_error, trigger_id, script_id,
first_attempt_at, last_attempt_at}`. I built the payload from the
**real** variant (which the brief itself instructs to "verify
serializes correctly"). No type change needed.
3. **`build_app` signature gained a `MasterKey` parameter.** Rather than
sourcing the key inside `build_app` (which would force every e2e test
to set process env), `main.rs` sources it and passes it in. The 3
existing `build_app` test callers pass a fixed test key.
4. **Pre-existing clippy warnings fixed (see §10).** Four warnings predate
this work; I fixed them in a dedicated commit so the `-D warnings`
gate is green, and flag them as a latent finding.
5. **Email-trigger retry settings** use the standard async defaults
(3 attempts, exponential, 1000 ms) — the brief didn't specify; matches
the cron/kv default shape.
No other deviations from prompt-specified defaults.
---
## 8. How to verify locally — §8 attestation (sourced from cargo's literal output)
All gates run on the handed-back HEAD (`a7d3dad`):
```sh
cargo fmt --all -- --check # clean
cargo clippy --all-targets --all-features -- -D warnings # clean (exit 0)
cd dashboard && npm run check # 0 ERRORS 0 WARNINGS (371 files)
```
Full test run **with `DATABASE_URL` set** so the DB-gated suites
(schema_snapshot, dispatcher_e2e ×9, email_inbound ×8) execute:
```sh
DATABASE_URL='postgres://picloud:picloud@127.0.0.1:15432/picloud' \
cargo test --workspace -- --test-threads=2
```
**Pass count, summed from cargo's literal output (NOT hand-counted):**
```sh
DATABASE_URL=... cargo test --workspace -- --test-threads=2 2>&1 | \
awk '/test result: ok\./ { gsub(";", ""); sum += $4 } END { print sum }'
# => 617
```
**617 passed, 0 failed** across the workspace (34 `test result:` lines,
0 `FAILED`). Largest binaries: 290 (manager-core lib), 74, 43, 32, 30;
plus `dispatcher_e2e` (9) and `email_inbound` (8).
**Bounded-parallelism note (`--test-threads=2`):** the picloud e2e
binaries each call `build_app`, which opens its own Postgres pool. Under
full default parallelism against the *shared dev* Postgres, ~9 concurrent
`build_app`s exhaust connections and a couple of e2e tests flake on
timeout (observed: `dispatcher_delivers_pubsub_to_handler`,
`dead_letter_handler_failure_does_not_recurse`). They pass reliably at
`--test-threads=2` and in isolation. CI's dedicated fresh `postgres:15`
(not a shared dev DB) does not hit this. Environmental, not a correctness
issue — flagged so the reviewer runs the DB-gated suite with bounded
parallelism (or on CI).
**Migrations:** apply cleanly on the v1.1.6 dev DB (0023→0025 applied
during the test run) and the schema-snapshot guardrail passes after
re-bless. The `BLESS` diff was exactly the new tables/columns/constraints
(secrets, email_trigger_details, app_secrets encrypted columns +
NULL-able plaintext, widened kind/source CHECKs, migrations 00230025) —
no unrelated drift.
**Manual smoke:** the e2e suite covers secrets set/get/delete/list,
inbound signed POST → handler fires with `ctx.event.email`, dead-letter
handler fires, realtime-key encryption + SSE. Outbound email to a live
relay (mailtrap) was NOT exercised (no SMTP configured in this
environment) — asserted instead via recording-transport unit tests
(To/From/Subject/body, multipart parts, cc/bcc, reply_to).
---
## 9. Open questions for the reviewer
1. **§8 bounded-parallelism caveat** — acceptable, or should the e2e
harness share a single `build_app`/pool across tests in a binary?
(Out of v1.1.7 scope; the existing v1.1.6 e2e tests have the same
shape.)
2. **`email::send` ignoring a stray `html` key** (forcing text-only) vs.
throwing — I chose forgiving text-only; happy to make it strict.
3. **Inbound `received_at`** is stamped by the receiver (`Utc::now()`),
not read from a provider header — confirm that's the intended
semantics.
---
## 10. Latent security / correctness findings
1. **`clippy --all-targets --all-features -- -D warnings` did NOT pass at
v1.1.6 HEAD** (verified by stashing this branch and re-running clippy
on the committed slice-1 tree). Four pre-existing warnings:
`double_must_use` on `realtime_router`, `map_unwrap_or` in
`pubsub_service`, `redundant_closure` in `topic_repo`,
`needless_raw_string_hashes` in a subscriber-token test. Fixed all four
(commit `2ea47eb`) so the gate is now green — flagging because it means
prior "clippy green" claims were likely run without `--all-targets`
(which compiles the test binaries).
2. **Inbound HMAC fails closed on decrypt error.** If a stored
`inbound_secret` can't be decrypted (e.g. `PICLOUD_SECRET_KEY`
rotated), the receiver returns 401 — it refuses the POST rather than
silently skipping verification. Intentional.
3. **No rate limiting on the public inbound-email endpoint.** Like every
public data-plane route, `/api/v1/email-inbound/...` is
unauthenticated by design (URL + HMAC are the gate). An unsigned
trigger (no `inbound_secret`) accepts any POST to its URL and enqueues
outbox rows — URL secrecy is the only guard, as documented. Mitigation
is operator-level (Caddy) rate limiting, the same answer as for other
public routes; no new gap introduced, but noted.
---
## 11. Deferred items (unchanged from brief)
Master-key rotation / per-app master key (v1.2); native SMTP listener
(v1.3+); provider-specific inbound unmarshallers, inbound attachments,
outbound SMTP connection pooling, per-app `from` validation / SPF / DKIM
(v1.2 / operator); dashboard inbound payload viewer (v1.2, PII); drop the
plaintext `realtime_signing_key` column (v1.1.8); secrets
versioning/history + secrets-change triggers (never); `users::*` (v1.1.8);
`queue::*` / `invoke()` (v1.1.9).
---
## 12. Known limitations
- Production `EmailTransport` is a per-call connection; high outbound
volume is connection-churn-bound until pooling (v1.2).
- Outbound `email::send` was not smoke-tested against a live relay in
this environment (no SMTP configured); the SMTP message contents are
asserted via recording-transport unit tests.
- The §8 DB-gated run requires bounded parallelism on a shared Postgres
(see §8); CI's dedicated Postgres does not.

382
HANDOFF.md Normal file
View File

@@ -0,0 +1,382 @@
# Handoff — 2026-06-05
Machine-switch handoff. This document is the entry point for picking up
PiCloud work on a different machine. It captures session state, what
shipped, what's queued, and how to continue.
---
## TL;DR
- **`main` is at v1.1.7** — seven minor releases (v1.1.1 → v1.1.7)
shipped this session via the dispatch-and-review workflow.
- Working tree is clean.
- Next release is **v1.1.8** (User Management). A draft dispatch prompt
is sketched in §6 below; ready to send to a dev agent.
- One dev Postgres container (`picloud-postgres-1` on port 15432) is
still running on the source machine — tear it down with
`docker compose down -v` before the source machine goes offline.
---
## How to resume on the new machine
```sh
git clone https://git.mc02.dev/fabi/PiCloud.git
cd PiCloud
git checkout main
git log --oneline -10 # should show v1.1.7 reviewer commit at HEAD
docker compose up -d # local Postgres for DB-gated tests
export DATABASE_URL='postgres://picloud:picloud@127.0.0.1:5432/picloud'
cargo test --workspace -- --test-threads=2
```
If you're starting from this branch (`handoff/2026-06-05`), it points at
the same `main` HEAD with this `HANDOFF.md` added; merge or just read it
and continue work on `main`.
For the master encryption key needed by v1.1.7+ secrets:
```sh
export PICLOUD_SECRET_KEY="$(openssl rand -base64 32)"
# OR, for dev only:
export PICLOUD_DEV_MODE=true
```
The dev fallback uses a deterministic key (`SHA-256` of a hardcoded
string) — fine for local testing, fatal for any real deployment.
---
## Session summary: v1.1.1 → v1.1.7
All seven minor releases completed in one session via the dispatch
workflow you set up: I draft a prompt, you dispatch it to a fresh
agent in another session, the agent implements and writes `HANDBACK.md`,
you bounce the report back to me, I audit the branch and write
`REVIEW.md` with a verdict, you bounce-back-for-fixes-if-needed, and on
approve I fast-forward merge into `main`.
| Release | Capability | Iterations | Status |
|---|---|---|---|
| **v1.1.1** | Storage & Events (KV + triggers framework + outbox + dispatcher + NATS-style sync HTTP + dead-letter table + dashboard surface) | 1 | ✅ merged |
| **v1.1.2** | Documents (`docs::*` SDK + query DSL + `docs:*` triggers) | 2 | ✅ merged (iteration 2 fixed a fmt diff) |
| **v1.1.3** | Modules (`scripts.kind` + `PicloudModuleResolver` + AST caches + `script_imports`) | 1 | ✅ merged |
| **v1.1.4** | Outbound HTTP & Scheduled Tasks (`http::*` with SSRF deny-list + cron triggers) | 1 | ✅ merged |
| **v1.1.5** | Files & Pub/Sub (filesystem-backed blobs + `pubsub::publish_durable` + first CI workflow) | 1 | ✅ merged |
| **v1.1.6** | Realtime Channels & Client Library (SSE + topics + HMAC subscriber tokens + `@picloud/client@1.0.0`) | 1 | ✅ merged |
| **v1.1.7** | Configuration & Email (encrypted secrets + outbound/inbound email + dead-letter handler fix) | 1 | ✅ merged |
**Versioning state on `main`:**
- Workspace `1.1.7`
- SDK schema `1.8`
- Dashboard `0.13.0`
- `@picloud/client` `1.0.0`
- Migrations applied through `0025`
**Test counts at HEAD:** `cargo test --workspace --test-threads=2` with
`DATABASE_URL` set → **617 passed / 0 failed**. The `--test-threads=2`
is required on shared dev Postgres (~9 concurrent `build_app`s
otherwise exhaust connections); CI's dedicated Postgres doesn't hit
this.
---
## Branches on this machine
### v1.1.x feature branches (all merged into main, kept locally for traceability)
| Branch | HEAD | What it contains |
|---|---|---|
| `feat/v1.1.1-storage-and-events` | `2796f36` | v1.1.1 work + HANDBACK + REVIEW |
| `feat/v1.1.2-documents` | `5bbbc26` | v1.1.2 work (2 iterations) + HANDBACK + REVIEW |
| `feat/v1.1.3-modules` | `6f17259` | v1.1.3 work + HANDBACK + REVIEW |
| `feat/v1.1.4-http-cron` | `03d03ea` | v1.1.4 work + HANDBACK + REVIEW |
| `feat/v1.1.5-files-pubsub` | `d064681` | v1.1.5 work + HANDBACK + REVIEW |
| `feat/v1.1.6-realtime-client` | `64ad978` | v1.1.6 work + HANDBACK + REVIEW |
| `feat/v1.1.7-secrets-email` | `5cbb6ca` | v1.1.7 work + HANDBACK + REVIEW |
All seven HEADs are reachable from `main` (fast-forward merges). Keeping
the branches makes it easy to inspect the per-release commit slice
without git log filtering.
### Older branches predating this session (state uncertain)
These appeared in `git branch` at session start and weren't touched by
v1.1.x work. I don't know which are abandoned, in-flight, or already
merged under different names. **On the new machine, decide for each:**
| Branch | Last commit | Tracking |
|---|---|---|
| `chore/ui-hardening` | `b42e273 fix(test): admin_is_implicit_app_admin uses force=true on app delete` | local-only |
| `feat/app-members` | `e6fc6e6 test(picloud): close two app_members test gaps` | local-only |
| `feat/cli` | `5d08974 style(cli): re-fmt one stray format! line in the integration test` | tracks `origin/feat/cli` (up to date) |
| `feat/multi-app-scoping` | `a393f11 feat(dashboard): auto-slug app names and infer route host kind from input` | tracks `origin/feat/multi-app-scoping` (ahead 3) |
| `feat/users-and-keys-ui` | `6eb32a7 feat(dashboard): adopt ActionMenu for user row actions` | local-only |
| `feat/users-authz` | `2aab92a style: cargo fmt across Phase 3.5 changes` | local-only |
| `test/cli-journeys` | `e4851b3 test(cli): extract shared Fixture into tests/common` | tracks `origin/test/cli-journeys` (up to date) |
| `test/frontend-e2e` | `ec3c768 test(dashboard): add full-stack integration specs` | local-only |
**Push these if you want them mirrored on the new machine** — see §3
below for the push commands. If any are obsolete, delete them locally
before resuming.
---
## §3 — Push instructions
Push was denied in this session (sandbox restriction). Run these on the
source machine to mirror state to `origin`:
```sh
# 1. The v1.1.x releases on main (55 commits)
git push origin main
# 2. The seven v1.1.x feature branches (preserves per-release history)
git push origin feat/v1.1.1-storage-and-events
git push origin feat/v1.1.2-documents
git push origin feat/v1.1.3-modules
git push origin feat/v1.1.4-http-cron
git push origin feat/v1.1.5-files-pubsub
git push origin feat/v1.1.6-realtime-client
git push origin feat/v1.1.7-secrets-email
# 3. This handoff branch
git push -u origin handoff/2026-06-05
# 4. OPTIONAL — push the older branches you want on the new machine
# (decide per-branch; some may be abandoned)
git push origin chore/ui-hardening
git push origin feat/app-members
git push origin feat/multi-app-scoping # ahead 3 of remote
git push origin feat/users-and-keys-ui
git push origin feat/users-authz
git push origin test/frontend-e2e
```
After pushing, on the new machine: `git fetch --all` brings everything
down. `git checkout main` puts you at v1.1.7 HEAD.
---
## §4 — Workflow context (read before dispatching v1.1.8)
The dispatch-and-review workflow you've been using:
1. **You ask me to draft the dispatch prompt** for the next release.
2. **I draft the prompt** based on:
- The roadmap in [`docs/v1.1.x-design-notes.md` §7](docs/v1.1.x-design-notes.md)
- Three or so follow-ups identified in the prior release's REVIEW.md
- Discipline lessons carried forward from prior retros
3. **You dispatch the prompt to a fresh agent in another session**
that agent gets no prior conversation context; the prompt + the
docs it points at are everything they have.
4. **The agent implements + writes `HANDBACK.md`** at the repo root,
then stops.
5. **You bounce the HANDBACK back to me.**
6. **I audit the branch and write `REVIEW.md`** with a verdict
(`APPROVE` or `NEEDS CHANGES`).
7. **If `NEEDS CHANGES`:** you bounce the REVIEW back to the agent;
they iterate; back to step 5.
8. **If `APPROVE`:** I fast-forward merge the branch into `main` and
pause for your next instruction.
What's worked well across seven releases:
- The discipline reminders compound. Each release's retro identifies
one small habit the agent dropped (§8 attestation hand-counting,
silent prompt-default deviations, brief-internal contradictions
silently reinterpreted, clippy run without `--all-targets`); the
next release's prompt explicitly addresses it. By v1.1.7 the agent
was catching their own latent findings without prompting.
- Explicit "deviations beyond the brief" sections in HANDBACK make
audits fast — every meaningful judgment call is in one place.
- The "this is the deferrable piece under scope pressure" clause in
big releases (v1.1.6 client lib, v1.1.7 inbound email) gave the
agent a clean escape hatch they never actually needed but worked
as intended.
- Latent findings discovered during implementation (v1.1.3 cross-app
trigger gap, v1.1.4 SSRF literal-IP bypass, v1.1.6 dead_letter
handler never firing, v1.1.7 clippy regression at v1.1.6 HEAD) all
surfaced honestly rather than being silently worked around.
What to do differently in v1.1.8:
- **Walk through each code example in the prompt** before sending. v1.1.4
brief said `(url, opts)` but its example was `http::post(url, body)`
the agent had to fix it during implementation. v1.1.7 brief sketched
`TriggerEvent::DeadLetter` field names that didn't match the actual
variant. Both flagged correctly, but pre-resolution saves agent
effort.
- **Pin the clippy gate**: `cargo clean` before `cargo clippy
--all-targets` to defeat incremental-cache false-greens. See v1.1.7
REVIEW §3.3 for context.
---
## §5 — Pending follow-ups for v1.1.8
From the v1.1.7 REVIEW.md, three load-bearing items to fold into the
v1.1.8 dispatch prompt:
### 5.1 Drop the plaintext `realtime_signing_key` column
The v1.1.7 phase-2 commitment. v1.1.7 added NULL-able encrypted columns
+ DROP NOT NULL on the plaintext column; the startup task encrypts
existing rows. v1.1.8 drops the plaintext column entirely.
**Pre-flight check:** scan for any remaining non-NULL rows on the
plaintext column. If found, run the encryption migration before the
drop. If the v1.1.7 startup task ran on the operator's deploy, all
rows should already be encrypted.
**CHANGELOG must note** that v1.1.8 requires v1.1.7 to have been
applied first. No skipping versions.
### 5.2 Clippy `--all-targets` discipline refinement
The v1.1.7 audit caught a real regression: four warnings predated
v1.1.7 that the v1.1.6 audit reported as clippy-green. Likely cause:
cargo's incremental cache leaving test binaries unchecked.
v1.1.8 prompt should require either:
- `cargo clean` before `cargo clippy --all-targets`, OR
- Explicit verification that the clippy output includes `Checking`
lines for test crates.
CI's `.github/workflows/ci.yml` (added in v1.1.5) might also benefit
from a clippy-cache-check step.
### 5.3 `auth_mode = 'session'` for realtime subscriber tokens
v1.1.7's CHECK constraint on `topics.auth_mode` only allows
`('public', 'token')`. v1.1.8's `users::*` work needs to:
- Extend the CHECK to include `'session'`.
- Add a session-token validator alongside the existing HMAC validator
behind the unchanged `RealtimeAuthority` trait.
The trait shape from v1.1.6 already supports this — natural extension.
---
## §6 — Draft v1.1.8 dispatch prompt outline
Not the full prompt — just the scope sketch so you can ask me to expand
it on the new machine.
**v1.1.8 — User Management (`users::*`)**
Core scope:
- `users::create / get / find / update / delete / list` SDK
- Password hashing (argon2id)
- `users` table per-app
- Sessions: `users::login(email, password)` → returns a session token;
`users::verify(session_token)` returns the user or `()`
- Sessions table with TTL + revocation
- Email verification flow (uses v1.1.7 email::send)
- Password reset flow (uses v1.1.7 email::send + tokens)
- Invitations (admin creates an invite → email link → user accepts +
sets password)
- Roles: per-app role assignments on users
- `Capability::AppUsersRead/Write/Admin` mapped to existing scopes
- Dashboard: Users tab on app detail page (list, invite, role-edit)
Follow-ups from v1.1.7 retro (fold in):
- Drop plaintext `realtime_signing_key` column (phase-2)
- Clippy `--all-targets` discipline refinement
- `auth_mode = 'session'` for realtime subscriber tokens (uses v1.1.8
sessions)
Out of scope:
- OAuth providers (defer to v1.2+)
- 2FA / MFA (defer to v1.2+)
- SSO / SAML (defer)
- Password policy customization (defer; ship with sensible default)
- User-to-user messaging (defer; userland)
Ask me to expand this into a full prompt when you're ready.
---
## §7 — Environmental notes
- **Dev Postgres container** `picloud-postgres-1` (port 15432) was
running at session end on the source machine. The v1.1.5/v1.1.6/
v1.1.7 agents started it for live DB-gated tests. **Tear down with
`docker compose down -v` before the source machine goes offline**
if you want a clean state.
- **`PICLOUD_SECRET_KEY`** is required for v1.1.7+ to start. Pick one
with `openssl rand -base64 32` for production; use
`PICLOUD_DEV_MODE=true` (no master key needed) for local
development. The dev key is deterministic so secrets persist across
restarts in dev.
- **CI workflow** lives at [`.github/workflows/ci.yml`](.github/workflows/ci.yml)
(added in v1.1.5). Runs fmt + clippy + `cargo test --workspace`
against a `postgres:15` service, plus dashboard `npm run check`.
When you push to `main` or open a PR, CI will run. **First push
after this handoff will exercise it for the first time on real
workload — watch the run.**
---
## §8 — Key documents for orientation
- **[`CLAUDE.md`](CLAUDE.md)** — project conventions. Read first.
- **[`serverless_cloud_blueprint.md`](serverless_cloud_blueprint.md)** —
the authoritative architecture document.
- **[`docs/sdk-shape.md`](docs/sdk-shape.md)** — SDK conventions every
v1.1.x service follows.
- **[`docs/v1.1.x-design-notes.md`](docs/v1.1.x-design-notes.md)** —
the in-flight-decisions document. Sections §1§6 contain the
"Decided 2026-06-01" annotations from the design conversation that
preceded this session; §7 holds the v1.1.x roadmap; §14 are
candidates for pruning (their decisions shipped in v1.1.1).
- **[`docs/versioning.md`](docs/versioning.md)** — patch-bump policy
under the post-1.0 expansion-phase carve-out.
- **[`docs/git-workflow.md`](docs/git-workflow.md)** — trunk-based
workflow conventions.
- **[`CHANGELOG.md`](CHANGELOG.md)** — release notes for v1.1.1 onward.
v1.1.7's entry includes the retroactive dead_letter security note.
Per-release artifacts on `main`:
- `HANDBACK.md` at repo root — currently holds the v1.1.7 agent's
handback. Overwritten each release.
- `REVIEW.md` at repo root — currently holds the v1.1.7 reviewer's
audit. Overwritten each release.
If you want the full per-release HANDBACK + REVIEW history, the seven
`feat/v1.1.x-*` branches preserve them (each branch's `HEAD~1`
contains the HANDBACK and `HEAD` contains the REVIEW for that release).
---
## §9 — Quick smoke after resuming
After cloning + setting up the new machine:
```sh
# Basic gates
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --workspace
# DB-gated (needs Postgres)
docker compose up -d
export DATABASE_URL='postgres://picloud:picloud@127.0.0.1:5432/picloud'
export PICLOUD_DEV_MODE=true
cargo test --workspace --test-threads=2
# Dashboard
cd dashboard && npm install && npm run check
# Client library
cd clients/typescript && npm install && npm run lint && npm run test && npm run build
```
If all green: machine is ready. Resume v1.1.8 work by asking me for
the full dispatch prompt.
---
**Handoff written 2026-06-05.** Main HEAD: `5cbb6ca` (v1.1.7 reviewer
APPROVE).

227
REVIEW.md
View File

@@ -1,162 +1,183 @@
# v1.1.4 Audit & Review # v1.1.7 Audit & Review
**Branch:** `feat/v1.1.4-http-cron` **Branch:** `feat/v1.1.7-secrets-email`
**Base:** `main` (v1.1.3 head) **Base:** `main` (v1.1.6 head)
**Commits ahead:** 2 (1 substantive + handback) **Commits ahead:** 10 (8 substantive + 1 chore-clippy-fix + 1 handback)
**HEAD audited:** `6080fc6` **HEAD audited:** `3cfb795`
**Audited by:** reviewer (this report) **Audited by:** reviewer (this report)
**Audited against:** the v1.1.4 dispatch prompt + the v1.1.1v1.1.3 patterns it mandated **Audited against:** the v1.1.7 dispatch prompt + the v1.1.1v1.1.6 patterns it mandated
**Iterations:** 1 **Iterations:** 1
## Verdict ## Verdict
**APPROVE — ready to merge to `main` as v1.1.4.** **APPROVE — ready to merge to `main` as v1.1.7.**
The SSRF implementation is the substance of this release, and it is genuinely well-built — DNS rebinding defense via reqwest's resolver hook + literal-IP check at URL-validation time + per-hop re-validation on redirects + IPv4-mapped IPv6 re-check, with error strings that never leak the resolved address. The cron scheduler correctly implements the fire-once catch-up policy. All three v1.1.3 follow-ups landed. Static gates green; live-DB smoke went beyond the brief's "Done" criteria. Substantial release: encrypted per-app secrets, outbound + inbound email, the long-overdue dead-letter handler wiring fix, and the realtime signing key encryption migration. All scope-in items shipped (inbound email — the deferrable-under-scope-pressure piece — was implemented in full, not deferred). 617 tests pass via awk-summed cargo output (§8 attestation discipline from the v1.1.6 retro landed). Gates green.
Two divergences from the brief, both flagged explicitly by the agent: a three-arg `verb(url, body, opts)` HTTP shape (resolves a self-contradiction in the brief) and a stale schema-snapshot golden re-blessed (pre-existing drift from v1.1.1v1.1.3, surfaced and fixed). Both calls are correct. Three flagged items in HANDBACK §7/§9/§10, all transparent and correct calls:
The agent's discipline carried over cleanly from the v1.1.3 retro: every deviation from a prompt-default is called out explicitly in HANDBACK §7. The §8 attestation is taken on the implementation commit with an explicit note that the HANDBACK commit is pure markdown — same pattern as v1.1.3, acceptable. 1. **Brief-internal contradiction on `TriggerEvent::DeadLetter` field names** — agent built from the real variant (which the brief itself said to "verify serializes correctly"). The v1.1.6 retro discipline lesson (flag-don't-reinterpret) working again.
2. **`inbound_secret` stored encrypted** — user-approved deviation during planning. The brief recommended plaintext for hot-path latency reasons; encryption was the user's call. Trade-off honest (one AES-GCM decrypt per inbound POST, negligible vs the HMAC + DB round-trip already there).
3. **Latent finding: clippy `--all-targets` didn't pass at v1.1.6 HEAD** — four pre-existing warnings the previous gate runs missed (likely run without `--all-targets`). Fixed in a dedicated commit. **This is a real audit finding that affects every prior REVIEW.md from v1.1.1 onward.**
The dead-letter handler wiring bug from v1.1.1 (six releases) is finally fixed, with regression tests that assert handler-fire (not just row-creation).
--- ---
## 1. Static checks reproduced (HEAD `6080fc6`) ## 1. Static checks reproduced (HEAD `3cfb795`)
``` ```
cargo fmt --all -- --check ✅ exit 0 cargo fmt --all -- --check ✅ exit 0
cargo clippy --all-targets --all-features -- -D warnings ✅ exit 0 cargo clippy --all-targets --all-features -- -D warnings ✅ exit 0 (now actually green; see §5)
cargo test --workspace 427 passed / 0 failed cargo test --workspace (DATABASE_URL set, --test-threads=2)617 passed / 0 failed
+ 140 ignored (Postgres-gated)
``` ```
Per-suite test counts (delta from v1.1.3 baseline): Sum via the v1.1.7 discipline awk pattern:
- manager-core: 184 (was 131 → +53; SSRF policy 20 + HTTP client 16 + cron scheduler 11 + cron admin 6)
- executor-core/tests/sdk_http: 15 (NEW — bridge integration)
- executor-core/tests/sdk_docs: 15 (unchanged)
- executor-core/tests/modules: 23 (+0; one new redaction test bundled here)
- executor-core/tests/module_redaction_logging: 1 (NEW — captures the tracing subscriber)
- orchestrator-core: 62 (unchanged)
- stdlib: 43 (unchanged)
- sdk_contract: 30 (unchanged)
- picloud: 21 (unchanged)
- executor-core engine: 17 (unchanged)
- shared: 9 (unchanged)
- sdk_kv: 7 (unchanged)
69 net new tests (HANDBACK claims 70 — one test likely got renamed/moved; immaterial). Comfortably above the "50-70" brief target. ```sh
cargo test --workspace 2>&1 | awk '/test result: ok\./ { gsub(";", ""); sum += $4 } END { print sum }'
# => 617
```
Matches HANDBACK §8 exactly. **The §8 discipline refinement from the v1.1.6 retro is working.**
The bounded `--test-threads=2` is required on shared-dev Postgres (~9 concurrent `build_app`s exhaust connections) but not on CI's dedicated Postgres. Acceptable environmental nuance; flagged in HANDBACK §8.
## 2. Design conformance (spot-checks) ## 2. Design conformance (spot-checks)
| Decision / requirement | Where it lives | Verdict | | Decision / requirement | Where it lives | Verdict |
|---|---|---| |---|---|---|
| **SSRF deny-list covers every prompt CIDR** | [ssrf.rs:65-110](crates/manager-core/src/ssrf.rs#L65-L110) | ✅ All 13 prompt-specified ranges + `0.0.0.0/8` ("unspecified") + `::` (defensible superset) | | **AES-256-GCM with 12-byte CSPRNG nonce + 16-byte appended auth tag** | [shared/src/crypto.rs:71-85](crates/shared/src/crypto.rs#L71-L85) | ✅ Uses `aes-gcm 0.10`; nonce from `rand::thread_rng().fill_bytes`; RustCrypto Aead layout (tag appended) |
| **Policy applied to RESOLVED IP, not hostname (DNS rebinding defense)** | [SsrfResolver::resolve](crates/manager-core/src/ssrf.rs#L181-L221) plugged via `ClientBuilder::dns_resolver` | ✅ Filter runs at every connection (incl. each redirect hop, since redirects are followed manually). Test `dns_rebinding` exercises a mock resolver that returns public-then-private. | | `MasterKey` redacts Debug; cheap to clone | shared/src/crypto.rs MasterKey impl | ✅ Per HANDBACK §2 |
| **Literal-IP gap closed** (reqwest bypasses resolver for literal IPs) | [http_service.rs:303 validate_url](crates/manager-core/src/http_service.rs#L303) | ✅ Excellent catch — the agent identified this and added the parallel literal-IP check at URL-validation time, on every hop | | `PICLOUD_SECRET_KEY` required (fatal if missing); dev-mode fallback requires explicit `PICLOUD_DEV_MODE=true` | crypto.rs MasterKey::from_env + resolve | ✅ No quiet "unencrypted mode" path |
| **IPv4-mapped IPv6 re-check** | [ssrf.rs:87-92 check_v6 → to_ipv4_mapped → check_v4](crates/manager-core/src/ssrf.rs#L87-L92) | ✅ `::ffff:127.0.0.1` correctly denied as "loopback" | | `MasterKey` threaded into `build_app` (test-friendly) | [picloud/src/lib.rs:build_app](crates/picloud/src/lib.rs) | ✅ Parameter, not env-sourced — tests can pass a fixed key |
| **Script-visible error never leaks the IP** | [ssrf.rs:118-129 SsrfBlocked + reason categories](crates/manager-core/src/ssrf.rs#L118-L129) | ✅ Reason is a CIDR category (`loopback` / `private` / `link-local` / `carrier-grade-nat` / `multicast` / `reserved` / `unique-local` / `unspecified`); the resolved address is never serialized into the error | | 64 KB plaintext cap per secret | secrets_service::seal | ✅ `PICLOUD_SECRET_MAX_VALUE_BYTES` override |
| **Scheme restrictions (http/https only)** | `validate_url` | ✅ `file://`, `ftp://`, `gopher://` rejected | | Generic GCM auth-failure error (no wrong-key vs tampered distinction) | crypto.rs CryptoError::Decrypt | ✅ By design — leaking which failure case happened weakens the integrity guarantee |
| **Port restrictions (22, 25, 465, 587)** | `validate_url` | ✅ | | `secrets` table with `(app_id, name)` PK, encrypted bytea + 12-byte nonce | [0023_secrets.sql](crates/manager-core/migrations/0023_secrets.sql) | ✅ |
| **Body size caps (request + response, env-overridable)** | [http_service.rs HttpConfig](crates/manager-core/src/http_service.rs#L54-L90) | ✅ 10 MB default; `PICLOUD_HTTP_MAX_REQUEST_BODY_BYTES` / `PICLOUD_HTTP_MAX_RESPONSE_BODY_BYTES`; response cap stream-with-Content-Length, fallback to mid-stream check | | `secrets::*` SDK — collection-less, JSON type round-trip | [executor-core/src/sdk/secrets.rs](crates/executor-core/src/sdk/secrets.rs) + secrets_service.rs | ✅ String comes back as String (not JSON-quoted) |
| **Timeout layering (30s default / 60s max for total; 10s connect)** | DEFAULT_TIMEOUT_MS / MAX_TIMEOUT_MS / CONNECT_TIMEOUT | ✅ Above-max rejected (not silently clamped); test covers this | | Cross-app isolation in secrets | secrets_service via `cx.app_id` | ✅ Test asserts |
| **Default User-Agent `picloud/<v> (script:<id>)`** | `build_headers` (paraphrased from grep) | ✅ The agent added `script_id` to `SdkCallCx` to make this work — flagged explicitly as a decision in HANDBACK §7 #4 | | `Capability::AppSecretsRead/Write``script:read/write` | manager-core::authz | ✅ Seven-scope commitment held |
| **`PICLOUD_HTTP_ALLOW_PRIVATE` disables deny-list entirely + startup warning** | HttpConfig + picloud binary | ✅ | | No `ServiceEvent` emission for secret writes | secrets_service | ✅ Per brief — secret-change triggers are a footgun |
| **`Capability::AppHttpRequest(AppId)` mapped to `script:write`; script-as-gate** | [http_service.rs:147-158](crates/manager-core/src/http_service.rs#L147-L158) | ✅ Anonymous cx skips; member with role allowed; member without forbidden. Seven-scope commitment held — no new `Scope` variants | | Outbound email via `lettre 0.11`, per-call connection model | manager-core::email_service | ✅ Pooling deferred to v1.2 per brief |
| **HttpService trait pattern matches v1.1.1+ services** | shared::http::HttpService + manager-core::http_service::HttpServiceImpl | ✅ Same shape as KvService, DocsService | | Disabled mode when SMTP env vars missing | EmailServiceImpl::from_env | ✅ Startup warn; every `send` returns `NotConfigured` |
| **Cron Layout-E extension (migration 0017)** | [0017_cron_triggers.sql](crates/manager-core/migrations/0017_cron_triggers.sql) | ✅ Mirrors 0014's CHECK-widen + detail-table pattern exactly | | `email::send_html` builds MultiPart alternative_plain_html | email_service.rs send_html path | ✅ |
| **Cron scheduler: fire-once catch-up policy** | [cron_scheduler.rs:66-82 next_due](crates/manager-core/src/cron_scheduler.rs#L66-L82) | ✅ Returns single next slot via `Schedule::after(&base).next()`; after firing `last_fired_at = now` re-anchors; test `catch_up_fires_exactly_once_after_5_missed_windows` pins this | | `to/cc/bcc` accept String or Array of Strings | sdk/email.rs bridge | ✅ |
| **Scheduler uses ExecutionGate indirectly (enqueues to outbox)** | [cron_scheduler.rs:148-162](crates/manager-core/src/cron_scheduler.rs#L148-L162) | ✅ Scheduler INSERTs to outbox; existing dispatcher acquires gate on consume — same path as kv/docs/dead_letter | | 25 MB message cap, env-overridable | email_service | ✅ `PICLOUD_EMAIL_MAX_MESSAGE_BYTES` |
| **`last_fired_at` transactional with outbox write** | Both INSERTs inside `tx` (`FOR UPDATE OF d SKIP LOCKED` + outbox insert + UPDATE in same tx) | ✅ | | RFC 5322-ish pre-validation + lettre Mailbox parse | email_service::validate | ✅ |
| **Timezone via `chrono-tz`; invalid IANA name → 422 at admin endpoint** | triggers_api.rs cron handler | ✅ Test `unknown_timezone` covers | | Inbound webhook receiver `POST /api/v1/email-inbound/{app_id}/{trigger_id}` | crates/picloud/src/lib.rs or orchestrator-core | ✅ Per [picloud/tests/email_inbound.rs](crates/picloud/tests/email_inbound.rs) test coverage |
| **Cron rejects module targets and cross-app scripts (v1.1.3 regressions)** | `ensure_script_targetable` reused | ✅ Tests `module_target_rejected` + `cross_app_target_rejected` | | Inbound: 202 success, 401 HMAC fail, 404 missing/wrong-kind, 422 malformed | email_inbound.rs tests | ✅ All four status codes pinned by tests |
| **`ctx.event.cron` shape matches the brief** | trigger_event.rs Cron variant; engine.rs serialization | ✅ Schedule, timezone, scheduled_at, fired_at all present | | `email_trigger_details` schema with HMAC secret | [0024_email_triggers.sql](crates/manager-core/migrations/0024_email_triggers.sql) | ✅ |
| **Dispatcher routing extension is a one-line match arm change** | dispatcher.rs (`Kv \| DeadLetter \| Docs \| Cron`) | ✅ | | `TriggerEvent::Email` shape: from/to/cc/subject/text/html/received_at/message_id | trigger_event.rs | ✅ |
| **§10a Module backend error redaction** | [module_resolver.rs:334-349](crates/executor-core/src/module_resolver.rs#L334-L349) | ✅ Script-visible string is the generic `"module backend unavailable; check server logs"`; original logged at error level. Test `module_redaction_logging.rs` verifies the log path captures the original. | | **Dead-letter handler fix: `list_matching_dead_letter` called from `dispatcher::handle_failure`** | [dispatcher.rs:498-501 + fan_out_dead_letter](crates/manager-core/src/dispatcher.rs#L498-L501) | ✅ Wired exactly as specified; built from the real `TriggerEvent::DeadLetter` variant |
| **§10b rhai exact pin** | [Cargo.toml workspace deps](Cargo.toml) | ✅ `rhai = { version = "=1.24", features = ["sync", "serde"] }` | | Recursion-stop preserved: handler failures don't re-dead-letter | dispatcher.rs `is_dead_letter_handler` short-circuit at top of handle_failure | ✅ No new guard needed — the existing flag fires before reaching the exhaustion branch |
| **§10c CHANGELOG retroactive security note** | CHANGELOG.md v1.1.3 section | ✅ (per HANDBACK; I didn't re-read CHANGELOG end-to-end, but the agent claims it) | | Best-effort fan-out: lookup/insert failures logged, not propagated | fan_out_dead_letter at dispatcher.rs:541-545 + 562-565 | ✅ Dead-letter row durably written; handler fan-out is secondary |
| **Versions: workspace 1.1.3→1.1.4, SDK 1.4→1.5, dashboard 0.9.0→0.10.0** | Cargo.toml + shared/src/version.rs + dashboard/package.json | ✅ All bumped | | **Two-phase realtime key migration: encrypted columns added NULL-able + plaintext kept** | [0025_encrypt_realtime_keys.sql](crates/manager-core/migrations/0025_encrypt_realtime_keys.sql) | ✅ DROP NOT NULL on plaintext column; encrypted columns added NULL-able |
| Startup `migrate_plaintext_keys` task encrypts existing rows; idempotent | manager-core::app_secrets_repo | ✅ Per HANDBACK §6; runs once in build_app |
| Decode-side prefers encrypted, falls back to plaintext during compat window | `decode_signing_key` helper, unit-tested for all four precedence states | ✅ |
| Plaintext column drop deferred to v1.1.8 + documented | CHANGELOG + migration header | ✅ |
| Versions: workspace 1.1.6→1.1.7, SDK 1.7→1.8, dashboard 0.12.0→0.13.0 | Cargo.toml + version.rs + package.json | ✅ All bumped |
| Migrations 0023→0025 sequential | migrations/ | ✅ |
| Dashboard: Secrets tab + email trigger form + npm run check clean | dashboard/src/routes/apps/[slug]/+page.svelte | ✅ Per HANDBACK |
## 3. Substantive strengths ## 3. The three flagged items
**1. The literal-IP discovery is the kind of finding that justifies code review.** The prompt called for a DNS-resolver hook (which the agent implemented correctly), but the agent noticed *during implementation* that reqwest only routes hostnames through the custom resolver — a URL with a literal IP host bypasses it entirely. They added a parallel literal-IP check at `validate_url` time, applied on every hop including post-redirect. Test coverage exists for both paths (resolver and literal). This is exactly the kind of independent verification that distinguishes serious security work from box-ticking. ### 3.1 Brief-internal contradiction: `TriggerEvent::DeadLetter` field names (HANDBACK §7 #2)
**2. SSRF error format discipline.** The error reasons are stable CIDR categories (`loopback`, `private`, `link-local`, `unique-local`, `carrier-grade-nat`, `multicast`, `reserved`, `unspecified`) — never the resolved IP. A script that probes `http://internal-host.example.com/` and gets back `"http: blocked by SSRF policy: private"` learns the policy fired, not which RFC1918 range hides behind that hostname. The internal network shape stays opaque to the attacker. The brief's §6 sketched the payload as `{source, op, original_event_id, original_payload, attempt_count, last_error, ...}`. The actual variant in `crates/shared/src/trigger_event.rs` is `{dead_letter_id, original: Box<TriggerEvent>, attempts, last_error, trigger_id, script_id, first_attempt_at, last_attempt_at}`.
**3. The `SsrfBlocked` marker propagation pattern.** The resolver wraps "all addresses denied" in a marker error; the HTTP service walks the reqwest error source chain looking for the `SSRF_BLOCK_PREFIX` to distinguish "policy block" from "generic DNS error" and surface a clean `HttpError::Ssrf`. Without this, all-addresses-denied would surface as `"DNS resolution failed"` which is misleading. Subtle and correct. The agent built from the real variant (which the brief itself said to "verify serializes correctly") and flagged the contradiction rather than silently reinterpreting.
**4. Per-hop re-validation on redirects.** Redirects are followed manually (`redirect(Policy::none())`) so the per-request `follow_redirects` / `max_redirects` are honored AND every hop re-runs through `validate_url` (literal-IP + scheme + port) AND every connection re-runs through the resolver. A naive implementation would validate once at the entry URL and follow reqwest's automatic redirects — that's exploitable by a 301 to `http://10.0.0.1`. The agent didn't fall into this trap. **Verdict: correct call.** The v1.1.6 retro discipline lesson (flag-don't-reinterpret on brief-internal contradictions) is paying dividends — this is the second time it's caught a brief-vs-code mismatch and produced the right outcome. Worth folding into the v1.1.8 prompt: walk through each example in this prompt and verify against the actual code shape before sending.
**5. Cron fire-once catch-up is correctly implemented.** `next_due` returns the single next slot after `last_fired_at`, not a range. After firing, `last_fired_at = now` re-anchors. A trigger that missed 5 windows wakes up exactly once on the next scheduler tick. The test `catch_up_fires_exactly_once_after_5_missed_windows` pins it. This matches the brief and avoids the thundering-herd-on-restart anti-pattern. ### 3.2 `inbound_secret` stored encrypted (HANDBACK §7 #1)
**6. The transactional cron-tick pattern.** `FOR UPDATE OF d SKIP LOCKED` + outbox insert + `last_fired_at` UPDATE all in one tx. A scheduler crash mid-tick rolls back both the enqueue and the bump; the next tick sees the row un-fired and re-tries. Cluster mode (multiple schedulers) wouldn't double-fire because the row is locked. No correctness issues I can construct. User-approved deviation during planning per the user's summary message. The brief recommended plaintext storage for hot-path latency reasons; the user chose to encrypt via the same master-key infrastructure.
**7. The agent's discipline carryover from the v1.1.3 retro.** Every prompt-default deviation is called out explicitly in HANDBACK §7: the three-arg API shape, the `chrono-tz` choice (which the brief left open but worth pinning), the `SdkCallCx::script_id` addition, the `0.0.0.0/::` defensive superset, the cron crate choice. The v1.1.3 retro lesson stuck. The §8 attestation is correctly taken on the implementation commit with the explicit "this HANDBACK commit is pure markdown" note. **Trade-off honest:** one AES-GCM decrypt per inbound POST (microseconds) vs the HMAC verification + DB lookup already on that hot path (milliseconds). The decrypt is negligible.
## 4. The two flagged divergences (both correct) **Verdict: accept the deviation.** Encryption-at-rest of credentials is the correct default; the brief's plaintext recommendation was a premature optimization. The agent took the right path. The fail-closed behavior on decrypt error (returns 401 if the secret can't be decrypted) is correct — refusing the POST is safer than silently bypassing verification.
### 4.1 Three-arg `verb(url, body, opts)` instead of brief's two-arg ### 3.3 Latent finding: clippy `--all-targets` regression (HANDBACK §10 #1)
The agent diverged from the brief's documented shape and dropped `body_raw`. HANDBACK §7 #1 calls this out explicitly. Why this is correct: This is the most important finding in this review.
The brief's Slack example (`http::post(url, #{ text: "alert" })`) was self-contradictory: The agent verified by stashing v1.1.7 work and re-running clippy on v1.1.6 HEAD with `--all-targets --all-features -- -D warnings` — four pre-existing warnings surfaced:
- Two-arg rule: `(url, opts)``#{ text: "alert" }` would be parsed as `opts` and `text` would throw as an unknown opt key. - `double_must_use` on `realtime_router`
- What the brief actually meant: `#{ text: "alert" }` is the body. - `map_unwrap_or` in `pubsub_service`
- `redundant_closure` in `topic_repo`
- `needless_raw_string_hashes` in a subscriber-token test
The agent resolved the contradiction by promoting body to a positional argument. The shape is now: The warnings landed in v1.1.6 itself (the realtime_router was new). The clippy gate v1.1.6 claimed to pass (and that I personally re-ran during the v1.1.6 audit and reported as exit 0) was apparently run without `--all-targets`, which compiles test binaries. Test-only clippy warnings escape.
- `http::get(url)` / `http::get(url, opts)` — bodyless verbs
- `http::post(url)` / `http::post(url, body)` / `http::post(url, body, opts)` — body verbs
- Body type dispatch: Map/Array → JSON, String → text, `()` → no body
- `opts` vocabulary: `{headers, timeout_ms, follow_redirects, max_redirects}` — no `body`, no `body_raw`
`body_raw` is dropped because raw-text-body just uses the positional String body. Cleaner. The Slack example works unchanged. **This is a real audit oversight.** My v1.1.6 REVIEW.md §1 reported `cargo clippy --all-targets --all-features -- -D warnings ✅ exit 0`. Either the warning count was below the threshold at the moment I ran it (and `2ea47eb`'s introduction of new test code in v1.1.7 tipped it over), or I genuinely missed the warnings. Looking at the four warnings the agent fixed, three are in non-test code (`realtime_router`, `pubsub_service`, `topic_repo`) — those should have failed `--all-targets`.
**Verdict: accept the divergence.** The fix is principled, the unknown-opt-key typo guard stays intact, and the resulting API is simpler than the two-arg form would have been. Worth noting for the v1.1.5 prompt: brief-internal contradictions should be flagged for resolution before dispatch, not silently lived with. **Most likely explanation:** the clippy run during the v1.1.6 audit got compilation caching from an earlier `cargo clippy` (without `--all-targets`) and didn't recompile the test binaries. Cargo's incremental compilation cache + clippy's per-target check interaction can produce false-green results when the lib was clippy-clean but tests weren't recently checked.
### 4.2 Re-blessed stale `expected_schema.txt` golden **Action for the v1.1.8 prompt:** require a clean build before clippy:
The schema-snapshot test is `#[ignore]`'d (needs `DATABASE_URL`), so it never ran in CI. The committed golden was missing `abandoned_executions`, `dead_letters`, `dead_letter_trigger_details`, `docs_*`, etc. — all tables added in v1.1.1, v1.1.2, v1.1.3. The agent re-blessed the file as part of v1.1.4, producing a +217-line diff of which only `cron_trigger_details` + the two widened CHECK constraints are v1.1.4-new. ```sh
cargo clean -p picloud-manager-core picloud-orchestrator-core picloud-executor-core picloud-shared picloud
cargo clippy --all-targets --all-features -- -D warnings
```
The agent flagged this transparently in HANDBACK and recommended lifting `#[ignore]` with a CI DB service so the golden can't drift again. Or simpler: use `cargo clippy --workspace --all-targets --all-features --no-deps -- -D warnings` and verify that the test binary count matches what cargo says it compiled.
**Verdict: accept the re-bless in this PR; act on the follow-up recommendation.** The agent fixed all four warnings in `2ea47eb` and gated v1.1.7 against the re-verified `--all-targets` baseline. Future audits should follow suit.
The drift correction lives on `main` going forward. The deferred work — lift `#[ignore]` once CI has a Postgres service — is the right architectural fix. Worth folding into the v1.1.5 prompt as an explicit small task, since the cost of fixing it has been borne (the golden is current) and the only thing missing is the CI wiring. ## 4. Substantive strengths
## 5. Smaller observations (no action required) **1. The §8 attestation discipline lesson landed cleanly.** v1.1.6 retro called for sourcing the test count from cargo's literal output instead of hand-counting. The v1.1.7 HANDBACK §8 includes the literal awk command + the verified count of 617. My independent re-run matches exactly. Discipline working as designed.
- **Single-commit feature delivery.** The brief suggested split `feat(v1.1.4-http)` / `feat(v1.1.4-cron)` commits, but the two features cross shared files (`Cargo.toml`, `Services` bundle, `version.rs`, etc.) — separable per-theme commits would require interactive hunk staging that the agent's tooling didn't provide. They chose one coherent green commit over broken intermediates, with explicit acknowledgment + invitation to squash/relabel. Acceptable. **2. Encryption infrastructure correctly built.** AES-256-GCM with 12-byte CSPRNG nonces is the textbook GCM configuration. Auth tag appended (RustCrypto Aead trait standard). `Decrypt` error doesn't distinguish wrong-key vs corrupted vs tampered — by design, since GCM's IND-CCA security guarantee depends on attackers not learning *which* failure case happened. `MasterKey`'s redacted `Debug` impl prevents accidental log-leaks. Master key threaded into `build_app` as a parameter (test-friendly; doesn't mutate process env).
- **`SdkCallCx::script_id` addition.** Cross-cutting change (19 construction sites updated) needed because the default User-Agent template `picloud/<v> (script:<id>)` requires the id and the cx didn't carry it. Clean addition; doubles as the audit-attribution key the brief emphasizes. HANDBACK §7 #4 flagged it.
- **`0.0.0.0` and `::` defensive additions.** The brief listed `0.0.0.0/8` (covered) but didn't list `::` (the IPv6 unspecified). The agent added both with reason `"unspecified"`. Defensible superset; minimal additional surface.
- **Live DB smoke went beyond the brief.** The agent stood up dev Postgres on port 15432, applied migrations 0007→0017 against a v1.1.3-era DB, and watched the cron scheduler actually fire against real Postgres (`last_fired_at` advancing at tick cadence; outbox row consumed by dispatcher). This is well-above the "Done looks like" bar — the brief asked for unit tests + integration tests + a manual smoke, and they delivered a live smoke against a real DB.
- **Container left running.** Side effect of the live smoke. The agent flagged this transparently. Run `docker compose down` when convenient; no data loss either way.
## 6. Open questions answered **3. Dead-letter handler fix is faithful and adequately tested.** Six releases of silently-broken triggers, finally connected. The implementation is straightforward (the bug was structural, not logical): after `DeadLetterRepo::insert`, call `list_matching_dead_letter` and INSERT one outbox row per matching trigger. The agent's e2e tests assert handler-fire (not just row-creation), exercise the source-filter dimension, and prove the recursion-stop holds. The retroactive CHANGELOG note from the v1.1.7 prompt is in place.
The HANDBACK §9 raises two open questions: **4. Two-phase realtime key migration done right.** The migration adds NULL-able encrypted columns + DROPs NOT NULL on plaintext (so new keys can be encrypted-only); the application-side migration encrypts existing rows; the read path prefers encrypted but falls back to plaintext during the compat window; the plaintext column drop is deferred to v1.1.8 (documented in CHANGELOG + the migration header). Operator-friendly: rolling deploys work cleanly.
1. **Three-arg HTTP shape** — confirmed acceptable (§4.1 above). **5. Inbound email as webhook receiver was the right architectural call.** Native SMTP listener would have been a multi-week effort (port 25 binding, anti-spam, MX records, deliverability, TLS cert lifecycle). The webhook approach hands deliverability to providers (Mailgun/Postmark/SendGrid/SES) who are good at it, and PiCloud just normalizes the parsed payload. Reasonable v1.1.7 scope.
2. **Stale schema golden re-blessed** — confirmed acceptable (§4.2 above).
No further questions outstanding. **6. Disabled-mode for outbound SMTP.** When SMTP env vars aren't set, every `send` throws `NotConfigured` cleanly. The brief specified this; the agent implemented it cleanly. Avoids the failure mode where a misconfigured email path silently swallows messages.
**7. The agent caught and surfaced the v1.1.6 clippy regression.** This is exactly the latent-finding-discipline the previous retros tried to instill. The fix lives on this branch; the regression is documented; the discipline note for v1.1.8 is the only follow-up.
## 5. Open questions answered
HANDBACK §9 raises three:
1. **§8 bounded-parallelism (`--test-threads=2`)**: environmental, not a correctness issue. Shared dev Postgres has a connection limit; each `build_app` opens its own pool. CI's dedicated Postgres doesn't hit this. **Accept as-is.** A future refactor to share one pool across e2e tests in a binary would be cleaner, but that's a workspace-wide harness change worth doing once for all DB-gated tests, not piecemeal per release. Defer to a dedicated e2e-harness pass.
2. **`email::send` ignoring stray `html` key**: the agent chose forgiving (silently drop `html`); the alternative was strict (throw "unknown field: html for text-only send"). **My read: forgiving is fine.** The signature distinguishes `send` (text-only) from `send_html` (multipart), and a script that accidentally passes `html` to `send` will notice when their recipient sees no formatting. Strict-throwing is also defensible; not worth changing.
3. **Inbound `received_at` stamped by the receiver vs read from provider**: agent stamps with `Utc::now()`. The alternative is reading from provider-specific headers (X-Mailgun-Timestamp, X-Sendgrid-Received-At, etc.), which requires provider unmarshallers that v1.1.7 deferred to v1.2. **Accept as-is.** Reader-stamped is the honest choice when the receiver doesn't know the provider's clock format.
## 6. Smaller observations
- **`build_app` signature gained `MasterKey` parameter (HANDBACK §7 #3).** Threading the key in from `main.rs` instead of sourcing inside `build_app` is correct — tests pass a fixed key and don't mutate process env, which would create test-isolation problems. The 3 existing `build_app` test callers were updated.
- **Email trigger retry defaults (HANDBACK §7 #5).** Standard async defaults (3 attempts, exponential, 1000 ms). Matches kv/docs/files/cron/pubsub. Right call — the brief didn't specify, and consistency with siblings is the right default.
- **The 10-commit split is exemplary.** crypto → secrets → email-outbound → email-inbound → dead-letter fix → realtime-migration → version-bump → clippy-fix → schema-rebless → handback. Each commit independently green. Best commit hygiene in any v1.1.x release.
## 7. Versioning audit ## 7. Versioning audit
| File | Before | After | Status | | File | Before | After | Status |
|---|---|---|---| |---|---|---|---|
| Workspace `Cargo.toml` | 1.1.3 | 1.1.4 | ✅ | | Workspace `Cargo.toml` | 1.1.6 | 1.1.7 | ✅ |
| SDK schema (`shared/src/version.rs`) | 1.4 | 1.5 | ✅ correctly bumped — `HttpService` trait + `HttpRequest/Response/Error` + `TriggerEvent::Cron` added to public surface | | SDK schema (`shared/src/version.rs`) | 1.7 | 1.8 | ✅ correctly bumped — `SecretsService`, `EmailService`, `MasterKey`, `crypto::{encrypt, decrypt}`, `TriggerEvent::Email` added to public surface |
| Dashboard `package.json` | 0.9.0 | 0.10.0 | ✅ | | Dashboard `package.json` | 0.12.0 | 0.13.0 | ✅ |
| Migrations | 0001..0016 | 0017 added | ✅ sequential, no skips | | Migrations | 0001..0022 | 0023..0025 added | ✅ sequential, no skips |
| `rhai` pin | `"1.19"` | `"=1.24"` (workspace deps) | ✅ v1.1.3 follow-up §10b | | CHANGELOG.md | v1.1.6 entry | v1.1.7 entry + retroactive dead_letter security note | ✅ Per prompt |
| CHANGELOG.md | v1.1.3 entry | v1.1.4 entry + retroactive v1.1.3 security note | ✅ §10c done |
## 8. Recommended next steps (post-merge) ## 8. Recommended next steps (post-merge)
1. **Merge** `feat/v1.1.4-http-cron` into `main` (fast-forward; branch is linear ahead). 1. **Merge** `feat/v1.1.7-secrets-email` into `main` (fast-forward; branch is linear ahead).
2. **`docker compose down` when convenient** to tidy up the dev Postgres container the agent left running. 2. **`docker compose down` when convenient** to tear down the dev Postgres container.
3. **Pause** before dispatching v1.1.5 (Files & Pub/Sub). 3. **Pause** before dispatching v1.1.8 (User Management).
4. **For the v1.1.5 dispatch prompt**, consider including: 4. **For the v1.1.8 dispatch prompt**, fold in:
- **Lift `#[ignore]` on the schema-snapshot test** with a CI Postgres service so the golden can't silently drift again (§4.2). This is small, mechanical, and prevents a recurring problem. - **Drop the plaintext `realtime_signing_key` column** (the v1.1.7 phase-2 commitment). Pre-flight check: scan the column for any remaining non-NULL rows; if found, run the encryption migration before the drop migration. Add a CHANGELOG note that v1.1.8 requires v1.1.7 to have been applied first (no skipping versions).
- **Pre-resolve any brief-internal contradictions before dispatch.** The v1.1.4 brief's two-arg `(url, opts)` rule was contradicted by its own Slack example; the agent had to fix it during implementation. For v1.1.5, walk through each example in the prompt and confirm it's parseable under the documented rules before sending. - **Clippy --all-targets discipline refinement** (§3.3 finding). Require either a `cargo clean` before `cargo clippy --all-targets` OR explicit verification that test binaries are being checked. v1.1.6's silent regression shows the gate can produce false-green results under cargo's incremental cache. Specific recommendation: add a CI step that asserts the clippy run touched the test binaries (e.g. count `Checking` lines in the output and verify they include test crates).
- **The literal-IP-bypass pattern** is worth remembering for any v1.1.x service that fronts a network library — if reqwest has this gap, other libraries might too. The pattern: "policy applies to the resolved address, BUT verify the library actually routes literal IPs through your hook before relying on it." - **`auth_mode = 'session'` for realtime subscriber tokens** — v1.1.7's CHECK constraint on `topics.auth_mode` only allows `('public', 'token')`. v1.1.8 (users::*) needs to add `'session'` and a session-token validator alongside the existing HMAC validator behind the unchanged `RealtimeAuthority` trait.
- **Bounded e2e parallelism** — defer the workspace-wide harness refactor (shared pool per binary) until there's a dedicated test-infra release. Until then, CI just needs `--test-threads=2` or smaller for the picloud crate's e2e binaries.
5. **Awareness from §3.3**: the clippy regression in v1.1.6 was caught by v1.1.7's diligence, but every prior REVIEW.md from v1.1.1 onward should be re-checked if you want certainty that no test-only clippy warnings slipped through. The fix is forward-only — re-running clippy on v1.1.1 through v1.1.6 commits would just confirm the warnings were latent then too.
Branch is ready for merge. Verdict: **APPROVE**. Branch is ready for merge. Verdict: **APPROVE**.

3
clients/typescript/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
node_modules/
dist/
*.tsbuildinfo

View File

@@ -0,0 +1,111 @@
# @picloud/client
TypeScript client for [PiCloud](../../README.md). Three capabilities, all
**script-mediated** — there is no direct KV / docs / users access from the
browser (the hybrid model, by design):
1. **Typed HTTP** to dev-defined script endpoints.
2. **SSE realtime** subscriptions to externally-subscribable pub/sub topics.
3. **Auth-flow helpers** over your own dev-defined login/logout endpoints.
```ts
import { PicloudClient } from '@picloud/client';
const client = new PicloudClient({
baseURL: 'https://api.example.com',
getAuthToken: () => localStorage.getItem('auth_token')
});
// Typed HTTP
interface CreateUserReq { name: string; email?: string; role: string }
interface CreateUserRes { id: string; name: string; created_at: string }
const user = await client
.endpoint<CreateUserReq, CreateUserRes>('/api/users')
.post({ name: 'alice', role: 'admin' });
// SSE subscription
const unsubscribe = client.subscribe('chat-room-123', (event) => {
console.log('got event:', event.message);
});
unsubscribe();
// Token-gated topic (token obtained from one of YOUR script endpoints,
// which calls `pubsub::subscriber_token`)
client.subscribe('chat-room-123', cb, { token: 'eyJhbGc...' });
// Auth helpers (call dev-defined endpoints under the hood)
await client.auth.login('alice@example.com', 'password');
await client.auth.logout();
const token = client.auth.token;
```
## React
```tsx
import { PicloudProvider, useTopic, useEndpoint } from '@picloud/client/react';
// Wrap your tree once: <PicloudProvider client={client}>…</PicloudProvider>
function ChatRoom({ roomId }: { roomId: string }) {
const messages = useTopic<ChatMessage>(`chat-room-${roomId}`);
return <ul>{messages.map((m, i) => <li key={i}>{m.text}</li>)}</ul>;
}
function UserProfile({ id }: { id: string }) {
const { data, loading, error } = useEndpoint<UserRes>(`/api/users/${id}`).get();
if (loading) return <Spinner />;
if (error) return <ErrorView error={error} />;
return <div>{data?.name}</div>;
}
```
## Svelte
```ts
import { topicStore, endpointStore } from '@picloud/client/svelte';
const messages = topicStore<ChatMessage>(client, `chat-room-${roomId}`);
// $messages is an array that grows as events arrive
const userQuery = endpointStore<UserRes>(client, `/api/users/${id}`).get();
// $userQuery is { data, loading, error }
```
> The Svelte helpers take the `client` explicitly (a store isn't a component,
> so there's no React-style context to read).
## Optional runtime validation (zod / valibot)
No hard dependency — the adapter is the `{ parse(input): T }` shape. A Zod
schema satisfies it directly; wrap Valibot in one line:
```ts
import { z } from 'zod';
const UserSchema = z.object({ id: z.string(), name: z.string() });
const user = await client.endpoint('/api/users/1').get({ validate: UserSchema });
// valibot:
import * as v from 'valibot';
const schema = v.object({ id: v.string() });
const adapter = { parse: (i: unknown) => v.parse(schema, i) };
```
## Transport notes
- SSE is implemented over streaming `fetch` (not native `EventSource`) so the
client can refresh an expired token on a 401, send `Last-Event-ID` on resume,
and apply its own exponential backoff (1s → 2s → 4s … capped at 30s).
- **React Native** has no native `EventSource`, but it also can't stream
`fetch` bodies on all engines — if you target RN, supply a streaming-capable
`fetch` polyfill via the `fetch` option, or use a `react-native-sse`-based
adapter. (Server-side `Last-Event-ID` replay is not implemented in v1.1.6;
the client sends the header so it's ready when the server adds replay.)
## Build / test
```sh
npm install
npm run lint # tsc --noEmit (strict)
npm run test # vitest
npm run build # tsup → dist/ (ESM + CJS + .d.ts)
```

3580
clients/typescript/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,61 @@
{
"name": "@picloud/client",
"version": "1.0.0",
"description": "TypeScript client for PiCloud — typed HTTP to script endpoints, SSE realtime subscriptions, auth-flow helpers, and React/Svelte hooks.",
"license": "MIT OR Apache-2.0",
"type": "module",
"sideEffects": false,
"files": [
"dist"
],
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js",
"require": "./dist/index.cjs"
},
"./react": {
"types": "./dist/react/index.d.ts",
"import": "./dist/react/index.js",
"require": "./dist/react/index.cjs"
},
"./svelte": {
"types": "./dist/svelte/index.d.ts",
"import": "./dist/svelte/index.js",
"require": "./dist/svelte/index.cjs"
}
},
"main": "./dist/index.cjs",
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"scripts": {
"build": "tsup",
"test": "vitest run",
"test:watch": "vitest",
"lint": "tsc --noEmit"
},
"peerDependencies": {
"react": ">=17",
"svelte": ">=4"
},
"peerDependenciesMeta": {
"react": {
"optional": true
},
"svelte": {
"optional": true
}
},
"devDependencies": {
"@testing-library/dom": "^10.4.0",
"@testing-library/react": "^16.1.0",
"@types/react": "^18.3.0",
"jsdom": "^25.0.0",
"react": "^18.3.0",
"react-dom": "^18.3.0",
"svelte": "^4.2.0",
"tsup": "^8.3.0",
"typescript": "^5.6.0",
"vitest": "^2.1.0"
}
}

View File

@@ -0,0 +1,71 @@
import { Endpoint } from './endpoint.js';
import type { AuthTokenProvider } from './types.js';
export interface AuthClientConfig {
baseURL: string;
fetchImpl: typeof fetch;
/** Path of the dev-defined login endpoint (default `/api/auth/login`). */
loginPath?: string;
/** Path of the dev-defined logout endpoint (default `/api/auth/logout`). */
logoutPath?: string;
/** Called whenever the stored token changes (e.g. to persist it). */
onToken?: (token: string | null) => void;
}
interface LoginResponse {
token?: string;
}
/**
* Auth-flow helpers. These call **dev-defined** endpoints under the hood
* (the script layer owns the actual auth); the lib only standardizes the
* dance + in-memory token storage. There is no built-in identity model —
* `login` POSTs credentials and stores whatever `token` comes back.
*/
export class AuthClient {
private current: string | null = null;
constructor(private readonly cfg: AuthClientConfig) {}
/** The current bearer token, or null. */
get token(): string | null {
return this.current;
}
/** Suitable as `PicloudClientOptions.getAuthToken`. */
readonly provider: AuthTokenProvider = () => this.current;
/** POST credentials to the login endpoint; store the returned token. */
async login(email: string, password: string): Promise<string | null> {
const ep = new Endpoint<{ email: string; password: string }, LoginResponse>({
baseURL: this.cfg.baseURL,
path: this.cfg.loginPath ?? '/api/auth/login',
fetchImpl: this.cfg.fetchImpl
});
const res = await ep.post({ email, password });
this.setToken(typeof res?.token === 'string' ? res.token : null);
return this.current;
}
/** POST to the logout endpoint (best-effort) and clear the token. */
async logout(): Promise<void> {
const ep = new Endpoint<undefined, unknown>({
baseURL: this.cfg.baseURL,
path: this.cfg.logoutPath ?? '/api/auth/logout',
// Send the current token so the script can invalidate the session.
getAuthToken: () => this.current,
fetchImpl: this.cfg.fetchImpl
});
try {
await ep.post();
} finally {
this.setToken(null);
}
}
/** Manually set (or clear) the token — e.g. restoring from storage. */
setToken(token: string | null): void {
this.current = token;
this.cfg.onToken?.(token);
}
}

View File

@@ -0,0 +1,61 @@
import { AuthClient } from './auth.js';
import { Endpoint } from './endpoint.js';
import { subscribeTopic } from './subscribe.js';
import type {
PicloudClientOptions,
RealtimeEvent,
SubscribeOptions,
Unsubscribe
} from './types.js';
/**
* The PiCloud frontend client. Three capabilities, all script-mediated
* (the hybrid model — no direct KV/docs/users access from the browser):
*
* - `endpoint<Req, Res>(path)` — typed HTTP to a dev-defined route.
* - `subscribe(topic, cb, opts?)` — SSE realtime subscription.
* - `auth` — login/logout/token helpers over dev-defined endpoints.
*/
export class PicloudClient {
readonly auth: AuthClient;
private readonly baseURL: string;
private readonly fetchImpl: typeof fetch;
private readonly getAuthToken: PicloudClientOptions['getAuthToken'];
constructor(opts: PicloudClientOptions) {
if (!opts.baseURL) throw new Error('PicloudClient: baseURL is required');
this.baseURL = opts.baseURL;
const f = opts.fetch ?? globalThis.fetch;
if (typeof f !== 'function') {
throw new Error('PicloudClient: no fetch available — pass options.fetch');
}
// Bind to avoid "Illegal invocation" when calling a detached global.
this.fetchImpl = f.bind(globalThis);
this.getAuthToken = opts.getAuthToken;
this.auth = new AuthClient({ baseURL: this.baseURL, fetchImpl: this.fetchImpl });
}
/** A typed handle to a dev-defined endpoint. */
endpoint<Req = unknown, Res = unknown>(path: string): Endpoint<Req, Res> {
return new Endpoint<Req, Res>({
baseURL: this.baseURL,
path,
getAuthToken: this.getAuthToken,
fetchImpl: this.fetchImpl
});
}
/** Subscribe to a realtime topic. Returns an unsubscribe function. */
subscribe<T = unknown>(
topic: string,
onMessage: (event: RealtimeEvent<T>) => void,
opts?: SubscribeOptions<T>
): Unsubscribe {
return subscribeTopic<T>(
{ baseURL: this.baseURL, fetchImpl: this.fetchImpl },
topic,
onMessage,
opts
);
}
}

View File

@@ -0,0 +1,106 @@
import { PicloudHttpError, type AuthTokenProvider, type Validator } from './types.js';
type Method = 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE';
export interface EndpointConfig {
baseURL: string;
path: string;
getAuthToken?: AuthTokenProvider;
fetchImpl: typeof fetch;
}
export interface RequestOptions<Res> {
/** Extra headers merged over the defaults. */
headers?: Record<string, string>;
/** Optional runtime validation of the parsed response. */
validate?: Validator<Res>;
/** AbortSignal to cancel the request. */
signal?: AbortSignal;
}
/**
* Typed HTTP to a dev-defined script endpoint. Auth header injection +
* structured errors; the request/response types are caller-supplied
* generics (`endpoint<Req, Res>('/path')`). No service access — every
* call hits a route a script binds (the hybrid model).
*/
export class Endpoint<Req = unknown, Res = unknown> {
constructor(private readonly cfg: EndpointConfig) {}
get(opts?: RequestOptions<Res>): Promise<Res> {
return this.send('GET', undefined, opts);
}
post(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
return this.send('POST', body, opts);
}
put(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
return this.send('PUT', body, opts);
}
patch(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
return this.send('PATCH', body, opts);
}
delete(opts?: RequestOptions<Res>): Promise<Res> {
return this.send('DELETE', undefined, opts);
}
private async send(method: Method, body: Req | undefined, opts?: RequestOptions<Res>): Promise<Res> {
const headers: Record<string, string> = {
Accept: 'application/json',
...(opts?.headers ?? {})
};
if (body !== undefined) {
headers['Content-Type'] ??= 'application/json';
}
const token = this.cfg.getAuthToken ? await this.cfg.getAuthToken() : undefined;
if (token) {
headers['Authorization'] ??= `Bearer ${token}`;
}
const url = joinUrl(this.cfg.baseURL, this.cfg.path);
const init: RequestInit = { method, headers };
if (body !== undefined) {
init.body = JSON.stringify(body);
}
if (opts?.signal) {
init.signal = opts.signal;
}
const res = await this.cfg.fetchImpl(url, init);
const parsed = await parseBody(res);
if (!res.ok) {
const message =
(isRecord(parsed) && typeof parsed['error'] === 'string' && parsed['error']) ||
`${method} ${this.cfg.path} failed with ${res.status}`;
throw new PicloudHttpError(res.status, message, parsed);
}
return opts?.validate ? opts.validate.parse(parsed) : (parsed as Res);
}
}
async function parseBody(res: Response): Promise<unknown> {
const text = await res.text();
if (text.length === 0) return null;
const ct = res.headers.get('content-type') ?? '';
if (ct.includes('application/json')) {
try {
return JSON.parse(text);
} catch {
return text;
}
}
return text;
}
function isRecord(v: unknown): v is Record<string, unknown> {
return typeof v === 'object' && v !== null;
}
export function joinUrl(base: string, path: string): string {
const b = base.endsWith('/') ? base.slice(0, -1) : base;
const p = path.startsWith('/') ? path : `/${path}`;
return `${b}${p}`;
}

View File

@@ -0,0 +1,14 @@
export { PicloudClient } from './client.js';
export { Endpoint } from './endpoint.js';
export { AuthClient } from './auth.js';
export { subscribeTopic } from './subscribe.js';
export {
PicloudHttpError,
type PicloudClientOptions,
type AuthTokenProvider,
type RealtimeEvent,
type SubscribeOptions,
type Unsubscribe,
type Validator
} from './types.js';
export type { RequestOptions } from './endpoint.js';

View File

@@ -0,0 +1,101 @@
import {
createContext,
createElement,
useContext,
useEffect,
useState,
type ReactNode
} from 'react';
import type { PicloudClient } from '../client.js';
import type { SubscribeOptions } from '../types.js';
const PicloudContext = createContext<PicloudClient | null>(null);
export interface PicloudProviderProps {
client: PicloudClient;
children?: ReactNode;
}
/** Provides a `PicloudClient` to `useTopic` / `useEndpoint`. */
export function PicloudProvider(props: PicloudProviderProps) {
return createElement(PicloudContext.Provider, { value: props.client }, props.children);
}
/** The client from the nearest `PicloudProvider`. Throws if absent. */
export function usePicloud(): PicloudClient {
const client = useContext(PicloudContext);
if (!client) {
throw new Error('usePicloud: wrap your tree in <PicloudProvider client={...}>');
}
return client;
}
/**
* Subscribe to a realtime topic; returns the accumulated messages in
* arrival order. Re-subscribes when `topic` changes; unsubscribes on
* unmount.
*/
export function useTopic<T = unknown>(topic: string, opts?: SubscribeOptions<T>): T[] {
const client = usePicloud();
const [messages, setMessages] = useState<T[]>([]);
useEffect(() => {
setMessages([]);
const unsubscribe = client.subscribe<T>(
topic,
(event) => setMessages((prev) => [...prev, event.message]),
opts
);
return () => unsubscribe();
// `opts` is intentionally excluded: a new object literal each render
// would otherwise resubscribe every render.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [client, topic]);
return messages;
}
export interface QueryState<T> {
data: T | null;
loading: boolean;
error: unknown;
}
export interface EndpointHook<Req, Res> {
get: () => QueryState<Res>;
post: (body?: Req) => QueryState<Res>;
}
/**
* Typed endpoint hook. `useEndpoint<Res>(path).get()` fires a GET and
* returns `{ data, loading, error }`, re-running when `path` changes.
* `.post(body)` is the mutation variant (auto-fires once per mount).
*/
export function useEndpoint<Res = unknown, Req = unknown>(path: string): EndpointHook<Req, Res> {
const client = usePicloud();
return {
get: () => useResource<Res>(() => client.endpoint<Req, Res>(path).get(), path, 'GET'),
post: (body?: Req) =>
useResource<Res>(() => client.endpoint<Req, Res>(path).post(body), path, 'POST')
};
}
function useResource<Res>(run: () => Promise<Res>, key: string, method: string): QueryState<Res> {
const [state, setState] = useState<QueryState<Res>>({
data: null,
loading: true,
error: null
});
useEffect(() => {
let active = true;
setState({ data: null, loading: true, error: null });
run()
.then((data) => active && setState({ data, loading: false, error: null }))
.catch((error) => active && setState({ data: null, loading: false, error }));
return () => {
active = false;
};
// `run` is recreated each render; key it on path + method instead.
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [key, method]);
return state;
}

View File

@@ -0,0 +1,194 @@
import { joinUrl } from './endpoint.js';
import type { RealtimeEvent, SubscribeOptions, Unsubscribe } from './types.js';
interface SubscribeConfig {
baseURL: string;
fetchImpl: typeof fetch;
}
/**
* Subscribe to an app pub/sub topic over SSE.
*
* Implemented over streaming `fetch` (not native `EventSource`) so the
* lib can: detect a 401 on (re)connect and refresh the token, send a
* `Last-Event-ID` header on resume, and apply its own exponential
* backoff. See HANDBACK for the rationale. Returns an unsubscribe
* function that aborts the connection and stops reconnecting.
*/
export function subscribeTopic<T = unknown>(
cfg: SubscribeConfig,
topic: string,
onMessage: (event: RealtimeEvent<T>) => void,
opts: SubscribeOptions<T> = {}
): Unsubscribe {
const baseBackoff = opts.baseBackoffMs ?? 1_000;
const maxBackoff = opts.maxBackoffMs ?? 30_000;
let token = opts.token;
let stopped = false;
let attempt = 0;
let lastEventId: string | undefined;
let controller: AbortController | null = null;
let backoffTimer: ReturnType<typeof setTimeout> | null = null;
const stop = () => {
stopped = true;
if (backoffTimer) clearTimeout(backoffTimer);
controller?.abort();
};
const scheduleReconnect = () => {
if (stopped) return;
// Exponential backoff: base, 2x, 4x… capped at maxBackoff.
const delay = Math.min(maxBackoff, baseBackoff * 2 ** attempt);
attempt += 1;
backoffTimer = setTimeout(() => void connect(), delay);
};
const connect = async (): Promise<void> => {
if (stopped) return;
controller = new AbortController();
const url = buildUrl(cfg.baseURL, topic, token);
const headers: Record<string, string> = { Accept: 'text/event-stream' };
if (lastEventId) headers['Last-Event-ID'] = lastEventId;
let res: Response;
try {
res = await cfg.fetchImpl(url, { headers, signal: controller.signal });
} catch (err) {
if (stopped || isAbort(err)) return;
scheduleReconnect();
return;
}
if (res.status === 401) {
// Token expired / rejected — try to refresh, else give up.
const fresh = opts.onTokenExpired ? await opts.onTokenExpired() : null;
if (fresh) {
token = fresh;
attempt = 0; // fresh credential → reconnect immediately
void connect();
} else {
opts.onError?.(new Error('realtime subscribe unauthorized (401)'));
stop();
}
return;
}
if (!res.ok || !res.body) {
if (!stopped) scheduleReconnect();
return;
}
// Connected — reset backoff and stream frames until the body ends.
attempt = 0;
try {
await readStream(res.body, (frame) => {
if (frame.id !== undefined) lastEventId = frame.id;
if (frame.data === undefined) return; // comment / heartbeat
const parsed = parseEvent<T>(frame.data, opts);
if (parsed) onMessage(parsed);
});
} catch (err) {
if (stopped || isAbort(err)) return;
}
// Stream ended (server closed, e.g. topic deleted) → reconnect.
if (!stopped) scheduleReconnect();
};
void connect();
return stop;
}
function buildUrl(baseURL: string, topic: string, token?: string): string {
const url = joinUrl(baseURL, `/realtime/topics/${encodeURIComponent(topic)}`);
// EventSource can't set headers, so the token rides in the query
// string — the same path a raw EventSource would use.
return token ? `${url}?token=${encodeURIComponent(token)}` : url;
}
function parseEvent<T>(data: string, opts: SubscribeOptions<T>): RealtimeEvent<T> | null {
let json: unknown;
try {
json = JSON.parse(data);
} catch {
return null;
}
if (!isRealtimeShape(json)) return null;
const message = opts.validate ? opts.validate.parse(json.message) : (json.message as T);
return { topic: json.topic, message, published_at: json.published_at };
}
function isRealtimeShape(v: unknown): v is RealtimeEvent<unknown> {
return (
typeof v === 'object' &&
v !== null &&
typeof (v as Record<string, unknown>)['topic'] === 'string' &&
typeof (v as Record<string, unknown>)['published_at'] === 'string' &&
'message' in (v as Record<string, unknown>)
);
}
interface SseFrame {
data?: string;
id?: string;
}
/**
* Read an SSE response body, invoking `onFrame` per event. Minimal
* parser: accumulates `data:` lines (joined by `\n`) and `id:` until a
* blank line dispatches the frame. Lines starting with `:` are comments
* (heartbeats) — surfaced as a frame with no `data` so the id can still
* advance.
*/
async function readStream(
body: ReadableStream<Uint8Array>,
onFrame: (frame: SseFrame) => void
): Promise<void> {
const reader = body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let dataLines: string[] = [];
let id: string | undefined;
let sawComment = false;
const dispatch = () => {
if (dataLines.length > 0) {
onFrame({ data: dataLines.join('\n'), id });
} else if (sawComment) {
onFrame({ id });
}
dataLines = [];
sawComment = false;
};
for (;;) {
const { value, done } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
let nl: number;
while ((nl = buffer.indexOf('\n')) >= 0) {
const line = buffer.slice(0, nl).replace(/\r$/, '');
buffer = buffer.slice(nl + 1);
if (line === '') {
dispatch();
continue;
}
if (line.startsWith(':')) {
sawComment = true;
continue;
}
const colon = line.indexOf(':');
const field = colon === -1 ? line : line.slice(0, colon);
const rawVal = colon === -1 ? '' : line.slice(colon + 1);
const val = rawVal.startsWith(' ') ? rawVal.slice(1) : rawVal;
if (field === 'data') dataLines.push(val);
else if (field === 'id') id = val;
}
}
// Flush a trailing frame if the stream ended without a blank line.
dispatch();
}
function isAbort(err: unknown): boolean {
return typeof err === 'object' && err !== null && (err as { name?: string }).name === 'AbortError';
}

View File

@@ -0,0 +1,72 @@
import { readable, type Readable } from 'svelte/store';
import type { PicloudClient } from '../client.js';
import type { SubscribeOptions } from '../types.js';
/**
* A Svelte store of realtime messages for a topic. `$messages` is an
* array that grows as events arrive. The SSE connection opens on the
* first subscriber and closes when the last unsubscribes (standard
* `readable` lifecycle).
*
* The client is passed explicitly (Svelte stores aren't components, so
* there's no React-style context to read). See HANDBACK §7.
*/
export function topicStore<T = unknown>(
client: PicloudClient,
topic: string,
opts?: SubscribeOptions<T>
): Readable<T[]> {
return readable<T[]>([], (set) => {
let items: T[] = [];
const unsubscribe = client.subscribe<T>(
topic,
(event) => {
items = [...items, event.message];
set(items);
},
opts
);
return () => unsubscribe();
});
}
export interface QueryState<T> {
data: T | null;
loading: boolean;
error: unknown;
}
export interface EndpointStore<Req, Res> {
get: () => Readable<QueryState<Res>>;
post: (body?: Req) => Readable<QueryState<Res>>;
}
/**
* A Svelte store wrapper over a typed endpoint. `$query` is
* `{ data, loading, error }`. The request fires when the store gains its
* first subscriber.
*/
export function endpointStore<Res = unknown, Req = unknown>(
client: PicloudClient,
path: string
): EndpointStore<Req, Res> {
const run = (exec: () => Promise<Res>): Readable<QueryState<Res>> =>
readable<QueryState<Res>>({ data: null, loading: true, error: null }, (set) => {
let active = true;
exec()
.then((data) => {
if (active) set({ data, loading: false, error: null });
})
.catch((error) => {
if (active) set({ data: null, loading: false, error });
});
return () => {
active = false;
};
});
return {
get: () => run(() => client.endpoint<Req, Res>(path).get()),
post: (body?: Req) => run(() => client.endpoint<Req, Res>(path).post(body))
};
}

View File

@@ -0,0 +1,73 @@
// Shared types for @picloud/client.
/** Returns the current bearer token (or null) before each HTTP request. */
export type AuthTokenProvider = () => string | null | undefined | Promise<string | null | undefined>;
export interface PicloudClientOptions {
/** Base URL of the PiCloud deployment, e.g. `https://api.example.com`. */
baseURL: string;
/**
* Optional: returns the current bearer token, called before each
* request. The client doesn't manage tokens — it just sends them.
*/
getAuthToken?: AuthTokenProvider;
/**
* Optional fetch implementation (defaults to the global `fetch`).
* Injected mainly for tests / non-browser runtimes.
*/
fetch?: typeof fetch;
}
/** A realtime event as delivered over SSE. */
export interface RealtimeEvent<T = unknown> {
topic: string;
message: T;
published_at: string;
}
/**
* Minimal validator shape for the optional runtime-validation adapter.
* A Zod schema satisfies this directly (`schema.parse`); for Valibot,
* wrap it: `{ parse: (i) => v.parse(schema, i) }`. No hard dep on either.
*/
export interface Validator<T> {
parse: (input: unknown) => T;
}
/** Thrown when an endpoint call returns a non-2xx status. */
export class PicloudHttpError extends Error {
readonly status: number;
readonly body: unknown;
constructor(status: number, message: string, body: unknown) {
super(message);
this.name = 'PicloudHttpError';
this.status = status;
this.body = body;
}
}
export interface SubscribeOptions<T = unknown> {
/**
* Subscriber token for `auth_mode = 'token'` topics. Obtained from one
* of your app's script endpoints (which calls
* `pubsub::subscriber_token`). Sent as `?token=` (EventSource-parity).
*/
token?: string;
/**
* Called when a (re)connect is rejected with 401 — typically an
* expired token. Return a fresh token to retry immediately, or
* null/undefined to stop and surface the error.
*/
onTokenExpired?: () => string | null | undefined | Promise<string | null | undefined>;
/** Called on a terminal error (after retries are exhausted or aborted). */
onError?: (err: unknown) => void;
/** Optional runtime validation of each event's `message`. */
validate?: Validator<T>;
/** Max reconnect backoff in ms (default 30_000). */
maxBackoffMs?: number;
/** Base reconnect backoff in ms (default 1_000). */
baseBackoffMs?: number;
}
/** Cancels a realtime subscription. */
export type Unsubscribe = () => void;

View File

@@ -0,0 +1,41 @@
import { describe, expect, it, vi } from 'vitest';
import { PicloudClient } from '../src/index.js';
import { jsonResponse, lastUrl, type FetchArgs } from './helpers.js';
describe('auth', () => {
it('login POSTs credentials and stores the returned token', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ token: 'session-abc' })
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const token = await client.auth.login('alice@example.com', 'pw');
expect(token).toBe('session-abc');
expect(client.auth.token).toBe('session-abc');
expect(lastUrl(fetchMock)).toBe('https://api.test/api/auth/login');
const init = fetchMock.mock.calls[0]?.[1];
expect(JSON.parse(String(init?.body))).toEqual({
email: 'alice@example.com',
password: 'pw'
});
});
it('logout clears the stored token', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => jsonResponse({}));
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
client.auth.setToken('existing');
await client.auth.logout();
expect(client.auth.token).toBeNull();
});
it('provider returns the current token for getAuthToken wiring', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ token: 't' })
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
await client.auth.login('a@b.c', 'pw');
expect(client.auth.provider()).toBe('t');
});
});

View File

@@ -0,0 +1,82 @@
import { describe, expect, it, vi } from 'vitest';
import { PicloudClient, PicloudHttpError } from '../src/index.js';
import { headerOf, jsonResponse, lastInit, lastUrl, type FetchArgs } from './helpers.js';
describe('endpoint', () => {
it('post round-trips a typed request/response', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ id: '1', name: 'alice', created_at: 'now' }, 201)
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
interface Req {
name: string;
role: string;
}
interface Res {
id: string;
name: string;
created_at: string;
}
const res = await client.endpoint<Req, Res>('/api/users').post({ name: 'alice', role: 'admin' });
expect(res).toEqual({ id: '1', name: 'alice', created_at: 'now' });
expect(lastUrl(fetchMock)).toBe('https://api.test/api/users');
const init = lastInit(fetchMock);
expect(init.method).toBe('POST');
expect(JSON.parse(String(init.body))).toEqual({ name: 'alice', role: 'admin' });
expect(headerOf(init, 'Content-Type')).toBe('application/json');
});
it('get round-trips', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ name: 'bob' })
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const res = await client.endpoint<unknown, { name: string }>('/api/users/1').get();
expect(res.name).toBe('bob');
expect(lastInit(fetchMock).method).toBe('GET');
});
it('injects the auth token from getAuthToken', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => jsonResponse({ ok: true }));
const client = new PicloudClient({
baseURL: 'https://api.test',
fetch: fetchMock,
getAuthToken: () => 'tok-123'
});
await client.endpoint('/api/me').get();
expect(headerOf(lastInit(fetchMock), 'Authorization')).toBe('Bearer tok-123');
});
it('throws PicloudHttpError with status + body on non-2xx', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ error: 'bad input' }, 422)
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const err = await client
.endpoint('/api/x')
.get()
.catch((e: unknown) => e);
expect(err).toBeInstanceOf(PicloudHttpError);
expect((err as PicloudHttpError).status).toBe(422);
expect((err as PicloudHttpError).message).toBe('bad input');
});
it('applies an optional validator to the response', async () => {
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
jsonResponse({ id: 7 })
);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const validator = {
parse: (input: unknown) => {
const r = input as { id: number };
if (typeof r.id !== 'number') throw new Error('bad');
return r;
}
};
const res = await client.endpoint<unknown, { id: number }>('/api/x').get({ validate: validator });
expect(res.id).toBe(7);
});
});

View File

@@ -0,0 +1,54 @@
// Test helpers: build JSON + SSE Response objects and a typed fetch mock.
export function jsonResponse(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { 'content-type': 'application/json' }
});
}
export function emptyResponse(status = 200): Response {
return new Response(null, { status });
}
/** Build a text/event-stream Response from raw SSE frame strings. */
export function sseResponse(frames: string[], status = 200): Response {
const encoder = new TextEncoder();
const stream = new ReadableStream<Uint8Array>({
start(controller) {
for (const frame of frames) controller.enqueue(encoder.encode(frame));
controller.close();
}
});
return new Response(stream, {
status,
headers: { 'content-type': 'text/event-stream' }
});
}
/** One SSE `data:` event frame for a realtime payload. */
export function dataFrame(topic: string, message: unknown, publishedAt = '2026-06-04T00:00:00Z'): string {
const payload = JSON.stringify({ topic, message, published_at: publishedAt });
return `data: ${payload}\n\n`;
}
export type FetchArgs = [string | URL | Request, RequestInit?];
type MockLike = { mock: { calls: ReadonlyArray<ReadonlyArray<unknown>> } };
export function lastInit(mock: MockLike, i = 0): RequestInit {
const call = mock.mock.calls[i];
if (!call) throw new Error(`no fetch call at index ${i}`);
return (call[1] as RequestInit | undefined) ?? {};
}
export function lastUrl(mock: MockLike, i = 0): string {
const call = mock.mock.calls[i];
if (!call) throw new Error(`no fetch call at index ${i}`);
return String(call[0]);
}
export function headerOf(init: RequestInit, name: string): string | undefined {
const h = init.headers as Record<string, string> | undefined;
return h?.[name];
}

View File

@@ -0,0 +1,41 @@
import { act, renderHook } from '@testing-library/react';
import type { ReactNode } from 'react';
import { describe, expect, it, vi } from 'vitest';
import type { PicloudClient, RealtimeEvent, Unsubscribe } from '../src/index.js';
import { PicloudProvider, useTopic } from '../src/react/index.js';
type Cb = (e: RealtimeEvent<unknown>) => void;
function fakeClient() {
const unsubscribe = vi.fn();
let captured: Cb | null = null;
const subscribe = vi.fn(
(_topic: string, cb: Cb): Unsubscribe => {
captured = cb;
return unsubscribe as unknown as Unsubscribe;
}
);
const client = { subscribe } as unknown as PicloudClient;
return { client, subscribe, unsubscribe, emit: (e: RealtimeEvent<unknown>) => captured?.(e) };
}
describe('react useTopic', () => {
it('subscribes on mount, accumulates messages, unsubscribes on unmount', () => {
const { client, subscribe, unsubscribe, emit } = fakeClient();
const wrapper = ({ children }: { children: ReactNode }) =>
PicloudProvider({ client, children });
const { result, unmount } = renderHook(() => useTopic<{ n: number }>('chat'), { wrapper });
expect(subscribe).toHaveBeenCalledTimes(1);
expect(result.current).toEqual([]);
act(() => emit({ topic: 'chat', message: { n: 1 }, published_at: 't' }));
act(() => emit({ topic: 'chat', message: { n: 2 }, published_at: 't' }));
expect(result.current).toEqual([{ n: 1 }, { n: 2 }]);
unmount();
expect(unsubscribe).toHaveBeenCalledTimes(1);
});
});

View File

@@ -0,0 +1,99 @@
import { describe, expect, it, vi } from 'vitest';
import { PicloudClient, type RealtimeEvent } from '../src/index.js';
import { dataFrame, emptyResponse, lastUrl, sseResponse, type FetchArgs } from './helpers.js';
/** A fetch mock that plays through a queue of response factories. */
function queuedFetch(responders: Array<() => Promise<Response>>) {
let i = 0;
return vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => {
const idx = Math.min(i, responders.length - 1);
i += 1;
const r = responders[idx];
if (!r) throw new Error('no responder');
return r();
});
}
describe('subscribe', () => {
it('connects to the SSE endpoint and delivers events', async () => {
const fetchMock = queuedFetch([async () => sseResponse([dataFrame('chat', { hi: 1 })])]);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const received: Array<RealtimeEvent<{ hi: number }>> = [];
const unsubscribe = client.subscribe<{ hi: number }>('chat', (e) => received.push(e));
await vi.waitFor(() => expect(received.length).toBe(1));
unsubscribe();
expect(received[0]?.topic).toBe('chat');
expect(received[0]?.message).toEqual({ hi: 1 });
expect(lastUrl(fetchMock)).toBe('https://api.test/realtime/topics/chat');
});
it('passes a token via the query string', async () => {
const fetchMock = queuedFetch([async () => sseResponse([dataFrame('chat', 1)])]);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const unsubscribe = client.subscribe('chat', () => {}, { token: 'abc.def' });
await vi.waitFor(() => expect(fetchMock).toHaveBeenCalled());
unsubscribe();
expect(lastUrl(fetchMock)).toBe('https://api.test/realtime/topics/chat?token=abc.def');
});
it('reconnects with backoff after an initial connection failure', async () => {
const fetchMock = queuedFetch([
async () => {
throw new Error('network down');
},
async () => sseResponse([dataFrame('chat', { ok: true })])
]);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const received: unknown[] = [];
const unsubscribe = client.subscribe('chat', (e) => received.push(e.message), {
baseBackoffMs: 5,
maxBackoffMs: 20
});
await vi.waitFor(() => expect(received.length).toBeGreaterThanOrEqual(1), { timeout: 1000 });
unsubscribe();
expect(fetchMock.mock.calls.length).toBeGreaterThanOrEqual(2);
expect(received[0]).toEqual({ ok: true });
});
it('refreshes the token after a 401 and reconnects', async () => {
const fetchMock = queuedFetch([
async () => emptyResponse(401),
async () => sseResponse([dataFrame('chat', { v: 2 })])
]);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const onTokenExpired = vi.fn(() => 'fresh-token');
const received: unknown[] = [];
const unsubscribe = client.subscribe('chat', (e) => received.push(e.message), {
token: 'stale',
onTokenExpired,
baseBackoffMs: 5
});
await vi.waitFor(() => expect(received.length).toBeGreaterThanOrEqual(1), { timeout: 1000 });
unsubscribe();
expect(onTokenExpired).toHaveBeenCalled();
// Second connect carries the refreshed token.
expect(lastUrl(fetchMock, 1)).toContain('token=fresh-token');
expect(received[0]).toEqual({ v: 2 });
});
it('stops and reports when a 401 cannot be refreshed', async () => {
const fetchMock = queuedFetch([async () => emptyResponse(401)]);
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
const onError = vi.fn();
const unsubscribe = client.subscribe('chat', () => {}, {
onTokenExpired: () => null,
onError
});
await vi.waitFor(() => expect(onError).toHaveBeenCalled());
unsubscribe();
});
});

View File

@@ -0,0 +1,34 @@
import { get } from 'svelte/store';
import { describe, expect, it, vi } from 'vitest';
import type { PicloudClient, RealtimeEvent, Unsubscribe } from '../src/index.js';
import { topicStore } from '../src/svelte/index.js';
type Cb = (e: RealtimeEvent<unknown>) => void;
describe('svelte topicStore', () => {
it('subscribes on first subscriber and unsubscribes on last', () => {
const unsubscribe = vi.fn();
const holder: { cb: Cb | null } = { cb: null };
const subscribe = vi.fn((_topic: string, cb: Cb): Unsubscribe => {
holder.cb = cb;
return unsubscribe as unknown as Unsubscribe;
});
const client = { subscribe } as unknown as PicloudClient;
const store = topicStore<{ x: number }>(client, 'chat');
// No SSE connection until someone subscribes (readable lifecycle).
expect(subscribe).not.toHaveBeenCalled();
let value: { x: number }[] = [];
const stop = store.subscribe((v) => (value = v));
expect(subscribe).toHaveBeenCalledTimes(1);
holder.cb?.({ topic: 'chat', message: { x: 1 }, published_at: 't' });
expect(value).toEqual([{ x: 1 }]);
expect(get(store)).toEqual([{ x: 1 }]);
stop();
expect(unsubscribe).toHaveBeenCalledTimes(1);
});
});

View File

@@ -0,0 +1,21 @@
{
"compilerOptions": {
"target": "ES2020",
"module": "ESNext",
"moduleResolution": "Bundler",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"jsx": "react-jsx",
"strict": true,
"noUncheckedIndexedAccess": true,
"exactOptionalPropertyTypes": false,
"noImplicitOverride": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"esModuleInterop": true,
"skipLibCheck": true,
"declaration": true,
"noEmit": true,
"types": []
},
"include": ["src", "tests"]
}

View File

@@ -0,0 +1,18 @@
import { defineConfig } from 'tsup';
// Dual ESM + CJS emit with .d.ts for the main entry and the two
// framework subpath exports. React and Svelte are peer deps — kept
// external so the lib never bundles a framework copy.
export default defineConfig({
entry: {
index: 'src/index.ts',
'react/index': 'src/react/index.ts',
'svelte/index': 'src/svelte/index.ts'
},
format: ['esm', 'cjs'],
dts: true,
clean: true,
sourcemap: true,
treeshake: true,
external: ['react', 'svelte', 'svelte/store']
});

View File

@@ -0,0 +1,11 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
// jsdom so the React/Svelte hook tests have a DOM; the core
// endpoint/subscribe/auth tests are environment-agnostic.
environment: 'jsdom',
globals: true,
include: ['tests/**/*.test.ts', 'tests/**/*.test.tsx']
}
});

View File

@@ -348,6 +348,7 @@ fn build_ctx_map(req: &ExecRequest) -> Map {
/// `docs/v1.1.x-design-notes.md` §4 (the dead-letter sub-shape) and /// `docs/v1.1.x-design-notes.md` §4 (the dead-letter sub-shape) and
/// §2/blueprint §9 (KV). Each variant becomes a Rhai map with a /// §2/blueprint §9 (KV). Each variant becomes a Rhai map with a
/// `source` discriminant plus per-source fields. /// `source` discriminant plus per-source fields.
#[allow(clippy::too_many_lines)]
fn trigger_event_to_dynamic(event: &TriggerEvent) -> Dynamic { fn trigger_event_to_dynamic(event: &TriggerEvent) -> Dynamic {
let mut m = Map::new(); let mut m = Map::new();
m.insert("source".into(), event.source().into()); m.insert("source".into(), event.source().into());
@@ -406,6 +407,81 @@ fn trigger_event_to_dynamic(event: &TriggerEvent) -> Dynamic {
cron_map.insert("fired_at".into(), fired_at.to_rfc3339().into()); cron_map.insert("fired_at".into(), fired_at.to_rfc3339().into());
m.insert("cron".into(), cron_map.into()); m.insert("cron".into(), cron_map.into());
} }
TriggerEvent::Files {
op,
collection,
id,
name,
content_type,
size,
checksum,
prev,
} => {
m.insert("op".into(), op.as_str().into());
let mut files_map = Map::new();
files_map.insert("collection".into(), collection.clone().into());
files_map.insert("id".into(), id.clone().into());
files_map.insert("name".into(), name.clone().into());
files_map.insert("content_type".into(), content_type.clone().into());
files_map.insert(
"size".into(),
i64::try_from(*size).unwrap_or(i64::MAX).into(),
);
files_map.insert("checksum".into(), checksum.clone().into());
files_map.insert(
"prev".into(),
prev.clone().map_or(Dynamic::UNIT, json_to_dynamic),
);
m.insert("files".into(), files_map.into());
}
TriggerEvent::Pubsub {
topic,
message,
published_at,
} => {
// `ctx.event.op` is always "publish" for pub/sub (the only
// op a publish produces).
m.insert("op".into(), "publish".into());
let mut ps = Map::new();
ps.insert("topic".into(), topic.clone().into());
ps.insert("message".into(), json_to_dynamic(message.clone()));
ps.insert("published_at".into(), published_at.to_rfc3339().into());
m.insert("pubsub".into(), ps.into());
}
TriggerEvent::Email {
from,
to,
cc,
subject,
text,
html,
received_at,
message_id,
} => {
// `ctx.event.op` is always "receive" for inbound email.
m.insert("op".into(), "receive".into());
let mut em = Map::new();
em.insert("from".into(), from.clone().into());
let to_arr: rhai::Array = to.iter().map(|a| Dynamic::from(a.clone())).collect();
em.insert("to".into(), to_arr.into());
let cc_arr: rhai::Array = cc.iter().map(|a| Dynamic::from(a.clone())).collect();
em.insert("cc".into(), cc_arr.into());
em.insert("subject".into(), subject.clone().into());
em.insert(
"text".into(),
text.clone().map_or(Dynamic::UNIT, Dynamic::from),
);
em.insert(
"html".into(),
html.clone().map_or(Dynamic::UNIT, Dynamic::from),
);
em.insert("received_at".into(), received_at.to_rfc3339().into());
em.insert(
"message_id".into(),
message_id.clone().map_or(Dynamic::UNIT, Dynamic::from),
);
m.insert("email".into(), em.into());
}
TriggerEvent::DeadLetter { TriggerEvent::DeadLetter {
dead_letter_id, dead_letter_id,
original, original,

View File

@@ -0,0 +1,150 @@
//! `email::` Rhai bridge — outbound email (v1.1.7).
//!
//! ```rhai
//! email::send(#{
//! to: "alice@example.com", // String or Array of String
//! from: "alerts@myapp.com",
//! subject: "Build complete",
//! text: "Your deploy finished."
//! });
//!
//! email::send_html(#{
//! to: ["alice@x.com", "bob@y.com"],
//! cc: ["dave@z.com"],
//! bcc: ["audit@myapp.com"],
//! from: "alerts@myapp.com",
//! reply_to: "support@myapp.com", // optional; defaults to `from`
//! subject: "Build complete",
//! text: "Your deploy finished.", // plain-text fallback
//! html: "<p>Your deploy <b>finished</b>.</p>"
//! });
//! ```
//!
//! Both map onto `EmailService::send`. `email::send` forces a text-only
//! message (any `html` key is ignored); `email::send_html` requires an
//! `html` part. `app_id` is derived from `cx.app_id` in the service.
use std::sync::Arc;
use picloud_shared::{EmailError, OutboundEmail, SdkCallCx, Services};
use rhai::{Array, Engine as RhaiEngine, EvalAltResult, Map, Module};
use tokio::runtime::Handle as TokioHandle;
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
let svc = services.email.clone();
let mut module = Module::new();
// email::send(#{...}) — plain text (html ignored).
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn("send", move |opts: Map| -> Result<(), Box<EvalAltResult>> {
let mut email = parse_email(&opts)?;
email.html = None; // text-only path
let svc = svc.clone();
let cx = cx.clone();
block_on(async move { svc.send(&cx, email).await })
});
}
// email::send_html(#{...}) — multipart text + html (html required).
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"send_html",
move |opts: Map| -> Result<(), Box<EvalAltResult>> {
let email = parse_email(&opts)?;
if email.html.as_ref().is_none_or(String::is_empty) {
return Err(runtime_err(
"email::send_html: an 'html' field is required (use email::send for text-only)",
));
}
let svc = svc.clone();
let cx = cx.clone();
block_on(async move { svc.send(&cx, email).await })
},
);
}
engine.register_static_module("email", module.into());
}
/// Parse the Rhai options map into an [`OutboundEmail`]. Field-level
/// validation (required fields, address shape) happens in the service;
/// here we only do type coercion (String/Array → Vec<String>).
fn parse_email(opts: &Map) -> Result<OutboundEmail, Box<EvalAltResult>> {
Ok(OutboundEmail {
to: addresses(opts, "to")?,
cc: addresses(opts, "cc")?,
bcc: addresses(opts, "bcc")?,
from: string_field(opts, "from").unwrap_or_default(),
reply_to: string_field(opts, "reply_to"),
subject: string_field(opts, "subject").unwrap_or_default(),
text: string_field(opts, "text"),
html: string_field(opts, "html"),
})
}
/// Read a string field. Missing or `()` → `None`.
fn string_field(opts: &Map, key: &str) -> Option<String> {
match opts.get(key) {
None => None,
Some(d) if d.is_unit() => None,
Some(d) if d.is_string() => Some(d.clone().into_string().unwrap_or_default()),
// Coerce non-string scalars via display (numbers, etc.).
Some(d) => Some(d.to_string()),
}
}
/// Read an address list: a String becomes a one-element list; an Array
/// of Strings becomes a list; missing/`()` is empty.
fn addresses(opts: &Map, key: &str) -> Result<Vec<String>, Box<EvalAltResult>> {
match opts.get(key) {
None => Ok(Vec::new()),
Some(d) if d.is_unit() => Ok(Vec::new()),
Some(d) if d.is_string() => Ok(vec![d.clone().into_string().unwrap_or_default()]),
Some(d) => {
if let Some(arr) = d.clone().try_cast::<Array>() {
let mut out = Vec::with_capacity(arr.len());
for el in arr {
if !el.is_string() {
return Err(runtime_err(&format!(
"email: '{key}' array must contain only strings"
)));
}
out.push(el.into_string().unwrap_or_default());
}
Ok(out)
} else {
Err(runtime_err(&format!(
"email: '{key}' must be a string or an array of strings"
)))
}
}
}
}
#[allow(clippy::unnecessary_box_returns)]
fn runtime_err(msg: &str) -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(msg.into(), rhai::Position::NONE).into()
}
/// Run an `EmailService` future inside the synchronous Rhai context,
/// mapping any `EmailError` to a Rhai runtime error. Mirrors
/// `kv::block_on`.
fn block_on<F>(fut: F) -> Result<(), Box<EvalAltResult>>
where
F: std::future::Future<Output = Result<(), EmailError>> + Send,
{
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
format!("email: no tokio runtime available: {e}").into(),
rhai::Position::NONE,
)
.into()
})?;
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(format!("email: {err}").into(), rhai::Position::NONE).into()
})
}

View File

@@ -0,0 +1,281 @@
//! `files::` Rhai bridge — collection-scoped handle pattern (v1.1.5).
//!
//! ```rhai
//! let avatars = files::collection("avatars");
//! let id = avatars.create(#{ name: "a.jpg", content_type: "image/jpeg", data: blob });
//! let meta = avatars.head(id); // metadata map or ()
//! let bytes = avatars.get(id); // Blob or ()
//! avatars.update(id, #{ data: new_bytes });
//! let gone = avatars.delete(id); // bool (was-present)
//! let page = avatars.list(); // #{ files: [...], next_cursor: () }
//! ```
//!
//! The `FilesHandle` custom Rhai type captures the collection name once
//! and routes each call through the injected `Arc<dyn FilesService>`
//! with the per-call `Arc<SdkCallCx>`. **The service derives `app_id`
//! from `cx.app_id` — it never appears in any signature script-side,
//! preserving cross-app isolation.**
//!
//! Error convention (per `docs/sdk-shape.md`): `create`/`update`/
//! `delete` throw on failure; `get`/`head` return `()` for a missing
//! file; `delete` returns `bool` (was-present). The blob bytes are a
//! Rhai `Blob` (byte array) in both directions.
use std::sync::Arc;
use picloud_shared::{
FileMeta, FileUpdate, FilesError, FilesService, NewFile, SdkCallCx, Services,
};
use rhai::{Array, Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
use tokio::runtime::Handle as TokioHandle;
/// Per-call handle captured by the Rhai SDK. Cheap to clone (two Arcs
/// plus an owned string).
#[derive(Clone)]
pub struct FilesHandle {
collection: String,
service: Arc<dyn FilesService>,
cx: Arc<SdkCallCx>,
}
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
let files_service = services.files.clone();
let mut module = Module::new();
{
let files_service = files_service.clone();
let cx = cx.clone();
module.set_native_fn(
"collection",
move |name: &str| -> Result<FilesHandle, Box<EvalAltResult>> {
if name.is_empty() {
return Err("files::collection name must not be empty".into());
}
Ok(FilesHandle {
collection: name.to_string(),
service: files_service.clone(),
cx: cx.clone(),
})
},
);
}
engine.register_static_module("files", module.into());
engine.register_type_with_name::<FilesHandle>("FilesHandle");
register_create(engine);
register_head(engine);
register_get(engine);
register_update(engine);
register_delete(engine);
register_list(engine);
}
fn register_create(engine: &mut RhaiEngine) {
engine.register_fn(
"create",
|handle: &mut FilesHandle, meta: Map| -> Result<String, Box<EvalAltResult>> {
let name = require_string(&meta, "name")?;
let content_type = require_string(&meta, "content_type")?;
let data = require_blob(&meta, "data")?;
let h = handle.clone();
let new = NewFile {
name,
content_type,
data,
};
let id = block_on(async move { h.service.create(&h.cx, &h.collection, new).await })?;
Ok(id.to_string())
},
);
}
fn register_head(engine: &mut RhaiEngine) {
engine.register_fn(
"head",
|handle: &mut FilesHandle, id: &str| -> Result<Dynamic, Box<EvalAltResult>> {
let h = handle.clone();
let id = id.to_string();
let meta = block_on(async move { h.service.head(&h.cx, &h.collection, &id).await })?;
Ok(meta.map_or(Dynamic::UNIT, |m| file_meta_to_map(&m).into()))
},
);
}
fn register_get(engine: &mut RhaiEngine) {
engine.register_fn(
"get",
|handle: &mut FilesHandle, id: &str| -> Result<Dynamic, Box<EvalAltResult>> {
let h = handle.clone();
let id = id.to_string();
let bytes = block_on(async move { h.service.get(&h.cx, &h.collection, &id).await })?;
Ok(bytes.map_or(Dynamic::UNIT, Dynamic::from_blob))
},
);
}
fn register_update(engine: &mut RhaiEngine) {
engine.register_fn(
"update",
|handle: &mut FilesHandle, id: &str, meta: Map| -> Result<(), Box<EvalAltResult>> {
let data = require_blob(&meta, "data")?;
let name = optional_string(&meta, "name")?;
let content_type = optional_string(&meta, "content_type")?;
let h = handle.clone();
let id = id.to_string();
let upd = FileUpdate {
data,
name,
content_type,
};
block_on(async move { h.service.update(&h.cx, &h.collection, &id, upd).await })
},
);
}
fn register_delete(engine: &mut RhaiEngine) {
engine.register_fn(
"delete",
|handle: &mut FilesHandle, id: &str| -> Result<bool, Box<EvalAltResult>> {
let h = handle.clone();
let id = id.to_string();
block_on(async move { h.service.delete(&h.cx, &h.collection, &id).await })
},
);
}
fn register_list(engine: &mut RhaiEngine) {
engine.register_fn(
"list",
|handle: &mut FilesHandle| -> Result<Map, Box<EvalAltResult>> {
list_call(handle, None, 0)
},
);
engine.register_fn(
"list",
|handle: &mut FilesHandle, cursor: &str| -> Result<Map, Box<EvalAltResult>> {
list_call(handle, Some(cursor.to_string()), 0)
},
);
engine.register_fn(
"list",
|handle: &mut FilesHandle, cursor: &str, limit: i64| -> Result<Map, Box<EvalAltResult>> {
let limit = u32::try_from(limit.max(0)).unwrap_or(0);
list_call(handle, Some(cursor.to_string()), limit)
},
);
// `list(#{ cursor, limit })` — the map form documented in the brief.
engine.register_fn(
"list",
|handle: &mut FilesHandle, opts: Map| -> Result<Map, Box<EvalAltResult>> {
let cursor = match opts.get("cursor") {
Some(v) if !v.is_unit() => {
Some(v.clone().into_string().map_err(|_| -> Box<EvalAltResult> {
"files: list cursor must be a string".into()
})?)
}
_ => None,
};
let limit = match opts.get("limit") {
Some(v) if !v.is_unit() => {
u32::try_from(v.as_int().unwrap_or(0).max(0)).unwrap_or(0)
}
_ => 0,
};
list_call(handle, cursor, limit)
},
);
}
fn list_call(
handle: &FilesHandle,
cursor: Option<String>,
limit: u32,
) -> Result<Map, Box<EvalAltResult>> {
let h = handle.clone();
let page = block_on(async move {
h.service
.list(&h.cx, &h.collection, cursor.as_deref(), limit)
.await
})?;
let mut m = Map::new();
let files: Array = page
.files
.iter()
.map(|meta| Dynamic::from(file_meta_to_map(meta)))
.collect();
m.insert("files".into(), files.into());
m.insert(
"next_cursor".into(),
page.next_cursor.map_or(Dynamic::UNIT, Dynamic::from),
);
Ok(m)
}
/// Render a `FileMeta` into the Rhai map shape scripts see from
/// `head` / `list`.
fn file_meta_to_map(meta: &FileMeta) -> Map {
let mut m = Map::new();
m.insert("id".into(), meta.id.to_string().into());
m.insert("collection".into(), meta.collection.clone().into());
m.insert("name".into(), meta.name.clone().into());
m.insert("content_type".into(), meta.content_type.clone().into());
m.insert(
"size".into(),
i64::try_from(meta.size).unwrap_or(i64::MAX).into(),
);
m.insert("checksum".into(), meta.checksum.clone().into());
m.insert("created_at".into(), meta.created_at.to_rfc3339().into());
m.insert("updated_at".into(), meta.updated_at.to_rfc3339().into());
m
}
/// Pull a required string field out of a Rhai map; throw naming the
/// field if it's absent or not a string.
fn require_string(meta: &Map, field: &'static str) -> Result<String, Box<EvalAltResult>> {
match meta.get(field) {
Some(v) if v.is_string() => Ok(v.clone().into_string().unwrap_or_default()),
Some(_) => Err(format!("files::create: field '{field}' must be a string").into()),
None => Err(format!("files::create: missing required field '{field}'").into()),
}
}
/// Pull an optional string field; `None` when the key is absent or unit.
fn optional_string(meta: &Map, field: &'static str) -> Result<Option<String>, Box<EvalAltResult>> {
match meta.get(field) {
None => Ok(None),
Some(v) if v.is_unit() => Ok(None),
Some(v) if v.is_string() => Ok(Some(v.clone().into_string().unwrap_or_default())),
Some(_) => Err(format!("files::update: field '{field}' must be a string").into()),
}
}
/// Pull a required blob (`data`) out of a Rhai map; throw naming the
/// field if it's absent or not a blob.
fn require_blob(meta: &Map, field: &'static str) -> Result<Vec<u8>, Box<EvalAltResult>> {
match meta.get(field) {
Some(v) if v.is_blob() => Ok(v.clone().into_blob().unwrap_or_default()),
Some(_) => Err(format!("files: field '{field}' must be a Blob (byte array)").into()),
None => Err(format!("files: missing required field '{field}'").into()),
}
}
/// Run an async future inside the synchronous Rhai context. Mirrors
/// `kv::block_on`; safe because `LocalExecutorClient` runs the script
/// under `spawn_blocking`, so a runtime handle is reachable.
fn block_on<F, T>(fut: F) -> Result<T, Box<EvalAltResult>>
where
F: std::future::Future<Output = Result<T, FilesError>> + Send,
T: Send,
{
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
format!("files: no tokio runtime available: {e}").into(),
rhai::Position::NONE,
)
.into()
})?;
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(format!("files: {err}").into(), rhai::Position::NONE).into()
})
}

View File

@@ -15,8 +15,12 @@ pub mod bridge;
pub mod cx; pub mod cx;
pub mod dead_letters; pub mod dead_letters;
pub mod docs; pub mod docs;
pub mod email;
pub mod files;
pub mod http; pub mod http;
pub mod kv; pub mod kv;
pub mod pubsub;
pub mod secrets;
pub mod stdlib; pub mod stdlib;
pub use bridge::{dynamic_to_json, json_to_dynamic}; pub use bridge::{dynamic_to_json, json_to_dynamic};
@@ -37,5 +41,9 @@ pub fn register_all(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCal
kv::register(engine, services, cx.clone()); kv::register(engine, services, cx.clone());
docs::register(engine, services, cx.clone()); docs::register(engine, services, cx.clone());
dead_letters::register(engine, services, cx.clone()); dead_letters::register(engine, services, cx.clone());
http::register(engine, services, cx); http::register(engine, services, cx.clone());
files::register(engine, services, cx.clone());
pubsub::register(engine, services, cx.clone());
secrets::register(engine, services, cx.clone());
email::register(engine, services, cx);
} }

View File

@@ -0,0 +1,176 @@
//! `pubsub::` Rhai bridge — durable publish (v1.1.5).
//!
//! ```rhai
//! pubsub::publish_durable("user.created", #{ user_id: "abc" });
//! pubsub::publish_durable("metric", 42);
//! ```
//!
//! No handle pattern (topics ARE the grouping unit, so there's no
//! `::collection(...)`). The message is any JSON-serializable Rhai value
//! — Maps, Arrays, strings, numbers, bools, unit, and **Blobs (which
//! encode as base64 strings** so trigger handlers see them as base64 on
//! the wire). Nested blobs are encoded at any depth.
//!
//! `app_id` is derived from `cx.app_id` in the service — it never
//! appears in the script-side signature, preserving cross-app
//! isolation.
use std::sync::Arc;
use base64::engine::general_purpose::STANDARD;
use base64::Engine as _;
use picloud_shared::{PubsubError, SdkCallCx, Services};
use rhai::{Array, Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
use serde_json::Value as Json;
use tokio::runtime::Handle as TokioHandle;
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
let svc = services.pubsub.clone();
let mut module = Module::new();
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"publish_durable",
move |topic: &str, message: Dynamic| -> Result<(), Box<EvalAltResult>> {
let json = message_to_json(&message);
let svc = svc.clone();
let cx = cx.clone();
block_on(async move { svc.publish_durable(&cx, topic, json).await })
},
);
}
// `pubsub::subscriber_token(topics)` — uses the configured default
// TTL.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"subscriber_token",
move |topics: Array| -> Result<String, Box<EvalAltResult>> {
mint_token(&svc, &cx, topics, None)
},
);
}
// `pubsub::subscriber_token(topics, ttl)` — `ttl` is an integer
// (seconds) or `()` for the default.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"subscriber_token",
move |topics: Array, ttl: Dynamic| -> Result<String, Box<EvalAltResult>> {
let ttl = ttl_from_dynamic(&ttl)?;
mint_token(&svc, &cx, topics, ttl)
},
);
}
engine.register_static_module("pubsub", module.into());
}
/// Interpret the optional `ttl` argument: `()` → use the default,
/// integer → that many seconds, anything else → throw.
fn ttl_from_dynamic(ttl: &Dynamic) -> Result<Option<i64>, Box<EvalAltResult>> {
if ttl.is_unit() {
return Ok(None);
}
ttl.as_int().map(Some).map_err(|_| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
"pubsub::subscriber_token: ttl must be an integer (seconds) or ()".into(),
rhai::Position::NONE,
)
.into()
})
}
fn mint_token(
svc: &Arc<dyn picloud_shared::PubsubService>,
cx: &Arc<SdkCallCx>,
topics: Array,
ttl: Option<i64>,
) -> Result<String, Box<EvalAltResult>> {
// Every element must be a string; surface a clear error otherwise.
let mut names = Vec::with_capacity(topics.len());
for t in topics {
if !t.is_string() {
return Err(EvalAltResult::ErrorRuntime(
"pubsub::subscriber_token: topics must be an array of strings".into(),
rhai::Position::NONE,
)
.into());
}
names.push(t.into_string().unwrap_or_default());
}
let svc = svc.clone();
let cx = cx.clone();
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
format!("pubsub: no tokio runtime available: {e}").into(),
rhai::Position::NONE,
)
.into()
})?;
// SubscriberToken errors already carry the full
// "pubsub::subscriber_token: …" wording, so surface them verbatim.
handle
.block_on(async move { svc.mint_subscriber_token(&cx, names, ttl).await })
.map_err(|err| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(format!("{err}").into(), rhai::Position::NONE).into()
})
}
/// Convert a Rhai `Dynamic` message into JSON, base64-encoding any
/// `Blob` (at any nesting depth). Mirrors `bridge::dynamic_to_json` but
/// adds the blob arm the pub/sub wire contract requires.
fn message_to_json(value: &Dynamic) -> Json {
// Blob must be checked before the generic array path (a Blob is a
// `Vec<u8>`, distinct from a Rhai `Array`).
if value.is_blob() {
let blob = value.clone().into_blob().unwrap_or_default();
return Json::String(STANDARD.encode(&blob));
}
if value.is_unit() {
return Json::Null;
}
if let Ok(b) = value.as_bool() {
return Json::Bool(b);
}
if let Ok(i) = value.as_int() {
return Json::Number(i.into());
}
if let Ok(f) = value.as_float() {
return serde_json::Number::from_f64(f).map_or(Json::Null, Json::Number);
}
if value.is_string() {
return Json::String(value.clone().into_string().unwrap_or_default());
}
if let Some(arr) = value.clone().try_cast::<Array>() {
return Json::Array(arr.iter().map(message_to_json).collect());
}
if let Some(map) = value.clone().try_cast::<Map>() {
let mut out = serde_json::Map::new();
for (k, v) in map {
out.insert(k.to_string(), message_to_json(&v));
}
return Json::Object(out);
}
Json::String(value.to_string())
}
/// Run an async future inside the synchronous Rhai context. Mirrors
/// `kv::block_on`.
fn block_on<F>(fut: F) -> Result<(), Box<EvalAltResult>>
where
F: std::future::Future<Output = Result<(), PubsubError>> + Send,
{
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
format!("pubsub: no tokio runtime available: {e}").into(),
rhai::Position::NONE,
)
.into()
})?;
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(format!("pubsub: {err}").into(), rhai::Position::NONE).into()
})
}

View File

@@ -0,0 +1,153 @@
//! `secrets::` Rhai bridge — encrypted per-app secrets (v1.1.7).
//!
//! ```rhai
//! secrets::set("stripe_key", "sk_live_xxx");
//! secrets::set("oauth", #{ client_id: "abc", client_secret: "xyz" });
//! let key = secrets::get("stripe_key"); // value or ()
//! let removed = secrets::delete("stripe_key"); // bool
//! let page = secrets::list(#{ cursor: (), limit: 100 });
//! // page = #{ names: [...], next_cursor: () | "..." }
//! ```
//!
//! Collection-less (secrets are per-app, like pubsub topics) so there's
//! no `::collection(...)`. Values are any JSON-serializable Rhai value
//! (String/Map/Array/number/bool); a String round-trips back as a
//! String. `app_id` is derived from `cx.app_id` in the service — it
//! never appears in the script-side signature, preserving cross-app
//! isolation.
use std::sync::Arc;
use picloud_shared::{SdkCallCx, SecretsError, SecretsListPage, Services};
use rhai::{Array, Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
use tokio::runtime::Handle as TokioHandle;
use super::bridge::{dynamic_to_json, json_to_dynamic};
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
let svc = services.secrets.clone();
let mut module = Module::new();
// secrets::set(name, value) — overwrites if present.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"set",
move |name: &str, value: Dynamic| -> Result<(), Box<EvalAltResult>> {
let json = dynamic_to_json(&value);
let svc = svc.clone();
let cx = cx.clone();
block_on(async move { svc.set(&cx, name, json).await })
},
);
}
// secrets::get(name) — decoded value, or () if missing.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"get",
move |name: &str| -> Result<Dynamic, Box<EvalAltResult>> {
let svc = svc.clone();
let cx = cx.clone();
let opt = block_on(async move { svc.get(&cx, name).await })?;
Ok(opt.map_or(Dynamic::UNIT, json_to_dynamic))
},
);
}
// secrets::delete(name) — bool was-present.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"delete",
move |name: &str| -> Result<bool, Box<EvalAltResult>> {
let svc = svc.clone();
let cx = cx.clone();
block_on(async move { svc.delete(&cx, name).await })
},
);
}
// secrets::list(#{ cursor, limit }) — names only, cursor-paginated.
{
let svc = svc.clone();
let cx = cx.clone();
module.set_native_fn(
"list",
move |opts: Map| -> Result<Map, Box<EvalAltResult>> {
let (cursor, limit) = parse_list_opts(&opts)?;
let svc = svc.clone();
let cx = cx.clone();
let page: SecretsListPage =
block_on(async move { svc.list(&cx, cursor.as_deref(), limit).await })?;
Ok(list_page_to_map(page))
},
);
}
engine.register_static_module("secrets", module.into());
}
/// Pull `cursor` (string or `()`) and `limit` (int or `()`) out of the
/// options map. Unknown/extra keys are ignored.
fn parse_list_opts(opts: &Map) -> Result<(Option<String>, u32), Box<EvalAltResult>> {
let cursor = match opts.get("cursor") {
None => None,
Some(d) if d.is_unit() => None,
Some(d) if d.is_string() => Some(d.clone().into_string().unwrap_or_default()),
Some(_) => return Err(runtime_err("secrets::list: cursor must be a string or ()")),
};
let limit = match opts.get("limit") {
None => 0,
Some(d) if d.is_unit() => 0,
Some(d) => {
let n = d
.as_int()
.map_err(|_| runtime_err("secrets::list: limit must be an integer or ()"))?;
u32::try_from(n.max(0)).unwrap_or(u32::MAX)
}
};
Ok((cursor, limit))
}
fn list_page_to_map(page: SecretsListPage) -> Map {
let mut m = Map::new();
let names: Array = page.names.into_iter().map(Dynamic::from).collect();
m.insert("names".into(), names.into());
m.insert(
"next_cursor".into(),
page.next_cursor.map_or(Dynamic::UNIT, Dynamic::from),
);
m
}
// Returns the boxed error directly because every caller needs a
// `Box<EvalAltResult>` (Rhai's error type), matching the other bridges.
#[allow(clippy::unnecessary_box_returns)]
fn runtime_err(msg: &str) -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(msg.into(), rhai::Position::NONE).into()
}
/// Run a `SecretsService` future inside the synchronous Rhai context,
/// mapping any `SecretsError` to a Rhai runtime error. Mirrors
/// `kv::block_on` / `pubsub::block_on`.
fn block_on<T, F>(fut: F) -> Result<T, Box<EvalAltResult>>
where
F: std::future::Future<Output = Result<T, SecretsError>> + Send,
T: Send,
{
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(
format!("secrets: no tokio runtime available: {e}").into(),
rhai::Position::NONE,
)
.into()
})?;
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
EvalAltResult::ErrorRuntime(format!("secrets: {err}").into(), rhai::Position::NONE).into()
})
}

View File

@@ -99,6 +99,10 @@ async fn original_backend_error_is_logged_at_error_level() {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
Arc::new(FailingSource), Arc::new(FailingSource),
Arc::new(NoopHttpService), Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
); );
let engine = Engine::new(Limits::default(), services); let engine = Engine::new(Limits::default(), services);

View File

@@ -97,6 +97,10 @@ fn services_with(modules: Arc<dyn ModuleSource>) -> Services {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
modules, modules,
Arc::new(NoopHttpService), Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
) )
} }

View File

@@ -228,6 +228,10 @@ fn make_engine() -> Arc<Engine> {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource), Arc::new(NoopModuleSource),
Arc::new(NoopHttpService), Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
); );
Arc::new(Engine::new(Limits::default(), services)) Arc::new(Engine::new(Limits::default(), services))
} }

View File

@@ -0,0 +1,209 @@
//! `email::` SDK bridge integration tests — runs a real Rhai engine
//! against a recording `EmailService`. Verifies the Rhai map → DTO
//! plumbing (address coercion, the text-only vs multipart split). The
//! SMTP transport, validation, and authz are unit-tested at the service
//! layer in `manager-core::email_service`.
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
use picloud_shared::{
AppId, EmailError, EmailService, ExecutionId, NoopDeadLetterService, NoopDocsService,
NoopEventEmitter, NoopHttpService, NoopKvService, NoopModuleSource, OutboundEmail, RequestId,
ScriptId, ScriptSandbox, SdkCallCx, Services, TriggerEvent,
};
use serde_json::{json, Value};
#[derive(Default)]
struct RecordingEmail {
sent: Mutex<Vec<OutboundEmail>>,
}
#[async_trait]
impl EmailService for RecordingEmail {
async fn send(&self, _cx: &SdkCallCx, email: OutboundEmail) -> Result<(), EmailError> {
self.sent.lock().unwrap().push(email);
Ok(())
}
}
fn engine_with(rec: Arc<RecordingEmail>) -> Arc<Engine> {
let services = Services::new(
Arc::new(NoopKvService),
Arc::new(NoopDocsService),
Arc::new(NoopDeadLetterService),
Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource),
Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
rec,
);
Arc::new(Engine::new(Limits::default(), services))
}
fn baseline_request(app_id: AppId) -> ExecRequest {
let execution_id = ExecutionId::new();
ExecRequest {
execution_id,
request_id: RequestId::new(),
script_id: ScriptId::new(),
script_name: "email-test".into(),
invocation_type: InvocationType::Http,
path: "/email-test".into(),
headers: BTreeMap::new(),
body: Value::Null,
params: BTreeMap::new(),
query: BTreeMap::new(),
rest: String::new(),
sandbox_overrides: ScriptSandbox::default(),
app_id,
principal: None,
trigger_depth: 0,
root_execution_id: execution_id,
is_dead_letter_handler: false,
event: None,
}
}
async fn run(engine: Arc<Engine>, src: &str) -> Result<(), ()> {
let src = src.to_string();
let app = AppId::new();
tokio::task::spawn_blocking(move || engine.execute(&src, baseline_request(app)))
.await
.expect("spawn_blocking")
.map(|_| ())
.map_err(|_| ())
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn send_parses_single_recipient_text() {
let rec = Arc::new(RecordingEmail::default());
let engine = engine_with(rec.clone());
run(
engine,
r#"
email::send(#{
to: "alice@example.com",
from: "alerts@myapp.com",
subject: "Build complete",
text: "done"
});
#{ ok: true }
"#,
)
.await
.unwrap();
let g = rec.sent.lock().unwrap();
let e = g.last().unwrap();
assert_eq!(e.to, vec!["alice@example.com".to_string()]);
assert_eq!(e.from, "alerts@myapp.com");
assert_eq!(e.subject, "Build complete");
assert_eq!(e.text.as_deref(), Some("done"));
// email::send forces text-only even if html were present.
assert!(e.html.is_none());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn send_html_carries_both_parts_and_lists() {
let rec = Arc::new(RecordingEmail::default());
let engine = engine_with(rec.clone());
run(
engine,
r#"
email::send_html(#{
to: ["alice@x.com", "bob@y.com"],
cc: ["dave@z.com"],
bcc: ["audit@myapp.com"],
from: "alerts@myapp.com",
reply_to: "support@myapp.com",
subject: "hi",
text: "plain",
html: "<p>rich</p>"
});
#{ ok: true }
"#,
)
.await
.unwrap();
let g = rec.sent.lock().unwrap();
let e = g.last().unwrap();
assert_eq!(
e.to,
vec!["alice@x.com".to_string(), "bob@y.com".to_string()]
);
assert_eq!(e.cc, vec!["dave@z.com".to_string()]);
assert_eq!(e.bcc, vec!["audit@myapp.com".to_string()]);
assert_eq!(e.reply_to.as_deref(), Some("support@myapp.com"));
assert_eq!(e.text.as_deref(), Some("plain"));
assert_eq!(e.html.as_deref(), Some("<p>rich</p>"));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn inbound_email_event_visible_to_handler() {
// A handler invoked by an email:receive trigger sees the normalized
// message at ctx.event.email (built by the engine's ctx renderer).
let rec = Arc::new(RecordingEmail::default());
let engine = engine_with(rec);
let mut req = baseline_request(AppId::new());
req.event = Some(TriggerEvent::Email {
from: "sender@external.com".into(),
to: vec!["alice@myapp.com".into()],
cc: vec!["bob@myapp.com".into()],
subject: "Re: question".into(),
text: Some("hello".into()),
html: None,
received_at: chrono::DateTime::parse_from_rfc3339("2026-08-15T12:00:00Z")
.unwrap()
.with_timezone(&chrono::Utc),
message_id: Some("<abc@external.com>".into()),
});
let src = r#"
let e = ctx.event;
#{
source: e.source,
op: e.op,
from: e.email.from,
to0: e.email.to[0],
cc0: e.email.cc[0],
subject: e.email.subject,
text: e.email.text,
html_is_unit: type_of(e.email.html) == "()",
message_id: e.email.message_id
}
"#;
let src = src.to_string();
let body = tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.unwrap()
.unwrap()
.body;
assert_eq!(body["source"], json!("email"));
assert_eq!(body["op"], json!("receive"));
assert_eq!(body["from"], json!("sender@external.com"));
assert_eq!(body["to0"], json!("alice@myapp.com"));
assert_eq!(body["cc0"], json!("bob@myapp.com"));
assert_eq!(body["subject"], json!("Re: question"));
assert_eq!(body["text"], json!("hello"));
assert_eq!(body["html_is_unit"], json!(true));
assert_eq!(body["message_id"], json!("<abc@external.com>"));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn send_html_without_html_throws() {
let rec = Arc::new(RecordingEmail::default());
let engine = engine_with(rec.clone());
let res = run(
engine,
r#"
email::send_html(#{ to: "a@b.com", from: "c@d.com", subject: "x", text: "y" });
#{ ok: true }
"#,
)
.await;
assert!(res.is_err(), "send_html without html must throw");
assert!(rec.sent.lock().unwrap().is_empty());
}

View File

@@ -0,0 +1,336 @@
//! `files::` SDK bridge integration tests — runs a real Rhai engine
//! against an in-memory `FilesService` impl. Mirrors `tests/sdk_kv.rs`:
//! `tokio::task::spawn_blocking` so the bridge's `block_on` has a
//! reachable runtime. Exercises the actual Rhai surface — blob in/out,
//! the metadata map shape, and the missing-required-field throw.
use std::collections::BTreeMap;
use std::sync::Arc;
use async_trait::async_trait;
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
use picloud_shared::{
AppId, ExecutionId, FileMeta, FileUpdate, FilesError, FilesListPage, FilesService, NewFile,
NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopHttpService, NoopKvService,
NoopModuleSource, RequestId, ScriptId, ScriptSandbox, SdkCallCx, Services,
};
use serde_json::{json, Value};
use tokio::sync::Mutex;
use uuid::Uuid;
#[derive(Default)]
struct InMemoryFiles {
#[allow(clippy::type_complexity)]
data: Mutex<BTreeMap<(AppId, String, Uuid), (FileMeta, Vec<u8>)>>,
}
/// The in-memory fake doesn't exercise the real checksum path (the
/// `FsFilesRepo` tempdir tests in manager-core cover SHA-256); a stable
/// placeholder keeps the metadata map non-empty.
fn fake_checksum(bytes: &[u8]) -> String {
format!("len-{}", bytes.len())
}
#[async_trait]
impl FilesService for InMemoryFiles {
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError> {
if collection.is_empty() {
return Err(FilesError::InvalidCollection("empty".into()));
}
new.validate(100 * 1024 * 1024)?;
let id = Uuid::new_v4();
let now = chrono::Utc::now();
let meta = FileMeta {
id,
collection: collection.to_string(),
name: new.name.clone(),
content_type: new.content_type.clone(),
size: new.data.len() as u64,
checksum: fake_checksum(&new.data),
created_at: now,
updated_at: now,
};
self.data
.lock()
.await
.insert((cx.app_id, collection.to_string(), id), (meta, new.data));
Ok(id)
}
async fn head(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<FileMeta>, FilesError> {
let Ok(uuid) = Uuid::parse_str(id) else {
return Ok(None);
};
Ok(self
.data
.lock()
.await
.get(&(cx.app_id, collection.to_string(), uuid))
.map(|(m, _)| m.clone()))
}
async fn get(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<Vec<u8>>, FilesError> {
let Ok(uuid) = Uuid::parse_str(id) else {
return Ok(None);
};
Ok(self
.data
.lock()
.await
.get(&(cx.app_id, collection.to_string(), uuid))
.map(|(_, b)| b.clone()))
}
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
upd: FileUpdate,
) -> Result<(), FilesError> {
upd.validate(100 * 1024 * 1024)?;
let Ok(uuid) = Uuid::parse_str(id) else {
return Err(FilesError::NotFound);
};
let mut data = self.data.lock().await;
let key = (cx.app_id, collection.to_string(), uuid);
let Some((meta, _)) = data.get(&key).cloned() else {
return Err(FilesError::NotFound);
};
let mut meta = meta;
if let Some(n) = upd.name {
meta.name = n;
}
if let Some(ct) = upd.content_type {
meta.content_type = ct;
}
meta.size = upd.data.len() as u64;
meta.checksum = fake_checksum(&upd.data);
data.insert(key, (meta, upd.data));
Ok(())
}
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError> {
let Ok(uuid) = Uuid::parse_str(id) else {
return Ok(false);
};
Ok(self
.data
.lock()
.await
.remove(&(cx.app_id, collection.to_string(), uuid))
.is_some())
}
async fn list(
&self,
cx: &SdkCallCx,
collection: &str,
_cursor: Option<&str>,
_limit: u32,
) -> Result<FilesListPage, FilesError> {
let data = self.data.lock().await;
let files: Vec<FileMeta> = data
.iter()
.filter(|((a, c, _), _)| *a == cx.app_id && c == collection)
.map(|(_, (m, _))| m.clone())
.collect();
Ok(FilesListPage {
files,
next_cursor: None,
})
}
}
fn make_engine() -> Arc<Engine> {
let services = Services::new(
Arc::new(NoopKvService),
Arc::new(NoopDocsService),
Arc::new(NoopDeadLetterService),
Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource),
Arc::new(NoopHttpService),
Arc::new(InMemoryFiles::default()),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
);
Arc::new(Engine::new(Limits::default(), services))
}
fn baseline_request(app_id: AppId) -> ExecRequest {
let execution_id = ExecutionId::new();
ExecRequest {
execution_id,
request_id: RequestId::new(),
script_id: ScriptId::new(),
script_name: "files-test".into(),
invocation_type: InvocationType::Http,
path: "/files-test".into(),
headers: BTreeMap::new(),
body: Value::Null,
params: BTreeMap::new(),
query: BTreeMap::new(),
rest: String::new(),
sandbox_overrides: ScriptSandbox::default(),
app_id,
principal: None,
trigger_depth: 0,
root_execution_id: execution_id,
is_dead_letter_handler: false,
event: None,
}
}
async fn run_script(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
let src = src.to_string();
tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic")
.expect("script execution should succeed")
.body
}
async fn run_script_err(engine: Arc<Engine>, src: &str, req: ExecRequest) -> String {
let src = src.to_string();
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic");
format!("{:?}", res.expect_err("script should error"))
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_create_get_round_trip_via_blob() {
let engine = make_engine();
let app = AppId::new();
// base64("hello") = "aGVsbG8="; decode → blob; create; get back; encode.
let src = r#"
let c = files::collection("avatars");
let data = base64::decode("aGVsbG8=");
let id = c.create(#{ name: "a.txt", content_type: "text/plain", data: data });
let back = c.get(id);
base64::encode(back)
"#;
let body = run_script(engine, src, baseline_request(app)).await;
assert_eq!(body, json!("aGVsbG8="));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_head_returns_metadata_map() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
let data = base64::decode("aGVsbG8=");
let id = c.create(#{ name: "a.txt", content_type: "text/plain", data: data });
let meta = c.head(id);
#{ name: meta.name, content_type: meta.content_type, size: meta.size, has_checksum: meta.checksum != () }
"#;
let body = run_script(engine, src, baseline_request(app)).await;
assert_eq!(
body,
json!({ "name": "a.txt", "content_type": "text/plain", "size": 5, "has_checksum": true })
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_get_and_head_missing_return_unit() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
let g = c.get("00000000-0000-0000-0000-000000000000");
let h = c.head("00000000-0000-0000-0000-000000000000");
#{ g: g == (), h: h == () }
"#;
let body = run_script(engine, src, baseline_request(app)).await;
assert_eq!(body, json!({ "g": true, "h": true }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_update_then_delete() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
let id = c.create(#{ name: "a", content_type: "text/plain", data: base64::decode("YQ==") });
c.update(id, #{ data: base64::decode("YmM=") }); // "bc"
let after = base64::encode(c.get(id));
let removed = c.delete(id);
let gone = c.delete(id);
#{ after: after, removed: removed, gone: gone }
"#;
let body = run_script(engine, src, baseline_request(app)).await;
assert_eq!(
body,
json!({ "after": "YmM=", "removed": true, "gone": false })
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_create_missing_data_throws_naming_field() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
c.create(#{ name: "a", content_type: "text/plain" })
"#;
let err = run_script_err(engine, src, baseline_request(app)).await;
assert!(
err.contains("data"),
"error should name the missing field: {err}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_create_missing_name_throws_naming_field() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
c.create(#{ content_type: "text/plain", data: base64::decode("YQ==") })
"#;
let err = run_script_err(engine, src, baseline_request(app)).await;
assert!(
err.contains("name"),
"error should name the missing field: {err}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_empty_collection_name_throws() {
let engine = make_engine();
let app = AppId::new();
let err = run_script_err(engine, r#"files::collection("")"#, baseline_request(app)).await;
assert!(err.to_lowercase().contains("empty"), "got {err}");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn files_list_returns_files_array() {
let engine = make_engine();
let app = AppId::new();
let src = r#"
let c = files::collection("avatars");
c.create(#{ name: "a", content_type: "text/plain", data: base64::decode("YQ==") });
c.create(#{ name: "b", content_type: "text/plain", data: base64::decode("Yg==") });
let page = c.list();
page.files.len()
"#;
let body = run_script(engine, src, baseline_request(app)).await;
assert_eq!(body, json!(2));
}

View File

@@ -88,6 +88,10 @@ fn engine_with(http: Arc<dyn HttpService>) -> Arc<Engine> {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource), Arc::new(NoopModuleSource),
http, http,
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
); );
Arc::new(Engine::new(Limits::default(), services)) Arc::new(Engine::new(Limits::default(), services))
} }

View File

@@ -107,6 +107,10 @@ fn make_engine() -> Arc<Engine> {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource), Arc::new(NoopModuleSource),
Arc::new(NoopHttpService), Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
); );
Arc::new(Engine::new(Limits::default(), services)) Arc::new(Engine::new(Limits::default(), services))
} }

View File

@@ -0,0 +1,159 @@
//! `pubsub::` SDK bridge integration tests — runs a real Rhai engine
//! against an in-memory `PubsubService` that records the published
//! `(topic, message)`. Verifies the message JSON encoding the wire
//! contract requires: Maps, Arrays, strings, numbers, bool, null, and
//! **Blob → base64**, including nesting.
use std::collections::BTreeMap;
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
use picloud_shared::{
AppId, ExecutionId, NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopFilesService,
NoopHttpService, NoopKvService, NoopModuleSource, PubsubError, PubsubService, RequestId,
ScriptId, ScriptSandbox, SdkCallCx, Services,
};
use serde_json::{json, Value};
#[derive(Default)]
struct RecordingPubsub {
last: Mutex<Option<(String, Value)>>,
}
#[async_trait]
impl PubsubService for RecordingPubsub {
async fn publish_durable(
&self,
_cx: &SdkCallCx,
topic: &str,
message: Value,
) -> Result<(), PubsubError> {
if topic.trim().is_empty() {
return Err(PubsubError::EmptyTopic);
}
*self.last.lock().unwrap() = Some((topic.to_string(), message));
Ok(())
}
}
fn make_engine(svc: Arc<RecordingPubsub>) -> Arc<Engine> {
let services = Services::new(
Arc::new(NoopKvService),
Arc::new(NoopDocsService),
Arc::new(NoopDeadLetterService),
Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource),
Arc::new(NoopHttpService),
Arc::new(NoopFilesService),
svc,
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
);
Arc::new(Engine::new(Limits::default(), services))
}
fn baseline_request(app_id: AppId) -> ExecRequest {
let execution_id = ExecutionId::new();
ExecRequest {
execution_id,
request_id: RequestId::new(),
script_id: ScriptId::new(),
script_name: "pubsub-test".into(),
invocation_type: InvocationType::Http,
path: "/pubsub-test".into(),
headers: BTreeMap::new(),
body: Value::Null,
params: BTreeMap::new(),
query: BTreeMap::new(),
rest: String::new(),
sandbox_overrides: ScriptSandbox::default(),
app_id,
principal: None,
trigger_depth: 0,
root_execution_id: execution_id,
is_dead_letter_handler: false,
event: None,
}
}
async fn run(engine: Arc<Engine>, src: &str, req: ExecRequest) {
let src = src.to_string();
tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic")
.expect("script execution should succeed");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_map_message() {
let svc = Arc::new(RecordingPubsub::default());
let engine = make_engine(svc.clone());
run(
engine,
r#"pubsub::publish_durable("user.created", #{ user_id: "abc", n: 7, ok: true });"#,
baseline_request(AppId::new()),
)
.await;
let (topic, msg) = svc.last.lock().unwrap().clone().unwrap();
assert_eq!(topic, "user.created");
assert_eq!(msg, json!({ "user_id": "abc", "n": 7, "ok": true }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_scalar_and_array_and_null() {
let svc = Arc::new(RecordingPubsub::default());
let engine = make_engine(svc.clone());
run(
engine,
r#"pubsub::publish_durable("a", [1, "two", false, ()]);"#,
baseline_request(AppId::new()),
)
.await;
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
assert_eq!(msg, json!([1, "two", false, null]));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_number_scalar() {
let svc = Arc::new(RecordingPubsub::default());
let engine = make_engine(svc.clone());
run(
engine,
r#"pubsub::publish_durable("metric", 42);"#,
baseline_request(AppId::new()),
)
.await;
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
assert_eq!(msg, json!(42));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_blob_encodes_base64_including_nested() {
let svc = Arc::new(RecordingPubsub::default());
let engine = make_engine(svc.clone());
// base64("hello") = "aGVsbG8=" (STANDARD, padded).
run(
engine,
r#"
let data = base64::decode("aGVsbG8=");
pubsub::publish_durable("blobs", #{ raw: data, list: [data] });
"#,
baseline_request(AppId::new()),
)
.await;
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
assert_eq!(msg, json!({ "raw": "aGVsbG8=", "list": ["aGVsbG8="] }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_empty_topic_throws() {
let svc = Arc::new(RecordingPubsub::default());
let engine = make_engine(svc.clone());
let src = r#"pubsub::publish_durable("", 1);"#.to_string();
let req = baseline_request(AppId::new());
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic");
assert!(res.is_err(), "empty topic should throw");
}

View File

@@ -0,0 +1,213 @@
//! `secrets::` SDK bridge integration tests — runs a real Rhai engine
//! against an in-memory `SecretsService` impl. Mirrors `sdk_kv.rs`: the
//! engine runs under `spawn_blocking` so the bridge's `block_on` has a
//! reachable runtime.
//!
//! This exercises the Rhai⇄JSON plumbing + the static `secrets` module
//! (set/get/delete/list, the missing→() contract, and the
//! String/Map/Array type round-trip). Encryption + authz + the
//! cross-app boundary are unit-tested at the service layer in
//! `manager-core::secrets_service`.
use std::collections::BTreeMap;
use std::sync::Arc;
use async_trait::async_trait;
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
use picloud_shared::{
AppId, ExecutionId, NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopHttpService,
NoopKvService, NoopModuleSource, RequestId, ScriptId, ScriptSandbox, SdkCallCx, SecretsError,
SecretsListPage, SecretsService, Services,
};
use serde_json::{json, Value};
use tokio::sync::Mutex;
/// In-memory secrets store keyed by `(app_id, name)`. Stores the JSON
/// value directly — the bridge test only cares about the Rhai plumbing,
/// not the at-rest encryption (which the service layer owns).
#[derive(Default)]
struct InMemorySecrets {
data: Mutex<BTreeMap<(AppId, String), Value>>,
}
#[async_trait]
impl SecretsService for InMemorySecrets {
async fn get(&self, cx: &SdkCallCx, name: &str) -> Result<Option<Value>, SecretsError> {
picloud_shared::validate_secret_name(name)?;
Ok(self
.data
.lock()
.await
.get(&(cx.app_id, name.to_string()))
.cloned())
}
async fn set(&self, cx: &SdkCallCx, name: &str, value: Value) -> Result<(), SecretsError> {
picloud_shared::validate_secret_name(name)?;
self.data
.lock()
.await
.insert((cx.app_id, name.to_string()), value);
Ok(())
}
async fn delete(&self, cx: &SdkCallCx, name: &str) -> Result<bool, SecretsError> {
picloud_shared::validate_secret_name(name)?;
Ok(self
.data
.lock()
.await
.remove(&(cx.app_id, name.to_string()))
.is_some())
}
async fn list(
&self,
cx: &SdkCallCx,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsListPage, SecretsError> {
let data = self.data.lock().await;
let mut names: Vec<String> = data
.iter()
.filter(|((a, _), _)| *a == cx.app_id)
.map(|((_, n), _)| n.clone())
.filter(|n| cursor.is_none_or(|c| n.as_str() > c))
.collect();
names.sort();
let take = if limit == 0 {
usize::MAX
} else {
limit as usize
};
let next_cursor = if names.len() > take {
names.truncate(take);
names.last().cloned()
} else {
None
};
Ok(SecretsListPage { names, next_cursor })
}
}
fn make_engine() -> Arc<Engine> {
let services = Services::new(
Arc::new(NoopKvService),
Arc::new(NoopDocsService),
Arc::new(NoopDeadLetterService),
Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource),
Arc::new(NoopHttpService),
Arc::new(picloud_shared::NoopFilesService),
Arc::new(picloud_shared::NoopPubsubService),
Arc::new(InMemorySecrets::default()),
Arc::new(picloud_shared::NoopEmailService),
);
Arc::new(Engine::new(Limits::default(), services))
}
fn baseline_request(app_id: AppId) -> ExecRequest {
let execution_id = ExecutionId::new();
ExecRequest {
execution_id,
request_id: RequestId::new(),
script_id: ScriptId::new(),
script_name: "secrets-test".into(),
invocation_type: InvocationType::Http,
path: "/secrets-test".into(),
headers: BTreeMap::new(),
body: Value::Null,
params: BTreeMap::new(),
query: BTreeMap::new(),
rest: String::new(),
sandbox_overrides: ScriptSandbox::default(),
app_id,
principal: None,
trigger_depth: 0,
root_execution_id: execution_id,
is_dead_letter_handler: false,
event: None,
}
}
async fn run_script(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
let src = src.to_string();
tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic")
.expect("script execution should succeed")
.body
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn set_then_get_string_round_trips() {
let engine = make_engine();
let src = r#"
secrets::set("stripe_key", "sk_live_xxx");
secrets::get("stripe_key")
"#;
let body = run_script(engine, src, baseline_request(AppId::new())).await;
// A String comes back a String, not a JSON-quoted "\"sk_live_xxx\"".
assert_eq!(body, json!("sk_live_xxx"));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn set_then_get_map_round_trips() {
let engine = make_engine();
let src = r#"
secrets::set("oauth", #{ client_id: "abc", client_secret: "xyz" });
secrets::get("oauth")
"#;
let body = run_script(engine, src, baseline_request(AppId::new())).await;
assert_eq!(body, json!({ "client_id": "abc", "client_secret": "xyz" }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn get_missing_returns_unit() {
let engine = make_engine();
let src = r#"
let v = secrets::get("nope");
#{ is_unit: type_of(v) == "()" }
"#;
let body = run_script(engine, src, baseline_request(AppId::new())).await;
assert_eq!(body, json!({ "is_unit": true }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn delete_returns_was_present() {
let engine = make_engine();
let src = r#"
secrets::set("k", "v");
let first = secrets::delete("k");
let second = secrets::delete("k");
#{ first: first, second: second }
"#;
let body = run_script(engine, src, baseline_request(AppId::new())).await;
assert_eq!(body, json!({ "first": true, "second": false }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn list_returns_names_and_cursor() {
let engine = make_engine();
let src = r#"
secrets::set("a", 1);
secrets::set("b", 2);
secrets::set("c", 3);
let page = secrets::list(#{ cursor: (), limit: 2 });
page
"#;
let body = run_script(engine, src, baseline_request(AppId::new())).await;
assert_eq!(body["names"], json!(["a", "b"]));
assert_eq!(body["next_cursor"], json!("b"));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn empty_name_throws() {
let engine = make_engine();
let src = r#" secrets::set("", "v"); #{ ok: true } "#;
let app = AppId::new();
let out = tokio::task::spawn_blocking(move || engine.execute(src, baseline_request(app)))
.await
.expect("spawn_blocking");
assert!(out.is_err(), "empty secret name must throw");
}

View File

@@ -0,0 +1,244 @@
//! `pubsub::subscriber_token` SDK bridge integration tests (v1.1.6).
//!
//! Runs a real Rhai engine against a fake `PubsubService` whose
//! `mint_subscriber_token` mirrors the production validation (principal
//! required, non-empty topics, ttl clamp, externally-subscribable check)
//! and signs a real token. These cover the bridge surface: array →
//! `Vec<String>` forwarding, the omitted/`()`/integer ttl handling, and
//! errors surfacing as thrown Rhai errors. The authoritative validation
//! logic is unit-tested in `manager-core::pubsub_service`.
use std::collections::BTreeMap;
use std::sync::Arc;
use async_trait::async_trait;
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
use picloud_shared::subscriber_token::{self, TokenClaims};
use picloud_shared::{
AdminUserId, AppId, ExecutionId, InstanceRole, NoopDeadLetterService, NoopDocsService,
NoopEventEmitter, NoopFilesService, NoopHttpService, NoopKvService, NoopModuleSource,
Principal, PubsubError, PubsubService, RequestId, ScriptId, ScriptSandbox, SdkCallCx, Services,
};
use serde_json::Value;
const FAKE_KEY: [u8; 32] = [7u8; 32];
const MIN_TTL: i64 = 10;
const MAX_TTL: i64 = 86_400;
const DEFAULT_TTL: i64 = 3_600;
/// Fake that mirrors the production mint rules and signs with FAKE_KEY.
#[derive(Default)]
struct FakeMintPubsub;
#[async_trait]
impl PubsubService for FakeMintPubsub {
async fn publish_durable(
&self,
_cx: &SdkCallCx,
_topic: &str,
_message: Value,
) -> Result<(), PubsubError> {
Ok(())
}
async fn mint_subscriber_token(
&self,
cx: &SdkCallCx,
topics: Vec<String>,
ttl_seconds: Option<i64>,
) -> Result<String, PubsubError> {
if cx.principal.is_none() {
return Err(PubsubError::SubscriberToken(
"pubsub::subscriber_token: requires an authenticated principal".into(),
));
}
if topics.is_empty() {
return Err(PubsubError::SubscriberToken(
"pubsub::subscriber_token: topics list must not be empty".into(),
));
}
let ttl = ttl_seconds.unwrap_or(DEFAULT_TTL);
if !(MIN_TTL..=MAX_TTL).contains(&ttl) {
return Err(PubsubError::SubscriberToken(format!(
"pubsub::subscriber_token: ttl_seconds must be between {MIN_TTL} and {MAX_TTL}"
)));
}
for name in &topics {
// Only "chat" and "notify" are "registered" in this fake.
if name != "chat" && name != "notify" {
return Err(PubsubError::SubscriberToken(format!(
"pubsub::subscriber_token: topic {name} is not externally subscribable"
)));
}
}
let now = 1_000_000;
Ok(subscriber_token::sign(
&FAKE_KEY,
&TokenClaims {
app_id: cx.app_id,
topics,
exp: now + ttl,
iat: now,
},
))
}
}
fn make_engine() -> Arc<Engine> {
let services = Services::new(
Arc::new(NoopKvService),
Arc::new(NoopDocsService),
Arc::new(NoopDeadLetterService),
Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource),
Arc::new(NoopHttpService),
Arc::new(NoopFilesService),
Arc::new(FakeMintPubsub),
Arc::new(picloud_shared::NoopSecretsService),
Arc::new(picloud_shared::NoopEmailService),
);
Arc::new(Engine::new(Limits::default(), services))
}
fn request(app_id: AppId, with_principal: bool) -> ExecRequest {
let execution_id = ExecutionId::new();
ExecRequest {
execution_id,
request_id: RequestId::new(),
script_id: ScriptId::new(),
script_name: "token-test".into(),
invocation_type: InvocationType::Http,
path: "/token-test".into(),
headers: BTreeMap::new(),
body: Value::Null,
params: BTreeMap::new(),
query: BTreeMap::new(),
rest: String::new(),
sandbox_overrides: ScriptSandbox::default(),
app_id,
principal: with_principal.then(|| Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Owner,
scopes: None,
app_binding: None,
}),
trigger_depth: 0,
root_execution_id: execution_id,
is_dead_letter_handler: false,
event: None,
}
}
async fn run_ok(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
let src = src.to_string();
tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic")
.expect("script execution should succeed")
.body
}
async fn run_err(engine: Arc<Engine>, src: &str, req: ExecRequest) {
let src = src.to_string();
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
.await
.expect("spawn_blocking should not panic");
assert!(res.is_err(), "expected script to throw");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn token_contains_topics_and_expiry() {
let app = AppId::new();
let body = run_ok(
make_engine(),
r#"#{ token: pubsub::subscriber_token(["chat", "notify"], 120) }"#,
request(app, true),
)
.await;
let token = body["token"].as_str().expect("token string");
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
assert_eq!(claims.app_id, app);
assert_eq!(
claims.topics,
vec!["chat".to_string(), "notify".to_string()]
);
assert_eq!(claims.exp - claims.iat, 120);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn omitted_ttl_uses_default() {
let app = AppId::new();
let body = run_ok(
make_engine(),
r#"#{ token: pubsub::subscriber_token(["chat"]) }"#,
request(app, true),
)
.await;
let token = body["token"].as_str().unwrap();
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
assert_eq!(claims.exp - claims.iat, DEFAULT_TTL);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn unit_ttl_uses_default() {
let app = AppId::new();
let body = run_ok(
make_engine(),
r#"#{ token: pubsub::subscriber_token(["chat"], ()) }"#,
request(app, true),
)
.await;
let token = body["token"].as_str().unwrap();
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
assert_eq!(claims.exp - claims.iat, DEFAULT_TTL);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn empty_topics_throws() {
run_err(
make_engine(),
r"pubsub::subscriber_token([], 60)",
request(AppId::new(), true),
)
.await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn ttl_below_min_throws() {
run_err(
make_engine(),
r#"pubsub::subscriber_token(["chat"], 5)"#,
request(AppId::new(), true),
)
.await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn ttl_above_max_throws() {
run_err(
make_engine(),
r#"pubsub::subscriber_token(["chat"], 90000)"#,
request(AppId::new(), true),
)
.await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn anonymous_principal_throws() {
run_err(
make_engine(),
r#"pubsub::subscriber_token(["chat"], 60)"#,
request(AppId::new(), false),
)
.await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn unregistered_topic_throws() {
run_err(
make_engine(),
r#"pubsub::subscriber_token(["chat", "secret"], 60)"#,
request(AppId::new(), true),
)
.await;
}

View File

@@ -31,8 +31,13 @@ reqwest.workspace = true
argon2.workspace = true argon2.workspace = true
sha2.workspace = true sha2.workspace = true
# HMAC-SHA256 verification of inbound-email provider signatures (v1.1.7).
hmac.workspace = true
hex.workspace = true
base64.workspace = true base64.workspace = true
data-encoding.workspace = true data-encoding.workspace = true
# Outbound SMTP email (v1.1.7 email::send / send_html).
lettre.workspace = true
[dev-dependencies] [dev-dependencies]
tokio.workspace = true tokio.workspace = true

View File

@@ -0,0 +1,25 @@
-- v1.1.5: filesystem-backed blob storage. The row holds metadata +
-- the SHA-256 checksum; the blob bytes live on disk at
-- <PICLOUD_FILES_ROOT>/files/<app_id>/<collection>/<id[0:2]>/<id>
-- (never in Postgres). Identity tuple is (app_id, collection, id) per
-- docs/sdk-shape.md, matching KV/docs collection scoping.
--
-- The checksum is computed in a single pass during the atomic write and
-- re-verified on read (FilesError::Corrupted on mismatch). Per-app
-- quotas are deferred to v1.2; only the per-file size cap is enforced
-- (in the service, not the schema).
CREATE TABLE files (
app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
collection TEXT NOT NULL,
id UUID NOT NULL,
name TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
checksum_sha256 TEXT NOT NULL, -- hex, 64 chars, lowercase
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (app_id, collection, id)
);
-- List + cursor pagination scans by (app_id, collection).
CREATE INDEX idx_files_app_collection ON files (app_id, collection);

View File

@@ -0,0 +1,29 @@
-- v1.1.5: extend the triggers framework to recognise `files` as the
-- fifth concrete kind (after `kv`/`dead_letter` v1.1.1, `docs` v1.1.2,
-- `cron` v1.1.4). Mirrors the 0014/0017 extensions exactly: two CHECK
-- constraints widen (strictly gaining `'files'`), one new detail table.
--
-- Files rows route through the SAME generic dispatcher path as the
-- other event kinds (single match-arm extension on the Rust side). The
-- only new machinery is the FilesServiceImpl emitting ServiceEvents
-- that the OutboxEventEmitter fans out — identical to KV/docs.
-- Extend triggers.kind to include 'files'. No existing row carries a
-- value outside the widened set, so the drop+add is safe.
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron', 'files'));
-- Extend outbox.source_kind to include 'files'.
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs', 'cron', 'files'));
-- One row per files trigger. Mirrors kv_trigger_details:
-- collection_glob — "*", "exact", or "prefix*"
-- ops — subset of {create, update, delete}, empty = any
CREATE TABLE files_trigger_details (
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
collection_glob TEXT NOT NULL,
ops TEXT[] NOT NULL
);

View File

@@ -0,0 +1,34 @@
-- v1.1.5: extend the triggers framework to recognise `pubsub` as the
-- sixth concrete kind. Same Layout-E shape as files (0019): two CHECK
-- constraints widen, one new detail table.
--
-- Pub/sub fans out at PUBLISH time (one outbox row per matching trigger,
-- written by the PubsubServiceImpl), so the dispatcher needs no pubsub-
-- specific branching — a pubsub outbox row dispatches like any other
-- async trigger.
-- Extend triggers.kind to include 'pubsub'.
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron', 'files', 'pubsub'));
-- Extend outbox.source_kind to include 'pubsub'.
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs',
'cron', 'files', 'pubsub'));
-- One row per pubsub trigger. `topic_pattern` is "exact", "prefix.*",
-- or "*" — validated in Rust at trigger creation. Topics are implicit
-- on first publish; the external-subscribable `topics` table is v1.1.6.
CREATE TABLE pubsub_trigger_details (
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
topic_pattern TEXT NOT NULL
);
-- Hot lookup for fan-out: "all enabled pubsub triggers in app X".
-- Third partial index of its kind (after v1.1.1's idx_triggers_app_kind_
-- enabled); partial indexes are tiny and the planner picks the narrowest.
CREATE INDEX idx_triggers_app_pubsub_enabled
ON triggers (app_id, kind)
WHERE enabled = TRUE AND kind = 'pubsub';

View File

@@ -0,0 +1,31 @@
-- v1.1.6: Explicit registration for externally-subscribable topics.
--
-- Internal-only topics remain implicit per the §5 design-notes
-- decision: anyone can publish_durable("any.topic", msg) and triggers
-- can subscribe without a row here. This table only holds topics that
-- have been explicitly externalized — external SSE subscribers can
-- only subscribe to topics with a row here AND external_subscribable
-- = TRUE.
--
-- The publish path (v1.1.5's publish_durable) does NOT consult this
-- table: publishing to a topic with no row still fans out to triggers
-- and to any in-process external subscribers (none exist for an
-- unregistered topic, since external subscribers can't subscribe to
-- one). The topics table is read by the SSE subscribe path only.
--
-- auth_mode values: 'public' + 'token' in v1.1.6. 'session' arrives in
-- v1.1.8 (users-SDK); 'script' arrives in v1.2 (script-mediated auth).
-- The CHECK constraint extends in those releases.
CREATE TABLE topics (
app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
name TEXT NOT NULL,
external_subscribable BOOL NOT NULL DEFAULT FALSE,
auth_mode TEXT NOT NULL DEFAULT 'public'
CHECK (auth_mode IN ('public', 'token')),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (app_id, name)
);
-- Hot lookup: "is topic T in app X externally subscribable?" The PK
-- (app_id, name) already covers this; an explicit index is redundant.

View File

@@ -0,0 +1,19 @@
-- v1.1.6: per-app secret material. Currently holds the HMAC signing key
-- used to mint + verify realtime subscriber tokens
-- (pubsub::subscriber_token → SSE /realtime/topics handshake).
--
-- The key is:
-- * stable across restarts (issued tokens stay valid until expiry),
-- * per-app (a token signed by app A is rejected by app B),
-- * never script-accessible (scripts can't print/exfiltrate it — the
-- SDK only mints tokens, it never returns the key).
--
-- The row is created lazily on the first pubsub::subscriber_token call
-- for an app (32 random bytes). This table is the natural home for
-- v1.1.7's encrypted per-app secrets work.
CREATE TABLE app_secrets (
app_id UUID PRIMARY KEY REFERENCES apps(id) ON DELETE CASCADE,
realtime_signing_key BYTEA NOT NULL, -- 32 random bytes
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

View File

@@ -0,0 +1,24 @@
-- v1.1.7: encrypted per-app secrets.
--
-- Operational config (API keys, OAuth tokens, webhook signing keys)
-- encrypted at rest with the process master key (AES-256-GCM). Both the
-- ciphertext (16-byte GCM auth tag appended) and the 12-byte nonce are
-- stored; the master key itself never lives in the database. See
-- `picloud_shared::crypto` + `manager-core::secrets_service`.
--
-- This is the user-facing `secrets::*` store. It is intentionally
-- separate from `app_secrets` (the one-row-per-app realtime signing
-- key, 0022): different cardinality (many named rows per app), and the
-- realtime key is encrypted in place by migration 0025.
CREATE TABLE secrets (
app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
name TEXT NOT NULL,
encrypted_value BYTEA NOT NULL, -- ciphertext incl. 16-byte GCM auth tag
nonce BYTEA NOT NULL, -- 12 bytes
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
PRIMARY KEY (app_id, name)
);
CREATE INDEX idx_secrets_app ON secrets (app_id);

View File

@@ -0,0 +1,32 @@
-- v1.1.7: inbound email triggers (email:receive).
--
-- A configured provider (Mailgun / Postmark / SendGrid / SES) POSTs
-- inbound email to POST /api/v1/email-inbound/{app_id}/{trigger_id};
-- the receiver normalizes it into a TriggerEvent::Email and enqueues an
-- outbox row for the trigger's handler. v1.1.7 ships the webhook path;
-- a native SMTP listener is v1.3+.
-- Widen the trigger-kind + outbox-source CHECK constraints to admit
-- 'email'.
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron',
'files', 'pubsub', 'email'));
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs',
'cron', 'files', 'pubsub', 'email'));
-- Per-trigger inbound config. The HMAC secret used to verify provider
-- signatures is stored ENCRYPTED at rest (AES-256-GCM under the process
-- master key) — a deviation from the original brief's plaintext column,
-- chosen to keep all operationally-secret material encrypted. The
-- receiver decrypts it per inbound request. NULL columns mean the
-- trigger has no signature verification (accepts any POST to its URL —
-- relies on URL secrecy).
CREATE TABLE email_trigger_details (
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
inbound_secret_encrypted BYTEA, -- ciphertext incl. GCM auth tag (NULL = unsigned)
inbound_secret_nonce BYTEA -- 12 bytes (NULL = unsigned)
);

View File

@@ -0,0 +1,24 @@
-- v1.1.7: encrypt the realtime signing key at rest (two-phase).
--
-- Phase 1 (this migration + the v1.1.7 startup task):
-- * add NULL-able encrypted columns,
-- * drop the NOT NULL on the plaintext column so newly-generated keys
-- can be stored encrypted-only,
-- * the application startup task `migrate_plaintext_keys` encrypts each
-- existing plaintext key into the new columns (plaintext is LEFT in
-- place during the compat window for rollback safety).
--
-- The `RealtimeAuthorityImpl` read path prefers the encrypted columns and
-- falls back to plaintext, so SSE keeps working throughout.
--
-- Phase 2 (v1.1.8): once all rows are migrated, a follow-up migration
-- drops the plaintext `realtime_signing_key` column.
ALTER TABLE app_secrets
ADD COLUMN realtime_signing_key_encrypted BYTEA,
ADD COLUMN realtime_signing_key_nonce BYTEA;
-- New keys (post-v1.1.7) are stored encrypted-only, so the plaintext
-- column must accept NULL.
ALTER TABLE app_secrets
ALTER COLUMN realtime_signing_key DROP NOT NULL;

View File

@@ -0,0 +1,241 @@
//! `AppSecretsRepo` — per-app secret material (v1.1.6, encrypted v1.1.7).
//!
//! Holds the HMAC signing key for realtime subscriber tokens. The key is
//! generated lazily (32 random bytes) on the first
//! `pubsub::subscriber_token` call for an app and never changes
//! thereafter (no rotation API yet). The key is never exposed to
//! scripts: the SDK mints tokens, it never returns the key.
//!
//! **v1.1.7 at-rest encryption (two-phase).** The key is now sealed with
//! the process master key (AES-256-GCM). New keys are written
//! encrypted-only; the startup task [`PostgresAppSecretsRepo::migrate_plaintext_keys`]
//! encrypts any pre-existing plaintext rows. The read path prefers the
//! encrypted columns and falls back to the plaintext column during the
//! compat window (migration 0025 made it NULL-able; v1.1.8 drops it).
use async_trait::async_trait;
use picloud_shared::{crypto, AppId, MasterKey};
use rand::RngCore;
use sqlx::PgPool;
use uuid::Uuid;
/// Length of a freshly-generated realtime signing key.
pub const SIGNING_KEY_LEN: usize = 32;
#[derive(Debug, thiserror::Error)]
pub enum AppSecretsRepoError {
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
/// A stored encrypted signing key could not be decrypted — corrupted
/// row or a master-key mismatch (e.g. `PICLOUD_SECRET_KEY` changed).
#[error("realtime signing key could not be decrypted (corrupted row or master-key mismatch)")]
Crypto,
}
#[async_trait]
pub trait AppSecretsRepo: Send + Sync {
/// Fetch the app's realtime signing key, generating + persisting one
/// (32 random bytes, encrypted) if absent. Idempotent under
/// concurrency: a racing creator's `ON CONFLICT DO NOTHING` insert is
/// a no-op and the existing key is returned.
async fn get_or_create_signing_key(
&self,
app_id: AppId,
) -> Result<Vec<u8>, AppSecretsRepoError>;
/// Fetch the signing key if it exists, WITHOUT creating one. The SSE
/// verify path uses this: a missing key means no token was ever
/// minted for the app, so any presented token must be rejected.
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError>;
}
pub struct PostgresAppSecretsRepo {
pool: PgPool,
master_key: MasterKey,
}
impl PostgresAppSecretsRepo {
#[must_use]
pub fn new(pool: PgPool, master_key: MasterKey) -> Self {
Self { pool, master_key }
}
/// Startup task (v1.1.7): encrypt every row that still has a
/// plaintext key but no encrypted key. Plaintext is left in place
/// (the read path prefers the encrypted columns); the plaintext
/// column is dropped in v1.1.8. Returns the number of rows migrated.
///
/// # Errors
///
/// Propagates database errors.
pub async fn migrate_plaintext_keys(&self) -> Result<usize, AppSecretsRepoError> {
let rows: Vec<(Uuid, Vec<u8>)> = sqlx::query_as(
"SELECT app_id, realtime_signing_key FROM app_secrets \
WHERE realtime_signing_key_encrypted IS NULL \
AND realtime_signing_key IS NOT NULL",
)
.fetch_all(&self.pool)
.await?;
let mut migrated = 0;
for (app_id, plaintext) in rows {
let enc = crypto::encrypt(&plaintext, self.master_key.as_bytes());
sqlx::query(
"UPDATE app_secrets \
SET realtime_signing_key_encrypted = $2, \
realtime_signing_key_nonce = $3, \
updated_at = NOW() \
WHERE app_id = $1 AND realtime_signing_key_encrypted IS NULL",
)
.bind(app_id)
.bind(&enc.ciphertext)
.bind(&enc.nonce[..])
.execute(&self.pool)
.await?;
migrated += 1;
}
Ok(migrated)
}
fn decode(
&self,
encrypted: Option<Vec<u8>>,
nonce: Option<Vec<u8>>,
plaintext: Option<Vec<u8>>,
) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
decode_signing_key(&self.master_key, encrypted, nonce, plaintext)
}
}
/// Resolve the signing key from a row's three columns. **Encrypted wins**
/// when present; otherwise fall back to the plaintext column (compat for
/// un-migrated rows / the post-v1.1.8 dropped-plaintext state).
fn decode_signing_key(
master_key: &MasterKey,
encrypted: Option<Vec<u8>>,
nonce: Option<Vec<u8>>,
plaintext: Option<Vec<u8>>,
) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
match (encrypted, nonce) {
(Some(ct), Some(n)) => {
let key = crypto::decrypt(&ct, &n, master_key.as_bytes())
.map_err(|_| AppSecretsRepoError::Crypto)?;
Ok(Some(key))
}
_ => Ok(plaintext),
}
}
#[async_trait]
impl AppSecretsRepo for PostgresAppSecretsRepo {
async fn get_or_create_signing_key(
&self,
app_id: AppId,
) -> Result<Vec<u8>, AppSecretsRepoError> {
let mut fresh = vec![0u8; SIGNING_KEY_LEN];
rand::thread_rng().fill_bytes(&mut fresh);
let enc = crypto::encrypt(&fresh, self.master_key.as_bytes());
// Insert-if-absent (encrypted-only). The racing-creator's insert
// is a no-op; the SELECT always returns the winning row.
sqlx::query(
"INSERT INTO app_secrets \
(app_id, realtime_signing_key_encrypted, realtime_signing_key_nonce) \
VALUES ($1, $2, $3) ON CONFLICT (app_id) DO NOTHING",
)
.bind(app_id.into_inner())
.bind(&enc.ciphertext)
.bind(&enc.nonce[..])
.execute(&self.pool)
.await?;
let row: (Option<Vec<u8>>, Option<Vec<u8>>, Option<Vec<u8>>) = sqlx::query_as(
"SELECT realtime_signing_key_encrypted, realtime_signing_key_nonce, \
realtime_signing_key \
FROM app_secrets WHERE app_id = $1",
)
.bind(app_id.into_inner())
.fetch_one(&self.pool)
.await?;
// A row exists by construction, so a key must decode.
self.decode(row.0, row.1, row.2)?
.ok_or(AppSecretsRepoError::Crypto)
}
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
let row: Option<(Option<Vec<u8>>, Option<Vec<u8>>, Option<Vec<u8>>)> = sqlx::query_as(
"SELECT realtime_signing_key_encrypted, realtime_signing_key_nonce, \
realtime_signing_key \
FROM app_secrets WHERE app_id = $1",
)
.bind(app_id.into_inner())
.fetch_optional(&self.pool)
.await?;
match row {
Some((e, n, p)) => self.decode(e, n, p),
None => Ok(None),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn key() -> MasterKey {
MasterKey::from_bytes([9u8; 32])
}
#[test]
fn encrypted_wins_over_plaintext() {
let mk = key();
let secret = vec![1u8, 2, 3, 4];
let enc = crypto::encrypt(&secret, mk.as_bytes());
// Both present → the encrypted value is returned (not the bogus
// plaintext).
let got = decode_signing_key(
&mk,
Some(enc.ciphertext),
Some(enc.nonce.to_vec()),
Some(vec![0xff; 32]),
)
.unwrap();
assert_eq!(got, Some(secret));
}
#[test]
fn falls_back_to_plaintext_when_encrypted_absent() {
let mk = key();
let plaintext = vec![7u8; 32];
let got = decode_signing_key(&mk, None, None, Some(plaintext.clone())).unwrap();
assert_eq!(got, Some(plaintext));
}
#[test]
fn encrypted_present_plaintext_null_works() {
// Post-v1.1.8 state: only the encrypted columns are populated.
let mk = key();
let secret = vec![5u8; 32];
let enc = crypto::encrypt(&secret, mk.as_bytes());
let got =
decode_signing_key(&mk, Some(enc.ciphertext), Some(enc.nonce.to_vec()), None).unwrap();
assert_eq!(got, Some(secret));
}
#[test]
fn missing_everything_is_none() {
let got = decode_signing_key(&key(), None, None, None).unwrap();
assert_eq!(got, None);
}
#[test]
fn wrong_master_key_is_crypto_error() {
let secret = vec![3u8; 32];
let enc = crypto::encrypt(&secret, key().as_bytes());
let other = MasterKey::from_bytes([1u8; 32]);
let err = decode_signing_key(&other, Some(enc.ciphertext), Some(enc.nonce.to_vec()), None)
.unwrap_err();
assert!(matches!(err, AppSecretsRepoError::Crypto));
}
}

View File

@@ -78,6 +78,29 @@ pub enum Capability {
/// so the conservative write mapping is correct. Splitting /// so the conservative write mapping is correct. Splitting
/// read/write is a v1.2+ refinement. Granted to `editor`+. /// read/write is a v1.2+ refinement. Granted to `editor`+.
AppHttpRequest(AppId), AppHttpRequest(AppId),
/// Read blobs from this app's files store (v1.1.5). Same trust
/// shape as KV/docs read — granted to `viewer`+, maps to
/// `script:read` on API keys. Honors the seven-scope commitment.
AppFilesRead(AppId),
/// Write blobs to this app's files store (v1.1.5). Granted to
/// `editor`+, maps to `script:write` on API keys.
AppFilesWrite(AppId),
/// Publish a durable pub/sub message from a script in this app
/// (v1.1.5). Maps to `script:write` on API keys (a publish is a
/// write that fans out to subscribers). Granted to `editor`+.
AppPubsubPublish(AppId),
/// Read a decrypted secret from this app's secrets store (v1.1.7).
/// Same trust shape as KV/docs/files read — granted to `viewer`+,
/// maps to `script:read` on API keys. Honors the seven-scope
/// commitment.
AppSecretsRead(AppId),
/// Write (set/delete) a secret in this app's secrets store (v1.1.7).
/// Granted to `editor`+, maps to `script:write` on API keys.
AppSecretsWrite(AppId),
/// Send an outbound email from a script in this app (v1.1.7). Maps
/// to `script:write` on API keys (sending mail is an outbound
/// side-effect like an HTTP request). Granted to `editor`+.
AppEmailSend(AppId),
/// Create / list / delete triggers for this app (v1.1.1). Maps to /// Create / list / delete triggers for this app (v1.1.1). Maps to
/// `app:admin` on API keys — triggers are app-configuration acts /// `app:admin` on API keys — triggers are app-configuration acts
/// rather than data-plane access. Granted to `app_admin`+. /// rather than data-plane access. Granted to `app_admin`+.
@@ -86,6 +109,12 @@ pub enum Capability {
/// to `app:admin` on API keys. Public-HTTP scripts (principal None) /// to `app:admin` on API keys. Public-HTTP scripts (principal None)
/// fail this check — managing dead letters is an admin act. /// fail this check — managing dead letters is an admin act.
AppDeadLetterManage(AppId), AppDeadLetterManage(AppId),
/// Register / list / update / delete externally-subscribable topics
/// for this app (v1.1.6). Maps to `app:admin` on API keys —
/// externalizing a topic is an app-configuration act with security
/// weight (it opens an internal pub/sub topic to outside SSE
/// subscribers). Granted to `app_admin`+.
AppTopicManage(AppId),
} }
impl Capability { impl Capability {
@@ -108,8 +137,15 @@ impl Capability {
| Self::AppDocsRead(id) | Self::AppDocsRead(id)
| Self::AppDocsWrite(id) | Self::AppDocsWrite(id)
| Self::AppHttpRequest(id) | Self::AppHttpRequest(id)
| Self::AppFilesRead(id)
| Self::AppFilesWrite(id)
| Self::AppPubsubPublish(id)
| Self::AppSecretsRead(id)
| Self::AppSecretsWrite(id)
| Self::AppEmailSend(id)
| Self::AppManageTriggers(id) | Self::AppManageTriggers(id)
| Self::AppDeadLetterManage(id) => Some(id), | Self::AppDeadLetterManage(id)
| Self::AppTopicManage(id) => Some(id),
} }
} }
@@ -124,16 +160,25 @@ impl Capability {
Self::InstanceCreateApp | Self::InstanceManageUsers | Self::InstanceManageSettings => { Self::InstanceCreateApp | Self::InstanceManageUsers | Self::InstanceManageSettings => {
Scope::InstanceAdmin Scope::InstanceAdmin
} }
Self::AppRead(_) | Self::AppKvRead(_) | Self::AppDocsRead(_) => Scope::ScriptRead, Self::AppRead(_)
| Self::AppKvRead(_)
| Self::AppDocsRead(_)
| Self::AppFilesRead(_)
| Self::AppSecretsRead(_) => Scope::ScriptRead,
Self::AppWriteScript(_) Self::AppWriteScript(_)
| Self::AppKvWrite(_) | Self::AppKvWrite(_)
| Self::AppDocsWrite(_) | Self::AppDocsWrite(_)
| Self::AppHttpRequest(_) => Scope::ScriptWrite, | Self::AppHttpRequest(_)
| Self::AppFilesWrite(_)
| Self::AppPubsubPublish(_)
| Self::AppSecretsWrite(_)
| Self::AppEmailSend(_) => Scope::ScriptWrite,
Self::AppWriteRoute(_) => Scope::RouteWrite, Self::AppWriteRoute(_) => Scope::RouteWrite,
Self::AppManageDomains(_) => Scope::DomainManage, Self::AppManageDomains(_) => Scope::DomainManage,
Self::AppAdmin(_) | Self::AppManageTriggers(_) | Self::AppDeadLetterManage(_) => { Self::AppAdmin(_)
Scope::AppAdmin | Self::AppManageTriggers(_)
} | Self::AppDeadLetterManage(_)
| Self::AppTopicManage(_) => Scope::AppAdmin,
Self::AppLogRead(_) => Scope::LogRead, Self::AppLogRead(_) => Scope::LogRead,
} }
} }
@@ -277,6 +322,8 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
| Capability::AppLogRead(_) | Capability::AppLogRead(_)
| Capability::AppKvRead(_) | Capability::AppKvRead(_)
| Capability::AppDocsRead(_) | Capability::AppDocsRead(_)
| Capability::AppFilesRead(_)
| Capability::AppSecretsRead(_)
); );
let in_editor = in_viewer let in_editor = in_viewer
|| matches!( || matches!(
@@ -286,6 +333,10 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
| Capability::AppKvWrite(_) | Capability::AppKvWrite(_)
| Capability::AppDocsWrite(_) | Capability::AppDocsWrite(_)
| Capability::AppHttpRequest(_) | Capability::AppHttpRequest(_)
| Capability::AppFilesWrite(_)
| Capability::AppPubsubPublish(_)
| Capability::AppSecretsWrite(_)
| Capability::AppEmailSend(_)
); );
let in_app_admin = in_editor let in_app_admin = in_editor
|| matches!( || matches!(
@@ -294,6 +345,7 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
| Capability::AppAdmin(_) | Capability::AppAdmin(_)
| Capability::AppManageTriggers(_) | Capability::AppManageTriggers(_)
| Capability::AppDeadLetterManage(_) | Capability::AppDeadLetterManage(_)
| Capability::AppTopicManage(_)
); );
match role { match role {
AppRole::Viewer => in_viewer, AppRole::Viewer => in_viewer,
@@ -637,6 +689,35 @@ mod tests {
); );
} }
#[tokio::test]
async fn topic_manage_requires_app_admin() {
let repo = InMemoryAuthzRepo::default();
let app = AppId::new();
// Maps to the app:admin scope, not a new one.
assert_eq!(
Capability::AppTopicManage(app).required_scope(),
Scope::AppAdmin
);
// Member with only Editor role cannot manage topics.
let p = principal(InstanceRole::Member);
repo.grant(p.user_id, app, AppRole::Editor).await;
assert_eq!(
can(&repo, &p, Capability::AppTopicManage(app))
.await
.unwrap(),
Decision::Deny,
);
// App-admin role can.
let admin = principal(InstanceRole::Member);
repo.grant(admin.user_id, app, AppRole::AppAdmin).await;
assert!(can(&repo, &admin, Capability::AppTopicManage(app))
.await
.unwrap()
.is_allow());
}
#[test] #[test]
fn capability_app_id_extraction() { fn capability_app_id_extraction() {
let app = AppId::new(); let app = AppId::new();

View File

@@ -23,19 +23,19 @@
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
use chrono::Utc; use chrono::{DateTime, Utc};
use picloud_executor_core::{ExecError, ExecRequest, ExecResponse, InvocationType}; use picloud_executor_core::{ExecError, ExecRequest, ExecResponse, InvocationType};
use picloud_orchestrator_core::{ExecutionGate, ExecutorClient}; use picloud_orchestrator_core::{ExecutionGate, ExecutorClient};
use picloud_shared::{ use picloud_shared::{
ExecResponseSummary, ExecutionId, HttpDispatchPayload, InboxDeliveryOutcome, InboxFailureKind, DeadLetterId, ExecResponseSummary, ExecutionId, HttpDispatchPayload, InboxDeliveryOutcome,
InboxResolver, InboxResult, RequestId, ScriptId, ScriptSandbox, TriggerEvent, InboxFailureKind, InboxResolver, InboxResult, RequestId, ScriptId, ScriptSandbox, TriggerEvent,
}; };
use rand::Rng; use rand::Rng;
use uuid::Uuid; use uuid::Uuid;
use crate::abandoned_repo::{AbandonedRepo, NewAbandonedExecution}; use crate::abandoned_repo::{AbandonedRepo, NewAbandonedExecution};
use crate::dead_letter_repo::{DeadLetterRepo, NewDeadLetter}; use crate::dead_letter_repo::{DeadLetterRepo, NewDeadLetter};
use crate::outbox_repo::{OutboxRepo, OutboxRow, OutboxSourceKind}; use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxRow, OutboxSourceKind};
use crate::principal_resolver::PrincipalResolver; use crate::principal_resolver::PrincipalResolver;
use crate::repo::ScriptRepository; use crate::repo::ScriptRepository;
use crate::trigger_config::{BackoffShape, TriggerConfig}; use crate::trigger_config::{BackoffShape, TriggerConfig};
@@ -166,7 +166,10 @@ impl Dispatcher {
OutboxSourceKind::Kv OutboxSourceKind::Kv
| OutboxSourceKind::Docs | OutboxSourceKind::Docs
| OutboxSourceKind::DeadLetter | OutboxSourceKind::DeadLetter
| OutboxSourceKind::Cron => { | OutboxSourceKind::Cron
| OutboxSourceKind::Files
| OutboxSourceKind::Pubsub
| OutboxSourceKind::Email => {
let resolved = self.resolve_trigger(&row).await?; let resolved = self.resolve_trigger(&row).await?;
let req = match self.build_exec_request(&row, &resolved).await { let req = match self.build_exec_request(&row, &resolved).await {
Ok(req) => req, Ok(req) => req,
@@ -460,12 +463,12 @@ impl Dispatcher {
// Exhausted retries → dead-letter. // Exhausted retries → dead-letter.
let (op, source) = describe_event(&row.payload); let (op, source) = describe_event(&row.payload);
let now = Utc::now(); let now = Utc::now();
if let Err(e) = self let dl_id = match self
.dead_letters .dead_letters
.insert(NewDeadLetter { .insert(NewDeadLetter {
app_id: row.app_id, app_id: row.app_id,
original_event_id: row.id, original_event_id: row.id,
source, source: source.clone(),
op, op,
trigger_id: row.trigger_id, trigger_id: row.trigger_id,
script_id: Some(resolved.script_id), script_id: Some(resolved.script_id),
@@ -477,8 +480,26 @@ impl Dispatcher {
}) })
.await .await
{ {
Ok(id) => Some(id),
Err(e) => {
tracing::error!(?e, "failed to write dead-letter row"); tracing::error!(?e, "failed to write dead-letter row");
None
} }
};
// v1.1.7 fix: fan the dead-letter out to matching handler triggers.
// This was missing since v1.1.1 — the row was written but
// `list_matching_dead_letter` had no production caller, so
// registered dead_letter handlers never fired. The recursion-stop
// (a dead-letter handler's own failure is not re-dead-lettered)
// is upheld by the `is_dead_letter_handler` short-circuit at the
// top of this function, so this fan-out is only reached for
// non-handler executions.
if let Some(dl_id) = dl_id {
self.fan_out_dead_letter(row, resolved, dl_id, &source, attempt, &err, now)
.await;
}
self.outbox self.outbox
.delete(row.id) .delete(row.id)
.await .await
@@ -486,6 +507,82 @@ impl Dispatcher {
Ok(()) Ok(())
} }
/// Enqueue one outbox row per matching `dead_letter` trigger so its
/// handler script runs with the dead-letter event as `ctx.event`.
/// Best-effort: a lookup/insert failure is logged, not propagated
/// (the dead-letter row itself is already durably written).
#[allow(clippy::too_many_arguments)]
async fn fan_out_dead_letter(
&self,
row: &OutboxRow,
resolved: &ResolvedTrigger,
dead_letter_id: DeadLetterId,
source: &str,
attempt: u32,
err: &ExecError,
now: DateTime<Utc>,
) {
// The DL event nests the original verbatim; if the payload can't
// be decoded back into a TriggerEvent we can't build the nested
// `original`, so skip the fan-out (the DL row is still written).
let Ok(original) = serde_json::from_value::<TriggerEvent>(row.payload.clone()) else {
tracing::warn!(
outbox_id = %row.id,
"dead-letter payload is not a TriggerEvent; skipping handler fan-out"
);
return;
};
let matches = match self
.triggers
.list_matching_dead_letter(row.app_id, source, row.trigger_id, Some(resolved.script_id))
.await
{
Ok(m) => m,
Err(e) => {
tracing::error!(?e, "dead-letter trigger lookup failed");
return;
}
};
for m in matches {
let event = TriggerEvent::DeadLetter {
dead_letter_id,
original: Box::new(original.clone()),
attempts: attempt,
last_error: err.to_string(),
trigger_id: row.trigger_id,
script_id: Some(resolved.script_id),
first_attempt_at: row.created_at,
last_attempt_at: now,
};
let payload = match serde_json::to_value(&event) {
Ok(p) => p,
Err(e) => {
tracing::error!(?e, "failed to serialize dead-letter event");
continue;
}
};
if let Err(e) = self
.outbox
.insert(NewOutboxRow {
app_id: row.app_id,
source_kind: OutboxSourceKind::DeadLetter,
trigger_id: Some(m.trigger_id),
script_id: Some(m.script_id),
reply_to: None,
payload,
origin_principal: Some(m.registered_by_principal),
trigger_depth: row.trigger_depth.saturating_add(1),
root_execution_id: row.root_execution_id,
})
.await
{
tracing::error!(?e, "failed to enqueue dead-letter handler delivery");
}
}
}
async fn deliver_inbox(&self, row: &OutboxRow, inbox_id: Uuid, result: InboxResult) { async fn deliver_inbox(&self, row: &OutboxRow, inbox_id: Uuid, result: InboxResult) {
match self.inbox.deliver(inbox_id, result.clone()).await { match self.inbox.deliver(inbox_id, result.clone()).await {
InboxDeliveryOutcome::Delivered => {} InboxDeliveryOutcome::Delivered => {}

View File

@@ -0,0 +1,307 @@
//! `POST /api/v1/email-inbound/{app_id}/{trigger_id}` — the inbound-email
//! webhook receiver (v1.1.7).
//!
//! A configured provider (Mailgun / Postmark / SendGrid / SES) POSTs a
//! normalized JSON message here; the receiver verifies the optional HMAC
//! signature, builds a `TriggerEvent::Email`, and enqueues an outbox row
//! the dispatcher picks up like any other async trigger.
//!
//! This is a PUBLIC endpoint (no admin auth) — the trigger URL itself,
//! plus the per-trigger HMAC secret, are the security boundary. It is
//! mounted OUTSIDE the `require_authenticated` layer.
//!
//! Status codes:
//! * 202 — accepted + enqueued
//! * 401 — HMAC required but missing/invalid
//! * 404 — trigger missing, disabled, not `kind=email`, or app mismatch
//! * 422 — body is not the expected JSON shape
//!
//! Only the generic provider-agnostic JSON shape is accepted in v1.1.7
//! (see [`InboundPayload`]); provider-specific unmarshallers are v1.2.
use std::sync::Arc;
use axum::body::Bytes;
use axum::extract::{Path, State};
use axum::http::{HeaderMap, StatusCode};
use axum::response::{IntoResponse, Json, Response};
use axum::routing::post;
use axum::Router;
use hmac::{Hmac, Mac};
use picloud_shared::{AppId, MasterKey, TriggerEvent, TriggerId};
use serde::Deserialize;
use serde_json::json;
use sha2::Sha256;
use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxSourceKind};
use crate::secrets_repo::StoredSecret;
use crate::secrets_service::open;
use crate::trigger_repo::TriggerRepo;
type HmacSha256 = Hmac<Sha256>;
/// Header the provider's HMAC signature is read from. The signature is
/// the lowercase hex of `HMAC-SHA256(inbound_secret, raw_body)`.
const SIGNATURE_HEADER: &str = "x-picloud-signature";
#[derive(Clone)]
pub struct EmailInboundState {
pub triggers: Arc<dyn TriggerRepo>,
pub outbox: Arc<dyn OutboxRepo>,
pub master_key: MasterKey,
}
pub fn email_inbound_router(state: EmailInboundState) -> Router {
Router::new()
.route(
"/email-inbound/{app_id}/{trigger_id}",
post(receive_inbound_email),
)
.with_state(state)
}
/// The generic provider-agnostic inbound shape. Users configure their
/// provider's webhook templating to POST this. `from` is required;
/// everything else defaults.
#[derive(Debug, Deserialize)]
struct InboundPayload {
from: String,
#[serde(default)]
to: Vec<String>,
#[serde(default)]
cc: Vec<String>,
#[serde(default)]
subject: String,
#[serde(default)]
text: Option<String>,
#[serde(default)]
html: Option<String>,
#[serde(default)]
message_id: Option<String>,
}
async fn receive_inbound_email(
State(s): State<EmailInboundState>,
Path((app_id, trigger_id)): Path<(AppId, TriggerId)>,
headers: HeaderMap,
body: Bytes,
) -> Result<StatusCode, EmailInboundError> {
// Resolve the trigger. 404 covers missing / wrong-kind / cross-app /
// disabled — all "this URL doesn't address a live email trigger".
let target = s
.triggers
.email_inbound_target(trigger_id)
.await
.map_err(|e| EmailInboundError::Backend(e.to_string()))?
.ok_or(EmailInboundError::NotFound)?;
if target.app_id != app_id || !target.enabled {
return Err(EmailInboundError::NotFound);
}
// HMAC verification (only when the trigger has a secret configured).
if let (Some(ct), Some(nonce)) = (
target.inbound_secret_encrypted.as_ref(),
target.inbound_secret_nonce.as_ref(),
) {
let secret = decrypt_secret(&s.master_key, ct, nonce)?;
verify_signature(&headers, &body, secret.as_bytes())?;
}
// Parse the generic JSON shape. Malformed → 422.
let payload: InboundPayload =
serde_json::from_slice(&body).map_err(|e| EmailInboundError::Malformed(e.to_string()))?;
let event = TriggerEvent::Email {
from: payload.from,
to: payload.to,
cc: payload.cc,
subject: payload.subject,
text: payload.text,
html: payload.html,
received_at: chrono::Utc::now(),
message_id: payload.message_id,
};
let payload_json = serde_json::to_value(&event)
.map_err(|e| EmailInboundError::Backend(format!("serialize event: {e}")))?;
s.outbox
.insert(NewOutboxRow {
app_id,
source_kind: OutboxSourceKind::Email,
trigger_id: Some(trigger_id),
script_id: Some(target.script_id),
reply_to: None,
payload: payload_json,
origin_principal: Some(target.registered_by_principal),
// Inbound email is the root of a trigger chain (depth 1).
trigger_depth: 1,
root_execution_id: None,
})
.await
.map_err(|e| EmailInboundError::Backend(e.to_string()))?;
Ok(StatusCode::ACCEPTED)
}
/// Decrypt the stored inbound secret back to its raw string. It was
/// sealed as a JSON string by the admin layer, so `open` yields a
/// `Value::String`.
fn decrypt_secret(
master_key: &MasterKey,
ciphertext: &[u8],
nonce: &[u8],
) -> Result<String, EmailInboundError> {
let stored = StoredSecret {
encrypted_value: ciphertext.to_vec(),
nonce: nonce.to_vec(),
};
let value = open(master_key, &stored).map_err(|_| {
// Corrupted secret means we can't verify — fail closed (401).
EmailInboundError::Unauthorized
})?;
value
.as_str()
.map(str::to_string)
.ok_or(EmailInboundError::Unauthorized)
}
/// Constant-time HMAC-SHA256 verification of the body against the
/// `X-Picloud-Signature` header (lowercase hex).
fn verify_signature(
headers: &HeaderMap,
body: &[u8],
secret: &[u8],
) -> Result<(), EmailInboundError> {
let provided_hex = headers
.get(SIGNATURE_HEADER)
.and_then(|h| h.to_str().ok())
.ok_or(EmailInboundError::Unauthorized)?;
let provided = hex::decode(provided_hex.trim()).map_err(|_| EmailInboundError::Unauthorized)?;
let mut mac =
HmacSha256::new_from_slice(secret).map_err(|_| EmailInboundError::Unauthorized)?;
mac.update(body);
mac.verify_slice(&provided)
.map_err(|_| EmailInboundError::Unauthorized)
}
#[derive(Debug, thiserror::Error)]
pub enum EmailInboundError {
#[error("trigger not found")]
NotFound,
#[error("invalid signature")]
Unauthorized,
#[error("malformed body: {0}")]
Malformed(String),
#[error("backend: {0}")]
Backend(String),
}
impl IntoResponse for EmailInboundError {
fn into_response(self) -> Response {
let (status, body) = match &self {
Self::NotFound => (
StatusCode::NOT_FOUND,
json!({ "error": "trigger not found" }),
),
Self::Unauthorized => (
StatusCode::UNAUTHORIZED,
json!({ "error": "invalid or missing signature" }),
),
Self::Malformed(m) => (
StatusCode::UNPROCESSABLE_ENTITY,
json!({ "error": format!("malformed inbound email body: {m}") }),
),
Self::Backend(e) => {
tracing::error!(error = %e, "inbound email receiver backend error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
};
(status, Json(body)).into_response()
}
}
#[cfg(test)]
mod tests {
//! Unit tests for the security-critical helpers (HMAC verify, secret
//! round-trip, payload parsing). The full request flow — 202 / 401 /
//! 404 / 422 / cross-app — is exercised end-to-end against a real
//! Postgres in `crates/picloud/tests/email_inbound.rs`.
use super::*;
use crate::secrets_service::seal;
use crate::secrets_service::DEFAULT_SECRET_MAX_VALUE_BYTES;
fn sign(secret: &[u8], body: &[u8]) -> String {
let mut mac = HmacSha256::new_from_slice(secret).unwrap();
mac.update(body);
hex::encode(mac.finalize().into_bytes())
}
fn headers_with_sig(sig: &str) -> HeaderMap {
let mut h = HeaderMap::new();
h.insert(SIGNATURE_HEADER, sig.parse().unwrap());
h
}
#[test]
fn valid_signature_verifies() {
let secret = b"shhh";
let body = br#"{"from":"a@b.com"}"#;
let sig = sign(secret, body);
assert!(verify_signature(&headers_with_sig(&sig), body, secret).is_ok());
}
#[test]
fn wrong_signature_rejected() {
let body = br#"{"from":"a@b.com"}"#;
let sig = sign(b"shhh", body);
let err = verify_signature(&headers_with_sig(&sig), body, b"different").unwrap_err();
assert!(matches!(err, EmailInboundError::Unauthorized));
}
#[test]
fn missing_signature_header_rejected() {
let err = verify_signature(&HeaderMap::new(), b"body", b"secret").unwrap_err();
assert!(matches!(err, EmailInboundError::Unauthorized));
}
#[test]
fn tampered_body_fails_verification() {
let secret = b"shhh";
let sig = sign(secret, b"original");
let err = verify_signature(&headers_with_sig(&sig), b"tampered", secret).unwrap_err();
assert!(matches!(err, EmailInboundError::Unauthorized));
}
#[test]
fn secret_round_trips_through_seal_open() {
let key = MasterKey::from_bytes([3u8; 32]);
let (ct, nonce) = seal(
&key,
&serde_json::Value::String("provider-secret".into()),
DEFAULT_SECRET_MAX_VALUE_BYTES,
)
.unwrap();
let recovered = decrypt_secret(&key, &ct, &nonce).unwrap();
assert_eq!(recovered, "provider-secret");
// And a signature made with the recovered secret verifies.
let body = br#"{"from":"x@y.com"}"#;
let sig = sign(recovered.as_bytes(), body);
assert!(verify_signature(&headers_with_sig(&sig), body, recovered.as_bytes()).is_ok());
}
#[test]
fn payload_requires_from_but_defaults_rest() {
let ok: Result<InboundPayload, _> = serde_json::from_slice(br#"{"from":"a@b.com"}"#);
let p = ok.expect("from-only payload parses");
assert_eq!(p.from, "a@b.com");
assert!(p.to.is_empty() && p.cc.is_empty() && p.text.is_none());
// Missing `from` → malformed.
let bad: Result<InboundPayload, _> = serde_json::from_slice(br#"{"subject":"hi"}"#);
assert!(bad.is_err());
}
}

View File

@@ -0,0 +1,597 @@
//! `EmailServiceImpl` — outbound email over an SMTP relay (`lettre`),
//! behind the `picloud_shared::EmailService` trait scripts reach via the
//! Rhai `email::{send,send_html}` bridge.
//!
//! Layers added here:
//!
//! 1. **Script-as-gate authz**: `AppEmailSend` checked when
//! `cx.principal.is_some()`; skipped for public-HTTP (`None`).
//! 2. Required-field + RFC 5322-ish address validation at the boundary.
//! 3. Per-message size cap (default 25 MB).
//! 4. **Disabled mode**: if no SMTP relay is configured (HOST/USER/
//! PASSWORD not all set) every `send` returns `NotConfigured` and
//! startup logs a warning — there is no silent drop.
//!
//! Connection model: one connection per call (lettre's default). A
//! pooled transport is a v1.2+ optimization. Per-app `from` validation /
//! SPF / DKIM are the operator's responsibility at the relay (v1.1.7
//! does not restrict the `from` address).
use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
use lettre::message::{Mailbox, Message, MultiPart, SinglePart};
use lettre::transport::smtp::authentication::Credentials;
use lettre::{AsyncSmtpTransport, AsyncTransport, Tokio1Executor};
use picloud_shared::{EmailError, EmailService, OutboundEmail, SdkCallCx};
use crate::authz::{self, AuthzRepo, Capability};
/// Default per-message size cap (25 MB) — matches most providers.
/// Override with `PICLOUD_EMAIL_MAX_MESSAGE_BYTES`.
pub const DEFAULT_EMAIL_MAX_MESSAGE_BYTES: usize = 25 * 1024 * 1024;
/// Generous upper bound on a single address string (RFC 5321 caps the
/// path at 256; 320 covers local@domain comfortably).
const ADDRESS_MAX_LEN: usize = 320;
/// Process config for the email service.
#[derive(Debug, Clone, Copy)]
pub struct EmailConfig {
pub max_message_bytes: usize,
}
impl EmailConfig {
#[must_use]
pub const fn conservative() -> Self {
Self {
max_message_bytes: DEFAULT_EMAIL_MAX_MESSAGE_BYTES,
}
}
#[must_use]
pub fn from_env() -> Self {
let mut c = Self::conservative();
if let Ok(v) = std::env::var("PICLOUD_EMAIL_MAX_MESSAGE_BYTES") {
match v.trim().parse::<usize>() {
Ok(n) if n > 0 => c.max_message_bytes = n,
_ => tracing::warn!(
value = %v,
"ignoring invalid PICLOUD_EMAIL_MAX_MESSAGE_BYTES (want a positive integer)"
),
}
}
c
}
}
impl Default for EmailConfig {
fn default() -> Self {
Self::conservative()
}
}
/// TLS mode for the SMTP relay connection.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SmtpTls {
/// STARTTLS upgrade on a plaintext port (typically 587). Default.
Starttls,
/// Implicit TLS from connect (typically 465).
Implicit,
/// No TLS — plaintext. Dev/test only.
None,
}
/// SMTP relay connection settings, sourced from env.
#[derive(Debug, Clone)]
pub struct SmtpConfig {
pub host: String,
pub port: u16,
pub user: String,
pub password: String,
pub tls: SmtpTls,
pub timeout_secs: u64,
}
impl SmtpConfig {
/// Read SMTP settings from env. Returns `None` (→ disabled mode) when
/// any of HOST / USER / PASSWORD is missing or empty.
#[must_use]
pub fn from_env() -> Option<Self> {
let host = non_empty_env("PICLOUD_SMTP_HOST")?;
let user = non_empty_env("PICLOUD_SMTP_USER")?;
let password = non_empty_env("PICLOUD_SMTP_PASSWORD")?;
let tls = match std::env::var("PICLOUD_SMTP_TLS")
.unwrap_or_default()
.trim()
.to_ascii_lowercase()
.as_str()
{
"implicit" => SmtpTls::Implicit,
"none" => SmtpTls::None,
// Default + explicit "starttls" + anything unrecognized.
_ => SmtpTls::Starttls,
};
let default_port = match tls {
SmtpTls::Implicit => 465,
SmtpTls::Starttls | SmtpTls::None => 587,
};
let port = std::env::var("PICLOUD_SMTP_PORT")
.ok()
.and_then(|v| v.trim().parse::<u16>().ok())
.unwrap_or(default_port);
let timeout_secs = std::env::var("PICLOUD_SMTP_TIMEOUT_SECS")
.ok()
.and_then(|v| v.trim().parse::<u64>().ok())
.filter(|n| *n > 0)
.unwrap_or(30);
Some(Self {
host,
port,
user,
password,
tls,
timeout_secs,
})
}
}
fn non_empty_env(key: &str) -> Option<String> {
std::env::var(key).ok().filter(|v| !v.trim().is_empty())
}
/// Internal transport seam so the service can be tested without a live
/// SMTP server. The production impl is [`LettreEmailTransport`]; tests
/// use a recording fake.
#[async_trait]
pub trait EmailTransport: Send + Sync {
async fn send(&self, message: &Message) -> Result<(), EmailError>;
}
/// Production transport: a per-call lettre SMTP connection.
pub struct LettreEmailTransport {
inner: AsyncSmtpTransport<Tokio1Executor>,
}
impl LettreEmailTransport {
/// Build the transport from settings.
///
/// # Errors
///
/// Returns the lettre SMTP error string if the relay descriptor is
/// invalid (e.g. TLS setup fails).
pub fn build(cfg: &SmtpConfig) -> Result<Self, String> {
let builder = match cfg.tls {
SmtpTls::Implicit => {
AsyncSmtpTransport::<Tokio1Executor>::relay(&cfg.host).map_err(|e| e.to_string())?
}
SmtpTls::Starttls => AsyncSmtpTransport::<Tokio1Executor>::starttls_relay(&cfg.host)
.map_err(|e| e.to_string())?,
SmtpTls::None => AsyncSmtpTransport::<Tokio1Executor>::builder_dangerous(&cfg.host),
};
let inner = builder
.port(cfg.port)
.credentials(Credentials::new(cfg.user.clone(), cfg.password.clone()))
.timeout(Some(Duration::from_secs(cfg.timeout_secs)))
.build();
Ok(Self { inner })
}
}
#[async_trait]
impl EmailTransport for LettreEmailTransport {
async fn send(&self, message: &Message) -> Result<(), EmailError> {
// lettre's `AsyncTransport::send` consumes the `Message`; clone so
// the caller keeps ownership (it needs it for the size check).
self.inner
.send(message.clone())
.await
.map(|_| ())
.map_err(|e| EmailError::Transport(e.to_string()))
}
}
pub struct EmailServiceImpl {
/// `None` → disabled mode (every send returns `NotConfigured`).
transport: Option<Arc<dyn EmailTransport>>,
authz: Arc<dyn AuthzRepo>,
config: EmailConfig,
}
impl EmailServiceImpl {
#[must_use]
pub fn new(
transport: Option<Arc<dyn EmailTransport>>,
authz: Arc<dyn AuthzRepo>,
config: EmailConfig,
) -> Self {
Self {
transport,
authz,
config,
}
}
/// Construct from env: builds a lettre SMTP transport if the relay is
/// configured, otherwise runs in disabled mode (with a warning). A
/// malformed relay descriptor is logged and also yields disabled mode
/// — email is non-critical and must not block startup.
#[must_use]
pub fn from_env(authz: Arc<dyn AuthzRepo>) -> Self {
let config = EmailConfig::from_env();
let transport: Option<Arc<dyn EmailTransport>> = match SmtpConfig::from_env() {
None => {
tracing::warn!(
"email is DISABLED: set PICLOUD_SMTP_HOST/USER/PASSWORD to enable \
email::send. Scripts calling email::send will get an error."
);
None
}
Some(cfg) => match LettreEmailTransport::build(&cfg) {
Ok(t) => {
tracing::info!(host = %cfg.host, port = cfg.port, "outbound email enabled");
Some(Arc::new(t))
}
Err(e) => {
tracing::error!(error = %e, "failed to build SMTP transport; email DISABLED");
None
}
},
};
Self::new(transport, authz, config)
}
async fn check_send(&self, cx: &SdkCallCx) -> Result<(), EmailError> {
if let Some(ref principal) = cx.principal {
authz::require(&*self.authz, principal, Capability::AppEmailSend(cx.app_id))
.await
.map_err(|_| EmailError::Forbidden)?;
}
Ok(())
}
}
#[async_trait]
impl EmailService for EmailServiceImpl {
async fn send(&self, cx: &SdkCallCx, email: OutboundEmail) -> Result<(), EmailError> {
self.check_send(cx).await?;
let Some(transport) = self.transport.as_ref() else {
return Err(EmailError::NotConfigured);
};
let message = build_message(&email)?;
let formatted = message.formatted();
if formatted.len() > self.config.max_message_bytes {
return Err(EmailError::TooLarge {
limit: self.config.max_message_bytes,
actual: formatted.len(),
});
}
transport.send(&message).await
}
}
/// Validate the required fields + addresses and assemble a lettre
/// `Message`. Pure (no I/O) so it's unit-testable on its own.
fn build_message(email: &OutboundEmail) -> Result<Message, EmailError> {
if email.from.trim().is_empty() {
return Err(EmailError::MissingField("from".into()));
}
if email.to.iter().all(|a| a.trim().is_empty()) {
return Err(EmailError::MissingField("to".into()));
}
if email.subject.trim().is_empty() {
return Err(EmailError::MissingField("subject".into()));
}
let has_text = email.text.as_ref().is_some_and(|t| !t.is_empty());
let has_html = email.html.as_ref().is_some_and(|h| !h.is_empty());
if !has_text && !has_html {
return Err(EmailError::MissingField("text or html".into()));
}
let mut builder = Message::builder()
.from(parse_address(&email.from)?)
.subject(email.subject.clone());
for addr in non_empty(&email.to) {
builder = builder.to(parse_address(addr)?);
}
for addr in non_empty(&email.cc) {
builder = builder.cc(parse_address(addr)?);
}
for addr in non_empty(&email.bcc) {
builder = builder.bcc(parse_address(addr)?);
}
// reply_to defaults to `from` when not supplied.
let reply_to = email.reply_to.as_deref().unwrap_or(&email.from);
builder = builder.reply_to(parse_address(reply_to)?);
// `has_text` / `has_html` were validated above (at least one is set).
let text = email.text.clone().unwrap_or_default();
let html = email.html.clone().unwrap_or_default();
let message = if has_text && has_html {
builder.multipart(MultiPart::alternative_plain_html(text, html))
} else if has_html {
builder.singlepart(SinglePart::html(html))
} else {
builder.singlepart(SinglePart::plain(text))
}
.map_err(|e| EmailError::Transport(e.to_string()))?;
Ok(message)
}
fn non_empty(addrs: &[String]) -> impl Iterator<Item = &String> {
addrs.iter().filter(|a| !a.trim().is_empty())
}
/// Hand-rolled RFC 5322-ish address check, then a `lettre::Mailbox`
/// parse (the authoritative validator). We do NOT check deliverability —
/// that's the SMTP layer's job.
fn parse_address(addr: &str) -> Result<Mailbox, EmailError> {
let trimmed = addr.trim();
if trimmed.is_empty() {
return Err(EmailError::InvalidAddress("empty address".into()));
}
if trimmed.len() > ADDRESS_MAX_LEN {
return Err(EmailError::InvalidAddress(format!(
"address exceeds {ADDRESS_MAX_LEN} bytes"
)));
}
// Must have a single-ish @ with a non-empty local part and a domain
// that contains a dot (rejects "a@b" and bare tokens).
match trimmed.rsplit_once('@') {
Some((local, domain)) if !local.is_empty() && domain.contains('.') => {}
_ => {
return Err(EmailError::InvalidAddress(format!(
"{trimmed:?} is not a valid email address"
)))
}
}
trimmed.parse::<Mailbox>().map_err(|_| {
EmailError::InvalidAddress(format!("{trimmed:?} is not a valid email address"))
})
}
// ----------------------------------------------------------------------------
// Tests — recording transport so unit tests need no live SMTP server.
// ----------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::authz::{AuthzError, AuthzRepo};
use async_trait::async_trait;
use picloud_shared::{
AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, Principal, RequestId, ScriptId,
UserId,
};
use std::sync::Mutex as StdMutex;
#[derive(Default)]
struct RecordingTransport {
sent: StdMutex<Vec<Vec<u8>>>,
}
#[async_trait]
impl EmailTransport for RecordingTransport {
async fn send(&self, message: &Message) -> Result<(), EmailError> {
self.sent.lock().unwrap().push(message.formatted());
Ok(())
}
}
#[derive(Default)]
struct DenyAuthz;
#[async_trait]
impl AuthzRepo for DenyAuthz {
async fn membership(&self, _: UserId, _: AppId) -> Result<Option<AppRole>, AuthzError> {
Ok(None)
}
}
struct GrantAuthz {
app: AppId,
role: AppRole,
}
#[async_trait]
impl AuthzRepo for GrantAuthz {
async fn membership(
&self,
_: UserId,
app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok((app_id == self.app).then_some(self.role))
}
}
fn svc_with(
transport: Option<Arc<dyn EmailTransport>>,
authz: Arc<dyn AuthzRepo>,
) -> EmailServiceImpl {
EmailServiceImpl::new(transport, authz, EmailConfig::conservative())
}
fn recording() -> (EmailServiceImpl, Arc<RecordingTransport>) {
let rec = Arc::new(RecordingTransport::default());
let svc = svc_with(Some(rec.clone()), Arc::new(DenyAuthz));
(svc, rec)
}
fn cx_with(app_id: AppId, principal: Option<Principal>) -> SdkCallCx {
SdkCallCx {
app_id,
script_id: ScriptId::new(),
principal,
execution_id: ExecutionId::new(),
request_id: RequestId::new(),
trigger_depth: 0,
root_execution_id: ExecutionId::new(),
is_dead_letter_handler: false,
event: None,
}
}
fn anon(app: AppId) -> SdkCallCx {
cx_with(app, None)
}
fn principal(role: InstanceRole) -> Principal {
Principal {
user_id: AdminUserId::new(),
instance_role: role,
scopes: None,
app_binding: None,
}
}
fn base_email() -> OutboundEmail {
OutboundEmail {
to: vec!["alice@example.com".into()],
from: "alerts@myapp.com".into(),
subject: "Build complete".into(),
text: Some("Your deploy finished.".into()),
..Default::default()
}
}
fn last_message(rec: &RecordingTransport) -> String {
let g = rec.sent.lock().unwrap();
String::from_utf8_lossy(g.last().expect("a message was sent")).into_owned()
}
#[tokio::test]
async fn send_text_includes_headers_and_body() {
let (svc, rec) = recording();
svc.send(&anon(AppId::new()), base_email()).await.unwrap();
let msg = last_message(&rec);
assert!(msg.contains("To: alice@example.com"), "{msg}");
assert!(msg.contains("From: alerts@myapp.com"), "{msg}");
assert!(msg.contains("Subject: Build complete"), "{msg}");
assert!(msg.contains("Your deploy finished."), "{msg}");
}
#[tokio::test]
async fn send_html_is_multipart_with_both_parts() {
let (svc, rec) = recording();
let mut e = base_email();
e.text = Some("plain fallback".into());
e.html = Some("<p>rich <b>body</b></p>".into());
svc.send(&anon(AppId::new()), e).await.unwrap();
let msg = last_message(&rec);
assert!(msg.contains("multipart/alternative"), "{msg}");
assert!(msg.contains("plain fallback"), "{msg}");
// HTML part is quoted-printable encoded, but the tag survives.
assert!(msg.contains("text/html"), "{msg}");
}
#[tokio::test]
async fn multiple_recipients_and_cc_bcc() {
let (svc, rec) = recording();
let mut e = base_email();
e.to = vec!["alice@x.com".into(), "bob@y.com".into()];
e.cc = vec!["dave@z.com".into()];
e.bcc = vec!["audit@myapp.com".into()];
svc.send(&anon(AppId::new()), e).await.unwrap();
let msg = last_message(&rec);
assert!(
msg.contains("alice@x.com") && msg.contains("bob@y.com"),
"{msg}"
);
assert!(msg.contains("Cc: dave@z.com"), "{msg}");
// Bcc is intentionally NOT serialized into the visible headers.
assert!(
!msg.contains("Bcc:"),
"bcc must not appear in headers: {msg}"
);
}
#[tokio::test]
async fn reply_to_populated() {
let (svc, rec) = recording();
let mut e = base_email();
e.reply_to = Some("support@myapp.com".into());
svc.send(&anon(AppId::new()), e).await.unwrap();
assert!(last_message(&rec).contains("Reply-To: support@myapp.com"));
}
#[tokio::test]
async fn missing_required_field_throws() {
let (svc, _) = recording();
let mut e = base_email();
e.subject = String::new();
let err = svc.send(&anon(AppId::new()), e).await.unwrap_err();
assert!(matches!(err, EmailError::MissingField(f) if f == "subject"));
let (svc, _) = recording();
let mut e = base_email();
e.text = None;
e.html = None;
let err = svc.send(&anon(AppId::new()), e).await.unwrap_err();
assert!(matches!(err, EmailError::MissingField(_)));
}
#[tokio::test]
async fn invalid_address_throws() {
let (svc, _) = recording();
let mut e = base_email();
e.to = vec!["not-an-email".into()];
let err = svc.send(&anon(AppId::new()), e).await.unwrap_err();
assert!(matches!(err, EmailError::InvalidAddress(_)));
}
#[tokio::test]
async fn message_size_cap_enforced() {
let rec = Arc::new(RecordingTransport::default());
let svc = EmailServiceImpl::new(
Some(rec),
Arc::new(DenyAuthz),
EmailConfig {
max_message_bytes: 64,
},
);
let err = svc
.send(&anon(AppId::new()), base_email())
.await
.unwrap_err();
assert!(matches!(err, EmailError::TooLarge { limit: 64, .. }));
}
#[tokio::test]
async fn not_configured_throws() {
let svc = svc_with(None, Arc::new(DenyAuthz));
let err = svc
.send(&anon(AppId::new()), base_email())
.await
.unwrap_err();
assert!(matches!(err, EmailError::NotConfigured));
}
#[tokio::test]
async fn anonymous_skips_authz() {
// DenyAuthz would deny an authed principal; anon skips the check.
let (svc, _) = recording();
svc.send(&anon(AppId::new()), base_email()).await.unwrap();
}
#[tokio::test]
async fn member_with_editor_role_allowed() {
let app = AppId::new();
let rec = Arc::new(RecordingTransport::default());
let svc = svc_with(
Some(rec),
Arc::new(GrantAuthz {
app,
role: AppRole::Editor,
}),
);
let cx = cx_with(app, Some(principal(InstanceRole::Member)));
svc.send(&cx, base_email()).await.unwrap();
}
#[tokio::test]
async fn member_without_role_forbidden() {
let (svc, _) = recording();
let cx = cx_with(AppId::new(), Some(principal(InstanceRole::Member)));
let err = svc.send(&cx, base_email()).await.unwrap_err();
assert!(matches!(err, EmailError::Forbidden));
}
}

View File

@@ -0,0 +1,215 @@
//! `/api/v1/admin/apps/{id}/files*` — minimal files admin endpoints
//! backing the dashboard's files view (v1.1.5).
//!
//! Two operations only, both operator-facing:
//! * `GET /apps/{id}/files?collection=<c>&cursor=&limit=` — list file
//! metadata for a collection (cursor-paginated).
//! * `DELETE /apps/{id}/files/{collection}/{file_id}` — remove a file.
//!
//! These talk to the `FilesRepo` directly (like `triggers_api` talks to
//! `TriggerRepo`), guarded by the same capability model as the SDK
//! (`AppFilesRead` / `AppFilesWrite`). **Admin deletes do NOT emit a
//! `files:delete` trigger event** — they're operator cleanup actions,
//! not script mutations (see HANDBACK §7). The capability binds to the
//! resource's `app_id` after the app is loaded.
use std::sync::Arc;
use axum::extract::{Path, Query, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Json, Response};
use axum::routing::{delete, get};
use axum::{Extension, Router};
use picloud_shared::{AppId, Principal};
use serde::{Deserialize, Serialize};
use serde_json::json;
use uuid::Uuid;
use crate::app_repo::AppRepository;
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
use crate::files_repo::{FilesRepo, FilesRepoError};
#[derive(Clone)]
pub struct FilesAdminState {
pub files: Arc<dyn FilesRepo>,
pub apps: Arc<dyn AppRepository>,
pub authz: Arc<dyn AuthzRepo>,
}
pub fn files_admin_router(state: FilesAdminState) -> Router {
Router::new()
.route("/apps/{app_id}/files", get(list_files))
.route(
"/apps/{app_id}/files/{collection}/{file_id}",
delete(delete_file),
)
.with_state(state)
}
#[derive(Debug, Deserialize)]
pub struct ListFilesQuery {
pub collection: String,
#[serde(default)]
pub cursor: Option<String>,
#[serde(default)]
pub limit: Option<u32>,
}
#[derive(Debug, Serialize)]
struct FileMetaDto {
id: String,
collection: String,
name: String,
content_type: String,
size: u64,
checksum: String,
created_at: String,
updated_at: String,
}
#[derive(Debug, Serialize)]
struct ListFilesResponse {
files: Vec<FileMetaDto>,
next_cursor: Option<String>,
}
async fn list_files(
State(s): State<FilesAdminState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Query(q): Query<ListFilesQuery>,
) -> Result<Json<ListFilesResponse>, FilesApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppFilesRead(app_id),
)
.await?;
if q.collection.trim().is_empty() {
return Err(FilesApiError::Invalid(
"collection must not be empty".into(),
));
}
let page = s
.files
.list(
app_id,
&q.collection,
q.cursor.as_deref(),
q.limit.unwrap_or(0),
)
.await?;
let files = page
.files
.into_iter()
.map(|m| FileMetaDto {
id: m.id.to_string(),
collection: m.collection,
name: m.name,
content_type: m.content_type,
size: m.size,
checksum: m.checksum,
created_at: m.created_at.to_rfc3339(),
updated_at: m.updated_at.to_rfc3339(),
})
.collect();
Ok(Json(ListFilesResponse {
files,
next_cursor: page.next_cursor,
}))
}
async fn delete_file(
State(s): State<FilesAdminState>,
Extension(principal): Extension<Principal>,
Path((app_id, collection, file_id)): Path<(AppId, String, String)>,
) -> Result<StatusCode, FilesApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppFilesWrite(app_id),
)
.await?;
let id = Uuid::parse_str(&file_id).map_err(|_| FilesApiError::NotFound)?;
if s.files.delete(app_id, &collection, id).await?.is_none() {
return Err(FilesApiError::NotFound);
}
Ok(StatusCode::NO_CONTENT)
}
async fn ensure_app_exists(apps: &dyn AppRepository, app_id: AppId) -> Result<(), FilesApiError> {
apps.get_by_id(app_id)
.await
.map_err(|e| FilesApiError::Backend(e.to_string()))?
.ok_or(FilesApiError::AppNotFound)?;
Ok(())
}
#[derive(Debug, thiserror::Error)]
pub enum FilesApiError {
#[error("app not found")]
AppNotFound,
#[error("file not found")]
NotFound,
#[error("invalid request: {0}")]
Invalid(String),
#[error("forbidden")]
Forbidden,
#[error("authorization repo error: {0}")]
AuthzRepo(String),
#[error("files backend: {0}")]
Backend(String),
}
impl From<AuthzDenied> for FilesApiError {
fn from(d: AuthzDenied) -> Self {
match d {
AuthzDenied::Denied => Self::Forbidden,
AuthzDenied::Repo(e) => Self::AuthzRepo(e.to_string()),
}
}
}
impl From<AuthzError> for FilesApiError {
fn from(e: AuthzError) -> Self {
Self::AuthzRepo(e.to_string())
}
}
impl From<FilesRepoError> for FilesApiError {
fn from(e: FilesRepoError) -> Self {
Self::Backend(e.to_string())
}
}
impl IntoResponse for FilesApiError {
fn into_response(self) -> Response {
let (status, body) = match &self {
Self::AppNotFound | Self::NotFound => {
(StatusCode::NOT_FOUND, json!({ "error": self.to_string() }))
}
Self::Invalid(_) => (
StatusCode::UNPROCESSABLE_ENTITY,
json!({ "error": self.to_string() }),
),
Self::Forbidden => (StatusCode::FORBIDDEN, json!({ "error": self.to_string() })),
Self::AuthzRepo(e) => {
tracing::error!(error = %e, "files admin authz repo error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
Self::Backend(e) => {
tracing::error!(error = %e, "files admin backend error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
};
(status, Json(body)).into_response()
}
}

View File

@@ -0,0 +1,759 @@
//! `FilesRepo` — the metadata row (Postgres) + blob bytes (filesystem)
//! storage layer for the v1.1.5 `files::*` SDK.
//!
//! Unlike KV/docs, this repo owns BOTH halves of a file: the `files`
//! row (metadata + SHA-256 checksum) and the bytes on disk at
//! `<root>/files/<app_id>/<collection>/<id[0:2]>/<id>`.
//! It owns both because the write must be atomic across them — a crash
//! mid-write must never leave a readable half-written file.
//!
//! ## Atomic write protocol (`create` / `update`)
//! 1. Validate (collection path-safety; caps live one layer up).
//! 2. `create_dir_all` the shard dir with `0o700`.
//! 3. SHA-256 the in-memory bytes (single pass) while writing to
//! `<final>.tmp.<unique>`.
//! 4. `fsync` the temp file.
//! 5. `rename` temp → final (atomic on POSIX).
//! 6. `fsync` the parent dir (so the rename is durable).
//! 7. INSERT / UPDATE the DB row.
//!
//! A crash between 15 leaves an orphan `*.tmp.*` (never read). A crash
//! between 57 leaves a file with no row — never reachable via the SDK
//! (reads start from the row). Both are reclaimed by a future orphan
//! sweep (deferred to v1.1.6+; see HANDBACK §7).
//!
//! ## Atomic delete protocol
//! 1. SELECT + DELETE the row inside one transaction; commit.
//! 2. `unlink` the file (outside the tx). A failure here leaves an
//! orphan; a failure before the commit changes nothing.
//!
//! ## Checksum-on-read
//! `get` reads the file, hashes it, and compares against the stored
//! checksum — returning `FilesError::Corrupted` (and logging the path
//! at error level) on a mismatch. It never auto-deletes; the operator
//! decides what to do with a metadata-vs-bytes divergence.
use std::env;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
use async_trait::async_trait;
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine as _;
use chrono::{DateTime, Utc};
use picloud_shared::{AppId, FileMeta, FileUpdate, FilesListPage, NewFile};
use sha2::{Digest, Sha256};
use sqlx::PgPool;
use uuid::Uuid;
/// 100 MB default per-file cap.
pub const DEFAULT_MAX_FILE_SIZE_BYTES: usize = 100 * 1024 * 1024;
/// Default filesystem root (relative to the process CWD).
pub const DEFAULT_FILES_ROOT: &str = "./data";
const FILES_LIST_MAX_LIMIT: u32 = 1_000;
const FILES_LIST_DEFAULT_LIMIT: u32 = 100;
/// Monotonic counter feeding unique temp-file suffixes (combined with
/// the pid). Avoids `rand` in the storage layer per the brief.
static TMP_COUNTER: AtomicU64 = AtomicU64::new(0);
#[derive(Debug, thiserror::Error)]
pub enum FilesRepoError {
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
#[error("filesystem error: {0}")]
Io(String),
#[error("invalid collection name: {0}")]
InvalidCollection(String),
/// The bytes on disk no longer match the stored checksum (or are
/// missing entirely while the row persists).
#[error("file content corrupted (checksum mismatch)")]
Corrupted,
#[error("invalid pagination cursor")]
InvalidCursor,
}
/// Outbound-files tunables. Env-overridable following the same pattern
/// as `HttpConfig::from_env`.
#[derive(Debug, Clone)]
pub struct FilesConfig {
pub root: PathBuf,
pub max_file_size_bytes: usize,
}
impl FilesConfig {
#[must_use]
pub fn conservative() -> Self {
Self {
root: PathBuf::from(DEFAULT_FILES_ROOT),
max_file_size_bytes: DEFAULT_MAX_FILE_SIZE_BYTES,
}
}
#[must_use]
pub fn from_env() -> Self {
let mut c = Self::conservative();
if let Ok(v) = env::var("PICLOUD_FILES_ROOT") {
if !v.trim().is_empty() {
c.root = PathBuf::from(v);
}
}
if let Ok(v) = env::var("PICLOUD_FILES_MAX_FILE_SIZE_BYTES") {
match v.parse::<usize>() {
Ok(n) => c.max_file_size_bytes = n,
Err(e) => {
tracing::warn!(error = %e, "ignoring invalid PICLOUD_FILES_MAX_FILE_SIZE_BYTES");
}
}
}
c
}
}
impl Default for FilesConfig {
fn default() -> Self {
Self::conservative()
}
}
/// The new+prior metadata returned from a successful `update`, so the
/// service can emit a `ServiceEvent` with the change-data-capture
/// surface (`old_payload`).
#[derive(Debug, Clone)]
pub struct FileUpdated {
pub new: FileMeta,
pub prev: FileMeta,
}
#[async_trait]
pub trait FilesRepo: Send + Sync {
async fn create(
&self,
app_id: AppId,
collection: &str,
new: NewFile,
) -> Result<FileMeta, FilesRepoError>;
async fn head(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError>;
/// Reads + checksum-verifies the bytes. `Ok(None)` when no row
/// exists; `Err(Corrupted)` when the row exists but the bytes are
/// missing or mismatched.
async fn get(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<Vec<u8>>, FilesRepoError>;
/// `Ok(None)` when no row exists (the SDK turns this into
/// `FilesError::NotFound`).
async fn update(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<Option<FileUpdated>, FilesRepoError>;
/// Returns the deleted row's metadata if present, `None` otherwise.
async fn delete(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError>;
async fn list(
&self,
app_id: AppId,
collection: &str,
cursor: Option<&str>,
limit: u32,
) -> Result<FilesListPage, FilesRepoError>;
}
/// Filesystem-bytes + Postgres-metadata repo.
pub struct FsFilesRepo {
pool: PgPool,
config: FilesConfig,
}
impl FsFilesRepo {
#[must_use]
pub fn new(pool: PgPool, config: FilesConfig) -> Self {
Self { pool, config }
}
/// Defensive path-component guard. The service already validates the
/// collection at the SDK boundary; this is belt-and-suspenders so a
/// future caller can't smuggle a traversal sequence onto disk.
fn guard_collection(collection: &str) -> Result<(), FilesRepoError> {
if collection.is_empty()
|| collection.contains('/')
|| collection.contains('\\')
|| collection.contains("..")
|| collection.contains('\0')
{
return Err(FilesRepoError::InvalidCollection(collection.to_string()));
}
Ok(())
}
fn final_path(&self, app_id: AppId, collection: &str, id: Uuid) -> PathBuf {
final_path_at(&self.config.root, app_id, collection, id)
}
fn write_atomic(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
bytes: &[u8],
) -> Result<String, FilesRepoError> {
write_atomic_at(&self.config.root, app_id, collection, id, bytes)
}
}
fn shard_dir_at(root: &Path, app_id: AppId, collection: &str, id_str: &str) -> PathBuf {
root.join("files")
.join(app_id.into_inner().to_string())
.join(collection)
.join(&id_str[..2])
}
fn final_path_at(root: &Path, app_id: AppId, collection: &str, id: Uuid) -> PathBuf {
let id_str = id.to_string();
shard_dir_at(root, app_id, collection, &id_str).join(&id_str)
}
/// Steps 26 of the atomic-write protocol. Returns the lowercase hex
/// SHA-256 of the bytes (computed in a single pass over the in-memory
/// buffer — the file is never re-read). Free function so the fs
/// mechanics are unit-testable without a Postgres pool.
fn write_atomic_at(
root: &Path,
app_id: AppId,
collection: &str,
id: Uuid,
bytes: &[u8],
) -> Result<String, FilesRepoError> {
use std::io::Write as _;
let id_str = id.to_string();
let dir = shard_dir_at(root, app_id, collection, &id_str);
create_dir_all_secure(&dir)?;
// Single-pass checksum over the in-memory buffer.
let mut hasher = Sha256::new();
hasher.update(bytes);
let checksum = hex_lower(&hasher.finalize());
let seq = TMP_COUNTER.fetch_add(1, Ordering::Relaxed);
let tmp = dir.join(format!("{id_str}.tmp.{}-{seq}", std::process::id()));
let final_path = dir.join(&id_str);
{
let mut f = std::fs::File::create(&tmp).map_err(io_err)?;
f.write_all(bytes).map_err(io_err)?;
f.sync_all().map_err(io_err)?; // fsync temp
}
std::fs::rename(&tmp, &final_path).map_err(io_err)?; // atomic
// fsync the parent dir so the rename is durable.
if let Ok(dirf) = std::fs::File::open(&dir) {
let _ = dirf.sync_all();
}
Ok(checksum)
}
/// Read + checksum-verify the bytes at the given path-set. Free
/// function mirror of the `get` read path. Returns `Corrupted` when the
/// bytes are missing or don't match `expected_checksum`.
fn read_verify_at(
root: &Path,
app_id: AppId,
collection: &str,
id: Uuid,
expected_checksum: &str,
) -> Result<Vec<u8>, FilesRepoError> {
let path = final_path_at(root, app_id, collection, id);
let bytes = match std::fs::read(&path) {
Ok(b) => b,
Err(e) => {
tracing::error!(
path = %path.display(), error = %e,
"files: row exists but bytes are unreadable — treating as corrupted"
);
return Err(FilesRepoError::Corrupted);
}
};
let mut hasher = Sha256::new();
hasher.update(&bytes);
let actual = hex_lower(&hasher.finalize());
if actual != expected_checksum {
tracing::error!(
path = %path.display(), expected = %expected_checksum, actual = %actual,
"files: checksum mismatch on read — content corrupted"
);
return Err(FilesRepoError::Corrupted);
}
Ok(bytes)
}
#[async_trait]
impl FilesRepo for FsFilesRepo {
async fn create(
&self,
app_id: AppId,
collection: &str,
new: NewFile,
) -> Result<FileMeta, FilesRepoError> {
Self::guard_collection(collection)?;
let id = Uuid::new_v4();
let size = i64::try_from(new.data.len()).unwrap_or(i64::MAX);
let checksum = self.write_atomic(app_id, collection, id, &new.data)?;
let row: FileRow = sqlx::query_as(
"INSERT INTO files \
(app_id, collection, id, name, content_type, size_bytes, checksum_sha256) \
VALUES ($1, $2, $3, $4, $5, $6, $7) \
RETURNING id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.bind(&new.name)
.bind(&new.content_type)
.bind(size)
.bind(&checksum)
.fetch_one(&self.pool)
.await?;
Ok(row.into_meta())
}
async fn head(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError> {
let row: Option<FileRow> = sqlx::query_as(
"SELECT id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at \
FROM files WHERE app_id = $1 AND collection = $2 AND id = $3",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(&self.pool)
.await?;
Ok(row.map(FileRow::into_meta))
}
async fn get(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<Vec<u8>>, FilesRepoError> {
let row: Option<(String,)> = sqlx::query_as(
"SELECT checksum_sha256 FROM files \
WHERE app_id = $1 AND collection = $2 AND id = $3",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(&self.pool)
.await?;
let Some((stored_checksum,)) = row else {
return Ok(None);
};
let bytes = read_verify_at(&self.config.root, app_id, collection, id, &stored_checksum)?;
Ok(Some(bytes))
}
async fn update(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<Option<FileUpdated>, FilesRepoError> {
Self::guard_collection(collection)?;
// Read the prior row first (existence check + CDC surface).
let Some(prev) = self.head(app_id, collection, id).await? else {
return Ok(None);
};
let size = i64::try_from(upd.data.len()).unwrap_or(i64::MAX);
let checksum = self.write_atomic(app_id, collection, id, &upd.data)?;
let row: FileRow = sqlx::query_as(
"UPDATE files SET \
name = COALESCE($4, name), \
content_type = COALESCE($5, content_type), \
size_bytes = $6, \
checksum_sha256 = $7, \
updated_at = NOW() \
WHERE app_id = $1 AND collection = $2 AND id = $3 \
RETURNING id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.bind(upd.name.as_deref())
.bind(upd.content_type.as_deref())
.bind(size)
.bind(&checksum)
.fetch_one(&self.pool)
.await?;
Ok(Some(FileUpdated {
new: row.into_meta(),
prev,
}))
}
async fn delete(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError> {
// SELECT + DELETE in one tx; unlink afterwards (outside the tx).
let mut tx = self.pool.begin().await?;
let row: Option<FileRow> = sqlx::query_as(
"SELECT id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at \
FROM files WHERE app_id = $1 AND collection = $2 AND id = $3 \
FOR UPDATE",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.fetch_optional(&mut *tx)
.await?;
let Some(row) = row else {
tx.rollback().await?;
return Ok(None);
};
sqlx::query("DELETE FROM files WHERE app_id = $1 AND collection = $2 AND id = $3")
.bind(app_id.into_inner())
.bind(collection)
.bind(id)
.execute(&mut *tx)
.await?;
tx.commit().await?;
// Row is gone; unlink the bytes. A failure here leaves an orphan
// file (reclaimed by a future sweep) — not fatal.
let path = self.final_path(app_id, collection, id);
if let Err(e) = std::fs::remove_file(&path) {
if e.kind() != std::io::ErrorKind::NotFound {
tracing::warn!(path = %path.display(), error = %e, "files: unlink after delete failed (orphan)");
}
}
Ok(Some(row.into_meta()))
}
async fn list(
&self,
app_id: AppId,
collection: &str,
cursor: Option<&str>,
limit: u32,
) -> Result<FilesListPage, FilesRepoError> {
let limit = if limit == 0 {
FILES_LIST_DEFAULT_LIMIT
} else {
limit.min(FILES_LIST_MAX_LIMIT)
};
let last_id = match cursor {
Some(c) => Some(decode_cursor(c)?),
None => None,
};
let take = i64::from(limit) + 1;
let rows: Vec<FileRow> = sqlx::query_as(
"SELECT id, collection, name, content_type, size_bytes, \
checksum_sha256, created_at, updated_at \
FROM files \
WHERE app_id = $1 AND collection = $2 \
AND ($3::uuid IS NULL OR id > $3) \
ORDER BY id ASC \
LIMIT $4",
)
.bind(app_id.into_inner())
.bind(collection)
.bind(last_id)
.bind(take)
.fetch_all(&self.pool)
.await?;
let mut files: Vec<FileMeta> = rows.into_iter().map(FileRow::into_meta).collect();
let next_cursor = if files.len() > limit as usize {
files.truncate(limit as usize);
files.last().map(|m| encode_cursor(m.id))
} else {
None
};
Ok(FilesListPage { files, next_cursor })
}
}
// ----------------------------------------------------------------------------
// Helpers
// ----------------------------------------------------------------------------
fn io_err(e: std::io::Error) -> FilesRepoError {
FilesRepoError::Io(e.to_string())
}
/// `create_dir_all` with `0o700` on the created tree (Unix). On other
/// platforms it falls back to the default permissions.
fn create_dir_all_secure(dir: &Path) -> Result<(), FilesRepoError> {
#[cfg(unix)]
{
use std::os::unix::fs::DirBuilderExt as _;
std::fs::DirBuilder::new()
.recursive(true)
.mode(0o700)
.create(dir)
.map_err(io_err)
}
#[cfg(not(unix))]
{
std::fs::create_dir_all(dir).map_err(io_err)
}
}
fn hex_lower(bytes: &[u8]) -> String {
let mut s = String::with_capacity(bytes.len() * 2);
for b in bytes {
use std::fmt::Write as _;
let _ = write!(s, "{b:02x}");
}
s
}
fn encode_cursor(last_id: Uuid) -> String {
URL_SAFE_NO_PAD.encode(last_id.to_string().as_bytes())
}
fn decode_cursor(cursor: &str) -> Result<Uuid, FilesRepoError> {
let bytes = URL_SAFE_NO_PAD
.decode(cursor)
.map_err(|_| FilesRepoError::InvalidCursor)?;
let s = String::from_utf8(bytes).map_err(|_| FilesRepoError::InvalidCursor)?;
Uuid::parse_str(&s).map_err(|_| FilesRepoError::InvalidCursor)
}
#[derive(sqlx::FromRow)]
struct FileRow {
id: Uuid,
collection: String,
name: String,
content_type: String,
size_bytes: i64,
checksum_sha256: String,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
impl FileRow {
fn into_meta(self) -> FileMeta {
FileMeta {
id: self.id,
collection: self.collection,
name: self.name,
content_type: self.content_type,
size: u64::try_from(self.size_bytes).unwrap_or(0),
checksum: self.checksum_sha256,
created_at: self.created_at,
updated_at: self.updated_at,
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn hex_lower_matches_known_sha256_vector() {
// SHA-256("abc") — NIST known-answer vector.
let mut h = Sha256::new();
h.update(b"abc");
assert_eq!(
hex_lower(&h.finalize()),
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
);
}
#[test]
fn hex_lower_of_empty_is_known_vector() {
let mut h = Sha256::new();
h.update(b"");
assert_eq!(
hex_lower(&h.finalize()),
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
);
}
#[test]
fn cursor_round_trips() {
let id = Uuid::new_v4();
let enc = encode_cursor(id);
assert_eq!(decode_cursor(&enc).unwrap(), id);
assert!(matches!(
decode_cursor("!!not-base64!!"),
Err(FilesRepoError::InvalidCursor)
));
}
#[test]
fn guard_collection_rejects_traversal() {
assert!(FsFilesRepo::guard_collection("avatars").is_ok());
assert!(FsFilesRepo::guard_collection("a/b").is_err());
assert!(FsFilesRepo::guard_collection("..").is_err());
assert!(FsFilesRepo::guard_collection("a..b").is_err());
assert!(FsFilesRepo::guard_collection("").is_err());
assert!(FsFilesRepo::guard_collection("a\0b").is_err());
}
#[test]
fn config_from_env_defaults_are_conservative() {
let c = FilesConfig::conservative();
assert_eq!(c.max_file_size_bytes, DEFAULT_MAX_FILE_SIZE_BYTES);
assert_eq!(c.root, PathBuf::from(DEFAULT_FILES_ROOT));
}
// ------------------------------------------------------------------
// Tempdir-backed filesystem mechanics — exercise the atomic write,
// single-pass checksum, and checksum-on-read tamper detection
// without needing a Postgres pool.
// ------------------------------------------------------------------
use picloud_shared::AppId;
/// Process-unique scratch dir under the system temp dir. Cleaned up
/// by each test via `remove_dir_all`.
fn unique_tmp_root() -> PathBuf {
let seq = TMP_COUNTER.fetch_add(1, Ordering::Relaxed);
let dir =
std::env::temp_dir().join(format!("picloud-files-test-{}-{seq}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir
}
#[test]
fn write_atomic_then_read_verify_round_trips() {
let root = unique_tmp_root();
let app = AppId::new();
let id = Uuid::new_v4();
let bytes = b"hello picloud files".to_vec();
let checksum = write_atomic_at(&root, app, "avatars", id, &bytes).unwrap();
// Single-pass checksum matches an independent hash of the bytes.
let mut h = Sha256::new();
h.update(&bytes);
assert_eq!(checksum, hex_lower(&h.finalize()));
let read = read_verify_at(&root, app, "avatars", id, &checksum).unwrap();
assert_eq!(read, bytes);
std::fs::remove_dir_all(&root).ok();
}
#[test]
fn read_verify_detects_tampering_as_corrupted() {
let root = unique_tmp_root();
let app = AppId::new();
let id = Uuid::new_v4();
let checksum = write_atomic_at(&root, app, "c", id, b"original").unwrap();
// Mutate the bytes behind the repo's back.
let path = final_path_at(&root, app, "c", id);
std::fs::write(&path, b"tampered").unwrap();
let err = read_verify_at(&root, app, "c", id, &checksum).unwrap_err();
assert!(matches!(err, FilesRepoError::Corrupted));
std::fs::remove_dir_all(&root).ok();
}
#[test]
fn read_verify_missing_bytes_is_corrupted() {
let root = unique_tmp_root();
let app = AppId::new();
let id = Uuid::new_v4();
// No write — the file never existed.
let err = read_verify_at(&root, app, "c", id, "deadbeef").unwrap_err();
assert!(matches!(err, FilesRepoError::Corrupted));
std::fs::remove_dir_all(&root).ok();
}
#[test]
fn atomic_write_leaves_no_tmp_file_after_success() {
let root = unique_tmp_root();
let app = AppId::new();
let id = Uuid::new_v4();
write_atomic_at(&root, app, "c", id, b"data").unwrap();
let id_str = id.to_string();
let dir = shard_dir_at(&root, app, "c", &id_str);
let entries: Vec<_> = std::fs::read_dir(&dir)
.unwrap()
.filter_map(Result::ok)
.map(|e| e.file_name().to_string_lossy().into_owned())
.collect();
// Exactly the final file is visible — no `*.tmp.*` orphan.
assert_eq!(entries, vec![id_str]);
assert!(!entries.iter().any(|n| n.contains(".tmp.")));
std::fs::remove_dir_all(&root).ok();
}
#[test]
fn id_shard_uses_first_two_chars() {
let root = PathBuf::from("/tmp/x");
let app = AppId::new();
let id = Uuid::new_v4();
let id_str = id.to_string();
let path = final_path_at(&root, app, "col", id);
let shard = &id_str[..2];
assert!(path
.to_string_lossy()
.contains(&format!("/col/{shard}/{id_str}")));
}
#[cfg(unix)]
#[test]
fn shard_tree_created_with_0700() {
use std::os::unix::fs::PermissionsExt as _;
let root = unique_tmp_root();
let app = AppId::new();
let id = Uuid::new_v4();
write_atomic_at(&root, app, "c", id, b"data").unwrap();
let id_str = id.to_string();
let dir = shard_dir_at(&root, app, "c", &id_str);
let mode = std::fs::metadata(&dir).unwrap().permissions().mode();
assert_eq!(mode & 0o777, 0o700, "shard dir should be 0o700");
std::fs::remove_dir_all(&root).ok();
}
}

View File

@@ -0,0 +1,833 @@
//! `FilesServiceImpl` — wires the `FilesRepo` underneath the
//! `picloud_shared::FilesService` trait scripts see via the Rhai
//! bridge.
//!
//! Layers added here (vs the raw repo), mirroring `KvServiceImpl`:
//! 1. Collection validation (empty + path-traversal) and field /
//! size-cap validation at the SDK boundary.
//! 2. **Script-as-gate authz**: when `cx.principal.is_some()` we run
//! `authz::require(...)`; when it's `None` (public HTTP) we skip.
//! Cross-app isolation is unaffected — every repo call is keyed by
//! `cx.app_id`, never an argument.
//! 3. `ServiceEvent` emission after each mutation (`create` /
//! `update` / `delete`). The payload is the file **metadata**, not
//! the blob bytes (files are too big for trigger payloads).
use std::sync::Arc;
use async_trait::async_trait;
use picloud_shared::{
validate_files_collection, FileMeta, FileUpdate, FilesError, FilesListPage, FilesService,
NewFile, SdkCallCx, ServiceEvent, ServiceEventEmitter,
};
use uuid::Uuid;
use crate::authz::{self, AuthzRepo, Capability};
use crate::files_repo::{FileUpdated, FilesRepo, FilesRepoError};
pub struct FilesServiceImpl {
repo: Arc<dyn FilesRepo>,
authz: Arc<dyn AuthzRepo>,
events: Arc<dyn ServiceEventEmitter>,
max_file_size_bytes: usize,
}
impl FilesServiceImpl {
#[must_use]
pub fn new(
repo: Arc<dyn FilesRepo>,
authz: Arc<dyn AuthzRepo>,
events: Arc<dyn ServiceEventEmitter>,
max_file_size_bytes: usize,
) -> Self {
Self {
repo,
authz,
events,
max_file_size_bytes,
}
}
async fn check_read(&self, cx: &SdkCallCx) -> Result<(), FilesError> {
if let Some(ref principal) = cx.principal {
authz::require(&*self.authz, principal, Capability::AppFilesRead(cx.app_id))
.await
.map_err(|_| FilesError::Forbidden)?;
}
Ok(())
}
async fn check_write(&self, cx: &SdkCallCx) -> Result<(), FilesError> {
if let Some(ref principal) = cx.principal {
authz::require(
&*self.authz,
principal,
Capability::AppFilesWrite(cx.app_id),
)
.await
.map_err(|_| FilesError::Forbidden)?;
}
Ok(())
}
/// Best-effort `ServiceEvent` emission. A failed emit is logged but
/// never rolls back the (already-durable) file write.
async fn emit(
&self,
cx: &SdkCallCx,
op: &'static str,
collection: &str,
meta: &FileMeta,
old: Option<&FileMeta>,
) {
let payload = serde_json::to_value(meta).ok();
let old_payload = old.and_then(|m| serde_json::to_value(m).ok());
if let Err(e) = self
.events
.emit(
cx,
ServiceEvent {
source: "files",
op,
collection: Some(collection.to_string()),
key: Some(meta.id.to_string()),
payload,
old_payload,
},
)
.await
{
tracing::warn!(error = %e, source = "files", op, "event emit failed");
}
}
}
/// Parse a script-supplied id. Invalid UUIDs aren't an error shape the
/// SDK exposes — for reads/deletes they simply mean "no such file".
fn parse_id(id: &str) -> Option<Uuid> {
Uuid::parse_str(id).ok()
}
impl From<FilesRepoError> for FilesError {
fn from(e: FilesRepoError) -> Self {
match e {
FilesRepoError::Corrupted => Self::Corrupted,
FilesRepoError::InvalidCollection(c) => Self::InvalidCollection(c),
other => Self::Backend(other.to_string()),
}
}
}
#[async_trait]
impl FilesService for FilesServiceImpl {
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError> {
validate_files_collection(collection)?;
self.check_write(cx).await?;
new.validate(self.max_file_size_bytes)?;
let meta = self.repo.create(cx.app_id, collection, new).await?;
self.emit(cx, "create", collection, &meta, None).await;
Ok(meta.id)
}
async fn head(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<FileMeta>, FilesError> {
validate_files_collection(collection)?;
self.check_read(cx).await?;
let Some(uuid) = parse_id(id) else {
return Ok(None);
};
Ok(self.repo.head(cx.app_id, collection, uuid).await?)
}
async fn get(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<Vec<u8>>, FilesError> {
validate_files_collection(collection)?;
self.check_read(cx).await?;
let Some(uuid) = parse_id(id) else {
return Ok(None);
};
Ok(self.repo.get(cx.app_id, collection, uuid).await?)
}
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
upd: FileUpdate,
) -> Result<(), FilesError> {
validate_files_collection(collection)?;
self.check_write(cx).await?;
upd.validate(self.max_file_size_bytes)?;
let Some(uuid) = parse_id(id) else {
return Err(FilesError::NotFound);
};
match self.repo.update(cx.app_id, collection, uuid, upd).await? {
Some(FileUpdated { new, prev }) => {
self.emit(cx, "update", collection, &new, Some(&prev)).await;
Ok(())
}
None => Err(FilesError::NotFound),
}
}
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError> {
validate_files_collection(collection)?;
self.check_write(cx).await?;
let Some(uuid) = parse_id(id) else {
return Ok(false);
};
match self.repo.delete(cx.app_id, collection, uuid).await? {
Some(meta) => {
// On delete, the top-level metadata AND `prev` both carry
// the deleted row (per docs/v1.1.x design + the brief).
self.emit(cx, "delete", collection, &meta, Some(&meta))
.await;
Ok(true)
}
None => Ok(false),
}
}
async fn list(
&self,
cx: &SdkCallCx,
collection: &str,
cursor: Option<&str>,
limit: u32,
) -> Result<FilesListPage, FilesError> {
validate_files_collection(collection)?;
self.check_read(cx).await?;
Ok(self.repo.list(cx.app_id, collection, cursor, limit).await?)
}
}
// ----------------------------------------------------------------------------
// Tests — in-memory FilesRepo so unit tests need neither Postgres nor a
// filesystem. The on-disk atomic-write / checksum mechanics are covered
// by the tempdir tests in `files_repo.rs`.
// ----------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::authz::{AuthzError, AuthzRepo};
use async_trait::async_trait;
use chrono::Utc;
use picloud_shared::{
AdminUserId, AppId, AppRole, EmitError, ExecutionId, InstanceRole, Principal, RequestId,
ScriptId, ServiceEvent, UserId,
};
use std::collections::BTreeMap;
use std::sync::Mutex as StdMutex;
use tokio::sync::Mutex;
/// In-memory FilesRepo keyed by (app, collection, id). Stores the
/// metadata + bytes together so cross-app isolation and round-trips
/// can be checked without disk.
#[derive(Default)]
struct InMemoryFilesRepo {
#[allow(clippy::type_complexity)]
data: Mutex<BTreeMap<(AppId, String, Uuid), (FileMeta, Vec<u8>)>>,
}
fn sha256_hex(bytes: &[u8]) -> String {
use sha2::{Digest, Sha256};
let mut h = Sha256::new();
h.update(bytes);
let out = h.finalize();
let mut s = String::new();
for b in out {
use std::fmt::Write as _;
let _ = write!(s, "{b:02x}");
}
s
}
#[async_trait]
impl FilesRepo for InMemoryFilesRepo {
async fn create(
&self,
app_id: AppId,
collection: &str,
new: NewFile,
) -> Result<FileMeta, FilesRepoError> {
let id = Uuid::new_v4();
let now = Utc::now();
let meta = FileMeta {
id,
collection: collection.to_string(),
name: new.name.clone(),
content_type: new.content_type.clone(),
size: new.data.len() as u64,
checksum: sha256_hex(&new.data),
created_at: now,
updated_at: now,
};
self.data.lock().await.insert(
(app_id, collection.to_string(), id),
(meta.clone(), new.data),
);
Ok(meta)
}
async fn head(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError> {
Ok(self
.data
.lock()
.await
.get(&(app_id, collection.to_string(), id))
.map(|(m, _)| m.clone()))
}
async fn get(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<Vec<u8>>, FilesRepoError> {
Ok(self
.data
.lock()
.await
.get(&(app_id, collection.to_string(), id))
.map(|(_, b)| b.clone()))
}
async fn update(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
upd: FileUpdate,
) -> Result<Option<FileUpdated>, FilesRepoError> {
let mut data = self.data.lock().await;
let key = (app_id, collection.to_string(), id);
let Some((prev_meta, _)) = data.get(&key).cloned() else {
return Ok(None);
};
let now = Utc::now();
let new_meta = FileMeta {
id,
collection: collection.to_string(),
name: upd.name.clone().unwrap_or_else(|| prev_meta.name.clone()),
content_type: upd
.content_type
.clone()
.unwrap_or_else(|| prev_meta.content_type.clone()),
size: upd.data.len() as u64,
checksum: sha256_hex(&upd.data),
created_at: prev_meta.created_at,
updated_at: now,
};
data.insert(key, (new_meta.clone(), upd.data));
Ok(Some(FileUpdated {
new: new_meta,
prev: prev_meta,
}))
}
async fn delete(
&self,
app_id: AppId,
collection: &str,
id: Uuid,
) -> Result<Option<FileMeta>, FilesRepoError> {
Ok(self
.data
.lock()
.await
.remove(&(app_id, collection.to_string(), id))
.map(|(m, _)| m))
}
async fn list(
&self,
app_id: AppId,
collection: &str,
cursor: Option<&str>,
limit: u32,
) -> Result<FilesListPage, FilesRepoError> {
let data = self.data.lock().await;
let after = cursor.and_then(|c| Uuid::parse_str(c).ok());
let mut metas: Vec<FileMeta> = data
.iter()
.filter(|((a, c, _), _)| *a == app_id && c == collection)
.map(|(_, (m, _))| m.clone())
.filter(|m| after.is_none_or(|a| m.id > a))
.collect();
metas.sort_by_key(|m| m.id);
let take = (limit.max(1)) as usize;
let next_cursor = if metas.len() > take {
metas.truncate(take);
metas.last().map(|m| m.id.to_string())
} else {
None
};
Ok(FilesListPage {
files: metas,
next_cursor,
})
}
}
/// Captures emitted events so tests can assert on fan-out shape.
#[derive(Default)]
struct CapturingEmitter {
events: StdMutex<Vec<ServiceEvent>>,
}
#[async_trait]
impl ServiceEventEmitter for CapturingEmitter {
async fn emit(&self, _cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
self.events.lock().unwrap().push(event);
Ok(())
}
}
#[derive(Default)]
struct DenyingAuthzRepo;
#[async_trait]
impl AuthzRepo for DenyingAuthzRepo {
async fn membership(
&self,
_user_id: UserId,
_app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok(None)
}
}
#[derive(Default)]
struct EditorAuthzRepo;
#[async_trait]
impl AuthzRepo for EditorAuthzRepo {
async fn membership(
&self,
_user_id: UserId,
_app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok(Some(AppRole::Editor))
}
}
fn anon_cx(app_id: AppId) -> SdkCallCx {
SdkCallCx {
app_id,
script_id: ScriptId::new(),
principal: None,
execution_id: ExecutionId::new(),
request_id: RequestId::new(),
trigger_depth: 0,
root_execution_id: ExecutionId::new(),
is_dead_letter_handler: false,
event: None,
}
}
fn member_cx(app_id: AppId) -> SdkCallCx {
SdkCallCx {
principal: Some(Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Member,
scopes: None,
app_binding: None,
}),
..anon_cx(app_id)
}
}
fn svc_with(authz: Arc<dyn AuthzRepo>, emitter: Arc<CapturingEmitter>) -> FilesServiceImpl {
FilesServiceImpl::new(
Arc::new(InMemoryFilesRepo::default()),
authz,
emitter,
10 * 1024 * 1024,
)
}
fn svc() -> FilesServiceImpl {
svc_with(
Arc::new(DenyingAuthzRepo),
Arc::new(CapturingEmitter::default()),
)
}
fn new_file(name: &str, data: &[u8]) -> NewFile {
NewFile {
name: name.to_string(),
content_type: "application/octet-stream".to_string(),
data: data.to_vec(),
}
}
#[tokio::test]
async fn create_then_get_head_round_trips() {
let files = svc();
let cx = anon_cx(AppId::new());
let id = files
.create(&cx, "avatars", new_file("a.bin", b"hello"))
.await
.unwrap();
let bytes = files.get(&cx, "avatars", &id.to_string()).await.unwrap();
assert_eq!(bytes, Some(b"hello".to_vec()));
let meta = files
.head(&cx, "avatars", &id.to_string())
.await
.unwrap()
.unwrap();
assert_eq!(meta.name, "a.bin");
assert_eq!(meta.size, 5);
assert_eq!(meta.checksum, sha256_hex(b"hello"));
}
#[tokio::test]
async fn get_and_head_missing_return_none() {
let files = svc();
let cx = anon_cx(AppId::new());
let missing = Uuid::new_v4().to_string();
assert_eq!(files.get(&cx, "c", &missing).await.unwrap(), None);
assert!(files.head(&cx, "c", &missing).await.unwrap().is_none());
// Non-UUID id is also "missing", not an error.
assert_eq!(files.get(&cx, "c", "not-a-uuid").await.unwrap(), None);
}
#[tokio::test]
async fn update_replaces_content_and_keeps_metadata_when_omitted() {
let files = svc();
let cx = anon_cx(AppId::new());
let id = files
.create(&cx, "c", new_file("v1.txt", b"one"))
.await
.unwrap();
files
.update(
&cx,
"c",
&id.to_string(),
FileUpdate {
data: b"two!!".to_vec(),
name: None,
content_type: None,
},
)
.await
.unwrap();
let meta = files
.head(&cx, "c", &id.to_string())
.await
.unwrap()
.unwrap();
assert_eq!(meta.name, "v1.txt"); // kept
assert_eq!(meta.size, 5);
assert_eq!(
files.get(&cx, "c", &id.to_string()).await.unwrap(),
Some(b"two!!".to_vec())
);
}
#[tokio::test]
async fn update_missing_throws_not_found() {
let files = svc();
let cx = anon_cx(AppId::new());
let err = files
.update(
&cx,
"c",
&Uuid::new_v4().to_string(),
FileUpdate {
data: b"x".to_vec(),
name: None,
content_type: None,
},
)
.await
.unwrap_err();
assert!(matches!(err, FilesError::NotFound));
}
#[tokio::test]
async fn delete_returns_was_present() {
let files = svc();
let cx = anon_cx(AppId::new());
let id = files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
assert!(files.delete(&cx, "c", &id.to_string()).await.unwrap());
assert!(!files.delete(&cx, "c", &id.to_string()).await.unwrap());
assert!(!files.delete(&cx, "c", "not-a-uuid").await.unwrap());
}
#[tokio::test]
async fn empty_collection_rejected() {
let files = svc();
let cx = anon_cx(AppId::new());
let err = files
.create(&cx, "", new_file("f", b"x"))
.await
.unwrap_err();
assert!(matches!(err, FilesError::InvalidCollection(_)));
}
#[tokio::test]
async fn traversal_collection_rejected() {
let files = svc();
let cx = anon_cx(AppId::new());
for bad in ["../etc", "a/b", "a..b", "x\0y"] {
let err = files
.create(&cx, bad, new_file("f", b"x"))
.await
.unwrap_err();
assert!(
matches!(err, FilesError::InvalidCollection(_)),
"expected reject for {bad:?}"
);
}
}
#[tokio::test]
async fn missing_required_fields_have_field_specific_messages() {
let files = svc();
let cx = anon_cx(AppId::new());
// name
let err = files
.create(
&cx,
"c",
NewFile {
name: " ".into(),
content_type: "text/plain".into(),
data: b"x".to_vec(),
},
)
.await
.unwrap_err();
assert!(matches!(err, FilesError::MissingField("name")));
// content_type
let err = files
.create(
&cx,
"c",
NewFile {
name: "f".into(),
content_type: String::new(),
data: b"x".to_vec(),
},
)
.await
.unwrap_err();
assert!(matches!(err, FilesError::MissingField("content_type")));
// Empty `data` is NO LONGER rejected (v1.1.6 relaxation) — see
// `empty_file_round_trips`.
}
#[tokio::test]
async fn empty_file_round_trips() {
// v1.1.6: a zero-byte blob is a valid stored state (sentinels,
// placeholders). Create with empty data, then read it back.
let files = svc();
let cx = anon_cx(AppId::new());
let id = files
.create(
&cx,
"c",
NewFile {
name: "empty.bin".into(),
content_type: "application/octet-stream".into(),
data: vec![],
},
)
.await
.expect("empty file create should succeed");
let bytes = files.get(&cx, "c", &id.to_string()).await.unwrap();
assert_eq!(bytes, Some(Vec::new()));
let meta = files
.head(&cx, "c", &id.to_string())
.await
.unwrap()
.expect("metadata present");
assert_eq!(meta.size, 0);
}
#[tokio::test]
async fn name_and_content_type_length_caps_enforced() {
let files = svc();
let cx = anon_cx(AppId::new());
let long_name = "x".repeat(256);
let err = files
.create(&cx, "c", new_file(&long_name, b"x"))
.await
.unwrap_err();
assert!(matches!(err, FilesError::NameTooLong(256)));
let err = files
.create(
&cx,
"c",
NewFile {
name: "f".into(),
content_type: "x".repeat(128),
data: b"x".to_vec(),
},
)
.await
.unwrap_err();
assert!(matches!(err, FilesError::ContentTypeTooLong(128)));
}
#[tokio::test]
async fn per_file_size_cap_enforced() {
let files = FilesServiceImpl::new(
Arc::new(InMemoryFilesRepo::default()),
Arc::new(DenyingAuthzRepo),
Arc::new(CapturingEmitter::default()),
8, // tiny cap
);
let cx = anon_cx(AppId::new());
let err = files
.create(&cx, "c", new_file("big", b"123456789"))
.await
.unwrap_err();
assert!(matches!(err, FilesError::TooLarge { limit: 8, .. }));
}
#[tokio::test]
async fn cross_app_isolation() {
let files = svc();
let app_a = AppId::new();
let app_b = AppId::new();
let cx_a = anon_cx(app_a);
let cx_b = anon_cx(app_b);
let id = files
.create(&cx_a, "shared", new_file("f", b"from-a"))
.await
.unwrap();
// app B cannot see app A's file by id.
assert_eq!(
files.get(&cx_b, "shared", &id.to_string()).await.unwrap(),
None
);
assert!(files
.head(&cx_b, "shared", &id.to_string())
.await
.unwrap()
.is_none());
let page_b = files.list(&cx_b, "shared", None, 100).await.unwrap();
assert!(page_b.files.is_empty());
// app A still sees it.
assert!(files
.get(&cx_a, "shared", &id.to_string())
.await
.unwrap()
.is_some());
}
#[tokio::test]
async fn anonymous_cx_skips_authz() {
let files = svc(); // DenyingAuthzRepo
let cx = anon_cx(AppId::new());
// No principal → no authz check, even with a denying repo.
files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
}
#[tokio::test]
async fn member_without_role_is_forbidden() {
let files = svc(); // DenyingAuthzRepo
let cx = member_cx(AppId::new());
let err = files
.create(&cx, "c", new_file("f", b"x"))
.await
.unwrap_err();
assert!(matches!(err, FilesError::Forbidden));
}
#[tokio::test]
async fn member_with_editor_role_allowed() {
let files = svc_with(
Arc::new(EditorAuthzRepo),
Arc::new(CapturingEmitter::default()),
);
let cx = member_cx(AppId::new());
files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
}
#[tokio::test]
async fn mutations_emit_events_with_correct_prev() {
let emitter = Arc::new(CapturingEmitter::default());
let files = svc_with(Arc::new(DenyingAuthzRepo), emitter.clone());
let cx = anon_cx(AppId::new());
let id = files.create(&cx, "c", new_file("f", b"one")).await.unwrap();
files
.update(
&cx,
"c",
&id.to_string(),
FileUpdate {
data: b"two".to_vec(),
name: None,
content_type: None,
},
)
.await
.unwrap();
files.delete(&cx, "c", &id.to_string()).await.unwrap();
let events = emitter.events.lock().unwrap();
assert_eq!(events.len(), 3);
// create: prev is None
assert_eq!(events[0].op, "create");
assert_eq!(events[0].source, "files");
assert!(events[0].old_payload.is_none());
assert!(events[0].payload.is_some());
// update: prev is the prior metadata
assert_eq!(events[1].op, "update");
assert!(events[1].old_payload.is_some());
// delete: prev is the deleted metadata (payload == old_payload)
assert_eq!(events[2].op, "delete");
assert_eq!(events[2].payload, events[2].old_payload);
assert!(events[2].payload.is_some());
}
#[tokio::test]
async fn list_cursor_paginates() {
let files = svc();
let cx = anon_cx(AppId::new());
for i in 0..5 {
files
.create(&cx, "c", new_file(&format!("f{i}"), b"x"))
.await
.unwrap();
}
let p1 = files.list(&cx, "c", None, 2).await.unwrap();
assert_eq!(p1.files.len(), 2);
assert!(p1.next_cursor.is_some());
let p2 = files
.list(&cx, "c", p1.next_cursor.as_deref(), 2)
.await
.unwrap();
assert_eq!(p2.files.len(), 2);
let p3 = files
.list(&cx, "c", p2.next_cursor.as_deref(), 2)
.await
.unwrap();
assert_eq!(p3.files.len(), 1);
assert!(p3.next_cursor.is_none());
}
}

View File

@@ -0,0 +1,185 @@
//! Orphan `*.tmp.*` blob sweeper (v1.1.6, v1.1.5 follow-up).
//!
//! The files repo writes blobs atomically: it streams into a
//! `<id>.tmp.<pid>-<seq>` temp file, fsyncs, then renames to the final
//! `<id>` path. A crash between create and rename leaves an orphan temp
//! file that is never read and never reclaimed. This sweeper deletes
//! those: every `PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC` (default 6h) it
//! walks `<root>/files/` and unlinks any `*.tmp.*` file older than
//! `PICLOUD_FILES_ORPHAN_TMP_TTL_SEC` (default 1h).
//!
//! Deliberately bounded: it does NOT cross-check on-disk files against DB
//! rows (the full reconciling sweeper is v1.3+). It only targets the temp
//! files, which are unambiguously orphans once past the TTL — no live
//! writer keeps one around for an hour.
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime};
const ENV_INTERVAL: &str = "PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC";
const ENV_TMP_TTL: &str = "PICLOUD_FILES_ORPHAN_TMP_TTL_SEC";
const DEFAULT_INTERVAL_SECS: u64 = 21_600; // 6h
const DEFAULT_TMP_TTL_SECS: u64 = 3_600; // 1h
/// Marker that identifies a temp blob (`<id>.tmp.<pid>-<seq>`). A final
/// blob is named just `<id>` (a UUID), so it never contains this.
const TMP_MARKER: &str = ".tmp.";
#[derive(Debug, Default, Clone, Copy)]
pub struct SweepStats {
pub dirs_walked: u64,
pub files_deleted: u64,
pub bytes_reclaimed: u64,
}
/// Spawn the periodic orphan sweep. Spawned at startup alongside the
/// cron scheduler and the realtime/cache GC tasks.
pub fn spawn_files_orphan_sweep(files_root: PathBuf) {
let interval = Duration::from_secs(read_secs(ENV_INTERVAL, DEFAULT_INTERVAL_SECS));
let ttl = Duration::from_secs(read_secs(ENV_TMP_TTL, DEFAULT_TMP_TTL_SECS));
tokio::spawn(async move {
let mut ticker = tokio::time::interval(interval);
ticker.tick().await; // skip the immediate first fire
loop {
ticker.tick().await;
let root = files_root.clone();
// Blocking filesystem walk off the async worker.
let stats = tokio::task::spawn_blocking(move || sweep_orphan_tmp_files(&root, ttl))
.await
.unwrap_or_default();
tracing::info!(
dirs_walked = stats.dirs_walked,
files_deleted = stats.files_deleted,
bytes_reclaimed = stats.bytes_reclaimed,
"files orphan sweep complete"
);
}
});
}
/// Walk `<files_root>/files/` and delete `*.tmp.*` files older than
/// `ttl`. Missing root is not an error (returns zeroed stats). Pure +
/// synchronous so it's unit-testable without a runtime.
#[must_use]
pub fn sweep_orphan_tmp_files(files_root: &Path, ttl: Duration) -> SweepStats {
let mut stats = SweepStats::default();
let blobs_dir = files_root.join("files");
if !blobs_dir.is_dir() {
return stats;
}
let now = SystemTime::now();
walk(&blobs_dir, ttl, now, &mut stats);
stats
}
fn walk(dir: &Path, ttl: Duration, now: SystemTime, stats: &mut SweepStats) {
stats.dirs_walked += 1;
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let Ok(ft) = entry.file_type() else {
continue;
};
let path = entry.path();
if ft.is_dir() {
walk(&path, ttl, now, stats);
continue;
}
if !ft.is_file() {
continue;
}
if !entry.file_name().to_string_lossy().contains(TMP_MARKER) {
continue;
}
let Ok(meta) = entry.metadata() else {
continue;
};
let age = meta
.modified()
.ok()
.and_then(|m| now.duration_since(m).ok())
.unwrap_or(Duration::ZERO);
if age >= ttl {
let size = meta.len();
if std::fs::remove_file(&path).is_ok() {
stats.files_deleted += 1;
stats.bytes_reclaimed += size;
}
}
}
}
fn read_secs(key: &str, default: u64) -> u64 {
match std::env::var(key) {
Err(_) => default,
Ok(v) => match v.parse::<u64>() {
Ok(n) if n > 0 => n,
_ => {
tracing::warn!(env = key, value = %v, "invalid; using default");
default
}
},
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::{AtomicU64, Ordering};
static SEQ: AtomicU64 = AtomicU64::new(0);
fn tmp_root() -> PathBuf {
let n = SEQ.fetch_add(1, Ordering::Relaxed);
let dir =
std::env::temp_dir().join(format!("picloud-sweep-test-{}-{n}", std::process::id()));
std::fs::create_dir_all(dir.join("files").join("ab")).unwrap();
dir
}
fn touch(path: &Path) {
std::fs::write(path, b"x").unwrap();
}
#[test]
fn deletes_old_tmp_files() {
let root = tmp_root();
let tmp = root.join("files/ab/uuid.tmp.123-0");
touch(&tmp);
// ttl 0 → any tmp file counts as old.
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
assert_eq!(stats.files_deleted, 1);
assert!(!tmp.exists());
assert!(stats.bytes_reclaimed >= 1);
}
#[test]
fn keeps_young_tmp_files() {
let root = tmp_root();
let tmp = root.join("files/ab/uuid.tmp.123-0");
touch(&tmp);
// Large TTL → the just-created file is too young to reap.
let stats = sweep_orphan_tmp_files(&root, Duration::from_secs(3600));
assert_eq!(stats.files_deleted, 0);
assert!(tmp.exists());
}
#[test]
fn keeps_non_tmp_files() {
let root = tmp_root();
let blob = root.join("files/ab/0123456789abcdef");
touch(&blob);
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
assert_eq!(stats.files_deleted, 0);
assert!(blob.exists());
}
#[test]
fn missing_root_does_not_panic() {
let root = std::env::temp_dir().join("picloud-sweep-nonexistent-xyz");
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
assert_eq!(stats.files_deleted, 0);
assert_eq!(stats.dirs_walked, 0);
}
}

View File

@@ -16,6 +16,7 @@ pub mod app_domain_repo;
pub mod app_members_api; pub mod app_members_api;
pub mod app_members_repo; pub mod app_members_repo;
pub mod app_repo; pub mod app_repo;
pub mod app_secrets_repo;
pub mod apps_api; pub mod apps_api;
pub mod auth; pub mod auth;
pub mod auth_api; pub mod auth_api;
@@ -30,6 +31,12 @@ pub mod dispatcher;
pub mod docs_filter; pub mod docs_filter;
pub mod docs_repo; pub mod docs_repo;
pub mod docs_service; pub mod docs_service;
pub mod email_inbound_api;
pub mod email_service;
pub mod files_api;
pub mod files_repo;
pub mod files_service;
pub mod files_sweep;
pub mod gc; pub mod gc;
pub mod http_service; pub mod http_service;
pub mod kv_repo; pub mod kv_repo;
@@ -40,12 +47,20 @@ pub mod module_source;
pub mod outbox_event_emitter; pub mod outbox_event_emitter;
pub mod outbox_repo; pub mod outbox_repo;
pub mod principal_resolver; pub mod principal_resolver;
pub mod pubsub_repo;
pub mod pubsub_service;
pub mod realtime_authority;
pub mod repo; pub mod repo;
pub mod route_admin; pub mod route_admin;
pub mod route_repo; pub mod route_repo;
pub mod sandbox; pub mod sandbox;
pub mod scheduler; pub mod scheduler;
pub mod secrets_api;
pub mod secrets_repo;
pub mod secrets_service;
pub mod ssrf; pub mod ssrf;
pub mod topic_repo;
pub mod topics_api;
pub mod trigger_config; pub mod trigger_config;
pub mod trigger_repo; pub mod trigger_repo;
pub mod triggers_api; pub mod triggers_api;
@@ -76,6 +91,9 @@ pub use app_members_repo::{
PostgresAppMembersRepository, PostgresAppMembersRepository,
}; };
pub use app_repo::{resolve_app, AppLookup, AppRepository, PostgresAppRepository}; pub use app_repo::{resolve_app, AppLookup, AppRepository, PostgresAppRepository};
pub use app_secrets_repo::{
AppSecretsRepo, AppSecretsRepoError, PostgresAppSecretsRepo, SIGNING_KEY_LEN,
};
pub use apps_api::{apps_router, AppsState}; pub use apps_api::{apps_router, AppsState};
pub use auth_api::auth_router; pub use auth_api::auth_router;
pub use auth_bootstrap::{ pub use auth_bootstrap::{
@@ -96,6 +114,15 @@ pub use dead_letters_api::{dead_letters_router, DeadLettersApiError, DeadLetters
pub use dispatcher::{compute_backoff, Dispatcher, DispatcherError}; pub use dispatcher::{compute_backoff, Dispatcher, DispatcherError};
pub use docs_repo::{DocsRepo, DocsRepoError, PostgresDocsRepo}; pub use docs_repo::{DocsRepo, DocsRepoError, PostgresDocsRepo};
pub use docs_service::DocsServiceImpl; pub use docs_service::DocsServiceImpl;
pub use email_inbound_api::{email_inbound_router, EmailInboundError, EmailInboundState};
pub use email_service::{
EmailConfig, EmailServiceImpl, EmailTransport, LettreEmailTransport, SmtpConfig, SmtpTls,
DEFAULT_EMAIL_MAX_MESSAGE_BYTES,
};
pub use files_api::{files_admin_router, FilesAdminState};
pub use files_repo::{FilesConfig, FilesRepo, FilesRepoError, FsFilesRepo};
pub use files_service::FilesServiceImpl;
pub use files_sweep::{spawn_files_orphan_sweep, sweep_orphan_tmp_files, SweepStats};
pub use gc::{spawn_abandoned_gc, spawn_dead_letter_gc}; pub use gc::{spawn_abandoned_gc, spawn_dead_letter_gc};
pub use http_service::{HttpConfig, HttpServiceImpl}; pub use http_service::{HttpConfig, HttpServiceImpl};
pub use kv_repo::{KvRepo, KvRepoError, PostgresKvRepo}; pub use kv_repo::{KvRepo, KvRepoError, PostgresKvRepo};
@@ -107,6 +134,9 @@ pub use outbox_repo::{
NewOutboxRow, OutboxRepo, OutboxRepoError, OutboxRow, OutboxSourceKind, PostgresOutboxRepo, NewOutboxRow, OutboxRepo, OutboxRepoError, OutboxRow, OutboxSourceKind, PostgresOutboxRepo,
}; };
pub use principal_resolver::{AdminPrincipalResolver, PrincipalResolver, PrincipalResolverError}; pub use principal_resolver::{AdminPrincipalResolver, PrincipalResolver, PrincipalResolverError};
pub use pubsub_repo::{PostgresPubsubRepo, PublishCtx, PubsubRepo, PubsubRepoError};
pub use pubsub_service::{PubsubServiceImpl, SubscriberTokenConfig};
pub use realtime_authority::RealtimeAuthorityImpl;
pub use repo::{ pub use repo::{
ExecutionLogRepository, NewScript, PostgresExecutionLogRepository, PostgresScriptRepository, ExecutionLogRepository, NewScript, PostgresExecutionLogRepository, PostgresScriptRepository,
RepoResolver, ScriptPatch, ScriptRepository, ScriptRepositoryError, RepoResolver, ScriptPatch, ScriptRepository, ScriptRepositoryError,
@@ -114,10 +144,22 @@ pub use repo::{
pub use route_admin::{compile_routes, route_admin_router, RouteAdminState}; pub use route_admin::{compile_routes, route_admin_router, RouteAdminState};
pub use route_repo::{NewRoute, PostgresRouteRepository, RouteRepository}; pub use route_repo::{NewRoute, PostgresRouteRepository, RouteRepository};
pub use sandbox::{CeilingError, SandboxCeiling}; pub use sandbox::{CeilingError, SandboxCeiling};
pub use secrets_api::{secrets_router, SecretsApiError, SecretsState};
pub use secrets_repo::{
PostgresSecretsRepo, SecretMeta, SecretsMetaPage, SecretsNamePage, SecretsRepo,
SecretsRepoError, StoredSecret,
};
pub use secrets_service::{
open as open_secret, seal as seal_secret, SecretsConfig, SecretsServiceImpl,
DEFAULT_SECRET_MAX_VALUE_BYTES,
};
pub use topic_repo::{PostgresTopicRepo, Topic, TopicAuthMode, TopicRepo, TopicRepoError};
pub use topics_api::{topics_router, TopicsApiError, TopicsState};
pub use trigger_config::{BackoffShape, TriggerConfig}; pub use trigger_config::{BackoffShape, TriggerConfig};
pub use trigger_repo::{ pub use trigger_repo::{
collection_matches, CreateDeadLetterTrigger, CreateDocsTrigger, CreateKvTrigger, collection_matches, CreateDeadLetterTrigger, CreateDocsTrigger, CreateEmailTrigger,
DeadLetterTriggerMatch, DocsTriggerMatch, KvTriggerMatch, PostgresTriggerRepo, Trigger, CreateFilesTrigger, CreateKvTrigger, CreatePubsubTrigger, DeadLetterTriggerMatch,
TriggerDetails, TriggerDispatchMode, TriggerKind, TriggerRepo, TriggerRepoError, DocsTriggerMatch, EmailInboundTarget, FilesTriggerMatch, KvTriggerMatch, PostgresTriggerRepo,
Trigger, TriggerDetails, TriggerDispatchMode, TriggerKind, TriggerRepo, TriggerRepoError,
}; };
pub use triggers_api::{triggers_router, TriggersApiError, TriggersState}; pub use triggers_api::{triggers_router, TriggersApiError, TriggersState};

View File

@@ -19,7 +19,8 @@ use std::sync::Arc;
use async_trait::async_trait; use async_trait::async_trait;
use picloud_shared::{ use picloud_shared::{
DocsEventOp, EmitError, KvEventOp, SdkCallCx, ServiceEvent, ServiceEventEmitter, TriggerEvent, DocsEventOp, EmitError, FileMeta, FilesEventOp, KvEventOp, SdkCallCx, ServiceEvent,
ServiceEventEmitter, TriggerEvent,
}; };
use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxSourceKind}; use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxSourceKind};
@@ -43,6 +44,7 @@ impl ServiceEventEmitter for OutboxEventEmitter {
match event.source { match event.source {
"kv" => self.emit_kv(cx, event).await, "kv" => self.emit_kv(cx, event).await,
"docs" => self.emit_docs(cx, event).await, "docs" => self.emit_docs(cx, event).await,
"files" => self.emit_files(cx, event).await,
// Future sources land here. For now, silently drop — the // Future sources land here. For now, silently drop — the
// SDK calls `events.emit(...)` unconditionally for forward // SDK calls `events.emit(...)` unconditionally for forward
// compat, so swallowing without an error is correct. // compat, so swallowing without an error is correct.
@@ -154,4 +156,68 @@ impl OutboxEventEmitter {
} }
Ok(()) Ok(())
} }
/// v1.1.5. Fan out a files mutation across matching files triggers.
/// The `ServiceEvent.payload` is the file **metadata** (never the
/// blob bytes); `old_payload` is the prior metadata (the deleted
/// row's metadata on delete). The `TriggerEvent::Files` carries the
/// metadata fields explicitly + `prev` for the change-data-capture
/// surface.
async fn emit_files(&self, cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
let Some(op) = FilesEventOp::from_wire(event.op) else {
return Ok(());
};
let Some(collection) = event.collection.clone() else {
return Ok(());
};
// The payload is the FileMeta JSON the FilesServiceImpl emitted.
let Some(meta) = event
.payload
.clone()
.and_then(|v| serde_json::from_value::<FileMeta>(v).ok())
else {
return Ok(());
};
let matches = self
.triggers
.list_matching_files(cx.app_id, &collection, op)
.await
.map_err(|e| EmitError::Unavailable(format!("trigger lookup: {e}")))?;
if matches.is_empty() {
return Ok(());
}
let trigger_event = TriggerEvent::Files {
op,
collection,
id: meta.id.to_string(),
name: meta.name,
content_type: meta.content_type,
size: meta.size,
checksum: meta.checksum,
prev: event.old_payload.clone(),
};
let payload = serde_json::to_value(&trigger_event)
.map_err(|e| EmitError::Rejected(format!("event serialize: {e}")))?;
for m in matches {
self.outbox
.insert(NewOutboxRow {
app_id: cx.app_id,
source_kind: OutboxSourceKind::Files,
trigger_id: Some(m.trigger_id),
script_id: Some(m.script_id),
reply_to: None,
payload: payload.clone(),
origin_principal: cx.principal.as_ref().map(|p| p.user_id),
trigger_depth: cx.trigger_depth.saturating_add(1),
root_execution_id: Some(cx.root_execution_id),
})
.await
.map_err(|e| EmitError::Unavailable(format!("outbox insert: {e}")))?;
}
Ok(())
}
} }

View File

@@ -27,6 +27,12 @@ pub enum OutboxSourceKind {
DeadLetter, DeadLetter,
/// v1.1.4. /// v1.1.4.
Cron, Cron,
/// v1.1.5.
Files,
/// v1.1.5.
Pubsub,
/// v1.1.7. Inbound email POSTed to the webhook receiver.
Email,
} }
impl OutboxSourceKind { impl OutboxSourceKind {
@@ -38,6 +44,9 @@ impl OutboxSourceKind {
Self::Docs => "docs", Self::Docs => "docs",
Self::DeadLetter => "dead_letter", Self::DeadLetter => "dead_letter",
Self::Cron => "cron", Self::Cron => "cron",
Self::Files => "files",
Self::Pubsub => "pubsub",
Self::Email => "email",
} }
} }
@@ -49,6 +58,9 @@ impl OutboxSourceKind {
"docs" => Some(Self::Docs), "docs" => Some(Self::Docs),
"dead_letter" => Some(Self::DeadLetter), "dead_letter" => Some(Self::DeadLetter),
"cron" => Some(Self::Cron), "cron" => Some(Self::Cron),
"files" => Some(Self::Files),
"pubsub" => Some(Self::Pubsub),
"email" => Some(Self::Email),
_ => None, _ => None,
} }
} }

View File

@@ -0,0 +1,118 @@
//! `PubsubRepo` — publish-time fan-out for the v1.1.5 `pubsub::*` SDK.
//!
//! `publish_durable` writes one outbox row per matching enabled `pubsub`
//! trigger, all inside a single transaction so a partial fan-out (some
//! subscribers got rows, others didn't, then a crash) can't happen.
//! Each delivery row then retries / dead-letters independently through
//! the existing dispatcher — no pub/sub-specific dispatch branching.
//!
//! Topic pattern matching runs in Rust (`picloud_shared::topic_matches`)
//! against the small set of the app's enabled pubsub triggers, keeping
//! the SELECT trivial. v1.2 can add a topic-trie index if fan-out
//! becomes a hot path.
use async_trait::async_trait;
use picloud_shared::{topic_matches, AdminUserId, AppId, ExecutionId};
use sqlx::PgPool;
use uuid::Uuid;
#[derive(Debug, thiserror::Error)]
pub enum PubsubRepoError {
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
}
/// The execution-context bits a fan-out needs to stamp onto each outbox
/// row. Derived from the publishing script's `SdkCallCx`.
#[derive(Debug, Clone, Copy)]
pub struct PublishCtx {
pub app_id: AppId,
pub origin_principal: Option<AdminUserId>,
pub trigger_depth: u32,
pub root_execution_id: ExecutionId,
}
#[async_trait]
pub trait PubsubRepo: Send + Sync {
/// Fan out a publish to every matching enabled pubsub trigger in
/// `ctx.app_id`, inserting one outbox row each in a SINGLE
/// transaction. `event_payload` is the serialized
/// `TriggerEvent::Pubsub`. Returns the number of delivery rows
/// written (0 when no trigger matched — the publish still succeeds).
async fn fan_out_publish(
&self,
ctx: PublishCtx,
topic: &str,
event_payload: serde_json::Value,
) -> Result<u32, PubsubRepoError>;
}
pub struct PostgresPubsubRepo {
pool: PgPool,
}
impl PostgresPubsubRepo {
#[must_use]
pub fn new(pool: PgPool) -> Self {
Self { pool }
}
}
#[derive(sqlx::FromRow)]
struct PubsubTriggerRow {
id: Uuid,
script_id: Uuid,
topic_pattern: String,
}
#[async_trait]
impl PubsubRepo for PostgresPubsubRepo {
async fn fan_out_publish(
&self,
ctx: PublishCtx,
topic: &str,
event_payload: serde_json::Value,
) -> Result<u32, PubsubRepoError> {
let mut tx = self.pool.begin().await?;
// Load all enabled pubsub triggers for the app; filter by topic
// pattern in Rust (keeps the query simple, honours the
// empty/`*`/prefix semantics without teaching SQL about globs).
let rows: Vec<PubsubTriggerRow> = sqlx::query_as(
"SELECT t.id, t.script_id, d.topic_pattern \
FROM triggers t \
JOIN pubsub_trigger_details d ON d.trigger_id = t.id \
WHERE t.app_id = $1 AND t.kind = 'pubsub' AND t.enabled = TRUE",
)
.bind(ctx.app_id.into_inner())
.fetch_all(&mut *tx)
.await?;
let mut written: u32 = 0;
for r in rows {
if !topic_matches(&r.topic_pattern, topic) {
continue;
}
sqlx::query(
"INSERT INTO outbox ( \
app_id, source_kind, trigger_id, script_id, reply_to, \
payload, origin_principal, trigger_depth, root_execution_id \
) VALUES ($1, 'pubsub', $2, $3, NULL, $4, $5, $6, $7)",
)
.bind(ctx.app_id.into_inner())
.bind(r.id)
.bind(r.script_id)
.bind(&event_payload)
.bind(ctx.origin_principal.map(AdminUserId::into_inner))
.bind(i32::try_from(ctx.trigger_depth.saturating_add(1)).unwrap_or(1))
.bind(ctx.root_execution_id.into_inner())
.execute(&mut *tx)
.await?;
written += 1;
}
// Commit once — all rows or none.
tx.commit().await?;
Ok(written)
}
}

View File

@@ -0,0 +1,726 @@
//! `PubsubServiceImpl` — wires `PubsubRepo` underneath the
//! `picloud_shared::PubsubService` trait scripts see via the Rhai
//! bridge.
//!
//! Mirrors the other stateful services: script-as-gate authz
//! (`AppPubsubPublish`, skipped when `cx.principal` is `None`), with the
//! backend doing a publish-time outbox fan-out instead of a row write.
//! No `ServiceEventEmitter` here — pub/sub publishes directly to the
//! outbox; it doesn't mutate local data that other triggers observe.
use std::sync::Arc;
use async_trait::async_trait;
use picloud_shared::subscriber_token::{self, TokenClaims};
use picloud_shared::{
PubsubError, PubsubService, RealtimeBroadcaster, RealtimeEvent, SdkCallCx, TriggerEvent,
};
use crate::app_secrets_repo::AppSecretsRepo;
use crate::authz::{self, AuthzRepo, Capability};
use crate::pubsub_repo::{PublishCtx, PubsubRepo, PubsubRepoError};
use crate::topic_repo::TopicRepo;
/// TTL bounds (seconds) for `pubsub::subscriber_token`. Env-overridable
/// via `PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`.
#[derive(Debug, Clone, Copy)]
pub struct SubscriberTokenConfig {
pub min_ttl: i64,
pub max_ttl: i64,
pub default_ttl: i64,
}
impl SubscriberTokenConfig {
#[must_use]
pub const fn conservative() -> Self {
Self {
min_ttl: 10,
max_ttl: 86_400,
default_ttl: 3_600,
}
}
/// Load from env, falling back to the conservative defaults for any
/// missing / invalid value.
#[must_use]
pub fn from_env() -> Self {
let mut c = Self::conservative();
load_i64(&mut c.min_ttl, "PICLOUD_SUBSCRIBER_TOKEN_TTL_MIN_SEC");
load_i64(&mut c.max_ttl, "PICLOUD_SUBSCRIBER_TOKEN_TTL_MAX_SEC");
load_i64(
&mut c.default_ttl,
"PICLOUD_SUBSCRIBER_TOKEN_TTL_DEFAULT_SEC",
);
c
}
}
impl Default for SubscriberTokenConfig {
fn default() -> Self {
Self::conservative()
}
}
fn load_i64(dst: &mut i64, key: &str) {
if let Ok(v) = std::env::var(key) {
match v.parse::<i64>() {
Ok(n) if n > 0 => *dst = n,
_ => tracing::warn!(env = key, value = %v, "ignoring invalid token-ttl value"),
}
}
}
pub struct PubsubServiceImpl {
repo: Arc<dyn PubsubRepo>,
authz: Arc<dyn AuthzRepo>,
// Realtime extras (v1.1.6) — optional so the existing two-arg
// constructor (and its unit tests) keep working without them. The
// production binary attaches them via `with_realtime`.
realtime: Option<Arc<dyn RealtimeBroadcaster>>,
topics: Option<Arc<dyn TopicRepo>>,
secrets: Option<Arc<dyn AppSecretsRepo>>,
token_config: SubscriberTokenConfig,
}
impl PubsubServiceImpl {
#[must_use]
pub fn new(repo: Arc<dyn PubsubRepo>, authz: Arc<dyn AuthzRepo>) -> Self {
Self {
repo,
authz,
realtime: None,
topics: None,
secrets: None,
token_config: SubscriberTokenConfig::conservative(),
}
}
/// Attach the v1.1.6 realtime surface: the in-process broadcaster
/// (publish fan-out to SSE subscribers), the topic registry +
/// app-secrets repo (subscriber-token minting), and the TTL config.
#[must_use]
pub fn with_realtime(
mut self,
broadcaster: Arc<dyn RealtimeBroadcaster>,
topics: Arc<dyn TopicRepo>,
secrets: Arc<dyn AppSecretsRepo>,
token_config: SubscriberTokenConfig,
) -> Self {
self.realtime = Some(broadcaster);
self.topics = Some(topics);
self.secrets = Some(secrets);
self.token_config = token_config;
self
}
async fn check_publish(&self, cx: &SdkCallCx) -> Result<(), PubsubError> {
if let Some(ref principal) = cx.principal {
authz::require(
&*self.authz,
principal,
Capability::AppPubsubPublish(cx.app_id),
)
.await
.map_err(|_| PubsubError::Forbidden)?;
}
Ok(())
}
}
impl From<PubsubRepoError> for PubsubError {
fn from(e: PubsubRepoError) -> Self {
Self::Unavailable(e.to_string())
}
}
#[async_trait]
impl PubsubService for PubsubServiceImpl {
async fn publish_durable(
&self,
cx: &SdkCallCx,
topic: &str,
message: serde_json::Value,
) -> Result<(), PubsubError> {
if topic.trim().is_empty() {
return Err(PubsubError::EmptyTopic);
}
self.check_publish(cx).await?;
// `published_at` is stamped once on the manager side so every
// delivery path — durable triggers AND the realtime broadcast —
// agrees on one instant. The message is cloned into the trigger
// event so the realtime path can reuse the original.
let published_at = chrono::Utc::now();
let event = TriggerEvent::Pubsub {
topic: topic.to_string(),
message: message.clone(),
published_at,
};
let payload = serde_json::to_value(&event)
.map_err(|e| PubsubError::Rejected(format!("event serialize: {e}")))?;
let publish_ctx = PublishCtx {
app_id: cx.app_id,
origin_principal: cx.principal.as_ref().map(|p| p.user_id),
trigger_depth: cx.trigger_depth,
root_execution_id: cx.root_execution_id,
};
// Order (design notes §8): transactional outbox fan-out + commit
// FIRST; only then the best-effort realtime broadcast. If the
// fan-out fails, the publish throws and no broadcast happens. If
// the broadcast fails after a committed fan-out, trigger
// deliveries still happen and only SSE subscribers miss this
// event (no replay in v1.1.6).
//
// No matching triggers → 0 rows written, publish still succeeds.
self.repo
.fan_out_publish(publish_ctx, topic, payload)
.await?;
// Non-transactional, best-effort fan-out to in-process SSE
// subscribers. Run on a child task so a panicking broadcaster
// (or a future cluster-mode resolver fault) becomes a warn log,
// never a failed publish — the durable deliveries already
// committed above.
if let Some(realtime) = self.realtime.clone() {
let app_id = cx.app_id;
let topic_owned = topic.to_string();
let realtime_event = RealtimeEvent {
topic: topic_owned.clone(),
message,
published_at,
};
let handle = tokio::spawn(async move {
realtime.publish(app_id, &topic_owned, realtime_event).await;
});
if let Err(e) = handle.await {
tracing::warn!(error = %e, "realtime broadcast failed; publish unaffected");
}
}
Ok(())
}
async fn mint_subscriber_token(
&self,
cx: &SdkCallCx,
topics: Vec<String>,
ttl_seconds: Option<i64>,
) -> Result<String, PubsubError> {
// Anonymous (public-HTTP) scripts can't mint — that would bypass
// the token-minting authz boundary.
let Some(principal) = cx.principal.as_ref() else {
return Err(PubsubError::SubscriberToken(
"pubsub::subscriber_token: requires an authenticated principal \
(a script on a public route cannot mint tokens)"
.into(),
));
};
// Minting reuses the existing pub/sub publish capability (no new
// scope — the seven-scope commitment holds).
authz::require(
&*self.authz,
principal,
Capability::AppPubsubPublish(cx.app_id),
)
.await
.map_err(|_| PubsubError::Forbidden)?;
let (Some(topic_repo), Some(secrets)) = (self.topics.as_ref(), self.secrets.as_ref())
else {
return Err(PubsubError::Unavailable(
"subscriber tokens are not wired in".into(),
));
};
if topics.is_empty() {
return Err(PubsubError::SubscriberToken(
"pubsub::subscriber_token: topics list must not be empty".into(),
));
}
let ttl = ttl_seconds.unwrap_or(self.token_config.default_ttl);
if ttl < self.token_config.min_ttl || ttl > self.token_config.max_ttl {
return Err(PubsubError::SubscriberToken(format!(
"pubsub::subscriber_token: ttl_seconds must be between {} and {}",
self.token_config.min_ttl, self.token_config.max_ttl
)));
}
// Every requested topic must be registered as externally
// subscribable in this app — fail fast rather than mint a token
// that won't work.
for name in &topics {
let registered = topic_repo
.get(cx.app_id, name)
.await
.map_err(|e| PubsubError::Unavailable(e.to_string()))?;
if !registered.is_some_and(|t| t.external_subscribable) {
return Err(PubsubError::SubscriberToken(format!(
"pubsub::subscriber_token: topic {name} is not externally subscribable"
)));
}
}
let key = secrets
.get_or_create_signing_key(cx.app_id)
.await
.map_err(|e| PubsubError::Unavailable(e.to_string()))?;
let now = chrono::Utc::now().timestamp();
let claims = TokenClaims {
app_id: cx.app_id,
topics,
exp: now.saturating_add(ttl),
iat: now,
};
Ok(subscriber_token::sign(&key, &claims))
}
}
// ----------------------------------------------------------------------------
// Tests — in-memory PubsubRepo so unit tests don't need Postgres. The
// real transactional fan-out is covered against a live DB by the
// integration suite; the in-memory fake models the all-or-nothing
// commit so the rollback semantics can be asserted without a DB.
// ----------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::authz::{AuthzError, AuthzRepo};
use async_trait::async_trait;
use picloud_shared::{
topic_matches, AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, Principal,
RequestId, ScriptId, UserId,
};
use std::sync::Mutex;
/// In-memory pubsub repo. Holds a set of `(app, pattern)`
/// subscriptions and records the outbox rows a publish would write.
/// `fail_at` simulates a mid-fan-out INSERT failure: when set to
/// `Some(n)`, the n-th (1-indexed) matching row errors and NOTHING
/// is recorded — modelling the single-transaction rollback.
struct InMemoryPubsubRepo {
subs: Vec<(AppId, String)>,
written: Mutex<Vec<(AppId, String)>>,
fail_at: Option<usize>,
}
impl InMemoryPubsubRepo {
fn new(subs: Vec<(AppId, String)>) -> Self {
Self {
subs,
written: Mutex::new(Vec::new()),
fail_at: None,
}
}
fn written_count(&self) -> usize {
self.written.lock().unwrap().len()
}
}
#[async_trait]
impl PubsubRepo for InMemoryPubsubRepo {
async fn fan_out_publish(
&self,
ctx: PublishCtx,
topic: &str,
_event_payload: serde_json::Value,
) -> Result<u32, PubsubRepoError> {
let matches: Vec<&(AppId, String)> = self
.subs
.iter()
.filter(|(a, pat)| *a == ctx.app_id && topic_matches(pat, topic))
.collect();
let mut staged = Vec::new();
for (i, _) in matches.iter().enumerate() {
if self.fail_at == Some(i + 1) {
// Rollback: nothing was committed.
return Err(PubsubRepoError::Db(sqlx::Error::Protocol(
"simulated insert failure".into(),
)));
}
staged.push((ctx.app_id, topic.to_string()));
}
let n = staged.len();
self.written.lock().unwrap().extend(staged);
Ok(u32::try_from(n).unwrap_or(u32::MAX))
}
}
#[derive(Default)]
struct DenyingAuthzRepo;
#[async_trait]
impl AuthzRepo for DenyingAuthzRepo {
async fn membership(
&self,
_user_id: UserId,
_app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok(None)
}
}
#[derive(Default)]
struct EditorAuthzRepo;
#[async_trait]
impl AuthzRepo for EditorAuthzRepo {
async fn membership(
&self,
_user_id: UserId,
_app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok(Some(AppRole::Editor))
}
}
fn anon_cx(app_id: AppId) -> SdkCallCx {
SdkCallCx {
app_id,
script_id: ScriptId::new(),
principal: None,
execution_id: ExecutionId::new(),
request_id: RequestId::new(),
trigger_depth: 0,
root_execution_id: ExecutionId::new(),
is_dead_letter_handler: false,
event: None,
}
}
fn member_cx(app_id: AppId) -> SdkCallCx {
SdkCallCx {
principal: Some(Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Member,
scopes: None,
app_binding: None,
}),
..anon_cx(app_id)
}
}
fn svc(repo: Arc<dyn PubsubRepo>, authz: Arc<dyn AuthzRepo>) -> PubsubServiceImpl {
PubsubServiceImpl::new(repo, authz)
}
#[tokio::test]
async fn publish_writes_one_row_per_matching_trigger() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![
(app, "user.*".into()),
(app, "user.created".into()),
(app, "order.*".into()), // does not match
]));
let files = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
files
.publish_durable(&anon_cx(app), "user.created", serde_json::json!({"id": 1}))
.await
.unwrap();
// Two of the three subscriptions match "user.created".
assert_eq!(repo.written_count(), 2);
}
#[tokio::test]
async fn no_matching_trigger_succeeds_silently() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![(app, "order.*".into())]));
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
svc.publish_durable(&anon_cx(app), "user.created", serde_json::json!(1))
.await
.unwrap();
assert_eq!(repo.written_count(), 0);
}
#[tokio::test]
async fn empty_topic_rejected() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
let err = svc
.publish_durable(&anon_cx(app), " ", serde_json::json!(1))
.await
.unwrap_err();
assert!(matches!(err, PubsubError::EmptyTopic));
}
#[tokio::test]
async fn cross_app_isolation() {
let app_a = AppId::new();
let app_b = AppId::new();
// The only subscription belongs to app B.
let repo = Arc::new(InMemoryPubsubRepo::new(vec![(app_b, "*".into())]));
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
// App A publishes — app B's trigger must NOT fire.
svc.publish_durable(&anon_cx(app_a), "user.created", serde_json::json!(1))
.await
.unwrap();
assert_eq!(repo.written_count(), 0);
}
#[tokio::test]
async fn fan_out_is_transactional_all_or_nothing() {
let app = AppId::new();
let mut repo = InMemoryPubsubRepo::new(vec![
(app, "*".into()),
(app, "user.*".into()),
(app, "user.created".into()),
]);
repo.fail_at = Some(3); // fail on the 3rd matching insert
let repo = Arc::new(repo);
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
let err = svc
.publish_durable(&anon_cx(app), "user.created", serde_json::json!(1))
.await
.unwrap_err();
assert!(matches!(err, PubsubError::Unavailable(_)));
// Rollback: no partial fan-out survived.
assert_eq!(repo.written_count(), 0);
}
#[tokio::test]
async fn anonymous_cx_skips_authz() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
// No principal → no authz check even with a denying repo.
svc.publish_durable(&anon_cx(app), "t", serde_json::json!(1))
.await
.unwrap();
}
#[tokio::test]
async fn member_without_role_is_forbidden() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
let err = svc
.publish_durable(&member_cx(app), "t", serde_json::json!(1))
.await
.unwrap_err();
assert!(matches!(err, PubsubError::Forbidden));
}
#[tokio::test]
async fn member_with_editor_role_allowed() {
let app = AppId::new();
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
let svc = svc(repo, Arc::new(EditorAuthzRepo));
svc.publish_durable(&member_cx(app), "t", serde_json::json!(1))
.await
.unwrap();
}
// ------------------------------------------------------------------
// v1.1.6 realtime broadcast + subscriber-token minting
// ------------------------------------------------------------------
use crate::app_secrets_repo::{AppSecretsRepo, AppSecretsRepoError};
use crate::topic_repo::{Topic, TopicAuthMode, TopicRepo, TopicRepoError};
use picloud_orchestrator_core::InProcessBroadcaster;
use picloud_shared::{RealtimeBroadcaster, RealtimeEvent};
/// Topic repo fake: returns the configured topics as registered +
/// externally-subscribable (unless absent).
struct FakeTopicRepo(Vec<String>);
#[async_trait]
impl TopicRepo for FakeTopicRepo {
async fn create(
&self,
_: AppId,
_: &str,
_: bool,
_: TopicAuthMode,
) -> Result<Topic, TopicRepoError> {
unimplemented!()
}
async fn list(&self, _: AppId) -> Result<Vec<Topic>, TopicRepoError> {
unimplemented!()
}
async fn get(&self, _: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
Ok(self.0.iter().any(|t| t == name).then(|| Topic {
name: name.to_string(),
external_subscribable: true,
auth_mode: TopicAuthMode::Token,
created_at: chrono::Utc::now(),
updated_at: chrono::Utc::now(),
}))
}
async fn update(
&self,
_: AppId,
_: &str,
_: Option<bool>,
_: Option<TopicAuthMode>,
) -> Result<Option<Topic>, TopicRepoError> {
unimplemented!()
}
async fn delete(&self, _: AppId, _: &str) -> Result<bool, TopicRepoError> {
unimplemented!()
}
}
#[derive(Default)]
struct FakeSecrets;
#[async_trait]
impl AppSecretsRepo for FakeSecrets {
async fn get_or_create_signing_key(
&self,
_: AppId,
) -> Result<Vec<u8>, AppSecretsRepoError> {
Ok(vec![42u8; 32])
}
async fn signing_key(&self, _: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
Ok(Some(vec![42u8; 32]))
}
}
/// Broadcaster that panics on publish — proves a broadcast fault
/// can't fail the publish.
struct PanicBroadcaster;
#[async_trait]
impl RealtimeBroadcaster for PanicBroadcaster {
async fn subscribe(
&self,
_: AppId,
_: &str,
) -> Result<tokio::sync::broadcast::Receiver<RealtimeEvent>, picloud_shared::BroadcasterError>
{
unimplemented!()
}
async fn publish(&self, _: AppId, _: &str, _: RealtimeEvent) {
panic!("boom");
}
async fn drop_topic(&self, _: AppId, _: &str) {}
}
fn realtime_svc(
repo: Arc<dyn PubsubRepo>,
broadcaster: Arc<dyn RealtimeBroadcaster>,
topics: Vec<String>,
) -> PubsubServiceImpl {
PubsubServiceImpl::new(repo, Arc::new(EditorAuthzRepo)).with_realtime(
broadcaster,
Arc::new(FakeTopicRepo(topics)),
Arc::new(FakeSecrets),
SubscriberTokenConfig::conservative(),
)
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn publish_broadcasts_to_in_process_subscribers() {
let app = AppId::new();
let broadcaster = Arc::new(InProcessBroadcaster::new(16));
let mut rx = broadcaster.subscribe(app, "chat").await.unwrap();
let svc = realtime_svc(
Arc::new(InMemoryPubsubRepo::new(vec![])),
broadcaster,
vec![],
);
svc.publish_durable(&anon_cx(app), "chat", serde_json::json!({ "hi": 1 }))
.await
.unwrap();
let ev = rx.recv().await.unwrap();
assert_eq!(ev.topic, "chat");
assert_eq!(ev.message, serde_json::json!({ "hi": 1 }));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn panicking_broadcaster_does_not_fail_publish() {
let app = AppId::new();
let svc = realtime_svc(
Arc::new(InMemoryPubsubRepo::new(vec![])),
Arc::new(PanicBroadcaster),
vec![],
);
// The outbox fan-out committed; the broadcast panic is swallowed.
svc.publish_durable(&anon_cx(app), "chat", serde_json::json!(1))
.await
.expect("publish must succeed despite broadcast panic");
}
fn mint_svc(topics: Vec<String>) -> PubsubServiceImpl {
realtime_svc(
Arc::new(InMemoryPubsubRepo::new(vec![])),
Arc::new(picloud_shared::NoopRealtimeBroadcaster),
topics,
)
}
#[tokio::test]
async fn mint_returns_token_scoped_to_topics() {
let app = AppId::new();
let svc = mint_svc(vec!["chat".into(), "notify".into()]);
let token = svc
.mint_subscriber_token(
&member_cx(app),
vec!["chat".into(), "notify".into()],
Some(120),
)
.await
.unwrap();
// Verify with the fake key; claims carry the topics + expiry.
let claims = subscriber_token::verify(&[42u8; 32], &token, chrono::Utc::now().timestamp())
.expect("token verifies");
assert_eq!(claims.app_id, app);
assert!(claims.allows_topic("chat") && claims.allows_topic("notify"));
assert!(claims.exp > claims.iat);
}
#[tokio::test]
async fn mint_anonymous_principal_throws() {
let app = AppId::new();
let svc = mint_svc(vec!["chat".into()]);
let err = svc
.mint_subscriber_token(&anon_cx(app), vec!["chat".into()], None)
.await
.unwrap_err();
assert!(matches!(err, PubsubError::SubscriberToken(_)));
}
#[tokio::test]
async fn mint_empty_topics_throws() {
let app = AppId::new();
let svc = mint_svc(vec!["chat".into()]);
let err = svc
.mint_subscriber_token(&member_cx(app), vec![], None)
.await
.unwrap_err();
assert!(matches!(err, PubsubError::SubscriberToken(_)));
}
#[tokio::test]
async fn mint_ttl_below_min_and_above_max_throw() {
let app = AppId::new();
let svc = mint_svc(vec!["chat".into()]);
for bad in [Some(5), Some(90_000)] {
let err = svc
.mint_subscriber_token(&member_cx(app), vec!["chat".into()], bad)
.await
.unwrap_err();
assert!(
matches!(err, PubsubError::SubscriberToken(_)),
"ttl {bad:?}"
);
}
}
#[tokio::test]
async fn mint_unregistered_topic_throws_with_message() {
let app = AppId::new();
// "chat" registered; "secret" is not.
let svc = mint_svc(vec!["chat".into()]);
let err = svc
.mint_subscriber_token(&member_cx(app), vec!["chat".into(), "secret".into()], None)
.await
.unwrap_err();
match err {
PubsubError::SubscriberToken(msg) => {
assert!(
msg.contains("topic secret is not externally subscribable"),
"got: {msg}"
);
}
other => panic!("expected SubscriberToken, got {other:?}"),
}
}
}

View File

@@ -0,0 +1,338 @@
//! `RealtimeAuthorityImpl` — the DB-backed SSE subscribe gate (v1.1.6).
//!
//! Backs the [`picloud_shared::RealtimeAuthority`] trait the SSE handler
//! in orchestrator-core calls. All `topics`-table reads and signing-key
//! material stay inside this impl so the data-plane crate never touches
//! the key.
//!
//! Verdict mapping (see [`SubscribeDenied`]):
//! * topic missing OR not externally subscribable → `NotFound` (404).
//! Both collapse to 404 so the endpoint can't probe internal topics.
//! * `auth_mode = 'public'` → allow.
//! * `auth_mode = 'token'` → verify the HMAC token (present, signed by
//! this app's key, unexpired, scoped to this topic) → allow, else
//! `Unauthorized` (401, generic — never says which check failed).
//!
//! Signing keys never change in v1.1.6 (no rotation API), so a small
//! in-memory cache avoids a per-subscribe DB read once an app's key has
//! been seen. The cache is purely an optimization — a cold miss reads
//! the row.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use async_trait::async_trait;
use picloud_shared::{subscriber_token, AppId, RealtimeAuthority, SubscribeDenied};
use crate::app_secrets_repo::AppSecretsRepo;
use crate::topic_repo::{TopicAuthMode, TopicRepo};
pub struct RealtimeAuthorityImpl {
topics: Arc<dyn TopicRepo>,
secrets: Arc<dyn AppSecretsRepo>,
key_cache: Mutex<HashMap<AppId, Vec<u8>>>,
}
impl RealtimeAuthorityImpl {
#[must_use]
pub fn new(topics: Arc<dyn TopicRepo>, secrets: Arc<dyn AppSecretsRepo>) -> Self {
Self {
topics,
secrets,
key_cache: Mutex::new(HashMap::new()),
}
}
/// Fetch the app's signing key, consulting the cache first. Returns
/// `None` when the app has no key (no token ever minted) — which the
/// caller maps to `Unauthorized`.
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, SubscribeDenied> {
if let Ok(cache) = self.key_cache.lock() {
if let Some(k) = cache.get(&app_id) {
return Ok(Some(k.clone()));
}
}
let key = self
.secrets
.signing_key(app_id)
.await
.map_err(|e| SubscribeDenied::Backend(e.to_string()))?;
if let Some(ref k) = key {
if let Ok(mut cache) = self.key_cache.lock() {
cache.insert(app_id, k.clone());
}
}
Ok(key)
}
}
#[async_trait]
impl RealtimeAuthority for RealtimeAuthorityImpl {
async fn authorize_subscribe(
&self,
app_id: AppId,
topic: &str,
token: Option<&str>,
) -> Result<(), SubscribeDenied> {
let registered = self
.topics
.get(app_id, topic)
.await
.map_err(|e| SubscribeDenied::Backend(e.to_string()))?;
// Missing topic AND internal-only topic both 404 — don't leak
// which internal topics exist.
let Some(t) = registered.filter(|t| t.external_subscribable) else {
return Err(SubscribeDenied::NotFound);
};
match t.auth_mode {
TopicAuthMode::Public => Ok(()),
TopicAuthMode::Token => {
let token = token.ok_or(SubscribeDenied::Unauthorized)?;
let key = self
.signing_key(app_id)
.await?
.ok_or(SubscribeDenied::Unauthorized)?;
let now = chrono::Utc::now().timestamp();
let claims = subscriber_token::verify(&key, token, now)
.map_err(|_| SubscribeDenied::Unauthorized)?;
// Per-app key already makes a cross-app token fail the
// signature check; this is belt-and-suspenders.
if claims.app_id != app_id || !claims.allows_topic(topic) {
return Err(SubscribeDenied::Unauthorized);
}
Ok(())
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::app_secrets_repo::AppSecretsRepoError;
use crate::topic_repo::{Topic, TopicRepoError};
use chrono::Utc;
use picloud_shared::subscriber_token::{sign, TokenClaims};
struct FakeTopics(Vec<(AppId, Topic)>);
#[async_trait]
impl TopicRepo for FakeTopics {
async fn create(
&self,
_: AppId,
_: &str,
_: bool,
_: TopicAuthMode,
) -> Result<Topic, TopicRepoError> {
unimplemented!()
}
async fn list(&self, _: AppId) -> Result<Vec<Topic>, TopicRepoError> {
unimplemented!()
}
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
Ok(self
.0
.iter()
.find(|(a, t)| *a == app_id && t.name == name)
.map(|(_, t)| t.clone()))
}
async fn update(
&self,
_: AppId,
_: &str,
_: Option<bool>,
_: Option<TopicAuthMode>,
) -> Result<Option<Topic>, TopicRepoError> {
unimplemented!()
}
async fn delete(&self, _: AppId, _: &str) -> Result<bool, TopicRepoError> {
unimplemented!()
}
}
struct FakeSecrets(AppId, Vec<u8>);
#[async_trait]
impl AppSecretsRepo for FakeSecrets {
async fn get_or_create_signing_key(
&self,
_: AppId,
) -> Result<Vec<u8>, AppSecretsRepoError> {
Ok(self.1.clone())
}
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
Ok((app_id == self.0).then(|| self.1.clone()))
}
}
fn topic(name: &str, external: bool, mode: TopicAuthMode) -> Topic {
Topic {
name: name.to_string(),
external_subscribable: external,
auth_mode: mode,
created_at: Utc::now(),
updated_at: Utc::now(),
}
}
fn authority(
topics: Vec<(AppId, Topic)>,
key_app: AppId,
key: Vec<u8>,
) -> RealtimeAuthorityImpl {
RealtimeAuthorityImpl::new(
Arc::new(FakeTopics(topics)),
Arc::new(FakeSecrets(key_app, key)),
)
}
#[tokio::test]
async fn missing_topic_is_not_found() {
let app = AppId::new();
let a = authority(vec![], app, vec![0u8; 32]);
assert_eq!(
a.authorize_subscribe(app, "ghost", None).await,
Err(SubscribeDenied::NotFound)
);
}
#[tokio::test]
async fn internal_only_topic_is_not_found() {
let app = AppId::new();
let a = authority(
vec![(app, topic("internal", false, TopicAuthMode::Public))],
app,
vec![0u8; 32],
);
assert_eq!(
a.authorize_subscribe(app, "internal", None).await,
Err(SubscribeDenied::NotFound)
);
}
#[tokio::test]
async fn public_topic_allows_without_token() {
let app = AppId::new();
let a = authority(
vec![(app, topic("news", true, TopicAuthMode::Public))],
app,
vec![0u8; 32],
);
assert!(a.authorize_subscribe(app, "news", None).await.is_ok());
}
#[tokio::test]
async fn token_topic_without_token_is_unauthorized() {
let app = AppId::new();
let a = authority(
vec![(app, topic("chat", true, TopicAuthMode::Token))],
app,
vec![7u8; 32],
);
assert_eq!(
a.authorize_subscribe(app, "chat", None).await,
Err(SubscribeDenied::Unauthorized)
);
}
#[tokio::test]
async fn token_topic_with_valid_token_allows() {
let app = AppId::new();
let key = vec![9u8; 32];
let a = authority(
vec![(app, topic("chat", true, TopicAuthMode::Token))],
app,
key.clone(),
);
let token = sign(
&key,
&TokenClaims {
app_id: app,
topics: vec!["chat".into()],
iat: Utc::now().timestamp(),
exp: Utc::now().timestamp() + 60,
},
);
assert!(a
.authorize_subscribe(app, "chat", Some(&token))
.await
.is_ok());
}
#[tokio::test]
async fn token_for_other_topic_is_unauthorized() {
let app = AppId::new();
let key = vec![9u8; 32];
let a = authority(
vec![(app, topic("chat", true, TopicAuthMode::Token))],
app,
key.clone(),
);
let token = sign(
&key,
&TokenClaims {
app_id: app,
topics: vec!["other".into()],
iat: Utc::now().timestamp(),
exp: Utc::now().timestamp() + 60,
},
);
assert_eq!(
a.authorize_subscribe(app, "chat", Some(&token)).await,
Err(SubscribeDenied::Unauthorized)
);
}
#[tokio::test]
async fn expired_token_is_unauthorized() {
let app = AppId::new();
let key = vec![9u8; 32];
let a = authority(
vec![(app, topic("chat", true, TopicAuthMode::Token))],
app,
key.clone(),
);
let token = sign(
&key,
&TokenClaims {
app_id: app,
topics: vec!["chat".into()],
iat: Utc::now().timestamp() - 120,
exp: Utc::now().timestamp() - 60,
},
);
assert_eq!(
a.authorize_subscribe(app, "chat", Some(&token)).await,
Err(SubscribeDenied::Unauthorized)
);
}
#[tokio::test]
async fn token_signed_by_other_app_key_is_unauthorized() {
let app_a = AppId::new();
let app_b = AppId::new();
let key_a = vec![1u8; 32];
let key_b = vec![2u8; 32];
// Authority for app B; its key is key_b.
let a = authority(
vec![(app_b, topic("chat", true, TopicAuthMode::Token))],
app_b,
key_b,
);
// Token signed by app A's key, claiming app A.
let token = sign(
&key_a,
&TokenClaims {
app_id: app_a,
topics: vec!["chat".into()],
iat: Utc::now().timestamp(),
exp: Utc::now().timestamp() + 60,
},
);
assert_eq!(
a.authorize_subscribe(app_b, "chat", Some(&token)).await,
Err(SubscribeDenied::Unauthorized)
);
}
}

View File

@@ -0,0 +1,232 @@
//! `/api/v1/admin/apps/{id}/secrets*` — secrets admin endpoints
//! (v1.1.7).
//!
//! * `GET /apps/{id}/secrets` — list names + updated_at
//! (NEVER values).
//! * `POST /apps/{id}/secrets` — set/overwrite a secret.
//! * `DELETE /apps/{id}/secrets/{name}` — delete a secret.
//!
//! Set/delete are gated by `AppSecretsWrite` (→ `script:write`); list by
//! `AppSecretsRead` (→ `script:read`). The list surface deliberately
//! returns only names + timestamps — the dashboard never receives
//! plaintext. Values are encrypted with the process master key before
//! they touch the database (same envelope as the script `secrets::set`).
use std::sync::Arc;
use axum::extract::{Path, Query, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Json, Response};
use axum::routing::get;
use axum::{Extension, Router};
use picloud_shared::{validate_secret_name, AppId, MasterKey, Principal, SecretsError};
use serde::Deserialize;
use serde_json::json;
use crate::app_repo::AppRepository;
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
use crate::secrets_repo::{SecretsRepo, SecretsRepoError};
use crate::secrets_service::seal;
#[derive(Clone)]
pub struct SecretsState {
pub repo: Arc<dyn SecretsRepo>,
pub apps: Arc<dyn AppRepository>,
pub authz: Arc<dyn AuthzRepo>,
pub master_key: MasterKey,
pub max_value_bytes: usize,
}
pub fn secrets_router(state: SecretsState) -> Router {
Router::new()
.route("/apps/{app_id}/secrets", get(list_secrets).post(set_secret))
.route(
"/apps/{app_id}/secrets/{name}",
axum::routing::delete(delete_secret),
)
.with_state(state)
}
#[derive(Debug, Deserialize)]
pub struct ListQuery {
#[serde(default)]
pub cursor: Option<String>,
#[serde(default)]
pub limit: Option<u32>,
}
#[derive(Debug, serde::Serialize)]
struct SecretItem {
name: String,
updated_at: chrono::DateTime<chrono::Utc>,
}
#[derive(Debug, serde::Serialize)]
struct ListSecretsResponse {
secrets: Vec<SecretItem>,
next_cursor: Option<String>,
}
async fn list_secrets(
State(s): State<SecretsState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Query(q): Query<ListQuery>,
) -> Result<Json<ListSecretsResponse>, SecretsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppSecretsRead(app_id),
)
.await?;
let page = s
.repo
.list_meta(app_id, q.cursor.as_deref(), q.limit.unwrap_or(0))
.await?;
Ok(Json(ListSecretsResponse {
secrets: page
.items
.into_iter()
.map(|m| SecretItem {
name: m.name,
updated_at: m.updated_at,
})
.collect(),
next_cursor: page.next_cursor,
}))
}
#[derive(Debug, Deserialize)]
pub struct SetSecretRequest {
pub name: String,
/// Any JSON value — the dashboard sends a single-line string, but
/// maps/arrays/numbers round-trip too (matching `secrets::set`).
pub value: serde_json::Value,
}
async fn set_secret(
State(s): State<SecretsState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Json(input): Json<SetSecretRequest>,
) -> Result<StatusCode, SecretsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppSecretsWrite(app_id),
)
.await?;
validate_secret_name(&input.name)?;
let (ciphertext, nonce) = seal(&s.master_key, &input.value, s.max_value_bytes)?;
s.repo.set(app_id, &input.name, &ciphertext, &nonce).await?;
Ok(StatusCode::NO_CONTENT)
}
async fn delete_secret(
State(s): State<SecretsState>,
Extension(principal): Extension<Principal>,
Path((app_id, name)): Path<(AppId, String)>,
) -> Result<StatusCode, SecretsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppSecretsWrite(app_id),
)
.await?;
if !s.repo.delete(app_id, &name).await? {
return Err(SecretsApiError::NotFound);
}
Ok(StatusCode::NO_CONTENT)
}
async fn ensure_app_exists(apps: &dyn AppRepository, app_id: AppId) -> Result<(), SecretsApiError> {
apps.get_by_id(app_id)
.await
.map_err(|e| SecretsApiError::Backend(e.to_string()))?
.ok_or(SecretsApiError::AppNotFound)?;
Ok(())
}
#[derive(Debug, thiserror::Error)]
pub enum SecretsApiError {
#[error("app not found")]
AppNotFound,
#[error("secret not found")]
NotFound,
#[error("invalid request: {0}")]
Invalid(String),
#[error("forbidden")]
Forbidden,
#[error("authorization repo error: {0}")]
AuthzRepo(String),
#[error("secrets backend: {0}")]
Backend(String),
}
impl From<AuthzDenied> for SecretsApiError {
fn from(d: AuthzDenied) -> Self {
match d {
AuthzDenied::Denied => Self::Forbidden,
AuthzDenied::Repo(e) => Self::AuthzRepo(e.to_string()),
}
}
}
impl From<AuthzError> for SecretsApiError {
fn from(e: AuthzError) -> Self {
Self::AuthzRepo(e.to_string())
}
}
impl From<SecretsRepoError> for SecretsApiError {
fn from(e: SecretsRepoError) -> Self {
match e {
SecretsRepoError::InvalidCursor => Self::Invalid("invalid pagination cursor".into()),
SecretsRepoError::Db(e) => Self::Backend(e.to_string()),
}
}
}
impl From<SecretsError> for SecretsApiError {
fn from(e: SecretsError) -> Self {
match e {
SecretsError::InvalidName(m) => Self::Invalid(m),
SecretsError::TooLarge { .. } => Self::Invalid(e.to_string()),
SecretsError::Forbidden => Self::Forbidden,
other => Self::Backend(other.to_string()),
}
}
}
impl IntoResponse for SecretsApiError {
fn into_response(self) -> Response {
let (status, body) = match &self {
Self::AppNotFound | Self::NotFound => {
(StatusCode::NOT_FOUND, json!({ "error": self.to_string() }))
}
Self::Invalid(_) => (
StatusCode::UNPROCESSABLE_ENTITY,
json!({ "error": self.to_string() }),
),
Self::Forbidden => (StatusCode::FORBIDDEN, json!({ "error": self.to_string() })),
Self::AuthzRepo(e) => {
tracing::error!(error = %e, "secrets admin authz repo error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
Self::Backend(e) => {
tracing::error!(error = %e, "secrets admin backend error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
};
(status, Json(body)).into_response()
}
}

View File

@@ -0,0 +1,246 @@
//! Low-level Postgres CRUD over `secrets`. Storage-only: it moves
//! opaque ciphertext + nonce blobs in and out. Encryption, JSON
//! encoding, authorization, name validation, and the value-size cap all
//! live one layer up in `SecretsServiceImpl` / `secrets_api`.
use async_trait::async_trait;
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
use base64::Engine as _;
use chrono::{DateTime, Utc};
use picloud_shared::AppId;
use sqlx::PgPool;
#[derive(Debug, thiserror::Error)]
pub enum SecretsRepoError {
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
#[error("invalid pagination cursor")]
InvalidCursor,
}
/// An encrypted secret as it lives on disk: ciphertext (auth tag
/// appended) plus the nonce it was sealed with.
#[derive(Debug, Clone)]
pub struct StoredSecret {
pub encrypted_value: Vec<u8>,
pub nonce: Vec<u8>,
}
/// Admin-surface metadata for one secret. Values are never returned —
/// only the name and the last-modified timestamp.
#[derive(Debug, Clone)]
pub struct SecretMeta {
pub name: String,
pub updated_at: DateTime<Utc>,
}
/// One page of names (SDK `list`).
#[derive(Debug, Clone)]
pub struct SecretsNamePage {
pub names: Vec<String>,
pub next_cursor: Option<String>,
}
/// One page of name + updated_at (admin `GET`).
#[derive(Debug, Clone)]
pub struct SecretsMetaPage {
pub items: Vec<SecretMeta>,
pub next_cursor: Option<String>,
}
/// Repo surface. Exposed as a trait so the service unit tests can
/// substitute an in-memory backing without Postgres.
#[async_trait]
pub trait SecretsRepo: Send + Sync {
async fn get(
&self,
app_id: AppId,
name: &str,
) -> Result<Option<StoredSecret>, SecretsRepoError>;
/// Upsert (overwrite if present).
async fn set(
&self,
app_id: AppId,
name: &str,
encrypted_value: &[u8],
nonce: &[u8],
) -> Result<(), SecretsRepoError>;
/// Delete; returns whether a row was present.
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, SecretsRepoError>;
/// Names only — the SDK `list` surface.
async fn list_names(
&self,
app_id: AppId,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsNamePage, SecretsRepoError>;
/// Name + updated_at — the admin `GET` surface.
async fn list_meta(
&self,
app_id: AppId,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsMetaPage, SecretsRepoError>;
}
pub struct PostgresSecretsRepo {
pool: PgPool,
}
impl PostgresSecretsRepo {
#[must_use]
pub fn new(pool: PgPool) -> Self {
Self { pool }
}
}
const SECRETS_LIST_MAX_LIMIT: u32 = 1_000;
const SECRETS_LIST_DEFAULT_LIMIT: u32 = 100;
fn clamp_limit(limit: u32) -> u32 {
if limit == 0 {
SECRETS_LIST_DEFAULT_LIMIT
} else {
limit.min(SECRETS_LIST_MAX_LIMIT)
}
}
/// Opaque keyset cursor: base64url of the last `name` returned.
pub(crate) fn encode_cursor(last_name: &str) -> String {
URL_SAFE_NO_PAD.encode(last_name.as_bytes())
}
pub(crate) fn decode_cursor(cursor: &str) -> Result<String, SecretsRepoError> {
let bytes = URL_SAFE_NO_PAD
.decode(cursor)
.map_err(|_| SecretsRepoError::InvalidCursor)?;
String::from_utf8(bytes).map_err(|_| SecretsRepoError::InvalidCursor)
}
#[async_trait]
impl SecretsRepo for PostgresSecretsRepo {
async fn get(
&self,
app_id: AppId,
name: &str,
) -> Result<Option<StoredSecret>, SecretsRepoError> {
let row: Option<(Vec<u8>, Vec<u8>)> = sqlx::query_as(
"SELECT encrypted_value, nonce FROM secrets WHERE app_id = $1 AND name = $2",
)
.bind(app_id.into_inner())
.bind(name)
.fetch_optional(&self.pool)
.await?;
Ok(row.map(|(encrypted_value, nonce)| StoredSecret {
encrypted_value,
nonce,
}))
}
async fn set(
&self,
app_id: AppId,
name: &str,
encrypted_value: &[u8],
nonce: &[u8],
) -> Result<(), SecretsRepoError> {
sqlx::query(
"INSERT INTO secrets (app_id, name, encrypted_value, nonce) \
VALUES ($1, $2, $3, $4) \
ON CONFLICT (app_id, name) DO UPDATE \
SET encrypted_value = EXCLUDED.encrypted_value, \
nonce = EXCLUDED.nonce, \
updated_at = NOW()",
)
.bind(app_id.into_inner())
.bind(name)
.bind(encrypted_value)
.bind(nonce)
.execute(&self.pool)
.await?;
Ok(())
}
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, SecretsRepoError> {
let res = sqlx::query("DELETE FROM secrets WHERE app_id = $1 AND name = $2")
.bind(app_id.into_inner())
.bind(name)
.execute(&self.pool)
.await?;
Ok(res.rows_affected() > 0)
}
async fn list_names(
&self,
app_id: AppId,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsNamePage, SecretsRepoError> {
let limit = clamp_limit(limit);
let last_name = match cursor {
Some(c) => Some(decode_cursor(c)?),
None => None,
};
let take = i64::from(limit) + 1;
let rows: Vec<(String,)> = sqlx::query_as(
"SELECT name FROM secrets \
WHERE app_id = $1 AND ($2::text IS NULL OR name > $2) \
ORDER BY name ASC LIMIT $3",
)
.bind(app_id.into_inner())
.bind(last_name.as_deref())
.bind(take)
.fetch_all(&self.pool)
.await?;
let mut names: Vec<String> = rows.into_iter().map(|(n,)| n).collect();
let next_cursor = if names.len() > limit as usize {
names.truncate(limit as usize);
names.last().map(|n| encode_cursor(n))
} else {
None
};
Ok(SecretsNamePage { names, next_cursor })
}
async fn list_meta(
&self,
app_id: AppId,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsMetaPage, SecretsRepoError> {
let limit = clamp_limit(limit);
let last_name = match cursor {
Some(c) => Some(decode_cursor(c)?),
None => None,
};
let take = i64::from(limit) + 1;
let rows: Vec<(String, DateTime<Utc>)> = sqlx::query_as(
"SELECT name, updated_at FROM secrets \
WHERE app_id = $1 AND ($2::text IS NULL OR name > $2) \
ORDER BY name ASC LIMIT $3",
)
.bind(app_id.into_inner())
.bind(last_name.as_deref())
.bind(take)
.fetch_all(&self.pool)
.await?;
let mut items: Vec<SecretMeta> = rows
.into_iter()
.map(|(name, updated_at)| SecretMeta { name, updated_at })
.collect();
let next_cursor = if items.len() > limit as usize {
items.truncate(limit as usize);
items.last().map(|m| encode_cursor(&m.name))
} else {
None
};
Ok(SecretsMetaPage { items, next_cursor })
}
}

View File

@@ -0,0 +1,574 @@
//! `SecretsServiceImpl` — wires the `SecretsRepo` underneath the
//! `picloud_shared::SecretsService` trait that scripts see via the Rhai
//! bridge.
//!
//! Layers added here (vs the raw repo):
//!
//! 1. Name validation (non-empty, ≤255 bytes) at the SDK boundary.
//! 2. **Script-as-gate authz**: when `cx.principal.is_some()` we run
//! `authz::require(...)`; when it's `None` (public unauthenticated
//! HTTP) we skip the check. Cross-app isolation is unaffected — every
//! query is keyed by `cx.app_id`, never an argument.
//! 3. **JSON ⇄ ciphertext**: `set` serializes the value to JSON bytes,
//! enforces the per-secret size cap, and AES-256-GCM-seals it; `get`
//! decrypts and deserializes back to the same JSON shape (a String
//! round-trips to a String, not a JSON-quoted `"\"…\""`).
//!
//! Deliberately **no `ServiceEvent` emission** — secret writes do not
//! fire triggers (footgun avoidance; see `docs/sdk-shape.md` + the
//! v1.1.7 brief §2).
use std::sync::Arc;
use async_trait::async_trait;
use picloud_shared::{
crypto, validate_secret_name, MasterKey, SdkCallCx, SecretsError, SecretsListPage,
SecretsService,
};
use crate::authz::{self, AuthzRepo, Capability};
use crate::secrets_repo::{SecretsRepo, SecretsRepoError, StoredSecret};
/// Default per-secret plaintext cap (64 KB). Override with
/// `PICLOUD_SECRET_MAX_VALUE_BYTES`.
pub const DEFAULT_SECRET_MAX_VALUE_BYTES: usize = 64 * 1024;
/// Process config for the secrets service.
#[derive(Debug, Clone, Copy)]
pub struct SecretsConfig {
/// Maximum size of the JSON-encoded plaintext, in bytes.
pub max_value_bytes: usize,
}
impl SecretsConfig {
#[must_use]
pub const fn conservative() -> Self {
Self {
max_value_bytes: DEFAULT_SECRET_MAX_VALUE_BYTES,
}
}
/// Read `PICLOUD_SECRET_MAX_VALUE_BYTES`; invalid values are ignored
/// with a warning (keeps the conservative default).
#[must_use]
pub fn from_env() -> Self {
let mut c = Self::conservative();
if let Ok(v) = std::env::var("PICLOUD_SECRET_MAX_VALUE_BYTES") {
match v.trim().parse::<usize>() {
Ok(n) if n > 0 => c.max_value_bytes = n,
_ => tracing::warn!(
value = %v,
"ignoring invalid PICLOUD_SECRET_MAX_VALUE_BYTES (want a positive integer)"
),
}
}
c
}
}
impl Default for SecretsConfig {
fn default() -> Self {
Self::conservative()
}
}
/// Serialize + size-check + encrypt a value into `(ciphertext, nonce)`.
///
/// # Errors
///
/// [`SecretsError::TooLarge`] when the encoded plaintext exceeds
/// `max_value_bytes`; [`SecretsError::Backend`] on a serialization
/// failure (should not happen for a `serde_json::Value`).
pub fn seal(
master_key: &MasterKey,
value: &serde_json::Value,
max_value_bytes: usize,
) -> Result<(Vec<u8>, [u8; crypto::NONCE_LEN]), SecretsError> {
let plaintext = serde_json::to_vec(value)
.map_err(|e| SecretsError::Backend(format!("encode secret value: {e}")))?;
if plaintext.len() > max_value_bytes {
return Err(SecretsError::TooLarge {
limit: max_value_bytes,
actual: plaintext.len(),
});
}
let enc = crypto::encrypt(&plaintext, master_key.as_bytes());
Ok((enc.ciphertext, enc.nonce))
}
/// Decrypt + deserialize a stored secret back to its JSON value.
///
/// # Errors
///
/// [`SecretsError::Corrupted`] when decryption or JSON decoding fails.
pub fn open(
master_key: &MasterKey,
stored: &StoredSecret,
) -> Result<serde_json::Value, SecretsError> {
let plaintext = crypto::decrypt(
&stored.encrypted_value,
&stored.nonce,
master_key.as_bytes(),
)
.map_err(|_| SecretsError::Corrupted)?;
serde_json::from_slice(&plaintext).map_err(|_| SecretsError::Corrupted)
}
pub struct SecretsServiceImpl {
repo: Arc<dyn SecretsRepo>,
authz: Arc<dyn AuthzRepo>,
master_key: MasterKey,
max_value_bytes: usize,
}
impl SecretsServiceImpl {
#[must_use]
pub fn new(
repo: Arc<dyn SecretsRepo>,
authz: Arc<dyn AuthzRepo>,
master_key: MasterKey,
config: SecretsConfig,
) -> Self {
Self {
repo,
authz,
master_key,
max_value_bytes: config.max_value_bytes,
}
}
async fn check_read(&self, cx: &SdkCallCx) -> Result<(), SecretsError> {
if let Some(ref principal) = cx.principal {
authz::require(
&*self.authz,
principal,
Capability::AppSecretsRead(cx.app_id),
)
.await
.map_err(|_| SecretsError::Forbidden)?;
}
Ok(())
}
async fn check_write(&self, cx: &SdkCallCx) -> Result<(), SecretsError> {
if let Some(ref principal) = cx.principal {
authz::require(
&*self.authz,
principal,
Capability::AppSecretsWrite(cx.app_id),
)
.await
.map_err(|_| SecretsError::Forbidden)?;
}
Ok(())
}
}
impl From<SecretsRepoError> for SecretsError {
fn from(e: SecretsRepoError) -> Self {
Self::Backend(e.to_string())
}
}
#[async_trait]
impl SecretsService for SecretsServiceImpl {
async fn get(
&self,
cx: &SdkCallCx,
name: &str,
) -> Result<Option<serde_json::Value>, SecretsError> {
validate_secret_name(name)?;
self.check_read(cx).await?;
let Some(stored) = self.repo.get(cx.app_id, name).await? else {
return Ok(None);
};
match open(&self.master_key, &stored) {
Ok(value) => Ok(Some(value)),
Err(e) => {
// A decrypt failure is operationally significant — surface
// the affected (app_id, name) so an operator can find the
// bad row, but never log the ciphertext or key material.
tracing::error!(
app_id = %cx.app_id,
secret = %name,
"secret could not be decrypted (corrupted row or master-key mismatch)"
);
Err(e)
}
}
}
async fn set(
&self,
cx: &SdkCallCx,
name: &str,
value: serde_json::Value,
) -> Result<(), SecretsError> {
validate_secret_name(name)?;
self.check_write(cx).await?;
let (ciphertext, nonce) = seal(&self.master_key, &value, self.max_value_bytes)?;
self.repo.set(cx.app_id, name, &ciphertext, &nonce).await?;
Ok(())
}
async fn delete(&self, cx: &SdkCallCx, name: &str) -> Result<bool, SecretsError> {
validate_secret_name(name)?;
self.check_write(cx).await?;
Ok(self.repo.delete(cx.app_id, name).await?)
}
async fn list(
&self,
cx: &SdkCallCx,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsListPage, SecretsError> {
self.check_read(cx).await?;
let page = self.repo.list_names(cx.app_id, cursor, limit).await?;
Ok(SecretsListPage {
names: page.names,
next_cursor: page.next_cursor,
})
}
}
// ----------------------------------------------------------------------------
// Tests — in-memory SecretsRepo so unit tests don't need Postgres.
// ----------------------------------------------------------------------------
#[cfg(test)]
mod tests {
use super::*;
use crate::authz::{AuthzError, AuthzRepo};
use crate::secrets_repo::{SecretsMetaPage, SecretsNamePage};
use async_trait::async_trait;
use picloud_shared::{
AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, Principal, RequestId, ScriptId,
UserId,
};
use std::collections::BTreeMap;
use tokio::sync::Mutex;
#[derive(Default)]
struct InMemorySecretsRepo {
data: Mutex<BTreeMap<(AppId, String), StoredSecret>>,
}
#[async_trait]
impl SecretsRepo for InMemorySecretsRepo {
async fn get(
&self,
app_id: AppId,
name: &str,
) -> Result<Option<StoredSecret>, SecretsRepoError> {
Ok(self
.data
.lock()
.await
.get(&(app_id, name.to_string()))
.cloned())
}
async fn set(
&self,
app_id: AppId,
name: &str,
encrypted_value: &[u8],
nonce: &[u8],
) -> Result<(), SecretsRepoError> {
self.data.lock().await.insert(
(app_id, name.to_string()),
StoredSecret {
encrypted_value: encrypted_value.to_vec(),
nonce: nonce.to_vec(),
},
);
Ok(())
}
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, SecretsRepoError> {
Ok(self
.data
.lock()
.await
.remove(&(app_id, name.to_string()))
.is_some())
}
async fn list_names(
&self,
app_id: AppId,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsNamePage, SecretsRepoError> {
let data = self.data.lock().await;
let last = cursor.map(std::string::ToString::to_string);
let mut names: Vec<String> = data
.iter()
.filter(|((a, _), _)| *a == app_id)
.map(|((_, n), _)| n.clone())
.filter(|n| last.as_ref().is_none_or(|l| n > l))
.collect();
names.sort();
let take = (limit as usize).max(1);
let next_cursor = if names.len() > take {
names.truncate(take);
names.last().cloned()
} else {
None
};
Ok(SecretsNamePage { names, next_cursor })
}
async fn list_meta(
&self,
_app_id: AppId,
_cursor: Option<&str>,
_limit: u32,
) -> Result<SecretsMetaPage, SecretsRepoError> {
unimplemented!("admin-only; not exercised in service tests")
}
}
#[derive(Default)]
struct DenyingAuthzRepo;
#[async_trait]
impl AuthzRepo for DenyingAuthzRepo {
async fn membership(
&self,
_user_id: UserId,
_app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok(None)
}
}
fn key() -> MasterKey {
MasterKey::from_bytes([0x5au8; 32])
}
fn svc() -> SecretsServiceImpl {
SecretsServiceImpl::new(
Arc::new(InMemorySecretsRepo::default()),
Arc::new(DenyingAuthzRepo),
key(),
SecretsConfig::conservative(),
)
}
fn cx_with(app_id: AppId, principal: Option<Principal>) -> SdkCallCx {
SdkCallCx {
app_id,
script_id: ScriptId::new(),
principal,
execution_id: ExecutionId::new(),
request_id: RequestId::new(),
trigger_depth: 0,
root_execution_id: ExecutionId::new(),
is_dead_letter_handler: false,
event: None,
}
}
fn anon_cx(app_id: AppId) -> SdkCallCx {
cx_with(app_id, None)
}
fn member_no_role_cx(app_id: AppId) -> SdkCallCx {
cx_with(
app_id,
Some(Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Member,
scopes: None,
app_binding: None,
}),
)
}
fn owner_cx(app_id: AppId) -> SdkCallCx {
cx_with(
app_id,
Some(Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Owner,
scopes: None,
app_binding: None,
}),
)
}
#[tokio::test]
async fn set_get_delete_round_trip() {
let s = svc();
let cx = anon_cx(AppId::new());
s.set(&cx, "stripe_key", serde_json::json!("sk_live_xxx"))
.await
.unwrap();
assert_eq!(
s.get(&cx, "stripe_key").await.unwrap(),
Some(serde_json::json!("sk_live_xxx"))
);
assert!(s.delete(&cx, "stripe_key").await.unwrap());
assert_eq!(s.get(&cx, "stripe_key").await.unwrap(), None);
// Idempotent delete.
assert!(!s.delete(&cx, "stripe_key").await.unwrap());
}
#[tokio::test]
async fn get_missing_returns_none() {
let s = svc();
let cx = anon_cx(AppId::new());
assert_eq!(s.get(&cx, "nope").await.unwrap(), None);
}
#[tokio::test]
async fn empty_name_rejected() {
let s = svc();
let cx = anon_cx(AppId::new());
let err = s.set(&cx, "", serde_json::json!("x")).await.unwrap_err();
assert!(matches!(err, SecretsError::InvalidName(_)));
let err = s.get(&cx, "").await.unwrap_err();
assert!(matches!(err, SecretsError::InvalidName(_)));
}
#[tokio::test]
async fn name_length_capped() {
let s = svc();
let cx = anon_cx(AppId::new());
let long = "a".repeat(256);
let err = s.set(&cx, &long, serde_json::json!(1)).await.unwrap_err();
assert!(matches!(err, SecretsError::InvalidName(_)));
// Exactly 255 is allowed.
let ok = "b".repeat(255);
s.set(&cx, &ok, serde_json::json!(1)).await.unwrap();
}
#[tokio::test]
async fn value_over_cap_rejected() {
let s = SecretsServiceImpl::new(
Arc::new(InMemorySecretsRepo::default()),
Arc::new(DenyingAuthzRepo),
key(),
SecretsConfig {
max_value_bytes: 16,
},
);
let cx = anon_cx(AppId::new());
let big = serde_json::json!("x".repeat(64));
let err = s.set(&cx, "k", big).await.unwrap_err();
assert!(matches!(err, SecretsError::TooLarge { limit: 16, .. }));
}
#[tokio::test]
async fn cross_app_isolation() {
let s = svc();
let a = AppId::new();
let b = AppId::new();
s.set(&anon_cx(a), "shared", serde_json::json!("from-a"))
.await
.unwrap();
s.set(&anon_cx(b), "shared", serde_json::json!("from-b"))
.await
.unwrap();
assert_eq!(
s.get(&anon_cx(a), "shared").await.unwrap(),
Some(serde_json::json!("from-a"))
);
assert_eq!(
s.get(&anon_cx(b), "shared").await.unwrap(),
Some(serde_json::json!("from-b"))
);
}
#[tokio::test]
async fn anonymous_skips_authz() {
let s = svc();
// DenyingAuthzRepo would deny an authed principal; anon skips it.
s.set(&anon_cx(AppId::new()), "k", serde_json::json!(1))
.await
.unwrap();
}
#[tokio::test]
async fn authed_member_without_role_forbidden() {
let s = svc();
let err = s
.set(&member_no_role_cx(AppId::new()), "k", serde_json::json!(1))
.await
.unwrap_err();
assert!(matches!(err, SecretsError::Forbidden));
}
#[tokio::test]
async fn owner_can_write() {
let s = svc();
s.set(&owner_cx(AppId::new()), "k", serde_json::json!(1))
.await
.unwrap();
}
/// Type round-trip: a String comes back a String, a Map a Map, an
/// Array an Array — the JSON encoding is transparent.
#[tokio::test]
async fn type_round_trip_preserves_shape() {
let s = svc();
let cx = anon_cx(AppId::new());
s.set(&cx, "str", serde_json::json!("sk_live_xxx"))
.await
.unwrap();
assert_eq!(
s.get(&cx, "str").await.unwrap(),
Some(serde_json::json!("sk_live_xxx"))
);
let map = serde_json::json!({ "client_id": "abc", "client_secret": "xyz" });
s.set(&cx, "oauth", map.clone()).await.unwrap();
assert_eq!(s.get(&cx, "oauth").await.unwrap(), Some(map));
let arr = serde_json::json!([1, 2, 3]);
s.set(&cx, "arr", arr.clone()).await.unwrap();
assert_eq!(s.get(&cx, "arr").await.unwrap(), Some(arr));
}
#[tokio::test]
async fn corrupted_ciphertext_surfaces_error() {
let repo = Arc::new(InMemorySecretsRepo::default());
let s = SecretsServiceImpl::new(
repo.clone(),
Arc::new(DenyingAuthzRepo),
key(),
SecretsConfig::conservative(),
);
let app = AppId::new();
s.set(&anon_cx(app), "k", serde_json::json!("v"))
.await
.unwrap();
// Corrupt the stored ciphertext directly.
repo.data
.lock()
.await
.get_mut(&(app, "k".to_string()))
.unwrap()
.encrypted_value[0] ^= 0xff;
let err = s.get(&anon_cx(app), "k").await.unwrap_err();
assert!(matches!(err, SecretsError::Corrupted));
}
#[tokio::test]
async fn list_returns_names_paginated() {
let s = svc();
let cx = anon_cx(AppId::new());
for i in 0..5 {
s.set(&cx, &format!("k{i:02}"), serde_json::json!(i))
.await
.unwrap();
}
let p1 = s.list(&cx, None, 2).await.unwrap();
assert_eq!(p1.names, vec!["k00".to_string(), "k01".to_string()]);
assert!(p1.next_cursor.is_some());
let p2 = s.list(&cx, p1.next_cursor.as_deref(), 10).await.unwrap();
assert_eq!(
p2.names,
vec!["k02".to_string(), "k03".to_string(), "k04".to_string()]
);
assert!(p2.next_cursor.is_none());
}
}

View File

@@ -0,0 +1,212 @@
//! `TopicRepo` — CRUD for the `topics` table (v1.1.6).
//!
//! This table holds ONLY topics that have been explicitly externalized
//! for SSE subscription (design notes §5). Internal-only pub/sub topics
//! stay implicit — they never get a row here, and the publish path never
//! consults this table. The two readers are the topic admin endpoints
//! ([`crate::topics_api`]) and the SSE subscribe authorization
//! ([`crate::realtime_authority`]).
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use picloud_shared::AppId;
use serde::{Deserialize, Serialize};
use sqlx::PgPool;
/// External-subscriber auth gate for a topic. `'public'` + `'token'` in
/// v1.1.6; `'session'` (v1.1.8) and `'script'` (v1.2) extend the DB
/// CHECK constraint and this enum later.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum TopicAuthMode {
Public,
Token,
}
impl TopicAuthMode {
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Self::Public => "public",
Self::Token => "token",
}
}
fn from_db(s: &str) -> Result<Self, TopicRepoError> {
match s {
"public" => Ok(Self::Public),
"token" => Ok(Self::Token),
other => Err(TopicRepoError::Backend(format!(
"unknown auth_mode in DB: {other}"
))),
}
}
}
/// A registered, externally-subscribable topic row.
#[derive(Debug, Clone, Serialize)]
pub struct Topic {
pub name: String,
pub external_subscribable: bool,
pub auth_mode: TopicAuthMode,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
#[derive(Debug, thiserror::Error)]
pub enum TopicRepoError {
#[error("a topic named {0:?} already exists in this app")]
AlreadyExists(String),
#[error("database error: {0}")]
Db(#[from] sqlx::Error),
#[error("topic backend error: {0}")]
Backend(String),
}
#[async_trait]
pub trait TopicRepo: Send + Sync {
/// Register a topic. Errors `AlreadyExists` on PK conflict.
async fn create(
&self,
app_id: AppId,
name: &str,
external_subscribable: bool,
auth_mode: TopicAuthMode,
) -> Result<Topic, TopicRepoError>;
/// List every registered topic in the app, ordered by name.
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError>;
/// Fetch one topic by name, `None` if not registered.
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError>;
/// Update `external_subscribable` and/or `auth_mode` (each `None`
/// leaves the column unchanged). `None` return = no such topic.
async fn update(
&self,
app_id: AppId,
name: &str,
external_subscribable: Option<bool>,
auth_mode: Option<TopicAuthMode>,
) -> Result<Option<Topic>, TopicRepoError>;
/// Unregister a topic. Returns `true` if a row was removed.
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError>;
}
#[derive(sqlx::FromRow)]
struct TopicRow {
name: String,
external_subscribable: bool,
auth_mode: String,
created_at: DateTime<Utc>,
updated_at: DateTime<Utc>,
}
impl TopicRow {
fn into_topic(self) -> Result<Topic, TopicRepoError> {
Ok(Topic {
auth_mode: TopicAuthMode::from_db(&self.auth_mode)?,
name: self.name,
external_subscribable: self.external_subscribable,
created_at: self.created_at,
updated_at: self.updated_at,
})
}
}
const SELECT_COLS: &str = "name, external_subscribable, auth_mode, created_at, updated_at";
pub struct PostgresTopicRepo {
pool: PgPool,
}
impl PostgresTopicRepo {
#[must_use]
pub fn new(pool: PgPool) -> Self {
Self { pool }
}
}
#[async_trait]
impl TopicRepo for PostgresTopicRepo {
async fn create(
&self,
app_id: AppId,
name: &str,
external_subscribable: bool,
auth_mode: TopicAuthMode,
) -> Result<Topic, TopicRepoError> {
let row: Option<TopicRow> = sqlx::query_as(&format!(
"INSERT INTO topics (app_id, name, external_subscribable, auth_mode) \
VALUES ($1, $2, $3, $4) \
ON CONFLICT (app_id, name) DO NOTHING \
RETURNING {SELECT_COLS}"
))
.bind(app_id.into_inner())
.bind(name)
.bind(external_subscribable)
.bind(auth_mode.as_str())
.fetch_optional(&self.pool)
.await?;
match row {
Some(r) => r.into_topic(),
None => Err(TopicRepoError::AlreadyExists(name.to_string())),
}
}
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError> {
let rows: Vec<TopicRow> = sqlx::query_as(&format!(
"SELECT {SELECT_COLS} FROM topics WHERE app_id = $1 ORDER BY name"
))
.bind(app_id.into_inner())
.fetch_all(&self.pool)
.await?;
rows.into_iter().map(TopicRow::into_topic).collect()
}
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
let row: Option<TopicRow> = sqlx::query_as(&format!(
"SELECT {SELECT_COLS} FROM topics WHERE app_id = $1 AND name = $2"
))
.bind(app_id.into_inner())
.bind(name)
.fetch_optional(&self.pool)
.await?;
row.map(TopicRow::into_topic).transpose()
}
async fn update(
&self,
app_id: AppId,
name: &str,
external_subscribable: Option<bool>,
auth_mode: Option<TopicAuthMode>,
) -> Result<Option<Topic>, TopicRepoError> {
// COALESCE leaves a column untouched when its bind is NULL.
let row: Option<TopicRow> = sqlx::query_as(&format!(
"UPDATE topics SET \
external_subscribable = COALESCE($3, external_subscribable), \
auth_mode = COALESCE($4, auth_mode), \
updated_at = NOW() \
WHERE app_id = $1 AND name = $2 \
RETURNING {SELECT_COLS}"
))
.bind(app_id.into_inner())
.bind(name)
.bind(external_subscribable)
.bind(auth_mode.map(TopicAuthMode::as_str))
.fetch_optional(&self.pool)
.await?;
row.map(TopicRow::into_topic).transpose()
}
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError> {
let res = sqlx::query("DELETE FROM topics WHERE app_id = $1 AND name = $2")
.bind(app_id.into_inner())
.bind(name)
.execute(&self.pool)
.await?;
Ok(res.rows_affected() > 0)
}
}

View File

@@ -0,0 +1,629 @@
//! `/api/v1/admin/apps/{id}/topics*` — topic registration admin
//! endpoints (v1.1.6).
//!
//! These manage the `topics` table: the explicit registry of which
//! pub/sub topics are externally subscribable over SSE (design notes
//! §5). Internal-only topics never appear here.
//!
//! * `POST /apps/{id}/topics` — register a topic.
//! * `GET /apps/{id}/topics` — list registered topics.
//! * `PATCH /apps/{id}/topics/{name}` — flip external/auth_mode.
//! * `DELETE /apps/{id}/topics/{name}` — unregister + disconnect.
//!
//! The PATCH endpoint is deliberately its OWN surface (not folded into a
//! generic topic update) so every change to externally-subscribable
//! status is a discrete, watchable/auditable API call (§5 commitment).
//!
//! Create / update / delete are gated by `AppTopicManage` (→ `app:admin`
//! scope); list is gated by the existing `AppRead`. DELETE also drops
//! the topic's in-process broadcast channel so live SSE subscribers
//! disconnect cleanly.
use std::sync::Arc;
use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Json, Response};
use axum::routing::{get, patch};
use axum::{Extension, Router};
use picloud_shared::{AppId, Principal, RealtimeBroadcaster};
use serde::Deserialize;
use serde_json::json;
use crate::app_repo::AppRepository;
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
use crate::topic_repo::{Topic, TopicAuthMode, TopicRepo, TopicRepoError};
#[derive(Clone)]
pub struct TopicsState {
pub topics: Arc<dyn TopicRepo>,
pub apps: Arc<dyn AppRepository>,
pub authz: Arc<dyn AuthzRepo>,
pub broadcaster: Arc<dyn RealtimeBroadcaster>,
}
pub fn topics_router(state: TopicsState) -> Router {
Router::new()
.route("/apps/{app_id}/topics", get(list_topics).post(create_topic))
.route(
"/apps/{app_id}/topics/{name}",
patch(update_topic).delete(delete_topic),
)
.with_state(state)
}
#[derive(Debug, Deserialize)]
pub struct CreateTopicRequest {
pub name: String,
#[serde(default)]
pub external_subscribable: bool,
#[serde(default = "default_auth_mode")]
pub auth_mode: TopicAuthMode,
}
const fn default_auth_mode() -> TopicAuthMode {
TopicAuthMode::Public
}
#[derive(Debug, Deserialize)]
pub struct UpdateTopicRequest {
#[serde(default)]
pub external_subscribable: Option<bool>,
#[serde(default)]
pub auth_mode: Option<TopicAuthMode>,
}
/// Topic names are concrete (external pattern subscription is v1.2), so
/// reject empties and `*` wildcards at registration.
fn validate_topic_name(name: &str) -> Result<(), TopicsApiError> {
if name.trim().is_empty() {
return Err(TopicsApiError::Invalid(
"topic name must not be empty".into(),
));
}
if name.contains('*') {
return Err(TopicsApiError::Invalid(
"topic name must be a concrete topic, not a pattern (no '*')".into(),
));
}
Ok(())
}
async fn create_topic(
State(s): State<TopicsState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Json(input): Json<CreateTopicRequest>,
) -> Result<(StatusCode, Json<Topic>), TopicsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppTopicManage(app_id),
)
.await?;
validate_topic_name(&input.name)?;
let topic = s
.topics
.create(
app_id,
input.name.trim(),
input.external_subscribable,
input.auth_mode,
)
.await?;
Ok((StatusCode::CREATED, Json(topic)))
}
#[derive(Debug, serde::Serialize)]
struct ListTopicsResponse {
topics: Vec<Topic>,
}
async fn list_topics(
State(s): State<TopicsState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
) -> Result<Json<ListTopicsResponse>, TopicsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(s.authz.as_ref(), &principal, Capability::AppRead(app_id)).await?;
let topics = s.topics.list(app_id).await?;
Ok(Json(ListTopicsResponse { topics }))
}
async fn update_topic(
State(s): State<TopicsState>,
Extension(principal): Extension<Principal>,
Path((app_id, name)): Path<(AppId, String)>,
Json(input): Json<UpdateTopicRequest>,
) -> Result<Json<Topic>, TopicsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppTopicManage(app_id),
)
.await?;
let topic = s
.topics
.update(app_id, &name, input.external_subscribable, input.auth_mode)
.await?
.ok_or(TopicsApiError::NotFound)?;
Ok(Json(topic))
}
async fn delete_topic(
State(s): State<TopicsState>,
Extension(principal): Extension<Principal>,
Path((app_id, name)): Path<(AppId, String)>,
) -> Result<StatusCode, TopicsApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppTopicManage(app_id),
)
.await?;
if !s.topics.delete(app_id, &name).await? {
return Err(TopicsApiError::NotFound);
}
// Disconnect any live SSE subscribers for the now-unregistered topic.
s.broadcaster.drop_topic(app_id, &name).await;
Ok(StatusCode::NO_CONTENT)
}
async fn ensure_app_exists(apps: &dyn AppRepository, app_id: AppId) -> Result<(), TopicsApiError> {
apps.get_by_id(app_id)
.await
.map_err(|e| TopicsApiError::Backend(e.to_string()))?
.ok_or(TopicsApiError::AppNotFound)?;
Ok(())
}
#[derive(Debug, thiserror::Error)]
pub enum TopicsApiError {
#[error("app not found")]
AppNotFound,
#[error("topic not found")]
NotFound,
#[error("{0}")]
AlreadyExists(String),
#[error("invalid request: {0}")]
Invalid(String),
#[error("forbidden")]
Forbidden,
#[error("authorization repo error: {0}")]
AuthzRepo(String),
#[error("topics backend: {0}")]
Backend(String),
}
impl From<AuthzDenied> for TopicsApiError {
fn from(d: AuthzDenied) -> Self {
match d {
AuthzDenied::Denied => Self::Forbidden,
AuthzDenied::Repo(e) => Self::AuthzRepo(e.to_string()),
}
}
}
impl From<AuthzError> for TopicsApiError {
fn from(e: AuthzError) -> Self {
Self::AuthzRepo(e.to_string())
}
}
impl From<TopicRepoError> for TopicsApiError {
fn from(e: TopicRepoError) -> Self {
match e {
TopicRepoError::AlreadyExists(name) => {
Self::AlreadyExists(format!("a topic named {name:?} already exists in this app"))
}
other => Self::Backend(other.to_string()),
}
}
}
impl IntoResponse for TopicsApiError {
fn into_response(self) -> Response {
let (status, body) = match &self {
Self::AppNotFound | Self::NotFound => {
(StatusCode::NOT_FOUND, json!({ "error": self.to_string() }))
}
Self::AlreadyExists(_) => (StatusCode::CONFLICT, json!({ "error": self.to_string() })),
Self::Invalid(_) => (
StatusCode::UNPROCESSABLE_ENTITY,
json!({ "error": self.to_string() }),
),
Self::Forbidden => (StatusCode::FORBIDDEN, json!({ "error": self.to_string() })),
Self::AuthzRepo(e) => {
tracing::error!(error = %e, "topics admin authz repo error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
Self::Backend(e) => {
tracing::error!(error = %e, "topics admin backend error");
(
StatusCode::INTERNAL_SERVER_ERROR,
json!({ "error": "internal error" }),
)
}
};
(status, Json(body)).into_response()
}
}
#[cfg(test)]
mod tests {
//! In-memory handler tests: capability enforcement, the
//! `external_subscribable` default, the flip being its own endpoint,
//! cross-app isolation, and DELETE disconnecting subscribers. The
//! Postgres repo is exercised by the schema + integration suites.
use super::*;
use crate::repo::ScriptRepositoryError;
use async_trait::async_trait;
use chrono::Utc;
use picloud_shared::{
AdminUserId, App, AppRole, BroadcasterError, InstanceRole, RealtimeEvent, UserId,
};
use std::collections::HashMap;
use std::sync::Mutex as StdMutex;
use tokio::sync::Mutex;
#[derive(Default)]
struct InMemoryTopicRepo {
inner: Mutex<HashMap<(AppId, String), Topic>>,
}
#[async_trait]
impl TopicRepo for InMemoryTopicRepo {
async fn create(
&self,
app_id: AppId,
name: &str,
external_subscribable: bool,
auth_mode: TopicAuthMode,
) -> Result<Topic, TopicRepoError> {
let mut g = self.inner.lock().await;
if g.contains_key(&(app_id, name.to_string())) {
return Err(TopicRepoError::AlreadyExists(name.to_string()));
}
let now = Utc::now();
let t = Topic {
name: name.to_string(),
external_subscribable,
auth_mode,
created_at: now,
updated_at: now,
};
g.insert((app_id, name.to_string()), t.clone());
Ok(t)
}
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError> {
let g = self.inner.lock().await;
let mut v: Vec<Topic> = g
.iter()
.filter(|((a, _), _)| *a == app_id)
.map(|(_, t)| t.clone())
.collect();
v.sort_by(|a, b| a.name.cmp(&b.name));
Ok(v)
}
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
Ok(self
.inner
.lock()
.await
.get(&(app_id, name.to_string()))
.cloned())
}
async fn update(
&self,
app_id: AppId,
name: &str,
external_subscribable: Option<bool>,
auth_mode: Option<TopicAuthMode>,
) -> Result<Option<Topic>, TopicRepoError> {
let mut g = self.inner.lock().await;
let Some(t) = g.get_mut(&(app_id, name.to_string())) else {
return Ok(None);
};
if let Some(e) = external_subscribable {
t.external_subscribable = e;
}
if let Some(m) = auth_mode {
t.auth_mode = m;
}
t.updated_at = Utc::now();
Ok(Some(t.clone()))
}
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError> {
Ok(self
.inner
.lock()
.await
.remove(&(app_id, name.to_string()))
.is_some())
}
}
struct InMemoryAppRepo(AppId);
#[async_trait]
impl AppRepository for InMemoryAppRepo {
async fn list(&self) -> Result<Vec<App>, ScriptRepositoryError> {
unimplemented!()
}
async fn list_for_user(&self, _: AdminUserId) -> Result<Vec<App>, ScriptRepositoryError> {
unimplemented!()
}
async fn get_by_id(&self, id: AppId) -> Result<Option<App>, ScriptRepositoryError> {
if id != self.0 {
return Ok(None);
}
let now = Utc::now();
Ok(Some(App {
id,
slug: "test".into(),
name: "test".into(),
description: None,
created_at: now,
updated_at: now,
}))
}
async fn get_by_slug(&self, _: &str) -> Result<Option<App>, ScriptRepositoryError> {
unimplemented!()
}
async fn get_by_slug_or_history(
&self,
_: &str,
) -> Result<Option<crate::app_repo::AppLookup>, ScriptRepositoryError> {
unimplemented!()
}
async fn slug_in_history(&self, _: &str) -> Result<Option<App>, ScriptRepositoryError> {
unimplemented!()
}
async fn create(
&self,
_: &str,
_: &str,
_: Option<&str>,
) -> Result<App, ScriptRepositoryError> {
unimplemented!()
}
async fn create_with_takeover(
&self,
_: &str,
_: &str,
_: Option<&str>,
) -> Result<App, ScriptRepositoryError> {
unimplemented!()
}
async fn update(
&self,
_: AppId,
_: Option<&str>,
_: Option<Option<&str>>,
) -> Result<App, ScriptRepositoryError> {
unimplemented!()
}
async fn rename_slug(
&self,
_: AppId,
_: &str,
_: bool,
) -> Result<App, ScriptRepositoryError> {
unimplemented!()
}
async fn delete(&self, _: AppId) -> Result<(), ScriptRepositoryError> {
unimplemented!()
}
async fn delete_cascade(&self, _: AppId) -> Result<(), ScriptRepositoryError> {
unimplemented!()
}
async fn count_scripts_in_app(&self, _: AppId) -> Result<i64, ScriptRepositoryError> {
unimplemented!()
}
}
/// Grants `AppAdmin` only for `granted_app`; denies elsewhere — used
/// for the cross-app isolation test.
struct PerAppAuthzRepo {
granted_app: AppId,
}
#[async_trait]
impl AuthzRepo for PerAppAuthzRepo {
async fn membership(
&self,
_: UserId,
app_id: AppId,
) -> Result<Option<AppRole>, AuthzError> {
Ok((app_id == self.granted_app).then_some(AppRole::AppAdmin))
}
}
struct DenyAuthzRepo;
#[async_trait]
impl AuthzRepo for DenyAuthzRepo {
async fn membership(&self, _: UserId, _: AppId) -> Result<Option<AppRole>, AuthzError> {
Ok(None)
}
}
#[derive(Default)]
struct RecordingBroadcaster {
dropped: StdMutex<Vec<(AppId, String)>>,
}
#[async_trait]
impl RealtimeBroadcaster for RecordingBroadcaster {
async fn subscribe(
&self,
_: AppId,
_: &str,
) -> Result<tokio::sync::broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
unimplemented!()
}
async fn publish(&self, _: AppId, _: &str, _: RealtimeEvent) {}
async fn drop_topic(&self, app_id: AppId, topic: &str) {
self.dropped
.lock()
.unwrap()
.push((app_id, topic.to_string()));
}
}
fn member() -> Principal {
Principal {
user_id: AdminUserId::new(),
instance_role: InstanceRole::Member,
scopes: None,
app_binding: None,
}
}
fn state(app_id: AppId, authz: Arc<dyn AuthzRepo>) -> (TopicsState, Arc<RecordingBroadcaster>) {
let bc = Arc::new(RecordingBroadcaster::default());
let state = TopicsState {
topics: Arc::new(InMemoryTopicRepo::default()),
apps: Arc::new(InMemoryAppRepo(app_id)),
authz,
broadcaster: bc.clone(),
};
(state, bc)
}
#[tokio::test]
async fn register_defaults_external_subscribable_false() {
let app = AppId::new();
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
let (status, Json(topic)) = create_topic(
State(s),
Extension(member()),
Path(app),
Json(CreateTopicRequest {
name: "chat".into(),
external_subscribable: false,
auth_mode: TopicAuthMode::Public,
}),
)
.await
.unwrap();
assert_eq!(status, StatusCode::CREATED);
assert!(!topic.external_subscribable);
assert_eq!(topic.name, "chat");
}
#[tokio::test]
async fn flip_requires_app_admin_role() {
let app = AppId::new();
// Topic exists; the caller has no role → PATCH is forbidden.
let (s, _) = state(app, Arc::new(DenyAuthzRepo));
s.topics
.create(app, "chat", false, TopicAuthMode::Public)
.await
.unwrap();
let err = update_topic(
State(s),
Extension(member()),
Path((app, "chat".to_string())),
Json(UpdateTopicRequest {
external_subscribable: Some(true),
auth_mode: None,
}),
)
.await
.unwrap_err();
assert!(matches!(err, TopicsApiError::Forbidden));
}
#[tokio::test]
async fn flip_is_its_own_endpoint_and_toggles_external() {
// The PATCH handler is a distinct surface from create; flipping
// external_subscribable false→true is a single discrete call.
let app = AppId::new();
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
s.topics
.create(app, "chat", false, TopicAuthMode::Public)
.await
.unwrap();
let Json(updated) = update_topic(
State(s),
Extension(member()),
Path((app, "chat".to_string())),
Json(UpdateTopicRequest {
external_subscribable: Some(true),
auth_mode: Some(TopicAuthMode::Token),
}),
)
.await
.unwrap();
assert!(updated.external_subscribable);
assert_eq!(updated.auth_mode, TopicAuthMode::Token);
}
#[tokio::test]
async fn delete_disconnects_subscribers() {
let app = AppId::new();
let (s, bc) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
s.topics
.create(app, "chat", true, TopicAuthMode::Public)
.await
.unwrap();
let status = delete_topic(
State(s),
Extension(member()),
Path((app, "chat".to_string())),
)
.await
.unwrap();
assert_eq!(status, StatusCode::NO_CONTENT);
assert_eq!(
bc.dropped.lock().unwrap().as_slice(),
&[(app, "chat".to_string())]
);
}
#[tokio::test]
async fn cross_app_admin_cannot_manage_other_app() {
let app_a = AppId::new();
let app_b = AppId::new();
// Caller is admin of app A only; both apps exist via separate state.
let authz = Arc::new(PerAppAuthzRepo { granted_app: app_a });
// App-B-scoped state, but the caller only has A's grant.
let (s, _) = state(app_b, authz);
let err = create_topic(
State(s),
Extension(member()),
Path(app_b),
Json(CreateTopicRequest {
name: "chat".into(),
external_subscribable: true,
auth_mode: TopicAuthMode::Public,
}),
)
.await
.unwrap_err();
assert!(matches!(err, TopicsApiError::Forbidden));
}
#[tokio::test]
async fn pattern_name_rejected() {
let app = AppId::new();
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
let err = create_topic(
State(s),
Extension(member()),
Path(app),
Json(CreateTopicRequest {
name: "user.*".into(),
external_subscribable: true,
auth_mode: TopicAuthMode::Public,
}),
)
.await
.unwrap_err();
assert!(matches!(err, TopicsApiError::Invalid(_)));
}
}

View File

@@ -5,7 +5,9 @@
use async_trait::async_trait; use async_trait::async_trait;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use picloud_shared::{AdminUserId, AppId, DocsEventOp, KvEventOp, ScriptId, TriggerId}; use picloud_shared::{
AdminUserId, AppId, DocsEventOp, FilesEventOp, KvEventOp, ScriptId, TriggerId,
};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use sqlx::PgPool; use sqlx::PgPool;
use uuid::Uuid; use uuid::Uuid;
@@ -51,6 +53,12 @@ pub enum TriggerKind {
DeadLetter, DeadLetter,
/// v1.1.4. /// v1.1.4.
Cron, Cron,
/// v1.1.5.
Files,
/// v1.1.5.
Pubsub,
/// v1.1.7. Inbound email via the webhook receiver.
Email,
} }
impl TriggerKind { impl TriggerKind {
@@ -61,6 +69,9 @@ impl TriggerKind {
Self::Docs => "docs", Self::Docs => "docs",
Self::DeadLetter => "dead_letter", Self::DeadLetter => "dead_letter",
Self::Cron => "cron", Self::Cron => "cron",
Self::Files => "files",
Self::Pubsub => "pubsub",
Self::Email => "email",
} }
} }
@@ -71,6 +82,9 @@ impl TriggerKind {
"docs" => Some(Self::Docs), "docs" => Some(Self::Docs),
"dead_letter" => Some(Self::DeadLetter), "dead_letter" => Some(Self::DeadLetter),
"cron" => Some(Self::Cron), "cron" => Some(Self::Cron),
"files" => Some(Self::Files),
"pubsub" => Some(Self::Pubsub),
"email" => Some(Self::Email),
_ => None, _ => None,
} }
} }
@@ -120,6 +134,17 @@ pub enum TriggerDetails {
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
last_fired_at: Option<DateTime<Utc>>, last_fired_at: Option<DateTime<Utc>>,
}, },
/// v1.1.5. Same shape as KV/docs: a collection glob + op subset.
Files {
collection_glob: String,
ops: Vec<FilesEventOp>,
},
/// v1.1.5. A topic pattern: exact, `<prefix>.*`, or `*`.
Pubsub { topic_pattern: String },
/// v1.1.7. Inbound email. The HMAC `inbound_secret` is never
/// surfaced (it's encrypted at rest); we expose only whether one is
/// configured so the admin UI can show "signed" vs "unsigned".
Email { has_inbound_secret: bool },
} }
/// Create payload for a KV trigger. Defaults applied at the admin /// Create payload for a KV trigger. Defaults applied at the admin
@@ -175,6 +200,73 @@ pub struct CreateCronTrigger {
pub registered_by_principal: AdminUserId, pub registered_by_principal: AdminUserId,
} }
/// Create payload for a files trigger (v1.1.5). Same shape as KV with
/// `FilesEventOp` ops.
#[derive(Debug, Clone)]
pub struct CreateFilesTrigger {
pub script_id: ScriptId,
pub collection_glob: String,
pub ops: Vec<FilesEventOp>,
pub dispatch_mode: TriggerDispatchMode,
pub retry_max_attempts: u32,
pub retry_backoff: BackoffShape,
pub retry_base_ms: u32,
pub registered_by_principal: AdminUserId,
}
/// One match for the dispatcher's files trigger fan-out lookup
/// (v1.1.5). Same shape as `KvTriggerMatch`.
#[derive(Debug, Clone)]
pub struct FilesTriggerMatch {
pub trigger_id: TriggerId,
pub script_id: ScriptId,
pub dispatch_mode: TriggerDispatchMode,
pub retry_max_attempts: u32,
pub retry_backoff: BackoffShape,
pub retry_base_ms: u32,
pub registered_by_principal: AdminUserId,
}
/// Create payload for a pubsub trigger (v1.1.5). `topic_pattern` is
/// validated (exact / `<prefix>.*` / `*`) before insert.
#[derive(Debug, Clone)]
pub struct CreatePubsubTrigger {
pub script_id: ScriptId,
pub topic_pattern: String,
pub dispatch_mode: TriggerDispatchMode,
pub retry_max_attempts: u32,
pub retry_backoff: BackoffShape,
pub retry_base_ms: u32,
pub registered_by_principal: AdminUserId,
}
/// Create payload for an email trigger (v1.1.7). `inbound_secret_*` is
/// the already-encrypted HMAC secret (sealed by the admin layer with the
/// process master key) or `None` for an unsigned trigger.
#[derive(Debug, Clone)]
pub struct CreateEmailTrigger {
pub script_id: ScriptId,
pub inbound_secret_encrypted: Option<Vec<u8>>,
pub inbound_secret_nonce: Option<Vec<u8>>,
pub registered_by_principal: AdminUserId,
}
/// What the inbound-email webhook receiver needs to verify + dispatch a
/// POST. Returned by `email_inbound_target`; `None` when the trigger
/// doesn't exist or isn't `kind = 'email'`.
#[derive(Debug, Clone)]
pub struct EmailInboundTarget {
pub app_id: AppId,
pub script_id: ScriptId,
pub enabled: bool,
pub dispatch_mode: TriggerDispatchMode,
pub registered_by_principal: AdminUserId,
/// Encrypted HMAC secret + nonce; both `None` for an unsigned
/// trigger (accepts any POST).
pub inbound_secret_encrypted: Option<Vec<u8>>,
pub inbound_secret_nonce: Option<Vec<u8>>,
}
/// One match for the dispatcher's "which KV triggers fire on this /// One match for the dispatcher's "which KV triggers fire on this
/// event" lookup. Carries everything the dispatcher needs to construct /// event" lookup. Carries everything the dispatcher needs to construct
/// the outbox row. /// the outbox row.
@@ -242,6 +334,37 @@ pub trait TriggerRepo: Send + Sync {
req: CreateCronTrigger, req: CreateCronTrigger,
) -> Result<Trigger, TriggerRepoError>; ) -> Result<Trigger, TriggerRepoError>;
/// v1.1.5.
async fn create_files_trigger(
&self,
app_id: AppId,
req: CreateFilesTrigger,
) -> Result<Trigger, TriggerRepoError>;
/// v1.1.5. `topic_pattern` is validated before insert.
async fn create_pubsub_trigger(
&self,
app_id: AppId,
req: CreatePubsubTrigger,
) -> Result<Trigger, TriggerRepoError>;
/// v1.1.7. Inbound email trigger. The `inbound_secret` is stored
/// already-encrypted (the admin layer seals it).
async fn create_email_trigger(
&self,
app_id: AppId,
req: CreateEmailTrigger,
) -> Result<Trigger, TriggerRepoError>;
/// v1.1.7. The webhook receiver's hot-path lookup: resolve a
/// `kind = 'email'` trigger to its app, handler script, dispatch
/// mode, and (encrypted) HMAC secret. Returns `None` when the
/// trigger doesn't exist or isn't an email trigger.
async fn email_inbound_target(
&self,
trigger_id: TriggerId,
) -> Result<Option<EmailInboundTarget>, TriggerRepoError>;
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError>; async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError>;
async fn get(&self, id: TriggerId) -> Result<Option<Trigger>, TriggerRepoError>; async fn get(&self, id: TriggerId) -> Result<Option<Trigger>, TriggerRepoError>;
@@ -269,6 +392,16 @@ pub trait TriggerRepo: Send + Sync {
op: DocsEventOp, op: DocsEventOp,
) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError>; ) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError>;
/// Dispatcher hot path for files fan-out (v1.1.5). Mirrors the KV
/// fan-out logic: pull every enabled files trigger, filter glob +
/// ops in Rust (empty ops array means "any op").
async fn list_matching_files(
&self,
app_id: AppId,
collection: &str,
op: FilesEventOp,
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError>;
/// Dispatcher hot path for dead-letter fan-out. Filters: source /// Dispatcher hot path for dead-letter fan-out. Filters: source
/// (or any-source), originating trigger_id (or any), originating /// (or any-source), originating trigger_id (or any), originating
/// script_id (or any). Each filter is "match OR is_null". /// script_id (or any). Each filter is "match OR is_null".
@@ -555,6 +688,214 @@ impl TriggerRepo for PostgresTriggerRepo {
}) })
} }
async fn create_files_trigger(
&self,
app_id: AppId,
req: CreateFilesTrigger,
) -> Result<Trigger, TriggerRepoError> {
if req.collection_glob.is_empty() {
return Err(TriggerRepoError::Invalid(
"collection_glob must not be empty".into(),
));
}
let mut tx = self.pool.begin().await?;
let parent: TriggerRow = sqlx::query_as(
"INSERT INTO triggers ( \
app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal \
) VALUES ($1, $2, 'files', TRUE, $3, $4, $5, $6, $7) \
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal, created_at, updated_at",
)
.bind(app_id.into_inner())
.bind(req.script_id.into_inner())
.bind(req.dispatch_mode.as_str())
.bind(i32::try_from(req.retry_max_attempts).unwrap_or(3))
.bind(req.retry_backoff.as_str())
.bind(i32::try_from(req.retry_base_ms).unwrap_or(1000))
.bind(req.registered_by_principal.into_inner())
.fetch_one(&mut *tx)
.await?;
let ops_str: Vec<String> = req.ops.iter().map(|o| o.as_str().to_string()).collect();
sqlx::query(
"INSERT INTO files_trigger_details (trigger_id, collection_glob, ops) \
VALUES ($1, $2, $3)",
)
.bind(parent.id)
.bind(&req.collection_glob)
.bind(&ops_str)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(Trigger {
id: parent.id.into(),
app_id: parent.app_id.into(),
script_id: parent.script_id.into(),
kind: TriggerKind::Files,
enabled: parent.enabled,
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
.unwrap_or(BackoffShape::Exponential),
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
registered_by_principal: parent.registered_by_principal.into(),
created_at: parent.created_at,
updated_at: parent.updated_at,
details: TriggerDetails::Files {
collection_glob: req.collection_glob,
ops: req.ops,
},
})
}
async fn create_pubsub_trigger(
&self,
app_id: AppId,
req: CreatePubsubTrigger,
) -> Result<Trigger, TriggerRepoError> {
// Defense-in-depth validation (the admin endpoint validates too).
picloud_shared::validate_topic_pattern(&req.topic_pattern)
.map_err(TriggerRepoError::Invalid)?;
let mut tx = self.pool.begin().await?;
let parent: TriggerRow = sqlx::query_as(
"INSERT INTO triggers ( \
app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal \
) VALUES ($1, $2, 'pubsub', TRUE, $3, $4, $5, $6, $7) \
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal, created_at, updated_at",
)
.bind(app_id.into_inner())
.bind(req.script_id.into_inner())
.bind(req.dispatch_mode.as_str())
.bind(i32::try_from(req.retry_max_attempts).unwrap_or(3))
.bind(req.retry_backoff.as_str())
.bind(i32::try_from(req.retry_base_ms).unwrap_or(1000))
.bind(req.registered_by_principal.into_inner())
.fetch_one(&mut *tx)
.await?;
sqlx::query(
"INSERT INTO pubsub_trigger_details (trigger_id, topic_pattern) VALUES ($1, $2)",
)
.bind(parent.id)
.bind(&req.topic_pattern)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(Trigger {
id: parent.id.into(),
app_id: parent.app_id.into(),
script_id: parent.script_id.into(),
kind: TriggerKind::Pubsub,
enabled: parent.enabled,
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
.unwrap_or(BackoffShape::Exponential),
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
registered_by_principal: parent.registered_by_principal.into(),
created_at: parent.created_at,
updated_at: parent.updated_at,
details: TriggerDetails::Pubsub {
topic_pattern: req.topic_pattern,
},
})
}
async fn create_email_trigger(
&self,
app_id: AppId,
req: CreateEmailTrigger,
) -> Result<Trigger, TriggerRepoError> {
let has_inbound_secret = req.inbound_secret_encrypted.is_some();
let mut tx = self.pool.begin().await?;
// Inbound email is delivered async like every other fan-out
// event; the receiver enqueues an outbox row the dispatcher
// picks up. Retry settings use the standard defaults.
let parent: TriggerRow = sqlx::query_as(
"INSERT INTO triggers ( \
app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal \
) VALUES ($1, $2, 'email', TRUE, 'async', 3, 'exponential', 1000, $3) \
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
retry_max_attempts, retry_backoff, retry_base_ms, \
registered_by_principal, created_at, updated_at",
)
.bind(app_id.into_inner())
.bind(req.script_id.into_inner())
.bind(req.registered_by_principal.into_inner())
.fetch_one(&mut *tx)
.await?;
sqlx::query(
"INSERT INTO email_trigger_details \
(trigger_id, inbound_secret_encrypted, inbound_secret_nonce) \
VALUES ($1, $2, $3)",
)
.bind(parent.id)
.bind(req.inbound_secret_encrypted.as_deref())
.bind(req.inbound_secret_nonce.as_deref())
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok(Trigger {
id: parent.id.into(),
app_id: parent.app_id.into(),
script_id: parent.script_id.into(),
kind: TriggerKind::Email,
enabled: parent.enabled,
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
.unwrap_or(BackoffShape::Exponential),
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
registered_by_principal: parent.registered_by_principal.into(),
created_at: parent.created_at,
updated_at: parent.updated_at,
details: TriggerDetails::Email { has_inbound_secret },
})
}
async fn email_inbound_target(
&self,
trigger_id: TriggerId,
) -> Result<Option<EmailInboundTarget>, TriggerRepoError> {
let row: Option<EmailInboundRow> = sqlx::query_as(
"SELECT t.app_id, t.script_id, t.enabled, t.dispatch_mode, \
t.registered_by_principal, \
d.inbound_secret_encrypted, d.inbound_secret_nonce \
FROM triggers t \
JOIN email_trigger_details d ON d.trigger_id = t.id \
WHERE t.id = $1 AND t.kind = 'email'",
)
.bind(trigger_id.into_inner())
.fetch_optional(&self.pool)
.await?;
Ok(row.map(|r| EmailInboundTarget {
app_id: r.app_id.into(),
script_id: r.script_id.into(),
enabled: r.enabled,
dispatch_mode: dispatch_from_str(&r.dispatch_mode),
registered_by_principal: r.registered_by_principal.into(),
inbound_secret_encrypted: r.inbound_secret_encrypted,
inbound_secret_nonce: r.inbound_secret_nonce,
}))
}
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> { async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> {
let parents: Vec<TriggerRow> = sqlx::query_as( let parents: Vec<TriggerRow> = sqlx::query_as(
"SELECT id, app_id, script_id, kind, enabled, dispatch_mode, \ "SELECT id, app_id, script_id, kind, enabled, dispatch_mode, \
@@ -693,6 +1034,51 @@ impl TriggerRepo for PostgresTriggerRepo {
Ok(out) Ok(out)
} }
async fn list_matching_files(
&self,
app_id: AppId,
collection: &str,
op: FilesEventOp,
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError> {
// Mirrors list_matching_kv: pull every enabled files trigger,
// filter glob + ops in Rust (empty ops array means "any op").
let rows: Vec<KvMatchRow> = sqlx::query_as(
"SELECT t.id, t.script_id, t.dispatch_mode, \
t.retry_max_attempts, t.retry_backoff, t.retry_base_ms, \
t.registered_by_principal, \
d.collection_glob, d.ops \
FROM triggers t \
JOIN files_trigger_details d ON d.trigger_id = t.id \
WHERE t.app_id = $1 AND t.kind = 'files' AND t.enabled = TRUE",
)
.bind(app_id.into_inner())
.fetch_all(&self.pool)
.await?;
let op_str = op.as_str();
let mut out = Vec::new();
for r in rows {
if !collection_matches(&r.collection_glob, collection) {
continue;
}
let any_op = r.ops.is_empty();
if !any_op && !r.ops.iter().any(|o| o == op_str) {
continue;
}
out.push(FilesTriggerMatch {
trigger_id: r.id.into(),
script_id: r.script_id.into(),
dispatch_mode: dispatch_from_str(&r.dispatch_mode),
retry_max_attempts: u32::try_from(r.retry_max_attempts).unwrap_or(3),
retry_backoff: BackoffShape::from_wire(&r.retry_backoff)
.unwrap_or(BackoffShape::Exponential),
retry_base_ms: u32::try_from(r.retry_base_ms).unwrap_or(1000),
registered_by_principal: r.registered_by_principal.into(),
});
}
Ok(out)
}
async fn list_matching_dead_letter( async fn list_matching_dead_letter(
&self, &self,
app_id: AppId, app_id: AppId,
@@ -729,6 +1115,7 @@ impl TriggerRepo for PostgresTriggerRepo {
} }
} }
#[allow(clippy::too_many_lines)]
async fn hydrate_one(pool: &PgPool, parent: TriggerRow) -> Result<Trigger, TriggerRepoError> { async fn hydrate_one(pool: &PgPool, parent: TriggerRow) -> Result<Trigger, TriggerRepoError> {
let kind = TriggerKind::from_wire(&parent.kind).ok_or_else(|| { let kind = TriggerKind::from_wire(&parent.kind).ok_or_else(|| {
TriggerRepoError::Invalid(format!("unknown trigger kind {}", parent.kind)) TriggerRepoError::Invalid(format!("unknown trigger kind {}", parent.kind))
@@ -797,6 +1184,45 @@ async fn hydrate_one(pool: &PgPool, parent: TriggerRow) -> Result<Trigger, Trigg
last_fired_at: row.last_fired_at, last_fired_at: row.last_fired_at,
} }
} }
TriggerKind::Files => {
let row: KvDetailRow = sqlx::query_as(
"SELECT collection_glob, ops FROM files_trigger_details WHERE trigger_id = $1",
)
.bind(parent.id)
.fetch_one(pool)
.await?;
let ops = row
.ops
.iter()
.filter_map(|s| FilesEventOp::from_wire(s))
.collect();
TriggerDetails::Files {
collection_glob: row.collection_glob,
ops,
}
}
TriggerKind::Pubsub => {
let row: PubsubDetailRow = sqlx::query_as(
"SELECT topic_pattern FROM pubsub_trigger_details WHERE trigger_id = $1",
)
.bind(parent.id)
.fetch_one(pool)
.await?;
TriggerDetails::Pubsub {
topic_pattern: row.topic_pattern,
}
}
TriggerKind::Email => {
let row: EmailDetailRow = sqlx::query_as(
"SELECT inbound_secret_encrypted FROM email_trigger_details WHERE trigger_id = $1",
)
.bind(parent.id)
.fetch_one(pool)
.await?;
TriggerDetails::Email {
has_inbound_secret: row.inbound_secret_encrypted.is_some(),
}
}
}; };
Ok(Trigger { Ok(Trigger {
@@ -869,6 +1295,27 @@ struct CronDetailRow {
last_fired_at: Option<DateTime<Utc>>, last_fired_at: Option<DateTime<Utc>>,
} }
#[derive(sqlx::FromRow)]
struct PubsubDetailRow {
topic_pattern: String,
}
#[derive(sqlx::FromRow)]
struct EmailDetailRow {
inbound_secret_encrypted: Option<Vec<u8>>,
}
#[derive(sqlx::FromRow)]
struct EmailInboundRow {
app_id: Uuid,
script_id: Uuid,
enabled: bool,
dispatch_mode: String,
registered_by_principal: Uuid,
inbound_secret_encrypted: Option<Vec<u8>>,
inbound_secret_nonce: Option<Vec<u8>>,
}
#[derive(sqlx::FromRow)] #[derive(sqlx::FromRow)]
#[allow(clippy::struct_field_names)] #[allow(clippy::struct_field_names)]
struct DlDetailRow { struct DlDetailRow {

View File

@@ -16,17 +16,22 @@ use axum::http::StatusCode;
use axum::response::{IntoResponse, Json, Response}; use axum::response::{IntoResponse, Json, Response};
use axum::routing::{delete, get, post}; use axum::routing::{delete, get, post};
use axum::{Extension, Router}; use axum::{Extension, Router};
use picloud_shared::{AppId, DocsEventOp, KvEventOp, Principal, ScriptId, ScriptKind, TriggerId}; use picloud_shared::{
AppId, DocsEventOp, FilesEventOp, KvEventOp, MasterKey, Principal, ScriptId, ScriptKind,
TriggerId,
};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use serde_json::json; use serde_json::json;
use crate::app_repo::AppRepository; use crate::app_repo::AppRepository;
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability}; use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
use crate::repo::{ScriptRepository, ScriptRepositoryError}; use crate::repo::{ScriptRepository, ScriptRepositoryError};
use crate::secrets_service::seal;
use crate::trigger_config::{BackoffShape, TriggerConfig}; use crate::trigger_config::{BackoffShape, TriggerConfig};
use crate::trigger_repo::{ use crate::trigger_repo::{
CreateCronTrigger, CreateDeadLetterTrigger, CreateDocsTrigger, CreateKvTrigger, Trigger, CreateCronTrigger, CreateDeadLetterTrigger, CreateDocsTrigger, CreateEmailTrigger,
TriggerDispatchMode, TriggerRepo, TriggerRepoError, CreateFilesTrigger, CreateKvTrigger, CreatePubsubTrigger, Trigger, TriggerDispatchMode,
TriggerRepo, TriggerRepoError,
}; };
#[derive(Clone)] #[derive(Clone)]
@@ -43,6 +48,9 @@ pub struct TriggersState {
/// retry settings. Kept on the state struct so tests can swap /// retry settings. Kept on the state struct so tests can swap
/// in a stricter / looser config without env tinkering. /// in a stricter / looser config without env tinkering.
pub config: TriggerConfig, pub config: TriggerConfig,
/// v1.1.7: master key used to encrypt an email trigger's inbound HMAC
/// secret before it's stored.
pub master_key: MasterKey,
} }
pub fn triggers_router(state: TriggersState) -> Router { pub fn triggers_router(state: TriggersState) -> Router {
@@ -54,10 +62,16 @@ pub fn triggers_router(state: TriggersState) -> Router {
.route("/apps/{app_id}/triggers/kv", post(create_kv_trigger)) .route("/apps/{app_id}/triggers/kv", post(create_kv_trigger))
.route("/apps/{app_id}/triggers/docs", post(create_docs_trigger)) .route("/apps/{app_id}/triggers/docs", post(create_docs_trigger))
.route("/apps/{app_id}/triggers/cron", post(create_cron_trigger)) .route("/apps/{app_id}/triggers/cron", post(create_cron_trigger))
.route("/apps/{app_id}/triggers/files", post(create_files_trigger))
.route(
"/apps/{app_id}/triggers/pubsub",
post(create_pubsub_trigger),
)
.route( .route(
"/apps/{app_id}/triggers/dead_letter", "/apps/{app_id}/triggers/dead_letter",
post(create_dl_trigger), post(create_dl_trigger),
) )
.route("/apps/{app_id}/triggers/email", post(create_email_trigger))
.route( .route(
"/apps/{app_id}/triggers/{trigger_id}", "/apps/{app_id}/triggers/{trigger_id}",
delete(delete_trigger), delete(delete_trigger),
@@ -139,6 +153,24 @@ fn default_timezone() -> String {
"UTC".to_string() "UTC".to_string()
} }
/// v1.1.5 files trigger. Mirrors `CreateKvTriggerRequest`; `ops` uses
/// `FilesEventOp` (`create` / `update` / `delete`).
#[derive(Debug, Deserialize)]
pub struct CreateFilesTriggerRequest {
pub script_id: ScriptId,
pub collection_glob: String,
#[serde(default)]
pub ops: Vec<FilesEventOp>,
#[serde(default = "default_dispatch")]
pub dispatch_mode: TriggerDispatchMode,
#[serde(default)]
pub retry_max_attempts: Option<u32>,
#[serde(default)]
pub retry_backoff: Option<BackoffShape>,
#[serde(default)]
pub retry_base_ms: Option<u32>,
}
#[derive(Debug, Deserialize)] #[derive(Debug, Deserialize)]
pub struct CreateDeadLetterTriggerRequest { pub struct CreateDeadLetterTriggerRequest {
pub script_id: ScriptId, pub script_id: ScriptId,
@@ -328,6 +360,94 @@ async fn create_cron_trigger(
Ok((StatusCode::CREATED, Json(created))) Ok((StatusCode::CREATED, Json(created)))
} }
/// v1.1.5 pubsub trigger. `topic_pattern` is validated to be exact /
/// `<prefix>.*` / `*`.
#[derive(Debug, Deserialize)]
pub struct CreatePubsubTriggerRequest {
pub script_id: ScriptId,
pub topic_pattern: String,
#[serde(default = "default_dispatch")]
pub dispatch_mode: TriggerDispatchMode,
#[serde(default)]
pub retry_max_attempts: Option<u32>,
#[serde(default)]
pub retry_backoff: Option<BackoffShape>,
#[serde(default)]
pub retry_base_ms: Option<u32>,
}
async fn create_pubsub_trigger(
State(s): State<TriggersState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Json(input): Json<CreatePubsubTriggerRequest>,
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppManageTriggers(app_id),
)
.await?;
// Validate the topic pattern before touching the script repo so a
// bad pattern fails fast with a clear 422.
picloud_shared::validate_topic_pattern(&input.topic_pattern)
.map_err(TriggersApiError::Invalid)?;
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
let req = CreatePubsubTrigger {
script_id: input.script_id,
topic_pattern: input.topic_pattern,
dispatch_mode: input.dispatch_mode,
retry_max_attempts: input
.retry_max_attempts
.unwrap_or(s.config.retry_max_attempts),
retry_backoff: input.retry_backoff.unwrap_or(s.config.retry_backoff),
retry_base_ms: input.retry_base_ms.unwrap_or(s.config.retry_base_ms),
registered_by_principal: principal.user_id,
};
let created = s.triggers.create_pubsub_trigger(app_id, req).await?;
Ok((StatusCode::CREATED, Json(created)))
}
async fn create_files_trigger(
State(s): State<TriggersState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Json(input): Json<CreateFilesTriggerRequest>,
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppManageTriggers(app_id),
)
.await?;
if input.collection_glob.trim().is_empty() {
return Err(TriggersApiError::Invalid(
"collection_glob must not be empty".into(),
));
}
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
let req = CreateFilesTrigger {
script_id: input.script_id,
collection_glob: input.collection_glob,
ops: input.ops,
dispatch_mode: input.dispatch_mode,
retry_max_attempts: input
.retry_max_attempts
.unwrap_or(s.config.retry_max_attempts),
retry_backoff: input.retry_backoff.unwrap_or(s.config.retry_backoff),
retry_base_ms: input.retry_base_ms.unwrap_or(s.config.retry_base_ms),
registered_by_principal: principal.user_id,
};
let created = s.triggers.create_files_trigger(app_id, req).await?;
Ok((StatusCode::CREATED, Json(created)))
}
async fn create_dl_trigger( async fn create_dl_trigger(
State(s): State<TriggersState>, State(s): State<TriggersState>,
Extension(principal): Extension<Principal>, Extension(principal): Extension<Principal>,
@@ -353,6 +473,60 @@ async fn create_dl_trigger(
Ok((StatusCode::CREATED, Json(created))) Ok((StatusCode::CREATED, Json(created)))
} }
#[derive(Debug, Deserialize)]
struct CreateEmailTriggerRequest {
script_id: ScriptId,
/// Shared HMAC secret the provider signs inbound POSTs with. `null`
/// (or absent) means the trigger accepts unsigned POSTs.
#[serde(default)]
inbound_secret: Option<String>,
}
async fn create_email_trigger(
State(s): State<TriggersState>,
Extension(principal): Extension<Principal>,
Path(app_id): Path<AppId>,
Json(input): Json<CreateEmailTriggerRequest>,
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
ensure_app_exists(&*s.apps, app_id).await?;
require(
s.authz.as_ref(),
&principal,
Capability::AppManageTriggers(app_id),
)
.await?;
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
// Encrypt the inbound HMAC secret at rest (user-approved deviation
// from the brief's plaintext column). An empty/whitespace secret is
// treated as "no secret" (unsigned trigger).
let (inbound_secret_encrypted, inbound_secret_nonce) = match input.inbound_secret {
Some(secret) if !secret.trim().is_empty() => {
// 64 KB cap is irrelevant for a signing secret, but `seal`
// takes one; reuse the secrets default.
let (ct, nonce) = seal(
&s.master_key,
&serde_json::Value::String(secret),
crate::secrets_service::DEFAULT_SECRET_MAX_VALUE_BYTES,
)
.map_err(|e| {
TriggersApiError::Invalid(format!("could not seal inbound_secret: {e}"))
})?;
(Some(ct), Some(nonce.to_vec()))
}
_ => (None, None),
};
let req = CreateEmailTrigger {
script_id: input.script_id,
inbound_secret_encrypted,
inbound_secret_nonce,
registered_by_principal: principal.user_id,
};
let created = s.triggers.create_email_trigger(app_id, req).await?;
Ok((StatusCode::CREATED, Json(created)))
}
async fn delete_trigger( async fn delete_trigger(
State(s): State<TriggersState>, State(s): State<TriggersState>,
Extension(principal): Extension<Principal>, Extension(principal): Extension<Principal>,
@@ -484,13 +658,15 @@ mod tests {
use super::*; use super::*;
use crate::app_repo::{AppLookup, AppRepository}; use crate::app_repo::{AppLookup, AppRepository};
use crate::trigger_repo::{ use crate::trigger_repo::{
CreateCronTrigger, DeadLetterTriggerMatch, DocsTriggerMatch, KvTriggerMatch, Trigger, CreateCronTrigger, CreateEmailTrigger, CreateFilesTrigger, CreatePubsubTrigger,
TriggerDetails, TriggerRepo, TriggerRepoError, DeadLetterTriggerMatch, DocsTriggerMatch, EmailInboundTarget, FilesTriggerMatch,
KvTriggerMatch, Trigger, TriggerDetails, TriggerKind, TriggerRepo, TriggerRepoError,
}; };
use async_trait::async_trait; use async_trait::async_trait;
use chrono::Utc; use chrono::Utc;
use picloud_shared::{ use picloud_shared::{
AdminUserId, App, AppRole, DocsEventOp, KvEventOp, ScriptId, TriggerId, UserId, AdminUserId, App, AppRole, DocsEventOp, FilesEventOp, KvEventOp, ScriptId, TriggerId,
UserId,
}; };
use std::collections::HashMap; use std::collections::HashMap;
use tokio::sync::Mutex; use tokio::sync::Mutex;
@@ -587,6 +763,50 @@ mod tests {
self.inner.lock().await.insert(id, trigger.clone()); self.inner.lock().await.insert(id, trigger.clone());
Ok(trigger) Ok(trigger)
} }
async fn create_email_trigger(
&self,
app_id: AppId,
req: CreateEmailTrigger,
) -> Result<Trigger, TriggerRepoError> {
let now = Utc::now();
let id = TriggerId::new();
let trigger = Trigger {
id,
app_id,
script_id: req.script_id,
kind: TriggerKind::Email,
enabled: true,
dispatch_mode: TriggerDispatchMode::Async,
retry_max_attempts: 3,
retry_backoff: BackoffShape::Exponential,
retry_base_ms: 1000,
registered_by_principal: req.registered_by_principal,
created_at: now,
updated_at: now,
details: TriggerDetails::Email {
has_inbound_secret: req.inbound_secret_encrypted.is_some(),
},
};
self.inner.lock().await.insert(id, trigger.clone());
Ok(trigger)
}
async fn email_inbound_target(
&self,
trigger_id: TriggerId,
) -> Result<Option<EmailInboundTarget>, TriggerRepoError> {
let g = self.inner.lock().await;
Ok(g.get(&trigger_id)
.filter(|t| t.kind == TriggerKind::Email)
.map(|t| EmailInboundTarget {
app_id: t.app_id,
script_id: t.script_id,
enabled: t.enabled,
dispatch_mode: t.dispatch_mode,
registered_by_principal: t.registered_by_principal,
inbound_secret_encrypted: None,
inbound_secret_nonce: None,
}))
}
async fn create_cron_trigger( async fn create_cron_trigger(
&self, &self,
app_id: AppId, app_id: AppId,
@@ -616,6 +836,61 @@ mod tests {
self.inner.lock().await.insert(id, trigger.clone()); self.inner.lock().await.insert(id, trigger.clone());
Ok(trigger) Ok(trigger)
} }
async fn create_files_trigger(
&self,
app_id: AppId,
req: CreateFilesTrigger,
) -> Result<Trigger, TriggerRepoError> {
let now = Utc::now();
let id = TriggerId::new();
let trigger = Trigger {
id,
app_id,
script_id: req.script_id,
kind: crate::trigger_repo::TriggerKind::Files,
enabled: true,
dispatch_mode: req.dispatch_mode,
retry_max_attempts: req.retry_max_attempts,
retry_backoff: req.retry_backoff,
retry_base_ms: req.retry_base_ms,
registered_by_principal: req.registered_by_principal,
created_at: now,
updated_at: now,
details: TriggerDetails::Files {
collection_glob: req.collection_glob,
ops: req.ops,
},
};
self.inner.lock().await.insert(id, trigger.clone());
Ok(trigger)
}
async fn create_pubsub_trigger(
&self,
app_id: AppId,
req: CreatePubsubTrigger,
) -> Result<Trigger, TriggerRepoError> {
let now = Utc::now();
let id = TriggerId::new();
let trigger = Trigger {
id,
app_id,
script_id: req.script_id,
kind: crate::trigger_repo::TriggerKind::Pubsub,
enabled: true,
dispatch_mode: req.dispatch_mode,
retry_max_attempts: req.retry_max_attempts,
retry_backoff: req.retry_backoff,
retry_base_ms: req.retry_base_ms,
registered_by_principal: req.registered_by_principal,
created_at: now,
updated_at: now,
details: TriggerDetails::Pubsub {
topic_pattern: req.topic_pattern,
},
};
self.inner.lock().await.insert(id, trigger.clone());
Ok(trigger)
}
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> { async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> {
Ok(self Ok(self
.inner .inner
@@ -648,6 +923,14 @@ mod tests {
) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError> { ) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError> {
Ok(vec![]) Ok(vec![])
} }
async fn list_matching_files(
&self,
_app_id: AppId,
_collection: &str,
_op: FilesEventOp,
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError> {
Ok(vec![])
}
async fn list_matching_dead_letter( async fn list_matching_dead_letter(
&self, &self,
_app_id: AppId, _app_id: AppId,
@@ -922,6 +1205,7 @@ mod tests {
authz, authz,
scripts: InMemoryScriptRepo::empty(), scripts: InMemoryScriptRepo::empty(),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
} }
} }
@@ -939,6 +1223,7 @@ mod tests {
authz, authz,
scripts: InMemoryScriptRepo::with_endpoint(app_id, script_id), scripts: InMemoryScriptRepo::with_endpoint(app_id, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
} }
} }
@@ -1211,6 +1496,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id), scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_kv_trigger( let res = create_kv_trigger(
State(state), State(state),
@@ -1248,6 +1534,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id), scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_docs_trigger( let res = create_docs_trigger(
State(state), State(state),
@@ -1282,6 +1569,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id), scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_dl_trigger( let res = create_dl_trigger(
State(state), State(state),
@@ -1347,6 +1635,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts, scripts,
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_kv_trigger( let res = create_kv_trigger(
State(state), State(state),
@@ -1477,6 +1766,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id), scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_cron_trigger( let res = create_cron_trigger(
State(state), State(state),
@@ -1506,6 +1796,7 @@ mod tests {
authz: Arc::new(AlwaysAllowAuthzRepo), authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id), scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
config: TriggerConfig::conservative(), config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
}; };
let res = create_cron_trigger( let res = create_cron_trigger(
State(state), State(state),
@@ -1560,4 +1851,262 @@ mod tests {
.expect("endpoint target should succeed"); .expect("endpoint target should succeed");
assert_eq!(status, StatusCode::CREATED); assert_eq!(status, StatusCode::CREATED);
} }
// ----------------------------------------------------------------
// v1.1.5: files + pubsub trigger create (Layout-E reject coverage).
// ----------------------------------------------------------------
fn files_req(script_id: ScriptId, glob: &str) -> CreateFilesTriggerRequest {
CreateFilesTriggerRequest {
script_id,
collection_glob: glob.into(),
ops: vec![FilesEventOp::Create],
dispatch_mode: TriggerDispatchMode::Async,
retry_max_attempts: None,
retry_backoff: None,
retry_base_ms: None,
}
}
#[tokio::test]
async fn files_trigger_create_succeeds() {
let app_id = AppId::new();
let script_id = ScriptId::new();
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
let (status, Json(trigger)) = create_files_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(files_req(script_id, "avatars")),
)
.await
.unwrap();
assert_eq!(status, StatusCode::CREATED);
assert!(matches!(
trigger.kind,
crate::trigger_repo::TriggerKind::Files
));
match trigger.details {
TriggerDetails::Files {
collection_glob,
ops,
} => {
assert_eq!(collection_glob, "avatars");
assert_eq!(ops, vec![FilesEventOp::Create]);
}
other => panic!("expected Files details, got {other:?}"),
}
}
#[tokio::test]
async fn files_trigger_empty_glob_rejected() {
let app_id = AppId::new();
let state = state_with(Arc::new(AlwaysAllowAuthzRepo), app_id);
let res = create_files_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(files_req(ScriptId::new(), " ")),
)
.await;
assert!(matches!(
res.expect_err("empty glob"),
TriggersApiError::Invalid(_)
));
}
#[tokio::test]
async fn files_trigger_rejects_module_target() {
let app_id = AppId::new();
let script_id = ScriptId::new();
let state = TriggersState {
triggers: Arc::new(InMemoryTriggerRepo::default()),
apps: InMemoryAppRepo::with(app_id),
authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
};
let res = create_files_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(files_req(script_id, "avatars")),
)
.await;
let msg = match res.expect_err("module rejected") {
TriggersApiError::Invalid(m) => m,
other => panic!("expected Invalid, got {other:?}"),
};
assert!(msg.to_lowercase().contains("module"));
}
#[tokio::test]
async fn files_trigger_rejects_cross_app_script() {
let app_a = AppId::new();
let app_b = AppId::new();
let script_id = ScriptId::new();
let state = TriggersState {
triggers: Arc::new(InMemoryTriggerRepo::default()),
apps: InMemoryAppRepo::with(app_a),
authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
};
let res = create_files_trigger(
State(state),
Extension(member_principal()),
Path(app_a),
Json(files_req(script_id, "avatars")),
)
.await;
let msg = match res.expect_err("cross-app rejected") {
TriggersApiError::Invalid(m) => m,
other => panic!("expected Invalid, got {other:?}"),
};
assert!(msg.to_lowercase().contains("does not belong"));
}
#[tokio::test]
async fn files_trigger_member_without_role_is_forbidden() {
let app_id = AppId::new();
let state = state_with(Arc::new(AlwaysDenyAuthzRepo), app_id);
let res = create_files_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(files_req(ScriptId::new(), "avatars")),
)
.await;
assert!(matches!(
res.expect_err("forbidden"),
TriggersApiError::Forbidden
));
}
fn pubsub_req(script_id: ScriptId, pattern: &str) -> CreatePubsubTriggerRequest {
CreatePubsubTriggerRequest {
script_id,
topic_pattern: pattern.into(),
dispatch_mode: TriggerDispatchMode::Async,
retry_max_attempts: None,
retry_backoff: None,
retry_base_ms: None,
}
}
#[tokio::test]
async fn pubsub_trigger_create_succeeds() {
let app_id = AppId::new();
let script_id = ScriptId::new();
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
let (status, Json(trigger)) = create_pubsub_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(pubsub_req(script_id, "user.*")),
)
.await
.unwrap();
assert_eq!(status, StatusCode::CREATED);
match trigger.details {
TriggerDetails::Pubsub { topic_pattern } => assert_eq!(topic_pattern, "user.*"),
other => panic!("expected Pubsub details, got {other:?}"),
}
}
#[tokio::test]
async fn pubsub_trigger_rejects_bad_pattern() {
let app_id = AppId::new();
let script_id = ScriptId::new();
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
for bad in ["*.created", "a.*.b", "**"] {
let res = create_pubsub_trigger(
State(state.clone()),
Extension(member_principal()),
Path(app_id),
Json(pubsub_req(script_id, bad)),
)
.await;
let msg = match res.expect_err("bad pattern") {
TriggersApiError::Invalid(m) => m,
other => panic!("expected Invalid, got {other:?}"),
};
assert!(
msg.contains("unsupported pubsub topic pattern"),
"got {msg} for {bad}"
);
}
}
#[tokio::test]
async fn pubsub_trigger_rejects_module_target() {
let app_id = AppId::new();
let script_id = ScriptId::new();
let state = TriggersState {
triggers: Arc::new(InMemoryTriggerRepo::default()),
apps: InMemoryAppRepo::with(app_id),
authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_module(app_id, script_id),
config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
};
let res = create_pubsub_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(pubsub_req(script_id, "user.*")),
)
.await;
let msg = match res.expect_err("module rejected") {
TriggersApiError::Invalid(m) => m,
other => panic!("expected Invalid, got {other:?}"),
};
assert!(msg.to_lowercase().contains("module"));
}
#[tokio::test]
async fn pubsub_trigger_rejects_cross_app_script() {
let app_a = AppId::new();
let app_b = AppId::new();
let script_id = ScriptId::new();
let state = TriggersState {
triggers: Arc::new(InMemoryTriggerRepo::default()),
apps: InMemoryAppRepo::with(app_a),
authz: Arc::new(AlwaysAllowAuthzRepo),
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
config: TriggerConfig::conservative(),
master_key: picloud_shared::MasterKey::from_bytes([0u8; 32]),
};
let res = create_pubsub_trigger(
State(state),
Extension(member_principal()),
Path(app_a),
Json(pubsub_req(script_id, "user.*")),
)
.await;
let msg = match res.expect_err("cross-app rejected") {
TriggersApiError::Invalid(m) => m,
other => panic!("expected Invalid, got {other:?}"),
};
assert!(msg.to_lowercase().contains("does not belong"));
}
#[tokio::test]
async fn pubsub_trigger_member_without_role_is_forbidden() {
let app_id = AppId::new();
let state = state_with(Arc::new(AlwaysDenyAuthzRepo), app_id);
let res = create_pubsub_trigger(
State(state),
Extension(member_principal()),
Path(app_id),
Json(pubsub_req(ScriptId::new(), "user.*")),
)
.await;
assert!(matches!(
res.expect_err("forbidden"),
TriggersApiError::Forbidden
));
}
} }

View File

@@ -58,6 +58,14 @@ table: app_members
role: text NOT NULL role: text NOT NULL
created_at: timestamp with time zone NOT NULL default=now() created_at: timestamp with time zone NOT NULL default=now()
table: app_secrets
app_id: uuid NOT NULL
realtime_signing_key: bytea NULL
created_at: timestamp with time zone NOT NULL default=now()
updated_at: timestamp with time zone NOT NULL default=now()
realtime_signing_key_encrypted: bytea NULL
realtime_signing_key_nonce: bytea NULL
table: app_slug_history table: app_slug_history
slug: text NOT NULL slug: text NOT NULL
current_app_id: uuid NOT NULL current_app_id: uuid NOT NULL
@@ -113,6 +121,11 @@ table: docs_trigger_details
collection_glob: text NOT NULL collection_glob: text NOT NULL
ops: ARRAY NOT NULL ops: ARRAY NOT NULL
table: email_trigger_details
trigger_id: uuid NOT NULL
inbound_secret_encrypted: bytea NULL
inbound_secret_nonce: bytea NULL
table: execution_logs table: execution_logs
id: uuid NOT NULL default=gen_random_uuid() id: uuid NOT NULL default=gen_random_uuid()
script_id: uuid NOT NULL script_id: uuid NOT NULL
@@ -128,6 +141,22 @@ table: execution_logs
created_at: timestamp with time zone NOT NULL default=now() created_at: timestamp with time zone NOT NULL default=now()
app_id: uuid NOT NULL app_id: uuid NOT NULL
table: files
app_id: uuid NOT NULL
collection: text NOT NULL
id: uuid NOT NULL
name: text NOT NULL
content_type: text NOT NULL
size_bytes: bigint NOT NULL
checksum_sha256: text NOT NULL
created_at: timestamp with time zone NOT NULL default=now()
updated_at: timestamp with time zone NOT NULL default=now()
table: files_trigger_details
trigger_id: uuid NOT NULL
collection_glob: text NOT NULL
ops: ARRAY NOT NULL
table: kv_entries table: kv_entries
app_id: uuid NOT NULL app_id: uuid NOT NULL
collection: text NOT NULL collection: text NOT NULL
@@ -158,6 +187,10 @@ table: outbox
claimed_by: text NULL claimed_by: text NULL
created_at: timestamp with time zone NOT NULL default=now() created_at: timestamp with time zone NOT NULL default=now()
table: pubsub_trigger_details
trigger_id: uuid NOT NULL
topic_pattern: text NOT NULL
table: routes table: routes
id: uuid NOT NULL default=gen_random_uuid() id: uuid NOT NULL default=gen_random_uuid()
script_id: uuid NOT NULL script_id: uuid NOT NULL
@@ -191,6 +224,22 @@ table: scripts
app_id: uuid NOT NULL app_id: uuid NOT NULL
kind: text NOT NULL default='endpoint'::text kind: text NOT NULL default='endpoint'::text
table: secrets
app_id: uuid NOT NULL
name: text NOT NULL
encrypted_value: bytea NOT NULL
nonce: bytea NOT NULL
created_at: timestamp with time zone NOT NULL default=now()
updated_at: timestamp with time zone NOT NULL default=now()
table: topics
app_id: uuid NOT NULL
name: text NOT NULL
external_subscribable: boolean NOT NULL default=false
auth_mode: text NOT NULL default='public'::text
created_at: timestamp with time zone NOT NULL default=now()
updated_at: timestamp with time zone NOT NULL default=now()
table: triggers table: triggers
id: uuid NOT NULL default=gen_random_uuid() id: uuid NOT NULL default=gen_random_uuid()
app_id: uuid NOT NULL app_id: uuid NOT NULL
@@ -236,6 +285,9 @@ indexes on app_members:
app_members_pkey: public.app_members USING btree (app_id, user_id) app_members_pkey: public.app_members USING btree (app_id, user_id)
app_members_user_id_idx: public.app_members USING btree (user_id) app_members_user_id_idx: public.app_members USING btree (user_id)
indexes on app_secrets:
app_secrets_pkey: public.app_secrets USING btree (app_id)
indexes on app_slug_history: indexes on app_slug_history:
app_slug_history_pkey: public.app_slug_history USING btree (slug) app_slug_history_pkey: public.app_slug_history USING btree (slug)
@@ -263,11 +315,21 @@ indexes on docs:
indexes on docs_trigger_details: indexes on docs_trigger_details:
docs_trigger_details_pkey: public.docs_trigger_details USING btree (trigger_id) docs_trigger_details_pkey: public.docs_trigger_details USING btree (trigger_id)
indexes on email_trigger_details:
email_trigger_details_pkey: public.email_trigger_details USING btree (trigger_id)
indexes on execution_logs: indexes on execution_logs:
execution_logs_app_id_created_at_idx: public.execution_logs USING btree (app_id, created_at DESC) execution_logs_app_id_created_at_idx: public.execution_logs USING btree (app_id, created_at DESC)
execution_logs_pkey: public.execution_logs USING btree (id) execution_logs_pkey: public.execution_logs USING btree (id)
execution_logs_script_id_created_at_idx: public.execution_logs USING btree (script_id, created_at DESC) execution_logs_script_id_created_at_idx: public.execution_logs USING btree (script_id, created_at DESC)
indexes on files:
files_pkey: public.files USING btree (app_id, collection, id)
idx_files_app_collection: public.files USING btree (app_id, collection)
indexes on files_trigger_details:
files_trigger_details_pkey: public.files_trigger_details USING btree (trigger_id)
indexes on kv_entries: indexes on kv_entries:
idx_kv_entries_app_collection: public.kv_entries USING btree (app_id, collection) idx_kv_entries_app_collection: public.kv_entries USING btree (app_id, collection)
kv_entries_pkey: public.kv_entries USING btree (app_id, collection, key) kv_entries_pkey: public.kv_entries USING btree (app_id, collection, key)
@@ -280,6 +342,9 @@ indexes on outbox:
idx_outbox_due: public.outbox USING btree (next_attempt_at) WHERE (claimed_at IS NULL) idx_outbox_due: public.outbox USING btree (next_attempt_at) WHERE (claimed_at IS NULL)
outbox_pkey: public.outbox USING btree (id) outbox_pkey: public.outbox USING btree (id)
indexes on pubsub_trigger_details:
pubsub_trigger_details_pkey: public.pubsub_trigger_details USING btree (trigger_id)
indexes on routes: indexes on routes:
routes_app_id_idx: public.routes USING btree (app_id) routes_app_id_idx: public.routes USING btree (app_id)
routes_lookup_idx: public.routes USING btree (host_kind, host) routes_lookup_idx: public.routes USING btree (host_kind, host)
@@ -298,8 +363,16 @@ indexes on scripts:
scripts_name_uidx: public.scripts USING btree (app_id, lower(name)) scripts_name_uidx: public.scripts USING btree (app_id, lower(name))
scripts_pkey: public.scripts USING btree (id) scripts_pkey: public.scripts USING btree (id)
indexes on secrets:
idx_secrets_app: public.secrets USING btree (app_id)
secrets_pkey: public.secrets USING btree (app_id, name)
indexes on topics:
topics_pkey: public.topics USING btree (app_id, name)
indexes on triggers: indexes on triggers:
idx_triggers_app_kind_enabled: public.triggers USING btree (app_id, kind) WHERE (enabled = true) idx_triggers_app_kind_enabled: public.triggers USING btree (app_id, kind) WHERE (enabled = true)
idx_triggers_app_pubsub_enabled: public.triggers USING btree (app_id, kind) WHERE ((enabled = true) AND (kind = 'pubsub'::text))
triggers_pkey: public.triggers USING btree (id) triggers_pkey: public.triggers USING btree (id)
## constraints ## constraints
@@ -335,6 +408,10 @@ constraints on app_members:
[FOREIGN KEY] app_members_user_id_fkey: FOREIGN KEY (user_id) REFERENCES admin_users(id) ON DELETE CASCADE [FOREIGN KEY] app_members_user_id_fkey: FOREIGN KEY (user_id) REFERENCES admin_users(id) ON DELETE CASCADE
[PRIMARY KEY] app_members_pkey: PRIMARY KEY (app_id, user_id) [PRIMARY KEY] app_members_pkey: PRIMARY KEY (app_id, user_id)
constraints on app_secrets:
[FOREIGN KEY] app_secrets_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] app_secrets_pkey: PRIMARY KEY (app_id)
constraints on app_slug_history: constraints on app_slug_history:
[FOREIGN KEY] app_slug_history_current_app_id_fkey: FOREIGN KEY (current_app_id) REFERENCES apps(id) ON DELETE CASCADE [FOREIGN KEY] app_slug_history_current_app_id_fkey: FOREIGN KEY (current_app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] app_slug_history_pkey: PRIMARY KEY (slug) [PRIMARY KEY] app_slug_history_pkey: PRIMARY KEY (slug)
@@ -364,12 +441,24 @@ constraints on docs_trigger_details:
[FOREIGN KEY] docs_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE [FOREIGN KEY] docs_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
[PRIMARY KEY] docs_trigger_details_pkey: PRIMARY KEY (trigger_id) [PRIMARY KEY] docs_trigger_details_pkey: PRIMARY KEY (trigger_id)
constraints on email_trigger_details:
[FOREIGN KEY] email_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
[PRIMARY KEY] email_trigger_details_pkey: PRIMARY KEY (trigger_id)
constraints on execution_logs: constraints on execution_logs:
[CHECK] execution_logs_status_check: CHECK ((status = ANY (ARRAY['success'::text, 'error'::text, 'timeout'::text, 'budget_exceeded'::text]))) [CHECK] execution_logs_status_check: CHECK ((status = ANY (ARRAY['success'::text, 'error'::text, 'timeout'::text, 'budget_exceeded'::text])))
[FOREIGN KEY] execution_logs_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE [FOREIGN KEY] execution_logs_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[FOREIGN KEY] execution_logs_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE CASCADE [FOREIGN KEY] execution_logs_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE CASCADE
[PRIMARY KEY] execution_logs_pkey: PRIMARY KEY (id) [PRIMARY KEY] execution_logs_pkey: PRIMARY KEY (id)
constraints on files:
[FOREIGN KEY] files_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] files_pkey: PRIMARY KEY (app_id, collection, id)
constraints on files_trigger_details:
[FOREIGN KEY] files_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
[PRIMARY KEY] files_trigger_details_pkey: PRIMARY KEY (trigger_id)
constraints on kv_entries: constraints on kv_entries:
[FOREIGN KEY] kv_entries_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE [FOREIGN KEY] kv_entries_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] kv_entries_pkey: PRIMARY KEY (app_id, collection, key) [PRIMARY KEY] kv_entries_pkey: PRIMARY KEY (app_id, collection, key)
@@ -379,10 +468,14 @@ constraints on kv_trigger_details:
[PRIMARY KEY] kv_trigger_details_pkey: PRIMARY KEY (trigger_id) [PRIMARY KEY] kv_trigger_details_pkey: PRIMARY KEY (trigger_id)
constraints on outbox: constraints on outbox:
[CHECK] outbox_source_kind_check: CHECK ((source_kind = ANY (ARRAY['http'::text, 'kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text]))) [CHECK] outbox_source_kind_check: CHECK ((source_kind = ANY (ARRAY['http'::text, 'kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text, 'files'::text, 'pubsub'::text, 'email'::text])))
[FOREIGN KEY] outbox_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE [FOREIGN KEY] outbox_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] outbox_pkey: PRIMARY KEY (id) [PRIMARY KEY] outbox_pkey: PRIMARY KEY (id)
constraints on pubsub_trigger_details:
[FOREIGN KEY] pubsub_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
[PRIMARY KEY] pubsub_trigger_details_pkey: PRIMARY KEY (trigger_id)
constraints on routes: constraints on routes:
[CHECK] routes_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text]))) [CHECK] routes_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text])))
[CHECK] routes_host_kind_check: CHECK ((host_kind = ANY (ARRAY['any'::text, 'strict'::text, 'wildcard'::text]))) [CHECK] routes_host_kind_check: CHECK ((host_kind = ANY (ARRAY['any'::text, 'strict'::text, 'wildcard'::text])))
@@ -405,9 +498,18 @@ constraints on scripts:
[FOREIGN KEY] scripts_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE RESTRICT [FOREIGN KEY] scripts_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE RESTRICT
[PRIMARY KEY] scripts_pkey: PRIMARY KEY (id) [PRIMARY KEY] scripts_pkey: PRIMARY KEY (id)
constraints on secrets:
[FOREIGN KEY] secrets_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] secrets_pkey: PRIMARY KEY (app_id, name)
constraints on topics:
[CHECK] topics_auth_mode_check: CHECK ((auth_mode = ANY (ARRAY['public'::text, 'token'::text])))
[FOREIGN KEY] topics_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[PRIMARY KEY] topics_pkey: PRIMARY KEY (app_id, name)
constraints on triggers: constraints on triggers:
[CHECK] triggers_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text]))) [CHECK] triggers_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text])))
[CHECK] triggers_kind_check: CHECK ((kind = ANY (ARRAY['kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text]))) [CHECK] triggers_kind_check: CHECK ((kind = ANY (ARRAY['kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text, 'files'::text, 'pubsub'::text, 'email'::text])))
[CHECK] triggers_retry_backoff_check: CHECK ((retry_backoff = ANY (ARRAY['exponential'::text, 'linear'::text, 'constant'::text]))) [CHECK] triggers_retry_backoff_check: CHECK ((retry_backoff = ANY (ARRAY['exponential'::text, 'linear'::text, 'constant'::text])))
[FOREIGN KEY] triggers_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE [FOREIGN KEY] triggers_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
[FOREIGN KEY] triggers_registered_by_principal_fkey: FOREIGN KEY (registered_by_principal) REFERENCES admin_users(id) ON DELETE CASCADE [FOREIGN KEY] triggers_registered_by_principal_fkey: FOREIGN KEY (registered_by_principal) REFERENCES admin_users(id) ON DELETE CASCADE
@@ -432,3 +534,11 @@ constraints on triggers:
0015: scripts kind 0015: scripts kind
0016: script imports 0016: script imports
0017: cron triggers 0017: cron triggers
0018: files
0019: files triggers
0020: pubsub triggers
0021: topics
0022: app secrets
0023: secrets
0024: email triggers
0025: encrypt realtime keys

View File

@@ -25,22 +25,46 @@
//! //!
//! Review the resulting diff in the same PR as the new migration. //! Review the resulting diff in the same PR as the new migration.
//! //!
//! Like the orchestrator integration tests, this is `#[ignore]`'d by //! v1.1.5: this test is no longer `#[ignore]`'d. It runs whenever
//! default so plain `cargo test --workspace` stays green without //! `DATABASE_URL` is set (CI wires a `postgres:15` service) and **skips
//! infrastructure. //! cleanly** when it's absent, so plain `cargo test --workspace` stays
//! green on machines without Postgres. Unlike the previous
//! `#[sqlx::test]` form (which spun up an isolated throwaway database),
//! it now applies the migrations against the `DATABASE_URL` database
//! directly — migrations are forward-only and idempotent, and CI's
//! Postgres is fresh, so the structural dump is identical either way.
use std::fmt::Write as _; use std::fmt::Write as _;
use std::path::PathBuf; use std::path::PathBuf;
use sqlx::postgres::PgPoolOptions;
use sqlx::{PgPool, Row}; use sqlx::{PgPool, Row};
const SCHEMA: &str = "public"; const SCHEMA: &str = "public";
const SNAPSHOT_PATH: &str = "tests/expected_schema.txt"; const SNAPSHOT_PATH: &str = "tests/expected_schema.txt";
#[ignore = "needs DATABASE_URL pointing at a running Postgres"] #[tokio::test]
#[sqlx::test(migrations = "./migrations")] async fn schema_after_replay_matches_snapshot() {
async fn schema_after_replay_matches_snapshot(pool: PgPool) { // Skip cleanly when DATABASE_URL is unset so `cargo test --workspace`
// stays green without Postgres. CI sets it (postgres:15 service).
let Ok(url) = std::env::var("DATABASE_URL") else {
eprintln!(
"schema_snapshot: DATABASE_URL unset — skipping. Set it (e.g. \
postgres://picloud:picloud@localhost:5432/picloud) to run this guardrail."
);
return;
};
let pool = PgPoolOptions::new()
.max_connections(1)
.connect(&url)
.await
.expect("connect to DATABASE_URL");
sqlx::migrate!("./migrations")
.run(&pool)
.await
.expect("apply migrations");
let actual = dump_schema(&pool).await; let actual = dump_schema(&pool).await;
let snapshot_file = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(SNAPSHOT_PATH); let snapshot_file = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(SNAPSHOT_PATH);

View File

@@ -23,8 +23,13 @@ chrono.workspace = true
reqwest.workspace = true reqwest.workspace = true
rhai.workspace = true rhai.workspace = true
tokio.workspace = true tokio.workspace = true
tokio-stream.workspace = true
urlencoding.workspace = true urlencoding.workspace = true
# v1.1.3 — top-level script AST cache lives in orchestrator-core's # v1.1.3 — top-level script AST cache lives in orchestrator-core's
# LocalExecutorClient; key is ScriptId, value is `(updated_at, Arc<rhai::AST>)`. # LocalExecutorClient; key is ScriptId, value is `(updated_at, Arc<rhai::AST>)`.
lru.workspace = true lru.workspace = true
[dev-dependencies]
# `ServiceExt::oneshot` for driving the SSE router in unit tests.
tower.workspace = true

View File

@@ -12,6 +12,8 @@ pub mod api;
pub mod client; pub mod client;
pub mod gate; pub mod gate;
pub mod inbox; pub mod inbox;
pub mod realtime;
pub mod realtime_api;
pub mod resolver; pub mod resolver;
pub mod routing; pub mod routing;
@@ -19,4 +21,8 @@ pub use api::{data_plane_router, user_routes_router, DataPlaneState};
pub use client::{ExecutorClient, LocalExecutorClient, RemoteExecutorClient, ScriptIdentity}; pub use client::{ExecutorClient, LocalExecutorClient, RemoteExecutorClient, ScriptIdentity};
pub use gate::{AcquireError, ExecutionGate}; pub use gate::{AcquireError, ExecutionGate};
pub use inbox::InboxRegistry; pub use inbox::InboxRegistry;
pub use realtime::{spawn_realtime_gc, InProcessBroadcaster, DEFAULT_BROADCAST_CAPACITY};
pub use realtime_api::{
heartbeat_secs_from_env, realtime_router, RealtimeState, DEFAULT_HEARTBEAT_SECS,
};
pub use resolver::{ResolverError, ScriptResolver}; pub use resolver::{ResolverError, ScriptResolver};

View File

@@ -0,0 +1,242 @@
//! In-process `RealtimeBroadcaster` — the SSE fan-out registry (v1.1.6).
//!
//! Sibling of [`crate::inbox::InboxRegistry`], but multi-receiver and
//! repeated-event: a `Mutex<HashMap<(AppId, topic), broadcast::Sender>>`
//! over `tokio::sync::broadcast` instead of a oneshot map. The publish
//! side ([`PubsubServiceImpl`]) and the SSE subscribe side both hold one
//! shared `Arc<InProcessBroadcaster>`.
//!
//! Delivery is best-effort: each channel has a bounded buffer
//! (`PICLOUD_REALTIME_BROADCAST_CAPACITY`, default 64); a slow consumer
//! that falls behind sees the oldest events dropped (standard
//! `broadcast` lag semantics — the receiver gets `RecvError::Lagged`).
//! SSE's transport-layer auto-reconnect is the recovery path; there's no
//! server-side replay in v1.1.6.
//!
//! Channels are created lazily on first subscribe. A periodic GC task
//! ([`spawn_realtime_gc`]) drops senders whose receiver count has fallen
//! to zero so one-shot subscribers don't grow the map unboundedly.
//!
//! Cluster mode (v1.3+) swaps this for a Postgres `LISTEN/NOTIFY`-backed
//! resolver behind the same `RealtimeBroadcaster` trait.
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use async_trait::async_trait;
use picloud_shared::{AppId, BroadcasterError, RealtimeBroadcaster, RealtimeEvent};
use tokio::sync::broadcast;
/// Default per-channel broadcast buffer depth.
pub const DEFAULT_BROADCAST_CAPACITY: usize = 64;
const ENV_CAPACITY: &str = "PICLOUD_REALTIME_BROADCAST_CAPACITY";
/// Default GC sweep interval for empty channels.
pub const DEFAULT_GC_INTERVAL_SECS: u64 = 60;
pub struct InProcessBroadcaster {
inner: Mutex<HashMap<(AppId, String), broadcast::Sender<RealtimeEvent>>>,
capacity: usize,
}
impl InProcessBroadcaster {
#[must_use]
pub fn new(capacity: usize) -> Self {
Self {
inner: Mutex::new(HashMap::new()),
capacity: capacity.max(1),
}
}
/// Build from `PICLOUD_REALTIME_BROADCAST_CAPACITY` (default 64).
#[must_use]
pub fn from_env() -> Self {
let capacity = match std::env::var(ENV_CAPACITY) {
Err(_) => DEFAULT_BROADCAST_CAPACITY,
Ok(v) => match v.parse::<usize>() {
Ok(n) if n > 0 => n,
Ok(_) => {
tracing::warn!(env = ENV_CAPACITY, value = %v, "must be > 0; using default");
DEFAULT_BROADCAST_CAPACITY
}
Err(e) => {
tracing::warn!(env = ENV_CAPACITY, value = %v, error = %e, "invalid; using default");
DEFAULT_BROADCAST_CAPACITY
}
},
};
Self::new(capacity)
}
/// Number of live channels in the map (test/observability helper).
#[must_use]
pub fn channel_count(&self) -> usize {
self.inner.lock().map(|g| g.len()).unwrap_or(0)
}
/// Drop senders with zero receivers. Returns how many were removed.
/// Called periodically by [`spawn_realtime_gc`].
pub fn gc(&self) -> usize {
let Ok(mut g) = self.inner.lock() else {
return 0;
};
let before = g.len();
g.retain(|_, tx| tx.receiver_count() > 0);
before - g.len()
}
}
#[async_trait]
impl RealtimeBroadcaster for InProcessBroadcaster {
async fn subscribe(
&self,
app_id: AppId,
topic: &str,
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
let mut g = self
.inner
.lock()
.map_err(|_| BroadcasterError::Unavailable("broadcaster map poisoned".into()))?;
let tx = g
.entry((app_id, topic.to_string()))
.or_insert_with(|| broadcast::channel(self.capacity).0);
Ok(tx.subscribe())
}
async fn publish(&self, app_id: AppId, topic: &str, event: RealtimeEvent) {
let Ok(g) = self.inner.lock() else {
return;
};
// Only fan out to an existing channel: a topic with no live
// subscribers has no sender (publish never creates one). `send`
// returns Err iff every receiver has dropped — a benign no-op.
if let Some(tx) = g.get(&(app_id, topic.to_string())) {
let _ = tx.send(event);
}
}
async fn drop_topic(&self, app_id: AppId, topic: &str) {
if let Ok(mut g) = self.inner.lock() {
// Removing the sender closes the channel; existing receivers
// observe `RecvError::Closed` and disconnect cleanly.
g.remove(&(app_id, topic.to_string()));
}
}
}
/// Spawn the background GC sweep that drops empty channels every
/// `interval_secs` (default [`DEFAULT_GC_INTERVAL_SECS`]). Spawned at
/// startup alongside the other housekeeping tasks.
pub fn spawn_realtime_gc(broadcaster: Arc<InProcessBroadcaster>, interval_secs: u64) {
let period = Duration::from_secs(interval_secs.max(1));
tokio::spawn(async move {
let mut ticker = tokio::time::interval(period);
ticker.tick().await; // skip the immediate first fire
loop {
ticker.tick().await;
let removed = broadcaster.gc();
if removed > 0 {
tracing::debug!(removed, "realtime broadcaster GC dropped empty channels");
}
}
});
}
#[cfg(test)]
mod tests {
use super::*;
use chrono::Utc;
use serde_json::json;
fn event(topic: &str, n: i64) -> RealtimeEvent {
RealtimeEvent {
topic: topic.to_string(),
message: json!({ "n": n }),
published_at: Utc::now(),
}
}
#[tokio::test]
async fn multiple_subscribers_each_receive_each_event() {
let b = InProcessBroadcaster::new(16);
let app = AppId::new();
let mut rx1 = b.subscribe(app, "chat").await.unwrap();
let mut rx2 = b.subscribe(app, "chat").await.unwrap();
b.publish(app, "chat", event("chat", 1)).await;
b.publish(app, "chat", event("chat", 2)).await;
for rx in [&mut rx1, &mut rx2] {
assert_eq!(rx.recv().await.unwrap().message, json!({ "n": 1 }));
assert_eq!(rx.recv().await.unwrap().message, json!({ "n": 2 }));
}
}
#[tokio::test]
async fn dropped_subscriber_does_not_leak_after_gc() {
let b = InProcessBroadcaster::new(16);
let app = AppId::new();
let rx = b.subscribe(app, "t").await.unwrap();
assert_eq!(b.channel_count(), 1);
drop(rx);
// GC reclaims the now-empty channel.
assert_eq!(b.gc(), 1);
assert_eq!(b.channel_count(), 0);
}
#[tokio::test]
async fn drop_topic_disconnects_existing_subscribers() {
let b = InProcessBroadcaster::new(16);
let app = AppId::new();
let mut rx = b.subscribe(app, "t").await.unwrap();
b.drop_topic(app, "t").await;
// Sender gone → receiver observes a closed channel.
assert!(rx.recv().await.is_err());
assert_eq!(b.channel_count(), 0);
}
#[tokio::test]
async fn slow_consumer_loses_oldest_events() {
// Capacity 2: a consumer that never drains sees the oldest
// events dropped (broadcast Lagged semantics).
let b = InProcessBroadcaster::new(2);
let app = AppId::new();
let mut rx = b.subscribe(app, "t").await.unwrap();
for i in 0..5 {
b.publish(app, "t", event("t", i)).await;
}
// First recv reports the lag rather than event 0.
let first = rx.recv().await;
assert!(
matches!(first, Err(broadcast::error::RecvError::Lagged(_))),
"expected Lagged, got {first:?}"
);
// Subsequent recvs return the most recent buffered events.
let next = rx.recv().await.unwrap();
assert_eq!(next.message, json!({ "n": 3 }));
}
#[tokio::test]
async fn cross_app_isolation() {
let b = InProcessBroadcaster::new(16);
let app_a = AppId::new();
let app_b = AppId::new();
let mut rx_a = b.subscribe(app_a, "shared").await.unwrap();
let mut rx_b = b.subscribe(app_b, "shared").await.unwrap();
b.publish(app_a, "shared", event("shared", 1)).await;
// App B's subscriber must not see app A's publish.
assert_eq!(rx_a.recv().await.unwrap().message, json!({ "n": 1 }));
assert!(rx_b.try_recv().is_err());
}
#[tokio::test]
async fn publish_with_no_subscribers_is_noop() {
let b = InProcessBroadcaster::new(16);
let app = AppId::new();
// No subscriber → no sender created → no panic, nothing fanned out.
b.publish(app, "ghost", event("ghost", 1)).await;
assert_eq!(b.channel_count(), 0);
}
}

View File

@@ -0,0 +1,407 @@
//! SSE realtime endpoint — `GET /realtime/topics/{topic}` (v1.1.6).
//!
//! This is a data-plane surface, deliberately NOT under `/api/`
//! (realtime is its own versioning surface per the path scheme). It is
//! merged at the router root by the `picloud` binary alongside
//! `/healthz`, `/version`, and the user-route fallback.
//!
//! Handshake:
//! 1. Resolve `Host` → `app_id` (two-phase dispatch). No app → 404.
//! 2. Extract the token from `Authorization: Bearer <t>` OR `?token=<t>`
//! (EventSource can't set custom headers, so the query form is the
//! browser-compatible path).
//! 3. Ask the injected [`RealtimeAuthority`]: missing/internal topic →
//! 404, bad/absent token on a token-gated topic → 401, otherwise OK.
//! 4. Acquire a `broadcast::Receiver` and stream events as SSE until
//! the client disconnects (dropping the receiver — the broadcaster
//! cleans up on its own).
//!
//! Heartbeats (`:` comment lines) keep idle proxies from closing the
//! connection; interval is `PICLOUD_REALTIME_HEARTBEAT_SEC` (default 30).
use std::sync::Arc;
use std::time::Duration;
use axum::extract::{Path, Query, State};
use axum::http::{HeaderMap, StatusCode};
use axum::response::sse::{Event, KeepAlive, Sse};
use axum::response::{IntoResponse, Response};
use axum::routing::get;
use axum::Router;
use picloud_shared::{RealtimeAuthority, RealtimeBroadcaster, SubscribeDenied};
use serde::Deserialize;
use tokio_stream::wrappers::BroadcastStream;
use tokio_stream::{Stream, StreamExt};
use crate::routing::AppDomainTable;
/// Default heartbeat interval (seconds) for idle SSE connections.
pub const DEFAULT_HEARTBEAT_SECS: u64 = 30;
const ENV_HEARTBEAT: &str = "PICLOUD_REALTIME_HEARTBEAT_SEC";
#[derive(Clone)]
pub struct RealtimeState {
/// Host → app_id resolver (shared with the rest of the data plane).
pub app_domains: Arc<AppDomainTable>,
pub broadcaster: Arc<dyn RealtimeBroadcaster>,
pub authority: Arc<dyn RealtimeAuthority>,
pub heartbeat: Duration,
}
impl RealtimeState {
#[must_use]
pub fn new(
app_domains: Arc<AppDomainTable>,
broadcaster: Arc<dyn RealtimeBroadcaster>,
authority: Arc<dyn RealtimeAuthority>,
) -> Self {
Self {
app_domains,
broadcaster,
authority,
heartbeat: Duration::from_secs(heartbeat_secs_from_env()),
}
}
}
/// Read `PICLOUD_REALTIME_HEARTBEAT_SEC` (default 30, must be > 0).
#[must_use]
pub fn heartbeat_secs_from_env() -> u64 {
match std::env::var(ENV_HEARTBEAT) {
Err(_) => DEFAULT_HEARTBEAT_SECS,
Ok(v) => match v.parse::<u64>() {
Ok(n) if n > 0 => n,
_ => {
tracing::warn!(env = ENV_HEARTBEAT, value = %v, "invalid; using default");
DEFAULT_HEARTBEAT_SECS
}
},
}
}
/// Router for the realtime SSE surface. Merged at the router root.
pub fn realtime_router(state: RealtimeState) -> Router {
Router::new()
.route("/realtime/topics/{topic}", get(sse_topic))
.with_state(state)
}
#[derive(Debug, Deserialize)]
struct TokenQuery {
token: Option<String>,
}
async fn sse_topic(
State(state): State<RealtimeState>,
Path(topic): Path<String>,
Query(q): Query<TokenQuery>,
headers: HeaderMap,
) -> Response {
// 1. Host → app.
let host = headers
.get("host")
.and_then(|h| h.to_str().ok())
.unwrap_or("");
let Some(app_id) = state.app_domains.resolve_app(host) else {
return not_found("no app claims this host");
};
// 2. Token: Authorization: Bearer <t> takes precedence, else ?token=.
let token = bearer_token(&headers).or(q.token);
// 3. Authorize.
match state
.authority
.authorize_subscribe(app_id, &topic, token.as_deref())
.await
{
Ok(()) => {}
Err(SubscribeDenied::NotFound) => return not_found("topic not found"),
Err(SubscribeDenied::Unauthorized) => return unauthorized(),
Err(SubscribeDenied::Backend(e)) => {
tracing::error!(error = %e, "realtime authority backend error");
return (
StatusCode::INTERNAL_SERVER_ERROR,
axum::Json(serde_json::json!({ "error": "internal error" })),
)
.into_response();
}
}
// 4. Subscribe + stream.
let rx = match state.broadcaster.subscribe(app_id, &topic).await {
Ok(rx) => rx,
Err(e) => {
tracing::error!(error = %e, "failed to acquire realtime subscription");
return (
StatusCode::INTERNAL_SERVER_ERROR,
axum::Json(serde_json::json!({ "error": "internal error" })),
)
.into_response();
}
};
let stream = event_stream(rx);
let sse =
Sse::new(stream).keep_alive(KeepAlive::new().interval(state.heartbeat).text("heartbeat"));
// Sse sets Content-Type: text/event-stream + Cache-Control: no-cache.
// Add X-Accel-Buffering: no so an intermediate nginx doesn't buffer
// the stream (ignored by other proxies). Connection management is
// hyper's concern (and is hop-by-hop on HTTP/1.1, server-managed on
// HTTP/2), so we don't set Connection ourselves.
let mut resp = sse.into_response();
resp.headers_mut().insert(
"X-Accel-Buffering",
axum::http::HeaderValue::from_static("no"),
);
resp
}
/// Map the broadcast receiver into a stream of SSE events. Lagged
/// notifications (slow consumer) are skipped; a closed channel
/// (`drop_topic`, or all senders gone) ends the stream and the SSE
/// connection closes cleanly.
fn event_stream(
rx: tokio::sync::broadcast::Receiver<picloud_shared::RealtimeEvent>,
) -> impl Stream<Item = Result<Event, std::convert::Infallible>> {
BroadcastStream::new(rx).filter_map(|item| {
let ev = item.ok()?; // drop Lagged errors
let payload = serde_json::json!({
"topic": ev.topic,
"message": ev.message,
"published_at": ev.published_at.to_rfc3339(),
});
Some(Ok(Event::default().data(payload.to_string())))
})
}
fn bearer_token(headers: &HeaderMap) -> Option<String> {
let raw = headers
.get(axum::http::header::AUTHORIZATION)?
.to_str()
.ok()?;
raw.strip_prefix("Bearer ")
.map(|t| t.trim().to_string())
.filter(|t| !t.is_empty())
}
fn not_found(msg: &str) -> Response {
(
StatusCode::NOT_FOUND,
axum::Json(serde_json::json!({ "error": msg })),
)
.into_response()
}
fn unauthorized() -> Response {
// Generic — never leaks which check failed.
(
StatusCode::UNAUTHORIZED,
axum::Json(serde_json::json!({ "error": "unauthorized" })),
)
.into_response()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::realtime::InProcessBroadcaster;
use crate::routing::AppDomainTable;
use async_trait::async_trait;
use axum::body::Body;
use axum::http::Request;
use picloud_shared::{AppId, RealtimeEvent};
use tower::ServiceExt; // oneshot
/// Authority stub returning a fixed verdict.
struct StubAuthority(Result<(), SubscribeDenied>);
#[async_trait]
impl RealtimeAuthority for StubAuthority {
async fn authorize_subscribe(
&self,
_: AppId,
_: &str,
_: Option<&str>,
) -> Result<(), SubscribeDenied> {
self.0.clone()
}
}
/// App-domain table that maps a fixed host to a fixed app.
fn domains(host: &str, app: AppId) -> Arc<AppDomainTable> {
use crate::routing::{parse_app_domain, CompiledAppDomain};
let d = parse_app_domain(host).unwrap();
let table = AppDomainTable::new();
table.replace(vec![CompiledAppDomain {
app_id: app,
pattern: d.pattern,
shape_key: d.shape_key,
}]);
Arc::new(table)
}
fn state(
app: AppId,
host: &str,
verdict: Result<(), SubscribeDenied>,
broadcaster: Arc<dyn RealtimeBroadcaster>,
) -> RealtimeState {
RealtimeState {
app_domains: domains(host, app),
broadcaster,
authority: Arc::new(StubAuthority(verdict)),
heartbeat: Duration::from_millis(100),
}
}
async fn get_status(state: RealtimeState, host: &str, topic: &str) -> StatusCode {
let app = realtime_router(state);
let req = Request::builder()
.uri(format!("/realtime/topics/{topic}"))
.header("host", host)
.body(Body::empty())
.unwrap();
app.oneshot(req).await.unwrap().status()
}
#[tokio::test]
async fn unknown_host_is_404() {
let app = AppId::new();
let st = state(
app,
"app.example.com",
Ok(()),
Arc::new(InProcessBroadcaster::new(8)),
);
// Request a different host → no app claims it.
assert_eq!(
get_status(st, "other.example.com", "chat").await,
StatusCode::NOT_FOUND
);
}
#[tokio::test]
async fn not_found_topic_is_404() {
let app = AppId::new();
let st = state(
app,
"app.example.com",
Err(SubscribeDenied::NotFound),
Arc::new(InProcessBroadcaster::new(8)),
);
assert_eq!(
get_status(st, "app.example.com", "ghost").await,
StatusCode::NOT_FOUND
);
}
#[tokio::test]
async fn unauthorized_token_is_401() {
let app = AppId::new();
let st = state(
app,
"app.example.com",
Err(SubscribeDenied::Unauthorized),
Arc::new(InProcessBroadcaster::new(8)),
);
assert_eq!(
get_status(st, "app.example.com", "chat").await,
StatusCode::UNAUTHORIZED
);
}
#[tokio::test]
async fn public_topic_returns_event_stream() {
let app = AppId::new();
let st = state(
app,
"app.example.com",
Ok(()),
Arc::new(InProcessBroadcaster::new(8)),
);
let appr = realtime_router(st);
let req = Request::builder()
.uri("/realtime/topics/chat")
.header("host", "app.example.com")
.body(Body::empty())
.unwrap();
let resp = appr.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
let ct = resp
.headers()
.get(axum::http::header::CONTENT_TYPE)
.unwrap()
.to_str()
.unwrap();
assert!(ct.starts_with("text/event-stream"));
assert_eq!(resp.headers().get("x-accel-buffering").unwrap(), "no");
}
#[tokio::test]
async fn subscribe_receives_published_event() {
let app = AppId::new();
let broadcaster = Arc::new(InProcessBroadcaster::new(8));
let st = state(app, "app.example.com", Ok(()), broadcaster.clone());
let appr = realtime_router(st);
let req = Request::builder()
.uri("/realtime/topics/chat")
.header("host", "app.example.com")
.body(Body::empty())
.unwrap();
let resp = appr.oneshot(req).await.unwrap();
assert_eq!(resp.status(), StatusCode::OK);
// The handler has subscribed; publish and read the first chunk.
// Give the streaming task a beat to register its receiver.
let mut body = resp.into_body().into_data_stream();
tokio::time::sleep(Duration::from_millis(50)).await;
broadcaster
.publish(
app,
"chat",
RealtimeEvent {
topic: "chat".into(),
message: serde_json::json!({ "hi": 1 }),
published_at: chrono::Utc::now(),
},
)
.await;
let chunk = tokio::time::timeout(Duration::from_secs(2), body.next())
.await
.expect("a chunk within timeout")
.expect("stream item")
.expect("chunk ok");
let text = String::from_utf8_lossy(&chunk);
assert!(text.contains("data:"), "got: {text}");
assert!(text.contains("\"hi\":1"), "got: {text}");
}
#[tokio::test]
async fn heartbeat_fires_on_idle_connection() {
let app = AppId::new();
let broadcaster = Arc::new(InProcessBroadcaster::new(8));
// Hold a clone so the channel's sender outlives the router (which
// oneshot consumes) — otherwise the stream closes immediately.
let _keepalive = broadcaster.clone();
let st = state(app, "app.example.com", Ok(()), broadcaster);
let appr = realtime_router(st);
let req = Request::builder()
.uri("/realtime/topics/chat")
.header("host", "app.example.com")
.body(Body::empty())
.unwrap();
let resp = appr.oneshot(req).await.unwrap();
let mut body = resp.into_body().into_data_stream();
// No publish — with a 100ms heartbeat, a keep-alive comment must
// arrive well within a second.
let chunk = tokio::time::timeout(Duration::from_secs(1), body.next())
.await
.expect("heartbeat within timeout")
.expect("stream item")
.expect("chunk ok");
let text = String::from_utf8_lossy(&chunk);
assert!(text.starts_with(':'), "expected SSE comment, got: {text}");
}
}

View File

@@ -41,3 +41,7 @@ serde.workspace = true
serde_json.workspace = true serde_json.workspace = true
uuid.workspace = true uuid.workspace = true
chrono.workspace = true chrono.workspace = true
# Compute inbound-email HMAC signatures in the e2e receiver tests.
hmac.workspace = true
sha2.workspace = true
hex.workspace = true

View File

@@ -11,29 +11,36 @@ use axum::{routing::get, Json, Router};
use picloud_executor_core::{Engine, Limits}; use picloud_executor_core::{Engine, Limits};
use picloud_manager_core::{ use picloud_manager_core::{
admin_router, admins_router, api_keys_router, app_members_router, apps_api, apps_router, admin_router, admins_router, api_keys_router, app_members_router, apps_api, apps_router,
attach_principal_if_present, auth_router, compile_routes, dead_letters_router, migrations, attach_principal_if_present, auth_router, compile_routes, dead_letters_router,
require_authenticated, route_admin_router, triggers_router, AbandonedRepo, email_inbound_router, files_admin_router, migrations, require_authenticated,
route_admin_router, secrets_router, topics_router, triggers_router, AbandonedRepo,
AdminPrincipalResolver, AdminSessionRepository, AdminState, AdminUserRepository, AdminsState, AdminPrincipalResolver, AdminSessionRepository, AdminState, AdminUserRepository, AdminsState,
ApiKeyRepository, ApiKeysState, AppDomainRepository, AppMembersRepository, AppMembersState, ApiKeyRepository, ApiKeysState, AppDomainRepository, AppMembersRepository, AppMembersState,
AppRepository, AppsState, AuthState, AuthzRepo, DeadLetterRepo, DeadLettersState, Dispatcher, AppRepository, AppsState, AuthState, AuthzRepo, DeadLetterRepo, DeadLettersState, Dispatcher,
DocsServiceImpl, HttpConfig, HttpServiceImpl, KvServiceImpl, OutboxEventEmitter, OutboxRepo, DocsServiceImpl, EmailInboundState, EmailServiceImpl, FilesAdminState, FilesConfig,
PostgresAbandonedRepo, PostgresAdminSessionRepository, PostgresAdminUserRepository, FilesServiceImpl, FsFilesRepo, HttpConfig, HttpServiceImpl, KvServiceImpl, OutboxEventEmitter,
OutboxRepo, PostgresAbandonedRepo, PostgresAdminSessionRepository, PostgresAdminUserRepository,
PostgresApiKeyRepository, PostgresAppDomainRepository, PostgresAppMembersRepository, PostgresApiKeyRepository, PostgresAppDomainRepository, PostgresAppMembersRepository,
PostgresAppRepository, PostgresDeadLetterRepo, PostgresDeadLetterService, PostgresDocsRepo, PostgresAppRepository, PostgresAppSecretsRepo, PostgresDeadLetterRepo,
PostgresExecutionLogRepository, PostgresExecutionLogSink, PostgresKvRepo, PostgresOutboxRepo, PostgresDeadLetterService, PostgresDocsRepo, PostgresExecutionLogRepository,
PostgresRouteRepository, PostgresScriptRepository, PostgresTriggerRepo, PrincipalResolver, PostgresExecutionLogSink, PostgresKvRepo, PostgresOutboxRepo, PostgresPubsubRepo,
RepoResolver, RouteAdminState, RouteRepository, SandboxCeiling, ScriptRepository, PostgresRouteRepository, PostgresScriptRepository, PostgresSecretsRepo, PostgresTopicRepo,
TriggerConfig, TriggerRepo, TriggersState, PostgresTriggerRepo, PrincipalResolver, PubsubServiceImpl, RealtimeAuthorityImpl, RepoResolver,
RouteAdminState, RouteRepository, SandboxCeiling, ScriptRepository, SecretsConfig,
SecretsServiceImpl, SecretsState, SubscriberTokenConfig, TopicRepo, TopicsState, TriggerConfig,
TriggerRepo, TriggersState,
}; };
use picloud_orchestrator_core::realtime::DEFAULT_GC_INTERVAL_SECS;
use picloud_orchestrator_core::routing::{AppDomainTable, RouteTable}; use picloud_orchestrator_core::routing::{AppDomainTable, RouteTable};
use picloud_orchestrator_core::{ use picloud_orchestrator_core::{
data_plane_router, user_routes_router, DataPlaneState, ExecutionGate, InboxRegistry, data_plane_router, realtime_router, spawn_realtime_gc, user_routes_router, DataPlaneState,
LocalExecutorClient, ExecutionGate, InProcessBroadcaster, InboxRegistry, LocalExecutorClient, RealtimeState,
}; };
use picloud_shared::{ use picloud_shared::{
DeadLetterService, DocsService, ExecutionLogSink, HttpService, InboxResolver, KvService, DeadLetterService, DocsService, EmailService, ExecutionLogSink, FilesService, HttpService,
OutboxWriter, ScriptValidator, ServiceEventEmitter, Services, API_VERSION, PRODUCT_VERSION, InboxResolver, KvService, MasterKey, OutboxWriter, PubsubService, RealtimeAuthority,
SDK_VERSION, WIRE_VERSION, RealtimeBroadcaster, ScriptValidator, SecretsService, ServiceEventEmitter, Services,
API_VERSION, PRODUCT_VERSION, SDK_VERSION, WIRE_VERSION,
}; };
use sqlx::postgres::PgPoolOptions; use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool; use sqlx::PgPool;
@@ -89,7 +96,11 @@ fn read_session_ttl() -> Duration {
/// (`/api/v1/execute/{id}`, the user-route fallthrough, `/healthz`, /// (`/api/v1/execute/{id}`, the user-route fallthrough, `/healthz`,
/// `/version`) stays open — it's the public ingress for user scripts. /// `/version`) stays open — it's the public ingress for user scripts.
#[allow(clippy::too_many_lines)] #[allow(clippy::too_many_lines)]
pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> { pub async fn build_app(
pool: PgPool,
auth: AuthDeps,
master_key: MasterKey,
) -> anyhow::Result<Router> {
let script_repo = Arc::new(PostgresScriptRepository::new(pool.clone())); let script_repo = Arc::new(PostgresScriptRepository::new(pool.clone()));
let log_repo = Arc::new(PostgresExecutionLogRepository::new(pool.clone())); let log_repo = Arc::new(PostgresExecutionLogRepository::new(pool.clone()));
let log_sink: Arc<dyn ExecutionLogSink> = Arc::new(PostgresExecutionLogSink::new(pool.clone())); let log_sink: Arc<dyn ExecutionLogSink> = Arc::new(PostgresExecutionLogSink::new(pool.clone()));
@@ -157,7 +168,94 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
); );
} }
let http: Arc<dyn HttpService> = Arc::new(HttpServiceImpl::new(http_config, authz.clone())); let http: Arc<dyn HttpService> = Arc::new(HttpServiceImpl::new(http_config, authz.clone()));
let services = Services::new(kv, docs, dl_service.clone(), events, modules, http); // v1.1.5 filesystem-backed blob storage. Metadata lives in Postgres;
// the bytes live on disk under `PICLOUD_FILES_ROOT` (default ./data).
let files_config = FilesConfig::from_env();
let files_max_size = files_config.max_file_size_bytes;
// Kept for the v1.1.6 orphan sweeper (cleans stale `*.tmp.*` files).
let files_root = files_config.root.clone();
let files_repo = Arc::new(FsFilesRepo::new(pool.clone(), files_config));
let files: Arc<dyn FilesService> = Arc::new(FilesServiceImpl::new(
files_repo.clone(),
authz.clone(),
events.clone(),
files_max_size,
));
// v1.1.6 realtime: the in-process broadcaster is shared between the
// publish path (PubsubServiceImpl fans out to SSE subscribers after
// the durable outbox fan-out) and the SSE endpoint (subscribe side).
// The topic registry + app-secrets repo back the subscriber-token
// mint + SSE subscribe-authorization.
let broadcaster_concrete = Arc::new(InProcessBroadcaster::from_env());
let broadcaster: Arc<dyn RealtimeBroadcaster> = broadcaster_concrete.clone();
let topic_repo: Arc<dyn TopicRepo> = Arc::new(PostgresTopicRepo::new(pool.clone()));
let app_secrets_repo = Arc::new(PostgresAppSecretsRepo::new(
pool.clone(),
master_key.clone(),
));
// v1.1.7 two-phase migration: encrypt any plaintext realtime signing
// keys at rest. Idempotent — only touches rows not yet encrypted. The
// plaintext column is dropped in v1.1.8.
match app_secrets_repo.migrate_plaintext_keys().await {
Ok(0) => {}
Ok(n) => {
tracing::info!(
migrated = n,
"encrypted plaintext realtime signing keys at rest"
);
}
Err(e) => {
tracing::error!(error = %e, "failed to encrypt realtime signing keys (continuing)");
}
}
let realtime_authority: Arc<dyn RealtimeAuthority> = Arc::new(RealtimeAuthorityImpl::new(
topic_repo.clone(),
app_secrets_repo.clone(),
));
// v1.1.5 durable pub/sub, extended in v1.1.6 with the realtime
// broadcast + subscriber-token mint. Publishes fan out to matching
// pubsub triggers at publish time (one outbox row each, delivered by
// the same dispatcher as every other async trigger) AND, best-effort,
// to in-process SSE subscribers.
let pubsub_repo = Arc::new(PostgresPubsubRepo::new(pool.clone()));
let pubsub: Arc<dyn PubsubService> = Arc::new(
PubsubServiceImpl::new(pubsub_repo, authz.clone()).with_realtime(
broadcaster.clone(),
topic_repo.clone(),
app_secrets_repo,
SubscriberTokenConfig::from_env(),
),
);
// v1.1.7 encrypted per-app secrets. Values are AES-256-GCM-sealed
// with the process master key before they touch Postgres; the repo
// only ever sees ciphertext + nonce. The admin surface reuses the
// same repo + master key (see `secrets_state` below).
let secrets_config = SecretsConfig::from_env();
let secrets_max_value_bytes = secrets_config.max_value_bytes;
let secrets_repo: Arc<dyn picloud_manager_core::SecretsRepo> =
Arc::new(PostgresSecretsRepo::new(pool.clone()));
let secrets: Arc<dyn SecretsService> = Arc::new(SecretsServiceImpl::new(
secrets_repo.clone(),
authz.clone(),
master_key.clone(),
secrets_config,
));
// v1.1.7 outbound email. Builds a lettre SMTP transport from
// PICLOUD_SMTP_* env (disabled mode + warning if unconfigured).
let email: Arc<dyn EmailService> = Arc::new(EmailServiceImpl::from_env(authz.clone()));
let services = Services::new(
kv,
docs,
dl_service.clone(),
events,
modules,
http,
files,
pubsub,
secrets,
email,
);
let engine = Arc::new(Engine::new(Limits::default(), services)); let engine = Arc::new(Engine::new(Limits::default(), services));
// Compile the routes table once at startup; admin writes refresh it. // Compile the routes table once at startup; admin writes refresh it.
@@ -257,12 +355,24 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
// enqueues due triggers into the outbox; the dispatcher above // enqueues due triggers into the outbox; the dispatcher above
// delivers them like any other async trigger. // delivers them like any other async trigger.
picloud_manager_core::spawn_cron_scheduler(pool, trigger_config.cron_tick_interval_ms); picloud_manager_core::spawn_cron_scheduler(pool, trigger_config.cron_tick_interval_ms);
// v1.1.6: GC empty realtime broadcast channels (one-shot subscribers)
// and sweep orphaned `*.tmp.*` blobs left by crashed file writes.
spawn_realtime_gc(broadcaster_concrete, DEFAULT_GC_INTERVAL_SECS);
picloud_manager_core::spawn_files_orphan_sweep(files_root);
let triggers_state = TriggersState { let triggers_state = TriggersState {
triggers: trigger_repo, triggers: trigger_repo.clone(),
apps: apps_repo.clone(), apps: apps_repo.clone(),
authz: authz.clone(), authz: authz.clone(),
scripts: Arc::new(PostgresScriptRepoHandle(script_repo.clone())), scripts: Arc::new(PostgresScriptRepoHandle(script_repo.clone())),
config: trigger_config, config: trigger_config,
master_key: master_key.clone(),
};
// v1.1.7 public inbound-email receiver. Outside the admin auth layer
// (the URL + per-trigger HMAC secret are the security boundary).
let email_inbound_state = EmailInboundState {
triggers: trigger_repo,
outbox: outbox_repo.clone(),
master_key: master_key.clone(),
}; };
let dead_letters_state = DeadLettersState { let dead_letters_state = DeadLettersState {
repo: dl_repo, repo: dl_repo,
@@ -270,11 +380,29 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
apps: apps_repo.clone(), apps: apps_repo.clone(),
authz: authz.clone(), authz: authz.clone(),
}; };
let files_admin_state = FilesAdminState {
files: files_repo,
apps: apps_repo.clone(),
authz: authz.clone(),
};
let topics_state = TopicsState {
topics: topic_repo,
apps: apps_repo.clone(),
authz: authz.clone(),
broadcaster: broadcaster.clone(),
};
let secrets_state = SecretsState {
repo: secrets_repo,
apps: apps_repo.clone(),
authz: authz.clone(),
master_key,
max_value_bytes: secrets_max_value_bytes,
};
let apps_state = AppsState { let apps_state = AppsState {
apps: apps_repo, apps: apps_repo,
domains: domains_repo, domains: domains_repo,
routes: route_repo, routes: route_repo,
domain_table: app_domain_table, domain_table: app_domain_table.clone(),
authz: authz.clone(), authz: authz.clone(),
}; };
@@ -312,6 +440,9 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
.merge(app_members_router(app_members_state)) .merge(app_members_router(app_members_state))
.merge(api_keys_router(api_keys_state)) .merge(api_keys_router(api_keys_state))
.merge(triggers_router(triggers_state)) .merge(triggers_router(triggers_state))
.merge(files_admin_router(files_admin_state))
.merge(topics_router(topics_state))
.merge(secrets_router(secrets_state))
.merge(dead_letters_router(dead_letters_state)) .merge(dead_letters_router(dead_letters_state))
.layer(from_fn_with_state( .layer(from_fn_with_state(
auth_state.clone(), auth_state.clone(),
@@ -340,12 +471,24 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
let api_v1 = Router::new() let api_v1 = Router::new()
.nest("/admin", auth_router(auth_state)) .nest("/admin", auth_router(auth_state))
.nest("/admin", guarded_admin) .nest("/admin", guarded_admin)
.merge(email_inbound_router(email_inbound_state))
.merge(data_plane_routed); .merge(data_plane_routed);
// v1.1.6 SSE realtime surface, merged at the root (deliberately NOT
// under /api/ — realtime is its own versioning surface). Public auth
// is per-topic; no principal middleware (token verification is the
// gate, handled inside the authority).
let realtime = realtime_router(RealtimeState::new(
app_domain_table,
broadcaster,
realtime_authority,
));
Ok(Router::new() Ok(Router::new()
.route("/healthz", get(healthz)) .route("/healthz", get(healthz))
.route("/version", get(version)) .route("/version", get(version))
.nest(&format!("/api/v{API_VERSION}"), api_v1) .nest(&format!("/api/v{API_VERSION}"), api_v1)
.merge(realtime)
.merge(user_routes) .merge(user_routes)
.layer(TraceLayer::new_for_http())) .layer(TraceLayer::new_for_http()))
} }

View File

@@ -39,6 +39,11 @@ async fn run_server() -> anyhow::Result<()> {
let database_url = let database_url =
std::env::var("DATABASE_URL").map_err(|_| anyhow::anyhow!("DATABASE_URL is required"))?; std::env::var("DATABASE_URL").map_err(|_| anyhow::anyhow!("DATABASE_URL is required"))?;
// Source the process master key BEFORE doing any work — an unset or
// malformed PICLOUD_SECRET_KEY is fatal (v1.1.7). The only escape
// hatch is PICLOUD_DEV_MODE=true, which logs a prominent warning.
let master_key = picloud_shared::MasterKey::from_env()?;
let pool = init_db(&database_url).await?; let pool = init_db(&database_url).await?;
migrations::run(&pool).await?; migrations::run(&pool).await?;
tracing::info!("migrations applied"); tracing::info!("migrations applied");
@@ -69,7 +74,7 @@ async fn run_server() -> anyhow::Result<()> {
// so a delayed sweep can't extend session lifetimes. // so a delayed sweep can't extend session lifetimes.
spawn_session_pruner(auth.sessions.clone()); spawn_session_pruner(auth.sessions.clone());
let app = build_app(pool, auth).await?; let app = build_app(pool, auth, master_key).await?;
let listener = tokio::net::TcpListener::bind(addr).await?; let listener = tokio::net::TcpListener::bind(addr).await?;
tracing::info!(%addr, "picloud all-in-one listening"); tracing::info!(%addr, "picloud all-in-one listening");

View File

@@ -40,7 +40,13 @@ async fn server_with_app(pool: PgPool) -> (TestServer, String) {
.await .await
.expect("seed admin"); .expect("seed admin");
let app = picloud::build_app(pool, auth).await.expect("build_app"); let app = picloud::build_app(
pool,
auth,
picloud_shared::MasterKey::from_bytes([0x42u8; 32]),
)
.await
.expect("build_app");
let mut server = TestServer::new(app).expect("TestServer should build"); let mut server = TestServer::new(app).expect("TestServer should build");
let resp = server let resp = server

View File

@@ -57,7 +57,11 @@ async fn boot(pool: PgPool) -> Seeded {
.await .await
.expect("seed owner"); .expect("seed owner");
let app = picloud::build_app(pool.clone(), auth) let app = picloud::build_app(
pool.clone(),
auth,
picloud_shared::MasterKey::from_bytes([0x42u8; 32]),
)
.await .await
.expect("build_app"); .expect("build_app");
let server = TestServer::new(app).expect("TestServer"); let server = TestServer::new(app).expect("TestServer");

View File

@@ -0,0 +1,489 @@
//! End-to-end dispatcher tests — one per trigger kind (v1.1.5 follow-up,
//! landed in v1.1.6). Each test wires the full all-in-one app via
//! `build_app` (which spawns the real dispatcher + cron scheduler +
//! executor), creates an app + a logging handler script + a trigger,
//! causes the originating event, and polls for the handler's side effect.
//!
//! ## Gating
//!
//! These need a Postgres reachable via `DATABASE_URL`. They follow the
//! `schema_snapshot` pattern (NOT `#[ignore]`): when `DATABASE_URL` is
//! unset the test prints a notice and returns early, so plain
//! `cargo test` stays green locally while CI (which sets `DATABASE_URL`)
//! runs them.
//!
//! ## How "the handler fired" is observed
//!
//! The dispatcher does not write `execution_log` rows for trigger
//! handlers, so each handler instead records its `ctx.event` into a KV
//! marker (`collection = "e2e_markers"`, which no trigger watches — no
//! recursion). The test polls `kv_entries` for that marker and asserts
//! the event shape. See HANDBACK §deviations for why this lives in
//! `picloud/tests/` rather than `manager-core/tests/` (build_app lives in
//! the `picloud` crate) and for the `dead_letter` reinterpretation.
#![allow(clippy::needless_pass_by_value)]
use std::time::Duration;
use axum_test::TestServer;
use serde_json::{json, Value};
use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool;
use uuid::Uuid;
/// Connect + migrate, or return `None` (printing a skip notice) when
/// `DATABASE_URL` is unset — mirrors `schema_snapshot.rs`.
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
eprintln!("dispatcher_e2e: DATABASE_URL unset — skipping");
return None;
};
let pool = PgPoolOptions::new()
.max_connections(5)
.connect(&url)
.await
.expect("connect to DATABASE_URL");
sqlx::migrate!("../manager-core/migrations")
.run(&pool)
.await
.expect("apply migrations");
Some(pool)
}
/// Build the app over the shared pool with a uniquely-named owner admin,
/// log in, and create a fresh app. `suffix` must be unique per test (the
/// pool is shared, so names must not collide).
async fn server_for(pool: PgPool, suffix: &str) -> (TestServer, String) {
use picloud_manager_core::auth::hash_password;
use picloud_shared::InstanceRole;
let unique = format!("{suffix}-{}", Uuid::new_v4().simple());
let auth = picloud::AuthDeps::from_pool(pool.clone());
let username = format!("e2e-{unique}");
let hash = hash_password("pw").expect("hash");
auth.users
.create(&username, &hash, InstanceRole::Owner, None)
.await
.expect("seed admin");
let app = picloud::build_app(
pool,
auth,
picloud_shared::MasterKey::from_bytes([0x42u8; 32]),
)
.await
.expect("build_app");
let mut server = TestServer::new(app).expect("TestServer");
let resp = server
.post("/api/v1/admin/auth/login")
.json(&json!({ "username": username, "password": "pw" }))
.await;
resp.assert_status_ok();
let token = resp.json::<Value>()["token"]
.as_str()
.expect("login token")
.to_string();
server.add_header("authorization", format!("Bearer {token}"));
// A fresh app keeps each test's KV / events isolated from siblings.
let slug = format!("e2e-{unique}");
let created: Value = server
.post("/api/v1/admin/apps")
.json(&json!({ "slug": slug, "name": slug }))
.await
.json();
let app_id = created["id"].as_str().expect("app id").to_string();
(server, app_id)
}
async fn create_script(server: &TestServer, app_id: &str, name: &str, source: &str) -> String {
let created: Value = server
.post("/api/v1/admin/scripts")
.json(&json!({ "app_id": app_id, "name": name, "source": source }))
.await
.json();
created["id"].as_str().expect("script id").to_string()
}
/// A handler that records its `ctx.event` into a KV marker the test can
/// observe. The marker collection is watched by no trigger.
const MARKER_HANDLER: &str = r#"
let e = ctx.event;
kv::collection("e2e_markers").set("marker", e);
#{ ok: true }
"#;
/// Poll the marker KV key until present (or ~10s timeout).
async fn poll_marker(pool: &PgPool, app_id: &str) -> Option<Value> {
poll_marker_n(pool, app_id, 100).await
}
/// Poll the marker KV key for `iters` × 100ms.
async fn poll_marker_n(pool: &PgPool, app_id: &str, iters: u32) -> Option<Value> {
let app_uuid = Uuid::parse_str(app_id).expect("app uuid");
for _ in 0..iters {
let row: Option<(Value,)> = sqlx::query_as(
"SELECT value FROM kv_entries \
WHERE app_id = $1 AND collection = 'e2e_markers' AND key = 'marker'",
)
.bind(app_uuid)
.fetch_optional(pool)
.await
.expect("query marker");
if let Some((value,)) = row {
return Some(value);
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
None
}
async fn execute(server: &TestServer, script_id: &str) {
server
.post(&format!("/api/v1/execute/{script_id}"))
.json(&json!({}))
.await
.assert_status_ok();
}
#[tokio::test]
async fn dispatcher_delivers_kv_to_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "kv").await;
let handler = create_script(&server, &app_id, "kv-handler", MARKER_HANDLER).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
&server,
&app_id,
"kv-source",
r#"kv::collection("src").set("k", 42); #{ ok: true }"#,
)
.await;
execute(&server, &source).await;
let event = poll_marker(&pool, &app_id).await.expect("kv handler fired");
assert_eq!(event["source"], "kv");
assert_eq!(event["op"], "insert");
assert_eq!(event["kv"]["collection"], "src");
assert_eq!(event["kv"]["key"], "k");
}
#[tokio::test]
async fn dispatcher_delivers_docs_to_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "docs").await;
let handler = create_script(&server, &app_id, "docs-handler", MARKER_HANDLER).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/docs"))
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
&server,
&app_id,
"docs-source",
r#"docs::collection("src").create(#{ x: 1 }); #{ ok: true }"#,
)
.await;
execute(&server, &source).await;
let event = poll_marker(&pool, &app_id)
.await
.expect("docs handler fired");
assert_eq!(event["source"], "docs");
assert_eq!(event["op"], "create");
assert_eq!(event["docs"]["collection"], "src");
}
#[tokio::test]
async fn dispatcher_delivers_cron_to_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "cron").await;
let handler = create_script(&server, &app_id, "cron-handler", MARKER_HANDLER).await;
// Fire every second (6-field cron, seconds-resolution).
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/cron"))
.json(&json!({ "script_id": handler, "schedule": "* * * * * *", "timezone": "UTC" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
// No source — the scheduler enqueues the due tick on its own. The
// scheduler skips its first tick and then ticks every
// PICLOUD_CRON_TICK_INTERVAL_MS (default 30s), so poll past that
// (set the env var lower to speed CI up if desired).
let event = poll_marker_n(&pool, &app_id, 450)
.await
.expect("cron handler fired");
assert_eq!(event["source"], "cron");
assert_eq!(event["op"], "tick");
assert_eq!(event["cron"]["timezone"], "UTC");
}
#[tokio::test]
async fn dispatcher_delivers_files_to_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "files").await;
let handler = create_script(&server, &app_id, "files-handler", MARKER_HANDLER).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/files"))
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
&server,
&app_id,
"files-source",
r#"
let data = base64::decode("aGk=");
files::collection("src").create(#{ name: "f.txt", content_type: "text/plain", data: data });
#{ ok: true }
"#,
)
.await;
execute(&server, &source).await;
let event = poll_marker(&pool, &app_id)
.await
.expect("files handler fired");
assert_eq!(event["source"], "files");
assert_eq!(event["op"], "create");
assert_eq!(event["files"]["collection"], "src");
assert_eq!(event["files"]["name"], "f.txt");
}
#[tokio::test]
async fn dispatcher_delivers_pubsub_to_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "pubsub").await;
let handler = create_script(&server, &app_id, "pubsub-handler", MARKER_HANDLER).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/pubsub"))
.json(&json!({ "script_id": handler, "topic_pattern": "e2e.topic" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
&server,
&app_id,
"pubsub-source",
r#"pubsub::publish_durable("e2e.topic", #{ hello: 1 }); #{ ok: true }"#,
)
.await;
execute(&server, &source).await;
let event = poll_marker(&pool, &app_id)
.await
.expect("pubsub handler fired");
assert_eq!(event["source"], "pubsub");
assert_eq!(event["op"], "publish");
assert_eq!(event["pubsub"]["topic"], "e2e.topic");
assert_eq!(event["pubsub"]["message"]["hello"], 1);
}
/// Count dead_letters rows for an app.
async fn dead_letter_count(pool: &PgPool, app_id: &str) -> i64 {
let app_uuid = Uuid::parse_str(app_id).unwrap();
sqlx::query_scalar("SELECT COUNT(*) FROM dead_letters WHERE app_id = $1")
.bind(app_uuid)
.fetch_one(pool)
.await
.expect("count dead_letters")
}
async fn poll_dead_letter_count(pool: &PgPool, app_id: &str, want: i64) -> i64 {
let mut count = 0;
for _ in 0..100 {
count = dead_letter_count(pool, app_id).await;
if count >= want {
break;
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
count
}
/// Register a failing KV trigger on `dlsrc` (single attempt → immediate
/// dead-letter) and a `dead_letter` trigger pointing at the marker
/// handler, then cause the originating KV event. Returns when set up.
async fn setup_dead_letter(server: &TestServer, app_id: &str, dl_handler: &str) {
let failing = create_script(server, app_id, "dl-failing", r#"throw "boom";"#).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
.json(&json!({
"script_id": failing,
"collection_glob": "dlsrc",
"retry_max_attempts": 1,
"retry_base_ms": 0
}))
.await
.assert_status(axum::http::StatusCode::CREATED);
// The dead_letter trigger (no filters → matches any dead-letter).
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/dead_letter"))
.json(&json!({ "script_id": dl_handler }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
server,
app_id,
"dl-source",
r#"kv::collection("dlsrc").set("k", 1); #{ ok: true }"#,
)
.await;
execute(server, source.as_str()).await;
}
#[tokio::test]
async fn dispatcher_delivers_dead_letter_to_handler() {
// v1.1.7: the dead-letter fan-out is now wired
// (`dispatcher::handle_failure` → `list_matching_dead_letter` →
// outbox). This asserts BOTH that the `dead_letters` row is written
// AND that the registered `dead_letter`-kind handler actually fires
// (it was silently non-functional v1.1.1v1.1.6).
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "dl").await;
let handler = create_script(&server, &app_id, "dl-handler", MARKER_HANDLER).await;
setup_dead_letter(&server, &app_id, &handler).await;
// Row written.
assert!(
poll_dead_letter_count(&pool, &app_id, 1).await > 0,
"a dead-letter row should have been produced"
);
// Handler fired.
let event = poll_marker(&pool, &app_id)
.await
.expect("dead-letter handler fired");
assert_eq!(event["source"], "dead_letter");
}
#[tokio::test]
async fn dispatcher_delivers_dead_letter_to_handler_actually_fires() {
// Focused on the handler-fire side: the marker handler receives a
// fully-shaped dead-letter event (the original KV event nested under
// `ctx.event.dead_letter.original`, plus the failure metadata).
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "dlfire").await;
let handler = create_script(&server, &app_id, "dl-handler", MARKER_HANDLER).await;
setup_dead_letter(&server, &app_id, &handler).await;
let event = poll_marker(&pool, &app_id)
.await
.expect("dead-letter handler fired");
assert_eq!(event["source"], "dead_letter");
// The original KV event is nested verbatim.
assert_eq!(event["dead_letter"]["original"]["source"], "kv");
assert_eq!(
event["dead_letter"]["original"]["kv"]["collection"],
"dlsrc"
);
// Failure metadata is present.
assert!(event["dead_letter"]["last_error"]
.as_str()
.unwrap()
.contains("boom"));
assert!(event["dead_letter"]["attempts"].as_i64().unwrap() >= 1);
}
#[tokio::test]
async fn dead_letter_source_filter_excludes_nonmatching() {
// `list_matching_dead_letter` filters by source (among trigger_id /
// script_id). A dead_letter trigger whose `source_filter` is "docs"
// must NOT fire for a "kv"-sourced dead-letter — the row is still
// written, but no handler delivery is enqueued.
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "dlfilter").await;
let handler = create_script(&server, &app_id, "dl-handler", MARKER_HANDLER).await;
let failing = create_script(&server, &app_id, "dl-failing", r#"throw "boom";"#).await;
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
.json(&json!({
"script_id": failing,
"collection_glob": "dlsrc",
"retry_max_attempts": 1,
"retry_base_ms": 0
}))
.await
.assert_status(axum::http::StatusCode::CREATED);
// Filter to a different source so this handler must NOT match.
server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/dead_letter"))
.json(&json!({ "script_id": handler, "source_filter": "docs" }))
.await
.assert_status(axum::http::StatusCode::CREATED);
let source = create_script(
&server,
&app_id,
"dl-source",
r#"kv::collection("dlsrc").set("k", 1); #{ ok: true }"#,
)
.await;
execute(&server, &source).await;
// The dead-letter row is written…
assert!(poll_dead_letter_count(&pool, &app_id, 1).await >= 1);
// …but the source-filtered handler never fires.
let marker = poll_marker_n(&pool, &app_id, 8).await;
assert!(
marker.is_none(),
"source_filter='docs' must not fire for a kv dead-letter"
);
}
#[tokio::test]
async fn dead_letter_handler_failure_does_not_recurse() {
// Recursion-stop (design notes §4): a dead_letter handler that itself
// throws must NOT produce a second dead-letter row. The
// `is_dead_letter_handler` short-circuit annotates the original row
// and drops the outbox row without re-dead-lettering.
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "dlrec").await;
// The DL handler itself throws.
let throwing = create_script(&server, &app_id, "dl-throws", r#"throw "handler boom";"#).await;
setup_dead_letter(&server, &app_id, &throwing).await;
// One dead-letter row appears (the original). Give the throwing
// handler time to run + (not) recurse, then confirm the count stayed
// at exactly 1.
assert!(poll_dead_letter_count(&pool, &app_id, 1).await >= 1);
tokio::time::sleep(Duration::from_millis(800)).await;
assert_eq!(
dead_letter_count(&pool, &app_id).await,
1,
"a failing dead-letter handler must not create a new dead-letter row"
);
}

View File

@@ -0,0 +1,298 @@
//! End-to-end tests for the inbound-email webhook receiver (v1.1.7).
//!
//! Gated on `DATABASE_URL` like `dispatcher_e2e.rs`: when unset the test
//! prints a notice and returns early so plain `cargo test` stays green.
//!
//! Covers the receiver's status-code matrix (202 / 401 / 404 / 422),
//! cross-app path isolation, HMAC verification (signed + unsigned
//! triggers), the dispatcher routing the `email` outbox row, and the
//! handler actually firing with `ctx.event.email` populated. The
//! "handler fired" observation uses the same KV-marker pattern as
//! `dispatcher_e2e.rs`.
#![allow(clippy::needless_pass_by_value)]
use std::time::Duration;
use axum_test::TestServer;
use hmac::{Hmac, Mac};
use serde_json::{json, Value};
use sha2::Sha256;
use sqlx::postgres::PgPoolOptions;
use sqlx::PgPool;
use uuid::Uuid;
/// Fixed master key so the receiver decrypts the inbound_secret the
/// admin endpoint encrypted (same key feeds build_app + the admin path).
fn master_key() -> picloud_shared::MasterKey {
picloud_shared::MasterKey::from_bytes([0x42u8; 32])
}
async fn pool_or_skip() -> Option<PgPool> {
let Ok(url) = std::env::var("DATABASE_URL") else {
eprintln!("email_inbound: DATABASE_URL unset — skipping");
return None;
};
let pool = PgPoolOptions::new()
.max_connections(5)
.connect(&url)
.await
.expect("connect to DATABASE_URL");
sqlx::migrate!("../manager-core/migrations")
.run(&pool)
.await
.expect("apply migrations");
Some(pool)
}
async fn server_for(pool: PgPool, suffix: &str) -> (TestServer, String) {
use picloud_manager_core::auth::hash_password;
use picloud_shared::InstanceRole;
let unique = format!("{suffix}-{}", Uuid::new_v4().simple());
let auth = picloud::AuthDeps::from_pool(pool.clone());
let username = format!("eml-{unique}");
let hash = hash_password("pw").expect("hash");
auth.users
.create(&username, &hash, InstanceRole::Owner, None)
.await
.expect("seed admin");
let app = picloud::build_app(pool, auth, master_key())
.await
.expect("build_app");
let mut server = TestServer::new(app).expect("TestServer");
let resp = server
.post("/api/v1/admin/auth/login")
.json(&json!({ "username": username, "password": "pw" }))
.await;
resp.assert_status_ok();
let token = resp.json::<Value>()["token"]
.as_str()
.expect("login token")
.to_string();
server.add_header("authorization", format!("Bearer {token}"));
let slug = format!("eml-{unique}");
let created: Value = server
.post("/api/v1/admin/apps")
.json(&json!({ "slug": slug, "name": slug }))
.await
.json();
let app_id = created["id"].as_str().expect("app id").to_string();
(server, app_id)
}
async fn create_script(server: &TestServer, app_id: &str, name: &str, source: &str) -> String {
let created: Value = server
.post("/api/v1/admin/scripts")
.json(&json!({ "app_id": app_id, "name": name, "source": source }))
.await
.json();
created["id"].as_str().expect("script id").to_string()
}
const MARKER_HANDLER: &str = r#"
let e = ctx.event;
kv::collection("e2e_markers").set("marker", e);
#{ ok: true }
"#;
async fn create_email_trigger(
server: &TestServer,
app_id: &str,
script_id: &str,
secret: Option<&str>,
) -> String {
let created: Value = server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/email"))
.json(&json!({ "script_id": script_id, "inbound_secret": secret }))
.await
.json();
created["id"].as_str().expect("trigger id").to_string()
}
fn sign(secret: &str, body: &str) -> String {
let mut mac = Hmac::<Sha256>::new_from_slice(secret.as_bytes()).expect("hmac key");
mac.update(body.as_bytes());
hex::encode(mac.finalize().into_bytes())
}
async fn poll_marker(pool: &PgPool, app_id: &str) -> Option<Value> {
let app_uuid = Uuid::parse_str(app_id).expect("app uuid");
for _ in 0..100 {
let row: Option<(Value,)> = sqlx::query_as(
"SELECT value FROM kv_entries \
WHERE app_id = $1 AND collection = 'e2e_markers' AND key = 'marker'",
)
.bind(app_uuid)
.fetch_optional(pool)
.await
.expect("query marker");
if let Some((value,)) = row {
return Some(value);
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
None
}
const BODY: &str = r#"{"from":"sender@external.com","to":["alice@myapp.com"],"cc":["bob@myapp.com"],"subject":"Re: question","text":"hello there","message_id":"<abc@external.com>"}"#;
#[tokio::test]
async fn signed_post_accepts_and_fires_handler() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool.clone(), "signed").await;
let handler = create_script(&server, &app_id, "eml-handler", MARKER_HANDLER).await;
let trigger = create_email_trigger(&server, &app_id, &handler, Some("topsecret")).await;
let sig = sign("topsecret", BODY);
server
.post(&format!("/api/v1/email-inbound/{app_id}/{trigger}"))
.add_header("x-picloud-signature", sig)
.text(BODY)
.await
.assert_status(axum::http::StatusCode::ACCEPTED);
// Outbox row landed with source_kind = 'email'.
let app_uuid = Uuid::parse_str(&app_id).unwrap();
// The dispatcher deletes the row after delivery; instead assert the
// handler fired (which proves the email row was dispatched).
let event = poll_marker(&pool, &app_id).await.expect("handler fired");
assert_eq!(event["source"], "email");
assert_eq!(event["op"], "receive");
assert_eq!(event["email"]["from"], "sender@external.com");
assert_eq!(event["email"]["to"][0], "alice@myapp.com");
assert_eq!(event["email"]["cc"][0], "bob@myapp.com");
assert_eq!(event["email"]["subject"], "Re: question");
assert_eq!(event["email"]["text"], "hello there");
assert_eq!(event["email"]["message_id"], "<abc@external.com>");
let _ = app_uuid;
}
#[tokio::test]
async fn missing_signature_is_401_when_secret_configured() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "nosig").await;
let handler = create_script(&server, &app_id, "h", MARKER_HANDLER).await;
let trigger = create_email_trigger(&server, &app_id, &handler, Some("topsecret")).await;
server
.post(&format!("/api/v1/email-inbound/{app_id}/{trigger}"))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn wrong_signature_is_401() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "wrongsig").await;
let handler = create_script(&server, &app_id, "h", MARKER_HANDLER).await;
let trigger = create_email_trigger(&server, &app_id, &handler, Some("topsecret")).await;
server
.post(&format!("/api/v1/email-inbound/{app_id}/{trigger}"))
.add_header("x-picloud-signature", sign("WRONG", BODY))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::UNAUTHORIZED);
}
#[tokio::test]
async fn unsigned_trigger_accepts_without_signature() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "unsigned").await;
let handler = create_script(&server, &app_id, "h", MARKER_HANDLER).await;
let trigger = create_email_trigger(&server, &app_id, &handler, None).await;
server
.post(&format!("/api/v1/email-inbound/{app_id}/{trigger}"))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::ACCEPTED);
}
#[tokio::test]
async fn unknown_trigger_is_404() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "missing").await;
let missing = Uuid::new_v4();
server
.post(&format!("/api/v1/email-inbound/{app_id}/{missing}"))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::NOT_FOUND);
}
#[tokio::test]
async fn wrong_kind_trigger_is_404() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "wrongkind").await;
let handler = create_script(&server, &app_id, "h", MARKER_HANDLER).await;
// A KV trigger — not an email trigger.
let kv_trigger: Value = server
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
.json(&json!({ "script_id": handler, "collection_glob": "*" }))
.await
.json();
let kv_id = kv_trigger["id"].as_str().unwrap();
server
.post(&format!("/api/v1/email-inbound/{app_id}/{kv_id}"))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::NOT_FOUND);
}
#[tokio::test]
async fn malformed_body_is_422() {
let Some(pool) = pool_or_skip().await else {
return;
};
let (server, app_id) = server_for(pool, "malformed").await;
let handler = create_script(&server, &app_id, "h", MARKER_HANDLER).await;
// Unsigned so we reach the parse step.
let trigger = create_email_trigger(&server, &app_id, &handler, None).await;
server
.post(&format!("/api/v1/email-inbound/{app_id}/{trigger}"))
.text("not json at all")
.await
.assert_status(axum::http::StatusCode::UNPROCESSABLE_ENTITY);
}
#[tokio::test]
async fn cross_app_path_is_404() {
let Some(pool) = pool_or_skip().await else {
return;
};
// Two apps under the same server. A trigger created in app B must
// not be reachable via app A's path segment.
let (server, app_a) = server_for(pool.clone(), "xa").await;
let app_b: Value = server
.post("/api/v1/admin/apps")
.json(&json!({ "slug": format!("xb-{}", Uuid::new_v4().simple()), "name": "xb" }))
.await
.json();
let app_b_id = app_b["id"].as_str().unwrap().to_string();
let handler_b = create_script(&server, &app_b_id, "hb", MARKER_HANDLER).await;
let trigger_b = create_email_trigger(&server, &app_b_id, &handler_b, None).await;
// POST to app A's path with app B's trigger id → 404 (path-bound).
server
.post(&format!("/api/v1/email-inbound/{app_a}/{trigger_b}"))
.text(BODY)
.await
.assert_status(axum::http::StatusCode::NOT_FOUND);
}

View File

@@ -15,3 +15,16 @@ serde_json.workspace = true
thiserror.workspace = true thiserror.workspace = true
uuid.workspace = true uuid.workspace = true
chrono.workspace = true chrono.workspace = true
# Realtime broadcaster trait returns a broadcast::Receiver; subscriber
# tokens are HMAC-SHA256 over a base64url payload (v1.1.6).
tokio = { workspace = true, features = ["sync"] }
hmac.workspace = true
sha2.workspace = true
base64.workspace = true
# AES-256-GCM envelope + master-key sourcing (v1.1.7 crypto module).
aes-gcm.workspace = true
rand.workspace = true
tracing.workspace = true
[dev-dependencies]
tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "time", "sync"] }

358
crates/shared/src/crypto.rs Normal file
View File

@@ -0,0 +1,358 @@
//! AES-256-GCM encryption envelope + master-key sourcing (v1.1.7).
//!
//! Two responsibilities:
//!
//! 1. [`encrypt`] / [`decrypt`] — the at-rest envelope used by per-app
//! `secrets`, the encrypted `inbound_secret` on email triggers, and
//! the realtime signing key. `Aes256Gcm` with a 96-bit (12-byte)
//! random nonce and a 128-bit auth tag **appended to the
//! ciphertext** (the RustCrypto `Aead`-trait layout — `encrypt`
//! returns `ciphertext || tag`, `decrypt` consumes the same). Both
//! the ciphertext (tag included) and the nonce are stored.
//!
//! 2. [`MasterKey`] — the process-wide 32-byte key, sourced once at
//! startup from `PICLOUD_SECRET_KEY` (base64 of exactly 32 bytes).
//! A deterministic in-memory dev key is allowed ONLY when the env
//! var is unset AND `PICLOUD_DEV_MODE=true`; otherwise an unset key
//! is fatal (no quiet "your secrets are unencrypted" mode).
//!
//! **Key rotation is out of scope for v1.1.7.** Changing
//! `PICLOUD_SECRET_KEY` between deploys orphans every existing
//! ciphertext (it can no longer be decrypted). v1.2+ adds key-version
//! columns + a re-encryption pass.
use aes_gcm::aead::{Aead, KeyInit};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use base64::engine::general_purpose::STANDARD as B64;
use base64::Engine as _;
use rand::RngCore;
use sha2::{Digest, Sha256};
use thiserror::Error;
/// Master-key length in bytes (AES-256 → 32-byte key).
pub const KEY_LEN: usize = 32;
/// GCM nonce length in bytes (96-bit nonce, the AES-GCM standard).
pub const NONCE_LEN: usize = 12;
/// Output of [`encrypt`]: the ciphertext (auth tag appended) plus the
/// randomly-generated nonce. Both must be persisted; `decrypt` needs
/// the nonce to recover the plaintext.
#[derive(Debug, Clone)]
pub struct EncryptResult {
/// Ciphertext with the 16-byte GCM auth tag appended.
pub ciphertext: Vec<u8>,
/// The 12-byte nonce used for this encryption.
pub nonce: [u8; NONCE_LEN],
}
/// Errors from the encryption envelope.
#[derive(Debug, Error)]
pub enum CryptoError {
/// Authentication failed — wrong key, corrupted ciphertext, or a
/// tampered nonce/tag. GCM does not distinguish these (by design),
/// so neither do we.
#[error("decryption failed: authentication tag mismatch (wrong key, corrupted ciphertext, or tampered nonce)")]
Decrypt,
/// The stored nonce was not exactly [`NONCE_LEN`] bytes — a sign of
/// row corruption.
#[error("invalid nonce length: expected {NONCE_LEN} bytes, got {0}")]
InvalidNonce(usize),
}
/// Encrypt `plaintext` under `key`, generating a fresh random nonce.
///
/// The auth tag is appended to the returned ciphertext (RustCrypto
/// `Aead` layout). Encryption with a valid 32-byte key and 12-byte
/// nonce is infallible in `aes-gcm`, so this returns a value rather
/// than a `Result`.
#[must_use]
pub fn encrypt(plaintext: &[u8], key: &[u8; KEY_LEN]) -> EncryptResult {
let cipher = Aes256Gcm::new(Key::<Aes256Gcm>::from_slice(key));
let mut nonce_bytes = [0u8; NONCE_LEN];
// CSPRNG nonce. `thread_rng` is seeded from the OS CSPRNG; a fresh
// 96-bit nonce per encryption keeps the (key, nonce) pair unique.
rand::thread_rng().fill_bytes(&mut nonce_bytes);
let nonce = Nonce::from_slice(&nonce_bytes);
let ciphertext = cipher
.encrypt(nonce, plaintext)
.expect("AES-256-GCM encryption is infallible for a valid key + 12-byte nonce");
EncryptResult {
ciphertext,
nonce: nonce_bytes,
}
}
/// Decrypt `ciphertext` (auth tag appended) with the stored `nonce`
/// under `key`.
///
/// # Errors
///
/// Returns [`CryptoError::InvalidNonce`] if `nonce` is the wrong length,
/// or [`CryptoError::Decrypt`] if authentication fails for any reason
/// (wrong key, corruption, tampering).
pub fn decrypt(
ciphertext: &[u8],
nonce: &[u8],
key: &[u8; KEY_LEN],
) -> Result<Vec<u8>, CryptoError> {
if nonce.len() != NONCE_LEN {
return Err(CryptoError::InvalidNonce(nonce.len()));
}
let cipher = Aes256Gcm::new(Key::<Aes256Gcm>::from_slice(key));
let nonce = Nonce::from_slice(nonce);
cipher
.decrypt(nonce, ciphertext)
.map_err(|_| CryptoError::Decrypt)
}
/// The process-wide master key. Sourced once at startup and threaded
/// into the secrets service, the email-trigger receiver, and the
/// realtime signing-key migration.
///
/// Cheap to clone (32 bytes). `Debug` is redacted so the key never
/// lands in a log line.
#[derive(Clone)]
pub struct MasterKey {
key: [u8; KEY_LEN],
}
impl std::fmt::Debug for MasterKey {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("MasterKey")
.field("key", &"<redacted 32 bytes>")
.finish()
}
}
/// Failure modes for master-key sourcing. Every variant is a fatal
/// startup error — there is no fallback to a quiet plaintext mode.
#[derive(Debug, Error)]
pub enum MasterKeyError {
/// `PICLOUD_SECRET_KEY` is unset/empty and dev mode is off.
#[error(
"PICLOUD_SECRET_KEY is required but unset. Generate one with `openssl rand -base64 32`, \
or set PICLOUD_DEV_MODE=true to use an insecure deterministic dev key (never in production)."
)]
Missing,
/// `PICLOUD_SECRET_KEY` was not valid base64.
#[error("PICLOUD_SECRET_KEY is not valid base64 (expected base64 of 32 bytes — `openssl rand -base64 32`)")]
Malformed,
/// Decoded to the wrong number of bytes.
#[error("PICLOUD_SECRET_KEY must decode to exactly {KEY_LEN} bytes, got {0}")]
WrongLength(usize),
}
impl MasterKey {
/// Borrow the raw 32-byte key for the crypto envelope.
#[must_use]
pub const fn as_bytes(&self) -> &[u8; KEY_LEN] {
&self.key
}
/// Build a key directly from 32 bytes (used by the realtime
/// migration's tests and by [`Self::from_base64`]).
#[must_use]
pub const fn from_bytes(key: [u8; KEY_LEN]) -> Self {
Self { key }
}
/// Decode a base64-encoded 32-byte key.
///
/// # Errors
///
/// [`MasterKeyError::Malformed`] for non-base64 input,
/// [`MasterKeyError::WrongLength`] when the decoded length is not 32.
pub fn from_base64(s: &str) -> Result<Self, MasterKeyError> {
let decoded = B64
.decode(s.trim().as_bytes())
.map_err(|_| MasterKeyError::Malformed)?;
let len = decoded.len();
let key: [u8; KEY_LEN] = decoded
.try_into()
.map_err(|_| MasterKeyError::WrongLength(len))?;
Ok(Self { key })
}
/// Source the master key from the process environment per the
/// v1.1.7 rules. See [`Self::resolve`] for the decision logic.
///
/// # Errors
///
/// Propagates [`MasterKeyError`] when the key is absent (and dev
/// mode is off) or malformed.
pub fn from_env() -> Result<Self, MasterKeyError> {
let secret = std::env::var("PICLOUD_SECRET_KEY").ok();
let dev_mode = std::env::var("PICLOUD_DEV_MODE")
.map(|v| is_truthy(&v))
.unwrap_or(false);
Self::resolve(secret.as_deref(), dev_mode)
}
/// Pure resolution logic, factored out of [`Self::from_env`] so it's
/// testable without mutating process-global env vars.
///
/// * `secret` present + non-empty → parse it (fatal if malformed).
/// * `secret` absent/empty + `dev_mode` → deterministic dev key +
/// a prominent warning.
/// * `secret` absent/empty + no dev mode → fatal.
///
/// # Errors
///
/// See [`Self::from_env`].
pub fn resolve(secret: Option<&str>, dev_mode: bool) -> Result<Self, MasterKeyError> {
match secret {
Some(v) if !v.trim().is_empty() => Self::from_base64(v),
_ if dev_mode => {
tracing::warn!(
"PICLOUD_SECRET_KEY is unset and PICLOUD_DEV_MODE=true: using a DETERMINISTIC \
in-memory dev master key. At-rest secrets are NOT secure in this mode. \
Never run a real deployment without PICLOUD_SECRET_KEY."
);
Ok(Self::dev_key())
}
_ => Err(MasterKeyError::Missing),
}
}
/// Deterministic dev key: SHA-256 of a fixed label. Stable across
/// restarts so dev secrets survive a reboot, but obviously not a
/// real secret (the input is public).
#[must_use]
fn dev_key() -> Self {
let digest = Sha256::digest(b"picloud-dev-master-key-v1.1.7");
let mut key = [0u8; KEY_LEN];
key.copy_from_slice(&digest);
Self { key }
}
}
/// Common env-var truthiness check shared with the other config knobs.
fn is_truthy(v: &str) -> bool {
matches!(v.trim().to_ascii_lowercase().as_str(), "1" | "true" | "yes")
}
#[cfg(test)]
mod tests {
use super::*;
fn test_key() -> [u8; KEY_LEN] {
let mut k = [0u8; KEY_LEN];
for (i, b) in k.iter_mut().enumerate() {
*b = u8::try_from(i).unwrap_or(0);
}
k
}
#[test]
fn round_trip_recovers_plaintext() {
let key = test_key();
let plaintext = b"sk_live_super_secret_value";
let enc = encrypt(plaintext, &key);
let dec = decrypt(&enc.ciphertext, &enc.nonce, &key).unwrap();
assert_eq!(dec, plaintext);
// Tag is appended → ciphertext is longer than plaintext.
assert!(enc.ciphertext.len() > plaintext.len());
}
#[test]
fn round_trip_empty_plaintext() {
let key = test_key();
let enc = encrypt(b"", &key);
let dec = decrypt(&enc.ciphertext, &enc.nonce, &key).unwrap();
assert!(dec.is_empty());
}
#[test]
fn tampered_ciphertext_fails() {
let key = test_key();
let mut enc = encrypt(b"hello world", &key);
enc.ciphertext[0] ^= 0xff;
let err = decrypt(&enc.ciphertext, &enc.nonce, &key).unwrap_err();
assert!(matches!(err, CryptoError::Decrypt));
}
#[test]
fn tampered_nonce_fails() {
let key = test_key();
let enc = encrypt(b"hello world", &key);
let mut nonce = enc.nonce;
nonce[0] ^= 0xff;
let err = decrypt(&enc.ciphertext, &nonce, &key).unwrap_err();
assert!(matches!(err, CryptoError::Decrypt));
}
#[test]
fn wrong_key_fails() {
let key = test_key();
let mut other = test_key();
other[31] ^= 0xff;
let enc = encrypt(b"hello world", &key);
let err = decrypt(&enc.ciphertext, &enc.nonce, &other).unwrap_err();
assert!(matches!(err, CryptoError::Decrypt));
}
#[test]
fn wrong_length_nonce_rejected() {
let key = test_key();
let enc = encrypt(b"hi", &key);
let err = decrypt(&enc.ciphertext, &enc.nonce[..8], &key).unwrap_err();
assert!(matches!(err, CryptoError::InvalidNonce(8)));
}
#[test]
fn distinct_nonces_per_encryption() {
let key = test_key();
let a = encrypt(b"same plaintext", &key);
let b = encrypt(b"same plaintext", &key);
// Random nonce → ciphertext differs even for identical input.
assert_ne!(a.nonce, b.nonce);
assert_ne!(a.ciphertext, b.ciphertext);
}
#[test]
fn master_key_from_valid_base64() {
let raw = [7u8; KEY_LEN];
let b64 = B64.encode(raw);
let mk = MasterKey::from_base64(&b64).unwrap();
assert_eq!(mk.as_bytes(), &raw);
}
#[test]
fn master_key_malformed_base64() {
let err = MasterKey::from_base64("not valid base64 !!!").unwrap_err();
assert!(matches!(err, MasterKeyError::Malformed));
}
#[test]
fn master_key_wrong_length() {
let b64 = B64.encode([1u8; 16]); // 16 bytes, not 32
let err = MasterKey::from_base64(&b64).unwrap_err();
assert!(matches!(err, MasterKeyError::WrongLength(16)));
}
#[test]
fn resolve_missing_without_dev_is_fatal() {
let err = MasterKey::resolve(None, false).unwrap_err();
assert!(matches!(err, MasterKeyError::Missing));
// Empty string counts as missing too.
let err = MasterKey::resolve(Some(" "), false).unwrap_err();
assert!(matches!(err, MasterKeyError::Missing));
}
#[test]
fn resolve_dev_fallback_only_with_dev_mode() {
// Dev mode on + no key → deterministic dev key.
let a = MasterKey::resolve(None, true).unwrap();
let b = MasterKey::resolve(None, true).unwrap();
assert_eq!(a.as_bytes(), b.as_bytes(), "dev key must be deterministic");
// A real key always wins over dev mode.
let raw = [9u8; KEY_LEN];
let real = MasterKey::resolve(Some(&B64.encode(raw)), true).unwrap();
assert_eq!(real.as_bytes(), &raw);
assert_ne!(real.as_bytes(), a.as_bytes());
}
}

View File

@@ -0,0 +1,89 @@
//! `EmailService` — the v1.1.7 outbound email contract.
//!
//! Scripts get `email::send(#{...})` (plain text) and
//! `email::send_html(#{...})` (multipart text + HTML). Both route to the
//! single `send` trait method with an [`OutboundEmail`]; the bridge sets
//! `html` only for `send_html`.
//!
//! Lives in `picloud-shared` (not `manager-core`) so the Rhai bridge and
//! the impl share one trait. The impl (an SMTP relay over `lettre`)
//! lives in `manager-core::email_service`; `picloud-shared` stays free
//! of the `lettre` dependency.
//!
//! `app_id` is derived from `cx.app_id` (authz only — there is no
//! per-app `from` validation in v1.1.7; deliverability is the operator's
//! SMTP-relay concern).
use async_trait::async_trait;
use thiserror::Error;
use crate::SdkCallCx;
/// A single outbound message. `to`/`cc`/`bcc` are address lists (the
/// bridge accepts a String or an Array of Strings). At least one of
/// `text` / `html` must be present.
#[derive(Debug, Clone, Default)]
pub struct OutboundEmail {
pub to: Vec<String>,
pub cc: Vec<String>,
pub bcc: Vec<String>,
pub from: String,
/// Defaults to `from` when absent.
pub reply_to: Option<String>,
pub subject: String,
pub text: Option<String>,
pub html: Option<String>,
}
#[async_trait]
pub trait EmailService: Send + Sync {
/// Validate, build, and send the message. Returns `Ok(())` once the
/// SMTP relay has accepted it for delivery (not on actual delivery —
/// that's the relay's job).
async fn send(&self, cx: &SdkCallCx, email: OutboundEmail) -> Result<(), EmailError>;
}
/// Failure modes surfaced to the Rhai bridge.
#[derive(Debug, Error)]
pub enum EmailError {
/// Caller principal lacked `AppEmailSend`. Only raised when
/// `cx.principal.is_some()` (script-as-gate semantics).
#[error("forbidden")]
Forbidden,
/// A required field (`to`, `from`, `subject`, or one of `text`/`html`)
/// was missing or empty.
#[error("missing required email field: {0}")]
MissingField(String),
/// An address failed basic RFC 5322-ish validation.
#[error("invalid email address: {0}")]
InvalidAddress(String),
/// The assembled message exceeded the per-message size cap.
#[error("email too large: {actual} bytes exceeds the {limit}-byte limit")]
TooLarge { limit: usize, actual: usize },
/// No SMTP relay is configured (HOST/USER/PASSWORD unset). Every
/// `send` fails until the operator configures one.
#[error(
"email is not configured: set PICLOUD_SMTP_HOST/USER/PASSWORD to enable outbound email"
)]
NotConfigured,
/// The SMTP relay rejected the message or the connection failed.
#[error("email transport error: {0}")]
Transport(String),
}
/// Stub used by test harnesses that build a `Services` bundle without an
/// SMTP relay. Every call returns `EmailError::NotConfigured`.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoopEmailService;
#[async_trait]
impl EmailService for NoopEmailService {
async fn send(&self, _cx: &SdkCallCx, _email: OutboundEmail) -> Result<(), EmailError> {
Err(EmailError::NotConfigured)
}
}

339
crates/shared/src/files.rs Normal file
View File

@@ -0,0 +1,339 @@
//! `FilesService` — the v1.1.5 filesystem-backed blob store contract.
//!
//! Lives in `picloud-shared` (not `executor-core`) so the Rhai bridge,
//! the manager-core filesystem+Postgres impl, and any in-memory test
//! impl can all depend on the same trait without dragging
//! `executor-core` into a Postgres or filesystem dependency.
//!
//! Implementations MUST derive every storage `app_id` from `cx.app_id`
//! — never from a script-passed argument. That is the cross-app
//! isolation boundary; see `docs/sdk-shape.md`.
//!
//! `FilesService` is collection-scoped: scripts get a handle via
//! `files::collection(name)` and call
//! `create`/`head`/`get`/`update`/`delete`/`list` on it. The blob bytes
//! never travel through Postgres or through trigger payloads — the row
//! is metadata + a SHA-256 checksum; the bytes live on the filesystem.
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use thiserror::Error;
use uuid::Uuid;
use crate::SdkCallCx;
/// POSIX-portable filename cap (255 bytes).
pub const MAX_FILE_NAME_BYTES: usize = 255;
/// RFC 6838 puts a reasonable media-type ceiling around 127 chars.
pub const MAX_CONTENT_TYPE_BYTES: usize = 127;
/// Payload for `create` — a brand-new blob. The id is server-generated
/// (a UUID); scripts never supply it.
#[derive(Debug, Clone)]
pub struct NewFile {
pub name: String,
pub content_type: String,
pub data: Vec<u8>,
}
/// Payload for `update` — replacement bytes plus optional metadata. If
/// `name` / `content_type` are `None` the prior values are kept.
#[derive(Debug, Clone)]
pub struct FileUpdate {
pub data: Vec<u8>,
pub name: Option<String>,
pub content_type: Option<String>,
}
/// File metadata as scripts and triggers see it. Serialized into
/// `ServiceEvent.payload` (the blob bytes are NOT included — files are
/// too big to ship through trigger payloads), and surfaced to Rhai by
/// `head` / `list`.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct FileMeta {
pub id: Uuid,
pub collection: String,
pub name: String,
pub content_type: String,
pub size: u64,
/// Lowercase hex SHA-256 of the content.
pub checksum: String,
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
}
/// One page of file metadata from `FilesService::list`. `next_cursor`
/// is `Some` when more pages exist, `None` when exhausted.
#[derive(Debug, Clone)]
pub struct FilesListPage {
pub files: Vec<FileMeta>,
pub next_cursor: Option<String>,
}
#[async_trait]
pub trait FilesService: Send + Sync {
/// Create a new blob; returns its server-generated id. Throws on a
/// missing required field, an over-limit blob, or an invalid
/// collection name.
async fn create(
&self,
cx: &SdkCallCx,
collection: &str,
new: NewFile,
) -> Result<Uuid, FilesError>;
/// Metadata only — no body read. `None` if the file is missing.
async fn head(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<FileMeta>, FilesError>;
/// Full content. `None` if missing. Verifies the stored checksum
/// against the bytes on disk and returns `FilesError::Corrupted`
/// when they diverge.
async fn get(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
) -> Result<Option<Vec<u8>>, FilesError>;
/// Replace content (and optionally metadata). Throws `NotFound`
/// when the file doesn't exist.
async fn update(
&self,
cx: &SdkCallCx,
collection: &str,
id: &str,
upd: FileUpdate,
) -> Result<(), FilesError>;
/// Delete by id; returns whether the file was present.
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError>;
/// Cursor-paginated metadata listing (same shape as KV's list).
async fn list(
&self,
cx: &SdkCallCx,
collection: &str,
cursor: Option<&str>,
limit: u32,
) -> Result<FilesListPage, FilesError>;
}
/// Failure modes surfaced to the Rhai bridge. The bridge converts each
/// to a Rhai runtime error string; the discriminants exist so internal
/// callers (admin endpoints, tests) can react more precisely.
#[derive(Debug, Error)]
pub enum FilesError {
/// Empty collection name, or one containing a path separator / `..`
/// / NUL — rejected at the SDK boundary per `docs/sdk-shape.md`.
#[error("invalid collection name: {0}")]
InvalidCollection(String),
/// A required field on `create` was missing or empty. The string
/// names the field (`name` / `content_type` / `data`).
#[error("missing required field: {0}")]
MissingField(&'static str),
/// Blob exceeds the per-file size cap (default 100 MB,
/// `PICLOUD_FILES_MAX_FILE_SIZE_BYTES`).
#[error("file too large: {size} bytes exceeds limit of {limit} bytes")]
TooLarge { size: usize, limit: usize },
/// Filename exceeds `MAX_FILE_NAME_BYTES`.
#[error("file name too long: {0} bytes exceeds 255")]
NameTooLong(usize),
/// Content-type exceeds `MAX_CONTENT_TYPE_BYTES`.
#[error("content_type too long: {0} bytes exceeds 127")]
ContentTypeTooLong(usize),
/// `update` on a non-existent file.
#[error("file not found")]
NotFound,
/// The bytes on disk no longer match the stored checksum — the
/// filesystem corrupted or a backup was misconfigured. The operator
/// decides what to do with the metadata-vs-bytes mismatch; the repo
/// does NOT auto-delete.
#[error("file content corrupted (checksum mismatch)")]
Corrupted,
/// Caller principal lacked the required capability. Only raised when
/// `cx.principal.is_some()` — scripts running with `principal: None`
/// (public HTTP) operate under script-as-gate semantics and skip
/// the capability check.
#[error("forbidden")]
Forbidden,
/// Anything else — Postgres unavailable, filesystem I/O error, etc.
#[error("files backend error: {0}")]
Backend(String),
}
impl NewFile {
/// Validate required fields + length caps at the SDK boundary.
///
/// Empty `data` is **accepted** as a valid stored state (v1.1.6
/// relaxed the v1.1.5 rejection — empty files are a legitimate use
/// case: sentinels, placeholders, zero-byte uploads. See HANDBACK
/// §7). `name` and `content_type` are still required.
///
/// # Errors
///
/// Returns the field-specific [`FilesError`] for the first failing
/// check.
pub fn validate(&self, max_size: usize) -> Result<(), FilesError> {
if self.name.trim().is_empty() {
return Err(FilesError::MissingField("name"));
}
if self.content_type.trim().is_empty() {
return Err(FilesError::MissingField("content_type"));
}
if self.name.len() > MAX_FILE_NAME_BYTES {
return Err(FilesError::NameTooLong(self.name.len()));
}
if self.content_type.len() > MAX_CONTENT_TYPE_BYTES {
return Err(FilesError::ContentTypeTooLong(self.content_type.len()));
}
if self.data.len() > max_size {
return Err(FilesError::TooLarge {
size: self.data.len(),
limit: max_size,
});
}
Ok(())
}
}
impl FileUpdate {
/// Validate the replacement bytes + any supplied metadata.
///
/// # Errors
///
/// Returns the field-specific [`FilesError`] for the first failing
/// check.
pub fn validate(&self, max_size: usize) -> Result<(), FilesError> {
// Empty replacement bytes are accepted (v1.1.6 relaxation —
// consistent with NewFile::validate; updating a file to zero
// bytes is as legitimate as creating one).
if let Some(name) = &self.name {
if name.trim().is_empty() {
return Err(FilesError::MissingField("name"));
}
if name.len() > MAX_FILE_NAME_BYTES {
return Err(FilesError::NameTooLong(name.len()));
}
}
if let Some(ct) = &self.content_type {
if ct.trim().is_empty() {
return Err(FilesError::MissingField("content_type"));
}
if ct.len() > MAX_CONTENT_TYPE_BYTES {
return Err(FilesError::ContentTypeTooLong(ct.len()));
}
}
if self.data.len() > max_size {
return Err(FilesError::TooLarge {
size: self.data.len(),
limit: max_size,
});
}
Ok(())
}
}
/// Reject a collection name that is empty or could escape the per-app
/// files tree. UUID-shaped ids never produce traversal paths, but
/// collection names come from scripts so they're validated defensively
/// at both the SDK boundary and the repo.
///
/// # Errors
///
/// Returns [`FilesError::InvalidCollection`] when the name is empty or
/// contains `/`, `\`, `..`, or a NUL byte.
pub fn validate_collection(collection: &str) -> Result<(), FilesError> {
if collection.is_empty() {
return Err(FilesError::InvalidCollection("must not be empty".into()));
}
if collection.contains('/')
|| collection.contains('\\')
|| collection.contains("..")
|| collection.contains('\0')
{
return Err(FilesError::InvalidCollection(format!(
"collection {collection:?} must not contain '/', '\\', '..', or NUL"
)));
}
Ok(())
}
/// Stub used by the test harness so executor-core integration tests
/// (which don't touch files) can construct a `Services` bundle without
/// a filesystem or Postgres. Every call returns
/// `FilesError::Backend("...")` so accidental use surfaces clearly.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoopFilesService;
#[async_trait]
impl FilesService for NoopFilesService {
async fn create(
&self,
_cx: &SdkCallCx,
_collection: &str,
_new: NewFile,
) -> Result<Uuid, FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
async fn head(
&self,
_cx: &SdkCallCx,
_collection: &str,
_id: &str,
) -> Result<Option<FileMeta>, FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
async fn get(
&self,
_cx: &SdkCallCx,
_collection: &str,
_id: &str,
) -> Result<Option<Vec<u8>>, FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
async fn update(
&self,
_cx: &SdkCallCx,
_collection: &str,
_id: &str,
_upd: FileUpdate,
) -> Result<(), FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
async fn delete(
&self,
_cx: &SdkCallCx,
_collection: &str,
_id: &str,
) -> Result<bool, FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
async fn list(
&self,
_cx: &SdkCallCx,
_collection: &str,
_cursor: Option<&str>,
_limit: u32,
) -> Result<FilesListPage, FilesError> {
Err(FilesError::Backend("files is not wired in".into()))
}
}

View File

@@ -6,12 +6,15 @@
pub mod app; pub mod app;
pub mod auth; pub mod auth;
pub mod crypto;
pub mod dead_letters; pub mod dead_letters;
pub mod docs; pub mod docs;
pub mod email;
pub mod error; pub mod error;
pub mod events; pub mod events;
pub mod exec_summary; pub mod exec_summary;
pub mod execution_log; pub mod execution_log;
pub mod files;
pub mod http; pub mod http;
pub mod ids; pub mod ids;
pub mod inbox; pub mod inbox;
@@ -19,23 +22,34 @@ pub mod kv;
pub mod log_sink; pub mod log_sink;
pub mod modules; pub mod modules;
pub mod outbox_writer; pub mod outbox_writer;
pub mod pubsub;
pub mod realtime;
pub mod realtime_authority;
pub mod route; pub mod route;
pub mod sandbox; pub mod sandbox;
pub mod script; pub mod script;
pub mod sdk_cx; pub mod sdk_cx;
pub mod secrets;
pub mod services; pub mod services;
pub mod subscriber_token;
pub mod trigger_event; pub mod trigger_event;
pub mod validator; pub mod validator;
pub mod version; pub mod version;
pub use app::{App, AppDomain, DomainShape}; pub use app::{App, AppDomain, DomainShape};
pub use auth::{AppRole, InstanceRole, Principal, Scope, UserId}; pub use auth::{AppRole, InstanceRole, Principal, Scope, UserId};
pub use crypto::{decrypt, encrypt, CryptoError, EncryptResult, MasterKey, MasterKeyError};
pub use dead_letters::{DeadLetterError, DeadLetterId, DeadLetterService, NoopDeadLetterService}; pub use dead_letters::{DeadLetterError, DeadLetterId, DeadLetterService, NoopDeadLetterService};
pub use docs::{DocId, DocRow, DocsError, DocsListPage, DocsService, NoopDocsService}; pub use docs::{DocId, DocRow, DocsError, DocsListPage, DocsService, NoopDocsService};
pub use email::{EmailError, EmailService, NoopEmailService, OutboundEmail};
pub use error::Error; pub use error::Error;
pub use events::{EmitError, NoopEventEmitter, ServiceEvent, ServiceEventEmitter}; pub use events::{EmitError, NoopEventEmitter, ServiceEvent, ServiceEventEmitter};
pub use exec_summary::ExecResponseSummary; pub use exec_summary::ExecResponseSummary;
pub use execution_log::{ExecutionLog, ExecutionStatus}; pub use execution_log::{ExecutionLog, ExecutionStatus};
pub use files::{
validate_collection as validate_files_collection, FileMeta, FileUpdate, FilesError,
FilesListPage, FilesService, NewFile, NoopFilesService,
};
pub use http::{HttpError, HttpRequest, HttpResponse, HttpService, NoopHttpService}; pub use http::{HttpError, HttpRequest, HttpResponse, HttpService, NoopHttpService};
pub use ids::{AdminUserId, ApiKeyId, AppId, ExecutionId, RequestId, ScriptId, TriggerId}; pub use ids::{AdminUserId, ApiKeyId, AppId, ExecutionId, RequestId, ScriptId, TriggerId};
pub use inbox::{ pub use inbox::{
@@ -45,11 +59,22 @@ pub use kv::{KvError, KvListPage, KvService, NoopKvService};
pub use log_sink::{ExecutionLogSink, LogSinkError}; pub use log_sink::{ExecutionLogSink, LogSinkError};
pub use modules::{ModuleScript, ModuleSource, ModuleSourceError, NoopModuleSource}; pub use modules::{ModuleScript, ModuleSource, ModuleSourceError, NoopModuleSource};
pub use outbox_writer::{HttpDispatchPayload, NewHttpOutbox, OutboxWriter, OutboxWriterError}; pub use outbox_writer::{HttpDispatchPayload, NewHttpOutbox, OutboxWriter, OutboxWriterError};
pub use pubsub::{
topic_matches, validate_topic_pattern, NoopPubsubService, PubsubError, PubsubService,
};
pub use realtime::{BroadcasterError, NoopRealtimeBroadcaster, RealtimeBroadcaster, RealtimeEvent};
pub use realtime_authority::{DenyAllRealtimeAuthority, RealtimeAuthority, SubscribeDenied};
pub use route::{DispatchMode, HostKind, PathKind, Route}; pub use route::{DispatchMode, HostKind, PathKind, Route};
pub use sandbox::ScriptSandbox; pub use sandbox::ScriptSandbox;
pub use script::{Script, ScriptKind}; pub use script::{Script, ScriptKind};
pub use sdk_cx::SdkCallCx; pub use sdk_cx::SdkCallCx;
pub use secrets::{
validate_secret_name, NoopSecretsService, SecretsError, SecretsListPage, SecretsService,
SECRET_NAME_MAX_BYTES,
};
pub use services::Services; pub use services::Services;
pub use trigger_event::{DeadLetterEventDetail, DocsEventOp, KvEventOp, TriggerEvent}; pub use trigger_event::{
DeadLetterEventDetail, DocsEventOp, FilesEventOp, KvEventOp, TriggerEvent,
};
pub use validator::{ScriptValidator, ValidatedScript, ValidationError}; pub use validator::{ScriptValidator, ValidatedScript, ValidationError};
pub use version::{API_VERSION, PRODUCT_VERSION, SDK_VERSION, WIRE_VERSION}; pub use version::{API_VERSION, PRODUCT_VERSION, SDK_VERSION, WIRE_VERSION};

194
crates/shared/src/pubsub.rs Normal file
View File

@@ -0,0 +1,194 @@
//! `PubsubService` — the v1.1.5 durable pub/sub contract.
//!
//! `pubsub::publish_durable(topic, message)` writes to the universal
//! outbox; the publish-time fan-out inserts one delivery row per
//! matching `pubsub` trigger, and each delivery retries / dead-letters
//! independently (the dispatcher already handles one-row-equals-one-
//! dispatch — no dispatcher changes for pub/sub).
//!
//! `publish_ephemeral` is committed as a v1.2 addition — the suffix
//! naming exists now so users learn "durable by default" from day one.
//!
//! Topic pattern matching runs in Rust (not SQL) so the trigger-select
//! query stays simple. The matcher + validator live here in
//! `picloud-shared` so the manager-core publish path, the admin trigger
//! endpoint, and tests all agree on the rules.
use async_trait::async_trait;
use thiserror::Error;
use crate::SdkCallCx;
#[async_trait]
pub trait PubsubService: Send + Sync {
/// Durable publish: writes the message to the outbox, fanned out to
/// every matching enabled `pubsub` trigger in `cx.app_id`. Succeeds
/// silently (zero rows written) when no trigger matches the topic.
async fn publish_durable(
&self,
cx: &SdkCallCx,
topic: &str,
message: serde_json::Value,
) -> Result<(), PubsubError>;
/// Mint an HMAC-signed realtime subscriber token (v1.1.6). Backs the
/// `pubsub::subscriber_token(topics, ttl)` Rhai SDK call. The minted
/// token authorizes an external SSE client to subscribe to the given
/// `topics` for `ttl_seconds` (clamped to the configured bounds; the
/// configured default applies when `ttl_seconds` is `None`).
///
/// Every topic must already be registered as externally subscribable
/// in `cx.app_id`; `cx.principal` must be `Some` (anonymous
/// public-HTTP scripts can't mint). See [`PubsubError::SubscriberToken`]
/// for the rejection messages.
///
/// The default impl errors `Unavailable` so test fakes and the
/// `NoopPubsubService` keep compiling; the real minting lives in
/// manager-core's `PubsubServiceImpl`.
async fn mint_subscriber_token(
&self,
cx: &SdkCallCx,
topics: Vec<String>,
ttl_seconds: Option<i64>,
) -> Result<String, PubsubError> {
let _ = (cx, topics, ttl_seconds);
Err(PubsubError::Unavailable(
"subscriber tokens are not wired in".into(),
))
}
}
#[derive(Debug, Error)]
pub enum PubsubError {
/// Empty topic; rejected at the SDK boundary.
#[error("topic must not be empty")]
EmptyTopic,
/// Caller principal lacked the required capability. Only raised when
/// `cx.principal.is_some()` (script-as-gate; public HTTP skips it).
#[error("forbidden")]
Forbidden,
/// Serialization / validation failure on the message.
#[error("pubsub rejected: {0}")]
Rejected(String),
/// A `pubsub::subscriber_token` mint was rejected (empty topics,
/// unregistered topic, ttl out of range, anonymous caller). The
/// string is the full user-facing message; the SDK surfaces it
/// verbatim so scripts see the documented wording.
#[error("{0}")]
SubscriberToken(String),
/// Anything else — Postgres unavailable, etc.
#[error("pubsub backend error: {0}")]
Unavailable(String),
}
/// Match a stored `topic_pattern` against a published `topic`.
///
/// - `"*"` matches every topic.
/// - `"<prefix>.*"` matches any topic starting with `"<prefix>."`.
/// - anything else is an exact match.
///
/// Mid-pattern wildcards (`*.created`, `a.*.b`) are NOT supported — they
/// are rejected at trigger creation by [`validate_topic_pattern`], so
/// the only patterns reaching this matcher are exact / prefix / `*`.
#[must_use]
pub fn topic_matches(pattern: &str, topic: &str) -> bool {
if pattern == "*" {
return true;
}
if let Some(prefix) = pattern.strip_suffix('*') {
// `prefix` retains the trailing '.', e.g. "user." for "user.*".
return topic.starts_with(prefix);
}
pattern == topic
}
/// Validate a subscription topic pattern. Accepts exactly: `"*"`
/// (universal), `"<prefix>.*"` (prefix wildcard, single trailing star),
/// or a literal with no `*` (exact). Everything else — mid-pattern
/// wildcards, multiple stars, a star not at the end — is rejected.
///
/// # Errors
///
/// Returns `Err(message)` with `"unsupported pubsub topic pattern: …"`
/// for any unsupported shape (or an empty pattern).
pub fn validate_topic_pattern(pattern: &str) -> Result<(), String> {
if pattern.is_empty() {
return Err("unsupported pubsub topic pattern: <empty>".to_string());
}
if pattern == "*" {
return Ok(());
}
let stars = pattern.matches('*').count();
if stars == 0 {
return Ok(()); // exact
}
if stars == 1 && pattern.ends_with(".*") {
return Ok(()); // prefix wildcard
}
Err(format!("unsupported pubsub topic pattern: {pattern}"))
}
/// Stub for the test harness so executor-core integration tests can
/// build a `Services` bundle without a database. Every call errors.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoopPubsubService;
#[async_trait]
impl PubsubService for NoopPubsubService {
async fn publish_durable(
&self,
_cx: &SdkCallCx,
_topic: &str,
_message: serde_json::Value,
) -> Result<(), PubsubError> {
Err(PubsubError::Unavailable("pubsub is not wired in".into()))
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn exact_match() {
assert!(topic_matches("user.created", "user.created"));
assert!(!topic_matches("user.created", "user.deleted"));
assert!(!topic_matches("user.created", "user.created.x"));
}
#[test]
fn prefix_wildcard() {
assert!(topic_matches("user.*", "user.created"));
assert!(topic_matches("user.*", "user.deleted"));
assert!(!topic_matches("user.*", "users.created"));
assert!(!topic_matches("user.*", "order.created"));
}
#[test]
fn universal() {
assert!(topic_matches("*", "anything"));
assert!(topic_matches("*", "a.b.c"));
}
#[test]
fn validation_accepts_supported_shapes() {
assert!(validate_topic_pattern("*").is_ok());
assert!(validate_topic_pattern("user.created").is_ok());
assert!(validate_topic_pattern("user.*").is_ok());
assert!(validate_topic_pattern("a.b.c").is_ok());
}
#[test]
fn validation_rejects_unsupported_shapes() {
for bad in ["*.created", "**", "a.*.b", "user.*x", "*user", ""] {
assert!(
validate_topic_pattern(bad).is_err(),
"expected {bad:?} to be rejected"
);
}
}
}

View File

@@ -0,0 +1,86 @@
//! `RealtimeBroadcaster` — the in-process fan-out seam for SSE realtime
//! delivery (v1.1.6).
//!
//! Structurally a sibling of [`crate::inbox::InboxResolver`]: the trait
//! lives here in `picloud-shared` because the publish side
//! (`PubsubServiceImpl` in manager-core) and the subscribe side (the SSE
//! handler in orchestrator-core) live in different crates and both need
//! one shared instance. The in-process impl lives in orchestrator-core
//! (`Mutex<HashMap<(AppId, topic), broadcast::Sender>>`); cluster mode
//! (v1.3+) swaps it for a Postgres `LISTEN/NOTIFY`-backed resolver behind
//! the same trait without touching either caller.
//!
//! Delivery is **best-effort, at-most-once**: this is the realtime path,
//! NOT the durable one. Durable trigger fan-out (retry / dead-letter)
//! goes through the outbox and is the publish caller's separate concern.
//! A slow SSE consumer loses the oldest events (bounded broadcast
//! buffer); SSE's own transport-layer auto-reconnect is the recovery
//! mechanism (no server-side replay in v1.1.6).
use async_trait::async_trait;
use chrono::{DateTime, Utc};
use thiserror::Error;
use tokio::sync::broadcast;
use crate::AppId;
/// A single realtime event delivered to in-process SSE subscribers. The
/// SSE handler serializes this to `data: {...}\n\n` on the wire.
#[derive(Debug, Clone)]
pub struct RealtimeEvent {
pub topic: String,
pub message: serde_json::Value,
pub published_at: DateTime<Utc>,
}
#[derive(Debug, Error)]
pub enum BroadcasterError {
/// Reserved for backends that can fail to register a subscriber
/// (e.g. the cluster-mode `LISTEN/NOTIFY` resolver). The in-process
/// impl never returns this.
#[error("realtime broadcaster unavailable: {0}")]
Unavailable(String),
}
#[async_trait]
pub trait RealtimeBroadcaster: Send + Sync {
/// Subscribe to events on `(app_id, topic)`. Returns a receiver that
/// yields events until dropped. Channels are created lazily on first
/// subscribe.
async fn subscribe(
&self,
app_id: AppId,
topic: &str,
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError>;
/// Publish an event to in-process subscribers. NOT durable — the
/// outbox-backed durable fan-out is the publish caller's separate
/// concern. A publish with no live subscribers is a silent no-op.
async fn publish(&self, app_id: AppId, topic: &str, event: RealtimeEvent);
/// Drop every subscriber for a topic (called on topic DELETE). Live
/// receivers observe a closed channel and disconnect cleanly.
async fn drop_topic(&self, app_id: AppId, topic: &str);
}
/// Bootstrap / test impl: subscribe yields a receiver on a throwaway
/// channel, publish is a no-op. Lets a `Services`-style bundle build
/// without the real registry wired in.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoopRealtimeBroadcaster;
#[async_trait]
impl RealtimeBroadcaster for NoopRealtimeBroadcaster {
async fn subscribe(
&self,
_app_id: AppId,
_topic: &str,
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
let (_tx, rx) = broadcast::channel(1);
Ok(rx)
}
async fn publish(&self, _app_id: AppId, _topic: &str, _event: RealtimeEvent) {}
async fn drop_topic(&self, _app_id: AppId, _topic: &str) {}
}

View File

@@ -0,0 +1,70 @@
//! `RealtimeAuthority` — the SSE subscribe authorization seam (v1.1.6).
//!
//! The SSE endpoint (`GET /realtime/topics/{topic}`) is a data-plane
//! surface in orchestrator-core, but deciding whether a subscribe is
//! allowed needs a `topics` table read plus (for token-gated topics) an
//! HMAC verify against the app's signing key — both of which require DB
//! access and the signing-key material that must NOT leak into the
//! data-plane crate. This trait keeps all of that inside the manager-core
//! impl: orchestrator-core only ever sees the three-way verdict below.
//!
//! `NotFound` is deliberately returned for *both* "no such topic" and
//! "topic exists but isn't externally subscribable" so the endpoint
//! can't be used to probe which internal topics exist (design notes §5).
use async_trait::async_trait;
use crate::AppId;
/// Why a subscribe attempt was refused. The SSE handler maps these to
/// HTTP status codes.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SubscribeDenied {
/// No externally-subscribable topic by that name in this app → 404.
/// Used for genuinely-missing topics AND internal-only ones, so the
/// endpoint doesn't leak which internal topics exist.
NotFound,
/// The topic is token-gated and the presented token was missing,
/// malformed, badly signed, expired, or not scoped to this topic →
/// 401 (generic; never says which check failed).
Unauthorized,
/// Backend failure (DB unavailable, etc.) → 500.
Backend(String),
}
#[async_trait]
pub trait RealtimeAuthority: Send + Sync {
/// Decide whether an external client may subscribe to
/// `(app_id, topic)`. `token` is the bearer/query token if the
/// client presented one (`None` otherwise).
///
/// Returns `Ok(())` when the subscribe is permitted (public topic,
/// or token-gated topic with a valid token scoped to it).
///
/// # Errors
///
/// [`SubscribeDenied`] — see the variants for the status mapping.
async fn authorize_subscribe(
&self,
app_id: AppId,
topic: &str,
token: Option<&str>,
) -> Result<(), SubscribeDenied>;
}
/// Bootstrap impl: denies everything as `NotFound`. Replaced in
/// `build_app` with the manager-core DB-backed authority.
#[derive(Debug, Default, Clone, Copy)]
pub struct DenyAllRealtimeAuthority;
#[async_trait]
impl RealtimeAuthority for DenyAllRealtimeAuthority {
async fn authorize_subscribe(
&self,
_app_id: AppId,
_topic: &str,
_token: Option<&str>,
) -> Result<(), SubscribeDenied> {
Err(SubscribeDenied::NotFound)
}
}

View File

@@ -0,0 +1,166 @@
//! `SecretsService` — the v1.1.7 encrypted per-app secrets contract.
//!
//! Collection-less (per-app, like pubsub): the script API is the bare
//! `secrets::{get,set,delete,list}(name)` — there is no
//! `secrets::collection(...)`. Secrets are operational config (API keys,
//! OAuth tokens, webhook signing keys), encrypted at rest with the
//! process master key.
//!
//! Lives in `picloud-shared` (not `executor-core`) so the Rhai bridge,
//! the manager-core Postgres impl, and test fakes can all depend on the
//! same trait. Implementations MUST derive every storage `app_id` from
//! `cx.app_id` — never from a script-passed argument. That is the
//! cross-app isolation boundary; see `docs/sdk-shape.md`.
//!
//! Values are JSON internally: `set` accepts any `serde_json::Value`
//! (the bridge maps a Rhai String/Map/Array to JSON), encrypts the
//! encoded bytes, and `get` decrypts + decodes back to the same JSON
//! shape — so a String round-trips to a String, not a JSON-quoted
//! `"\"…\""`. There is deliberately **no `ServiceEvent` emission**:
//! firing triggers on secret writes is a footgun (every rotation would
//! fan out handler executions that might log the new value).
use async_trait::async_trait;
use thiserror::Error;
use crate::SdkCallCx;
/// Maximum secret name length in bytes (matches the brief: 255).
pub const SECRET_NAME_MAX_BYTES: usize = 255;
/// `SecretsService` is collection-less and per-app. Every method derives
/// the owning `app_id` from `cx.app_id`.
#[async_trait]
pub trait SecretsService: Send + Sync {
/// Decrypt and return the secret, or `None` if no secret with this
/// name exists for the app.
async fn get(
&self,
cx: &SdkCallCx,
name: &str,
) -> Result<Option<serde_json::Value>, SecretsError>;
/// Encrypt and store the secret, overwriting any existing value for
/// this name.
async fn set(
&self,
cx: &SdkCallCx,
name: &str,
value: serde_json::Value,
) -> Result<(), SecretsError>;
/// Delete the secret. Returns whether a secret was present.
async fn delete(&self, cx: &SdkCallCx, name: &str) -> Result<bool, SecretsError>;
/// List secret **names only** (never values), cursor-paginated like
/// KV/files `list`. `cursor` is opaque; `None` starts from the
/// beginning.
async fn list(
&self,
cx: &SdkCallCx,
cursor: Option<&str>,
limit: u32,
) -> Result<SecretsListPage, SecretsError>;
}
/// One page of secret names from `SecretsService::list`. `next_cursor`
/// is `Some` when more pages exist.
#[derive(Debug, Clone)]
pub struct SecretsListPage {
pub names: Vec<String>,
pub next_cursor: Option<String>,
}
/// Failure modes surfaced to the Rhai bridge. The bridge converts each
/// to a Rhai runtime error string.
#[derive(Debug, Error)]
pub enum SecretsError {
/// Empty name, or a name longer than [`SECRET_NAME_MAX_BYTES`].
#[error("{0}")]
InvalidName(String),
/// The encoded plaintext exceeded the configured per-secret cap.
#[error("secret value too large: {actual} bytes exceeds the {limit}-byte limit")]
TooLarge { limit: usize, actual: usize },
/// Caller principal lacked the required capability. Only raised when
/// `cx.principal.is_some()` — public-HTTP scripts (`principal: None`)
/// operate under script-as-gate semantics and skip the check.
#[error("forbidden")]
Forbidden,
/// The stored ciphertext could not be decrypted (corrupted row,
/// wrong master key, or tampering). The impl logs the affected
/// `(app_id, name)` at error level before returning this.
#[error("secret is corrupted or was encrypted with a different master key")]
Corrupted,
/// The process master key was unavailable. Startup should already
/// have failed; this is defense in depth.
#[error("master key is not configured")]
MasterKeyMissing,
/// Anything else — Postgres unavailable, serialization failure, etc.
#[error("secrets backend error: {0}")]
Backend(String),
}
/// Stub used by the executor-core test harness (which doesn't touch
/// secrets) so a `Services` bundle can be built without Postgres. Every
/// call returns `SecretsError::Backend(...)` so accidental use surfaces.
#[derive(Debug, Default, Clone, Copy)]
pub struct NoopSecretsService;
#[async_trait]
impl SecretsService for NoopSecretsService {
async fn get(
&self,
_cx: &SdkCallCx,
_name: &str,
) -> Result<Option<serde_json::Value>, SecretsError> {
Err(SecretsError::Backend("secrets is not wired in".into()))
}
async fn set(
&self,
_cx: &SdkCallCx,
_name: &str,
_value: serde_json::Value,
) -> Result<(), SecretsError> {
Err(SecretsError::Backend("secrets is not wired in".into()))
}
async fn delete(&self, _cx: &SdkCallCx, _name: &str) -> Result<bool, SecretsError> {
Err(SecretsError::Backend("secrets is not wired in".into()))
}
async fn list(
&self,
_cx: &SdkCallCx,
_cursor: Option<&str>,
_limit: u32,
) -> Result<SecretsListPage, SecretsError> {
Err(SecretsError::Backend("secrets is not wired in".into()))
}
}
/// Validate a secret name at the SDK/admin boundary: non-empty and at
/// most [`SECRET_NAME_MAX_BYTES`] bytes.
///
/// # Errors
///
/// Returns [`SecretsError::InvalidName`] when empty or too long.
pub fn validate_secret_name(name: &str) -> Result<(), SecretsError> {
if name.is_empty() {
return Err(SecretsError::InvalidName(
"secret name must not be empty".into(),
));
}
if name.len() > SECRET_NAME_MAX_BYTES {
return Err(SecretsError::InvalidName(format!(
"secret name must be at most {SECRET_NAME_MAX_BYTES} bytes, got {}",
name.len()
)));
}
Ok(())
}

View File

@@ -20,9 +20,10 @@
use std::sync::Arc; use std::sync::Arc;
use crate::{ use crate::{
DeadLetterService, DocsService, HttpService, KvService, ModuleSource, NoopDeadLetterService, DeadLetterService, DocsService, EmailService, FilesService, HttpService, KvService,
NoopDocsService, NoopEventEmitter, NoopHttpService, NoopKvService, NoopModuleSource, ModuleSource, NoopDeadLetterService, NoopDocsService, NoopEmailService, NoopEventEmitter,
ServiceEventEmitter, NoopFilesService, NoopHttpService, NoopKvService, NoopModuleSource, NoopPubsubService,
NoopSecretsService, PubsubService, SecretsService, ServiceEventEmitter,
}; };
/// SDK service bundle. See module docs for the lifecycle and the v1.1.x /// SDK service bundle. See module docs for the lifecycle and the v1.1.x
@@ -60,6 +61,31 @@ pub struct Services {
/// the picloud binary; `NoopHttpService` in tests that don't make /// the picloud binary; `NoopHttpService` in tests that don't make
/// network calls. /// network calls.
pub http: Arc<dyn HttpService>, pub http: Arc<dyn HttpService>,
/// Filesystem-backed blob storage (v1.1.5). Scripts get
/// `files::collection(name).{create,head,get,update,delete,list}`.
/// Backed by a Postgres-metadata + on-disk-bytes repo in the
/// picloud binary; `NoopFilesService` in tests that don't touch
/// files.
pub files: Arc<dyn FilesService>,
/// Durable pub/sub (v1.1.5). Scripts get
/// `pubsub::publish_durable(topic, message)`. Backed by a
/// publish-time outbox fan-out in the picloud binary;
/// `NoopPubsubService` in tests that don't publish.
pub pubsub: Arc<dyn PubsubService>,
/// Encrypted per-app secrets (v1.1.7). Scripts get
/// `secrets::{get,set,delete,list}(name)`. Backed by an
/// AES-256-GCM-at-rest Postgres repo in the picloud binary;
/// `NoopSecretsService` in tests that don't touch secrets.
pub secrets: Arc<dyn SecretsService>,
/// Outbound email (v1.1.7). Scripts get `email::{send,send_html}`.
/// Backed by an SMTP relay (lettre) in the picloud binary;
/// `NoopEmailService` (always `NotConfigured`) in tests that don't
/// send mail.
pub email: Arc<dyn EmailService>,
} }
impl Services { impl Services {
@@ -67,6 +93,7 @@ impl Services {
/// The picloud binary's `main` wires this up after the DB pool is /// The picloud binary's `main` wires this up after the DB pool is
/// open; tests build it from in-memory fakes. /// open; tests build it from in-memory fakes.
#[must_use] #[must_use]
#[allow(clippy::too_many_arguments)] // one Arc per stateful service; a builder would just move the noise
pub fn new( pub fn new(
kv: Arc<dyn KvService>, kv: Arc<dyn KvService>,
docs: Arc<dyn DocsService>, docs: Arc<dyn DocsService>,
@@ -74,6 +101,10 @@ impl Services {
events: Arc<dyn ServiceEventEmitter>, events: Arc<dyn ServiceEventEmitter>,
modules: Arc<dyn ModuleSource>, modules: Arc<dyn ModuleSource>,
http: Arc<dyn HttpService>, http: Arc<dyn HttpService>,
files: Arc<dyn FilesService>,
pubsub: Arc<dyn PubsubService>,
secrets: Arc<dyn SecretsService>,
email: Arc<dyn EmailService>,
) -> Self { ) -> Self {
Self { Self {
kv, kv,
@@ -82,6 +113,10 @@ impl Services {
events, events,
modules, modules,
http, http,
files,
pubsub,
secrets,
email,
} }
} }
@@ -99,6 +134,10 @@ impl Services {
Arc::new(NoopEventEmitter), Arc::new(NoopEventEmitter),
Arc::new(NoopModuleSource), Arc::new(NoopModuleSource),
Arc::new(NoopHttpService), Arc::new(NoopHttpService),
Arc::new(NoopFilesService),
Arc::new(NoopPubsubService),
Arc::new(NoopSecretsService),
Arc::new(NoopEmailService),
) )
} }
} }

Some files were not shown because too many files have changed in this diff Show More