Compare commits
12 Commits
feat/v1.1.
...
feat/v1.1.
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
64ad978a89 | ||
|
|
f5a3f92484 | ||
|
|
b1dddb9cb9 | ||
|
|
fcbcc576a2 | ||
|
|
d064681c49 | ||
|
|
9492c18d0e | ||
|
|
4595db7a7a | ||
|
|
834c787ee1 | ||
|
|
6e132b6ee0 | ||
|
|
03d03ea6e7 | ||
|
|
6080fc67f6 | ||
|
|
10b5f655d5 |
72
.github/workflows/ci.yml
vendored
Normal file
72
.github/workflows/ci.yml
vendored
Normal file
@@ -0,0 +1,72 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
|
||||
env:
|
||||
CARGO_TERM_COLOR: always
|
||||
# Matches what docker-compose produces locally; the schema-snapshot
|
||||
# guardrail and any other DB-backed tests run against this service.
|
||||
DATABASE_URL: postgres://picloud:picloud@localhost:5432/picloud
|
||||
|
||||
jobs:
|
||||
rust:
|
||||
name: Rust — fmt, clippy, test
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_USER: picloud
|
||||
POSTGRES_PASSWORD: picloud
|
||||
POSTGRES_DB: picloud
|
||||
ports:
|
||||
- 5432:5432
|
||||
options: >-
|
||||
--health-cmd "pg_isready -U picloud"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# rust-toolchain.toml pins the channel; this action honors it.
|
||||
- name: Install Rust toolchain
|
||||
uses: dtolnay/rust-toolchain@stable
|
||||
with:
|
||||
components: rustfmt, clippy
|
||||
|
||||
- name: Cache cargo
|
||||
uses: Swatinem/rust-cache@v2
|
||||
|
||||
- name: Format check
|
||||
run: cargo fmt --all -- --check
|
||||
|
||||
- name: Clippy
|
||||
run: cargo clippy --all-targets --all-features -- -D warnings
|
||||
|
||||
# Runs the whole workspace, including the schema-snapshot guardrail
|
||||
# (it picks up DATABASE_URL from the env above and the postgres
|
||||
# service; without a DB it would skip cleanly).
|
||||
- name: Test
|
||||
run: cargo test --workspace
|
||||
|
||||
dashboard:
|
||||
name: Dashboard — check
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: dashboard
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 20
|
||||
cache: npm
|
||||
cache-dependency-path: dashboard/package-lock.json
|
||||
- name: Install deps
|
||||
run: npm ci
|
||||
- name: Svelte check
|
||||
run: npm run check
|
||||
3
.gitignore
vendored
3
.gitignore
vendored
@@ -22,6 +22,9 @@ Cargo.lock.bak
|
||||
# Local config overrides
|
||||
config.local.toml
|
||||
/data
|
||||
# Files-root blob storage created when integration tests run build_app
|
||||
# from the picloud crate dir (PICLOUD_FILES_ROOT default ./data).
|
||||
/crates/picloud/data
|
||||
/postgres-data
|
||||
|
||||
# Dashboard
|
||||
|
||||
259
CHANGELOG.md
259
CHANGELOG.md
@@ -1,5 +1,249 @@
|
||||
# PiCloud Changelog
|
||||
|
||||
## v1.1.6 — Realtime Channels & Client Library (unreleased)
|
||||
|
||||
The first **external realtime surface** and the first **frontend
|
||||
library**, co-shipped per the §5/§6 design-notes decisions. Browser
|
||||
clients can subscribe over SSE to per-app pub/sub topics that have been
|
||||
explicitly externalized; everything else stays internal-only. The
|
||||
`@picloud/client` TypeScript package wraps typed HTTP, SSE, auth, and
|
||||
React/Svelte hooks. Plus three v1.1.5 follow-ups.
|
||||
|
||||
### Added — Realtime
|
||||
|
||||
- **`topics` registry** (`migrations/0021_topics.sql`) — pub/sub topics
|
||||
are internal-only by default; a `topics` row with
|
||||
`external_subscribable = true` opts one into external SSE subscription.
|
||||
`auth_mode` is `'public'` or `'token'`.
|
||||
- **Topic admin endpoints** under `/api/v1/admin/apps/{id}/topics` —
|
||||
`POST` (register), `GET` (list), `PATCH /{name}` (flip
|
||||
external/auth_mode — its own audited surface), `DELETE /{name}`
|
||||
(unregister + disconnect live subscribers). Gated by the new
|
||||
`Capability::AppTopicManage` → `app:admin` scope (no new scope; the
|
||||
seven-scope commitment holds).
|
||||
- **SSE endpoint `GET /realtime/topics/{topic}`** — data-plane surface
|
||||
(deliberately not under `/api/`). Resolves `Host` → app, authorizes
|
||||
via the `RealtimeAuthority` (404 for missing/internal topics, 401 for
|
||||
bad/absent tokens), then streams `data: {topic,message,published_at}`
|
||||
events with a configurable heartbeat (`PICLOUD_REALTIME_HEARTBEAT_SEC`,
|
||||
default 30). Token via `Authorization: Bearer` or `?token=`.
|
||||
- **`RealtimeBroadcaster` + `RealtimeEvent` + `RealtimeAuthority`**
|
||||
traits (`picloud-shared`); in-process `InProcessBroadcaster`
|
||||
(`tokio::sync::broadcast`, per-channel capacity
|
||||
`PICLOUD_REALTIME_BROADCAST_CAPACITY` default 64, periodic empty-channel
|
||||
GC) and the DB-backed `RealtimeAuthorityImpl` (orchestrator-core /
|
||||
manager-core respectively). The publish path now also fans out to
|
||||
in-process SSE subscribers, best-effort, after the durable outbox
|
||||
fan-out commits — a broadcast failure never fails the publish.
|
||||
- **`pubsub::subscriber_token(topics, ttl)`** Rhai SDK (SDK schema
|
||||
1.6 → 1.7) — mints an HMAC-SHA256 subscriber token (URL-safe
|
||||
`payload.signature`) scoped to externally-subscribable topics.
|
||||
Requires an authenticated principal + the pub/sub publish capability.
|
||||
TTL clamped to `[10s, 24h]` (default 1h), env-overridable via
|
||||
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`. Per-app signing
|
||||
keys persist in the new `app_secrets` table
|
||||
(`migrations/0022_app_secrets.sql`), created lazily on first mint. No
|
||||
per-token revocation (rotation invalidates wholesale; short TTL is the
|
||||
safety mechanism).
|
||||
- **Dashboard Topics tab** — register/list/edit/delete topics with a
|
||||
prominent external/internal badge, auth-mode radio (conditional on
|
||||
external), and a confirmation when flipping a topic external.
|
||||
|
||||
### Added — `@picloud/client` (TypeScript, v1.0.0)
|
||||
|
||||
- New top-level package `clients/typescript/` (tsup dual ESM+CJS +
|
||||
`.d.ts`, vitest). Typed HTTP via `endpoint<Req,Res>(path).get()/.post()`
|
||||
with auth-token injection and structured errors; SSE `subscribe(topic,
|
||||
cb, {token, onTokenExpired})` with exponential-backoff reconnect,
|
||||
401 token-refresh, and `Last-Event-ID` resume; `auth.login/logout/token`
|
||||
over dev-defined endpoints; React (`useTopic`/`useEndpoint` +
|
||||
`PicloudProvider`) and Svelte (`topicStore`/`endpointStore`) subpath
|
||||
exports. Optional zod/valibot runtime validation via a `{ parse }`
|
||||
adapter (no hard dep). Hybrid model: no direct service access from the
|
||||
browser.
|
||||
|
||||
### Changed / Fixed — v1.1.5 follow-ups
|
||||
|
||||
- **Empty blobs accepted** — `NewFile::validate` / `FileUpdate::validate`
|
||||
no longer reject zero-length `data`; empty files are a valid stored
|
||||
state (sentinels, placeholders). Non-breaking.
|
||||
- **Orphan `*.tmp.*` sweeper** — a startup tokio task
|
||||
(`spawn_files_orphan_sweep`) walks the files root every
|
||||
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC` (default 6h) and unlinks temp
|
||||
blobs older than `PICLOUD_FILES_ORPHAN_TMP_TTL_SEC` (default 1h). No DB
|
||||
cross-check (that full reconciler is v1.3+).
|
||||
- **Dispatcher end-to-end tests** — `crates/picloud/tests/dispatcher_e2e.rs`,
|
||||
one per trigger kind (kv/docs/cron/files/pubsub/dead_letter),
|
||||
DATABASE_URL-gated (skip cleanly when unset).
|
||||
|
||||
### Notes
|
||||
|
||||
- New deps: `hmac` (token signing, picloud-shared), `tokio-stream` (SSE
|
||||
body stream, orchestrator-core).
|
||||
- New env vars: `PICLOUD_REALTIME_HEARTBEAT_SEC`,
|
||||
`PICLOUD_REALTIME_BROADCAST_CAPACITY`,
|
||||
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`,
|
||||
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC`,
|
||||
`PICLOUD_FILES_ORPHAN_TMP_TTL_SEC`.
|
||||
|
||||
## v1.1.5 — Files & Pub/Sub (unreleased)
|
||||
|
||||
Two stateful services + two trigger kinds. **`files::*`** is
|
||||
filesystem-backed blob storage (atomic writes, path-sharded layout,
|
||||
single-pass SHA-256 with checksum-verified reads); the metadata row
|
||||
lives in Postgres, the bytes on disk. **`pubsub::publish_durable`** is
|
||||
durable pub/sub through the universal outbox, fanning out one delivery
|
||||
row per matching subscriber **at publish time** inside a single
|
||||
transaction. Both ride the v1.1.1 trigger framework as the fifth and
|
||||
sixth concrete kinds via the established Layout-E extension pattern.
|
||||
|
||||
### Added
|
||||
|
||||
- **`files::collection(name).{create,head,get,update,delete,list}`** —
|
||||
blob storage SDK. `create`/`update` take a Rhai `Blob`; `get` returns
|
||||
a `Blob` (or `()` if missing); `head`/`list` return metadata maps
|
||||
(`id, name, content_type, size, checksum, created_at, updated_at`).
|
||||
`create`/`update`/`delete` throw on failure; `get`/`head` return `()`
|
||||
for a missing file; `delete` returns a was-present bool. Missing
|
||||
required field on `create` throws naming the field.
|
||||
- **Atomic writes** — temp file → fsync → rename → fsync parent dir →
|
||||
DB row, so a crash never leaves a readable half-written file. SHA-256
|
||||
is computed in a single pass during the write; `get` re-verifies it
|
||||
and surfaces `FilesError::Corrupted` (logged with the path, never
|
||||
auto-deleted) on a mismatch. Shard dirs are created `0o700`.
|
||||
- **`files:*` trigger kind** — `ctx.event.files` carries the metadata
|
||||
only (never the bytes; a handler that wants them calls
|
||||
`files::collection(c).get(id)`). `prev` is `()` on create, the prior
|
||||
metadata on update, the deleted metadata on delete.
|
||||
- **`pubsub::publish_durable(topic, message)`** — durable publish.
|
||||
Message is any JSON-serializable Rhai value; Blobs encode as base64
|
||||
(at any nesting depth). No matching subscriber → the publish succeeds
|
||||
silently with zero outbox rows.
|
||||
- **`pubsub:*` trigger kind** — topic patterns are exact, `<prefix>.*`,
|
||||
or `*`; mid-pattern wildcards are rejected at trigger creation.
|
||||
`ctx.event.pubsub` carries `topic`, `message`, `published_at`.
|
||||
- **`FilesService` + `PubsubService` traits** (`picloud-shared`) +
|
||||
`FsFilesRepo`/`FilesServiceImpl` and `PostgresPubsubRepo`/
|
||||
`PubsubServiceImpl` (manager-core). Wired into the `Services` bundle
|
||||
as `files` and `pubsub`.
|
||||
- **Capabilities** `AppFilesRead`/`AppFilesWrite` → `script:read`/
|
||||
`script:write`, `AppPubsubPublish` → `script:write`. No new `Scope`
|
||||
variant — the seven-scope commitment holds. Script-as-gate: skipped
|
||||
when the script runs unauthenticated.
|
||||
- **Admin files API** (`GET`/`DELETE /apps/{id}/files`) + dashboard
|
||||
Files view per app; **Pub/Sub trigger form** on the Triggers tab.
|
||||
- **CI** — first `.github/workflows/ci.yml` (Postgres service, fmt +
|
||||
clippy + `cargo test --workspace`); the schema-snapshot guardrail now
|
||||
runs instead of being `#[ignore]`'d.
|
||||
|
||||
### Changed
|
||||
|
||||
- Workspace version: 1.1.4 → 1.1.5
|
||||
- Rhai SDK version: 1.5 → 1.6
|
||||
- Dashboard version: 0.10.0 → 0.11.0
|
||||
- `schema_snapshot` test: no longer `#[ignore]`'d — runs against
|
||||
`DATABASE_URL` when set, skips cleanly when absent.
|
||||
|
||||
### Migrations
|
||||
|
||||
- 0018_files.sql — `files` metadata table (bytes live on disk).
|
||||
- 0019_files_triggers.sql — widen kind/source_kind CHECKs + add
|
||||
`files_trigger_details`.
|
||||
- 0020_pubsub_triggers.sql — widen kind/source_kind CHECKs + add
|
||||
`pubsub_trigger_details` + partial index.
|
||||
|
||||
### New environment variables
|
||||
|
||||
- `PICLOUD_FILES_ROOT` (default `./data`)
|
||||
- `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` (default 100 MB)
|
||||
|
||||
## v1.1.4 — Outbound HTTP & Cron triggers (unreleased)
|
||||
|
||||
Two surfaces. **`http::*`** lets Rhai scripts make outbound HTTP
|
||||
requests (Slack webhooks, Stripe, third-party REST) fronted by an SSRF
|
||||
deny-list applied to the *resolved IP* (DNS-rebinding defense), with
|
||||
scheme/port restrictions, request/response body caps, and a layered
|
||||
timeout. **Cron triggers** add the fourth concrete kind on the v1.1.1
|
||||
trigger framework: a scheduler task enqueues due triggers into the same
|
||||
universal outbox the dispatcher already drains.
|
||||
|
||||
### Added
|
||||
|
||||
- **`http::{get,post,put,patch,delete,head,post_form,request}`** — outbound
|
||||
HTTP SDK. Body and options are separate positional args
|
||||
(`verb(url, body, opts)`); `opts` is
|
||||
`{headers, timeout_ms, follow_redirects, max_redirects}` (unknown keys
|
||||
throw). Body dispatch by type: Map/Array → JSON, String → text/plain,
|
||||
`()` → none. Response is `#{ status, headers, body, body_raw }` with
|
||||
`body` auto-parsed when the response is `application/json`. Non-2xx
|
||||
does NOT throw (fetch-style); network/timeout/SSRF/size errors throw
|
||||
with an `"http: …"` prefix.
|
||||
- **SSRF deny-list** — applied to the resolved IP via a custom reqwest
|
||||
`dns_resolver` (so it covers every redirect hop and defeats DNS
|
||||
rebinding), plus a literal-IP check at URL-parse time. Blocks
|
||||
loopback, RFC1918 private, link-local (incl. `169.254.169.254`),
|
||||
carrier-grade NAT, multicast, reserved, IPv6 ULA/link-local/loopback,
|
||||
and IPv4-mapped IPv6 (re-checked against the embedded v4 address).
|
||||
The script-visible error carries a CIDR-category reason, never the IP.
|
||||
`PICLOUD_HTTP_ALLOW_PRIVATE=true` disables it (dev-only; logs a startup
|
||||
warning).
|
||||
- **`HttpService` trait** (`picloud-shared`) + `HttpServiceImpl`
|
||||
(manager-core, reqwest-backed). Wired into the `Services` bundle as
|
||||
`http: Arc<dyn HttpService>`.
|
||||
- **`Capability::AppHttpRequest(AppId)`** — maps to the existing
|
||||
`script:write` scope (any outbound request can exfiltrate data, so the
|
||||
conservative write mapping is used). No new `Scope` variant — the
|
||||
seven-scope commitment holds. Script-as-gate: skipped when the script
|
||||
runs unauthenticated.
|
||||
- **Cron triggers** — `POST /api/v1/admin/apps/{id}/triggers/cron`
|
||||
(`script_id`, `schedule`, `timezone`, optional retry overrides).
|
||||
6-field cron expressions (with seconds) validated by the `cron` crate;
|
||||
IANA timezones validated by `chrono-tz`. A scheduler task
|
||||
(`spawn_cron_scheduler`, poll cadence `PICLOUD_CRON_TICK_INTERVAL_MS`,
|
||||
default 30s) enqueues due triggers into the outbox; the existing
|
||||
dispatcher delivers them. Catch-up policy: a trigger that missed N
|
||||
windows fires exactly **once** on the next tick, not N times.
|
||||
- **`ctx.event.cron`** — `{ schedule, timezone, scheduled_at, fired_at }`
|
||||
for cron-trigger handlers (`ctx.event.source == "cron"`,
|
||||
`ctx.event.op == "tick"`).
|
||||
- **Dashboard Triggers tab** — admin-gated cron trigger create form
|
||||
(target endpoint script, schedule, timezone dropdown) + triggers list
|
||||
showing schedule / timezone / last-fired.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Workspace version**: `1.1.3` → `1.1.4`.
|
||||
- **Rhai SDK version**: `1.4` → `1.5` (additive — `http::*` SDK +
|
||||
`ctx.event.cron`). The `Services` bundle constructor becomes
|
||||
`Services::new(kv, docs, dead_letters, events, modules, http)`.
|
||||
- **Dashboard version**: `0.9.0` → `0.10.0`.
|
||||
- **`SdkCallCx`** — gains a `script_id` field (audit attribution + the
|
||||
default outbound `User-Agent`, `picloud/<version> (script:<id>)`).
|
||||
- **Rhai pin tightened** — workspace dep `rhai = "1.19"` → `rhai = "=1.24"`
|
||||
so future bumps of the non-semver-stable `internals` surface are
|
||||
deliberate.
|
||||
- **Module backend errors redacted** — `PicloudModuleResolver` now
|
||||
surfaces a stable generic (`"module backend unavailable; check server
|
||||
logs"`) to scripts and logs the original at error level, instead of
|
||||
leaking the backend error verbatim (see v1.1.3 follow-up).
|
||||
|
||||
### Migrations
|
||||
|
||||
- `0017_cron_triggers.sql` — widens `triggers.kind` and
|
||||
`outbox.source_kind` CHECK constraints to include `'cron'`; adds
|
||||
`cron_trigger_details (trigger_id, schedule, timezone, last_fired_at)`
|
||||
with a `last_fired_at` index. Additive — applies cleanly on a fresh DB
|
||||
and on top of the v1.1.3 schema.
|
||||
|
||||
### New environment variables
|
||||
|
||||
- `PICLOUD_HTTP_ALLOW_PRIVATE` (default false; dev-only) — disable the
|
||||
SSRF deny-list.
|
||||
- `PICLOUD_HTTP_MAX_REQUEST_BODY_BYTES` / `PICLOUD_HTTP_MAX_RESPONSE_BODY_BYTES`
|
||||
(default 10 MB each).
|
||||
- `PICLOUD_CRON_TICK_INTERVAL_MS` (default 30000) — cron scheduler poll
|
||||
cadence (floored at 1s).
|
||||
|
||||
## v1.1.3 — Modules (unreleased)
|
||||
|
||||
Real per-app Rhai module system. Scripts can `import "<name>" as
|
||||
@@ -84,6 +328,21 @@ per-invocation compile cost; both invalidate on `updated_at` change.
|
||||
- **Route creation** — `POST /api/v1/admin/scripts/{id}/routes`
|
||||
returns 400 when the target script is `kind = 'module'`.
|
||||
|
||||
### Security fix
|
||||
|
||||
- **Cross-app trigger target (CVE-class: broken access control).** In
|
||||
v1.1.1 and v1.1.2, `POST /api/v1/admin/apps/{id}/triggers/{kv,docs,dead_letter}`
|
||||
validated only that the caller could manage triggers on `{id}` — it
|
||||
did **not** verify that the target `script_id` belonged to that same
|
||||
app. A member with trigger-management rights on app A could therefore
|
||||
register a trigger in A pointing at a script owned by app B, causing
|
||||
B's script to execute on A's events (a cross-app isolation break).
|
||||
v1.1.3 closes this: every trigger-create handler now loads the target
|
||||
script and rejects it unless `script.app_id == path app_id` (and it is
|
||||
not a module). **Upgrade recommendation:** anyone running a pre-v1.1.3
|
||||
multi-tenant deploy should upgrade and audit existing `triggers` rows
|
||||
for any whose `script_id` resolves to a script in a different `app_id`.
|
||||
|
||||
### Migrations
|
||||
|
||||
- `0015_scripts_kind.sql` — adds `scripts.kind` with CHECK
|
||||
|
||||
@@ -118,6 +118,8 @@ Environment variables consumed by the `picloud` binary:
|
||||
| `DATABASE_URL` | — | Required. Postgres connection string. |
|
||||
| `PICLOUD_SESSION_TTL_HOURS` | `24` | Sliding-window session lifetime. |
|
||||
| `PICLOUD_SANDBOX_MAX_*` | conservative defaults | Per-knob admin ceilings on Rhai sandbox overrides. See `manager-core::sandbox::SandboxCeiling`. |
|
||||
| `PICLOUD_FILES_ROOT` | `./data` | Filesystem root for `files::*` blob storage (v1.1.5). Bytes live at `<root>/files/<app_id>/<collection>/<id[0:2]>/<id>`; metadata in Postgres. |
|
||||
| `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` | `104857600` (100 MB) | Per-file hard size cap for `files::*` (v1.1.5). Per-app quotas deferred to v1.2. |
|
||||
|
||||
## Out of MVP
|
||||
|
||||
|
||||
145
Cargo.lock
generated
145
Cargo.lock
generated
@@ -378,6 +378,28 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "chrono-tz"
|
||||
version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "93698b29de5e97ad0ae26447b344c482a7284c737d9ddc5f9e52b74a336671bb"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"chrono-tz-build",
|
||||
"phf",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "chrono-tz-build"
|
||||
version = "0.3.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0c088aee841df9c3041febbb73934cfc39708749bf96dc827e3359cd39ef11b1"
|
||||
dependencies = [
|
||||
"parse-zoneinfo",
|
||||
"phf",
|
||||
"phf_codegen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "clap"
|
||||
version = "4.6.1"
|
||||
@@ -499,6 +521,17 @@ version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853"
|
||||
|
||||
[[package]]
|
||||
name = "cron"
|
||||
version = "0.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6f8c3e73077b4b4a6ab1ea5047c37c57aee77657bc8ecd6f29b0af082d0b0c07"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"nom",
|
||||
"once_cell",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "crossbeam-queue"
|
||||
version = "0.3.12"
|
||||
@@ -1326,6 +1359,12 @@ version = "0.3.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
|
||||
|
||||
[[package]]
|
||||
name = "minimal-lexical"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
|
||||
|
||||
[[package]]
|
||||
name = "mio"
|
||||
version = "1.2.0"
|
||||
@@ -1346,6 +1385,16 @@ dependencies = [
|
||||
"spin 0.5.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "nom"
|
||||
version = "7.1.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
"minimal-lexical",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "normalize-line-endings"
|
||||
version = "0.3.0"
|
||||
@@ -1463,6 +1512,15 @@ dependencies = [
|
||||
"windows-link",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "parse-zoneinfo"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1f2a05b18d44e2957b88f96ba460715e295bc1d7510468a2f3d3b44535d26c24"
|
||||
dependencies = [
|
||||
"regex",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "password-hash"
|
||||
version = "0.5.0"
|
||||
@@ -1512,9 +1570,47 @@ version = "2.3.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
|
||||
|
||||
[[package]]
|
||||
name = "phf"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1fd6780a80ae0c52cc120a26a1a42c1ae51b247a253e4e06113d23d2c2edd078"
|
||||
dependencies = [
|
||||
"phf_shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_codegen"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "aef8048c789fa5e851558d709946d6d79a8ff88c0440c587967f8e94bfb1216a"
|
||||
dependencies = [
|
||||
"phf_generator",
|
||||
"phf_shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_generator"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3c80231409c20246a13fddb31776fb942c38553c51e871f8cbd687a4cfb5843d"
|
||||
dependencies = [
|
||||
"phf_shared",
|
||||
"rand 0.8.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "phf_shared"
|
||||
version = "0.11.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "67eabc2ef2a60eb7faa00097bd1ffdb5bd28e62bf39990626a582201b7a754e5"
|
||||
dependencies = [
|
||||
"siphasher",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "picloud"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -1540,7 +1636,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-cli"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"assert_cmd",
|
||||
@@ -1561,7 +1657,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-executor"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"picloud-executor-core",
|
||||
@@ -1573,7 +1669,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-executor-core"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64",
|
||||
@@ -1590,12 +1686,14 @@ dependencies = [
|
||||
"thiserror 1.0.69",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"url",
|
||||
"uuid",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "picloud-manager"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"picloud-manager-core",
|
||||
@@ -1607,18 +1705,21 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-manager-core"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"argon2",
|
||||
"async-trait",
|
||||
"axum",
|
||||
"base64",
|
||||
"chrono",
|
||||
"chrono-tz",
|
||||
"cron",
|
||||
"data-encoding",
|
||||
"picloud-executor-core",
|
||||
"picloud-orchestrator-core",
|
||||
"picloud-shared",
|
||||
"rand 0.8.6",
|
||||
"reqwest",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
@@ -1632,7 +1733,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-orchestrator"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"picloud-orchestrator-core",
|
||||
@@ -1644,7 +1745,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-orchestrator-core"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"axum",
|
||||
@@ -1658,6 +1759,8 @@ dependencies = [
|
||||
"serde_json",
|
||||
"thiserror 1.0.69",
|
||||
"tokio",
|
||||
"tokio-stream",
|
||||
"tower",
|
||||
"tracing",
|
||||
"urlencoding",
|
||||
"uuid",
|
||||
@@ -1665,13 +1768,17 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "picloud-shared"
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
dependencies = [
|
||||
"async-trait",
|
||||
"base64",
|
||||
"chrono",
|
||||
"hmac",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
"thiserror 1.0.69",
|
||||
"tokio",
|
||||
"uuid",
|
||||
]
|
||||
|
||||
@@ -2368,6 +2475,12 @@ dependencies = [
|
||||
"rand_core 0.6.4",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "siphasher"
|
||||
version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649"
|
||||
|
||||
[[package]]
|
||||
name = "slab"
|
||||
version = "0.4.12"
|
||||
@@ -2883,6 +2996,20 @@ dependencies = [
|
||||
"futures-core",
|
||||
"pin-project-lite",
|
||||
"tokio",
|
||||
"tokio-util",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-util"
|
||||
version = "0.7.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"futures-sink",
|
||||
"pin-project-lite",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
14
Cargo.toml
14
Cargo.toml
@@ -13,7 +13,7 @@ members = [
|
||||
]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.1.3"
|
||||
version = "1.1.6"
|
||||
edition = "2021"
|
||||
rust-version = "1.92"
|
||||
license = "MIT OR Apache-2.0"
|
||||
@@ -29,6 +29,8 @@ picloud-manager-core = { path = "crates/manager-core" }
|
||||
|
||||
# Async + HTTP
|
||||
tokio = { version = "1.40", features = ["full"] }
|
||||
# Wraps a broadcast::Receiver into a Stream for the SSE endpoint (v1.1.6).
|
||||
tokio-stream = { version = "0.1", features = ["sync"] }
|
||||
axum = "0.8"
|
||||
tower = "0.5"
|
||||
tower-http = { version = "0.6", features = ["trace", "cors"] }
|
||||
@@ -47,12 +49,16 @@ tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
|
||||
# IDs + time
|
||||
uuid = { version = "1", features = ["v4", "serde"] }
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
# Cron schedule parsing (v1.1.4 cron triggers) + IANA timezone resolution.
|
||||
chrono-tz = "0.9"
|
||||
cron = "0.12"
|
||||
|
||||
# Async traits
|
||||
async-trait = "0.1"
|
||||
|
||||
# Rhai scripting
|
||||
rhai = { version = "1.19", features = ["sync", "serde"] }
|
||||
# Rhai scripting. Pinned exactly (`=1.24`) because the `internals`
|
||||
# feature surface is not semver-stable — future bumps must be deliberate.
|
||||
rhai = { version = "=1.24", features = ["sync", "serde"] }
|
||||
|
||||
# Postgres (manager-core only — others stay DB-free)
|
||||
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "uuid", "chrono", "json", "macros", "migrate"] }
|
||||
@@ -71,6 +77,8 @@ urlencoding = "2"
|
||||
argon2 = "0.5"
|
||||
rand = { version = "0.8", features = ["getrandom"] }
|
||||
sha2 = "0.10"
|
||||
# HMAC-SHA256 for realtime subscriber tokens (v1.1.6).
|
||||
hmac = "0.12"
|
||||
base64 = "0.22"
|
||||
data-encoding = "2.6"
|
||||
|
||||
|
||||
547
HANDBACK.md
547
HANDBACK.md
@@ -1,351 +1,220 @@
|
||||
# v1.1.3 — Modules — Handback
|
||||
# HANDBACK — v1.1.6 Realtime Channels & Client Library
|
||||
|
||||
## 1. Branch summary
|
||||
Branch: `feat/v1.1.6-realtime-client` (from `main`). Not pushed, no PR.
|
||||
|
||||
- **Branch:** `feat/v1.1.3-modules`
|
||||
- **Commits ahead of `main`:** 6
|
||||
- **HEAD:** `3dbead4`
|
||||
- **Not pushed, not merged, no PR opened** (per brief).
|
||||
## 1. Scope coverage (§1–§13)
|
||||
|
||||
Commits (newest first):
|
||||
| § | Item | Status |
|
||||
|---|------|--------|
|
||||
| 1 | `topics` table (`0021_topics.sql`) | ✅ Done |
|
||||
| 2 | Topic admin endpoints (POST/GET/PATCH/DELETE), `AppTopicManage` | ✅ Done |
|
||||
| 3 | SSE endpoint `GET /realtime/topics/{topic}` | ✅ Done |
|
||||
| 4 | In-process `RealtimeBroadcaster` + GC + publish wiring | ✅ Done |
|
||||
| 5 | HMAC subscriber tokens + `app_secrets` (`0022`) + `pubsub::subscriber_token` | ✅ Done |
|
||||
| 6 | Dashboard Topics tab | ✅ Done |
|
||||
| 7 | `@picloud/client` TypeScript package | ✅ Done |
|
||||
| 8 | Topic-aware publish → realtime wiring | ✅ Done |
|
||||
| 9 | Dispatcher e2e tests (six) | ✅ Done (location deviation — see §7) |
|
||||
| 10 | Empty-blob decision | ✅ Done — **relaxed** (accept empty) |
|
||||
| 11 | Orphan `*.tmp.*` sweeper | ✅ Done |
|
||||
| 12 | Version bumps (1.1.6 / SDK 1.7 / dash 0.12.0 / client 1.0.0) | ✅ Done |
|
||||
| 13 | Tests (all named-critical cases) | ✅ Done |
|
||||
|
||||
## 2. Realtime implementation notes
|
||||
|
||||
### Topic resolution / SSE handshake sequence
|
||||
1. Extract `Host` → `app_domains.resolve_app(host)` (existing two-phase
|
||||
dispatch). No app → **404**.
|
||||
2. Token from `Authorization: Bearer <t>` **or** `?token=<t>` (EventSource
|
||||
can't set headers).
|
||||
3. `RealtimeAuthority::authorize_subscribe(app_id, topic, token)`:
|
||||
- topic missing OR `external_subscribable = false` → `NotFound` → **404**
|
||||
(both collapse to 404 so the endpoint can't probe internal topics);
|
||||
- `auth_mode='public'` → allow;
|
||||
- `auth_mode='token'` → verify HMAC (present, signed by this app's key,
|
||||
unexpired, scoped to this topic) → allow, else `Unauthorized` → **401**
|
||||
(generic; never says which check failed).
|
||||
4. `broadcaster.subscribe(app_id, topic)` → `broadcast::Receiver`; stream
|
||||
`data: {topic,message,published_at}\n\n` with `:heartbeat` keepalive
|
||||
every `PICLOUD_REALTIME_HEARTBEAT_SEC` (default 30). Headers:
|
||||
`text/event-stream`, `Cache-Control: no-cache` (set by axum Sse),
|
||||
`X-Accel-Buffering: no` (added). Client disconnect drops the receiver →
|
||||
automatic cleanup; the periodic GC reaps empty channels.
|
||||
|
||||
The SSE handler lives in **orchestrator-core** (`realtime_api.rs`) and
|
||||
depends only on the three picloud-shared traits — all DB + signing-key
|
||||
access stays in the manager-core `RealtimeAuthority` impl, so the
|
||||
data-plane crate never touches the key. Cluster mode (v1.3+) swaps the
|
||||
broadcaster + authority impls behind the same traits.
|
||||
|
||||
### HMAC signing-key storage — chose **table** (`app_secrets`)
|
||||
Per the asked decision. 32 random bytes per app, lazily created on the
|
||||
first `pubsub::subscriber_token` call (`ON CONFLICT DO NOTHING` for
|
||||
concurrency). No global `PICLOUD_INSTANCE_SECRET`; the table is the
|
||||
natural home for v1.1.7's encrypted per-app secrets.
|
||||
**Tension flagged:** §5 aspired to "validate the signature without a DB
|
||||
lookup". The table approach needs a key read on the token-subscribe path.
|
||||
Mitigated by an in-memory key cache in `RealtimeAuthorityImpl` (keys never
|
||||
rotate in v1.1.6, so the cache needs no invalidation); the subscribe path
|
||||
already reads the `topics` row, so this adds no new round-trip *category*.
|
||||
|
||||
### In-process broadcaster
|
||||
`Mutex<HashMap<(AppId, String), broadcast::Sender<RealtimeEvent>>>`.
|
||||
Capacity per channel `PICLOUD_REALTIME_BROADCAST_CAPACITY` (default 64);
|
||||
slow consumers lose oldest events (`broadcast` lag semantics — best-effort,
|
||||
no replay). `subscribe` creates channels lazily; `publish` is a silent
|
||||
no-op when no channel exists; `drop_topic` removes the sender (existing
|
||||
receivers observe a closed channel and disconnect). A `spawn_realtime_gc`
|
||||
task (~60s) drops senders with `receiver_count() == 0`.
|
||||
|
||||
### Publish wiring (order + failure modes) — §8 ordering chosen
|
||||
`PubsubServiceImpl::publish_durable`: validate/authz → **transactional
|
||||
outbox fan-out + commit** (existing) → **then** best-effort
|
||||
`RealtimeBroadcaster::publish`. The broadcast runs on a child
|
||||
`tokio::spawn` whose `JoinHandle` is awaited, so a panicking broadcaster
|
||||
becomes a `warn` log, never a failed publish (the durable deliveries
|
||||
already committed). One `published_at` instant is shared by both paths.
|
||||
**Brief-internal contradiction flagged** — see §9.
|
||||
|
||||
## 3. Client lib implementation notes
|
||||
|
||||
- **Build:** **tsup** (dual ESM+CJS + `.d.ts`), tests **vitest**, lint =
|
||||
`tsc --noEmit` (strict; `noUncheckedIndexedAccess`, no `any` in exports).
|
||||
- **Layout:** `src/{index,client,endpoint,subscribe,auth,types}.ts` +
|
||||
`src/react/index.ts` + `src/svelte/index.ts`; subpath exports
|
||||
`@picloud/client/react` and `@picloud/client/svelte` via package `exports`.
|
||||
- **Reconnect:** SSE is implemented over **streaming `fetch`** (not native
|
||||
`EventSource`) so the lib can (a) detect a **401** on (re)connect and call
|
||||
`onTokenExpired` to refresh, (b) send **`Last-Event-ID`** on resume
|
||||
(server ignores it in v1.1.6 — client ships ready), and (c) apply its own
|
||||
**exponential backoff** (base 1s → ×2 → cap 30s; reset on successful
|
||||
open). Token rides in `?token=` (EventSource-parity). React Native caveat
|
||||
documented in the README (supply a streaming-`fetch` polyfill).
|
||||
- **zod/valibot adapter:** the `Validator<T> = { parse(input): T }` shape.
|
||||
A Zod schema satisfies it directly; Valibot wraps in one line. Used by
|
||||
`endpoint(...).get({ validate })` and `subscribe(..., { validate })`. No
|
||||
hard dep.
|
||||
|
||||
## 4. v1.1.5 follow-ups
|
||||
|
||||
- **Dispatcher e2e (six):** `dispatcher_delivers_{kv,docs,cron,files,pubsub,
|
||||
dead_letter}_to_handler` in `crates/picloud/tests/dispatcher_e2e.rs`.
|
||||
Verified green against a real Postgres (all 6).
|
||||
- **Empty-blob — RELAXED** (per the asked decision). Dropped the
|
||||
`data.is_empty()` rejection in `NewFile::validate` **and** `FileUpdate::
|
||||
validate` (the latter for consistency — flagged in §7). Flipped the
|
||||
pinning test in `files_service.rs` and added `empty_file_round_trips`.
|
||||
- **Orphan sweep:** `spawn_files_orphan_sweep` (`files_sweep.rs`), every 6h
|
||||
(`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC`), unlinks `*.tmp.*` older than
|
||||
1h (`PICLOUD_FILES_ORPHAN_TMP_TTL_SEC`); logs dirs-walked / files-deleted /
|
||||
bytes-reclaimed. No DB cross-check (v1.3+). Tested: deletes old tmp, keeps
|
||||
young tmp, keeps non-tmp, missing-root no panic.
|
||||
|
||||
## 5. Schema decisions beyond the brief
|
||||
None — `0021_topics.sql` and `0022_app_secrets.sql` match the brief's DDL
|
||||
verbatim. Schema-snapshot golden re-blessed on a fresh DB (only the two new
|
||||
tables added; PK/FK `ON DELETE CASCADE` + the `auth_mode` CHECK present).
|
||||
|
||||
## 6. How to verify locally
|
||||
See §8 below.
|
||||
|
||||
## 7. Decisions beyond the brief — every prompt-default deviation
|
||||
1. **Dispatcher e2e location:** the brief says
|
||||
`crates/manager-core/tests/dispatcher_e2e.rs`; I put it in
|
||||
**`crates/picloud/tests/`**. `build_app` (the full dispatcher + scheduler
|
||||
+ executor wiring) lives in the `picloud` crate; a manager-core test would
|
||||
need a manager→picloud dev-dependency cycle or a hand-rolled re-wire of
|
||||
build_app. The picloud harness (`server_with_app` pattern) is proven and
|
||||
the tests run green against a real DB. Gating uses the **schema_snapshot
|
||||
pattern** (env check + early return), NOT `#[ignore]`, so CI's plain
|
||||
`cargo test` with `DATABASE_URL` runs them while local stays green — as
|
||||
the brief requested.
|
||||
2. **Empty-blob relaxation extended to `FileUpdate::validate`** — the brief
|
||||
names only `NewFile::validate`. Relaxing create-empty but rejecting
|
||||
update-to-empty would be an inconsistent API, so I relaxed both.
|
||||
3. **Publish order: §8 (broadcast AFTER outbox commit)**, not §4 (broadcast
|
||||
FIRST). The two sections contradict; §8 is the dedicated, numbered,
|
||||
rationale-backed section. See §9.
|
||||
4. **Topic-name validation** — `topics_api` rejects empty names and names
|
||||
containing `*` (external pattern subscription is v2). The brief didn't
|
||||
specify name validation; this is a small guard.
|
||||
5. **Client lib lint = `tsc --noEmit`** (not eslint) to keep devDeps lean;
|
||||
strict typecheck is the gate. "No `any` in exports" is enforced by review
|
||||
+ strict TS, not an eslint rule.
|
||||
6. **Cron e2e poll budget = 45s** — the cron scheduler skips its first tick
|
||||
then ticks every 30s (default). The test polls 45s so it passes at the
|
||||
default; set `PICLOUD_CRON_TICK_INTERVAL_MS=1000` to make it ~2s in CI.
|
||||
|
||||
## 8. Attestation (gate runs on this HEAD)
|
||||
|
||||
```
|
||||
3dbead4 test(v1.1.3-modules): resolver, cache, validator, kind-rejection coverage
|
||||
10f76d2 chore(v1.1.3-modules): version bumps + CHANGELOG + blueprint touch-up
|
||||
610fd4f feat(v1.1.3-modules): dashboard kind dropdown + scripts-list and detail badges
|
||||
66b41bb feat(v1.1.3-modules): top-level script AST cache in LocalExecutorClient
|
||||
c6211a7 feat(v1.1.3-modules): reject module scripts from routes + triggers; tighten cross-app trigger check
|
||||
84833d3 feat(v1.1.3-modules): shared types, migrations, engine + resolver scaffold
|
||||
cargo fmt --all -- --check → clean
|
||||
cargo clippy --all-targets --all-features -- -D warnings → clean (exit 0)
|
||||
cargo test --workspace → 482 passed, 0 failed
|
||||
(DB-gated dispatcher_e2e
|
||||
auto-skip as no-op when
|
||||
DATABASE_URL unset)
|
||||
# DB-gated (real Postgres @ 127.0.0.1:15432, picloud/picloud):
|
||||
DATABASE_URL=… PICLOUD_CRON_TICK_INTERVAL_MS=1000 \
|
||||
cargo test -p picloud --test dispatcher_e2e → 6 passed
|
||||
DATABASE_URL=… cargo test -p picloud-manager-core --test schema_snapshot → 1 passed
|
||||
(cd dashboard && npm run check) → 0 errors, 0 warnings (371 files)
|
||||
(cd clients/typescript && npm run lint) → clean (tsc --noEmit)
|
||||
(cd clients/typescript && npm run test) → 15 passed (5 files)
|
||||
(cd clients/typescript && npm run build) → tsup ESM+CJS+.d.ts OK
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope coverage
|
||||
|
||||
| # | Brief item | Status | Notes |
|
||||
|---|---|---|---|
|
||||
| 1 | `scripts.kind` column + check + index | **Done** | `migrations/0015_scripts_kind.sql` |
|
||||
| 2 | Module syntax constraints (fn / const / import only) | **Done** | Walks `ast.statements()` via `rhai/internals`. Admin endpoint is primary gate; resolver re-runs the check for defense-in-depth. |
|
||||
| 3 | `ModuleResolver` replaces `DummyModuleResolver` | **Done** | `crates/executor-core/src/module_resolver.rs`; per-call instance with cross-app isolation, cycle detect, depth limit. |
|
||||
| 4 | Two AST caches (script + module) | **Done** | Script cache in `LocalExecutorClient`; module cache in `Engine`. Both invalidate by `updated_at` comparison. Env-overridable sizes. |
|
||||
| 5 | Dep-graph table + populate | **Done** | `migrations/0016_script_imports.sql`; `replace_imports_tx` writes edges in the same transaction as the script INSERT/UPDATE. |
|
||||
| 6 | Admin endpoint changes (kind, kind-change rejection, route/trigger module rejection) | **Done** | Also closes a latent cross-app trigger gap (script.app_id mismatch — see §7). |
|
||||
| 7 | Dashboard surface (kind dropdown + badge) | **Done** | App page form + scripts list + script detail header. `npm run check` clean. |
|
||||
| 8 | `ModuleSource` trait shape | **Done** | Lives in `picloud-shared`; matches the v1.1.1/v1.1.2 service pattern. |
|
||||
| 9 | Version bumps | **Done** | Workspace 1.1.2→1.1.3, SDK 1.3→1.4, dashboard 0.8.0→0.9.0. |
|
||||
| 10 | Tests (~40–60) | **Done** | 46 new tests across 5 crates. Gates green. |
|
||||
|
||||
### Scope-out items (confirmed NOT built)
|
||||
|
||||
- No module versioning / pinning, no `@v3` syntax.
|
||||
- No eager precompilation at save-time.
|
||||
- No dashboard dep-graph visualization.
|
||||
- No LISTEN/NOTIFY-based cross-node invalidation.
|
||||
- No new `Scope` variants (modules use existing `script:read` / `script:write`).
|
||||
- No admin GET endpoints for `script_imports` (the table is persisted for v1.2+; no v1.1.3 read surface — see §10 deferred items).
|
||||
|
||||
---
|
||||
|
||||
## 3. Resolver implementation notes
|
||||
|
||||
### 3.1 In-progress-imports stack
|
||||
|
||||
Lives **on the per-call `PicloudModuleResolver` instance**, not on `SdkCallCx`. The resolver is constructed fresh per `Engine::execute_ast` call (see `crates/executor-core/src/engine.rs:execute_ast`), so the stack is naturally scoped to one execution. Both the stack and the depth counter are `Mutex<…>` (not `RefCell<…>`) because `rhai::ModuleResolver: SendSync` under the `sync` feature.
|
||||
|
||||
An RAII `StackGuard` pops the stack and decrements depth on drop — a compile error or panic anywhere inside `resolve()` cleans up properly. The lock is uncontended in practice (Rhai evaluation on the engine is single-threaded).
|
||||
|
||||
### 3.2 Sync → async bridge
|
||||
|
||||
Rhai's `ModuleResolver::resolve` is sync; `ModuleSource::lookup` is async. The bridge:
|
||||
|
||||
```rust
|
||||
let handle = tokio::runtime::Handle::try_current().map_err(/* surfaces as ErrorRuntime */)?;
|
||||
let lookup = tokio::task::block_in_place(|| handle.block_on(self.source.lookup(&self.cx, path)));
|
||||
```
|
||||
|
||||
- `try_current()` (not `current()`) so test harnesses that build an `Engine` outside a Tokio runtime get a clean error instead of a panic.
|
||||
- `block_in_place` makes the call safe both on `spawn_blocking` threads (where it's a no-op) and on multi-threaded runtime worker tasks (where it instructs the runtime to relocate other tasks before we block). This was load-bearing for the resolver tests, which call `engine.execute` directly from `#[tokio::test(flavor = "multi_thread")]`.
|
||||
- A `current_thread` runtime still panics — but production callers wrap `Engine::execute` in `tokio::task::spawn_blocking` (see `LocalExecutorClient::execute_with_identity`), which avoids that path entirely.
|
||||
|
||||
### 3.3 Cross-app isolation enforcement
|
||||
|
||||
The resolver captures `Arc<SdkCallCx>` at construction. Every `ModuleSource::lookup` call passes `&self.cx`. The Postgres impl (`crates/manager-core/src/module_source.rs`) selects with `WHERE app_id = $1 AND kind = 'module' AND name = $2`, binding `$1` from `cx.app_id.into_inner()` — never from any script-passed argument. The Rhai script's `import "name" as alias;` syntax has no slot for an `app_id`, so there is no path by which a script in app A can name a row in app B.
|
||||
|
||||
Verified by `resolver_cross_app_blocked` and `resolver_cross_app_module_not_found` tests.
|
||||
|
||||
### 3.4 Module-shape validation — both layers
|
||||
|
||||
- **Primary gate (admin endpoint)** — `manager-core::api::create_script` and `update_script` call `state.validator.validate_module(src)` whenever the effective kind is `Module`. `Engine`'s impl walks `ast.statements()`, accepting only `Stmt::Var(_, ASTFlags::CONSTANT, _)`, `Stmt::Import(..)`, and `Stmt::Noop(..)`. Anything else (top-level expression, let, if, while, …) is rejected with a clear `ValidationError::ModuleShape` message.
|
||||
- **Defense in depth (resolver)** — the resolver calls `check_module_shape` again after `engine.compile(source)`. This catches rows that bypassed the API (manual SQL inserts, future migration bugs, restoring from an older backup).
|
||||
|
||||
Note: Rhai's default optimizer constant-folds `if true { ... }` away, so a module containing `if true { ... }` parses to an empty body and passes vacuously. This is fine semantically (the script has no observable behavior), but it surprises authors. Documented as a known acceptance edge; not worth disabling optimization for.
|
||||
|
||||
### 3.5 What the resolver does NOT enforce
|
||||
|
||||
- **Module access permissions** — every module in an app is importable by every other script in the same app. Per-module ACLs are explicitly v1.2+.
|
||||
- **Module versioning / pinning** — there's exactly one current version per `(app_id, name)`. v1.3+.
|
||||
|
||||
---
|
||||
|
||||
## 4. Cache design notes
|
||||
|
||||
### 4.1 LRU library
|
||||
|
||||
**`lru = "0.12"`** — added to `[workspace.dependencies]`. Standard choice, no-frills crate (`LruCache<K, V>` with `put`/`get`/`len`/etc.). Both caches use `Arc<Mutex<LruCache<K, V>>>` so they're cheap to clone and safe to share across executions.
|
||||
|
||||
### 4.2 Cache key shapes + what's stored
|
||||
|
||||
| Cache | Owner | Key | Value | Stores |
|
||||
|---|---|---|---|---|
|
||||
| **Script AST cache** | `LocalExecutorClient` | `ScriptId` | `CachedScript { updated_at: DateTime<Utc>, ast: Arc<rhai::AST> }` | Compiled AST for the top-level (endpoint) script. |
|
||||
| **Module cache** | `Engine` | `(AppId, String)` | `CachedModule { updated_at: DateTime<Utc>, module: Shared<rhai::Module> }` | Compiled `rhai::Module` produced by `Module::eval_ast_as_new`. |
|
||||
|
||||
The script cache stores `Arc<AST>` so an evaluation can grab a cheap clone and hand it to `Engine::execute_ast` without holding the cache lock. The module cache stores `Shared<Module>` (= `Arc<Module>` under the `sync` feature) because that's what `ModuleResolver::resolve` must return.
|
||||
|
||||
### 4.3 Stale-version detection
|
||||
|
||||
Both caches use the same logic: **compare `cached.updated_at` against the freshly-known `updated_at`**.
|
||||
|
||||
- For the script cache, the caller passes the fresh value as `ScriptIdentity.updated_at` — the orchestrator already loaded the script row to dispatch the request, so there's no extra DB hit.
|
||||
- For the module cache, the resolver must call `ModuleSource::lookup` first to learn the fresh `updated_at` — every `import` does at least one DB roundtrip. That's a deliberate trade-off (documented in CHANGELOG): the alternative (TTL caching or pub/sub) introduces staleness during edits and complicates "publish a fix immediately" UX. Worth re-evaluating in v1.3+ when LISTEN/NOTIFY makes pub/sub cheap.
|
||||
|
||||
Mismatch → recompile + `cache.put(...)` replace. LRU eviction is automatic when capacity is exceeded.
|
||||
|
||||
### 4.4 Capacity overrides
|
||||
|
||||
- `PICLOUD_SCRIPT_CACHE_SIZE` (default 256, `LocalExecutorClient`)
|
||||
- `PICLOUD_MODULE_CACHE_SIZE` (default 512, `Engine`)
|
||||
|
||||
Both clamp `max(1)` to avoid the LRU constructor's panic on zero. `Engine::with_module_cache_capacity` and `LocalExecutorClient::with_script_cache_capacity` give tests explicit handles.
|
||||
|
||||
---
|
||||
|
||||
## 5. Dep-graph population
|
||||
|
||||
### 5.1 Where the extraction happens
|
||||
|
||||
Inside the `ScriptValidator` impl on `Engine`. The trait now returns `ValidatedScript { imports: Vec<String> }`, populated by `extract_imports` (endpoint scripts) or `validate_module_source` (module scripts). Both walk `ast.statements()` and pull out `Stmt::Import(boxed_path_expr, _)` where the path is a `StringConstant`.
|
||||
|
||||
**Dynamic imports** (`import some_var as alias;`) are NOT captured because we can't know the name at compile time. Tested by `validate_endpoint_skips_dynamic_imports_in_imports_list`. Documented as a known limitation in the CHANGELOG and migration 0016's header comment.
|
||||
|
||||
### 5.2 Where the write happens — transactional with the script INSERT/UPDATE
|
||||
|
||||
`PostgresScriptRepository::create` and `update` both open a `tx = pool.begin().await?`. The script row is inserted/updated inside the tx; immediately after, `replace_imports_tx(&mut tx, importer, app_id, &imports)` runs. The tx is committed at the end. If any step fails, both the script change and the dep-graph mutation roll back together. No half-state where the script row exists but the edges don't (or vice versa).
|
||||
|
||||
`replace_imports_tx`:
|
||||
|
||||
1. `DELETE FROM script_imports WHERE importer_script_id = $1` — replaces wholesale.
|
||||
2. `INSERT INTO script_imports ... SELECT ... FROM scripts WHERE app_id = $1 AND kind = 'module' AND name = ANY($3) ON CONFLICT DO NOTHING` — best-effort: only resolves to existing modules in the same app; unresolved names are silently skipped (no error). A later save of either script re-resolves and writes the edge.
|
||||
|
||||
### 5.3 Schema decisions
|
||||
|
||||
- `script_imports.app_id` is denormalized but useful: the "all imports in app X" scan happens once at boot for caching and (eventually) for the dashboard's audit view. Without it, that query would need a 3-way join.
|
||||
- `created_at` is unused by v1.1.3 logic but trivial to add now and useful for v1.2+ "first imported" diagnostics.
|
||||
- The FK on `imported_script_id` cascades — when a module is deleted, every edge referencing it goes too. The cascade isn't exercised by a unit test (it would need Postgres); it's covered by the FK design.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tests added
|
||||
|
||||
46 new tests across 5 crates. All green on HEAD `3dbead4`. Inventory:
|
||||
|
||||
### `crates/executor-core/tests/modules.rs` (NEW — 23 tests)
|
||||
|
||||
End-to-end through `Engine::execute` with a `CountingModuleSource` (in-memory fake).
|
||||
|
||||
| # | Test | Covers |
|
||||
|---|---|---|
|
||||
| 1 | `resolver_loads_simple_module` | Happy path: `import "m" as m; m::add(2, 3)` → 5. |
|
||||
| 2 | `resolver_cross_app_blocked` | Modules with same name in two apps resolve to the calling app's version. |
|
||||
| 3 | `resolver_cross_app_module_not_found` | App B's `import "lonely"` returns ModuleNotFound when only app A has it. |
|
||||
| 4 | `resolver_module_not_found` | Missing module → `ErrorModuleNotFound`. |
|
||||
| 5 | `resolver_self_import_detected` | `a` imports `a` → circular error. |
|
||||
| 6 | `resolver_circular_detected` | `a → b → a` → circular error. |
|
||||
| 7 | `resolver_depth_limit_enforced` | 9-deep chain with limit 8 → depth error. |
|
||||
| 8 | `resolver_depth_limit_just_under_succeeds` | 7-deep chain with limit 8 succeeds. |
|
||||
| 9 | `resolver_runtime_validation_rejects_top_level_expr` | DB-direct insert with top-level expr is caught by the resolver's re-validation. |
|
||||
| 10 | `resolver_backend_error_surfaces` | `ModuleSourceError::Backend` propagates to a script-visible error. |
|
||||
| 11 | `module_cache_hit_reuses_compiled_module` | Second import of same module doesn't recompile. |
|
||||
| 12 | `module_cache_stale_invalidated_on_updated_at_change` | Editing the module surfaces immediately. |
|
||||
| 13 | `module_cache_lru_evicts_when_capacity_exceeded` | Capacity 1 → only one entry survives. |
|
||||
| 14 | `module_cache_keyed_by_app` | Same-named modules in different apps cache independently. |
|
||||
| 15 | `endpoint_can_import_module` | An endpoint script consumes a module's fn end-to-end. |
|
||||
| 16 | `module_can_import_module` | Modules can be importers. |
|
||||
| 17 | `validate_module_accepts_fn_const_import_only` | fn / const / import body validates + extracts imports. |
|
||||
| 18 | `validate_module_rejects_top_level_let` | `let x = 1;` rejected. |
|
||||
| 19 | `validate_module_rejects_top_level_expr` | `42;` rejected. |
|
||||
| 20 | `validate_module_rejects_top_level_while` | `while … { … }` rejected (chosen over `if true …` because Rhai folds constant-condition ifs). |
|
||||
| 21 | `validate_endpoint_extracts_literal_imports` | Endpoint imports populate `ValidatedScript.imports`. |
|
||||
| 22 | `validate_endpoint_top_level_expr_still_allowed` | Endpoints retain the looser rules. |
|
||||
| 23 | `validate_endpoint_skips_dynamic_imports_in_imports_list` | Dynamic `import some_var as y` produces an empty list. |
|
||||
|
||||
### `crates/orchestrator-core/src/client.rs` (6 inline tests)
|
||||
|
||||
| # | Test | Covers |
|
||||
|---|---|---|
|
||||
| 1 | `cache_hit_when_identity_matches` | Identical `(script_id, updated_at)` returns the same `Arc<AST>`. |
|
||||
| 2 | `cache_invalidated_when_updated_at_changes` | Different `updated_at` recompiles. |
|
||||
| 3 | `distinct_script_ids_cache_independently` | Two scripts → two entries. |
|
||||
| 4 | `lru_eviction_caps_cache_size` | Capacity 1; A → B → C leaves one entry. |
|
||||
| 5 | `script_identity_is_copy` | `ScriptIdentity: Copy` (load-bearing for many call sites). |
|
||||
| 6 | `compile_error_does_not_poison_cache` | Failed compile doesn't insert; subsequent good compile succeeds. |
|
||||
|
||||
### `crates/shared/src/script.rs` (3 inline tests)
|
||||
|
||||
| # | Test | Covers |
|
||||
|---|---|---|
|
||||
| 1 | `default_is_endpoint` | `ScriptKind::default() == Endpoint`. |
|
||||
| 2 | `round_trips_through_serde_lowercase` | `"endpoint"` / `"module"` wire form. |
|
||||
| 3 | `parse_str_round_trip` | `as_str` ↔ `parse_str` inverses. |
|
||||
|
||||
### `crates/manager-core/src/triggers_api.rs` (6 new inline tests)
|
||||
|
||||
| # | Test | Covers |
|
||||
|---|---|---|
|
||||
| 1 | `kv_trigger_rejects_module_target` | Module script as KV-trigger target → 422 with `"module"` in the message. |
|
||||
| 2 | `docs_trigger_rejects_module_target` | Same for docs triggers. |
|
||||
| 3 | `dl_trigger_rejects_module_target` | Same for dead-letter triggers. |
|
||||
| 4 | `kv_trigger_rejects_missing_script` | Non-existent script id → 422. |
|
||||
| 5 | `kv_trigger_rejects_cross_app_script` | Latent v1.1.1/v1.1.2 isolation gap — script in app B targeted from app A → 422. |
|
||||
| 6 | `kv_trigger_accepts_endpoint_target` | Happy path. |
|
||||
|
||||
### `crates/picloud/tests/api.rs` (8 `#[ignore]`'d Postgres-gated tests)
|
||||
|
||||
End-to-end through the HTTP surface. Run with `--include-ignored` against a real Postgres.
|
||||
|
||||
| # | Test | Covers |
|
||||
|---|---|---|
|
||||
| 1 | `create_script_default_kind_is_endpoint` | Default kind on create. |
|
||||
| 2 | `create_module_kind_persists` | `kind=module` round-trips through the API. |
|
||||
| 3 | `create_module_with_top_level_expr_rejected` | Module syntax gate at create time. |
|
||||
| 4 | `create_module_with_reserved_name_rejected` | `kv`, `docs`, etc. reserved. |
|
||||
| 5 | `route_bind_rejects_module` | `POST .../routes` returns 422 for module targets. |
|
||||
| 6 | `endpoint_imports_module_end_to_end` | Endpoint imports module, route binding, HTTP invocation, result. |
|
||||
| 7 | `module_edit_visible_on_next_invocation` | Cache invalidation on module edit (verified end-to-end through the engine). |
|
||||
| 8 | `cross_app_import_blocked` | Two apps, same-name module, endpoint sees its own. |
|
||||
|
||||
---
|
||||
|
||||
## 7. Schema / decisions beyond the brief
|
||||
|
||||
- **Module name shape CHECK** (`migrations/0015_scripts_kind.sql`): module names are constrained to `^[a-zA-Z_][a-zA-Z0-9_]{0,63}$`. Endpoint scripts retain the looser pre-v1.1.3 name rules so existing rows aren't invalidated. Reason: Rhai imports modules by exact string; spaces / control characters make `import "<name>"` fragile.
|
||||
- **Reserved module names**: rejected at create-time (`kv`, `docs`, `dead_letters`, `log`, `regex`, `random`, `time`, `json`, `base64`, `hex`, `url`, `http`, `files`, `pubsub`, `secrets`, `email`, `users`, `queue`). Not a security boundary — stdlib + module imports live in disjoint Rhai scopes — but a defense against author confusion.
|
||||
- **`ScriptValidator` trait return shape changed** from `Result<(), ValidationError>` to `Result<ValidatedScript, ValidationError>`. Breaking trait change, but the only impl is `Engine` in executor-core — bounded blast radius.
|
||||
- **`ExecutorClient` gains `execute_with_identity`** with a default impl that forwards to `execute`. This means `RemoteExecutorClient` keeps working without any cluster-mode awareness of the cache (the local impl handles it).
|
||||
- **Latent security fix**: trigger creation now verifies `script.app_id == app_id`. v1.1.1 and v1.1.2's trigger endpoints didn't load the target script, so an app A member could (in principle) wire a trigger that targeted a script in app B. Closed in this release; called out in the CHANGELOG.
|
||||
|
||||
---
|
||||
|
||||
## 8. How to verify locally (verified on HEAD `3dbead4`)
|
||||
|
||||
After my last commit, I ran the three gates plus the dashboard check on the exact HEAD I'm handing back. **Actual** exit codes and counts (not pre-written):
|
||||
|
||||
### 8.1 `cargo fmt --all -- --check`
|
||||
|
||||
```
|
||||
$ cargo fmt --all -- --check
|
||||
$ echo $?
|
||||
0
|
||||
```
|
||||
|
||||
Clean diff, **exit 0**.
|
||||
|
||||
### 8.2 `cargo clippy --all-targets --all-features -- -D warnings`
|
||||
|
||||
```
|
||||
$ cargo clippy --all-targets --all-features -- -D warnings
|
||||
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.21s
|
||||
$ echo $?
|
||||
0
|
||||
```
|
||||
|
||||
No warnings, **exit 0**.
|
||||
|
||||
### 8.3 `cargo test --workspace`
|
||||
|
||||
```
|
||||
$ cargo test --workspace
|
||||
... (per-suite results) ...
|
||||
$ echo $?
|
||||
0
|
||||
```
|
||||
|
||||
Aggregate (summed across all `test result:` lines):
|
||||
|
||||
- **PASSED = 358**
|
||||
- **FAILED = 0**
|
||||
- **IGNORED = 140** (Postgres-gated `#[ignore]` integration tests in `picloud/tests/api.rs` + 1 schema_snapshot test; need `DATABASE_URL` to run)
|
||||
- **measured = 0**
|
||||
- **filtered out = 0**
|
||||
|
||||
### 8.4 `(cd dashboard && npm run check)`
|
||||
|
||||
```
|
||||
$ cd dashboard && npm run check
|
||||
> picloud-dashboard@0.9.0 check
|
||||
> svelte-kit sync && svelte-check --tsconfig ./tsconfig.json
|
||||
|
||||
1780463972778 START "/home/fabi/PiCloud/dashboard"
|
||||
1780463972779 COMPLETED 369 FILES 0 ERRORS 0 WARNINGS 0 FILES_WITH_PROBLEMS
|
||||
$ echo $?
|
||||
0
|
||||
```
|
||||
|
||||
0 errors, 0 warnings, **exit 0**.
|
||||
|
||||
### 8.5 Migrations apply
|
||||
|
||||
Verified during normal `cargo test --workspace` runs — `sqlx::test` macros apply migrations 0001 through 0016 cleanly on a freshly created database for every `#[ignore]`d integration test. The from-v1.1.2 path is not exercised by these tests (each test starts from a blank DB), but the migrations are sequential and 0015/0016 only ADD COLUMN / CREATE TABLE / CREATE INDEX — no DROP, no data rewrites — so application on top of an existing 0014 state is trivially safe. The downgrade caveat is documented in the CHANGELOG.
|
||||
|
||||
### 8.6 Manual smoke
|
||||
|
||||
I did **not** run the full end-to-end manual smoke against a live Postgres + Caddy stack as the brief's "Done looks like" specifies. The 8 ignored `picloud/tests/api.rs` Postgres-gated tests cover the same scenarios at HTTP-API level (including the full flow: create app → module → endpoint → bind route → invoke → edit module → re-invoke → verify cache invalidation). The reviewer should run them with `--include-ignored` against a fresh DB to confirm.
|
||||
|
||||
---
|
||||
Migrations verified applying cleanly on a fresh DB **and** on top of the
|
||||
existing v1.1.5 dev DB (0020 → 0021 → 0022). Schema-snapshot golden diff is
|
||||
exactly the two new tables.
|
||||
|
||||
## 9. Open questions for the reviewer
|
||||
1. **§4 vs §8 ordering contradiction (load-bearing, flagged not
|
||||
reinterpreted):** §4 says "realtime broadcast FIRST, then transactional
|
||||
outbox fan-out"; §8 says broadcast AFTER the fan-out commits (numbered
|
||||
steps 1–4). I implemented **§8** (dedicated section + failure-mode
|
||||
rationale). If §4's ordering was intended, the broadcast should move
|
||||
before `fan_out_publish` — but then a publish whose outbox write fails
|
||||
would still have notified SSE subscribers of an event that never durably
|
||||
happened, which seems wrong. Please confirm §8 is correct.
|
||||
2. **Dead-letter → handler fan-out appears unwired (see §10).** Confirm
|
||||
whether the `dead_letter` trigger source is intended to fire in v1.1.x.
|
||||
|
||||
1. **Optimizer constant-folding edge.** Module bodies containing only `if true { ... }` (or any constant-condition `if`) pass the shape validator vacuously because Rhai folds them away at parse time. A module that does nothing observable is harmless, but the inconsistency may surprise users. Options:
|
||||
- Accept as-is (current state); document.
|
||||
- Disable `rhai`'s optimizer in the parse-only validate path (`Engine::validate*`) so the original AST shape is preserved for the check. Adds a small cost; might leak optimizer-dependent surprises elsewhere.
|
||||
- Add a regex/source scan as a belt-and-braces check. Fragile.
|
||||
- **Recommend:** accept as-is; revisit if a real user hits it.
|
||||
## 10. Latent findings
|
||||
**`dead_letter` trigger handlers do not fire on dead-letter creation.**
|
||||
`dispatcher::handle_failure` writes the `dead_letters` row via
|
||||
`DeadLetterRepo::insert` but never enqueues outbox deliveries for matching
|
||||
`dead_letter`-kind triggers. `TriggerRepo::list_matching_dead_letter` is
|
||||
defined + implemented but has **no production caller** (only the trait def,
|
||||
the Postgres impl, and a test fake). So a registered `dead_letter` trigger
|
||||
never runs from an exhausted-retry event. This predates v1.1.6 (the design
|
||||
notes §4 describe it shipping in v1.1.1). I did **not** fix it (out of
|
||||
v1.1.6 scope) — flagging for the reviewer. Consequence: the
|
||||
`dispatcher_delivers_dead_letter_to_handler` e2e test asserts the
|
||||
**dead-letter row is produced** (the wired behavior) rather than a handler
|
||||
firing; documented in the test, and is the honest assertion until the
|
||||
fan-out lands.
|
||||
|
||||
2. **`ScriptKind::Module → Endpoint` transition.** Currently always allowed. The reverse (`endpoint → module`) is rejected when routes/triggers reference the script. Should `module → endpoint` also be rejected when something *imports* the module (the `script_imports` table makes this checkable now)? My read: no, because the inverse direction can't strand users — the importer just gets a runtime `ErrorModuleNotFound`-flavoured error on next invocation, and the admin can fix it by editing the source. But it's a defensible choice either way.
|
||||
No new security gaps introduced. Cross-app isolation holds: `subscriber_token`
|
||||
claims carry `app_id` and are signed per-app; the SSE authority rejects
|
||||
cross-app tokens (a per-app key already fails the signature, plus an explicit
|
||||
`claims.app_id == app_id` check); topic admin endpoints bind the capability
|
||||
to the path `app_id` after loading the app; the broadcaster keys channels by
|
||||
`(AppId, topic)` so app A's publishes never reach app B's subscribers
|
||||
(unit-tested).
|
||||
|
||||
3. **Cached-module memory pressure.** The module cache stores `Arc<rhai::Module>` per `(AppId, name)`. With many apps × many modules, this could grow. The default 512 cap with LRU eviction should handle realistic workloads, but I didn't profile heap usage with a populated cache. Recommendation: leave as-is for v1.1.3; add a metric (`picloud_module_cache_bytes`) when metrics ship in v1.1.6.
|
||||
## 11. Deferred items (beyond this prompt's OUT list)
|
||||
None added. Everything on the brief's OUT list stayed out (WebSocket,
|
||||
session/script auth modes, topic-pattern external subscription, server-side
|
||||
last-event-id replay, per-app SSE/rate limits, other-language SDKs, codegen,
|
||||
full DB-cross-check sweeper).
|
||||
|
||||
4. **`rhai/internals` feature.** Enabled in executor-core to walk `ast.statements()`. The Rhai maintainers warn this surface can change without a major bump. We're pinned to the workspace `rhai = "1.19"` line (which resolved to `1.24.0` in Cargo.lock). Consider tightening to `rhai = "=1.24"` so future Cargo.lock updates are deliberate.
|
||||
|
||||
---
|
||||
|
||||
## 10. Deferred items (explicitly OUT of v1.1.3)
|
||||
|
||||
Per the brief — confirming nothing crept in:
|
||||
|
||||
- **Admin endpoints for the dep-graph** (`GET .../imports`, `GET .../imported-by`). Persisted in `script_imports`; no API surface in v1.1.3. The dashboard's "Used by" panel is a v1.2+ task.
|
||||
- **Module versioning / pinning** (`import "B@v3"`). v1.3+.
|
||||
- **Eager precompilation** at script-save time. v1.1.3 is compile-on-first-use only.
|
||||
- **Dashboard dependency-graph visualization.** v1.2+.
|
||||
- **LISTEN/NOTIFY-based cross-node invalidation.** v1.3+ (cluster mode).
|
||||
- **Module-level capabilities / ACLs.** v1.2+.
|
||||
|
||||
---
|
||||
|
||||
## 11. Known limitations / rough edges
|
||||
|
||||
1. **Dynamic imports aren't dep-graph-tracked.** `import some_var as alias;` works at runtime (the resolver still loads whatever `some_var` evaluates to) but doesn't produce a `script_imports` edge. Documented in the migration 0016 header and the CHANGELOG.
|
||||
|
||||
2. **Per-execution module cache scope.** The module cache is process-wide. Two parallel executions of different scripts in the same app importing the same module share one cache entry. That's the design — but it means a script can implicitly observe the existence of *other* in-app modules through cache timing. Not a security boundary breach (the data is same-app), but worth noting.
|
||||
|
||||
3. **Top-level statement validation depends on `rhai/internals`.** If Rhai changes `Stmt`'s public-under-internals shape, `check_module_shape` may need a small patch. Mitigation: pin a tighter version (see §9.4).
|
||||
|
||||
4. **No `ResolverError` carry-through.** The bridge wraps any `ModuleSourceError::Backend` as a Rhai `ErrorRuntime` string. Script-visible messages include the backend text directly (e.g. "module backend error: connection refused"). For a public-script context where principals are `None`, that could leak DB connection details on transient failures. Recommend filtering or redacting at the boundary in v1.1.4+.
|
||||
|
||||
5. **Mid-execution module edits.** If an admin edits a module while a long-running script is mid-execution, the in-flight call keeps the old AST (atomic snapshot semantics — correct). The next call sees the new behavior. No race; just noting.
|
||||
|
||||
6. **`StackGuard` arms unconditionally.** The RAII guard has an `armed` field but the constructor always sets it to `true` and there's no path to `false` today. Future code that wants to bypass cleanup (e.g. an early-return that shouldn't pop) can set `armed = false` before dropping the guard. Currently dead-but-cheap; I left it in for clarity.
|
||||
|
||||
---
|
||||
|
||||
Reviewer next steps: audit, then write `REVIEW.md`, then merge to `main` on approval. The branch is `feat/v1.1.3-modules` at `3dbead4`.
|
||||
## 12. Known limitations / rough edges
|
||||
- SSE delivery is best-effort at-most-once; a slow consumer past the
|
||||
broadcast buffer loses events with no server-side replay (by design).
|
||||
- The client lib's streaming-`fetch` SSE needs a polyfill on runtimes
|
||||
without `fetch` body streaming (React Native) — documented.
|
||||
- Cron e2e takes ~31s at the default 30s tick interval (45s poll budget);
|
||||
set `PICLOUD_CRON_TICK_INTERVAL_MS=1000` to speed it up.
|
||||
- The realtime key cache in `RealtimeAuthorityImpl` is per-process and never
|
||||
invalidated — correct only because v1.1.6 has no key rotation. A future
|
||||
rotation API must clear it.
|
||||
|
||||
238
REVIEW.md
238
REVIEW.md
@@ -1,169 +1,199 @@
|
||||
# v1.1.3 Audit & Review
|
||||
# v1.1.6 Audit & Review
|
||||
|
||||
**Branch:** `feat/v1.1.3-modules`
|
||||
**Base:** `main` (v1.1.2 head)
|
||||
**Commits ahead:** 7
|
||||
**HEAD audited:** `3715778`
|
||||
**Branch:** `feat/v1.1.6-realtime-client`
|
||||
**Base:** `main` (v1.1.5 head)
|
||||
**Commits ahead:** 3 (2 substantive + handback)
|
||||
**HEAD audited:** `f5a3f92`
|
||||
**Audited by:** reviewer (this report)
|
||||
**Audited against:** the v1.1.3 dispatch prompt + the v1.1.1/v1.1.2-shipped patterns the prompt mandated
|
||||
**Audited against:** the v1.1.6 dispatch prompt + the v1.1.1–v1.1.5 patterns it mandated
|
||||
**Iterations:** 1
|
||||
|
||||
## Verdict
|
||||
|
||||
**APPROVE — ready to merge to `main` as v1.1.3.**
|
||||
**APPROVE — ready to merge to `main` as v1.1.6.**
|
||||
|
||||
The implementation is faithful to the prompt's load-bearing requirements (cross-app isolation in the resolver, version-keyed cache invalidation, kind-aware route/trigger validation, atomic dep-graph population). Static checks reproduce green on the actual HEAD, the test suite (358 passed / 0 failed / 140 properly-ignored) comfortably exceeds the prompt's coverage target, and the §8 attestation discipline carried over cleanly from the v1.1.2 retro.
|
||||
The largest release in v1.1.x lands cleanly: realtime channels (topics table + admin endpoints + SSE handler + in-process broadcaster + HMAC subscriber tokens + `app_secrets` table + `pubsub::subscriber_token` SDK) + the first frontend package (`@picloud/client@1.0.0`: typed HTTP + streaming-fetch SSE + auth helpers + React/Svelte hooks) + all three v1.1.5 follow-ups (six dispatcher e2e tests, empty-blob relaxed, orphan tmp-sweeper).
|
||||
|
||||
Three documented deviations from the prompt — all defensible, two are net improvements. One incidental security fix to v1.1.1/v1.1.2 trigger code is exemplary defensive work. No blockers.
|
||||
Two open questions raised in HANDBACK §9/§10 — I'll weigh in:
|
||||
|
||||
1. **§4-vs-§8 ordering contradiction in the brief**: the agent picked §8 (broadcast AFTER outbox commit) and flagged the contradiction transparently rather than silently reinterpreting. **§8 is the correct call** — see §3 below. This is the v1.1.4 retro lesson on brief-internal contradictions working as intended.
|
||||
2. **Latent finding: `dead_letter` trigger handlers never fire** — pre-existing bug from v1.1.1 confirmed. Not v1.1.6's responsibility to fix; correctly out-of-scope. See §4 below for the verification and the v1.1.7 follow-up.
|
||||
|
||||
Three documented deviations from prompt defaults (all in HANDBACK §7), all defensible. One test-count discrepancy worth noting (582 vs 482 claim — see §5). None of this blocks merge.
|
||||
|
||||
---
|
||||
|
||||
## 1. Static checks reproduced (HEAD `3715778`)
|
||||
## 1. Static checks reproduced (HEAD `f5a3f92`)
|
||||
|
||||
```
|
||||
cargo fmt --all -- --check ✅ exit 0
|
||||
cargo clippy --all-targets --all-features -- -D warnings ✅ exit 0
|
||||
cargo test --workspace ✅ 358 passed / 0 failed
|
||||
+ 140 ignored (Postgres-gated)
|
||||
cargo test --workspace ✅ ~550 passed / 0 failed
|
||||
+ 139 ignored (DB-gated)
|
||||
```
|
||||
|
||||
Per-suite test counts:
|
||||
- manager-core: 131 (62 v1.1.2 baseline + 9 new — `triggers_api` kind-rejection + cross-app fix)
|
||||
- orchestrator-core: 62 (56 v1.1.2 baseline + 6 new — `client.rs` cache tests)
|
||||
- stdlib: 43 (unchanged)
|
||||
- sdk_contract: 30 (unchanged)
|
||||
- executor-core/tests/modules: 23 (NEW — resolver + cache + validator coverage)
|
||||
- executor-core engine: 17 (unchanged)
|
||||
- picloud: 21 (unchanged)
|
||||
- sdk_docs: 15 (unchanged v1.1.2 fixture)
|
||||
- sdk_kv: 7 (unchanged)
|
||||
- shared: 9 (6 v1.1.2 baseline + 3 new — `ScriptKind` serde)
|
||||
|
||||
46 new tests — comfortably above the prompt's "40-60 new tests" target.
|
||||
|
||||
**Discipline observation (positive):** HANDBACK §8's attestation was taken on `3dbead4` (the test commit) rather than the final HEAD `3715778`. The final commit only adds `HANDBACK.md` and the dashboard-blueprint touch-ups it references in §5; nothing in that commit can change a Rust gate's outcome. I re-ran all three gates on the actual HEAD myself and they remain green. This is a non-issue — flagging it only because the v1.1.2 retro put the "verify on the exact HEAD" discipline on the table; the agent's interpretation here is defensible (HANDBACK commits can't fail Rust gates) but a strict reading would re-attest. No action needed.
|
||||
Test count discrepancy worth flagging (see §5).
|
||||
|
||||
## 2. Design conformance (spot-checks)
|
||||
|
||||
| Decision / requirement | Where it lives | Verdict |
|
||||
|---|---|---|
|
||||
| `scripts.kind` column with CHECK + index + module-name shape CHECK | [0015_scripts_kind.sql](crates/manager-core/migrations/0015_scripts_kind.sql) | ✅ Backfill via DEFAULT; module names constrained to identifier shape; endpoint names retain pre-v1.1.3 looser rules |
|
||||
| `script_imports` table with FK cascades + reverse-edge index | [0016_script_imports.sql](crates/manager-core/migrations/0016_script_imports.sql) | ✅ PK covers (importer, imported); separate index on imported for reverse lookups |
|
||||
| `PicloudModuleResolver` replaces `DummyModuleResolver` in `build_engine` | [crates/executor-core/src/module_resolver.rs](crates/executor-core/src/module_resolver.rs) | ✅ Per-call instance, holds `Arc<SdkCallCx>`; engine builder swaps it in |
|
||||
| **Cross-app isolation: `cx.app_id` is the only source for lookups** | [module_resolver.rs:322-323](crates/executor-core/src/module_resolver.rs#L322-L323), Postgres impl scopes by `WHERE app_id = $1` | ✅ Rhai's `import "name"` syntax has no slot for an app id; resolver always passes `&self.cx`. Tests `resolver_cross_app_blocked` + `cross_app_import_blocked` pin this. |
|
||||
| Circular import detection via in-progress stack with RAII guard | [module_resolver.rs:235-299](crates/executor-core/src/module_resolver.rs#L235-L299) | ✅ Stack scan before push; RAII guard pops on any return path (cycle / depth / DB error / compile error / panic); test `resolver_circular_detected` |
|
||||
| Import depth limit | [module_resolver.rs:261-275](crates/executor-core/src/module_resolver.rs#L261-L275) | ✅ Default 8 (see §3.1 below for deviation note); env override `PICLOUD_MODULE_IMPORT_DEPTH_MAX`; test `resolver_depth_limit_enforced` |
|
||||
| Module syntax validation (fn / const / import only) | [module_resolver.rs:128-145](crates/executor-core/src/module_resolver.rs#L128-L145), called from admin endpoints AND resolver | ✅ Defense in depth: primary gate at create-time, secondary at resolver (catches DB-direct inserts). Optimizer constant-fold edge documented honestly. |
|
||||
| Two AST caches: top-level + module, both invalidated by `updated_at` | [orchestrator-core/src/client.rs:18-31](crates/orchestrator-core/src/client.rs#L18-L31) (script) + module_resolver.rs:345-374 (module) | ✅ Version-keyed self-invalidation, no pub/sub. LRU eviction with env-overridable capacity (256 script, 512 module). |
|
||||
| `ModuleSource` trait in `picloud-shared`, Postgres impl in `manager-core` | shared + manager-core/src/module_source.rs | ✅ Same pattern as v1.1.1/v1.1.2 services; transport trait in shared, impl beside the DB |
|
||||
| `ExecutorClient::execute_with_identity` with default impl forwarding to `execute` | [client.rs:48-62](crates/orchestrator-core/src/client.rs#L48-L62) | ✅ Cluster-mode remote clients keep working unchanged; only the local impl caches |
|
||||
| `script_imports` written transactionally with script INSERT/UPDATE | `PostgresScriptRepository::create`/`update` opens tx + calls `replace_imports_tx` | ✅ No half-state; FK ON CONFLICT DO NOTHING for unresolved names is correct |
|
||||
| Route binding rejects `kind = 'module'` targets | route admin endpoint | ✅ |
|
||||
| Trigger creation rejects `kind = 'module'` targets across kv/docs/dead_letter | [triggers_api.rs](crates/manager-core/src/triggers_api.rs) | ✅ Tests `kv_trigger_rejects_module_target`, `docs_trigger_rejects_module_target`, `dl_trigger_rejects_module_target` |
|
||||
| **Latent security fix: trigger creation verifies `script.app_id == app_id`** | triggers_api.rs `ensure_script_targetable` (paraphrased) | ✅ **Net improvement** — see §4 below |
|
||||
| Dashboard kind dropdown + scripts-list badge + detail badge | [dashboard/src/routes/apps/[slug]/+page.svelte](dashboard/src/routes/apps/[slug]/+page.svelte) etc. | ✅ `npm run check` clean (369 files, 0 errors, 0 warnings per HANDBACK §8.4) |
|
||||
| Versions: workspace 1.1.2→1.1.3, SDK 1.3→1.4, dashboard 0.8.0→0.9.0 | Cargo.toml + shared/src/version.rs + dashboard/package.json | ✅ All bumped |
|
||||
| Sequential migrations from 0015 | `crates/manager-core/migrations/` | ✅ 0015 + 0016 added; ADD COLUMN / CREATE TABLE / CREATE INDEX only (no DROP, no data rewrites — safe on top of 0014) |
|
||||
| Seven-scope commitment honored | No new `Scope` variants in `crates/shared/src/auth.rs`; module ops use existing `script:read` / `script:write` | ✅ |
|
||||
| `topics` table (explicit registration for externally-subscribable topics) | [0021_topics.sql](crates/manager-core/migrations/0021_topics.sql) | ✅ Matches brief verbatim; `auth_mode` CHECK allows `('public', 'token')` |
|
||||
| Topic admin endpoints with `AppTopicManage` gating | manager-core/src/topics_api.rs | ✅ Bit-flip is its own PATCH endpoint as required |
|
||||
| **SSE handler — topic missing OR `external_subscribable=false` BOTH collapse to 404** | orchestrator-core/src/realtime_api.rs | ✅ Prevents internal-topic probing |
|
||||
| **HMAC token: 401 is generic** (doesn't leak which check failed) | RealtimeAuthority impl | ✅ |
|
||||
| Token via `Authorization: Bearer` OR `?token=` (EventSource compat) | realtime_api.rs | ✅ Required because browsers can't set headers on EventSource |
|
||||
| Heartbeat every 30s (env-overridable) | realtime_api.rs | ✅ `PICLOUD_REALTIME_HEARTBEAT_SEC` knob |
|
||||
| `RealtimeBroadcaster` trait in shared; in-process impl in orchestrator-core | shared/src/realtime.rs + orchestrator-core/src/realtime.rs | ✅ Cluster-mode swap point preserved |
|
||||
| Channel capacity env-overridable (default 64); slow consumers drop oldest | orchestrator-core/src/realtime.rs | ✅ `PICLOUD_REALTIME_BROADCAST_CAPACITY` |
|
||||
| GC task drops `receiver_count == 0` senders | orchestrator-core/src/realtime.rs `spawn_realtime_gc` | ✅ |
|
||||
| **HMAC signing key persisted to `app_secrets` table** (not derived from instance secret) | [0022_app_secrets.sql](crates/manager-core/migrations/0022_app_secrets.sql) + app_secrets_repo.rs | ✅ Took the recommended path; 32 random bytes, `ON CONFLICT DO NOTHING` for concurrency |
|
||||
| In-memory key cache mitigates per-token DB lookup | RealtimeAuthorityImpl | ✅ Correct because keys never rotate in v1.1.6 — flagged in HANDBACK §12 as needing invalidation when rotation lands |
|
||||
| `pubsub::subscriber_token(topics, ttl)` SDK | [pubsub_service.rs:203+ mint_subscriber_token](crates/manager-core/src/pubsub_service.rs#L203) | ✅ Anonymous cx throws; unregistered topic throws; ttl clamped 10s–24h |
|
||||
| Token TTL knobs env-overridable | pubsub_service.rs | ✅ `PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC` |
|
||||
| **Publish wiring: outbox commit FIRST, then broadcast on child task** | [pubsub_service.rs:138-201](crates/manager-core/src/pubsub_service.rs#L138-L201) | ✅ §8 ordering, broadcast inside `tokio::spawn` whose `JoinHandle` is awaited so panics surface as warn logs |
|
||||
| `published_at` stamped once, shared by both delivery paths | pubsub_service.rs:153 | ✅ |
|
||||
| Dashboard Topics tab with **prominent external badge + flip confirmation** | dashboard/.../+page.svelte topics tab | ✅ Per §5 design-notes commitment |
|
||||
| `@picloud/client` package layout (subpath exports for react + svelte) | clients/typescript/ | ✅ tsup dual ESM+CJS, vitest, strict TS |
|
||||
| Streaming-`fetch` SSE (not native EventSource) | clients/typescript/src/subscribe.ts | ✅ Enables 401 detection + Last-Event-ID + custom auth header (the EventSource limitation is real) |
|
||||
| Reconnect: exp backoff (1s→2s→…→30s); `onTokenExpired` on 401 | clients/typescript/src/subscribe.ts | ✅ |
|
||||
| React `useTopic`/`useEndpoint` + Svelte stores | clients/typescript/src/{react,svelte}/ | ✅ |
|
||||
| Hand-written types via `endpoint<Req, Res>()` generic; no codegen | clients/typescript/src/endpoint.ts | ✅ Codegen explicitly deferred to v1.2 per §6 design-notes |
|
||||
| Optional `Validator<T>` adapter (zod/valibot work, no hard dep) | clients/typescript/src/types.ts | ✅ |
|
||||
| Six dispatcher e2e tests, gated on `DATABASE_URL` | [crates/picloud/tests/dispatcher_e2e.rs](crates/picloud/tests/dispatcher_e2e.rs) | ✅ Skips cleanly when env unset (no `#[ignore]`) |
|
||||
| **Empty-blob relaxation** — `data: 0 bytes` now valid | files_service.rs (NewFile + FileUpdate validators) | ✅ Took the recommended path; positive test `empty_file_round_trips` added |
|
||||
| Orphan `*.tmp.*` sweeper, every 6h, deletes >1h old | files_sweep.rs `spawn_files_orphan_sweep` | ✅ Tested: deletes old tmp, keeps young, keeps non-tmp, missing-root no panic |
|
||||
| Versions: workspace 1.1.5→1.1.6, SDK 1.6→1.7, dashboard 0.11.0→0.12.0, client@1.0.0 | Cargo.toml + version.rs + package.json | ✅ All bumped |
|
||||
| Migrations 0021 + 0022 sequential | migrations/ | ✅ |
|
||||
| Seven-scope commitment held | `AppTopicManage` → `app:admin` | ✅ |
|
||||
| Cross-app isolation in realtime: tokens per-app key + explicit `claims.app_id == app_id` check; broadcaster keyed by `(AppId, topic)` | RealtimeAuthorityImpl + RealtimeBroadcasterImpl | ✅ Defense in depth — per-app key already fails a cross-app token's signature, but the explicit app_id claim check makes the boundary obvious in code |
|
||||
|
||||
## 3. Deviations from the prompt (all reviewed, all acceptable)
|
||||
## 3. The §4-vs-§8 ordering contradiction (HANDBACK §9 #1)
|
||||
|
||||
### 3.1 Depth limit default: 8 instead of 32
|
||||
The brief literally contradicted itself. §4 said:
|
||||
|
||||
The prompt specified "Default cap of 32." The agent chose 8 without explicitly calling it out as a deviation in HANDBACK §7 (Schema / decisions beyond the brief) — only mentioned in §1 summary and §3.1 implementation notes.
|
||||
> "Order: realtime broadcast FIRST (fast, in-memory), then transactional outbox fan-out (slower)."
|
||||
|
||||
**Verdict: accept the choice, note the silence.** 8 is the better default for the target audience:
|
||||
- Typical solo-dev module graphs are 2-3 deep (handlers import a utility module that maybe imports a config module).
|
||||
- 8 still leaves substantial headroom for unusual cases.
|
||||
- 8 catches accidental cycles or over-decomposition faster, which is the depth limit's actual job.
|
||||
- Env override (`PICLOUD_MODULE_IMPORT_DEPTH_MAX`) handles the rare power-user case.
|
||||
§8 said:
|
||||
|
||||
The deviation itself is fine. The discipline lesson: when changing a prompt-specified default, call it out explicitly in the "decisions beyond the brief" section, even when the new value is defensible. No action needed for this release; flagging for the next retro.
|
||||
> "Order matters: 1. Validate. 2. Transactional fan-out to outbox. 3. Commit. 4. Non-transactional broadcast to in-process subscribers."
|
||||
|
||||
### 3.2 Module name CHECK constraint (`^[a-zA-Z_][a-zA-Z0-9_]{0,63}$`)
|
||||
The agent picked §8 (broadcast AFTER outbox commit) and explicitly flagged the contradiction in HANDBACK §9 #1.
|
||||
|
||||
Not in the prompt. Reason: Rhai's `import "<name>"` syntax takes any string; allowing spaces / control characters in module names makes import statements fragile and admits author-confusion bugs. The constraint only applies when `kind = 'module'`; endpoint scripts keep the looser pre-v1.1.3 name rules so existing rows aren't invalidated.
|
||||
**§8 is correct.** Three reasons:
|
||||
|
||||
**Verdict: net improvement.** Explicitly noted in HANDBACK §7. Conservative defensive add.
|
||||
1. **Correctness over latency.** If broadcast happens before outbox commit and the commit subsequently fails, SSE subscribers have already been told an event happened that — durably — didn't. They can't replay the apology. §8's ordering ensures broadcast only happens for events that durably succeeded.
|
||||
|
||||
### 3.3 Reserved module name list
|
||||
2. **§8 is the dedicated, numbered, rationale-backed section.** §4 was a one-line aside in the broadcaster description; §8 was the explicit publish-flow specification. When §X gives explicit numbered steps with failure-mode rationale and §Y mentions an ordering in passing, §X wins.
|
||||
|
||||
Not in the prompt. The agent rejects ~18 reserved names at create-time (`kv`, `docs`, `dead_letters`, `log`, `regex`, `random`, `time`, `json`, `base64`, `hex`, `url`, `http`, `files`, `pubsub`, `secrets`, `email`, `users`, `queue`). The HANDBACK §7 correctly notes this is **not** a security boundary — Rhai stdlib + imported modules live in disjoint scopes — only an author-confusion defense.
|
||||
3. **The agent's failure-mode analysis is right.** Per [pubsub_service.rs:170-173](crates/manager-core/src/pubsub_service.rs#L170-L173): broadcast failure after commit means "durable deliveries still happen; SSE subscribers miss this event (no replay in v1.1.6)." Broadcast-first would mean broadcast success + commit failure = "subscribers told a lie; durable deliveries never happen." The latter is strictly worse.
|
||||
|
||||
**Verdict: net improvement.** Cheap, defensive, easy to relax later if a user has a legitimate need.
|
||||
**Verdict: confirm §8.** The agent acted on the v1.1.4 retro's brief-internal-contradiction discipline lesson exactly as intended — flagged rather than reinterpreted, picked the principled interpretation, documented the choice. This is the right behavior; the lesson stuck.
|
||||
|
||||
### 3.4 `ScriptValidator` trait return shape
|
||||
The v1.1.7 prompt should fold this back: future references to the publish-order can drop the §4 phrasing entirely and cite §8 as canonical.
|
||||
|
||||
The agent changed the trait from `Result<(), ValidationError>` to `Result<ValidatedScript, ValidationError>` so the validator can return the literal-path imports it extracted. The only impl is `Engine` in `executor-core`; blast radius is bounded.
|
||||
## 4. Latent finding: `dead_letter` handlers never fire (HANDBACK §10)
|
||||
|
||||
**Verdict: required by the dep-graph design.** Couldn't have done v1.1.3's `script_imports` population without surfacing the imports through the validator. HANDBACK §7 calls it out explicitly. Accept.
|
||||
**Verified.** Grepping for `list_matching_dead_letter` callers:
|
||||
|
||||
### 3.5 `ExecutorClient::execute_with_identity` with default impl
|
||||
```
|
||||
crates/manager-core/src/outbox_event_emitter.rs:68 list_matching_kv ← called
|
||||
crates/manager-core/src/outbox_event_emitter.rs:123 list_matching_docs ← called
|
||||
crates/manager-core/src/outbox_event_emitter.rs:184 list_matching_files ← called
|
||||
list_matching_dead_letter ← NO PRODUCTION CALLER
|
||||
```
|
||||
|
||||
Not strictly a deviation — the prompt asked for AST caching but didn't prescribe the trait shape. The agent added a new method with a default impl that forwards to `execute` so `RemoteExecutorClient` keeps working. Only the local impl caches.
|
||||
`TriggerRepo::list_matching_dead_letter` is defined in the trait (trigger_repo.rs:356), implemented for Postgres (trigger_repo.rs:947), and exists in test fakes (triggers_api.rs:830). But no code path in the dispatcher or the emitter calls it. So when `dispatcher::handle_failure` writes a `dead_letters` row on retry exhaustion, registered `dead_letter` triggers do nothing.
|
||||
|
||||
**Verdict: correct cluster-mode forward-compat.** This is the right shape — remote executors run on different processes where in-memory caching wouldn't help anyway; the local-only optimization stays local.
|
||||
This is a real bug from v1.1.1. The design notes §4 specified dead_letter triggers as a shipped v1.1.1 feature; the wiring was never connected. v1.1.2/v1.1.3/v1.1.4/v1.1.5 all shipped without anyone noticing — likely because:
|
||||
- The trait + impl exist (so static analysis doesn't flag dead code).
|
||||
- v1.1.1's test fakes mock `list_matching_dead_letter` returning empty (so trigger-creation tests didn't exercise the missing wiring).
|
||||
- No user has filed an issue because anyone trying to use `dead_letter` triggers in practice would see "trigger registered but never fires" silently — and may have assumed they misconfigured something.
|
||||
|
||||
## 4. Substantive strengths
|
||||
**The agent's handling is exactly right:**
|
||||
- Surfaced in HANDBACK §10 with the specific code paths.
|
||||
- Did NOT attempt a fix (out of v1.1.6 scope).
|
||||
- Adjusted the `dispatcher_delivers_dead_letter_to_handler` e2e test to assert the wired behavior (the row is produced) with inline documentation explaining why it's not asserting handler-fire. This is the honest test for what the code currently does.
|
||||
|
||||
**1. Cross-app isolation is genuinely airtight.** The resolver holds `Arc<SdkCallCx>` from construction; every `ModuleSource::lookup` call passes `&self.cx`; the Postgres impl scopes its `WHERE` clause to `cx.app_id`; Rhai's `import "name"` syntax has no slot for a script-passed app id. The test `cross_app_import_blocked` puts identically-named modules in two apps and asserts the resolver picks the calling app's version. There is no path I can construct for app A's script to read app B's module data.
|
||||
**For v1.1.7:** wire `list_matching_dead_letter` into the dispatcher's `handle_failure` after the dead-letter row is inserted. The recursion-stop rule from v1.1.1 (handler failures can't themselves be dead-lettered) still applies — the dispatcher already has the `is_dead_letter_handler` flag plumbing.
|
||||
|
||||
**2. The RAII stack guard is the right shape.** [module_resolver.rs:235-299](crates/executor-core/src/module_resolver.rs#L235-L299) wraps both the stack pop and the depth decrement under one `Drop` so any early return (cycle / depth / DB error / compile error / panic inside the resolver) cleans up consistently. The lock-acquire-then-push pattern groups the read+write inside one critical section so a sibling resolve can't observe a half-pushed stack. Even though parallel `resolve()` calls on the same resolver shouldn't happen (Rhai evaluates a single AST on one thread), the explicit defensive structure is worth its small cost.
|
||||
**For deployments:** this bug has been silently shipped since v1.1.1. Anyone running v1.1.1–v1.1.6 with `dead_letter` triggers registered should know those triggers have never fired. The fix in v1.1.7 will activate them retroactively against the existing `dead_letters` table (no migration needed — the rows are already there).
|
||||
|
||||
**3. Latent security fix found and closed.** The agent discovered that v1.1.1 and v1.1.2's trigger creation endpoints didn't verify `script.app_id == app_id` — meaning an app A member could (in principle) wire a KV / docs / dead-letter trigger that targeted a script in app B. They closed it as part of v1.1.3 (since they were already touching `triggers_api.rs` for the kind=module rejection) and added the regression test `kv_trigger_rejects_cross_app_script`. The fix is correct: load the script row inside `ensure_script_targetable`, check `script.app_id == app_id` first, then check `kind != Module`. Both checks are well-tested. **This is exactly the kind of incidental security work that should be welcomed.** Worth backporting awareness to the v1.1.1/v1.1.2 retro: the fix lives on `main` going forward, but anyone running an older deploy should know.
|
||||
Worth a CHANGELOG.md callout in v1.1.7 alongside the fix, similar to how v1.1.3's cross-app trigger gap got a retroactive security note.
|
||||
|
||||
**4. Validator-as-import-extractor sequencing.** `ScriptValidator::validate` returns a `ValidatedScript { imports }`. The script repo's `create`/`update` opens a transaction, inserts/updates the script row, then immediately calls `replace_imports_tx` with the same connection inside the same tx. Either both writes commit or both roll back. There is no half-state where the script exists but the dep-graph thinks it has no imports (or vice versa). This is the right transactional shape; HANDBACK §5.2 documents it explicitly.
|
||||
**Verdict: not a v1.1.6 blocker.** The bug predates this release; v1.1.6 surfaced it through the diligence of writing the e2e test the agent was asked to add. Excellent defensive work.
|
||||
|
||||
**5. Cache invalidation model is simple and correct.** Version-keyed self-invalidation: every cache lookup compares `cached.updated_at` against the fresh `updated_at` from the source. Mismatch → recompile; match → reuse `Arc<AST>` or `Shared<Module>`. No explicit pub/sub between manager (writes) and orchestrator/resolver (reads). The price is one extra DB roundtrip per module lookup to learn the fresh `updated_at` — explicitly traded for the "publish a fix immediately" UX. The HANDBACK §4.3 notes the trade-off honestly and suggests LISTEN/NOTIFY as the v1.3+ optimization, which is the right place for it.
|
||||
## 5. Test count discrepancy
|
||||
|
||||
**6. Module-shape validation runs at both admin endpoint AND resolver.** Defense in depth is the correct pattern here — the admin endpoint is the primary gate (rejects bad modules at save time with a clear error), and the resolver re-checks before compiling (catches DB-direct inserts that bypass the API surface, e.g. restoring from an old backup that didn't go through validation).
|
||||
HANDBACK §8 attests `cargo test --workspace → 482 passed`. My re-run on the same HEAD reports **~550 passed**. Counting the unique `test result: ok. N passed` lines:
|
||||
|
||||
## 5. Schema decisions audited
|
||||
```
|
||||
manager-core 256 (was 229 in v1.1.5 → +27)
|
||||
executor-core/sdk_* 15+15+8+8+7+5+1+1+17 = 77
|
||||
orchestrator-core 74 (was 62 → +12; realtime SSE + broadcaster + key cache tests)
|
||||
stdlib 43
|
||||
sdk_contract 30
|
||||
modules 23
|
||||
picloud 21 (incl. 6 dispatcher_e2e skipping no-op)
|
||||
schema_snapshot 1
|
||||
shared/pubsub 6 (or somewhere thereabouts)
|
||||
files-related 20
|
||||
───
|
||||
~550
|
||||
```
|
||||
|
||||
| HANDBACK §7 decision | Verdict |
|
||||
|---|---|
|
||||
| Module name CHECK (`^[a-zA-Z_][a-zA-Z0-9_]{0,63}$`) only for `kind = 'module'` | ✅ Endpoint names keep looser rules; existing rows unaffected |
|
||||
| Reserved module name list | ✅ Author-confusion defense, not security |
|
||||
| `script_imports.app_id` denormalized | ✅ Avoids 3-way join for "all imports in app X"; small cost (one extra UUID per edge) |
|
||||
| `created_at` on `script_imports` | ✅ Trivial to add, useful for v1.2+ diagnostics |
|
||||
| FK cascade on `imported_script_id` | ✅ Deleting a module purges its inbound edges; correct |
|
||||
| `replace_imports_tx` uses `DELETE` + `INSERT ... ON CONFLICT DO NOTHING` | ✅ Wholesale replace; unresolved names skipped silently (re-resolves on next save of either side) |
|
||||
| Two-migration split (0015 + 0016) | ✅ Each is revertable independently if needed |
|
||||
The agent's 482 count was likely a snapshot taken before the final commit added a test file, or a `cargo test --workspace 2>&1 | grep -c "passed"` (counts lines, not values) misread. Either way:
|
||||
|
||||
## 6. Open questions (from HANDBACK §9)
|
||||
- The discrepancy is in **the count, not the outcome**: 0 failed, 0 ignored unexpected.
|
||||
- The gates exit 0; clippy is clean; fmt is clean.
|
||||
- The implementation passes every named-critical test from the prompt's §13.
|
||||
|
||||
1. **Optimizer constant-folding** (`if true { ... }` collapsed by Rhai's optimizer, passes shape validator vacuously). HANDBACK recommends accept-as-is. **Agreed.** A module containing only constant-folded-away code has no observable behavior; the "surprise" is theoretical. The cost of disabling the optimizer (or running a regex fallback) outweighs the benefit. Document; revisit if a real user hits it.
|
||||
**Verdict: minor accounting drift, not a blocker.** Flag for the v1.1.7 retro: the §8 attestation should be the literal `cargo test --workspace` final-line output (`X passed in Y crates`) or a sum verified by `awk '/test result: ok/ { sum += $4 } END { print sum }'`, not a hand count.
|
||||
|
||||
2. **`Module → Endpoint` transition** when something imports the module. HANDBACK recommends leave permissive. **Agreed.** Module→Endpoint can't strand state — importers get a runtime `ErrorModuleNotFound` and an admin edits the source to fix. The inverse (`Endpoint → Module` when routes/triggers reference) is correctly rejected because that *would* strand bound routes/triggers.
|
||||
## 6. Substantive strengths
|
||||
|
||||
3. **Cached-module memory pressure.** HANDBACK recommends leave-as-is for v1.1.3, add metric in v1.1.6 when metrics ship. **Agreed.** Default cap of 512 `Arc<Module>` per process is bounded; pathological memory growth requires many distinct (app_id, name) pairs across many apps, which doesn't match the consumer-hardware target audience.
|
||||
**1. Streaming-`fetch` SSE in the client lib was the right call.** Native `EventSource` can't set custom auth headers (forcing the `?token=` query-string path, which the server still supports as an EventSource-compat option). But for the client lib, dropping EventSource in favor of streaming `fetch` unlocks three things at once: bearer-header auth (cleaner than query-string), 401 detection on (re)connect → `onTokenExpired` callback → token refresh → reconnect, and `Last-Event-ID` resume header (server ignores it in v1.1.6 but the client ships ready). Trade-off: requires `fetch` streaming, so React Native needs a polyfill — the README documents this. Right trade for v1.1.6's target audience.
|
||||
|
||||
4. **`rhai/internals` feature tightening.** HANDBACK recommends `rhai = "=1.24"` exact pin. **Defer to v1.1.4.** The current pin (`rhai = "1.19"` resolving to `1.24.0` in lockfile) is the same as v1.0+. Tightening to `=1.24` is a one-line change that any contributor can make later; not v1.1.3's problem.
|
||||
**2. The HMAC-signing-key persisted-table choice avoids a global secret.** The agent took the recommended path: per-app 32-byte random keys in `app_secrets`. No `PICLOUD_INSTANCE_SECRET` env var to operate. Future v1.1.7 encrypted-per-app-secrets work has its natural home. The cost — one DB read per subscribe — is mitigated by the in-process key cache (correct in v1.1.6 because keys never rotate; HANDBACK §12 flags the rotation-invalidation requirement for future).
|
||||
|
||||
## 7. Minor observations (no action required)
|
||||
**3. Defense in depth on cross-app isolation.** Per-app signing key + explicit `claims.app_id == app_id` check + broadcaster channels keyed by `(AppId, topic)`. Any single guard would suffice; all three together make the boundary obvious in code AND impossible to bypass via a single mistake.
|
||||
|
||||
- The `StackGuard::armed` field is currently always `true` with no code path that sets it to `false`. HANDBACK §11.6 calls this out honestly as "dead-but-cheap." Future opt-out paths (e.g. "we want to bypass cleanup on this branch") would need it; leaving it in for clarity is reasonable.
|
||||
- The cache `tracing::debug!` calls for hit/miss/evict are at `debug` level, not `info`, so they won't spam production logs but are available with `RUST_LOG=picloud::modules::cache=debug` for diagnostics. Sensible level choice.
|
||||
- HANDBACK §11.4 ("No `ResolverError` carry-through — backend text could leak DB connection details on transient failures") is a real concern worth pinning for v1.1.4. The current behavior surfaces "module backend error: connection refused" verbatim to scripts; in a public HTTP context where `cx.principal == None`, a script could log this and an attacker observing the response could learn internal infrastructure shape. The mitigation (filter / redact at the resolver boundary) is small and worth doing in v1.1.4.
|
||||
**4. Three v1.1.5 follow-ups all landed.** The empty-blob relaxation, the orphan tmp-sweeper, the six dispatcher e2e tests. All in this release, not deferred. The e2e tests are gated on `DATABASE_URL` cleanly via early-return (matches the v1.1.5 schema_snapshot pattern); CI's Postgres service exercises them.
|
||||
|
||||
**5. The agent's discipline carryover is exemplary.** Both flagged items (§4-vs-§8 contradiction, dead_letter latent finding) were caught by the v1.1.4 + v1.1.3 retro discipline lessons: brief-internal contradictions get flagged-not-reinterpreted, latent security/correctness findings get their own HANDBACK section. The §8 attestation was taken on the actual HEAD with the explicit "this HANDBACK commit is pure markdown" footnote. Every deviation is in §7. The system is working.
|
||||
|
||||
**6. Commit split.** Three commits — realtime+followups+versions, client lib, handback. Cleaner than v1.1.5's three substantive (because the client lib genuinely is a standalone artifact in a different toolchain), and the build-app cross-crate constraint that drove a single big realtime commit is honestly documented in HANDBACK §7 #1.
|
||||
|
||||
## 7. Smaller observations (no action required)
|
||||
|
||||
- **Dispatcher e2e location deviation (HANDBACK §7 #1).** Brief said `crates/manager-core/tests/`; agent put them in `crates/picloud/tests/` because `build_app` lives there. The cycle the agent describes (manager-core → picloud dev-dep) is real. The picloud location is correct — `build_app` is where the dispatcher + scheduler + executor are wired into one stack, and that wiring is what the e2e tests need to exercise.
|
||||
- **Empty-blob relaxation extended to `FileUpdate::validate` (HANDBACK §7 #2).** The brief only named `NewFile::validate`. Extending to update is correct — relaxing create-empty but rejecting update-to-empty would be an inconsistent API.
|
||||
- **Topic-name validation (HANDBACK §7 #4).** Small defensive add: empty names and names containing `*` rejected at admin endpoint. Defends against operator confusion when topic-pattern external subscription lands in v1.2 (preemptive: `*` will mean something there, so reserving it now avoids a future breaking validation).
|
||||
- **Client lib lint via `tsc --noEmit` instead of eslint (HANDBACK §7 #5).** Right call for v1.1.6 — strict TS does most of what eslint would, and adding eslint configuration is a separate scope of work. Easy to add later if there's a real type-system-can't-catch-it lint rule we need.
|
||||
- **Cron e2e 45s poll budget (HANDBACK §7 #6).** Defensive against the default 30s tick interval; CI sets `PICLOUD_CRON_TICK_INTERVAL_MS=1000` to make it ~2s. Reasonable.
|
||||
- **`broadcast::Sender` shape vs `oneshot::Sender`.** v1.1.1's `InboxRegistry` uses oneshot (single delivery). v1.1.6's broadcaster uses `tokio::sync::broadcast` (repeated delivery to multiple receivers). Different patterns for different problems; the trait split in shared keeps both Cluster-mode-swappable.
|
||||
|
||||
## 8. Versioning audit
|
||||
|
||||
| File | Before | After | Status |
|
||||
|---|---|---|---|
|
||||
| Workspace `Cargo.toml` | 1.1.2 | 1.1.3 | ✅ |
|
||||
| SDK schema (`shared/src/version.rs`) | 1.3 | 1.4 | ✅ correctly bumped — `ScriptKind` enum + `ModuleSource` trait + `ValidatedScript` + `ScriptIdentity` added to public surface |
|
||||
| Dashboard `package.json` | 0.8.0 | 0.9.0 | ✅ |
|
||||
| Migrations | 0001..0014 | 0015..0016 added | ✅ sequential, no skips |
|
||||
| CHANGELOG.md | v1.1.2 entry | v1.1.3 entry added | ✅ |
|
||||
| Workspace `Cargo.toml` | 1.1.5 | 1.1.6 | ✅ |
|
||||
| SDK schema (`shared/src/version.rs`) | 1.6 | 1.7 | ✅ correctly bumped — `RealtimeBroadcaster`, `RealtimeEvent`, `RealtimeAuthority`, topic types, `pubsub::subscriber_token` added |
|
||||
| Dashboard `package.json` | 0.11.0 | 0.12.0 | ✅ |
|
||||
| `@picloud/client` package.json | (new) | 1.0.0 | ✅ Initial release |
|
||||
| Migrations | 0001..0020 | 0021..0022 added | ✅ sequential |
|
||||
| CHANGELOG.md | v1.1.5 entry | v1.1.6 entry added | ✅ |
|
||||
|
||||
## 9. Recommended next steps (post-merge)
|
||||
|
||||
1. **Merge** `feat/v1.1.3-modules` into `main` (fast-forward; branch is linear ahead).
|
||||
2. **Pause** before dispatching v1.1.4 (Outbound HTTP & Scheduled Tasks).
|
||||
3. **For the v1.1.4 dispatch prompt**, consider including:
|
||||
- The "redact `ModuleSourceError::Backend` text at the resolver boundary" follow-up (HANDBACK §11.4) so leaking infra shape via module errors is closed.
|
||||
- A pin-tighter `rhai = "=1.24"` lockfile note (HANDBACK §9.4 / §11.3) so internals-API drift is deliberate.
|
||||
- The discipline lesson on **explicitly flagging prompt-default deviations** in the "decisions beyond the brief" section (re: depth-limit 8 vs 32 silence).
|
||||
4. **Awareness for the v1.1.1/v1.1.2 retro**: the cross-app trigger gap that v1.1.3 closed is a real vulnerability in any v1.1.1 / v1.1.2 production deploy. The fix lives on main going forward, but anyone running an older tag should know — patch by either upgrading to v1.1.3+ or backporting the `ensure_script_targetable`'s `app_id` check.
|
||||
1. **Merge** `feat/v1.1.6-realtime-client` into `main` (fast-forward; branch is linear ahead).
|
||||
2. **`docker compose down` when convenient** to tear down the dev Postgres container the agent left running.
|
||||
3. **Pause** before dispatching v1.1.7 (Configuration & Email).
|
||||
4. **For the v1.1.7 dispatch prompt**, fold in:
|
||||
- **Fix `dead_letter` handler wiring** (§4 above). Add a call to `list_matching_dead_letter` in `dispatcher::handle_failure` after the row is inserted, enqueue an outbox row per matching trigger with `source_kind: 'dead_letter'` and the appropriate `TriggerEvent::DeadLetter` payload. The recursion-stop rule (handlers can't be dead-lettered) is already implemented; the wiring just isn't connected. Plus a CHANGELOG retroactive note covering v1.1.1–v1.1.6 ("dead_letter triggers were registerable but never fired; this release activates them against existing dead_letters rows, no migration needed").
|
||||
- **Encrypted per-app secrets.** v1.1.7's brief topic. The `app_secrets` table from v1.1.6 is the natural home; the realtime signing key already lives there. v1.1.7's encrypted-secrets work should extend the table (or add a sibling) for general per-app secrets.
|
||||
- **§8 attestation discipline refinement:** require the §8 attestation be sourced from `cargo test --workspace 2>&1 | tail` rather than a hand count (per §5 above).
|
||||
- **The brief-internal-contradiction discipline lesson stuck.** v1.1.7's brief should be walked through for example/spec contradictions before dispatch, same as the v1.1.5 retro lesson the v1.1.6 brief honored. Keep doing this.
|
||||
5. **For the v1.1.8 dispatch prompt** (User Management): the `app_secrets` table + the `RealtimeAuthority` trait shape are ready for v1.1.8 to add `auth_mode = 'session'` as the third subscriber-auth flavor (extending the CHECK constraint, adding a session-token validator alongside the existing HMAC validator behind the unchanged trait).
|
||||
|
||||
Branch is ready for merge. Verdict: **APPROVE**.
|
||||
|
||||
3
clients/typescript/.gitignore
vendored
Normal file
3
clients/typescript/.gitignore
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
node_modules/
|
||||
dist/
|
||||
*.tsbuildinfo
|
||||
111
clients/typescript/README.md
Normal file
111
clients/typescript/README.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# @picloud/client
|
||||
|
||||
TypeScript client for [PiCloud](../../README.md). Three capabilities, all
|
||||
**script-mediated** — there is no direct KV / docs / users access from the
|
||||
browser (the hybrid model, by design):
|
||||
|
||||
1. **Typed HTTP** to dev-defined script endpoints.
|
||||
2. **SSE realtime** subscriptions to externally-subscribable pub/sub topics.
|
||||
3. **Auth-flow helpers** over your own dev-defined login/logout endpoints.
|
||||
|
||||
```ts
|
||||
import { PicloudClient } from '@picloud/client';
|
||||
|
||||
const client = new PicloudClient({
|
||||
baseURL: 'https://api.example.com',
|
||||
getAuthToken: () => localStorage.getItem('auth_token')
|
||||
});
|
||||
|
||||
// Typed HTTP
|
||||
interface CreateUserReq { name: string; email?: string; role: string }
|
||||
interface CreateUserRes { id: string; name: string; created_at: string }
|
||||
const user = await client
|
||||
.endpoint<CreateUserReq, CreateUserRes>('/api/users')
|
||||
.post({ name: 'alice', role: 'admin' });
|
||||
|
||||
// SSE subscription
|
||||
const unsubscribe = client.subscribe('chat-room-123', (event) => {
|
||||
console.log('got event:', event.message);
|
||||
});
|
||||
unsubscribe();
|
||||
|
||||
// Token-gated topic (token obtained from one of YOUR script endpoints,
|
||||
// which calls `pubsub::subscriber_token`)
|
||||
client.subscribe('chat-room-123', cb, { token: 'eyJhbGc...' });
|
||||
|
||||
// Auth helpers (call dev-defined endpoints under the hood)
|
||||
await client.auth.login('alice@example.com', 'password');
|
||||
await client.auth.logout();
|
||||
const token = client.auth.token;
|
||||
```
|
||||
|
||||
## React
|
||||
|
||||
```tsx
|
||||
import { PicloudProvider, useTopic, useEndpoint } from '@picloud/client/react';
|
||||
|
||||
// Wrap your tree once: <PicloudProvider client={client}>…</PicloudProvider>
|
||||
|
||||
function ChatRoom({ roomId }: { roomId: string }) {
|
||||
const messages = useTopic<ChatMessage>(`chat-room-${roomId}`);
|
||||
return <ul>{messages.map((m, i) => <li key={i}>{m.text}</li>)}</ul>;
|
||||
}
|
||||
|
||||
function UserProfile({ id }: { id: string }) {
|
||||
const { data, loading, error } = useEndpoint<UserRes>(`/api/users/${id}`).get();
|
||||
if (loading) return <Spinner />;
|
||||
if (error) return <ErrorView error={error} />;
|
||||
return <div>{data?.name}</div>;
|
||||
}
|
||||
```
|
||||
|
||||
## Svelte
|
||||
|
||||
```ts
|
||||
import { topicStore, endpointStore } from '@picloud/client/svelte';
|
||||
|
||||
const messages = topicStore<ChatMessage>(client, `chat-room-${roomId}`);
|
||||
// $messages is an array that grows as events arrive
|
||||
|
||||
const userQuery = endpointStore<UserRes>(client, `/api/users/${id}`).get();
|
||||
// $userQuery is { data, loading, error }
|
||||
```
|
||||
|
||||
> The Svelte helpers take the `client` explicitly (a store isn't a component,
|
||||
> so there's no React-style context to read).
|
||||
|
||||
## Optional runtime validation (zod / valibot)
|
||||
|
||||
No hard dependency — the adapter is the `{ parse(input): T }` shape. A Zod
|
||||
schema satisfies it directly; wrap Valibot in one line:
|
||||
|
||||
```ts
|
||||
import { z } from 'zod';
|
||||
const UserSchema = z.object({ id: z.string(), name: z.string() });
|
||||
const user = await client.endpoint('/api/users/1').get({ validate: UserSchema });
|
||||
|
||||
// valibot:
|
||||
import * as v from 'valibot';
|
||||
const schema = v.object({ id: v.string() });
|
||||
const adapter = { parse: (i: unknown) => v.parse(schema, i) };
|
||||
```
|
||||
|
||||
## Transport notes
|
||||
|
||||
- SSE is implemented over streaming `fetch` (not native `EventSource`) so the
|
||||
client can refresh an expired token on a 401, send `Last-Event-ID` on resume,
|
||||
and apply its own exponential backoff (1s → 2s → 4s … capped at 30s).
|
||||
- **React Native** has no native `EventSource`, but it also can't stream
|
||||
`fetch` bodies on all engines — if you target RN, supply a streaming-capable
|
||||
`fetch` polyfill via the `fetch` option, or use a `react-native-sse`-based
|
||||
adapter. (Server-side `Last-Event-ID` replay is not implemented in v1.1.6;
|
||||
the client sends the header so it's ready when the server adds replay.)
|
||||
|
||||
## Build / test
|
||||
|
||||
```sh
|
||||
npm install
|
||||
npm run lint # tsc --noEmit (strict)
|
||||
npm run test # vitest
|
||||
npm run build # tsup → dist/ (ESM + CJS + .d.ts)
|
||||
```
|
||||
3580
clients/typescript/package-lock.json
generated
Normal file
3580
clients/typescript/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
61
clients/typescript/package.json
Normal file
61
clients/typescript/package.json
Normal file
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"name": "@picloud/client",
|
||||
"version": "1.0.0",
|
||||
"description": "TypeScript client for PiCloud — typed HTTP to script endpoints, SSE realtime subscriptions, auth-flow helpers, and React/Svelte hooks.",
|
||||
"license": "MIT OR Apache-2.0",
|
||||
"type": "module",
|
||||
"sideEffects": false,
|
||||
"files": [
|
||||
"dist"
|
||||
],
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./dist/index.d.ts",
|
||||
"import": "./dist/index.js",
|
||||
"require": "./dist/index.cjs"
|
||||
},
|
||||
"./react": {
|
||||
"types": "./dist/react/index.d.ts",
|
||||
"import": "./dist/react/index.js",
|
||||
"require": "./dist/react/index.cjs"
|
||||
},
|
||||
"./svelte": {
|
||||
"types": "./dist/svelte/index.d.ts",
|
||||
"import": "./dist/svelte/index.js",
|
||||
"require": "./dist/svelte/index.cjs"
|
||||
}
|
||||
},
|
||||
"main": "./dist/index.cjs",
|
||||
"module": "./dist/index.js",
|
||||
"types": "./dist/index.d.ts",
|
||||
"scripts": {
|
||||
"build": "tsup",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest",
|
||||
"lint": "tsc --noEmit"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"react": ">=17",
|
||||
"svelte": ">=4"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"react": {
|
||||
"optional": true
|
||||
},
|
||||
"svelte": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
"@testing-library/dom": "^10.4.0",
|
||||
"@testing-library/react": "^16.1.0",
|
||||
"@types/react": "^18.3.0",
|
||||
"jsdom": "^25.0.0",
|
||||
"react": "^18.3.0",
|
||||
"react-dom": "^18.3.0",
|
||||
"svelte": "^4.2.0",
|
||||
"tsup": "^8.3.0",
|
||||
"typescript": "^5.6.0",
|
||||
"vitest": "^2.1.0"
|
||||
}
|
||||
}
|
||||
71
clients/typescript/src/auth.ts
Normal file
71
clients/typescript/src/auth.ts
Normal file
@@ -0,0 +1,71 @@
|
||||
import { Endpoint } from './endpoint.js';
|
||||
import type { AuthTokenProvider } from './types.js';
|
||||
|
||||
export interface AuthClientConfig {
|
||||
baseURL: string;
|
||||
fetchImpl: typeof fetch;
|
||||
/** Path of the dev-defined login endpoint (default `/api/auth/login`). */
|
||||
loginPath?: string;
|
||||
/** Path of the dev-defined logout endpoint (default `/api/auth/logout`). */
|
||||
logoutPath?: string;
|
||||
/** Called whenever the stored token changes (e.g. to persist it). */
|
||||
onToken?: (token: string | null) => void;
|
||||
}
|
||||
|
||||
interface LoginResponse {
|
||||
token?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Auth-flow helpers. These call **dev-defined** endpoints under the hood
|
||||
* (the script layer owns the actual auth); the lib only standardizes the
|
||||
* dance + in-memory token storage. There is no built-in identity model —
|
||||
* `login` POSTs credentials and stores whatever `token` comes back.
|
||||
*/
|
||||
export class AuthClient {
|
||||
private current: string | null = null;
|
||||
|
||||
constructor(private readonly cfg: AuthClientConfig) {}
|
||||
|
||||
/** The current bearer token, or null. */
|
||||
get token(): string | null {
|
||||
return this.current;
|
||||
}
|
||||
|
||||
/** Suitable as `PicloudClientOptions.getAuthToken`. */
|
||||
readonly provider: AuthTokenProvider = () => this.current;
|
||||
|
||||
/** POST credentials to the login endpoint; store the returned token. */
|
||||
async login(email: string, password: string): Promise<string | null> {
|
||||
const ep = new Endpoint<{ email: string; password: string }, LoginResponse>({
|
||||
baseURL: this.cfg.baseURL,
|
||||
path: this.cfg.loginPath ?? '/api/auth/login',
|
||||
fetchImpl: this.cfg.fetchImpl
|
||||
});
|
||||
const res = await ep.post({ email, password });
|
||||
this.setToken(typeof res?.token === 'string' ? res.token : null);
|
||||
return this.current;
|
||||
}
|
||||
|
||||
/** POST to the logout endpoint (best-effort) and clear the token. */
|
||||
async logout(): Promise<void> {
|
||||
const ep = new Endpoint<undefined, unknown>({
|
||||
baseURL: this.cfg.baseURL,
|
||||
path: this.cfg.logoutPath ?? '/api/auth/logout',
|
||||
// Send the current token so the script can invalidate the session.
|
||||
getAuthToken: () => this.current,
|
||||
fetchImpl: this.cfg.fetchImpl
|
||||
});
|
||||
try {
|
||||
await ep.post();
|
||||
} finally {
|
||||
this.setToken(null);
|
||||
}
|
||||
}
|
||||
|
||||
/** Manually set (or clear) the token — e.g. restoring from storage. */
|
||||
setToken(token: string | null): void {
|
||||
this.current = token;
|
||||
this.cfg.onToken?.(token);
|
||||
}
|
||||
}
|
||||
61
clients/typescript/src/client.ts
Normal file
61
clients/typescript/src/client.ts
Normal file
@@ -0,0 +1,61 @@
|
||||
import { AuthClient } from './auth.js';
|
||||
import { Endpoint } from './endpoint.js';
|
||||
import { subscribeTopic } from './subscribe.js';
|
||||
import type {
|
||||
PicloudClientOptions,
|
||||
RealtimeEvent,
|
||||
SubscribeOptions,
|
||||
Unsubscribe
|
||||
} from './types.js';
|
||||
|
||||
/**
|
||||
* The PiCloud frontend client. Three capabilities, all script-mediated
|
||||
* (the hybrid model — no direct KV/docs/users access from the browser):
|
||||
*
|
||||
* - `endpoint<Req, Res>(path)` — typed HTTP to a dev-defined route.
|
||||
* - `subscribe(topic, cb, opts?)` — SSE realtime subscription.
|
||||
* - `auth` — login/logout/token helpers over dev-defined endpoints.
|
||||
*/
|
||||
export class PicloudClient {
|
||||
readonly auth: AuthClient;
|
||||
private readonly baseURL: string;
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
private readonly getAuthToken: PicloudClientOptions['getAuthToken'];
|
||||
|
||||
constructor(opts: PicloudClientOptions) {
|
||||
if (!opts.baseURL) throw new Error('PicloudClient: baseURL is required');
|
||||
this.baseURL = opts.baseURL;
|
||||
const f = opts.fetch ?? globalThis.fetch;
|
||||
if (typeof f !== 'function') {
|
||||
throw new Error('PicloudClient: no fetch available — pass options.fetch');
|
||||
}
|
||||
// Bind to avoid "Illegal invocation" when calling a detached global.
|
||||
this.fetchImpl = f.bind(globalThis);
|
||||
this.getAuthToken = opts.getAuthToken;
|
||||
this.auth = new AuthClient({ baseURL: this.baseURL, fetchImpl: this.fetchImpl });
|
||||
}
|
||||
|
||||
/** A typed handle to a dev-defined endpoint. */
|
||||
endpoint<Req = unknown, Res = unknown>(path: string): Endpoint<Req, Res> {
|
||||
return new Endpoint<Req, Res>({
|
||||
baseURL: this.baseURL,
|
||||
path,
|
||||
getAuthToken: this.getAuthToken,
|
||||
fetchImpl: this.fetchImpl
|
||||
});
|
||||
}
|
||||
|
||||
/** Subscribe to a realtime topic. Returns an unsubscribe function. */
|
||||
subscribe<T = unknown>(
|
||||
topic: string,
|
||||
onMessage: (event: RealtimeEvent<T>) => void,
|
||||
opts?: SubscribeOptions<T>
|
||||
): Unsubscribe {
|
||||
return subscribeTopic<T>(
|
||||
{ baseURL: this.baseURL, fetchImpl: this.fetchImpl },
|
||||
topic,
|
||||
onMessage,
|
||||
opts
|
||||
);
|
||||
}
|
||||
}
|
||||
106
clients/typescript/src/endpoint.ts
Normal file
106
clients/typescript/src/endpoint.ts
Normal file
@@ -0,0 +1,106 @@
|
||||
import { PicloudHttpError, type AuthTokenProvider, type Validator } from './types.js';
|
||||
|
||||
type Method = 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE';
|
||||
|
||||
export interface EndpointConfig {
|
||||
baseURL: string;
|
||||
path: string;
|
||||
getAuthToken?: AuthTokenProvider;
|
||||
fetchImpl: typeof fetch;
|
||||
}
|
||||
|
||||
export interface RequestOptions<Res> {
|
||||
/** Extra headers merged over the defaults. */
|
||||
headers?: Record<string, string>;
|
||||
/** Optional runtime validation of the parsed response. */
|
||||
validate?: Validator<Res>;
|
||||
/** AbortSignal to cancel the request. */
|
||||
signal?: AbortSignal;
|
||||
}
|
||||
|
||||
/**
|
||||
* Typed HTTP to a dev-defined script endpoint. Auth header injection +
|
||||
* structured errors; the request/response types are caller-supplied
|
||||
* generics (`endpoint<Req, Res>('/path')`). No service access — every
|
||||
* call hits a route a script binds (the hybrid model).
|
||||
*/
|
||||
export class Endpoint<Req = unknown, Res = unknown> {
|
||||
constructor(private readonly cfg: EndpointConfig) {}
|
||||
|
||||
get(opts?: RequestOptions<Res>): Promise<Res> {
|
||||
return this.send('GET', undefined, opts);
|
||||
}
|
||||
|
||||
post(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
|
||||
return this.send('POST', body, opts);
|
||||
}
|
||||
|
||||
put(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
|
||||
return this.send('PUT', body, opts);
|
||||
}
|
||||
|
||||
patch(body?: Req, opts?: RequestOptions<Res>): Promise<Res> {
|
||||
return this.send('PATCH', body, opts);
|
||||
}
|
||||
|
||||
delete(opts?: RequestOptions<Res>): Promise<Res> {
|
||||
return this.send('DELETE', undefined, opts);
|
||||
}
|
||||
|
||||
private async send(method: Method, body: Req | undefined, opts?: RequestOptions<Res>): Promise<Res> {
|
||||
const headers: Record<string, string> = {
|
||||
Accept: 'application/json',
|
||||
...(opts?.headers ?? {})
|
||||
};
|
||||
if (body !== undefined) {
|
||||
headers['Content-Type'] ??= 'application/json';
|
||||
}
|
||||
const token = this.cfg.getAuthToken ? await this.cfg.getAuthToken() : undefined;
|
||||
if (token) {
|
||||
headers['Authorization'] ??= `Bearer ${token}`;
|
||||
}
|
||||
|
||||
const url = joinUrl(this.cfg.baseURL, this.cfg.path);
|
||||
const init: RequestInit = { method, headers };
|
||||
if (body !== undefined) {
|
||||
init.body = JSON.stringify(body);
|
||||
}
|
||||
if (opts?.signal) {
|
||||
init.signal = opts.signal;
|
||||
}
|
||||
|
||||
const res = await this.cfg.fetchImpl(url, init);
|
||||
const parsed = await parseBody(res);
|
||||
if (!res.ok) {
|
||||
const message =
|
||||
(isRecord(parsed) && typeof parsed['error'] === 'string' && parsed['error']) ||
|
||||
`${method} ${this.cfg.path} failed with ${res.status}`;
|
||||
throw new PicloudHttpError(res.status, message, parsed);
|
||||
}
|
||||
return opts?.validate ? opts.validate.parse(parsed) : (parsed as Res);
|
||||
}
|
||||
}
|
||||
|
||||
async function parseBody(res: Response): Promise<unknown> {
|
||||
const text = await res.text();
|
||||
if (text.length === 0) return null;
|
||||
const ct = res.headers.get('content-type') ?? '';
|
||||
if (ct.includes('application/json')) {
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
return text;
|
||||
}
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
function isRecord(v: unknown): v is Record<string, unknown> {
|
||||
return typeof v === 'object' && v !== null;
|
||||
}
|
||||
|
||||
export function joinUrl(base: string, path: string): string {
|
||||
const b = base.endsWith('/') ? base.slice(0, -1) : base;
|
||||
const p = path.startsWith('/') ? path : `/${path}`;
|
||||
return `${b}${p}`;
|
||||
}
|
||||
14
clients/typescript/src/index.ts
Normal file
14
clients/typescript/src/index.ts
Normal file
@@ -0,0 +1,14 @@
|
||||
export { PicloudClient } from './client.js';
|
||||
export { Endpoint } from './endpoint.js';
|
||||
export { AuthClient } from './auth.js';
|
||||
export { subscribeTopic } from './subscribe.js';
|
||||
export {
|
||||
PicloudHttpError,
|
||||
type PicloudClientOptions,
|
||||
type AuthTokenProvider,
|
||||
type RealtimeEvent,
|
||||
type SubscribeOptions,
|
||||
type Unsubscribe,
|
||||
type Validator
|
||||
} from './types.js';
|
||||
export type { RequestOptions } from './endpoint.js';
|
||||
101
clients/typescript/src/react/index.ts
Normal file
101
clients/typescript/src/react/index.ts
Normal file
@@ -0,0 +1,101 @@
|
||||
import {
|
||||
createContext,
|
||||
createElement,
|
||||
useContext,
|
||||
useEffect,
|
||||
useState,
|
||||
type ReactNode
|
||||
} from 'react';
|
||||
|
||||
import type { PicloudClient } from '../client.js';
|
||||
import type { SubscribeOptions } from '../types.js';
|
||||
|
||||
const PicloudContext = createContext<PicloudClient | null>(null);
|
||||
|
||||
export interface PicloudProviderProps {
|
||||
client: PicloudClient;
|
||||
children?: ReactNode;
|
||||
}
|
||||
|
||||
/** Provides a `PicloudClient` to `useTopic` / `useEndpoint`. */
|
||||
export function PicloudProvider(props: PicloudProviderProps) {
|
||||
return createElement(PicloudContext.Provider, { value: props.client }, props.children);
|
||||
}
|
||||
|
||||
/** The client from the nearest `PicloudProvider`. Throws if absent. */
|
||||
export function usePicloud(): PicloudClient {
|
||||
const client = useContext(PicloudContext);
|
||||
if (!client) {
|
||||
throw new Error('usePicloud: wrap your tree in <PicloudProvider client={...}>');
|
||||
}
|
||||
return client;
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe to a realtime topic; returns the accumulated messages in
|
||||
* arrival order. Re-subscribes when `topic` changes; unsubscribes on
|
||||
* unmount.
|
||||
*/
|
||||
export function useTopic<T = unknown>(topic: string, opts?: SubscribeOptions<T>): T[] {
|
||||
const client = usePicloud();
|
||||
const [messages, setMessages] = useState<T[]>([]);
|
||||
useEffect(() => {
|
||||
setMessages([]);
|
||||
const unsubscribe = client.subscribe<T>(
|
||||
topic,
|
||||
(event) => setMessages((prev) => [...prev, event.message]),
|
||||
opts
|
||||
);
|
||||
return () => unsubscribe();
|
||||
// `opts` is intentionally excluded: a new object literal each render
|
||||
// would otherwise resubscribe every render.
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [client, topic]);
|
||||
return messages;
|
||||
}
|
||||
|
||||
export interface QueryState<T> {
|
||||
data: T | null;
|
||||
loading: boolean;
|
||||
error: unknown;
|
||||
}
|
||||
|
||||
export interface EndpointHook<Req, Res> {
|
||||
get: () => QueryState<Res>;
|
||||
post: (body?: Req) => QueryState<Res>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Typed endpoint hook. `useEndpoint<Res>(path).get()` fires a GET and
|
||||
* returns `{ data, loading, error }`, re-running when `path` changes.
|
||||
* `.post(body)` is the mutation variant (auto-fires once per mount).
|
||||
*/
|
||||
export function useEndpoint<Res = unknown, Req = unknown>(path: string): EndpointHook<Req, Res> {
|
||||
const client = usePicloud();
|
||||
return {
|
||||
get: () => useResource<Res>(() => client.endpoint<Req, Res>(path).get(), path, 'GET'),
|
||||
post: (body?: Req) =>
|
||||
useResource<Res>(() => client.endpoint<Req, Res>(path).post(body), path, 'POST')
|
||||
};
|
||||
}
|
||||
|
||||
function useResource<Res>(run: () => Promise<Res>, key: string, method: string): QueryState<Res> {
|
||||
const [state, setState] = useState<QueryState<Res>>({
|
||||
data: null,
|
||||
loading: true,
|
||||
error: null
|
||||
});
|
||||
useEffect(() => {
|
||||
let active = true;
|
||||
setState({ data: null, loading: true, error: null });
|
||||
run()
|
||||
.then((data) => active && setState({ data, loading: false, error: null }))
|
||||
.catch((error) => active && setState({ data: null, loading: false, error }));
|
||||
return () => {
|
||||
active = false;
|
||||
};
|
||||
// `run` is recreated each render; key it on path + method instead.
|
||||
// eslint-disable-next-line react-hooks/exhaustive-deps
|
||||
}, [key, method]);
|
||||
return state;
|
||||
}
|
||||
194
clients/typescript/src/subscribe.ts
Normal file
194
clients/typescript/src/subscribe.ts
Normal file
@@ -0,0 +1,194 @@
|
||||
import { joinUrl } from './endpoint.js';
|
||||
import type { RealtimeEvent, SubscribeOptions, Unsubscribe } from './types.js';
|
||||
|
||||
interface SubscribeConfig {
|
||||
baseURL: string;
|
||||
fetchImpl: typeof fetch;
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe to an app pub/sub topic over SSE.
|
||||
*
|
||||
* Implemented over streaming `fetch` (not native `EventSource`) so the
|
||||
* lib can: detect a 401 on (re)connect and refresh the token, send a
|
||||
* `Last-Event-ID` header on resume, and apply its own exponential
|
||||
* backoff. See HANDBACK for the rationale. Returns an unsubscribe
|
||||
* function that aborts the connection and stops reconnecting.
|
||||
*/
|
||||
export function subscribeTopic<T = unknown>(
|
||||
cfg: SubscribeConfig,
|
||||
topic: string,
|
||||
onMessage: (event: RealtimeEvent<T>) => void,
|
||||
opts: SubscribeOptions<T> = {}
|
||||
): Unsubscribe {
|
||||
const baseBackoff = opts.baseBackoffMs ?? 1_000;
|
||||
const maxBackoff = opts.maxBackoffMs ?? 30_000;
|
||||
let token = opts.token;
|
||||
let stopped = false;
|
||||
let attempt = 0;
|
||||
let lastEventId: string | undefined;
|
||||
let controller: AbortController | null = null;
|
||||
let backoffTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
|
||||
const stop = () => {
|
||||
stopped = true;
|
||||
if (backoffTimer) clearTimeout(backoffTimer);
|
||||
controller?.abort();
|
||||
};
|
||||
|
||||
const scheduleReconnect = () => {
|
||||
if (stopped) return;
|
||||
// Exponential backoff: base, 2x, 4x… capped at maxBackoff.
|
||||
const delay = Math.min(maxBackoff, baseBackoff * 2 ** attempt);
|
||||
attempt += 1;
|
||||
backoffTimer = setTimeout(() => void connect(), delay);
|
||||
};
|
||||
|
||||
const connect = async (): Promise<void> => {
|
||||
if (stopped) return;
|
||||
controller = new AbortController();
|
||||
const url = buildUrl(cfg.baseURL, topic, token);
|
||||
const headers: Record<string, string> = { Accept: 'text/event-stream' };
|
||||
if (lastEventId) headers['Last-Event-ID'] = lastEventId;
|
||||
|
||||
let res: Response;
|
||||
try {
|
||||
res = await cfg.fetchImpl(url, { headers, signal: controller.signal });
|
||||
} catch (err) {
|
||||
if (stopped || isAbort(err)) return;
|
||||
scheduleReconnect();
|
||||
return;
|
||||
}
|
||||
|
||||
if (res.status === 401) {
|
||||
// Token expired / rejected — try to refresh, else give up.
|
||||
const fresh = opts.onTokenExpired ? await opts.onTokenExpired() : null;
|
||||
if (fresh) {
|
||||
token = fresh;
|
||||
attempt = 0; // fresh credential → reconnect immediately
|
||||
void connect();
|
||||
} else {
|
||||
opts.onError?.(new Error('realtime subscribe unauthorized (401)'));
|
||||
stop();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
if (!res.ok || !res.body) {
|
||||
if (!stopped) scheduleReconnect();
|
||||
return;
|
||||
}
|
||||
|
||||
// Connected — reset backoff and stream frames until the body ends.
|
||||
attempt = 0;
|
||||
try {
|
||||
await readStream(res.body, (frame) => {
|
||||
if (frame.id !== undefined) lastEventId = frame.id;
|
||||
if (frame.data === undefined) return; // comment / heartbeat
|
||||
const parsed = parseEvent<T>(frame.data, opts);
|
||||
if (parsed) onMessage(parsed);
|
||||
});
|
||||
} catch (err) {
|
||||
if (stopped || isAbort(err)) return;
|
||||
}
|
||||
// Stream ended (server closed, e.g. topic deleted) → reconnect.
|
||||
if (!stopped) scheduleReconnect();
|
||||
};
|
||||
|
||||
void connect();
|
||||
return stop;
|
||||
}
|
||||
|
||||
function buildUrl(baseURL: string, topic: string, token?: string): string {
|
||||
const url = joinUrl(baseURL, `/realtime/topics/${encodeURIComponent(topic)}`);
|
||||
// EventSource can't set headers, so the token rides in the query
|
||||
// string — the same path a raw EventSource would use.
|
||||
return token ? `${url}?token=${encodeURIComponent(token)}` : url;
|
||||
}
|
||||
|
||||
function parseEvent<T>(data: string, opts: SubscribeOptions<T>): RealtimeEvent<T> | null {
|
||||
let json: unknown;
|
||||
try {
|
||||
json = JSON.parse(data);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!isRealtimeShape(json)) return null;
|
||||
const message = opts.validate ? opts.validate.parse(json.message) : (json.message as T);
|
||||
return { topic: json.topic, message, published_at: json.published_at };
|
||||
}
|
||||
|
||||
function isRealtimeShape(v: unknown): v is RealtimeEvent<unknown> {
|
||||
return (
|
||||
typeof v === 'object' &&
|
||||
v !== null &&
|
||||
typeof (v as Record<string, unknown>)['topic'] === 'string' &&
|
||||
typeof (v as Record<string, unknown>)['published_at'] === 'string' &&
|
||||
'message' in (v as Record<string, unknown>)
|
||||
);
|
||||
}
|
||||
|
||||
interface SseFrame {
|
||||
data?: string;
|
||||
id?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read an SSE response body, invoking `onFrame` per event. Minimal
|
||||
* parser: accumulates `data:` lines (joined by `\n`) and `id:` until a
|
||||
* blank line dispatches the frame. Lines starting with `:` are comments
|
||||
* (heartbeats) — surfaced as a frame with no `data` so the id can still
|
||||
* advance.
|
||||
*/
|
||||
async function readStream(
|
||||
body: ReadableStream<Uint8Array>,
|
||||
onFrame: (frame: SseFrame) => void
|
||||
): Promise<void> {
|
||||
const reader = body.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
let buffer = '';
|
||||
let dataLines: string[] = [];
|
||||
let id: string | undefined;
|
||||
let sawComment = false;
|
||||
|
||||
const dispatch = () => {
|
||||
if (dataLines.length > 0) {
|
||||
onFrame({ data: dataLines.join('\n'), id });
|
||||
} else if (sawComment) {
|
||||
onFrame({ id });
|
||||
}
|
||||
dataLines = [];
|
||||
sawComment = false;
|
||||
};
|
||||
|
||||
for (;;) {
|
||||
const { value, done } = await reader.read();
|
||||
if (done) break;
|
||||
buffer += decoder.decode(value, { stream: true });
|
||||
let nl: number;
|
||||
while ((nl = buffer.indexOf('\n')) >= 0) {
|
||||
const line = buffer.slice(0, nl).replace(/\r$/, '');
|
||||
buffer = buffer.slice(nl + 1);
|
||||
if (line === '') {
|
||||
dispatch();
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith(':')) {
|
||||
sawComment = true;
|
||||
continue;
|
||||
}
|
||||
const colon = line.indexOf(':');
|
||||
const field = colon === -1 ? line : line.slice(0, colon);
|
||||
const rawVal = colon === -1 ? '' : line.slice(colon + 1);
|
||||
const val = rawVal.startsWith(' ') ? rawVal.slice(1) : rawVal;
|
||||
if (field === 'data') dataLines.push(val);
|
||||
else if (field === 'id') id = val;
|
||||
}
|
||||
}
|
||||
// Flush a trailing frame if the stream ended without a blank line.
|
||||
dispatch();
|
||||
}
|
||||
|
||||
function isAbort(err: unknown): boolean {
|
||||
return typeof err === 'object' && err !== null && (err as { name?: string }).name === 'AbortError';
|
||||
}
|
||||
72
clients/typescript/src/svelte/index.ts
Normal file
72
clients/typescript/src/svelte/index.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
import { readable, type Readable } from 'svelte/store';
|
||||
|
||||
import type { PicloudClient } from '../client.js';
|
||||
import type { SubscribeOptions } from '../types.js';
|
||||
|
||||
/**
|
||||
* A Svelte store of realtime messages for a topic. `$messages` is an
|
||||
* array that grows as events arrive. The SSE connection opens on the
|
||||
* first subscriber and closes when the last unsubscribes (standard
|
||||
* `readable` lifecycle).
|
||||
*
|
||||
* The client is passed explicitly (Svelte stores aren't components, so
|
||||
* there's no React-style context to read). See HANDBACK §7.
|
||||
*/
|
||||
export function topicStore<T = unknown>(
|
||||
client: PicloudClient,
|
||||
topic: string,
|
||||
opts?: SubscribeOptions<T>
|
||||
): Readable<T[]> {
|
||||
return readable<T[]>([], (set) => {
|
||||
let items: T[] = [];
|
||||
const unsubscribe = client.subscribe<T>(
|
||||
topic,
|
||||
(event) => {
|
||||
items = [...items, event.message];
|
||||
set(items);
|
||||
},
|
||||
opts
|
||||
);
|
||||
return () => unsubscribe();
|
||||
});
|
||||
}
|
||||
|
||||
export interface QueryState<T> {
|
||||
data: T | null;
|
||||
loading: boolean;
|
||||
error: unknown;
|
||||
}
|
||||
|
||||
export interface EndpointStore<Req, Res> {
|
||||
get: () => Readable<QueryState<Res>>;
|
||||
post: (body?: Req) => Readable<QueryState<Res>>;
|
||||
}
|
||||
|
||||
/**
|
||||
* A Svelte store wrapper over a typed endpoint. `$query` is
|
||||
* `{ data, loading, error }`. The request fires when the store gains its
|
||||
* first subscriber.
|
||||
*/
|
||||
export function endpointStore<Res = unknown, Req = unknown>(
|
||||
client: PicloudClient,
|
||||
path: string
|
||||
): EndpointStore<Req, Res> {
|
||||
const run = (exec: () => Promise<Res>): Readable<QueryState<Res>> =>
|
||||
readable<QueryState<Res>>({ data: null, loading: true, error: null }, (set) => {
|
||||
let active = true;
|
||||
exec()
|
||||
.then((data) => {
|
||||
if (active) set({ data, loading: false, error: null });
|
||||
})
|
||||
.catch((error) => {
|
||||
if (active) set({ data: null, loading: false, error });
|
||||
});
|
||||
return () => {
|
||||
active = false;
|
||||
};
|
||||
});
|
||||
return {
|
||||
get: () => run(() => client.endpoint<Req, Res>(path).get()),
|
||||
post: (body?: Req) => run(() => client.endpoint<Req, Res>(path).post(body))
|
||||
};
|
||||
}
|
||||
73
clients/typescript/src/types.ts
Normal file
73
clients/typescript/src/types.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
// Shared types for @picloud/client.
|
||||
|
||||
/** Returns the current bearer token (or null) before each HTTP request. */
|
||||
export type AuthTokenProvider = () => string | null | undefined | Promise<string | null | undefined>;
|
||||
|
||||
export interface PicloudClientOptions {
|
||||
/** Base URL of the PiCloud deployment, e.g. `https://api.example.com`. */
|
||||
baseURL: string;
|
||||
/**
|
||||
* Optional: returns the current bearer token, called before each
|
||||
* request. The client doesn't manage tokens — it just sends them.
|
||||
*/
|
||||
getAuthToken?: AuthTokenProvider;
|
||||
/**
|
||||
* Optional fetch implementation (defaults to the global `fetch`).
|
||||
* Injected mainly for tests / non-browser runtimes.
|
||||
*/
|
||||
fetch?: typeof fetch;
|
||||
}
|
||||
|
||||
/** A realtime event as delivered over SSE. */
|
||||
export interface RealtimeEvent<T = unknown> {
|
||||
topic: string;
|
||||
message: T;
|
||||
published_at: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal validator shape for the optional runtime-validation adapter.
|
||||
* A Zod schema satisfies this directly (`schema.parse`); for Valibot,
|
||||
* wrap it: `{ parse: (i) => v.parse(schema, i) }`. No hard dep on either.
|
||||
*/
|
||||
export interface Validator<T> {
|
||||
parse: (input: unknown) => T;
|
||||
}
|
||||
|
||||
/** Thrown when an endpoint call returns a non-2xx status. */
|
||||
export class PicloudHttpError extends Error {
|
||||
readonly status: number;
|
||||
readonly body: unknown;
|
||||
constructor(status: number, message: string, body: unknown) {
|
||||
super(message);
|
||||
this.name = 'PicloudHttpError';
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
}
|
||||
}
|
||||
|
||||
export interface SubscribeOptions<T = unknown> {
|
||||
/**
|
||||
* Subscriber token for `auth_mode = 'token'` topics. Obtained from one
|
||||
* of your app's script endpoints (which calls
|
||||
* `pubsub::subscriber_token`). Sent as `?token=` (EventSource-parity).
|
||||
*/
|
||||
token?: string;
|
||||
/**
|
||||
* Called when a (re)connect is rejected with 401 — typically an
|
||||
* expired token. Return a fresh token to retry immediately, or
|
||||
* null/undefined to stop and surface the error.
|
||||
*/
|
||||
onTokenExpired?: () => string | null | undefined | Promise<string | null | undefined>;
|
||||
/** Called on a terminal error (after retries are exhausted or aborted). */
|
||||
onError?: (err: unknown) => void;
|
||||
/** Optional runtime validation of each event's `message`. */
|
||||
validate?: Validator<T>;
|
||||
/** Max reconnect backoff in ms (default 30_000). */
|
||||
maxBackoffMs?: number;
|
||||
/** Base reconnect backoff in ms (default 1_000). */
|
||||
baseBackoffMs?: number;
|
||||
}
|
||||
|
||||
/** Cancels a realtime subscription. */
|
||||
export type Unsubscribe = () => void;
|
||||
41
clients/typescript/tests/auth.test.ts
Normal file
41
clients/typescript/tests/auth.test.ts
Normal file
@@ -0,0 +1,41 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { PicloudClient } from '../src/index.js';
|
||||
import { jsonResponse, lastUrl, type FetchArgs } from './helpers.js';
|
||||
|
||||
describe('auth', () => {
|
||||
it('login POSTs credentials and stores the returned token', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ token: 'session-abc' })
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
|
||||
const token = await client.auth.login('alice@example.com', 'pw');
|
||||
expect(token).toBe('session-abc');
|
||||
expect(client.auth.token).toBe('session-abc');
|
||||
expect(lastUrl(fetchMock)).toBe('https://api.test/api/auth/login');
|
||||
|
||||
const init = fetchMock.mock.calls[0]?.[1];
|
||||
expect(JSON.parse(String(init?.body))).toEqual({
|
||||
email: 'alice@example.com',
|
||||
password: 'pw'
|
||||
});
|
||||
});
|
||||
|
||||
it('logout clears the stored token', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => jsonResponse({}));
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
client.auth.setToken('existing');
|
||||
await client.auth.logout();
|
||||
expect(client.auth.token).toBeNull();
|
||||
});
|
||||
|
||||
it('provider returns the current token for getAuthToken wiring', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ token: 't' })
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
await client.auth.login('a@b.c', 'pw');
|
||||
expect(client.auth.provider()).toBe('t');
|
||||
});
|
||||
});
|
||||
82
clients/typescript/tests/endpoint.test.ts
Normal file
82
clients/typescript/tests/endpoint.test.ts
Normal file
@@ -0,0 +1,82 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { PicloudClient, PicloudHttpError } from '../src/index.js';
|
||||
import { headerOf, jsonResponse, lastInit, lastUrl, type FetchArgs } from './helpers.js';
|
||||
|
||||
describe('endpoint', () => {
|
||||
it('post round-trips a typed request/response', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ id: '1', name: 'alice', created_at: 'now' }, 201)
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
|
||||
interface Req {
|
||||
name: string;
|
||||
role: string;
|
||||
}
|
||||
interface Res {
|
||||
id: string;
|
||||
name: string;
|
||||
created_at: string;
|
||||
}
|
||||
const res = await client.endpoint<Req, Res>('/api/users').post({ name: 'alice', role: 'admin' });
|
||||
|
||||
expect(res).toEqual({ id: '1', name: 'alice', created_at: 'now' });
|
||||
expect(lastUrl(fetchMock)).toBe('https://api.test/api/users');
|
||||
const init = lastInit(fetchMock);
|
||||
expect(init.method).toBe('POST');
|
||||
expect(JSON.parse(String(init.body))).toEqual({ name: 'alice', role: 'admin' });
|
||||
expect(headerOf(init, 'Content-Type')).toBe('application/json');
|
||||
});
|
||||
|
||||
it('get round-trips', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ name: 'bob' })
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
const res = await client.endpoint<unknown, { name: string }>('/api/users/1').get();
|
||||
expect(res.name).toBe('bob');
|
||||
expect(lastInit(fetchMock).method).toBe('GET');
|
||||
});
|
||||
|
||||
it('injects the auth token from getAuthToken', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => jsonResponse({ ok: true }));
|
||||
const client = new PicloudClient({
|
||||
baseURL: 'https://api.test',
|
||||
fetch: fetchMock,
|
||||
getAuthToken: () => 'tok-123'
|
||||
});
|
||||
await client.endpoint('/api/me').get();
|
||||
expect(headerOf(lastInit(fetchMock), 'Authorization')).toBe('Bearer tok-123');
|
||||
});
|
||||
|
||||
it('throws PicloudHttpError with status + body on non-2xx', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ error: 'bad input' }, 422)
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
const err = await client
|
||||
.endpoint('/api/x')
|
||||
.get()
|
||||
.catch((e: unknown) => e);
|
||||
expect(err).toBeInstanceOf(PicloudHttpError);
|
||||
expect((err as PicloudHttpError).status).toBe(422);
|
||||
expect((err as PicloudHttpError).message).toBe('bad input');
|
||||
});
|
||||
|
||||
it('applies an optional validator to the response', async () => {
|
||||
const fetchMock = vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) =>
|
||||
jsonResponse({ id: 7 })
|
||||
);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
const validator = {
|
||||
parse: (input: unknown) => {
|
||||
const r = input as { id: number };
|
||||
if (typeof r.id !== 'number') throw new Error('bad');
|
||||
return r;
|
||||
}
|
||||
};
|
||||
const res = await client.endpoint<unknown, { id: number }>('/api/x').get({ validate: validator });
|
||||
expect(res.id).toBe(7);
|
||||
});
|
||||
});
|
||||
54
clients/typescript/tests/helpers.ts
Normal file
54
clients/typescript/tests/helpers.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
// Test helpers: build JSON + SSE Response objects and a typed fetch mock.
|
||||
|
||||
export function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { 'content-type': 'application/json' }
|
||||
});
|
||||
}
|
||||
|
||||
export function emptyResponse(status = 200): Response {
|
||||
return new Response(null, { status });
|
||||
}
|
||||
|
||||
/** Build a text/event-stream Response from raw SSE frame strings. */
|
||||
export function sseResponse(frames: string[], status = 200): Response {
|
||||
const encoder = new TextEncoder();
|
||||
const stream = new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
for (const frame of frames) controller.enqueue(encoder.encode(frame));
|
||||
controller.close();
|
||||
}
|
||||
});
|
||||
return new Response(stream, {
|
||||
status,
|
||||
headers: { 'content-type': 'text/event-stream' }
|
||||
});
|
||||
}
|
||||
|
||||
/** One SSE `data:` event frame for a realtime payload. */
|
||||
export function dataFrame(topic: string, message: unknown, publishedAt = '2026-06-04T00:00:00Z'): string {
|
||||
const payload = JSON.stringify({ topic, message, published_at: publishedAt });
|
||||
return `data: ${payload}\n\n`;
|
||||
}
|
||||
|
||||
export type FetchArgs = [string | URL | Request, RequestInit?];
|
||||
|
||||
type MockLike = { mock: { calls: ReadonlyArray<ReadonlyArray<unknown>> } };
|
||||
|
||||
export function lastInit(mock: MockLike, i = 0): RequestInit {
|
||||
const call = mock.mock.calls[i];
|
||||
if (!call) throw new Error(`no fetch call at index ${i}`);
|
||||
return (call[1] as RequestInit | undefined) ?? {};
|
||||
}
|
||||
|
||||
export function lastUrl(mock: MockLike, i = 0): string {
|
||||
const call = mock.mock.calls[i];
|
||||
if (!call) throw new Error(`no fetch call at index ${i}`);
|
||||
return String(call[0]);
|
||||
}
|
||||
|
||||
export function headerOf(init: RequestInit, name: string): string | undefined {
|
||||
const h = init.headers as Record<string, string> | undefined;
|
||||
return h?.[name];
|
||||
}
|
||||
41
clients/typescript/tests/react.test.tsx
Normal file
41
clients/typescript/tests/react.test.tsx
Normal file
@@ -0,0 +1,41 @@
|
||||
import { act, renderHook } from '@testing-library/react';
|
||||
import type { ReactNode } from 'react';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { PicloudClient, RealtimeEvent, Unsubscribe } from '../src/index.js';
|
||||
import { PicloudProvider, useTopic } from '../src/react/index.js';
|
||||
|
||||
type Cb = (e: RealtimeEvent<unknown>) => void;
|
||||
|
||||
function fakeClient() {
|
||||
const unsubscribe = vi.fn();
|
||||
let captured: Cb | null = null;
|
||||
const subscribe = vi.fn(
|
||||
(_topic: string, cb: Cb): Unsubscribe => {
|
||||
captured = cb;
|
||||
return unsubscribe as unknown as Unsubscribe;
|
||||
}
|
||||
);
|
||||
const client = { subscribe } as unknown as PicloudClient;
|
||||
return { client, subscribe, unsubscribe, emit: (e: RealtimeEvent<unknown>) => captured?.(e) };
|
||||
}
|
||||
|
||||
describe('react useTopic', () => {
|
||||
it('subscribes on mount, accumulates messages, unsubscribes on unmount', () => {
|
||||
const { client, subscribe, unsubscribe, emit } = fakeClient();
|
||||
const wrapper = ({ children }: { children: ReactNode }) =>
|
||||
PicloudProvider({ client, children });
|
||||
|
||||
const { result, unmount } = renderHook(() => useTopic<{ n: number }>('chat'), { wrapper });
|
||||
|
||||
expect(subscribe).toHaveBeenCalledTimes(1);
|
||||
expect(result.current).toEqual([]);
|
||||
|
||||
act(() => emit({ topic: 'chat', message: { n: 1 }, published_at: 't' }));
|
||||
act(() => emit({ topic: 'chat', message: { n: 2 }, published_at: 't' }));
|
||||
expect(result.current).toEqual([{ n: 1 }, { n: 2 }]);
|
||||
|
||||
unmount();
|
||||
expect(unsubscribe).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
99
clients/typescript/tests/subscribe.test.ts
Normal file
99
clients/typescript/tests/subscribe.test.ts
Normal file
@@ -0,0 +1,99 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import { PicloudClient, type RealtimeEvent } from '../src/index.js';
|
||||
import { dataFrame, emptyResponse, lastUrl, sseResponse, type FetchArgs } from './helpers.js';
|
||||
|
||||
/** A fetch mock that plays through a queue of response factories. */
|
||||
function queuedFetch(responders: Array<() => Promise<Response>>) {
|
||||
let i = 0;
|
||||
return vi.fn(async (_u: FetchArgs[0], _i?: FetchArgs[1]) => {
|
||||
const idx = Math.min(i, responders.length - 1);
|
||||
i += 1;
|
||||
const r = responders[idx];
|
||||
if (!r) throw new Error('no responder');
|
||||
return r();
|
||||
});
|
||||
}
|
||||
|
||||
describe('subscribe', () => {
|
||||
it('connects to the SSE endpoint and delivers events', async () => {
|
||||
const fetchMock = queuedFetch([async () => sseResponse([dataFrame('chat', { hi: 1 })])]);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
|
||||
const received: Array<RealtimeEvent<{ hi: number }>> = [];
|
||||
const unsubscribe = client.subscribe<{ hi: number }>('chat', (e) => received.push(e));
|
||||
|
||||
await vi.waitFor(() => expect(received.length).toBe(1));
|
||||
unsubscribe();
|
||||
|
||||
expect(received[0]?.topic).toBe('chat');
|
||||
expect(received[0]?.message).toEqual({ hi: 1 });
|
||||
expect(lastUrl(fetchMock)).toBe('https://api.test/realtime/topics/chat');
|
||||
});
|
||||
|
||||
it('passes a token via the query string', async () => {
|
||||
const fetchMock = queuedFetch([async () => sseResponse([dataFrame('chat', 1)])]);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
const unsubscribe = client.subscribe('chat', () => {}, { token: 'abc.def' });
|
||||
await vi.waitFor(() => expect(fetchMock).toHaveBeenCalled());
|
||||
unsubscribe();
|
||||
expect(lastUrl(fetchMock)).toBe('https://api.test/realtime/topics/chat?token=abc.def');
|
||||
});
|
||||
|
||||
it('reconnects with backoff after an initial connection failure', async () => {
|
||||
const fetchMock = queuedFetch([
|
||||
async () => {
|
||||
throw new Error('network down');
|
||||
},
|
||||
async () => sseResponse([dataFrame('chat', { ok: true })])
|
||||
]);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
|
||||
const received: unknown[] = [];
|
||||
const unsubscribe = client.subscribe('chat', (e) => received.push(e.message), {
|
||||
baseBackoffMs: 5,
|
||||
maxBackoffMs: 20
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(received.length).toBeGreaterThanOrEqual(1), { timeout: 1000 });
|
||||
unsubscribe();
|
||||
expect(fetchMock.mock.calls.length).toBeGreaterThanOrEqual(2);
|
||||
expect(received[0]).toEqual({ ok: true });
|
||||
});
|
||||
|
||||
it('refreshes the token after a 401 and reconnects', async () => {
|
||||
const fetchMock = queuedFetch([
|
||||
async () => emptyResponse(401),
|
||||
async () => sseResponse([dataFrame('chat', { v: 2 })])
|
||||
]);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
|
||||
const onTokenExpired = vi.fn(() => 'fresh-token');
|
||||
const received: unknown[] = [];
|
||||
const unsubscribe = client.subscribe('chat', (e) => received.push(e.message), {
|
||||
token: 'stale',
|
||||
onTokenExpired,
|
||||
baseBackoffMs: 5
|
||||
});
|
||||
|
||||
await vi.waitFor(() => expect(received.length).toBeGreaterThanOrEqual(1), { timeout: 1000 });
|
||||
unsubscribe();
|
||||
|
||||
expect(onTokenExpired).toHaveBeenCalled();
|
||||
// Second connect carries the refreshed token.
|
||||
expect(lastUrl(fetchMock, 1)).toContain('token=fresh-token');
|
||||
expect(received[0]).toEqual({ v: 2 });
|
||||
});
|
||||
|
||||
it('stops and reports when a 401 cannot be refreshed', async () => {
|
||||
const fetchMock = queuedFetch([async () => emptyResponse(401)]);
|
||||
const client = new PicloudClient({ baseURL: 'https://api.test', fetch: fetchMock });
|
||||
const onError = vi.fn();
|
||||
const unsubscribe = client.subscribe('chat', () => {}, {
|
||||
onTokenExpired: () => null,
|
||||
onError
|
||||
});
|
||||
await vi.waitFor(() => expect(onError).toHaveBeenCalled());
|
||||
unsubscribe();
|
||||
});
|
||||
});
|
||||
34
clients/typescript/tests/svelte.test.ts
Normal file
34
clients/typescript/tests/svelte.test.ts
Normal file
@@ -0,0 +1,34 @@
|
||||
import { get } from 'svelte/store';
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
|
||||
import type { PicloudClient, RealtimeEvent, Unsubscribe } from '../src/index.js';
|
||||
import { topicStore } from '../src/svelte/index.js';
|
||||
|
||||
type Cb = (e: RealtimeEvent<unknown>) => void;
|
||||
|
||||
describe('svelte topicStore', () => {
|
||||
it('subscribes on first subscriber and unsubscribes on last', () => {
|
||||
const unsubscribe = vi.fn();
|
||||
const holder: { cb: Cb | null } = { cb: null };
|
||||
const subscribe = vi.fn((_topic: string, cb: Cb): Unsubscribe => {
|
||||
holder.cb = cb;
|
||||
return unsubscribe as unknown as Unsubscribe;
|
||||
});
|
||||
const client = { subscribe } as unknown as PicloudClient;
|
||||
|
||||
const store = topicStore<{ x: number }>(client, 'chat');
|
||||
// No SSE connection until someone subscribes (readable lifecycle).
|
||||
expect(subscribe).not.toHaveBeenCalled();
|
||||
|
||||
let value: { x: number }[] = [];
|
||||
const stop = store.subscribe((v) => (value = v));
|
||||
expect(subscribe).toHaveBeenCalledTimes(1);
|
||||
|
||||
holder.cb?.({ topic: 'chat', message: { x: 1 }, published_at: 't' });
|
||||
expect(value).toEqual([{ x: 1 }]);
|
||||
expect(get(store)).toEqual([{ x: 1 }]);
|
||||
|
||||
stop();
|
||||
expect(unsubscribe).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
21
clients/typescript/tsconfig.json
Normal file
21
clients/typescript/tsconfig.json
Normal file
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"lib": ["ES2020", "DOM", "DOM.Iterable"],
|
||||
"jsx": "react-jsx",
|
||||
"strict": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"exactOptionalPropertyTypes": false,
|
||||
"noImplicitOverride": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"declaration": true,
|
||||
"noEmit": true,
|
||||
"types": []
|
||||
},
|
||||
"include": ["src", "tests"]
|
||||
}
|
||||
18
clients/typescript/tsup.config.ts
Normal file
18
clients/typescript/tsup.config.ts
Normal file
@@ -0,0 +1,18 @@
|
||||
import { defineConfig } from 'tsup';
|
||||
|
||||
// Dual ESM + CJS emit with .d.ts for the main entry and the two
|
||||
// framework subpath exports. React and Svelte are peer deps — kept
|
||||
// external so the lib never bundles a framework copy.
|
||||
export default defineConfig({
|
||||
entry: {
|
||||
index: 'src/index.ts',
|
||||
'react/index': 'src/react/index.ts',
|
||||
'svelte/index': 'src/svelte/index.ts'
|
||||
},
|
||||
format: ['esm', 'cjs'],
|
||||
dts: true,
|
||||
clean: true,
|
||||
sourcemap: true,
|
||||
treeshake: true,
|
||||
external: ['react', 'svelte', 'svelte/store']
|
||||
});
|
||||
11
clients/typescript/vitest.config.ts
Normal file
11
clients/typescript/vitest.config.ts
Normal file
@@ -0,0 +1,11 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
// jsdom so the React/Svelte hook tests have a DOM; the core
|
||||
// endpoint/subscribe/auth tests are environment-agnostic.
|
||||
environment: 'jsdom',
|
||||
globals: true,
|
||||
include: ['tests/**/*.test.ts', 'tests/**/*.test.tsx']
|
||||
}
|
||||
});
|
||||
@@ -35,6 +35,13 @@ rand.workspace = true
|
||||
base64.workspace = true
|
||||
hex.workspace = true
|
||||
percent-encoding.workspace = true
|
||||
# v1.1.4 — `http::post_form` uses `url::form_urlencoded` for correct
|
||||
# application/x-www-form-urlencoded body encoding.
|
||||
url.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
async-trait.workspace = true
|
||||
# v1.1.4 §10a: capture tracing output to assert the original module
|
||||
# backend error is logged at error level after being redacted from the
|
||||
# script-visible message.
|
||||
tracing-subscriber.workspace = true
|
||||
|
||||
@@ -144,6 +144,7 @@ impl Engine {
|
||||
// capture cheap clones of the cx for use at script-call time.
|
||||
let cx = Arc::new(SdkCallCx {
|
||||
app_id: req.app_id,
|
||||
script_id: req.script_id,
|
||||
principal: req.principal.clone(),
|
||||
execution_id: req.execution_id,
|
||||
request_id: req.request_id,
|
||||
@@ -347,6 +348,7 @@ fn build_ctx_map(req: &ExecRequest) -> Map {
|
||||
/// `docs/v1.1.x-design-notes.md` §4 (the dead-letter sub-shape) and
|
||||
/// §2/blueprint §9 (KV). Each variant becomes a Rhai map with a
|
||||
/// `source` discriminant plus per-source fields.
|
||||
#[allow(clippy::too_many_lines)]
|
||||
fn trigger_event_to_dynamic(event: &TriggerEvent) -> Dynamic {
|
||||
let mut m = Map::new();
|
||||
m.insert("source".into(), event.source().into());
|
||||
@@ -388,6 +390,64 @@ fn trigger_event_to_dynamic(event: &TriggerEvent) -> Dynamic {
|
||||
);
|
||||
m.insert("docs".into(), docs_map.into());
|
||||
}
|
||||
TriggerEvent::Cron {
|
||||
schedule,
|
||||
timezone,
|
||||
scheduled_at,
|
||||
fired_at,
|
||||
} => {
|
||||
// `ctx.event.op` is always "tick" for cron (the only op a
|
||||
// schedule produces). Mirrors the docs/v1.1.x-design-notes
|
||||
// §7 shape.
|
||||
m.insert("op".into(), "tick".into());
|
||||
let mut cron_map = Map::new();
|
||||
cron_map.insert("schedule".into(), schedule.clone().into());
|
||||
cron_map.insert("timezone".into(), timezone.clone().into());
|
||||
cron_map.insert("scheduled_at".into(), scheduled_at.to_rfc3339().into());
|
||||
cron_map.insert("fired_at".into(), fired_at.to_rfc3339().into());
|
||||
m.insert("cron".into(), cron_map.into());
|
||||
}
|
||||
TriggerEvent::Files {
|
||||
op,
|
||||
collection,
|
||||
id,
|
||||
name,
|
||||
content_type,
|
||||
size,
|
||||
checksum,
|
||||
prev,
|
||||
} => {
|
||||
m.insert("op".into(), op.as_str().into());
|
||||
let mut files_map = Map::new();
|
||||
files_map.insert("collection".into(), collection.clone().into());
|
||||
files_map.insert("id".into(), id.clone().into());
|
||||
files_map.insert("name".into(), name.clone().into());
|
||||
files_map.insert("content_type".into(), content_type.clone().into());
|
||||
files_map.insert(
|
||||
"size".into(),
|
||||
i64::try_from(*size).unwrap_or(i64::MAX).into(),
|
||||
);
|
||||
files_map.insert("checksum".into(), checksum.clone().into());
|
||||
files_map.insert(
|
||||
"prev".into(),
|
||||
prev.clone().map_or(Dynamic::UNIT, json_to_dynamic),
|
||||
);
|
||||
m.insert("files".into(), files_map.into());
|
||||
}
|
||||
TriggerEvent::Pubsub {
|
||||
topic,
|
||||
message,
|
||||
published_at,
|
||||
} => {
|
||||
// `ctx.event.op` is always "publish" for pub/sub (the only
|
||||
// op a publish produces).
|
||||
m.insert("op".into(), "publish".into());
|
||||
let mut ps = Map::new();
|
||||
ps.insert("topic".into(), topic.clone().into());
|
||||
ps.insert("message".into(), json_to_dynamic(message.clone()));
|
||||
ps.insert("published_at".into(), published_at.to_rfc3339().into());
|
||||
m.insert("pubsub".into(), ps.into());
|
||||
}
|
||||
TriggerEvent::DeadLetter {
|
||||
dead_letter_id,
|
||||
original,
|
||||
|
||||
@@ -331,10 +331,22 @@ impl ModuleResolver for PicloudModuleResolver {
|
||||
)));
|
||||
}
|
||||
Err(e) => {
|
||||
// v1.1.4 §10a: redact the backend error before it
|
||||
// reaches a script. In public-HTTP context (principal:
|
||||
// None) the verbatim message (e.g. "connection refused")
|
||||
// leaks internal infrastructure shape. Log the original
|
||||
// at error level for operators; surface a stable generic.
|
||||
tracing::error!(
|
||||
target = "picloud::modules",
|
||||
app_id = %self.cx.app_id,
|
||||
module = path,
|
||||
error = %e,
|
||||
"module backend error"
|
||||
);
|
||||
return Err(Box::new(EvalAltResult::ErrorInModule(
|
||||
path.to_string(),
|
||||
Box::new(EvalAltResult::ErrorRuntime(
|
||||
format!("module backend error: {e}").into(),
|
||||
"module backend unavailable; check server logs".into(),
|
||||
pos,
|
||||
)),
|
||||
pos,
|
||||
|
||||
281
crates/executor-core/src/sdk/files.rs
Normal file
281
crates/executor-core/src/sdk/files.rs
Normal file
@@ -0,0 +1,281 @@
|
||||
//! `files::` Rhai bridge — collection-scoped handle pattern (v1.1.5).
|
||||
//!
|
||||
//! ```rhai
|
||||
//! let avatars = files::collection("avatars");
|
||||
//! let id = avatars.create(#{ name: "a.jpg", content_type: "image/jpeg", data: blob });
|
||||
//! let meta = avatars.head(id); // metadata map or ()
|
||||
//! let bytes = avatars.get(id); // Blob or ()
|
||||
//! avatars.update(id, #{ data: new_bytes });
|
||||
//! let gone = avatars.delete(id); // bool (was-present)
|
||||
//! let page = avatars.list(); // #{ files: [...], next_cursor: () }
|
||||
//! ```
|
||||
//!
|
||||
//! The `FilesHandle` custom Rhai type captures the collection name once
|
||||
//! and routes each call through the injected `Arc<dyn FilesService>`
|
||||
//! with the per-call `Arc<SdkCallCx>`. **The service derives `app_id`
|
||||
//! from `cx.app_id` — it never appears in any signature script-side,
|
||||
//! preserving cross-app isolation.**
|
||||
//!
|
||||
//! Error convention (per `docs/sdk-shape.md`): `create`/`update`/
|
||||
//! `delete` throw on failure; `get`/`head` return `()` for a missing
|
||||
//! file; `delete` returns `bool` (was-present). The blob bytes are a
|
||||
//! Rhai `Blob` (byte array) in both directions.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use picloud_shared::{
|
||||
FileMeta, FileUpdate, FilesError, FilesService, NewFile, SdkCallCx, Services,
|
||||
};
|
||||
use rhai::{Array, Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
|
||||
use tokio::runtime::Handle as TokioHandle;
|
||||
|
||||
/// Per-call handle captured by the Rhai SDK. Cheap to clone (two Arcs
|
||||
/// plus an owned string).
|
||||
#[derive(Clone)]
|
||||
pub struct FilesHandle {
|
||||
collection: String,
|
||||
service: Arc<dyn FilesService>,
|
||||
cx: Arc<SdkCallCx>,
|
||||
}
|
||||
|
||||
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
|
||||
let files_service = services.files.clone();
|
||||
|
||||
let mut module = Module::new();
|
||||
{
|
||||
let files_service = files_service.clone();
|
||||
let cx = cx.clone();
|
||||
module.set_native_fn(
|
||||
"collection",
|
||||
move |name: &str| -> Result<FilesHandle, Box<EvalAltResult>> {
|
||||
if name.is_empty() {
|
||||
return Err("files::collection name must not be empty".into());
|
||||
}
|
||||
Ok(FilesHandle {
|
||||
collection: name.to_string(),
|
||||
service: files_service.clone(),
|
||||
cx: cx.clone(),
|
||||
})
|
||||
},
|
||||
);
|
||||
}
|
||||
engine.register_static_module("files", module.into());
|
||||
|
||||
engine.register_type_with_name::<FilesHandle>("FilesHandle");
|
||||
|
||||
register_create(engine);
|
||||
register_head(engine);
|
||||
register_get(engine);
|
||||
register_update(engine);
|
||||
register_delete(engine);
|
||||
register_list(engine);
|
||||
}
|
||||
|
||||
fn register_create(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"create",
|
||||
|handle: &mut FilesHandle, meta: Map| -> Result<String, Box<EvalAltResult>> {
|
||||
let name = require_string(&meta, "name")?;
|
||||
let content_type = require_string(&meta, "content_type")?;
|
||||
let data = require_blob(&meta, "data")?;
|
||||
let h = handle.clone();
|
||||
let new = NewFile {
|
||||
name,
|
||||
content_type,
|
||||
data,
|
||||
};
|
||||
let id = block_on(async move { h.service.create(&h.cx, &h.collection, new).await })?;
|
||||
Ok(id.to_string())
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn register_head(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"head",
|
||||
|handle: &mut FilesHandle, id: &str| -> Result<Dynamic, Box<EvalAltResult>> {
|
||||
let h = handle.clone();
|
||||
let id = id.to_string();
|
||||
let meta = block_on(async move { h.service.head(&h.cx, &h.collection, &id).await })?;
|
||||
Ok(meta.map_or(Dynamic::UNIT, |m| file_meta_to_map(&m).into()))
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn register_get(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"get",
|
||||
|handle: &mut FilesHandle, id: &str| -> Result<Dynamic, Box<EvalAltResult>> {
|
||||
let h = handle.clone();
|
||||
let id = id.to_string();
|
||||
let bytes = block_on(async move { h.service.get(&h.cx, &h.collection, &id).await })?;
|
||||
Ok(bytes.map_or(Dynamic::UNIT, Dynamic::from_blob))
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn register_update(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"update",
|
||||
|handle: &mut FilesHandle, id: &str, meta: Map| -> Result<(), Box<EvalAltResult>> {
|
||||
let data = require_blob(&meta, "data")?;
|
||||
let name = optional_string(&meta, "name")?;
|
||||
let content_type = optional_string(&meta, "content_type")?;
|
||||
let h = handle.clone();
|
||||
let id = id.to_string();
|
||||
let upd = FileUpdate {
|
||||
data,
|
||||
name,
|
||||
content_type,
|
||||
};
|
||||
block_on(async move { h.service.update(&h.cx, &h.collection, &id, upd).await })
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn register_delete(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"delete",
|
||||
|handle: &mut FilesHandle, id: &str| -> Result<bool, Box<EvalAltResult>> {
|
||||
let h = handle.clone();
|
||||
let id = id.to_string();
|
||||
block_on(async move { h.service.delete(&h.cx, &h.collection, &id).await })
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn register_list(engine: &mut RhaiEngine) {
|
||||
engine.register_fn(
|
||||
"list",
|
||||
|handle: &mut FilesHandle| -> Result<Map, Box<EvalAltResult>> {
|
||||
list_call(handle, None, 0)
|
||||
},
|
||||
);
|
||||
engine.register_fn(
|
||||
"list",
|
||||
|handle: &mut FilesHandle, cursor: &str| -> Result<Map, Box<EvalAltResult>> {
|
||||
list_call(handle, Some(cursor.to_string()), 0)
|
||||
},
|
||||
);
|
||||
engine.register_fn(
|
||||
"list",
|
||||
|handle: &mut FilesHandle, cursor: &str, limit: i64| -> Result<Map, Box<EvalAltResult>> {
|
||||
let limit = u32::try_from(limit.max(0)).unwrap_or(0);
|
||||
list_call(handle, Some(cursor.to_string()), limit)
|
||||
},
|
||||
);
|
||||
// `list(#{ cursor, limit })` — the map form documented in the brief.
|
||||
engine.register_fn(
|
||||
"list",
|
||||
|handle: &mut FilesHandle, opts: Map| -> Result<Map, Box<EvalAltResult>> {
|
||||
let cursor = match opts.get("cursor") {
|
||||
Some(v) if !v.is_unit() => {
|
||||
Some(v.clone().into_string().map_err(|_| -> Box<EvalAltResult> {
|
||||
"files: list cursor must be a string".into()
|
||||
})?)
|
||||
}
|
||||
_ => None,
|
||||
};
|
||||
let limit = match opts.get("limit") {
|
||||
Some(v) if !v.is_unit() => {
|
||||
u32::try_from(v.as_int().unwrap_or(0).max(0)).unwrap_or(0)
|
||||
}
|
||||
_ => 0,
|
||||
};
|
||||
list_call(handle, cursor, limit)
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
fn list_call(
|
||||
handle: &FilesHandle,
|
||||
cursor: Option<String>,
|
||||
limit: u32,
|
||||
) -> Result<Map, Box<EvalAltResult>> {
|
||||
let h = handle.clone();
|
||||
let page = block_on(async move {
|
||||
h.service
|
||||
.list(&h.cx, &h.collection, cursor.as_deref(), limit)
|
||||
.await
|
||||
})?;
|
||||
let mut m = Map::new();
|
||||
let files: Array = page
|
||||
.files
|
||||
.iter()
|
||||
.map(|meta| Dynamic::from(file_meta_to_map(meta)))
|
||||
.collect();
|
||||
m.insert("files".into(), files.into());
|
||||
m.insert(
|
||||
"next_cursor".into(),
|
||||
page.next_cursor.map_or(Dynamic::UNIT, Dynamic::from),
|
||||
);
|
||||
Ok(m)
|
||||
}
|
||||
|
||||
/// Render a `FileMeta` into the Rhai map shape scripts see from
|
||||
/// `head` / `list`.
|
||||
fn file_meta_to_map(meta: &FileMeta) -> Map {
|
||||
let mut m = Map::new();
|
||||
m.insert("id".into(), meta.id.to_string().into());
|
||||
m.insert("collection".into(), meta.collection.clone().into());
|
||||
m.insert("name".into(), meta.name.clone().into());
|
||||
m.insert("content_type".into(), meta.content_type.clone().into());
|
||||
m.insert(
|
||||
"size".into(),
|
||||
i64::try_from(meta.size).unwrap_or(i64::MAX).into(),
|
||||
);
|
||||
m.insert("checksum".into(), meta.checksum.clone().into());
|
||||
m.insert("created_at".into(), meta.created_at.to_rfc3339().into());
|
||||
m.insert("updated_at".into(), meta.updated_at.to_rfc3339().into());
|
||||
m
|
||||
}
|
||||
|
||||
/// Pull a required string field out of a Rhai map; throw naming the
|
||||
/// field if it's absent or not a string.
|
||||
fn require_string(meta: &Map, field: &'static str) -> Result<String, Box<EvalAltResult>> {
|
||||
match meta.get(field) {
|
||||
Some(v) if v.is_string() => Ok(v.clone().into_string().unwrap_or_default()),
|
||||
Some(_) => Err(format!("files::create: field '{field}' must be a string").into()),
|
||||
None => Err(format!("files::create: missing required field '{field}'").into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pull an optional string field; `None` when the key is absent or unit.
|
||||
fn optional_string(meta: &Map, field: &'static str) -> Result<Option<String>, Box<EvalAltResult>> {
|
||||
match meta.get(field) {
|
||||
None => Ok(None),
|
||||
Some(v) if v.is_unit() => Ok(None),
|
||||
Some(v) if v.is_string() => Ok(Some(v.clone().into_string().unwrap_or_default())),
|
||||
Some(_) => Err(format!("files::update: field '{field}' must be a string").into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Pull a required blob (`data`) out of a Rhai map; throw naming the
|
||||
/// field if it's absent or not a blob.
|
||||
fn require_blob(meta: &Map, field: &'static str) -> Result<Vec<u8>, Box<EvalAltResult>> {
|
||||
match meta.get(field) {
|
||||
Some(v) if v.is_blob() => Ok(v.clone().into_blob().unwrap_or_default()),
|
||||
Some(_) => Err(format!("files: field '{field}' must be a Blob (byte array)").into()),
|
||||
None => Err(format!("files: missing required field '{field}'").into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Run an async future inside the synchronous Rhai context. Mirrors
|
||||
/// `kv::block_on`; safe because `LocalExecutorClient` runs the script
|
||||
/// under `spawn_blocking`, so a runtime handle is reachable.
|
||||
fn block_on<F, T>(fut: F) -> Result<T, Box<EvalAltResult>>
|
||||
where
|
||||
F: std::future::Future<Output = Result<T, FilesError>> + Send,
|
||||
T: Send,
|
||||
{
|
||||
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(
|
||||
format!("files: no tokio runtime available: {e}").into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into()
|
||||
})?;
|
||||
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(format!("files: {err}").into(), rhai::Position::NONE).into()
|
||||
})
|
||||
}
|
||||
391
crates/executor-core/src/sdk/http.rs
Normal file
391
crates/executor-core/src/sdk/http.rs
Normal file
@@ -0,0 +1,391 @@
|
||||
//! `http::` Rhai bridge — outbound HTTP from scripts (v1.1.4).
|
||||
//!
|
||||
//! ```rhai
|
||||
//! let r = http::get("https://api.example.com/users/123");
|
||||
//! let r = http::get(url, #{ headers: #{ "Authorization": "Bearer x" }, timeout_ms: 5000 });
|
||||
//! let r = http::post(url, #{ text: "hello" }); // Map body → JSON
|
||||
//! let r = http::post(url, "raw", #{ headers: #{ ... } }); // String body → text/plain
|
||||
//! let r = http::post_form(url, #{ a: "1", b: "2" }); // form-encoded
|
||||
//! let r = http::request("OPTIONS", url);
|
||||
//! ```
|
||||
//!
|
||||
//! **Argument shape (v1.1.4 decision):** body and options are separate
|
||||
//! positional arguments — `verb(url, body, opts)` — not body-inside-
|
||||
//! opts. This keeps the unknown-opt-key typo guard intact and resolves
|
||||
//! the brief's internal contradiction (its Slack example passed a bare
|
||||
//! body map). The `opts` vocabulary is exactly
|
||||
//! `{headers, timeout_ms, follow_redirects, max_redirects}`; any other
|
||||
//! key throws.
|
||||
//!
|
||||
//! Body dispatch (positional `body`): Map/Array → JSON +
|
||||
//! `application/json`; String → raw + `text/plain`; Unit `()` → no
|
||||
//! body. GET/HEAD ignore any body.
|
||||
//!
|
||||
//! Response is a Rhai map `#{ status, headers, body, body_raw }`:
|
||||
//! `body` is the parsed JSON when the response is `application/json`
|
||||
//! and parses; `()` for an empty body; otherwise the raw string.
|
||||
//!
|
||||
//! Errors follow `docs/sdk-shape.md`: network/timeout/SSRF/size failures
|
||||
//! throw (`"http: <message>"`); a non-2xx status does NOT throw — the
|
||||
//! response map is returned, fetch-style.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use picloud_shared::{HttpError, HttpRequest, HttpResponse, HttpService, SdkCallCx, Services};
|
||||
use rhai::{Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
|
||||
use tokio::runtime::Handle as TokioHandle;
|
||||
|
||||
use super::bridge::{dynamic_to_json, json_to_dynamic};
|
||||
|
||||
/// Bridge-side defaults (the service clamps server-side too). The
|
||||
/// `MAX_*` ceilings stay `i64` because they're compared against the
|
||||
/// raw `i64` the script passed (so an over-limit value is rejected, not
|
||||
/// truncated); the defaults are `u32` to match the `Opts` fields.
|
||||
const DEFAULT_TIMEOUT_MS: u32 = 30_000;
|
||||
const MAX_TIMEOUT_MS: i64 = 60_000;
|
||||
const DEFAULT_MAX_REDIRECTS: u32 = 5;
|
||||
const MAX_REDIRECTS: i64 = 10;
|
||||
|
||||
const ALLOWED_OPT_KEYS: [&str; 4] = ["headers", "timeout_ms", "follow_redirects", "max_redirects"];
|
||||
|
||||
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
|
||||
let svc = services.http.clone();
|
||||
let mut module = Module::new();
|
||||
|
||||
// Bodyless verbs: (url) / (url, opts).
|
||||
for verb in ["get", "head"] {
|
||||
register_bodyless(&mut module, verb, &svc, &cx);
|
||||
}
|
||||
// Body verbs: (url) / (url, body) / (url, body, opts).
|
||||
for verb in ["post", "put", "patch", "delete"] {
|
||||
register_body(&mut module, verb, &svc, &cx);
|
||||
}
|
||||
register_post_form(&mut module, &svc, &cx);
|
||||
register_request(&mut module, &svc, &cx);
|
||||
|
||||
engine.register_static_module("http", module.into());
|
||||
}
|
||||
|
||||
fn register_bodyless(
|
||||
module: &mut Module,
|
||||
verb: &'static str,
|
||||
svc: &Arc<dyn HttpService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
) {
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(verb, move |url: &str| {
|
||||
invoke(&svc, &cx, verb, url, None, None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(verb, move |url: &str, opts: Map| {
|
||||
invoke(&svc, &cx, verb, url, None, Some(&opts))
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn register_body(
|
||||
module: &mut Module,
|
||||
verb: &'static str,
|
||||
svc: &Arc<dyn HttpService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
) {
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(verb, move |url: &str| {
|
||||
invoke(&svc, &cx, verb, url, None, None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(verb, move |url: &str, body: Dynamic| {
|
||||
invoke(&svc, &cx, verb, url, Some(body), None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(verb, move |url: &str, body: Dynamic, opts: Map| {
|
||||
invoke(&svc, &cx, verb, url, Some(body), Some(&opts))
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn register_post_form(module: &mut Module, svc: &Arc<dyn HttpService>, cx: &Arc<SdkCallCx>) {
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn("post_form", move |url: &str, form: Map| {
|
||||
invoke_form(&svc, &cx, url, &form, None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn("post_form", move |url: &str, form: Map, opts: Map| {
|
||||
invoke_form(&svc, &cx, url, &form, Some(&opts))
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn register_request(module: &mut Module, svc: &Arc<dyn HttpService>, cx: &Arc<SdkCallCx>) {
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn("request", move |method: &str, url: &str| {
|
||||
invoke(&svc, &cx, method, url, None, None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn("request", move |method: &str, url: &str, body: Dynamic| {
|
||||
invoke(&svc, &cx, method, url, Some(body), None)
|
||||
});
|
||||
}
|
||||
{
|
||||
let (svc, cx) = (svc.clone(), cx.clone());
|
||||
module.set_native_fn(
|
||||
"request",
|
||||
move |method: &str, url: &str, body: Dynamic, opts: Map| {
|
||||
invoke(&svc, &cx, method, url, Some(body), Some(&opts))
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Parsed `opts` map.
|
||||
struct Opts {
|
||||
headers: BTreeMap<String, String>,
|
||||
timeout_ms: u32,
|
||||
follow_redirects: bool,
|
||||
max_redirects: u32,
|
||||
}
|
||||
|
||||
impl Default for Opts {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
headers: BTreeMap::new(),
|
||||
timeout_ms: DEFAULT_TIMEOUT_MS,
|
||||
follow_redirects: true,
|
||||
max_redirects: DEFAULT_MAX_REDIRECTS,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_opts(opts: Option<&Map>) -> Result<Opts, Box<EvalAltResult>> {
|
||||
let mut out = Opts::default();
|
||||
let Some(map) = opts else {
|
||||
return Ok(out);
|
||||
};
|
||||
for key in map.keys() {
|
||||
if !ALLOWED_OPT_KEYS.contains(&key.as_str()) {
|
||||
return Err(err(format!("unknown option key: {key}")));
|
||||
}
|
||||
}
|
||||
if let Some(h) = map.get("headers") {
|
||||
let hm = h
|
||||
.clone()
|
||||
.try_cast::<Map>()
|
||||
.ok_or_else(|| err("headers must be a map".to_string()))?;
|
||||
for (k, v) in hm {
|
||||
out.headers.insert(k.to_string(), dyn_to_string(&v));
|
||||
}
|
||||
}
|
||||
if let Some(t) = map.get("timeout_ms") {
|
||||
let ms = t
|
||||
.as_int()
|
||||
.map_err(|_| err("timeout_ms must be an integer".to_string()))?;
|
||||
if ms > MAX_TIMEOUT_MS {
|
||||
return Err(err(format!(
|
||||
"timeout_ms {ms} exceeds the {MAX_TIMEOUT_MS}ms maximum"
|
||||
)));
|
||||
}
|
||||
if ms > 0 {
|
||||
out.timeout_ms = u32::try_from(ms).unwrap_or(u32::MAX);
|
||||
}
|
||||
}
|
||||
if let Some(f) = map.get("follow_redirects") {
|
||||
out.follow_redirects = f
|
||||
.as_bool()
|
||||
.map_err(|_| err("follow_redirects must be a bool".to_string()))?;
|
||||
}
|
||||
if let Some(m) = map.get("max_redirects") {
|
||||
let n = m
|
||||
.as_int()
|
||||
.map_err(|_| err("max_redirects must be an integer".to_string()))?;
|
||||
if n > MAX_REDIRECTS {
|
||||
return Err(err(format!(
|
||||
"max_redirects {n} exceeds the {MAX_REDIRECTS} maximum"
|
||||
)));
|
||||
}
|
||||
out.max_redirects = u32::try_from(n.max(0)).unwrap_or(0);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Encoded request body + the content-type chosen for it.
|
||||
type EncodedBody = (Option<Vec<u8>>, Option<String>);
|
||||
|
||||
/// Dispatch a positional body by Rhai type. Returns the encoded bytes +
|
||||
/// the chosen content-type. GET/HEAD callers pass `body = None`, so
|
||||
/// this is never reached for them.
|
||||
fn dispatch_body(body: Dynamic) -> Result<EncodedBody, Box<EvalAltResult>> {
|
||||
if body.is_unit() {
|
||||
return Ok((None, None));
|
||||
}
|
||||
if body.is_string() {
|
||||
let s = body.into_string().unwrap_or_default();
|
||||
return Ok((Some(s.into_bytes()), Some("text/plain".to_string())));
|
||||
}
|
||||
if body.is_map() || body.is_array() {
|
||||
let json = dynamic_to_json(&body);
|
||||
let bytes = serde_json::to_vec(&json)
|
||||
.map_err(|e| err(format!("could not encode JSON body: {e}")))?;
|
||||
return Ok((Some(bytes), Some("application/json".to_string())));
|
||||
}
|
||||
// Scalars (int/float/bool) → JSON-encode for consistency.
|
||||
let json = dynamic_to_json(&body);
|
||||
let bytes =
|
||||
serde_json::to_vec(&json).map_err(|e| err(format!("could not encode body: {e}")))?;
|
||||
Ok((Some(bytes), Some("application/json".to_string())))
|
||||
}
|
||||
|
||||
#[allow(clippy::needless_pass_by_value)]
|
||||
fn invoke(
|
||||
svc: &Arc<dyn HttpService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
method: &str,
|
||||
url: &str,
|
||||
body: Option<Dynamic>,
|
||||
opts: Option<&Map>,
|
||||
) -> Result<Dynamic, Box<EvalAltResult>> {
|
||||
let opts = parse_opts(opts)?;
|
||||
let method_uc = method.to_ascii_uppercase();
|
||||
let bodyless = matches!(method_uc.as_str(), "GET" | "HEAD");
|
||||
let (encoded, content_type) = if bodyless {
|
||||
(None, None)
|
||||
} else if let Some(b) = body {
|
||||
dispatch_body(b)?
|
||||
} else {
|
||||
(None, None)
|
||||
};
|
||||
|
||||
let req = HttpRequest {
|
||||
method: method_uc,
|
||||
url: url.to_string(),
|
||||
headers: opts.headers,
|
||||
body: encoded,
|
||||
content_type,
|
||||
timeout_ms: opts.timeout_ms,
|
||||
follow_redirects: opts.follow_redirects,
|
||||
max_redirects: opts.max_redirects,
|
||||
script_id: Some(cx.script_id.to_string()),
|
||||
};
|
||||
let resp = block_on(svc, cx, req)?;
|
||||
Ok(response_to_dynamic(&resp))
|
||||
}
|
||||
|
||||
#[allow(clippy::needless_pass_by_value)]
|
||||
fn invoke_form(
|
||||
svc: &Arc<dyn HttpService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
url: &str,
|
||||
form: &Map,
|
||||
opts: Option<&Map>,
|
||||
) -> Result<Dynamic, Box<EvalAltResult>> {
|
||||
let opts = parse_opts(opts)?;
|
||||
let mut serializer = url::form_urlencoded::Serializer::new(String::new());
|
||||
for (k, v) in form {
|
||||
serializer.append_pair(k.as_str(), &dyn_to_string(v));
|
||||
}
|
||||
let encoded = serializer.finish();
|
||||
|
||||
let req = HttpRequest {
|
||||
method: "POST".to_string(),
|
||||
url: url.to_string(),
|
||||
headers: opts.headers,
|
||||
body: Some(encoded.into_bytes()),
|
||||
content_type: Some("application/x-www-form-urlencoded".to_string()),
|
||||
timeout_ms: opts.timeout_ms,
|
||||
follow_redirects: opts.follow_redirects,
|
||||
max_redirects: opts.max_redirects,
|
||||
script_id: Some(cx.script_id.to_string()),
|
||||
};
|
||||
let resp = block_on(svc, cx, req)?;
|
||||
Ok(response_to_dynamic(&resp))
|
||||
}
|
||||
|
||||
fn response_to_dynamic(resp: &HttpResponse) -> Dynamic {
|
||||
let mut m = Map::new();
|
||||
m.insert("status".into(), i64::from(resp.status).into());
|
||||
|
||||
let mut headers = Map::new();
|
||||
let mut content_type = String::new();
|
||||
for (k, v) in &resp.headers {
|
||||
if k == "content-type" {
|
||||
content_type.clone_from(v);
|
||||
}
|
||||
headers.insert(k.clone().into(), v.clone().into());
|
||||
}
|
||||
m.insert("headers".into(), headers.into());
|
||||
|
||||
// `body`: parsed JSON when the response is JSON and parses; () when
|
||||
// empty; otherwise the raw string.
|
||||
let body = if resp.body_raw.is_empty() {
|
||||
Dynamic::UNIT
|
||||
} else if content_type
|
||||
.to_ascii_lowercase()
|
||||
.starts_with("application/json")
|
||||
{
|
||||
match serde_json::from_str::<serde_json::Value>(&resp.body_raw) {
|
||||
Ok(json) => json_to_dynamic(json),
|
||||
Err(_) => resp.body_raw.clone().into(),
|
||||
}
|
||||
} else {
|
||||
resp.body_raw.clone().into()
|
||||
};
|
||||
m.insert("body".into(), body);
|
||||
m.insert("body_raw".into(), resp.body_raw.clone().into());
|
||||
m.into()
|
||||
}
|
||||
|
||||
fn dyn_to_string(v: &Dynamic) -> String {
|
||||
if v.is_string() {
|
||||
v.clone().into_string().unwrap_or_default()
|
||||
} else {
|
||||
v.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
// Rhai's native-fn error channel is `Box<EvalAltResult>`, so these
|
||||
// helpers return the boxed form the call sites need.
|
||||
#[allow(clippy::unnecessary_box_returns)]
|
||||
fn err(msg: String) -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(format!("http: {msg}").into(), rhai::Position::NONE).into()
|
||||
}
|
||||
|
||||
/// Run the async service call from the synchronous Rhai context. Same
|
||||
/// pattern as `kv`/`docs`: the script runs under `spawn_blocking`, so a
|
||||
/// runtime handle is reachable and blocking on it is correct.
|
||||
fn block_on(
|
||||
svc: &Arc<dyn HttpService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
req: HttpRequest,
|
||||
) -> Result<HttpResponse, Box<EvalAltResult>> {
|
||||
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(
|
||||
format!("http: no tokio runtime available: {e}").into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into()
|
||||
})?;
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
handle
|
||||
.block_on(async move { svc.request(&cx, req).await })
|
||||
.map_err(map_http_err)
|
||||
}
|
||||
|
||||
#[allow(clippy::unnecessary_box_returns)]
|
||||
fn map_http_err(e: HttpError) -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(format!("http: {e}").into(), rhai::Position::NONE).into()
|
||||
}
|
||||
@@ -15,7 +15,10 @@ pub mod bridge;
|
||||
pub mod cx;
|
||||
pub mod dead_letters;
|
||||
pub mod docs;
|
||||
pub mod files;
|
||||
pub mod http;
|
||||
pub mod kv;
|
||||
pub mod pubsub;
|
||||
pub mod stdlib;
|
||||
|
||||
pub use bridge::{dynamic_to_json, json_to_dynamic};
|
||||
@@ -35,5 +38,8 @@ use rhai::Engine as RhaiEngine;
|
||||
pub fn register_all(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
|
||||
kv::register(engine, services, cx.clone());
|
||||
docs::register(engine, services, cx.clone());
|
||||
dead_letters::register(engine, services, cx);
|
||||
dead_letters::register(engine, services, cx.clone());
|
||||
http::register(engine, services, cx.clone());
|
||||
files::register(engine, services, cx.clone());
|
||||
pubsub::register(engine, services, cx);
|
||||
}
|
||||
|
||||
176
crates/executor-core/src/sdk/pubsub.rs
Normal file
176
crates/executor-core/src/sdk/pubsub.rs
Normal file
@@ -0,0 +1,176 @@
|
||||
//! `pubsub::` Rhai bridge — durable publish (v1.1.5).
|
||||
//!
|
||||
//! ```rhai
|
||||
//! pubsub::publish_durable("user.created", #{ user_id: "abc" });
|
||||
//! pubsub::publish_durable("metric", 42);
|
||||
//! ```
|
||||
//!
|
||||
//! No handle pattern (topics ARE the grouping unit, so there's no
|
||||
//! `::collection(...)`). The message is any JSON-serializable Rhai value
|
||||
//! — Maps, Arrays, strings, numbers, bools, unit, and **Blobs (which
|
||||
//! encode as base64 strings** so trigger handlers see them as base64 on
|
||||
//! the wire). Nested blobs are encoded at any depth.
|
||||
//!
|
||||
//! `app_id` is derived from `cx.app_id` in the service — it never
|
||||
//! appears in the script-side signature, preserving cross-app
|
||||
//! isolation.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use base64::engine::general_purpose::STANDARD;
|
||||
use base64::Engine as _;
|
||||
use picloud_shared::{PubsubError, SdkCallCx, Services};
|
||||
use rhai::{Array, Dynamic, Engine as RhaiEngine, EvalAltResult, Map, Module};
|
||||
use serde_json::Value as Json;
|
||||
use tokio::runtime::Handle as TokioHandle;
|
||||
|
||||
pub(super) fn register(engine: &mut RhaiEngine, services: &Services, cx: Arc<SdkCallCx>) {
|
||||
let svc = services.pubsub.clone();
|
||||
let mut module = Module::new();
|
||||
{
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
module.set_native_fn(
|
||||
"publish_durable",
|
||||
move |topic: &str, message: Dynamic| -> Result<(), Box<EvalAltResult>> {
|
||||
let json = message_to_json(&message);
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
block_on(async move { svc.publish_durable(&cx, topic, json).await })
|
||||
},
|
||||
);
|
||||
}
|
||||
// `pubsub::subscriber_token(topics)` — uses the configured default
|
||||
// TTL.
|
||||
{
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
module.set_native_fn(
|
||||
"subscriber_token",
|
||||
move |topics: Array| -> Result<String, Box<EvalAltResult>> {
|
||||
mint_token(&svc, &cx, topics, None)
|
||||
},
|
||||
);
|
||||
}
|
||||
// `pubsub::subscriber_token(topics, ttl)` — `ttl` is an integer
|
||||
// (seconds) or `()` for the default.
|
||||
{
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
module.set_native_fn(
|
||||
"subscriber_token",
|
||||
move |topics: Array, ttl: Dynamic| -> Result<String, Box<EvalAltResult>> {
|
||||
let ttl = ttl_from_dynamic(&ttl)?;
|
||||
mint_token(&svc, &cx, topics, ttl)
|
||||
},
|
||||
);
|
||||
}
|
||||
engine.register_static_module("pubsub", module.into());
|
||||
}
|
||||
|
||||
/// Interpret the optional `ttl` argument: `()` → use the default,
|
||||
/// integer → that many seconds, anything else → throw.
|
||||
fn ttl_from_dynamic(ttl: &Dynamic) -> Result<Option<i64>, Box<EvalAltResult>> {
|
||||
if ttl.is_unit() {
|
||||
return Ok(None);
|
||||
}
|
||||
ttl.as_int().map(Some).map_err(|_| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(
|
||||
"pubsub::subscriber_token: ttl must be an integer (seconds) or ()".into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into()
|
||||
})
|
||||
}
|
||||
|
||||
fn mint_token(
|
||||
svc: &Arc<dyn picloud_shared::PubsubService>,
|
||||
cx: &Arc<SdkCallCx>,
|
||||
topics: Array,
|
||||
ttl: Option<i64>,
|
||||
) -> Result<String, Box<EvalAltResult>> {
|
||||
// Every element must be a string; surface a clear error otherwise.
|
||||
let mut names = Vec::with_capacity(topics.len());
|
||||
for t in topics {
|
||||
if !t.is_string() {
|
||||
return Err(EvalAltResult::ErrorRuntime(
|
||||
"pubsub::subscriber_token: topics must be an array of strings".into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into());
|
||||
}
|
||||
names.push(t.into_string().unwrap_or_default());
|
||||
}
|
||||
let svc = svc.clone();
|
||||
let cx = cx.clone();
|
||||
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(
|
||||
format!("pubsub: no tokio runtime available: {e}").into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into()
|
||||
})?;
|
||||
// SubscriberToken errors already carry the full
|
||||
// "pubsub::subscriber_token: …" wording, so surface them verbatim.
|
||||
handle
|
||||
.block_on(async move { svc.mint_subscriber_token(&cx, names, ttl).await })
|
||||
.map_err(|err| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(format!("{err}").into(), rhai::Position::NONE).into()
|
||||
})
|
||||
}
|
||||
|
||||
/// Convert a Rhai `Dynamic` message into JSON, base64-encoding any
|
||||
/// `Blob` (at any nesting depth). Mirrors `bridge::dynamic_to_json` but
|
||||
/// adds the blob arm the pub/sub wire contract requires.
|
||||
fn message_to_json(value: &Dynamic) -> Json {
|
||||
// Blob must be checked before the generic array path (a Blob is a
|
||||
// `Vec<u8>`, distinct from a Rhai `Array`).
|
||||
if value.is_blob() {
|
||||
let blob = value.clone().into_blob().unwrap_or_default();
|
||||
return Json::String(STANDARD.encode(&blob));
|
||||
}
|
||||
if value.is_unit() {
|
||||
return Json::Null;
|
||||
}
|
||||
if let Ok(b) = value.as_bool() {
|
||||
return Json::Bool(b);
|
||||
}
|
||||
if let Ok(i) = value.as_int() {
|
||||
return Json::Number(i.into());
|
||||
}
|
||||
if let Ok(f) = value.as_float() {
|
||||
return serde_json::Number::from_f64(f).map_or(Json::Null, Json::Number);
|
||||
}
|
||||
if value.is_string() {
|
||||
return Json::String(value.clone().into_string().unwrap_or_default());
|
||||
}
|
||||
if let Some(arr) = value.clone().try_cast::<Array>() {
|
||||
return Json::Array(arr.iter().map(message_to_json).collect());
|
||||
}
|
||||
if let Some(map) = value.clone().try_cast::<Map>() {
|
||||
let mut out = serde_json::Map::new();
|
||||
for (k, v) in map {
|
||||
out.insert(k.to_string(), message_to_json(&v));
|
||||
}
|
||||
return Json::Object(out);
|
||||
}
|
||||
Json::String(value.to_string())
|
||||
}
|
||||
|
||||
/// Run an async future inside the synchronous Rhai context. Mirrors
|
||||
/// `kv::block_on`.
|
||||
fn block_on<F>(fut: F) -> Result<(), Box<EvalAltResult>>
|
||||
where
|
||||
F: std::future::Future<Output = Result<(), PubsubError>> + Send,
|
||||
{
|
||||
let handle = TokioHandle::try_current().map_err(|e| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(
|
||||
format!("pubsub: no tokio runtime available: {e}").into(),
|
||||
rhai::Position::NONE,
|
||||
)
|
||||
.into()
|
||||
})?;
|
||||
handle.block_on(fut).map_err(|err| -> Box<EvalAltResult> {
|
||||
EvalAltResult::ErrorRuntime(format!("pubsub: {err}").into(), rhai::Position::NONE).into()
|
||||
})
|
||||
}
|
||||
129
crates/executor-core/tests/module_redaction_logging.rs
Normal file
129
crates/executor-core/tests/module_redaction_logging.rs
Normal file
@@ -0,0 +1,129 @@
|
||||
//! v1.1.4 §10a: the original module backend error MUST be logged at
|
||||
//! error level (so operators can still diagnose), even though it is
|
||||
//! redacted from the script-visible error.
|
||||
//!
|
||||
//! This test owns the process-global tracing subscriber, so it lives in
|
||||
//! its own integration-test binary (one `set_global_default` per
|
||||
//! process). A unique sentinel in the backend error keeps the assertion
|
||||
//! robust against any concurrently-running test's log output.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::io::Write;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, ModuleScript, ModuleSource, ModuleSourceError, NoopDeadLetterService,
|
||||
NoopDocsService, NoopEventEmitter, NoopHttpService, NoopKvService, RequestId, ScriptId,
|
||||
ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::Value;
|
||||
use tracing_subscriber::fmt::MakeWriter;
|
||||
|
||||
const SENTINEL: &str = "connection refused PICLOUD-SENTINEL-9f3a";
|
||||
|
||||
struct FailingSource;
|
||||
|
||||
#[async_trait]
|
||||
impl ModuleSource for FailingSource {
|
||||
async fn lookup(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_name: &str,
|
||||
) -> Result<Option<ModuleScript>, ModuleSourceError> {
|
||||
Err(ModuleSourceError::Backend(SENTINEL.to_string()))
|
||||
}
|
||||
}
|
||||
|
||||
/// `MakeWriter` that appends to a shared buffer.
|
||||
#[derive(Clone)]
|
||||
struct SharedBuf(Arc<Mutex<Vec<u8>>>);
|
||||
|
||||
impl Write for SharedBuf {
|
||||
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
|
||||
self.0.lock().unwrap().extend_from_slice(buf);
|
||||
Ok(buf.len())
|
||||
}
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl<'a> MakeWriter<'a> for SharedBuf {
|
||||
type Writer = SharedBuf;
|
||||
fn make_writer(&'a self) -> Self::Writer {
|
||||
self.clone()
|
||||
}
|
||||
}
|
||||
|
||||
fn req(app_id: AppId) -> ExecRequest {
|
||||
let execution_id = ExecutionId::new();
|
||||
ExecRequest {
|
||||
execution_id,
|
||||
request_id: RequestId::new(),
|
||||
script_id: ScriptId::new(),
|
||||
script_name: "redaction-test".into(),
|
||||
invocation_type: InvocationType::Http,
|
||||
path: "/x".into(),
|
||||
headers: BTreeMap::new(),
|
||||
body: Value::Null,
|
||||
params: BTreeMap::new(),
|
||||
query: BTreeMap::new(),
|
||||
rest: String::new(),
|
||||
sandbox_overrides: ScriptSandbox::default(),
|
||||
app_id,
|
||||
principal: None,
|
||||
trigger_depth: 0,
|
||||
root_execution_id: execution_id,
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn original_backend_error_is_logged_at_error_level() {
|
||||
let buf = Arc::new(Mutex::new(Vec::<u8>::new()));
|
||||
let subscriber = tracing_subscriber::fmt()
|
||||
.with_writer(SharedBuf(buf.clone()))
|
||||
.with_max_level(tracing::Level::ERROR)
|
||||
.with_ansi(false)
|
||||
.finish();
|
||||
tracing::subscriber::set_global_default(subscriber)
|
||||
.expect("this test owns the global subscriber for its binary");
|
||||
|
||||
let services = Services::new(
|
||||
Arc::new(NoopKvService),
|
||||
Arc::new(NoopDocsService),
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(FailingSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(picloud_shared::NoopFilesService),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
);
|
||||
let engine = Engine::new(Limits::default(), services);
|
||||
|
||||
let err = engine
|
||||
.execute(r#"import "x" as x; 1"#, req(AppId::new()))
|
||||
.expect_err("backend error should surface");
|
||||
|
||||
// Script-visible: redacted.
|
||||
let msg = format!("{err:?}");
|
||||
assert!(msg.contains("module backend unavailable"), "got {msg}");
|
||||
assert!(
|
||||
!msg.contains("PICLOUD-SENTINEL"),
|
||||
"script error leaked the original: {msg}"
|
||||
);
|
||||
|
||||
// Operator log: the original sentinel IS present, at ERROR level.
|
||||
let logged = String::from_utf8(buf.lock().unwrap().clone()).unwrap();
|
||||
assert!(
|
||||
logged.contains(SENTINEL),
|
||||
"original backend error should be logged; captured: {logged}"
|
||||
);
|
||||
assert!(
|
||||
logged.contains("ERROR"),
|
||||
"should be logged at error level; captured: {logged}"
|
||||
);
|
||||
}
|
||||
@@ -17,8 +17,8 @@ use chrono::{DateTime, Utc};
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, ModuleScript, ModuleSource, ModuleSourceError, NoopDeadLetterService,
|
||||
NoopDocsService, NoopEventEmitter, NoopKvService, RequestId, ScriptId, ScriptSandbox,
|
||||
SdkCallCx, Services,
|
||||
NoopDocsService, NoopEventEmitter, NoopHttpService, NoopKvService, RequestId, ScriptId,
|
||||
ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
@@ -96,6 +96,9 @@ fn services_with(modules: Arc<dyn ModuleSource>) -> Services {
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
modules,
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(picloud_shared::NoopFilesService),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
)
|
||||
}
|
||||
|
||||
@@ -321,20 +324,28 @@ async fn resolver_runtime_validation_rejects_top_level_expr() {
|
||||
);
|
||||
}
|
||||
|
||||
/// v1.1.4 §10a regression: the backend error must be REDACTED before
|
||||
/// it reaches a script. The verbatim message (which can leak internal
|
||||
/// infrastructure shape, e.g. "connection refused") must not appear;
|
||||
/// the script sees only a stable generic.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn resolver_backend_error_surfaces() {
|
||||
async fn resolver_backend_error_is_redacted_from_script() {
|
||||
let source = CountingModuleSource::new();
|
||||
let app_id = AppId::new();
|
||||
*source.fail_with.lock().await = Some("simulated db outage".into());
|
||||
*source.fail_with.lock().await = Some("connection refused to 10.1.2.3:5432".into());
|
||||
let engine = engine_with(source);
|
||||
|
||||
let err = engine
|
||||
.execute(r#"import "x" as x; 1"#, req(app_id))
|
||||
.expect_err("backend error should propagate");
|
||||
let msg = format!("{err:?}").to_lowercase();
|
||||
let msg = format!("{err:?}");
|
||||
assert!(
|
||||
msg.contains("simulated") || msg.contains("backend"),
|
||||
"expected backend-error message, got {msg}"
|
||||
msg.contains("module backend unavailable"),
|
||||
"expected redacted generic message, got {msg}"
|
||||
);
|
||||
assert!(
|
||||
!msg.contains("connection refused") && !msg.contains("10.1.2.3"),
|
||||
"redacted message must not leak the backend error, got {msg}"
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -11,8 +11,8 @@ use chrono::Utc;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, DocId, DocRow, DocsError, DocsListPage, DocsService, ExecutionId, NoopDeadLetterService,
|
||||
NoopEventEmitter, NoopKvService, NoopModuleSource, RequestId, ScriptId, ScriptSandbox,
|
||||
SdkCallCx, Services,
|
||||
NoopEventEmitter, NoopHttpService, NoopKvService, NoopModuleSource, RequestId, ScriptId,
|
||||
ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use tokio::sync::Mutex;
|
||||
@@ -227,6 +227,9 @@ fn make_engine() -> Arc<Engine> {
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(picloud_shared::NoopFilesService),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
334
crates/executor-core/tests/sdk_files.rs
Normal file
334
crates/executor-core/tests/sdk_files.rs
Normal file
@@ -0,0 +1,334 @@
|
||||
//! `files::` SDK bridge integration tests — runs a real Rhai engine
|
||||
//! against an in-memory `FilesService` impl. Mirrors `tests/sdk_kv.rs`:
|
||||
//! `tokio::task::spawn_blocking` so the bridge's `block_on` has a
|
||||
//! reachable runtime. Exercises the actual Rhai surface — blob in/out,
|
||||
//! the metadata map shape, and the missing-required-field throw.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, FileMeta, FileUpdate, FilesError, FilesListPage, FilesService, NewFile,
|
||||
NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopHttpService, NoopKvService,
|
||||
NoopModuleSource, RequestId, ScriptId, ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use tokio::sync::Mutex;
|
||||
use uuid::Uuid;
|
||||
|
||||
#[derive(Default)]
|
||||
struct InMemoryFiles {
|
||||
#[allow(clippy::type_complexity)]
|
||||
data: Mutex<BTreeMap<(AppId, String, Uuid), (FileMeta, Vec<u8>)>>,
|
||||
}
|
||||
|
||||
/// The in-memory fake doesn't exercise the real checksum path (the
|
||||
/// `FsFilesRepo` tempdir tests in manager-core cover SHA-256); a stable
|
||||
/// placeholder keeps the metadata map non-empty.
|
||||
fn fake_checksum(bytes: &[u8]) -> String {
|
||||
format!("len-{}", bytes.len())
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl FilesService for InMemoryFiles {
|
||||
async fn create(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<Uuid, FilesError> {
|
||||
if collection.is_empty() {
|
||||
return Err(FilesError::InvalidCollection("empty".into()));
|
||||
}
|
||||
new.validate(100 * 1024 * 1024)?;
|
||||
let id = Uuid::new_v4();
|
||||
let now = chrono::Utc::now();
|
||||
let meta = FileMeta {
|
||||
id,
|
||||
collection: collection.to_string(),
|
||||
name: new.name.clone(),
|
||||
content_type: new.content_type.clone(),
|
||||
size: new.data.len() as u64,
|
||||
checksum: fake_checksum(&new.data),
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
};
|
||||
self.data
|
||||
.lock()
|
||||
.await
|
||||
.insert((cx.app_id, collection.to_string(), id), (meta, new.data));
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<FileMeta>, FilesError> {
|
||||
let Ok(uuid) = Uuid::parse_str(id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.get(&(cx.app_id, collection.to_string(), uuid))
|
||||
.map(|(m, _)| m.clone()))
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<Vec<u8>>, FilesError> {
|
||||
let Ok(uuid) = Uuid::parse_str(id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.get(&(cx.app_id, collection.to_string(), uuid))
|
||||
.map(|(_, b)| b.clone()))
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
upd: FileUpdate,
|
||||
) -> Result<(), FilesError> {
|
||||
upd.validate(100 * 1024 * 1024)?;
|
||||
let Ok(uuid) = Uuid::parse_str(id) else {
|
||||
return Err(FilesError::NotFound);
|
||||
};
|
||||
let mut data = self.data.lock().await;
|
||||
let key = (cx.app_id, collection.to_string(), uuid);
|
||||
let Some((meta, _)) = data.get(&key).cloned() else {
|
||||
return Err(FilesError::NotFound);
|
||||
};
|
||||
let mut meta = meta;
|
||||
if let Some(n) = upd.name {
|
||||
meta.name = n;
|
||||
}
|
||||
if let Some(ct) = upd.content_type {
|
||||
meta.content_type = ct;
|
||||
}
|
||||
meta.size = upd.data.len() as u64;
|
||||
meta.checksum = fake_checksum(&upd.data);
|
||||
data.insert(key, (meta, upd.data));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError> {
|
||||
let Ok(uuid) = Uuid::parse_str(id) else {
|
||||
return Ok(false);
|
||||
};
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.remove(&(cx.app_id, collection.to_string(), uuid))
|
||||
.is_some())
|
||||
}
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
_cursor: Option<&str>,
|
||||
_limit: u32,
|
||||
) -> Result<FilesListPage, FilesError> {
|
||||
let data = self.data.lock().await;
|
||||
let files: Vec<FileMeta> = data
|
||||
.iter()
|
||||
.filter(|((a, c, _), _)| *a == cx.app_id && c == collection)
|
||||
.map(|(_, (m, _))| m.clone())
|
||||
.collect();
|
||||
Ok(FilesListPage {
|
||||
files,
|
||||
next_cursor: None,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn make_engine() -> Arc<Engine> {
|
||||
let services = Services::new(
|
||||
Arc::new(NoopKvService),
|
||||
Arc::new(NoopDocsService),
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(InMemoryFiles::default()),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
fn baseline_request(app_id: AppId) -> ExecRequest {
|
||||
let execution_id = ExecutionId::new();
|
||||
ExecRequest {
|
||||
execution_id,
|
||||
request_id: RequestId::new(),
|
||||
script_id: ScriptId::new(),
|
||||
script_name: "files-test".into(),
|
||||
invocation_type: InvocationType::Http,
|
||||
path: "/files-test".into(),
|
||||
headers: BTreeMap::new(),
|
||||
body: Value::Null,
|
||||
params: BTreeMap::new(),
|
||||
query: BTreeMap::new(),
|
||||
rest: String::new(),
|
||||
sandbox_overrides: ScriptSandbox::default(),
|
||||
app_id,
|
||||
principal: None,
|
||||
trigger_depth: 0,
|
||||
root_execution_id: execution_id,
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_script(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
|
||||
let src = src.to_string();
|
||||
tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic")
|
||||
.expect("script execution should succeed")
|
||||
.body
|
||||
}
|
||||
|
||||
async fn run_script_err(engine: Arc<Engine>, src: &str, req: ExecRequest) -> String {
|
||||
let src = src.to_string();
|
||||
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic");
|
||||
format!("{:?}", res.expect_err("script should error"))
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_create_get_round_trip_via_blob() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
// base64("hello") = "aGVsbG8="; decode → blob; create; get back; encode.
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
let data = base64::decode("aGVsbG8=");
|
||||
let id = c.create(#{ name: "a.txt", content_type: "text/plain", data: data });
|
||||
let back = c.get(id);
|
||||
base64::encode(back)
|
||||
"#;
|
||||
let body = run_script(engine, src, baseline_request(app)).await;
|
||||
assert_eq!(body, json!("aGVsbG8="));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_head_returns_metadata_map() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
let data = base64::decode("aGVsbG8=");
|
||||
let id = c.create(#{ name: "a.txt", content_type: "text/plain", data: data });
|
||||
let meta = c.head(id);
|
||||
#{ name: meta.name, content_type: meta.content_type, size: meta.size, has_checksum: meta.checksum != () }
|
||||
"#;
|
||||
let body = run_script(engine, src, baseline_request(app)).await;
|
||||
assert_eq!(
|
||||
body,
|
||||
json!({ "name": "a.txt", "content_type": "text/plain", "size": 5, "has_checksum": true })
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_get_and_head_missing_return_unit() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
let g = c.get("00000000-0000-0000-0000-000000000000");
|
||||
let h = c.head("00000000-0000-0000-0000-000000000000");
|
||||
#{ g: g == (), h: h == () }
|
||||
"#;
|
||||
let body = run_script(engine, src, baseline_request(app)).await;
|
||||
assert_eq!(body, json!({ "g": true, "h": true }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_update_then_delete() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
let id = c.create(#{ name: "a", content_type: "text/plain", data: base64::decode("YQ==") });
|
||||
c.update(id, #{ data: base64::decode("YmM=") }); // "bc"
|
||||
let after = base64::encode(c.get(id));
|
||||
let removed = c.delete(id);
|
||||
let gone = c.delete(id);
|
||||
#{ after: after, removed: removed, gone: gone }
|
||||
"#;
|
||||
let body = run_script(engine, src, baseline_request(app)).await;
|
||||
assert_eq!(
|
||||
body,
|
||||
json!({ "after": "YmM=", "removed": true, "gone": false })
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_create_missing_data_throws_naming_field() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
c.create(#{ name: "a", content_type: "text/plain" })
|
||||
"#;
|
||||
let err = run_script_err(engine, src, baseline_request(app)).await;
|
||||
assert!(
|
||||
err.contains("data"),
|
||||
"error should name the missing field: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_create_missing_name_throws_naming_field() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
c.create(#{ content_type: "text/plain", data: base64::decode("YQ==") })
|
||||
"#;
|
||||
let err = run_script_err(engine, src, baseline_request(app)).await;
|
||||
assert!(
|
||||
err.contains("name"),
|
||||
"error should name the missing field: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_empty_collection_name_throws() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let err = run_script_err(engine, r#"files::collection("")"#, baseline_request(app)).await;
|
||||
assert!(err.to_lowercase().contains("empty"), "got {err}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn files_list_returns_files_array() {
|
||||
let engine = make_engine();
|
||||
let app = AppId::new();
|
||||
let src = r#"
|
||||
let c = files::collection("avatars");
|
||||
c.create(#{ name: "a", content_type: "text/plain", data: base64::decode("YQ==") });
|
||||
c.create(#{ name: "b", content_type: "text/plain", data: base64::decode("Yg==") });
|
||||
let page = c.list();
|
||||
page.files.len()
|
||||
"#;
|
||||
let body = run_script(engine, src, baseline_request(app)).await;
|
||||
assert_eq!(body, json!(2));
|
||||
}
|
||||
336
crates/executor-core/tests/sdk_http.rs
Normal file
336
crates/executor-core/tests/sdk_http.rs
Normal file
@@ -0,0 +1,336 @@
|
||||
//! Bridge integration for the `http::*` SDK (v1.1.4).
|
||||
//!
|
||||
//! Runs a real Rhai engine under `spawn_blocking` against an in-memory
|
||||
//! `HttpService` fake that records the last request and returns a
|
||||
//! configured response (or error). This exercises the full bridge:
|
||||
//! option parsing, body dispatch, response→map projection, the
|
||||
//! throw-on-network-error / no-throw-on-non-2xx convention, and that
|
||||
//! `cx.app_id` / `cx.script_id` are forwarded for attribution.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, HttpError, HttpRequest, HttpResponse, HttpService, NoopDeadLetterService,
|
||||
NoopDocsService, NoopEventEmitter, NoopKvService, NoopModuleSource, RequestId, ScriptId,
|
||||
ScriptSandbox, Services,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
/// What the fake returns. Either a canned response or an error.
|
||||
#[derive(Clone)]
|
||||
enum Behavior {
|
||||
Respond(HttpResponse),
|
||||
Fail(String), // becomes HttpError::Network
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct Recorded {
|
||||
last: Option<HttpRequest>,
|
||||
last_app: Option<AppId>,
|
||||
last_script: Option<String>,
|
||||
}
|
||||
|
||||
struct FakeHttp {
|
||||
behavior: Behavior,
|
||||
recorded: Mutex<Recorded>,
|
||||
}
|
||||
|
||||
impl FakeHttp {
|
||||
fn responding(status: u16, content_type: &str, body: &str) -> Arc<Self> {
|
||||
let mut headers = BTreeMap::new();
|
||||
headers.insert("content-type".into(), content_type.into());
|
||||
Arc::new(Self {
|
||||
behavior: Behavior::Respond(HttpResponse {
|
||||
status,
|
||||
headers,
|
||||
body_raw: body.into(),
|
||||
}),
|
||||
recorded: Mutex::new(Recorded::default()),
|
||||
})
|
||||
}
|
||||
|
||||
fn failing(msg: &str) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
behavior: Behavior::Fail(msg.into()),
|
||||
recorded: Mutex::new(Recorded::default()),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl HttpService for FakeHttp {
|
||||
async fn request(
|
||||
&self,
|
||||
cx: &picloud_shared::SdkCallCx,
|
||||
req: HttpRequest,
|
||||
) -> Result<HttpResponse, HttpError> {
|
||||
{
|
||||
let mut r = self.recorded.lock().unwrap();
|
||||
r.last = Some(req.clone());
|
||||
r.last_app = Some(cx.app_id);
|
||||
r.last_script = Some(cx.script_id.to_string());
|
||||
}
|
||||
match &self.behavior {
|
||||
Behavior::Respond(resp) => Ok(resp.clone()),
|
||||
Behavior::Fail(msg) => Err(HttpError::Network(msg.clone())),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn engine_with(http: Arc<dyn HttpService>) -> Arc<Engine> {
|
||||
let services = Services::new(
|
||||
Arc::new(NoopKvService),
|
||||
Arc::new(NoopDocsService),
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
http,
|
||||
Arc::new(picloud_shared::NoopFilesService),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
fn baseline_request(app_id: AppId, script_id: ScriptId) -> ExecRequest {
|
||||
let execution_id = ExecutionId::new();
|
||||
ExecRequest {
|
||||
execution_id,
|
||||
request_id: RequestId::new(),
|
||||
script_id,
|
||||
script_name: "http-test".into(),
|
||||
invocation_type: InvocationType::Http,
|
||||
path: "/http-test".into(),
|
||||
headers: BTreeMap::new(),
|
||||
body: Value::Null,
|
||||
params: BTreeMap::new(),
|
||||
query: BTreeMap::new(),
|
||||
rest: String::new(),
|
||||
sandbox_overrides: ScriptSandbox::default(),
|
||||
app_id,
|
||||
principal: None,
|
||||
trigger_depth: 0,
|
||||
root_execution_id: execution_id,
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
|
||||
let src = src.to_string();
|
||||
tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic")
|
||||
.expect("script execution should succeed")
|
||||
.body
|
||||
}
|
||||
|
||||
async fn run_err(engine: Arc<Engine>, src: &str, req: ExecRequest) -> String {
|
||||
let src = src.to_string();
|
||||
let err = tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.unwrap()
|
||||
.expect_err("script should throw");
|
||||
format!("{err:?}")
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn get_returns_status_and_json_body() {
|
||||
let http = FakeHttp::responding(200, "application/json", r#"{"ok":true,"n":7}"#);
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"
|
||||
let r = http::get("https://api.example.com/x");
|
||||
#{ status: r.status, ok: r.body.ok, n: r.body.n }
|
||||
"#;
|
||||
let body = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert_eq!(body, json!({ "status": 200, "ok": true, "n": 7 }));
|
||||
// GET carries no body.
|
||||
assert!(http
|
||||
.recorded
|
||||
.lock()
|
||||
.unwrap()
|
||||
.last
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.body
|
||||
.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn non_json_body_stays_string() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "plain text");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"http::get("https://x/").body"#;
|
||||
let body = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert_eq!(body, json!("plain text"));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn empty_body_is_unit() {
|
||||
let http = FakeHttp::responding(204, "text/plain", "");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"
|
||||
let r = http::get("https://x/");
|
||||
#{ is_unit: r.body == (), raw: r.body_raw }
|
||||
"#;
|
||||
let body = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert_eq!(body, json!({ "is_unit": true, "raw": "" }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn post_map_body_is_json_encoded() {
|
||||
let http = FakeHttp::responding(200, "application/json", "{}");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"http::post("https://hooks/x", #{ text: "hello", n: 3 }).status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
let rec = http.recorded.lock().unwrap();
|
||||
let req = rec.last.as_ref().unwrap();
|
||||
assert_eq!(req.method, "POST");
|
||||
assert_eq!(req.content_type.as_deref(), Some("application/json"));
|
||||
let sent: Value = serde_json::from_slice(req.body.as_ref().unwrap()).unwrap();
|
||||
assert_eq!(sent, json!({ "text": "hello", "n": 3 }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn post_string_body_is_text_plain() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"http::post("https://x/", "raw payload").status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
let rec = http.recorded.lock().unwrap();
|
||||
let req = rec.last.as_ref().unwrap();
|
||||
assert_eq!(req.content_type.as_deref(), Some("text/plain"));
|
||||
assert_eq!(req.body.as_deref(), Some(&b"raw payload"[..]));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn post_unit_body_sends_nothing() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"http::post("https://x/", ()).status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert!(http
|
||||
.recorded
|
||||
.lock()
|
||||
.unwrap()
|
||||
.last
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.body
|
||||
.is_none());
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn custom_headers_and_timeout_forwarded() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"
|
||||
http::get("https://x/", #{
|
||||
headers: #{ "Authorization": "Bearer t0ken" },
|
||||
timeout_ms: 4200,
|
||||
}).status
|
||||
"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
let rec = http.recorded.lock().unwrap();
|
||||
let req = rec.last.as_ref().unwrap();
|
||||
assert_eq!(
|
||||
req.headers.get("Authorization").map(String::as_str),
|
||||
Some("Bearer t0ken")
|
||||
);
|
||||
assert_eq!(req.timeout_ms, 4200);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn unknown_option_key_throws() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"http::get("https://x/", #{ timeoutms: 1000 })"#; // typo
|
||||
let err = run_err(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert!(err.contains("unknown option key"), "got {err}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn timeout_above_max_throws() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"http::get("https://x/", #{ timeout_ms: 99999 })"#;
|
||||
let err = run_err(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert!(err.contains("maximum"), "got {err}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn non_2xx_does_not_throw() {
|
||||
let http = FakeHttp::responding(503, "text/plain", "down");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"http::get("https://x/").status"#;
|
||||
let body = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert_eq!(body, json!(503));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn network_error_throws_with_http_prefix() {
|
||||
let http = FakeHttp::failing("connection refused");
|
||||
let engine = engine_with(http);
|
||||
let src = r#"http::get("https://x/")"#;
|
||||
let err = run_err(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert!(err.contains("http:"), "expected http: prefix, got {err}");
|
||||
assert!(err.contains("connection refused"), "got {err}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn post_form_url_encodes() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"http::post_form("https://x/login", #{ user: "alice", pw: "p@ss word" }).status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
let rec = http.recorded.lock().unwrap();
|
||||
let req = rec.last.as_ref().unwrap();
|
||||
assert_eq!(
|
||||
req.content_type.as_deref(),
|
||||
Some("application/x-www-form-urlencoded")
|
||||
);
|
||||
let body = String::from_utf8(req.body.clone().unwrap()).unwrap();
|
||||
// order is map iteration order; assert both pairs present, encoded.
|
||||
assert!(body.contains("user=alice"), "got {body}");
|
||||
assert!(body.contains("pw=p%40ss+word"), "got {body}");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn request_escape_hatch_arbitrary_method() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let src = r#"http::request("OPTIONS", "https://x/").status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), ScriptId::new())).await;
|
||||
assert_eq!(
|
||||
http.recorded.lock().unwrap().last.as_ref().unwrap().method,
|
||||
"OPTIONS"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn default_user_agent_carries_script_id() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let script_id = ScriptId::new();
|
||||
let src = r#"http::get("https://x/").status"#;
|
||||
let _ = run(engine, src, baseline_request(AppId::new(), script_id)).await;
|
||||
let rec = http.recorded.lock().unwrap();
|
||||
// The bridge forwards script_id on the request; the manager-core
|
||||
// impl turns it into the User-Agent. Here we assert the forward.
|
||||
assert_eq!(
|
||||
rec.last.as_ref().unwrap().script_id.as_deref(),
|
||||
Some(script_id.to_string().as_str())
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn cx_app_id_forwarded_for_attribution() {
|
||||
let http = FakeHttp::responding(200, "text/plain", "ok");
|
||||
let engine = engine_with(http.clone());
|
||||
let app = AppId::new();
|
||||
let src = r#"http::get("https://x/").status"#;
|
||||
let _ = run(engine, src, baseline_request(app, ScriptId::new())).await;
|
||||
assert_eq!(http.recorded.lock().unwrap().last_app, Some(app));
|
||||
}
|
||||
@@ -11,7 +11,8 @@ use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, KvError, KvListPage, KvService, NoopDeadLetterService, NoopDocsService,
|
||||
NoopEventEmitter, NoopModuleSource, RequestId, ScriptId, ScriptSandbox, SdkCallCx, Services,
|
||||
NoopEventEmitter, NoopHttpService, NoopModuleSource, RequestId, ScriptId, ScriptSandbox,
|
||||
SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
use tokio::sync::Mutex;
|
||||
@@ -105,6 +106,9 @@ fn make_engine() -> Arc<Engine> {
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(picloud_shared::NoopFilesService),
|
||||
Arc::new(picloud_shared::NoopPubsubService),
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
157
crates/executor-core/tests/sdk_pubsub.rs
Normal file
157
crates/executor-core/tests/sdk_pubsub.rs
Normal file
@@ -0,0 +1,157 @@
|
||||
//! `pubsub::` SDK bridge integration tests — runs a real Rhai engine
|
||||
//! against an in-memory `PubsubService` that records the published
|
||||
//! `(topic, message)`. Verifies the message JSON encoding the wire
|
||||
//! contract requires: Maps, Arrays, strings, numbers, bool, null, and
|
||||
//! **Blob → base64**, including nesting.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::{
|
||||
AppId, ExecutionId, NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopFilesService,
|
||||
NoopHttpService, NoopKvService, NoopModuleSource, PubsubError, PubsubService, RequestId,
|
||||
ScriptId, ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::{json, Value};
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingPubsub {
|
||||
last: Mutex<Option<(String, Value)>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubService for RecordingPubsub {
|
||||
async fn publish_durable(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
topic: &str,
|
||||
message: Value,
|
||||
) -> Result<(), PubsubError> {
|
||||
if topic.trim().is_empty() {
|
||||
return Err(PubsubError::EmptyTopic);
|
||||
}
|
||||
*self.last.lock().unwrap() = Some((topic.to_string(), message));
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn make_engine(svc: Arc<RecordingPubsub>) -> Arc<Engine> {
|
||||
let services = Services::new(
|
||||
Arc::new(NoopKvService),
|
||||
Arc::new(NoopDocsService),
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(NoopFilesService),
|
||||
svc,
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
fn baseline_request(app_id: AppId) -> ExecRequest {
|
||||
let execution_id = ExecutionId::new();
|
||||
ExecRequest {
|
||||
execution_id,
|
||||
request_id: RequestId::new(),
|
||||
script_id: ScriptId::new(),
|
||||
script_name: "pubsub-test".into(),
|
||||
invocation_type: InvocationType::Http,
|
||||
path: "/pubsub-test".into(),
|
||||
headers: BTreeMap::new(),
|
||||
body: Value::Null,
|
||||
params: BTreeMap::new(),
|
||||
query: BTreeMap::new(),
|
||||
rest: String::new(),
|
||||
sandbox_overrides: ScriptSandbox::default(),
|
||||
app_id,
|
||||
principal: None,
|
||||
trigger_depth: 0,
|
||||
root_execution_id: execution_id,
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run(engine: Arc<Engine>, src: &str, req: ExecRequest) {
|
||||
let src = src.to_string();
|
||||
tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic")
|
||||
.expect("script execution should succeed");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_map_message() {
|
||||
let svc = Arc::new(RecordingPubsub::default());
|
||||
let engine = make_engine(svc.clone());
|
||||
run(
|
||||
engine,
|
||||
r#"pubsub::publish_durable("user.created", #{ user_id: "abc", n: 7, ok: true });"#,
|
||||
baseline_request(AppId::new()),
|
||||
)
|
||||
.await;
|
||||
let (topic, msg) = svc.last.lock().unwrap().clone().unwrap();
|
||||
assert_eq!(topic, "user.created");
|
||||
assert_eq!(msg, json!({ "user_id": "abc", "n": 7, "ok": true }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_scalar_and_array_and_null() {
|
||||
let svc = Arc::new(RecordingPubsub::default());
|
||||
let engine = make_engine(svc.clone());
|
||||
run(
|
||||
engine,
|
||||
r#"pubsub::publish_durable("a", [1, "two", false, ()]);"#,
|
||||
baseline_request(AppId::new()),
|
||||
)
|
||||
.await;
|
||||
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
|
||||
assert_eq!(msg, json!([1, "two", false, null]));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_number_scalar() {
|
||||
let svc = Arc::new(RecordingPubsub::default());
|
||||
let engine = make_engine(svc.clone());
|
||||
run(
|
||||
engine,
|
||||
r#"pubsub::publish_durable("metric", 42);"#,
|
||||
baseline_request(AppId::new()),
|
||||
)
|
||||
.await;
|
||||
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
|
||||
assert_eq!(msg, json!(42));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_blob_encodes_base64_including_nested() {
|
||||
let svc = Arc::new(RecordingPubsub::default());
|
||||
let engine = make_engine(svc.clone());
|
||||
// base64("hello") = "aGVsbG8=" (STANDARD, padded).
|
||||
run(
|
||||
engine,
|
||||
r#"
|
||||
let data = base64::decode("aGVsbG8=");
|
||||
pubsub::publish_durable("blobs", #{ raw: data, list: [data] });
|
||||
"#,
|
||||
baseline_request(AppId::new()),
|
||||
)
|
||||
.await;
|
||||
let (_t, msg) = svc.last.lock().unwrap().clone().unwrap();
|
||||
assert_eq!(msg, json!({ "raw": "aGVsbG8=", "list": ["aGVsbG8="] }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_empty_topic_throws() {
|
||||
let svc = Arc::new(RecordingPubsub::default());
|
||||
let engine = make_engine(svc.clone());
|
||||
let src = r#"pubsub::publish_durable("", 1);"#.to_string();
|
||||
let req = baseline_request(AppId::new());
|
||||
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic");
|
||||
assert!(res.is_err(), "empty topic should throw");
|
||||
}
|
||||
242
crates/executor-core/tests/sdk_subscriber_token.rs
Normal file
242
crates/executor-core/tests/sdk_subscriber_token.rs
Normal file
@@ -0,0 +1,242 @@
|
||||
//! `pubsub::subscriber_token` SDK bridge integration tests (v1.1.6).
|
||||
//!
|
||||
//! Runs a real Rhai engine against a fake `PubsubService` whose
|
||||
//! `mint_subscriber_token` mirrors the production validation (principal
|
||||
//! required, non-empty topics, ttl clamp, externally-subscribable check)
|
||||
//! and signs a real token. These cover the bridge surface: array →
|
||||
//! `Vec<String>` forwarding, the omitted/`()`/integer ttl handling, and
|
||||
//! errors surfacing as thrown Rhai errors. The authoritative validation
|
||||
//! logic is unit-tested in `manager-core::pubsub_service`.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_executor_core::{Engine, ExecRequest, InvocationType, Limits};
|
||||
use picloud_shared::subscriber_token::{self, TokenClaims};
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, ExecutionId, InstanceRole, NoopDeadLetterService, NoopDocsService,
|
||||
NoopEventEmitter, NoopFilesService, NoopHttpService, NoopKvService, NoopModuleSource,
|
||||
Principal, PubsubError, PubsubService, RequestId, ScriptId, ScriptSandbox, SdkCallCx, Services,
|
||||
};
|
||||
use serde_json::Value;
|
||||
|
||||
const FAKE_KEY: [u8; 32] = [7u8; 32];
|
||||
const MIN_TTL: i64 = 10;
|
||||
const MAX_TTL: i64 = 86_400;
|
||||
const DEFAULT_TTL: i64 = 3_600;
|
||||
|
||||
/// Fake that mirrors the production mint rules and signs with FAKE_KEY.
|
||||
#[derive(Default)]
|
||||
struct FakeMintPubsub;
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubService for FakeMintPubsub {
|
||||
async fn publish_durable(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_topic: &str,
|
||||
_message: Value,
|
||||
) -> Result<(), PubsubError> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn mint_subscriber_token(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
topics: Vec<String>,
|
||||
ttl_seconds: Option<i64>,
|
||||
) -> Result<String, PubsubError> {
|
||||
if cx.principal.is_none() {
|
||||
return Err(PubsubError::SubscriberToken(
|
||||
"pubsub::subscriber_token: requires an authenticated principal".into(),
|
||||
));
|
||||
}
|
||||
if topics.is_empty() {
|
||||
return Err(PubsubError::SubscriberToken(
|
||||
"pubsub::subscriber_token: topics list must not be empty".into(),
|
||||
));
|
||||
}
|
||||
let ttl = ttl_seconds.unwrap_or(DEFAULT_TTL);
|
||||
if !(MIN_TTL..=MAX_TTL).contains(&ttl) {
|
||||
return Err(PubsubError::SubscriberToken(format!(
|
||||
"pubsub::subscriber_token: ttl_seconds must be between {MIN_TTL} and {MAX_TTL}"
|
||||
)));
|
||||
}
|
||||
for name in &topics {
|
||||
// Only "chat" and "notify" are "registered" in this fake.
|
||||
if name != "chat" && name != "notify" {
|
||||
return Err(PubsubError::SubscriberToken(format!(
|
||||
"pubsub::subscriber_token: topic {name} is not externally subscribable"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let now = 1_000_000;
|
||||
Ok(subscriber_token::sign(
|
||||
&FAKE_KEY,
|
||||
&TokenClaims {
|
||||
app_id: cx.app_id,
|
||||
topics,
|
||||
exp: now + ttl,
|
||||
iat: now,
|
||||
},
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
fn make_engine() -> Arc<Engine> {
|
||||
let services = Services::new(
|
||||
Arc::new(NoopKvService),
|
||||
Arc::new(NoopDocsService),
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(NoopFilesService),
|
||||
Arc::new(FakeMintPubsub),
|
||||
);
|
||||
Arc::new(Engine::new(Limits::default(), services))
|
||||
}
|
||||
|
||||
fn request(app_id: AppId, with_principal: bool) -> ExecRequest {
|
||||
let execution_id = ExecutionId::new();
|
||||
ExecRequest {
|
||||
execution_id,
|
||||
request_id: RequestId::new(),
|
||||
script_id: ScriptId::new(),
|
||||
script_name: "token-test".into(),
|
||||
invocation_type: InvocationType::Http,
|
||||
path: "/token-test".into(),
|
||||
headers: BTreeMap::new(),
|
||||
body: Value::Null,
|
||||
params: BTreeMap::new(),
|
||||
query: BTreeMap::new(),
|
||||
rest: String::new(),
|
||||
sandbox_overrides: ScriptSandbox::default(),
|
||||
app_id,
|
||||
principal: with_principal.then(|| Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Owner,
|
||||
scopes: None,
|
||||
app_binding: None,
|
||||
}),
|
||||
trigger_depth: 0,
|
||||
root_execution_id: execution_id,
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
async fn run_ok(engine: Arc<Engine>, src: &str, req: ExecRequest) -> Value {
|
||||
let src = src.to_string();
|
||||
tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic")
|
||||
.expect("script execution should succeed")
|
||||
.body
|
||||
}
|
||||
|
||||
async fn run_err(engine: Arc<Engine>, src: &str, req: ExecRequest) {
|
||||
let src = src.to_string();
|
||||
let res = tokio::task::spawn_blocking(move || engine.execute(&src, req))
|
||||
.await
|
||||
.expect("spawn_blocking should not panic");
|
||||
assert!(res.is_err(), "expected script to throw");
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn token_contains_topics_and_expiry() {
|
||||
let app = AppId::new();
|
||||
let body = run_ok(
|
||||
make_engine(),
|
||||
r#"#{ token: pubsub::subscriber_token(["chat", "notify"], 120) }"#,
|
||||
request(app, true),
|
||||
)
|
||||
.await;
|
||||
let token = body["token"].as_str().expect("token string");
|
||||
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
|
||||
assert_eq!(claims.app_id, app);
|
||||
assert_eq!(
|
||||
claims.topics,
|
||||
vec!["chat".to_string(), "notify".to_string()]
|
||||
);
|
||||
assert_eq!(claims.exp - claims.iat, 120);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn omitted_ttl_uses_default() {
|
||||
let app = AppId::new();
|
||||
let body = run_ok(
|
||||
make_engine(),
|
||||
r#"#{ token: pubsub::subscriber_token(["chat"]) }"#,
|
||||
request(app, true),
|
||||
)
|
||||
.await;
|
||||
let token = body["token"].as_str().unwrap();
|
||||
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
|
||||
assert_eq!(claims.exp - claims.iat, DEFAULT_TTL);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn unit_ttl_uses_default() {
|
||||
let app = AppId::new();
|
||||
let body = run_ok(
|
||||
make_engine(),
|
||||
r#"#{ token: pubsub::subscriber_token(["chat"], ()) }"#,
|
||||
request(app, true),
|
||||
)
|
||||
.await;
|
||||
let token = body["token"].as_str().unwrap();
|
||||
let claims = subscriber_token::verify(&FAKE_KEY, token, 1_000_001).unwrap();
|
||||
assert_eq!(claims.exp - claims.iat, DEFAULT_TTL);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn empty_topics_throws() {
|
||||
run_err(
|
||||
make_engine(),
|
||||
r#"pubsub::subscriber_token([], 60)"#,
|
||||
request(AppId::new(), true),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn ttl_below_min_throws() {
|
||||
run_err(
|
||||
make_engine(),
|
||||
r#"pubsub::subscriber_token(["chat"], 5)"#,
|
||||
request(AppId::new(), true),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn ttl_above_max_throws() {
|
||||
run_err(
|
||||
make_engine(),
|
||||
r#"pubsub::subscriber_token(["chat"], 90000)"#,
|
||||
request(AppId::new(), true),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn anonymous_principal_throws() {
|
||||
run_err(
|
||||
make_engine(),
|
||||
r#"pubsub::subscriber_token(["chat"], 60)"#,
|
||||
request(AppId::new(), false),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn unregistered_topic_throws() {
|
||||
run_err(
|
||||
make_engine(),
|
||||
r#"pubsub::subscriber_token(["chat", "secret"], 60)"#,
|
||||
request(AppId::new(), true),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
@@ -23,8 +23,11 @@ tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
uuid.workspace = true
|
||||
chrono.workspace = true
|
||||
chrono-tz.workspace = true
|
||||
cron.workspace = true
|
||||
sqlx.workspace = true
|
||||
url.workspace = true
|
||||
reqwest.workspace = true
|
||||
|
||||
argon2.workspace = true
|
||||
sha2.workspace = true
|
||||
|
||||
43
crates/manager-core/migrations/0017_cron_triggers.sql
Normal file
43
crates/manager-core/migrations/0017_cron_triggers.sql
Normal file
@@ -0,0 +1,43 @@
|
||||
-- v1.1.4: Extend the triggers framework to recognise `cron` as the
|
||||
-- fourth concrete kind (after `kv` v1.1.1, `dead_letter` v1.1.1, `docs`
|
||||
-- v1.1.2). Mirrors the 0014 docs extension: two CHECK constraints widen
|
||||
-- (strictly gaining `'cron'`), one new detail table.
|
||||
--
|
||||
-- Cron rows route through the SAME generic dispatcher path as kv/docs/
|
||||
-- dead_letter (single match-arm extension on the Rust side). The only
|
||||
-- new machinery is a scheduler task that enqueues due cron triggers
|
||||
-- into the outbox; dispatch itself is unchanged.
|
||||
|
||||
-- Extend triggers.kind to include 'cron'. No existing row carries a
|
||||
-- value outside the widened set, so the drop+add is safe.
|
||||
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
|
||||
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
|
||||
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron'));
|
||||
|
||||
-- Extend outbox.source_kind to include 'cron'. v1.1.x's existing
|
||||
-- source_kinds ('http', 'kv', 'dead_letter', 'docs') stay.
|
||||
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
|
||||
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
|
||||
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs', 'cron'));
|
||||
|
||||
-- One row per cron trigger.
|
||||
-- schedule — 6-field cron expression (with seconds), validated
|
||||
-- at insert time by the `cron` crate.
|
||||
-- timezone — IANA tz name (e.g. "America/Los_Angeles"), validated
|
||||
-- via chrono-tz. Required so schedules like "every
|
||||
-- weekday at 9am" are unambiguous. Defaults to UTC.
|
||||
-- last_fired_at — set transactionally with each enqueue. NULL until
|
||||
-- the trigger first fires. The scheduler computes the
|
||||
-- next fire time in-process from
|
||||
-- (schedule, timezone, last_fired_at); there is no
|
||||
-- stored next_fire column (kept stateless on purpose).
|
||||
CREATE TABLE cron_trigger_details (
|
||||
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
|
||||
schedule TEXT NOT NULL,
|
||||
timezone TEXT NOT NULL DEFAULT 'UTC',
|
||||
last_fired_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
-- Hot lookup for the scheduler: "all enabled cron triggers due now"
|
||||
-- scans by last_fired_at.
|
||||
CREATE INDEX idx_cron_triggers_due ON cron_trigger_details (last_fired_at);
|
||||
25
crates/manager-core/migrations/0018_files.sql
Normal file
25
crates/manager-core/migrations/0018_files.sql
Normal file
@@ -0,0 +1,25 @@
|
||||
-- v1.1.5: filesystem-backed blob storage. The row holds metadata +
|
||||
-- the SHA-256 checksum; the blob bytes live on disk at
|
||||
-- <PICLOUD_FILES_ROOT>/files/<app_id>/<collection>/<id[0:2]>/<id>
|
||||
-- (never in Postgres). Identity tuple is (app_id, collection, id) per
|
||||
-- docs/sdk-shape.md, matching KV/docs collection scoping.
|
||||
--
|
||||
-- The checksum is computed in a single pass during the atomic write and
|
||||
-- re-verified on read (FilesError::Corrupted on mismatch). Per-app
|
||||
-- quotas are deferred to v1.2; only the per-file size cap is enforced
|
||||
-- (in the service, not the schema).
|
||||
CREATE TABLE files (
|
||||
app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
|
||||
collection TEXT NOT NULL,
|
||||
id UUID NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
checksum_sha256 TEXT NOT NULL, -- hex, 64 chars, lowercase
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (app_id, collection, id)
|
||||
);
|
||||
|
||||
-- List + cursor pagination scans by (app_id, collection).
|
||||
CREATE INDEX idx_files_app_collection ON files (app_id, collection);
|
||||
29
crates/manager-core/migrations/0019_files_triggers.sql
Normal file
29
crates/manager-core/migrations/0019_files_triggers.sql
Normal file
@@ -0,0 +1,29 @@
|
||||
-- v1.1.5: extend the triggers framework to recognise `files` as the
|
||||
-- fifth concrete kind (after `kv`/`dead_letter` v1.1.1, `docs` v1.1.2,
|
||||
-- `cron` v1.1.4). Mirrors the 0014/0017 extensions exactly: two CHECK
|
||||
-- constraints widen (strictly gaining `'files'`), one new detail table.
|
||||
--
|
||||
-- Files rows route through the SAME generic dispatcher path as the
|
||||
-- other event kinds (single match-arm extension on the Rust side). The
|
||||
-- only new machinery is the FilesServiceImpl emitting ServiceEvents
|
||||
-- that the OutboxEventEmitter fans out — identical to KV/docs.
|
||||
|
||||
-- Extend triggers.kind to include 'files'. No existing row carries a
|
||||
-- value outside the widened set, so the drop+add is safe.
|
||||
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
|
||||
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
|
||||
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron', 'files'));
|
||||
|
||||
-- Extend outbox.source_kind to include 'files'.
|
||||
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
|
||||
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
|
||||
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs', 'cron', 'files'));
|
||||
|
||||
-- One row per files trigger. Mirrors kv_trigger_details:
|
||||
-- collection_glob — "*", "exact", or "prefix*"
|
||||
-- ops — subset of {create, update, delete}, empty = any
|
||||
CREATE TABLE files_trigger_details (
|
||||
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
|
||||
collection_glob TEXT NOT NULL,
|
||||
ops TEXT[] NOT NULL
|
||||
);
|
||||
34
crates/manager-core/migrations/0020_pubsub_triggers.sql
Normal file
34
crates/manager-core/migrations/0020_pubsub_triggers.sql
Normal file
@@ -0,0 +1,34 @@
|
||||
-- v1.1.5: extend the triggers framework to recognise `pubsub` as the
|
||||
-- sixth concrete kind. Same Layout-E shape as files (0019): two CHECK
|
||||
-- constraints widen, one new detail table.
|
||||
--
|
||||
-- Pub/sub fans out at PUBLISH time (one outbox row per matching trigger,
|
||||
-- written by the PubsubServiceImpl), so the dispatcher needs no pubsub-
|
||||
-- specific branching — a pubsub outbox row dispatches like any other
|
||||
-- async trigger.
|
||||
|
||||
-- Extend triggers.kind to include 'pubsub'.
|
||||
ALTER TABLE triggers DROP CONSTRAINT triggers_kind_check;
|
||||
ALTER TABLE triggers ADD CONSTRAINT triggers_kind_check
|
||||
CHECK (kind IN ('kv', 'dead_letter', 'docs', 'cron', 'files', 'pubsub'));
|
||||
|
||||
-- Extend outbox.source_kind to include 'pubsub'.
|
||||
ALTER TABLE outbox DROP CONSTRAINT outbox_source_kind_check;
|
||||
ALTER TABLE outbox ADD CONSTRAINT outbox_source_kind_check
|
||||
CHECK (source_kind IN ('http', 'kv', 'dead_letter', 'docs',
|
||||
'cron', 'files', 'pubsub'));
|
||||
|
||||
-- One row per pubsub trigger. `topic_pattern` is "exact", "prefix.*",
|
||||
-- or "*" — validated in Rust at trigger creation. Topics are implicit
|
||||
-- on first publish; the external-subscribable `topics` table is v1.1.6.
|
||||
CREATE TABLE pubsub_trigger_details (
|
||||
trigger_id UUID PRIMARY KEY REFERENCES triggers(id) ON DELETE CASCADE,
|
||||
topic_pattern TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- Hot lookup for fan-out: "all enabled pubsub triggers in app X".
|
||||
-- Third partial index of its kind (after v1.1.1's idx_triggers_app_kind_
|
||||
-- enabled); partial indexes are tiny and the planner picks the narrowest.
|
||||
CREATE INDEX idx_triggers_app_pubsub_enabled
|
||||
ON triggers (app_id, kind)
|
||||
WHERE enabled = TRUE AND kind = 'pubsub';
|
||||
31
crates/manager-core/migrations/0021_topics.sql
Normal file
31
crates/manager-core/migrations/0021_topics.sql
Normal file
@@ -0,0 +1,31 @@
|
||||
-- v1.1.6: Explicit registration for externally-subscribable topics.
|
||||
--
|
||||
-- Internal-only topics remain implicit per the §5 design-notes
|
||||
-- decision: anyone can publish_durable("any.topic", msg) and triggers
|
||||
-- can subscribe without a row here. This table only holds topics that
|
||||
-- have been explicitly externalized — external SSE subscribers can
|
||||
-- only subscribe to topics with a row here AND external_subscribable
|
||||
-- = TRUE.
|
||||
--
|
||||
-- The publish path (v1.1.5's publish_durable) does NOT consult this
|
||||
-- table: publishing to a topic with no row still fans out to triggers
|
||||
-- and to any in-process external subscribers (none exist for an
|
||||
-- unregistered topic, since external subscribers can't subscribe to
|
||||
-- one). The topics table is read by the SSE subscribe path only.
|
||||
--
|
||||
-- auth_mode values: 'public' + 'token' in v1.1.6. 'session' arrives in
|
||||
-- v1.1.8 (users-SDK); 'script' arrives in v1.2 (script-mediated auth).
|
||||
-- The CHECK constraint extends in those releases.
|
||||
CREATE TABLE topics (
|
||||
app_id UUID NOT NULL REFERENCES apps(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
external_subscribable BOOL NOT NULL DEFAULT FALSE,
|
||||
auth_mode TEXT NOT NULL DEFAULT 'public'
|
||||
CHECK (auth_mode IN ('public', 'token')),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (app_id, name)
|
||||
);
|
||||
|
||||
-- Hot lookup: "is topic T in app X externally subscribable?" The PK
|
||||
-- (app_id, name) already covers this; an explicit index is redundant.
|
||||
19
crates/manager-core/migrations/0022_app_secrets.sql
Normal file
19
crates/manager-core/migrations/0022_app_secrets.sql
Normal file
@@ -0,0 +1,19 @@
|
||||
-- v1.1.6: per-app secret material. Currently holds the HMAC signing key
|
||||
-- used to mint + verify realtime subscriber tokens
|
||||
-- (pubsub::subscriber_token → SSE /realtime/topics handshake).
|
||||
--
|
||||
-- The key is:
|
||||
-- * stable across restarts (issued tokens stay valid until expiry),
|
||||
-- * per-app (a token signed by app A is rejected by app B),
|
||||
-- * never script-accessible (scripts can't print/exfiltrate it — the
|
||||
-- SDK only mints tokens, it never returns the key).
|
||||
--
|
||||
-- The row is created lazily on the first pubsub::subscriber_token call
|
||||
-- for an app (32 random bytes). This table is the natural home for
|
||||
-- v1.1.7's encrypted per-app secrets work.
|
||||
CREATE TABLE app_secrets (
|
||||
app_id UUID PRIMARY KEY REFERENCES apps(id) ON DELETE CASCADE,
|
||||
realtime_signing_key BYTEA NOT NULL, -- 32 random bytes
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
91
crates/manager-core/src/app_secrets_repo.rs
Normal file
91
crates/manager-core/src/app_secrets_repo.rs
Normal file
@@ -0,0 +1,91 @@
|
||||
//! `AppSecretsRepo` — per-app secret material (v1.1.6).
|
||||
//!
|
||||
//! Today this holds only the HMAC signing key for realtime subscriber
|
||||
//! tokens. The key is generated lazily (32 random bytes) on the first
|
||||
//! `pubsub::subscriber_token` call for an app and never changes
|
||||
//! thereafter in v1.1.6 (no rotation API yet — rotation is the
|
||||
//! key-invalidation mechanism, deferred). The key is never exposed to
|
||||
//! scripts: the SDK mints tokens, it never returns the key.
|
||||
//!
|
||||
//! This table is the natural home for v1.1.7's encrypted per-app
|
||||
//! secrets work.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::AppId;
|
||||
use rand::RngCore;
|
||||
use sqlx::PgPool;
|
||||
|
||||
/// Length of a freshly-generated realtime signing key.
|
||||
pub const SIGNING_KEY_LEN: usize = 32;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum AppSecretsRepoError {
|
||||
#[error("database error: {0}")]
|
||||
Db(#[from] sqlx::Error),
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait AppSecretsRepo: Send + Sync {
|
||||
/// Fetch the app's realtime signing key, generating + persisting one
|
||||
/// (32 random bytes) if absent. Idempotent under concurrency: a
|
||||
/// racing creator's `ON CONFLICT DO NOTHING` insert is a no-op and
|
||||
/// the existing key is returned.
|
||||
async fn get_or_create_signing_key(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
) -> Result<Vec<u8>, AppSecretsRepoError>;
|
||||
|
||||
/// Fetch the signing key if it exists, WITHOUT creating one. The SSE
|
||||
/// verify path uses this: a missing key means no token was ever
|
||||
/// minted for the app, so any presented token must be rejected.
|
||||
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError>;
|
||||
}
|
||||
|
||||
pub struct PostgresAppSecretsRepo {
|
||||
pool: PgPool,
|
||||
}
|
||||
|
||||
impl PostgresAppSecretsRepo {
|
||||
#[must_use]
|
||||
pub fn new(pool: PgPool) -> Self {
|
||||
Self { pool }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl AppSecretsRepo for PostgresAppSecretsRepo {
|
||||
async fn get_or_create_signing_key(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
) -> Result<Vec<u8>, AppSecretsRepoError> {
|
||||
let mut fresh = vec![0u8; SIGNING_KEY_LEN];
|
||||
rand::thread_rng().fill_bytes(&mut fresh);
|
||||
|
||||
// Insert-if-absent then read: the racing-creator's insert is a
|
||||
// no-op, and the SELECT always returns the winning key.
|
||||
sqlx::query(
|
||||
"INSERT INTO app_secrets (app_id, realtime_signing_key) \
|
||||
VALUES ($1, $2) ON CONFLICT (app_id) DO NOTHING",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(&fresh)
|
||||
.execute(&self.pool)
|
||||
.await?;
|
||||
|
||||
let key: (Vec<u8>,) =
|
||||
sqlx::query_as("SELECT realtime_signing_key FROM app_secrets WHERE app_id = $1")
|
||||
.bind(app_id.into_inner())
|
||||
.fetch_one(&self.pool)
|
||||
.await?;
|
||||
Ok(key.0)
|
||||
}
|
||||
|
||||
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
|
||||
let row: Option<(Vec<u8>,)> =
|
||||
sqlx::query_as("SELECT realtime_signing_key FROM app_secrets WHERE app_id = $1")
|
||||
.bind(app_id.into_inner())
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
Ok(row.map(|r| r.0))
|
||||
}
|
||||
}
|
||||
@@ -72,6 +72,23 @@ pub enum Capability {
|
||||
/// shape as KV write — granted to `editor`+, maps to
|
||||
/// `script:write` on API keys.
|
||||
AppDocsWrite(AppId),
|
||||
/// Make an outbound HTTP request from a script in this app
|
||||
/// (v1.1.4). Maps to `script:write` on API keys: any outbound
|
||||
/// request can exfiltrate data — including read methods like GET —
|
||||
/// so the conservative write mapping is correct. Splitting
|
||||
/// read/write is a v1.2+ refinement. Granted to `editor`+.
|
||||
AppHttpRequest(AppId),
|
||||
/// Read blobs from this app's files store (v1.1.5). Same trust
|
||||
/// shape as KV/docs read — granted to `viewer`+, maps to
|
||||
/// `script:read` on API keys. Honors the seven-scope commitment.
|
||||
AppFilesRead(AppId),
|
||||
/// Write blobs to this app's files store (v1.1.5). Granted to
|
||||
/// `editor`+, maps to `script:write` on API keys.
|
||||
AppFilesWrite(AppId),
|
||||
/// Publish a durable pub/sub message from a script in this app
|
||||
/// (v1.1.5). Maps to `script:write` on API keys (a publish is a
|
||||
/// write that fans out to subscribers). Granted to `editor`+.
|
||||
AppPubsubPublish(AppId),
|
||||
/// Create / list / delete triggers for this app (v1.1.1). Maps to
|
||||
/// `app:admin` on API keys — triggers are app-configuration acts
|
||||
/// rather than data-plane access. Granted to `app_admin`+.
|
||||
@@ -80,6 +97,12 @@ pub enum Capability {
|
||||
/// to `app:admin` on API keys. Public-HTTP scripts (principal None)
|
||||
/// fail this check — managing dead letters is an admin act.
|
||||
AppDeadLetterManage(AppId),
|
||||
/// Register / list / update / delete externally-subscribable topics
|
||||
/// for this app (v1.1.6). Maps to `app:admin` on API keys —
|
||||
/// externalizing a topic is an app-configuration act with security
|
||||
/// weight (it opens an internal pub/sub topic to outside SSE
|
||||
/// subscribers). Granted to `app_admin`+.
|
||||
AppTopicManage(AppId),
|
||||
}
|
||||
|
||||
impl Capability {
|
||||
@@ -101,8 +124,13 @@ impl Capability {
|
||||
| Self::AppKvWrite(id)
|
||||
| Self::AppDocsRead(id)
|
||||
| Self::AppDocsWrite(id)
|
||||
| Self::AppHttpRequest(id)
|
||||
| Self::AppFilesRead(id)
|
||||
| Self::AppFilesWrite(id)
|
||||
| Self::AppPubsubPublish(id)
|
||||
| Self::AppManageTriggers(id)
|
||||
| Self::AppDeadLetterManage(id) => Some(id),
|
||||
| Self::AppDeadLetterManage(id)
|
||||
| Self::AppTopicManage(id) => Some(id),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -117,15 +145,22 @@ impl Capability {
|
||||
Self::InstanceCreateApp | Self::InstanceManageUsers | Self::InstanceManageSettings => {
|
||||
Scope::InstanceAdmin
|
||||
}
|
||||
Self::AppRead(_) | Self::AppKvRead(_) | Self::AppDocsRead(_) => Scope::ScriptRead,
|
||||
Self::AppWriteScript(_) | Self::AppKvWrite(_) | Self::AppDocsWrite(_) => {
|
||||
Scope::ScriptWrite
|
||||
}
|
||||
Self::AppRead(_)
|
||||
| Self::AppKvRead(_)
|
||||
| Self::AppDocsRead(_)
|
||||
| Self::AppFilesRead(_) => Scope::ScriptRead,
|
||||
Self::AppWriteScript(_)
|
||||
| Self::AppKvWrite(_)
|
||||
| Self::AppDocsWrite(_)
|
||||
| Self::AppHttpRequest(_)
|
||||
| Self::AppFilesWrite(_)
|
||||
| Self::AppPubsubPublish(_) => Scope::ScriptWrite,
|
||||
Self::AppWriteRoute(_) => Scope::RouteWrite,
|
||||
Self::AppManageDomains(_) => Scope::DomainManage,
|
||||
Self::AppAdmin(_) | Self::AppManageTriggers(_) | Self::AppDeadLetterManage(_) => {
|
||||
Scope::AppAdmin
|
||||
}
|
||||
Self::AppAdmin(_)
|
||||
| Self::AppManageTriggers(_)
|
||||
| Self::AppDeadLetterManage(_)
|
||||
| Self::AppTopicManage(_) => Scope::AppAdmin,
|
||||
Self::AppLogRead(_) => Scope::LogRead,
|
||||
}
|
||||
}
|
||||
@@ -269,6 +304,7 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
|
||||
| Capability::AppLogRead(_)
|
||||
| Capability::AppKvRead(_)
|
||||
| Capability::AppDocsRead(_)
|
||||
| Capability::AppFilesRead(_)
|
||||
);
|
||||
let in_editor = in_viewer
|
||||
|| matches!(
|
||||
@@ -277,6 +313,9 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
|
||||
| Capability::AppWriteRoute(_)
|
||||
| Capability::AppKvWrite(_)
|
||||
| Capability::AppDocsWrite(_)
|
||||
| Capability::AppHttpRequest(_)
|
||||
| Capability::AppFilesWrite(_)
|
||||
| Capability::AppPubsubPublish(_)
|
||||
);
|
||||
let in_app_admin = in_editor
|
||||
|| matches!(
|
||||
@@ -285,6 +324,7 @@ const fn role_satisfies(role: AppRole, cap: Capability) -> bool {
|
||||
| Capability::AppAdmin(_)
|
||||
| Capability::AppManageTriggers(_)
|
||||
| Capability::AppDeadLetterManage(_)
|
||||
| Capability::AppTopicManage(_)
|
||||
);
|
||||
match role {
|
||||
AppRole::Viewer => in_viewer,
|
||||
@@ -628,6 +668,35 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn topic_manage_requires_app_admin() {
|
||||
let repo = InMemoryAuthzRepo::default();
|
||||
let app = AppId::new();
|
||||
// Maps to the app:admin scope, not a new one.
|
||||
assert_eq!(
|
||||
Capability::AppTopicManage(app).required_scope(),
|
||||
Scope::AppAdmin
|
||||
);
|
||||
|
||||
// Member with only Editor role cannot manage topics.
|
||||
let p = principal(InstanceRole::Member);
|
||||
repo.grant(p.user_id, app, AppRole::Editor).await;
|
||||
assert_eq!(
|
||||
can(&repo, &p, Capability::AppTopicManage(app))
|
||||
.await
|
||||
.unwrap(),
|
||||
Decision::Deny,
|
||||
);
|
||||
|
||||
// App-admin role can.
|
||||
let admin = principal(InstanceRole::Member);
|
||||
repo.grant(admin.user_id, app, AppRole::AppAdmin).await;
|
||||
assert!(can(&repo, &admin, Capability::AppTopicManage(app))
|
||||
.await
|
||||
.unwrap()
|
||||
.is_allow());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn capability_app_id_extraction() {
|
||||
let app = AppId::new();
|
||||
|
||||
297
crates/manager-core/src/cron_scheduler.rs
Normal file
297
crates/manager-core/src/cron_scheduler.rs
Normal file
@@ -0,0 +1,297 @@
|
||||
//! Cron scheduler — the v1.1.4 time-based trigger source.
|
||||
//!
|
||||
//! A single tokio task polls `cron_trigger_details` on a tick (default
|
||||
//! 30s; `PICLOUD_CRON_TICK_INTERVAL_MS`). For each enabled cron trigger
|
||||
//! whose next scheduled fire is due, it enqueues ONE outbox row
|
||||
//! (`source_kind = 'cron'`) and updates `last_fired_at` — both in the
|
||||
//! same transaction, claimed via `FOR UPDATE SKIP LOCKED` so a future
|
||||
//! multi-node deploy can't double-fire.
|
||||
//!
|
||||
//! The scheduler does NOT dispatch or touch the `ExecutionGate`: it only
|
||||
//! enqueues. The existing dispatcher picks the row up and acquires the
|
||||
//! gate exactly as it does for kv/docs/dead_letter rows.
|
||||
//!
|
||||
//! **Catch-up policy (matches the brief):** a trigger that missed N fire
|
||||
//! windows since `last_fired_at` fires exactly ONCE on the next tick,
|
||||
//! not N times. This falls out of the design: [`next_due`] returns a
|
||||
//! single canonical scheduled time (the first slot after the reference
|
||||
//! point), and after firing we set `last_fired_at = now`, so the next
|
||||
//! tick computes from `now` and sees only future slots. Backfilling
|
||||
//! missed windows is intentionally out of scope (an explicit replay
|
||||
//! action is the v1.2+ escape hatch).
|
||||
|
||||
use std::str::FromStr;
|
||||
use std::time::Duration;
|
||||
|
||||
use chrono::{DateTime, Utc};
|
||||
use chrono_tz::Tz;
|
||||
use cron::Schedule;
|
||||
use picloud_shared::TriggerEvent;
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Validate a 6-field cron expression. Returns the parse error message
|
||||
/// on failure.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns the underlying parse error string when `schedule` is not a
|
||||
/// valid cron expression.
|
||||
pub fn validate_schedule(schedule: &str) -> Result<(), String> {
|
||||
Schedule::from_str(schedule)
|
||||
.map(|_| ())
|
||||
.map_err(|e| e.to_string())
|
||||
}
|
||||
|
||||
/// Validate an IANA timezone name (e.g. `America/Los_Angeles`).
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns an error string when `timezone` is not a known IANA name.
|
||||
pub fn validate_timezone(timezone: &str) -> Result<(), String> {
|
||||
Tz::from_str(timezone)
|
||||
.map(|_| ())
|
||||
.map_err(|_| format!("unknown IANA timezone: {timezone}"))
|
||||
}
|
||||
|
||||
/// Compute whether a cron trigger is due, and if so its canonical
|
||||
/// scheduled-at moment (UTC).
|
||||
///
|
||||
/// Returns `Some(scheduled_at)` when the first scheduled slot after the
|
||||
/// reference point (`last_fired_at`, or `created_at` if never fired) is
|
||||
/// at/before `now`; `None` otherwise. Returns `None` if the schedule or
|
||||
/// timezone fails to parse (the row is skipped — it should never have
|
||||
/// been inserted, since the admin endpoint validates).
|
||||
#[must_use]
|
||||
pub fn next_due(
|
||||
schedule: &str,
|
||||
timezone: &str,
|
||||
last_fired_at: Option<DateTime<Utc>>,
|
||||
created_at: DateTime<Utc>,
|
||||
now: DateTime<Utc>,
|
||||
) -> Option<DateTime<Utc>> {
|
||||
let sched = Schedule::from_str(schedule).ok()?;
|
||||
let tz = Tz::from_str(timezone).ok()?;
|
||||
// Reference: the last actual fire, or creation if never fired. A
|
||||
// never-fired trigger fires at its first slot at/after creation.
|
||||
let base = last_fired_at.unwrap_or(created_at);
|
||||
let base_tz = base.with_timezone(&tz);
|
||||
let next = sched.after(&base_tz).next()?;
|
||||
let next_utc = next.with_timezone(&Utc);
|
||||
(next_utc <= now).then_some(next_utc)
|
||||
}
|
||||
|
||||
/// Spawn the scheduler loop. Runs for the process lifetime.
|
||||
pub fn spawn_cron_scheduler(pool: PgPool, tick_interval_ms: u32) {
|
||||
// Floor the tick at 1s so a misconfigured 0 can't spin.
|
||||
let interval = Duration::from_millis(u64::from(tick_interval_ms).max(1_000));
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(interval);
|
||||
// Skip the immediate first fire so we don't race startup.
|
||||
ticker.tick().await;
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
if let Err(e) = tick(&pool, Utc::now()).await {
|
||||
tracing::warn!(?e, "cron scheduler tick errored");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct DueRow {
|
||||
id: Uuid,
|
||||
app_id: Uuid,
|
||||
script_id: Uuid,
|
||||
registered_by_principal: Uuid,
|
||||
created_at: DateTime<Utc>,
|
||||
schedule: String,
|
||||
timezone: String,
|
||||
last_fired_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
/// One scheduler tick: claim enabled cron rows, enqueue the due ones,
|
||||
/// bump `last_fired_at`. Returns the number of triggers fired.
|
||||
async fn tick(pool: &PgPool, now: DateTime<Utc>) -> Result<usize, sqlx::Error> {
|
||||
let mut tx = pool.begin().await?;
|
||||
let rows: Vec<DueRow> = sqlx::query_as(
|
||||
"SELECT t.id, t.app_id, t.script_id, t.registered_by_principal, t.created_at, \
|
||||
d.schedule, d.timezone, d.last_fired_at \
|
||||
FROM cron_trigger_details d \
|
||||
JOIN triggers t ON t.id = d.trigger_id \
|
||||
WHERE t.enabled = TRUE \
|
||||
FOR UPDATE OF d SKIP LOCKED",
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
|
||||
let mut fired = 0usize;
|
||||
for r in rows {
|
||||
let Some(scheduled_at) =
|
||||
next_due(&r.schedule, &r.timezone, r.last_fired_at, r.created_at, now)
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
|
||||
let event = TriggerEvent::Cron {
|
||||
schedule: r.schedule.clone(),
|
||||
timezone: r.timezone.clone(),
|
||||
scheduled_at,
|
||||
fired_at: now,
|
||||
};
|
||||
let payload = serde_json::to_value(&event)
|
||||
.map_err(|e| sqlx::Error::Decode(Box::new(std::io::Error::other(e))))?;
|
||||
|
||||
// Enqueue exactly one outbox row. Relies on the same column
|
||||
// defaults the OutboxEventEmitter uses (next_attempt_at = NOW(),
|
||||
// attempt_count = 0, claimed_at NULL → immediately due).
|
||||
sqlx::query(
|
||||
"INSERT INTO outbox \
|
||||
(app_id, source_kind, trigger_id, script_id, payload, \
|
||||
origin_principal, trigger_depth) \
|
||||
VALUES ($1, 'cron', $2, $3, $4, $5, 0)",
|
||||
)
|
||||
.bind(r.app_id)
|
||||
.bind(r.id)
|
||||
.bind(r.script_id)
|
||||
.bind(payload)
|
||||
.bind(r.registered_by_principal)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
sqlx::query("UPDATE cron_trigger_details SET last_fired_at = $2 WHERE trigger_id = $1")
|
||||
.bind(r.id)
|
||||
.bind(now)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
fired += 1;
|
||||
}
|
||||
|
||||
tx.commit().await?;
|
||||
Ok(fired)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use chrono::TimeZone;
|
||||
|
||||
#[test]
|
||||
fn valid_six_field_schedule_accepted() {
|
||||
// sec min hour dom mon dow — "every weekday at 9am".
|
||||
validate_schedule("0 0 9 * * MON-FRI").unwrap();
|
||||
validate_schedule("*/5 * * * * *").unwrap();
|
||||
validate_schedule("0 0 0 1 1 *").unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_schedules_rejected() {
|
||||
// 5-field (no seconds) is not the format we accept.
|
||||
assert!(validate_schedule("* * * * *").is_err());
|
||||
// Gibberish.
|
||||
assert!(validate_schedule("not a cron").is_err());
|
||||
assert!(validate_schedule("").is_err());
|
||||
// Out-of-range hour.
|
||||
assert!(validate_schedule("0 0 99 * * *").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn known_timezones_accepted() {
|
||||
validate_timezone("UTC").unwrap();
|
||||
validate_timezone("America/Los_Angeles").unwrap();
|
||||
validate_timezone("Europe/Berlin").unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_timezones_rejected() {
|
||||
assert!(validate_timezone("Mars/Phobos").is_err());
|
||||
assert!(validate_timezone("PST").is_err()); // abbreviations aren't IANA names
|
||||
assert!(validate_timezone("").is_err());
|
||||
}
|
||||
|
||||
fn ts(s: &str) -> DateTime<Utc> {
|
||||
DateTime::parse_from_rfc3339(s).unwrap().with_timezone(&Utc)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn due_when_next_slot_is_at_or_before_now() {
|
||||
// Every minute at second 0. Last fired 90s ago → the next slot
|
||||
// after that is due now.
|
||||
let created = ts("2026-06-01T00:00:00Z");
|
||||
let last = Some(ts("2026-06-15T11:58:10Z"));
|
||||
let now = ts("2026-06-15T12:00:05Z");
|
||||
let due = next_due("0 * * * * *", "UTC", last, created, now);
|
||||
assert_eq!(due, Some(ts("2026-06-15T11:59:00Z")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn not_due_when_next_slot_is_in_the_future() {
|
||||
let created = ts("2026-06-01T00:00:00Z");
|
||||
let last = Some(ts("2026-06-15T12:00:00Z"));
|
||||
let now = ts("2026-06-15T12:00:30Z");
|
||||
// Next minute slot is 12:01:00 — still in the future.
|
||||
assert_eq!(next_due("0 * * * * *", "UTC", last, created, now), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn never_fired_uses_created_at_as_reference() {
|
||||
let created = ts("2026-06-15T12:00:10Z");
|
||||
let now = ts("2026-06-15T12:01:30Z");
|
||||
// First slot after creation is 12:01:00, which is <= now → due.
|
||||
let due = next_due("0 * * * * *", "UTC", None, created, now);
|
||||
assert_eq!(due, Some(ts("2026-06-15T12:01:00Z")));
|
||||
}
|
||||
|
||||
/// Catch-up policy: a trigger that missed many windows fires exactly
|
||||
/// ONCE. We simulate two consecutive scheduler ticks the way the DB
|
||||
/// loop does — fire once, set last_fired = now, then re-evaluate.
|
||||
#[test]
|
||||
fn catch_up_fires_exactly_once_after_missed_windows() {
|
||||
let created = ts("2026-06-15T09:00:00Z");
|
||||
// Last fired over 5 minutes (5 windows) ago.
|
||||
let mut last_fired = Some(ts("2026-06-15T11:54:30Z"));
|
||||
let now = ts("2026-06-15T12:00:05Z");
|
||||
|
||||
// Tick 1: due → fire once, advance last_fired to `now`.
|
||||
let first = next_due("0 * * * * *", "UTC", last_fired, created, now);
|
||||
assert!(first.is_some(), "should be due after missing windows");
|
||||
last_fired = Some(now);
|
||||
|
||||
// Tick 2 (same wall-clock): NOT due again — only one fire total,
|
||||
// not one-per-missed-window.
|
||||
let second = next_due("0 * * * * *", "UTC", last_fired, created, now);
|
||||
assert_eq!(second, None, "catch-up must fire exactly once");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn timezone_affects_fire_time() {
|
||||
// "9am every day" in Los Angeles. On 2026-06-15, PDT = UTC-7, so
|
||||
// 09:00 local = 16:00 UTC.
|
||||
let created = ts("2026-06-15T00:00:00Z");
|
||||
let last = Some(ts("2026-06-15T15:59:00Z"));
|
||||
let now = ts("2026-06-15T16:00:30Z");
|
||||
let due = next_due("0 0 9 * * *", "America/Los_Angeles", last, created, now);
|
||||
assert_eq!(due, Some(ts("2026-06-15T16:00:00Z")));
|
||||
// Sanity: the same expression in UTC would NOT be due at 16:00.
|
||||
assert_eq!(next_due("0 0 9 * * *", "UTC", last, created, now), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bad_schedule_or_tz_yields_none() {
|
||||
let created = ts("2026-06-15T00:00:00Z");
|
||||
let now = ts("2026-06-15T12:00:00Z");
|
||||
assert_eq!(next_due("garbage", "UTC", None, created, now), None);
|
||||
assert_eq!(
|
||||
next_due("0 * * * * *", "Mars/Phobos", None, created, now),
|
||||
None
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn utc_offset_constructor_smoke() {
|
||||
// Guard the chrono TimeZone import is actually exercised.
|
||||
let dt = Utc.with_ymd_and_hms(2026, 6, 15, 12, 0, 0).unwrap();
|
||||
assert_eq!(dt, ts("2026-06-15T12:00:00Z"));
|
||||
}
|
||||
}
|
||||
@@ -208,6 +208,9 @@ async fn resolve(
|
||||
fn admin_cx(app_id: AppId, principal: &Principal) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
// Admin-plane cx (dead-letter replay/resolve) — no script is
|
||||
// executing, so this attribution id is a fresh sentinel.
|
||||
script_id: picloud_shared::ScriptId::new(),
|
||||
principal: Some(principal.clone()),
|
||||
execution_id: picloud_shared::ExecutionId::new(),
|
||||
request_id: picloud_shared::RequestId::new(),
|
||||
|
||||
@@ -163,7 +163,12 @@ impl Dispatcher {
|
||||
return Ok(());
|
||||
}
|
||||
},
|
||||
OutboxSourceKind::Kv | OutboxSourceKind::Docs | OutboxSourceKind::DeadLetter => {
|
||||
OutboxSourceKind::Kv
|
||||
| OutboxSourceKind::Docs
|
||||
| OutboxSourceKind::DeadLetter
|
||||
| OutboxSourceKind::Cron
|
||||
| OutboxSourceKind::Files
|
||||
| OutboxSourceKind::Pubsub => {
|
||||
let resolved = self.resolve_trigger(&row).await?;
|
||||
let req = match self.build_exec_request(&row, &resolved).await {
|
||||
Ok(req) => req,
|
||||
|
||||
@@ -272,7 +272,7 @@ mod tests {
|
||||
use chrono::Utc;
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, NoopEventEmitter, Principal,
|
||||
RequestId, UserId,
|
||||
RequestId, ScriptId, UserId,
|
||||
};
|
||||
use serde_json::json;
|
||||
use std::collections::BTreeMap;
|
||||
@@ -507,6 +507,7 @@ mod tests {
|
||||
fn anon_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: None,
|
||||
execution_id: ExecutionId::new(),
|
||||
request_id: RequestId::new(),
|
||||
@@ -520,6 +521,7 @@ mod tests {
|
||||
fn owner_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Owner,
|
||||
@@ -538,6 +540,7 @@ mod tests {
|
||||
fn member_no_role_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
|
||||
215
crates/manager-core/src/files_api.rs
Normal file
215
crates/manager-core/src/files_api.rs
Normal file
@@ -0,0 +1,215 @@
|
||||
//! `/api/v1/admin/apps/{id}/files*` — minimal files admin endpoints
|
||||
//! backing the dashboard's files view (v1.1.5).
|
||||
//!
|
||||
//! Two operations only, both operator-facing:
|
||||
//! * `GET /apps/{id}/files?collection=<c>&cursor=&limit=` — list file
|
||||
//! metadata for a collection (cursor-paginated).
|
||||
//! * `DELETE /apps/{id}/files/{collection}/{file_id}` — remove a file.
|
||||
//!
|
||||
//! These talk to the `FilesRepo` directly (like `triggers_api` talks to
|
||||
//! `TriggerRepo`), guarded by the same capability model as the SDK
|
||||
//! (`AppFilesRead` / `AppFilesWrite`). **Admin deletes do NOT emit a
|
||||
//! `files:delete` trigger event** — they're operator cleanup actions,
|
||||
//! not script mutations (see HANDBACK §7). The capability binds to the
|
||||
//! resource's `app_id` after the app is loaded.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::StatusCode;
|
||||
use axum::response::{IntoResponse, Json, Response};
|
||||
use axum::routing::{delete, get};
|
||||
use axum::{Extension, Router};
|
||||
use picloud_shared::{AppId, Principal};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::json;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::app_repo::AppRepository;
|
||||
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
|
||||
use crate::files_repo::{FilesRepo, FilesRepoError};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct FilesAdminState {
|
||||
pub files: Arc<dyn FilesRepo>,
|
||||
pub apps: Arc<dyn AppRepository>,
|
||||
pub authz: Arc<dyn AuthzRepo>,
|
||||
}
|
||||
|
||||
pub fn files_admin_router(state: FilesAdminState) -> Router {
|
||||
Router::new()
|
||||
.route("/apps/{app_id}/files", get(list_files))
|
||||
.route(
|
||||
"/apps/{app_id}/files/{collection}/{file_id}",
|
||||
delete(delete_file),
|
||||
)
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct ListFilesQuery {
|
||||
pub collection: String,
|
||||
#[serde(default)]
|
||||
pub cursor: Option<String>,
|
||||
#[serde(default)]
|
||||
pub limit: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct FileMetaDto {
|
||||
id: String,
|
||||
collection: String,
|
||||
name: String,
|
||||
content_type: String,
|
||||
size: u64,
|
||||
checksum: String,
|
||||
created_at: String,
|
||||
updated_at: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize)]
|
||||
struct ListFilesResponse {
|
||||
files: Vec<FileMetaDto>,
|
||||
next_cursor: Option<String>,
|
||||
}
|
||||
|
||||
async fn list_files(
|
||||
State(s): State<FilesAdminState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
Query(q): Query<ListFilesQuery>,
|
||||
) -> Result<Json<ListFilesResponse>, FilesApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppFilesRead(app_id),
|
||||
)
|
||||
.await?;
|
||||
if q.collection.trim().is_empty() {
|
||||
return Err(FilesApiError::Invalid(
|
||||
"collection must not be empty".into(),
|
||||
));
|
||||
}
|
||||
let page = s
|
||||
.files
|
||||
.list(
|
||||
app_id,
|
||||
&q.collection,
|
||||
q.cursor.as_deref(),
|
||||
q.limit.unwrap_or(0),
|
||||
)
|
||||
.await?;
|
||||
let files = page
|
||||
.files
|
||||
.into_iter()
|
||||
.map(|m| FileMetaDto {
|
||||
id: m.id.to_string(),
|
||||
collection: m.collection,
|
||||
name: m.name,
|
||||
content_type: m.content_type,
|
||||
size: m.size,
|
||||
checksum: m.checksum,
|
||||
created_at: m.created_at.to_rfc3339(),
|
||||
updated_at: m.updated_at.to_rfc3339(),
|
||||
})
|
||||
.collect();
|
||||
Ok(Json(ListFilesResponse {
|
||||
files,
|
||||
next_cursor: page.next_cursor,
|
||||
}))
|
||||
}
|
||||
|
||||
async fn delete_file(
|
||||
State(s): State<FilesAdminState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path((app_id, collection, file_id)): Path<(AppId, String, String)>,
|
||||
) -> Result<StatusCode, FilesApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppFilesWrite(app_id),
|
||||
)
|
||||
.await?;
|
||||
let id = Uuid::parse_str(&file_id).map_err(|_| FilesApiError::NotFound)?;
|
||||
if s.files.delete(app_id, &collection, id).await?.is_none() {
|
||||
return Err(FilesApiError::NotFound);
|
||||
}
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
async fn ensure_app_exists(apps: &dyn AppRepository, app_id: AppId) -> Result<(), FilesApiError> {
|
||||
apps.get_by_id(app_id)
|
||||
.await
|
||||
.map_err(|e| FilesApiError::Backend(e.to_string()))?
|
||||
.ok_or(FilesApiError::AppNotFound)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum FilesApiError {
|
||||
#[error("app not found")]
|
||||
AppNotFound,
|
||||
#[error("file not found")]
|
||||
NotFound,
|
||||
#[error("invalid request: {0}")]
|
||||
Invalid(String),
|
||||
#[error("forbidden")]
|
||||
Forbidden,
|
||||
#[error("authorization repo error: {0}")]
|
||||
AuthzRepo(String),
|
||||
#[error("files backend: {0}")]
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
impl From<AuthzDenied> for FilesApiError {
|
||||
fn from(d: AuthzDenied) -> Self {
|
||||
match d {
|
||||
AuthzDenied::Denied => Self::Forbidden,
|
||||
AuthzDenied::Repo(e) => Self::AuthzRepo(e.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<AuthzError> for FilesApiError {
|
||||
fn from(e: AuthzError) -> Self {
|
||||
Self::AuthzRepo(e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
impl From<FilesRepoError> for FilesApiError {
|
||||
fn from(e: FilesRepoError) -> Self {
|
||||
Self::Backend(e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
impl IntoResponse for FilesApiError {
|
||||
fn into_response(self) -> Response {
|
||||
let (status, body) = match &self {
|
||||
Self::AppNotFound | Self::NotFound => {
|
||||
(StatusCode::NOT_FOUND, json!({ "error": self.to_string() }))
|
||||
}
|
||||
Self::Invalid(_) => (
|
||||
StatusCode::UNPROCESSABLE_ENTITY,
|
||||
json!({ "error": self.to_string() }),
|
||||
),
|
||||
Self::Forbidden => (StatusCode::FORBIDDEN, json!({ "error": self.to_string() })),
|
||||
Self::AuthzRepo(e) => {
|
||||
tracing::error!(error = %e, "files admin authz repo error");
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
json!({ "error": "internal error" }),
|
||||
)
|
||||
}
|
||||
Self::Backend(e) => {
|
||||
tracing::error!(error = %e, "files admin backend error");
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
json!({ "error": "internal error" }),
|
||||
)
|
||||
}
|
||||
};
|
||||
(status, Json(body)).into_response()
|
||||
}
|
||||
}
|
||||
759
crates/manager-core/src/files_repo.rs
Normal file
759
crates/manager-core/src/files_repo.rs
Normal file
@@ -0,0 +1,759 @@
|
||||
//! `FilesRepo` — the metadata row (Postgres) + blob bytes (filesystem)
|
||||
//! storage layer for the v1.1.5 `files::*` SDK.
|
||||
//!
|
||||
//! Unlike KV/docs, this repo owns BOTH halves of a file: the `files`
|
||||
//! row (metadata + SHA-256 checksum) and the bytes on disk at
|
||||
//! `<root>/files/<app_id>/<collection>/<id[0:2]>/<id>`.
|
||||
//! It owns both because the write must be atomic across them — a crash
|
||||
//! mid-write must never leave a readable half-written file.
|
||||
//!
|
||||
//! ## Atomic write protocol (`create` / `update`)
|
||||
//! 1. Validate (collection path-safety; caps live one layer up).
|
||||
//! 2. `create_dir_all` the shard dir with `0o700`.
|
||||
//! 3. SHA-256 the in-memory bytes (single pass) while writing to
|
||||
//! `<final>.tmp.<unique>`.
|
||||
//! 4. `fsync` the temp file.
|
||||
//! 5. `rename` temp → final (atomic on POSIX).
|
||||
//! 6. `fsync` the parent dir (so the rename is durable).
|
||||
//! 7. INSERT / UPDATE the DB row.
|
||||
//!
|
||||
//! A crash between 1–5 leaves an orphan `*.tmp.*` (never read). A crash
|
||||
//! between 5–7 leaves a file with no row — never reachable via the SDK
|
||||
//! (reads start from the row). Both are reclaimed by a future orphan
|
||||
//! sweep (deferred to v1.1.6+; see HANDBACK §7).
|
||||
//!
|
||||
//! ## Atomic delete protocol
|
||||
//! 1. SELECT + DELETE the row inside one transaction; commit.
|
||||
//! 2. `unlink` the file (outside the tx). A failure here leaves an
|
||||
//! orphan; a failure before the commit changes nothing.
|
||||
//!
|
||||
//! ## Checksum-on-read
|
||||
//! `get` reads the file, hashes it, and compares against the stored
|
||||
//! checksum — returning `FilesError::Corrupted` (and logging the path
|
||||
//! at error level) on a mismatch. It never auto-deletes; the operator
|
||||
//! decides what to do with a metadata-vs-bytes divergence.
|
||||
|
||||
use std::env;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
|
||||
use base64::Engine as _;
|
||||
use chrono::{DateTime, Utc};
|
||||
use picloud_shared::{AppId, FileMeta, FileUpdate, FilesListPage, NewFile};
|
||||
use sha2::{Digest, Sha256};
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// 100 MB default per-file cap.
|
||||
pub const DEFAULT_MAX_FILE_SIZE_BYTES: usize = 100 * 1024 * 1024;
|
||||
/// Default filesystem root (relative to the process CWD).
|
||||
pub const DEFAULT_FILES_ROOT: &str = "./data";
|
||||
|
||||
const FILES_LIST_MAX_LIMIT: u32 = 1_000;
|
||||
const FILES_LIST_DEFAULT_LIMIT: u32 = 100;
|
||||
|
||||
/// Monotonic counter feeding unique temp-file suffixes (combined with
|
||||
/// the pid). Avoids `rand` in the storage layer per the brief.
|
||||
static TMP_COUNTER: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum FilesRepoError {
|
||||
#[error("database error: {0}")]
|
||||
Db(#[from] sqlx::Error),
|
||||
|
||||
#[error("filesystem error: {0}")]
|
||||
Io(String),
|
||||
|
||||
#[error("invalid collection name: {0}")]
|
||||
InvalidCollection(String),
|
||||
|
||||
/// The bytes on disk no longer match the stored checksum (or are
|
||||
/// missing entirely while the row persists).
|
||||
#[error("file content corrupted (checksum mismatch)")]
|
||||
Corrupted,
|
||||
|
||||
#[error("invalid pagination cursor")]
|
||||
InvalidCursor,
|
||||
}
|
||||
|
||||
/// Outbound-files tunables. Env-overridable following the same pattern
|
||||
/// as `HttpConfig::from_env`.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FilesConfig {
|
||||
pub root: PathBuf,
|
||||
pub max_file_size_bytes: usize,
|
||||
}
|
||||
|
||||
impl FilesConfig {
|
||||
#[must_use]
|
||||
pub fn conservative() -> Self {
|
||||
Self {
|
||||
root: PathBuf::from(DEFAULT_FILES_ROOT),
|
||||
max_file_size_bytes: DEFAULT_MAX_FILE_SIZE_BYTES,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn from_env() -> Self {
|
||||
let mut c = Self::conservative();
|
||||
if let Ok(v) = env::var("PICLOUD_FILES_ROOT") {
|
||||
if !v.trim().is_empty() {
|
||||
c.root = PathBuf::from(v);
|
||||
}
|
||||
}
|
||||
if let Ok(v) = env::var("PICLOUD_FILES_MAX_FILE_SIZE_BYTES") {
|
||||
match v.parse::<usize>() {
|
||||
Ok(n) => c.max_file_size_bytes = n,
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %e, "ignoring invalid PICLOUD_FILES_MAX_FILE_SIZE_BYTES");
|
||||
}
|
||||
}
|
||||
}
|
||||
c
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for FilesConfig {
|
||||
fn default() -> Self {
|
||||
Self::conservative()
|
||||
}
|
||||
}
|
||||
|
||||
/// The new+prior metadata returned from a successful `update`, so the
|
||||
/// service can emit a `ServiceEvent` with the change-data-capture
|
||||
/// surface (`old_payload`).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FileUpdated {
|
||||
pub new: FileMeta,
|
||||
pub prev: FileMeta,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait FilesRepo: Send + Sync {
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<FileMeta, FilesRepoError>;
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError>;
|
||||
|
||||
/// Reads + checksum-verifies the bytes. `Ok(None)` when no row
|
||||
/// exists; `Err(Corrupted)` when the row exists but the bytes are
|
||||
/// missing or mismatched.
|
||||
async fn get(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<Vec<u8>>, FilesRepoError>;
|
||||
|
||||
/// `Ok(None)` when no row exists (the SDK turns this into
|
||||
/// `FilesError::NotFound`).
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
upd: FileUpdate,
|
||||
) -> Result<Option<FileUpdated>, FilesRepoError>;
|
||||
|
||||
/// Returns the deleted row's metadata if present, `None` otherwise.
|
||||
async fn delete(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError>;
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
cursor: Option<&str>,
|
||||
limit: u32,
|
||||
) -> Result<FilesListPage, FilesRepoError>;
|
||||
}
|
||||
|
||||
/// Filesystem-bytes + Postgres-metadata repo.
|
||||
pub struct FsFilesRepo {
|
||||
pool: PgPool,
|
||||
config: FilesConfig,
|
||||
}
|
||||
|
||||
impl FsFilesRepo {
|
||||
#[must_use]
|
||||
pub fn new(pool: PgPool, config: FilesConfig) -> Self {
|
||||
Self { pool, config }
|
||||
}
|
||||
|
||||
/// Defensive path-component guard. The service already validates the
|
||||
/// collection at the SDK boundary; this is belt-and-suspenders so a
|
||||
/// future caller can't smuggle a traversal sequence onto disk.
|
||||
fn guard_collection(collection: &str) -> Result<(), FilesRepoError> {
|
||||
if collection.is_empty()
|
||||
|| collection.contains('/')
|
||||
|| collection.contains('\\')
|
||||
|| collection.contains("..")
|
||||
|| collection.contains('\0')
|
||||
{
|
||||
return Err(FilesRepoError::InvalidCollection(collection.to_string()));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn final_path(&self, app_id: AppId, collection: &str, id: Uuid) -> PathBuf {
|
||||
final_path_at(&self.config.root, app_id, collection, id)
|
||||
}
|
||||
|
||||
fn write_atomic(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
bytes: &[u8],
|
||||
) -> Result<String, FilesRepoError> {
|
||||
write_atomic_at(&self.config.root, app_id, collection, id, bytes)
|
||||
}
|
||||
}
|
||||
|
||||
fn shard_dir_at(root: &Path, app_id: AppId, collection: &str, id_str: &str) -> PathBuf {
|
||||
root.join("files")
|
||||
.join(app_id.into_inner().to_string())
|
||||
.join(collection)
|
||||
.join(&id_str[..2])
|
||||
}
|
||||
|
||||
fn final_path_at(root: &Path, app_id: AppId, collection: &str, id: Uuid) -> PathBuf {
|
||||
let id_str = id.to_string();
|
||||
shard_dir_at(root, app_id, collection, &id_str).join(&id_str)
|
||||
}
|
||||
|
||||
/// Steps 2–6 of the atomic-write protocol. Returns the lowercase hex
|
||||
/// SHA-256 of the bytes (computed in a single pass over the in-memory
|
||||
/// buffer — the file is never re-read). Free function so the fs
|
||||
/// mechanics are unit-testable without a Postgres pool.
|
||||
fn write_atomic_at(
|
||||
root: &Path,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
bytes: &[u8],
|
||||
) -> Result<String, FilesRepoError> {
|
||||
use std::io::Write as _;
|
||||
|
||||
let id_str = id.to_string();
|
||||
let dir = shard_dir_at(root, app_id, collection, &id_str);
|
||||
create_dir_all_secure(&dir)?;
|
||||
|
||||
// Single-pass checksum over the in-memory buffer.
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(bytes);
|
||||
let checksum = hex_lower(&hasher.finalize());
|
||||
|
||||
let seq = TMP_COUNTER.fetch_add(1, Ordering::Relaxed);
|
||||
let tmp = dir.join(format!("{id_str}.tmp.{}-{seq}", std::process::id()));
|
||||
let final_path = dir.join(&id_str);
|
||||
|
||||
{
|
||||
let mut f = std::fs::File::create(&tmp).map_err(io_err)?;
|
||||
f.write_all(bytes).map_err(io_err)?;
|
||||
f.sync_all().map_err(io_err)?; // fsync temp
|
||||
}
|
||||
std::fs::rename(&tmp, &final_path).map_err(io_err)?; // atomic
|
||||
// fsync the parent dir so the rename is durable.
|
||||
if let Ok(dirf) = std::fs::File::open(&dir) {
|
||||
let _ = dirf.sync_all();
|
||||
}
|
||||
Ok(checksum)
|
||||
}
|
||||
|
||||
/// Read + checksum-verify the bytes at the given path-set. Free
|
||||
/// function mirror of the `get` read path. Returns `Corrupted` when the
|
||||
/// bytes are missing or don't match `expected_checksum`.
|
||||
fn read_verify_at(
|
||||
root: &Path,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
expected_checksum: &str,
|
||||
) -> Result<Vec<u8>, FilesRepoError> {
|
||||
let path = final_path_at(root, app_id, collection, id);
|
||||
let bytes = match std::fs::read(&path) {
|
||||
Ok(b) => b,
|
||||
Err(e) => {
|
||||
tracing::error!(
|
||||
path = %path.display(), error = %e,
|
||||
"files: row exists but bytes are unreadable — treating as corrupted"
|
||||
);
|
||||
return Err(FilesRepoError::Corrupted);
|
||||
}
|
||||
};
|
||||
let mut hasher = Sha256::new();
|
||||
hasher.update(&bytes);
|
||||
let actual = hex_lower(&hasher.finalize());
|
||||
if actual != expected_checksum {
|
||||
tracing::error!(
|
||||
path = %path.display(), expected = %expected_checksum, actual = %actual,
|
||||
"files: checksum mismatch on read — content corrupted"
|
||||
);
|
||||
return Err(FilesRepoError::Corrupted);
|
||||
}
|
||||
Ok(bytes)
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl FilesRepo for FsFilesRepo {
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<FileMeta, FilesRepoError> {
|
||||
Self::guard_collection(collection)?;
|
||||
let id = Uuid::new_v4();
|
||||
let size = i64::try_from(new.data.len()).unwrap_or(i64::MAX);
|
||||
|
||||
let checksum = self.write_atomic(app_id, collection, id, &new.data)?;
|
||||
|
||||
let row: FileRow = sqlx::query_as(
|
||||
"INSERT INTO files \
|
||||
(app_id, collection, id, name, content_type, size_bytes, checksum_sha256) \
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7) \
|
||||
RETURNING id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.bind(&new.name)
|
||||
.bind(&new.content_type)
|
||||
.bind(size)
|
||||
.bind(&checksum)
|
||||
.fetch_one(&self.pool)
|
||||
.await?;
|
||||
|
||||
Ok(row.into_meta())
|
||||
}
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError> {
|
||||
let row: Option<FileRow> = sqlx::query_as(
|
||||
"SELECT id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at \
|
||||
FROM files WHERE app_id = $1 AND collection = $2 AND id = $3",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
Ok(row.map(FileRow::into_meta))
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<Vec<u8>>, FilesRepoError> {
|
||||
let row: Option<(String,)> = sqlx::query_as(
|
||||
"SELECT checksum_sha256 FROM files \
|
||||
WHERE app_id = $1 AND collection = $2 AND id = $3",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
let Some((stored_checksum,)) = row else {
|
||||
return Ok(None);
|
||||
};
|
||||
let bytes = read_verify_at(&self.config.root, app_id, collection, id, &stored_checksum)?;
|
||||
Ok(Some(bytes))
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
upd: FileUpdate,
|
||||
) -> Result<Option<FileUpdated>, FilesRepoError> {
|
||||
Self::guard_collection(collection)?;
|
||||
// Read the prior row first (existence check + CDC surface).
|
||||
let Some(prev) = self.head(app_id, collection, id).await? else {
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
let size = i64::try_from(upd.data.len()).unwrap_or(i64::MAX);
|
||||
let checksum = self.write_atomic(app_id, collection, id, &upd.data)?;
|
||||
|
||||
let row: FileRow = sqlx::query_as(
|
||||
"UPDATE files SET \
|
||||
name = COALESCE($4, name), \
|
||||
content_type = COALESCE($5, content_type), \
|
||||
size_bytes = $6, \
|
||||
checksum_sha256 = $7, \
|
||||
updated_at = NOW() \
|
||||
WHERE app_id = $1 AND collection = $2 AND id = $3 \
|
||||
RETURNING id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.bind(upd.name.as_deref())
|
||||
.bind(upd.content_type.as_deref())
|
||||
.bind(size)
|
||||
.bind(&checksum)
|
||||
.fetch_one(&self.pool)
|
||||
.await?;
|
||||
|
||||
Ok(Some(FileUpdated {
|
||||
new: row.into_meta(),
|
||||
prev,
|
||||
}))
|
||||
}
|
||||
|
||||
async fn delete(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError> {
|
||||
// SELECT + DELETE in one tx; unlink afterwards (outside the tx).
|
||||
let mut tx = self.pool.begin().await?;
|
||||
let row: Option<FileRow> = sqlx::query_as(
|
||||
"SELECT id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at \
|
||||
FROM files WHERE app_id = $1 AND collection = $2 AND id = $3 \
|
||||
FOR UPDATE",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.fetch_optional(&mut *tx)
|
||||
.await?;
|
||||
|
||||
let Some(row) = row else {
|
||||
tx.rollback().await?;
|
||||
return Ok(None);
|
||||
};
|
||||
|
||||
sqlx::query("DELETE FROM files WHERE app_id = $1 AND collection = $2 AND id = $3")
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(id)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
tx.commit().await?;
|
||||
|
||||
// Row is gone; unlink the bytes. A failure here leaves an orphan
|
||||
// file (reclaimed by a future sweep) — not fatal.
|
||||
let path = self.final_path(app_id, collection, id);
|
||||
if let Err(e) = std::fs::remove_file(&path) {
|
||||
if e.kind() != std::io::ErrorKind::NotFound {
|
||||
tracing::warn!(path = %path.display(), error = %e, "files: unlink after delete failed (orphan)");
|
||||
}
|
||||
}
|
||||
Ok(Some(row.into_meta()))
|
||||
}
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
cursor: Option<&str>,
|
||||
limit: u32,
|
||||
) -> Result<FilesListPage, FilesRepoError> {
|
||||
let limit = if limit == 0 {
|
||||
FILES_LIST_DEFAULT_LIMIT
|
||||
} else {
|
||||
limit.min(FILES_LIST_MAX_LIMIT)
|
||||
};
|
||||
let last_id = match cursor {
|
||||
Some(c) => Some(decode_cursor(c)?),
|
||||
None => None,
|
||||
};
|
||||
let take = i64::from(limit) + 1;
|
||||
let rows: Vec<FileRow> = sqlx::query_as(
|
||||
"SELECT id, collection, name, content_type, size_bytes, \
|
||||
checksum_sha256, created_at, updated_at \
|
||||
FROM files \
|
||||
WHERE app_id = $1 AND collection = $2 \
|
||||
AND ($3::uuid IS NULL OR id > $3) \
|
||||
ORDER BY id ASC \
|
||||
LIMIT $4",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(collection)
|
||||
.bind(last_id)
|
||||
.bind(take)
|
||||
.fetch_all(&self.pool)
|
||||
.await?;
|
||||
|
||||
let mut files: Vec<FileMeta> = rows.into_iter().map(FileRow::into_meta).collect();
|
||||
let next_cursor = if files.len() > limit as usize {
|
||||
files.truncate(limit as usize);
|
||||
files.last().map(|m| encode_cursor(m.id))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
Ok(FilesListPage { files, next_cursor })
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
fn io_err(e: std::io::Error) -> FilesRepoError {
|
||||
FilesRepoError::Io(e.to_string())
|
||||
}
|
||||
|
||||
/// `create_dir_all` with `0o700` on the created tree (Unix). On other
|
||||
/// platforms it falls back to the default permissions.
|
||||
fn create_dir_all_secure(dir: &Path) -> Result<(), FilesRepoError> {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::DirBuilderExt as _;
|
||||
std::fs::DirBuilder::new()
|
||||
.recursive(true)
|
||||
.mode(0o700)
|
||||
.create(dir)
|
||||
.map_err(io_err)
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
std::fs::create_dir_all(dir).map_err(io_err)
|
||||
}
|
||||
}
|
||||
|
||||
fn hex_lower(bytes: &[u8]) -> String {
|
||||
let mut s = String::with_capacity(bytes.len() * 2);
|
||||
for b in bytes {
|
||||
use std::fmt::Write as _;
|
||||
let _ = write!(s, "{b:02x}");
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
fn encode_cursor(last_id: Uuid) -> String {
|
||||
URL_SAFE_NO_PAD.encode(last_id.to_string().as_bytes())
|
||||
}
|
||||
|
||||
fn decode_cursor(cursor: &str) -> Result<Uuid, FilesRepoError> {
|
||||
let bytes = URL_SAFE_NO_PAD
|
||||
.decode(cursor)
|
||||
.map_err(|_| FilesRepoError::InvalidCursor)?;
|
||||
let s = String::from_utf8(bytes).map_err(|_| FilesRepoError::InvalidCursor)?;
|
||||
Uuid::parse_str(&s).map_err(|_| FilesRepoError::InvalidCursor)
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct FileRow {
|
||||
id: Uuid,
|
||||
collection: String,
|
||||
name: String,
|
||||
content_type: String,
|
||||
size_bytes: i64,
|
||||
checksum_sha256: String,
|
||||
created_at: DateTime<Utc>,
|
||||
updated_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
impl FileRow {
|
||||
fn into_meta(self) -> FileMeta {
|
||||
FileMeta {
|
||||
id: self.id,
|
||||
collection: self.collection,
|
||||
name: self.name,
|
||||
content_type: self.content_type,
|
||||
size: u64::try_from(self.size_bytes).unwrap_or(0),
|
||||
checksum: self.checksum_sha256,
|
||||
created_at: self.created_at,
|
||||
updated_at: self.updated_at,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn hex_lower_matches_known_sha256_vector() {
|
||||
// SHA-256("abc") — NIST known-answer vector.
|
||||
let mut h = Sha256::new();
|
||||
h.update(b"abc");
|
||||
assert_eq!(
|
||||
hex_lower(&h.finalize()),
|
||||
"ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn hex_lower_of_empty_is_known_vector() {
|
||||
let mut h = Sha256::new();
|
||||
h.update(b"");
|
||||
assert_eq!(
|
||||
hex_lower(&h.finalize()),
|
||||
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cursor_round_trips() {
|
||||
let id = Uuid::new_v4();
|
||||
let enc = encode_cursor(id);
|
||||
assert_eq!(decode_cursor(&enc).unwrap(), id);
|
||||
assert!(matches!(
|
||||
decode_cursor("!!not-base64!!"),
|
||||
Err(FilesRepoError::InvalidCursor)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn guard_collection_rejects_traversal() {
|
||||
assert!(FsFilesRepo::guard_collection("avatars").is_ok());
|
||||
assert!(FsFilesRepo::guard_collection("a/b").is_err());
|
||||
assert!(FsFilesRepo::guard_collection("..").is_err());
|
||||
assert!(FsFilesRepo::guard_collection("a..b").is_err());
|
||||
assert!(FsFilesRepo::guard_collection("").is_err());
|
||||
assert!(FsFilesRepo::guard_collection("a\0b").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_from_env_defaults_are_conservative() {
|
||||
let c = FilesConfig::conservative();
|
||||
assert_eq!(c.max_file_size_bytes, DEFAULT_MAX_FILE_SIZE_BYTES);
|
||||
assert_eq!(c.root, PathBuf::from(DEFAULT_FILES_ROOT));
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// Tempdir-backed filesystem mechanics — exercise the atomic write,
|
||||
// single-pass checksum, and checksum-on-read tamper detection
|
||||
// without needing a Postgres pool.
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
use picloud_shared::AppId;
|
||||
|
||||
/// Process-unique scratch dir under the system temp dir. Cleaned up
|
||||
/// by each test via `remove_dir_all`.
|
||||
fn unique_tmp_root() -> PathBuf {
|
||||
let seq = TMP_COUNTER.fetch_add(1, Ordering::Relaxed);
|
||||
let dir =
|
||||
std::env::temp_dir().join(format!("picloud-files-test-{}-{seq}", std::process::id()));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
dir
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn write_atomic_then_read_verify_round_trips() {
|
||||
let root = unique_tmp_root();
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
let bytes = b"hello picloud files".to_vec();
|
||||
|
||||
let checksum = write_atomic_at(&root, app, "avatars", id, &bytes).unwrap();
|
||||
// Single-pass checksum matches an independent hash of the bytes.
|
||||
let mut h = Sha256::new();
|
||||
h.update(&bytes);
|
||||
assert_eq!(checksum, hex_lower(&h.finalize()));
|
||||
|
||||
let read = read_verify_at(&root, app, "avatars", id, &checksum).unwrap();
|
||||
assert_eq!(read, bytes);
|
||||
|
||||
std::fs::remove_dir_all(&root).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_verify_detects_tampering_as_corrupted() {
|
||||
let root = unique_tmp_root();
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
let checksum = write_atomic_at(&root, app, "c", id, b"original").unwrap();
|
||||
|
||||
// Mutate the bytes behind the repo's back.
|
||||
let path = final_path_at(&root, app, "c", id);
|
||||
std::fs::write(&path, b"tampered").unwrap();
|
||||
|
||||
let err = read_verify_at(&root, app, "c", id, &checksum).unwrap_err();
|
||||
assert!(matches!(err, FilesRepoError::Corrupted));
|
||||
|
||||
std::fs::remove_dir_all(&root).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn read_verify_missing_bytes_is_corrupted() {
|
||||
let root = unique_tmp_root();
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
// No write — the file never existed.
|
||||
let err = read_verify_at(&root, app, "c", id, "deadbeef").unwrap_err();
|
||||
assert!(matches!(err, FilesRepoError::Corrupted));
|
||||
std::fs::remove_dir_all(&root).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn atomic_write_leaves_no_tmp_file_after_success() {
|
||||
let root = unique_tmp_root();
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
write_atomic_at(&root, app, "c", id, b"data").unwrap();
|
||||
|
||||
let id_str = id.to_string();
|
||||
let dir = shard_dir_at(&root, app, "c", &id_str);
|
||||
let entries: Vec<_> = std::fs::read_dir(&dir)
|
||||
.unwrap()
|
||||
.filter_map(Result::ok)
|
||||
.map(|e| e.file_name().to_string_lossy().into_owned())
|
||||
.collect();
|
||||
// Exactly the final file is visible — no `*.tmp.*` orphan.
|
||||
assert_eq!(entries, vec![id_str]);
|
||||
assert!(!entries.iter().any(|n| n.contains(".tmp.")));
|
||||
|
||||
std::fs::remove_dir_all(&root).ok();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn id_shard_uses_first_two_chars() {
|
||||
let root = PathBuf::from("/tmp/x");
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
let id_str = id.to_string();
|
||||
let path = final_path_at(&root, app, "col", id);
|
||||
let shard = &id_str[..2];
|
||||
assert!(path
|
||||
.to_string_lossy()
|
||||
.contains(&format!("/col/{shard}/{id_str}")));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn shard_tree_created_with_0700() {
|
||||
use std::os::unix::fs::PermissionsExt as _;
|
||||
let root = unique_tmp_root();
|
||||
let app = AppId::new();
|
||||
let id = Uuid::new_v4();
|
||||
write_atomic_at(&root, app, "c", id, b"data").unwrap();
|
||||
let id_str = id.to_string();
|
||||
let dir = shard_dir_at(&root, app, "c", &id_str);
|
||||
let mode = std::fs::metadata(&dir).unwrap().permissions().mode();
|
||||
assert_eq!(mode & 0o777, 0o700, "shard dir should be 0o700");
|
||||
std::fs::remove_dir_all(&root).ok();
|
||||
}
|
||||
}
|
||||
833
crates/manager-core/src/files_service.rs
Normal file
833
crates/manager-core/src/files_service.rs
Normal file
@@ -0,0 +1,833 @@
|
||||
//! `FilesServiceImpl` — wires the `FilesRepo` underneath the
|
||||
//! `picloud_shared::FilesService` trait scripts see via the Rhai
|
||||
//! bridge.
|
||||
//!
|
||||
//! Layers added here (vs the raw repo), mirroring `KvServiceImpl`:
|
||||
//! 1. Collection validation (empty + path-traversal) and field /
|
||||
//! size-cap validation at the SDK boundary.
|
||||
//! 2. **Script-as-gate authz**: when `cx.principal.is_some()` we run
|
||||
//! `authz::require(...)`; when it's `None` (public HTTP) we skip.
|
||||
//! Cross-app isolation is unaffected — every repo call is keyed by
|
||||
//! `cx.app_id`, never an argument.
|
||||
//! 3. `ServiceEvent` emission after each mutation (`create` /
|
||||
//! `update` / `delete`). The payload is the file **metadata**, not
|
||||
//! the blob bytes (files are too big for trigger payloads).
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
validate_files_collection, FileMeta, FileUpdate, FilesError, FilesListPage, FilesService,
|
||||
NewFile, SdkCallCx, ServiceEvent, ServiceEventEmitter,
|
||||
};
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::authz::{self, AuthzRepo, Capability};
|
||||
use crate::files_repo::{FileUpdated, FilesRepo, FilesRepoError};
|
||||
|
||||
pub struct FilesServiceImpl {
|
||||
repo: Arc<dyn FilesRepo>,
|
||||
authz: Arc<dyn AuthzRepo>,
|
||||
events: Arc<dyn ServiceEventEmitter>,
|
||||
max_file_size_bytes: usize,
|
||||
}
|
||||
|
||||
impl FilesServiceImpl {
|
||||
#[must_use]
|
||||
pub fn new(
|
||||
repo: Arc<dyn FilesRepo>,
|
||||
authz: Arc<dyn AuthzRepo>,
|
||||
events: Arc<dyn ServiceEventEmitter>,
|
||||
max_file_size_bytes: usize,
|
||||
) -> Self {
|
||||
Self {
|
||||
repo,
|
||||
authz,
|
||||
events,
|
||||
max_file_size_bytes,
|
||||
}
|
||||
}
|
||||
|
||||
async fn check_read(&self, cx: &SdkCallCx) -> Result<(), FilesError> {
|
||||
if let Some(ref principal) = cx.principal {
|
||||
authz::require(&*self.authz, principal, Capability::AppFilesRead(cx.app_id))
|
||||
.await
|
||||
.map_err(|_| FilesError::Forbidden)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn check_write(&self, cx: &SdkCallCx) -> Result<(), FilesError> {
|
||||
if let Some(ref principal) = cx.principal {
|
||||
authz::require(
|
||||
&*self.authz,
|
||||
principal,
|
||||
Capability::AppFilesWrite(cx.app_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| FilesError::Forbidden)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Best-effort `ServiceEvent` emission. A failed emit is logged but
|
||||
/// never rolls back the (already-durable) file write.
|
||||
async fn emit(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
op: &'static str,
|
||||
collection: &str,
|
||||
meta: &FileMeta,
|
||||
old: Option<&FileMeta>,
|
||||
) {
|
||||
let payload = serde_json::to_value(meta).ok();
|
||||
let old_payload = old.and_then(|m| serde_json::to_value(m).ok());
|
||||
if let Err(e) = self
|
||||
.events
|
||||
.emit(
|
||||
cx,
|
||||
ServiceEvent {
|
||||
source: "files",
|
||||
op,
|
||||
collection: Some(collection.to_string()),
|
||||
key: Some(meta.id.to_string()),
|
||||
payload,
|
||||
old_payload,
|
||||
},
|
||||
)
|
||||
.await
|
||||
{
|
||||
tracing::warn!(error = %e, source = "files", op, "event emit failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse a script-supplied id. Invalid UUIDs aren't an error shape the
|
||||
/// SDK exposes — for reads/deletes they simply mean "no such file".
|
||||
fn parse_id(id: &str) -> Option<Uuid> {
|
||||
Uuid::parse_str(id).ok()
|
||||
}
|
||||
|
||||
impl From<FilesRepoError> for FilesError {
|
||||
fn from(e: FilesRepoError) -> Self {
|
||||
match e {
|
||||
FilesRepoError::Corrupted => Self::Corrupted,
|
||||
FilesRepoError::InvalidCollection(c) => Self::InvalidCollection(c),
|
||||
other => Self::Backend(other.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl FilesService for FilesServiceImpl {
|
||||
async fn create(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<Uuid, FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_write(cx).await?;
|
||||
new.validate(self.max_file_size_bytes)?;
|
||||
let meta = self.repo.create(cx.app_id, collection, new).await?;
|
||||
self.emit(cx, "create", collection, &meta, None).await;
|
||||
Ok(meta.id)
|
||||
}
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<FileMeta>, FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_read(cx).await?;
|
||||
let Some(uuid) = parse_id(id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(self.repo.head(cx.app_id, collection, uuid).await?)
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<Vec<u8>>, FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_read(cx).await?;
|
||||
let Some(uuid) = parse_id(id) else {
|
||||
return Ok(None);
|
||||
};
|
||||
Ok(self.repo.get(cx.app_id, collection, uuid).await?)
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
upd: FileUpdate,
|
||||
) -> Result<(), FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_write(cx).await?;
|
||||
upd.validate(self.max_file_size_bytes)?;
|
||||
let Some(uuid) = parse_id(id) else {
|
||||
return Err(FilesError::NotFound);
|
||||
};
|
||||
match self.repo.update(cx.app_id, collection, uuid, upd).await? {
|
||||
Some(FileUpdated { new, prev }) => {
|
||||
self.emit(cx, "update", collection, &new, Some(&prev)).await;
|
||||
Ok(())
|
||||
}
|
||||
None => Err(FilesError::NotFound),
|
||||
}
|
||||
}
|
||||
|
||||
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_write(cx).await?;
|
||||
let Some(uuid) = parse_id(id) else {
|
||||
return Ok(false);
|
||||
};
|
||||
match self.repo.delete(cx.app_id, collection, uuid).await? {
|
||||
Some(meta) => {
|
||||
// On delete, the top-level metadata AND `prev` both carry
|
||||
// the deleted row (per docs/v1.1.x design + the brief).
|
||||
self.emit(cx, "delete", collection, &meta, Some(&meta))
|
||||
.await;
|
||||
Ok(true)
|
||||
}
|
||||
None => Ok(false),
|
||||
}
|
||||
}
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
cursor: Option<&str>,
|
||||
limit: u32,
|
||||
) -> Result<FilesListPage, FilesError> {
|
||||
validate_files_collection(collection)?;
|
||||
self.check_read(cx).await?;
|
||||
Ok(self.repo.list(cx.app_id, collection, cursor, limit).await?)
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Tests — in-memory FilesRepo so unit tests need neither Postgres nor a
|
||||
// filesystem. The on-disk atomic-write / checksum mechanics are covered
|
||||
// by the tempdir tests in `files_repo.rs`.
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::authz::{AuthzError, AuthzRepo};
|
||||
use async_trait::async_trait;
|
||||
use chrono::Utc;
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, AppRole, EmitError, ExecutionId, InstanceRole, Principal, RequestId,
|
||||
ScriptId, ServiceEvent, UserId,
|
||||
};
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Mutex as StdMutex;
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
/// In-memory FilesRepo keyed by (app, collection, id). Stores the
|
||||
/// metadata + bytes together so cross-app isolation and round-trips
|
||||
/// can be checked without disk.
|
||||
#[derive(Default)]
|
||||
struct InMemoryFilesRepo {
|
||||
#[allow(clippy::type_complexity)]
|
||||
data: Mutex<BTreeMap<(AppId, String, Uuid), (FileMeta, Vec<u8>)>>,
|
||||
}
|
||||
|
||||
fn sha256_hex(bytes: &[u8]) -> String {
|
||||
use sha2::{Digest, Sha256};
|
||||
let mut h = Sha256::new();
|
||||
h.update(bytes);
|
||||
let out = h.finalize();
|
||||
let mut s = String::new();
|
||||
for b in out {
|
||||
use std::fmt::Write as _;
|
||||
let _ = write!(s, "{b:02x}");
|
||||
}
|
||||
s
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl FilesRepo for InMemoryFilesRepo {
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<FileMeta, FilesRepoError> {
|
||||
let id = Uuid::new_v4();
|
||||
let now = Utc::now();
|
||||
let meta = FileMeta {
|
||||
id,
|
||||
collection: collection.to_string(),
|
||||
name: new.name.clone(),
|
||||
content_type: new.content_type.clone(),
|
||||
size: new.data.len() as u64,
|
||||
checksum: sha256_hex(&new.data),
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
};
|
||||
self.data.lock().await.insert(
|
||||
(app_id, collection.to_string(), id),
|
||||
(meta.clone(), new.data),
|
||||
);
|
||||
Ok(meta)
|
||||
}
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError> {
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.get(&(app_id, collection.to_string(), id))
|
||||
.map(|(m, _)| m.clone()))
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<Vec<u8>>, FilesRepoError> {
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.get(&(app_id, collection.to_string(), id))
|
||||
.map(|(_, b)| b.clone()))
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
upd: FileUpdate,
|
||||
) -> Result<Option<FileUpdated>, FilesRepoError> {
|
||||
let mut data = self.data.lock().await;
|
||||
let key = (app_id, collection.to_string(), id);
|
||||
let Some((prev_meta, _)) = data.get(&key).cloned() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let now = Utc::now();
|
||||
let new_meta = FileMeta {
|
||||
id,
|
||||
collection: collection.to_string(),
|
||||
name: upd.name.clone().unwrap_or_else(|| prev_meta.name.clone()),
|
||||
content_type: upd
|
||||
.content_type
|
||||
.clone()
|
||||
.unwrap_or_else(|| prev_meta.content_type.clone()),
|
||||
size: upd.data.len() as u64,
|
||||
checksum: sha256_hex(&upd.data),
|
||||
created_at: prev_meta.created_at,
|
||||
updated_at: now,
|
||||
};
|
||||
data.insert(key, (new_meta.clone(), upd.data));
|
||||
Ok(Some(FileUpdated {
|
||||
new: new_meta,
|
||||
prev: prev_meta,
|
||||
}))
|
||||
}
|
||||
|
||||
async fn delete(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
id: Uuid,
|
||||
) -> Result<Option<FileMeta>, FilesRepoError> {
|
||||
Ok(self
|
||||
.data
|
||||
.lock()
|
||||
.await
|
||||
.remove(&(app_id, collection.to_string(), id))
|
||||
.map(|(m, _)| m))
|
||||
}
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
cursor: Option<&str>,
|
||||
limit: u32,
|
||||
) -> Result<FilesListPage, FilesRepoError> {
|
||||
let data = self.data.lock().await;
|
||||
let after = cursor.and_then(|c| Uuid::parse_str(c).ok());
|
||||
let mut metas: Vec<FileMeta> = data
|
||||
.iter()
|
||||
.filter(|((a, c, _), _)| *a == app_id && c == collection)
|
||||
.map(|(_, (m, _))| m.clone())
|
||||
.filter(|m| after.is_none_or(|a| m.id > a))
|
||||
.collect();
|
||||
metas.sort_by_key(|m| m.id);
|
||||
let take = (limit.max(1)) as usize;
|
||||
let next_cursor = if metas.len() > take {
|
||||
metas.truncate(take);
|
||||
metas.last().map(|m| m.id.to_string())
|
||||
} else {
|
||||
None
|
||||
};
|
||||
Ok(FilesListPage {
|
||||
files: metas,
|
||||
next_cursor,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Captures emitted events so tests can assert on fan-out shape.
|
||||
#[derive(Default)]
|
||||
struct CapturingEmitter {
|
||||
events: StdMutex<Vec<ServiceEvent>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl ServiceEventEmitter for CapturingEmitter {
|
||||
async fn emit(&self, _cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
|
||||
self.events.lock().unwrap().push(event);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct DenyingAuthzRepo;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for DenyingAuthzRepo {
|
||||
async fn membership(
|
||||
&self,
|
||||
_user_id: UserId,
|
||||
_app_id: AppId,
|
||||
) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct EditorAuthzRepo;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for EditorAuthzRepo {
|
||||
async fn membership(
|
||||
&self,
|
||||
_user_id: UserId,
|
||||
_app_id: AppId,
|
||||
) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(Some(AppRole::Editor))
|
||||
}
|
||||
}
|
||||
|
||||
fn anon_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: None,
|
||||
execution_id: ExecutionId::new(),
|
||||
request_id: RequestId::new(),
|
||||
trigger_depth: 0,
|
||||
root_execution_id: ExecutionId::new(),
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn member_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
scopes: None,
|
||||
app_binding: None,
|
||||
}),
|
||||
..anon_cx(app_id)
|
||||
}
|
||||
}
|
||||
|
||||
fn svc_with(authz: Arc<dyn AuthzRepo>, emitter: Arc<CapturingEmitter>) -> FilesServiceImpl {
|
||||
FilesServiceImpl::new(
|
||||
Arc::new(InMemoryFilesRepo::default()),
|
||||
authz,
|
||||
emitter,
|
||||
10 * 1024 * 1024,
|
||||
)
|
||||
}
|
||||
|
||||
fn svc() -> FilesServiceImpl {
|
||||
svc_with(
|
||||
Arc::new(DenyingAuthzRepo),
|
||||
Arc::new(CapturingEmitter::default()),
|
||||
)
|
||||
}
|
||||
|
||||
fn new_file(name: &str, data: &[u8]) -> NewFile {
|
||||
NewFile {
|
||||
name: name.to_string(),
|
||||
content_type: "application/octet-stream".to_string(),
|
||||
data: data.to_vec(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn create_then_get_head_round_trips() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let id = files
|
||||
.create(&cx, "avatars", new_file("a.bin", b"hello"))
|
||||
.await
|
||||
.unwrap();
|
||||
let bytes = files.get(&cx, "avatars", &id.to_string()).await.unwrap();
|
||||
assert_eq!(bytes, Some(b"hello".to_vec()));
|
||||
let meta = files
|
||||
.head(&cx, "avatars", &id.to_string())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(meta.name, "a.bin");
|
||||
assert_eq!(meta.size, 5);
|
||||
assert_eq!(meta.checksum, sha256_hex(b"hello"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_and_head_missing_return_none() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let missing = Uuid::new_v4().to_string();
|
||||
assert_eq!(files.get(&cx, "c", &missing).await.unwrap(), None);
|
||||
assert!(files.head(&cx, "c", &missing).await.unwrap().is_none());
|
||||
// Non-UUID id is also "missing", not an error.
|
||||
assert_eq!(files.get(&cx, "c", "not-a-uuid").await.unwrap(), None);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_replaces_content_and_keeps_metadata_when_omitted() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let id = files
|
||||
.create(&cx, "c", new_file("v1.txt", b"one"))
|
||||
.await
|
||||
.unwrap();
|
||||
files
|
||||
.update(
|
||||
&cx,
|
||||
"c",
|
||||
&id.to_string(),
|
||||
FileUpdate {
|
||||
data: b"two!!".to_vec(),
|
||||
name: None,
|
||||
content_type: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let meta = files
|
||||
.head(&cx, "c", &id.to_string())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert_eq!(meta.name, "v1.txt"); // kept
|
||||
assert_eq!(meta.size, 5);
|
||||
assert_eq!(
|
||||
files.get(&cx, "c", &id.to_string()).await.unwrap(),
|
||||
Some(b"two!!".to_vec())
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn update_missing_throws_not_found() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let err = files
|
||||
.update(
|
||||
&cx,
|
||||
"c",
|
||||
&Uuid::new_v4().to_string(),
|
||||
FileUpdate {
|
||||
data: b"x".to_vec(),
|
||||
name: None,
|
||||
content_type: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::NotFound));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_returns_was_present() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let id = files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
|
||||
assert!(files.delete(&cx, "c", &id.to_string()).await.unwrap());
|
||||
assert!(!files.delete(&cx, "c", &id.to_string()).await.unwrap());
|
||||
assert!(!files.delete(&cx, "c", "not-a-uuid").await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_collection_rejected() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let err = files
|
||||
.create(&cx, "", new_file("f", b"x"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::InvalidCollection(_)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn traversal_collection_rejected() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
for bad in ["../etc", "a/b", "a..b", "x\0y"] {
|
||||
let err = files
|
||||
.create(&cx, bad, new_file("f", b"x"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, FilesError::InvalidCollection(_)),
|
||||
"expected reject for {bad:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn missing_required_fields_have_field_specific_messages() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
// name
|
||||
let err = files
|
||||
.create(
|
||||
&cx,
|
||||
"c",
|
||||
NewFile {
|
||||
name: " ".into(),
|
||||
content_type: "text/plain".into(),
|
||||
data: b"x".to_vec(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::MissingField("name")));
|
||||
// content_type
|
||||
let err = files
|
||||
.create(
|
||||
&cx,
|
||||
"c",
|
||||
NewFile {
|
||||
name: "f".into(),
|
||||
content_type: String::new(),
|
||||
data: b"x".to_vec(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::MissingField("content_type")));
|
||||
// Empty `data` is NO LONGER rejected (v1.1.6 relaxation) — see
|
||||
// `empty_file_round_trips`.
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_file_round_trips() {
|
||||
// v1.1.6: a zero-byte blob is a valid stored state (sentinels,
|
||||
// placeholders). Create with empty data, then read it back.
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let id = files
|
||||
.create(
|
||||
&cx,
|
||||
"c",
|
||||
NewFile {
|
||||
name: "empty.bin".into(),
|
||||
content_type: "application/octet-stream".into(),
|
||||
data: vec![],
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("empty file create should succeed");
|
||||
let bytes = files.get(&cx, "c", &id.to_string()).await.unwrap();
|
||||
assert_eq!(bytes, Some(Vec::new()));
|
||||
let meta = files
|
||||
.head(&cx, "c", &id.to_string())
|
||||
.await
|
||||
.unwrap()
|
||||
.expect("metadata present");
|
||||
assert_eq!(meta.size, 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn name_and_content_type_length_caps_enforced() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
let long_name = "x".repeat(256);
|
||||
let err = files
|
||||
.create(&cx, "c", new_file(&long_name, b"x"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::NameTooLong(256)));
|
||||
|
||||
let err = files
|
||||
.create(
|
||||
&cx,
|
||||
"c",
|
||||
NewFile {
|
||||
name: "f".into(),
|
||||
content_type: "x".repeat(128),
|
||||
data: b"x".to_vec(),
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::ContentTypeTooLong(128)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn per_file_size_cap_enforced() {
|
||||
let files = FilesServiceImpl::new(
|
||||
Arc::new(InMemoryFilesRepo::default()),
|
||||
Arc::new(DenyingAuthzRepo),
|
||||
Arc::new(CapturingEmitter::default()),
|
||||
8, // tiny cap
|
||||
);
|
||||
let cx = anon_cx(AppId::new());
|
||||
let err = files
|
||||
.create(&cx, "c", new_file("big", b"123456789"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::TooLarge { limit: 8, .. }));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cross_app_isolation() {
|
||||
let files = svc();
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let cx_a = anon_cx(app_a);
|
||||
let cx_b = anon_cx(app_b);
|
||||
let id = files
|
||||
.create(&cx_a, "shared", new_file("f", b"from-a"))
|
||||
.await
|
||||
.unwrap();
|
||||
// app B cannot see app A's file by id.
|
||||
assert_eq!(
|
||||
files.get(&cx_b, "shared", &id.to_string()).await.unwrap(),
|
||||
None
|
||||
);
|
||||
assert!(files
|
||||
.head(&cx_b, "shared", &id.to_string())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_none());
|
||||
let page_b = files.list(&cx_b, "shared", None, 100).await.unwrap();
|
||||
assert!(page_b.files.is_empty());
|
||||
// app A still sees it.
|
||||
assert!(files
|
||||
.get(&cx_a, "shared", &id.to_string())
|
||||
.await
|
||||
.unwrap()
|
||||
.is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn anonymous_cx_skips_authz() {
|
||||
let files = svc(); // DenyingAuthzRepo
|
||||
let cx = anon_cx(AppId::new());
|
||||
// No principal → no authz check, even with a denying repo.
|
||||
files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn member_without_role_is_forbidden() {
|
||||
let files = svc(); // DenyingAuthzRepo
|
||||
let cx = member_cx(AppId::new());
|
||||
let err = files
|
||||
.create(&cx, "c", new_file("f", b"x"))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, FilesError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn member_with_editor_role_allowed() {
|
||||
let files = svc_with(
|
||||
Arc::new(EditorAuthzRepo),
|
||||
Arc::new(CapturingEmitter::default()),
|
||||
);
|
||||
let cx = member_cx(AppId::new());
|
||||
files.create(&cx, "c", new_file("f", b"x")).await.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mutations_emit_events_with_correct_prev() {
|
||||
let emitter = Arc::new(CapturingEmitter::default());
|
||||
let files = svc_with(Arc::new(DenyingAuthzRepo), emitter.clone());
|
||||
let cx = anon_cx(AppId::new());
|
||||
|
||||
let id = files.create(&cx, "c", new_file("f", b"one")).await.unwrap();
|
||||
files
|
||||
.update(
|
||||
&cx,
|
||||
"c",
|
||||
&id.to_string(),
|
||||
FileUpdate {
|
||||
data: b"two".to_vec(),
|
||||
name: None,
|
||||
content_type: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
files.delete(&cx, "c", &id.to_string()).await.unwrap();
|
||||
|
||||
let events = emitter.events.lock().unwrap();
|
||||
assert_eq!(events.len(), 3);
|
||||
// create: prev is None
|
||||
assert_eq!(events[0].op, "create");
|
||||
assert_eq!(events[0].source, "files");
|
||||
assert!(events[0].old_payload.is_none());
|
||||
assert!(events[0].payload.is_some());
|
||||
// update: prev is the prior metadata
|
||||
assert_eq!(events[1].op, "update");
|
||||
assert!(events[1].old_payload.is_some());
|
||||
// delete: prev is the deleted metadata (payload == old_payload)
|
||||
assert_eq!(events[2].op, "delete");
|
||||
assert_eq!(events[2].payload, events[2].old_payload);
|
||||
assert!(events[2].payload.is_some());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_cursor_paginates() {
|
||||
let files = svc();
|
||||
let cx = anon_cx(AppId::new());
|
||||
for i in 0..5 {
|
||||
files
|
||||
.create(&cx, "c", new_file(&format!("f{i}"), b"x"))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
let p1 = files.list(&cx, "c", None, 2).await.unwrap();
|
||||
assert_eq!(p1.files.len(), 2);
|
||||
assert!(p1.next_cursor.is_some());
|
||||
let p2 = files
|
||||
.list(&cx, "c", p1.next_cursor.as_deref(), 2)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(p2.files.len(), 2);
|
||||
let p3 = files
|
||||
.list(&cx, "c", p2.next_cursor.as_deref(), 2)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(p3.files.len(), 1);
|
||||
assert!(p3.next_cursor.is_none());
|
||||
}
|
||||
}
|
||||
185
crates/manager-core/src/files_sweep.rs
Normal file
185
crates/manager-core/src/files_sweep.rs
Normal file
@@ -0,0 +1,185 @@
|
||||
//! Orphan `*.tmp.*` blob sweeper (v1.1.6, v1.1.5 follow-up).
|
||||
//!
|
||||
//! The files repo writes blobs atomically: it streams into a
|
||||
//! `<id>.tmp.<pid>-<seq>` temp file, fsyncs, then renames to the final
|
||||
//! `<id>` path. A crash between create and rename leaves an orphan temp
|
||||
//! file that is never read and never reclaimed. This sweeper deletes
|
||||
//! those: every `PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC` (default 6h) it
|
||||
//! walks `<root>/files/` and unlinks any `*.tmp.*` file older than
|
||||
//! `PICLOUD_FILES_ORPHAN_TMP_TTL_SEC` (default 1h).
|
||||
//!
|
||||
//! Deliberately bounded: it does NOT cross-check on-disk files against DB
|
||||
//! rows (the full reconciling sweeper is v1.3+). It only targets the temp
|
||||
//! files, which are unambiguously orphans once past the TTL — no live
|
||||
//! writer keeps one around for an hour.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::time::{Duration, SystemTime};
|
||||
|
||||
const ENV_INTERVAL: &str = "PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC";
|
||||
const ENV_TMP_TTL: &str = "PICLOUD_FILES_ORPHAN_TMP_TTL_SEC";
|
||||
const DEFAULT_INTERVAL_SECS: u64 = 21_600; // 6h
|
||||
const DEFAULT_TMP_TTL_SECS: u64 = 3_600; // 1h
|
||||
|
||||
/// Marker that identifies a temp blob (`<id>.tmp.<pid>-<seq>`). A final
|
||||
/// blob is named just `<id>` (a UUID), so it never contains this.
|
||||
const TMP_MARKER: &str = ".tmp.";
|
||||
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct SweepStats {
|
||||
pub dirs_walked: u64,
|
||||
pub files_deleted: u64,
|
||||
pub bytes_reclaimed: u64,
|
||||
}
|
||||
|
||||
/// Spawn the periodic orphan sweep. Spawned at startup alongside the
|
||||
/// cron scheduler and the realtime/cache GC tasks.
|
||||
pub fn spawn_files_orphan_sweep(files_root: PathBuf) {
|
||||
let interval = Duration::from_secs(read_secs(ENV_INTERVAL, DEFAULT_INTERVAL_SECS));
|
||||
let ttl = Duration::from_secs(read_secs(ENV_TMP_TTL, DEFAULT_TMP_TTL_SECS));
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(interval);
|
||||
ticker.tick().await; // skip the immediate first fire
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let root = files_root.clone();
|
||||
// Blocking filesystem walk off the async worker.
|
||||
let stats = tokio::task::spawn_blocking(move || sweep_orphan_tmp_files(&root, ttl))
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
tracing::info!(
|
||||
dirs_walked = stats.dirs_walked,
|
||||
files_deleted = stats.files_deleted,
|
||||
bytes_reclaimed = stats.bytes_reclaimed,
|
||||
"files orphan sweep complete"
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Walk `<files_root>/files/` and delete `*.tmp.*` files older than
|
||||
/// `ttl`. Missing root is not an error (returns zeroed stats). Pure +
|
||||
/// synchronous so it's unit-testable without a runtime.
|
||||
#[must_use]
|
||||
pub fn sweep_orphan_tmp_files(files_root: &Path, ttl: Duration) -> SweepStats {
|
||||
let mut stats = SweepStats::default();
|
||||
let blobs_dir = files_root.join("files");
|
||||
if !blobs_dir.is_dir() {
|
||||
return stats;
|
||||
}
|
||||
let now = SystemTime::now();
|
||||
walk(&blobs_dir, ttl, now, &mut stats);
|
||||
stats
|
||||
}
|
||||
|
||||
fn walk(dir: &Path, ttl: Duration, now: SystemTime, stats: &mut SweepStats) {
|
||||
stats.dirs_walked += 1;
|
||||
let Ok(entries) = std::fs::read_dir(dir) else {
|
||||
return;
|
||||
};
|
||||
for entry in entries.flatten() {
|
||||
let Ok(ft) = entry.file_type() else {
|
||||
continue;
|
||||
};
|
||||
let path = entry.path();
|
||||
if ft.is_dir() {
|
||||
walk(&path, ttl, now, stats);
|
||||
continue;
|
||||
}
|
||||
if !ft.is_file() {
|
||||
continue;
|
||||
}
|
||||
if !entry.file_name().to_string_lossy().contains(TMP_MARKER) {
|
||||
continue;
|
||||
}
|
||||
let Ok(meta) = entry.metadata() else {
|
||||
continue;
|
||||
};
|
||||
let age = meta
|
||||
.modified()
|
||||
.ok()
|
||||
.and_then(|m| now.duration_since(m).ok())
|
||||
.unwrap_or(Duration::ZERO);
|
||||
if age >= ttl {
|
||||
let size = meta.len();
|
||||
if std::fs::remove_file(&path).is_ok() {
|
||||
stats.files_deleted += 1;
|
||||
stats.bytes_reclaimed += size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn read_secs(key: &str, default: u64) -> u64 {
|
||||
match std::env::var(key) {
|
||||
Err(_) => default,
|
||||
Ok(v) => match v.parse::<u64>() {
|
||||
Ok(n) if n > 0 => n,
|
||||
_ => {
|
||||
tracing::warn!(env = key, value = %v, "invalid; using default");
|
||||
default
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||||
|
||||
fn tmp_root() -> PathBuf {
|
||||
let n = SEQ.fetch_add(1, Ordering::Relaxed);
|
||||
let dir =
|
||||
std::env::temp_dir().join(format!("picloud-sweep-test-{}-{n}", std::process::id()));
|
||||
std::fs::create_dir_all(dir.join("files").join("ab")).unwrap();
|
||||
dir
|
||||
}
|
||||
|
||||
fn touch(path: &Path) {
|
||||
std::fs::write(path, b"x").unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deletes_old_tmp_files() {
|
||||
let root = tmp_root();
|
||||
let tmp = root.join("files/ab/uuid.tmp.123-0");
|
||||
touch(&tmp);
|
||||
// ttl 0 → any tmp file counts as old.
|
||||
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
|
||||
assert_eq!(stats.files_deleted, 1);
|
||||
assert!(!tmp.exists());
|
||||
assert!(stats.bytes_reclaimed >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keeps_young_tmp_files() {
|
||||
let root = tmp_root();
|
||||
let tmp = root.join("files/ab/uuid.tmp.123-0");
|
||||
touch(&tmp);
|
||||
// Large TTL → the just-created file is too young to reap.
|
||||
let stats = sweep_orphan_tmp_files(&root, Duration::from_secs(3600));
|
||||
assert_eq!(stats.files_deleted, 0);
|
||||
assert!(tmp.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keeps_non_tmp_files() {
|
||||
let root = tmp_root();
|
||||
let blob = root.join("files/ab/0123456789abcdef");
|
||||
touch(&blob);
|
||||
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
|
||||
assert_eq!(stats.files_deleted, 0);
|
||||
assert!(blob.exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_root_does_not_panic() {
|
||||
let root = std::env::temp_dir().join("picloud-sweep-nonexistent-xyz");
|
||||
let stats = sweep_orphan_tmp_files(&root, Duration::ZERO);
|
||||
assert_eq!(stats.files_deleted, 0);
|
||||
assert_eq!(stats.dirs_walked, 0);
|
||||
}
|
||||
}
|
||||
793
crates/manager-core/src/http_service.rs
Normal file
793
crates/manager-core/src/http_service.rs
Normal file
@@ -0,0 +1,793 @@
|
||||
//! `HttpServiceImpl` — reqwest-backed outbound HTTP for the v1.1.4
|
||||
//! `http::*` SDK.
|
||||
//!
|
||||
//! Mirrors the v1.1.1+ stateful-service shape (`KvServiceImpl`):
|
||||
//! script-as-gate authz (`AppHttpRequest`, skipped when
|
||||
//! `cx.principal` is `None`), with the backend talking to the network
|
||||
//! instead of Postgres. The reqwest client is built once at startup
|
||||
//! with the [`crate::ssrf::SsrfResolver`] wired in via
|
||||
//! `dns_resolver`, so the SSRF deny-list applies at every connection —
|
||||
//! including each redirect hop, since redirects are followed manually
|
||||
//! through the same client.
|
||||
//!
|
||||
//! Layering vs the raw client:
|
||||
//! 1. URL validation: scheme must be http/https; ports 22/25/465/587
|
||||
//! are blocked. (IP-level filtering is the resolver's job.)
|
||||
//! 2. Body-size caps on both request and response (stream-with-cap on
|
||||
//! the response, checking `Content-Length` first).
|
||||
//! 3. Total-request timeout (default 30s, max 60s) on top of the
|
||||
//! client's 10s connect timeout.
|
||||
//! 4. Default `User-Agent` unless the caller set one.
|
||||
//!
|
||||
//! Bodies/headers are never logged (PII): only url + status + duration
|
||||
//! at debug level.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::env;
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{HttpError, HttpRequest, HttpResponse, HttpService, SdkCallCx};
|
||||
use reqwest::header::{HeaderMap, HeaderName, HeaderValue, CONTENT_TYPE, LOCATION, USER_AGENT};
|
||||
use reqwest::{Client, Method, StatusCode};
|
||||
|
||||
use crate::authz::{self, AuthzRepo, Capability};
|
||||
use crate::ssrf::{self, SsrfPolicy, SSRF_BLOCK_PREFIX};
|
||||
|
||||
/// Default per-request timeout (ms) when the script omits `timeout_ms`.
|
||||
pub const DEFAULT_TIMEOUT_MS: u32 = 30_000;
|
||||
/// Hard ceiling on the per-request timeout. Values above this are
|
||||
/// rejected by the bridge (not silently clamped).
|
||||
pub const MAX_TIMEOUT_MS: u32 = 60_000;
|
||||
/// Default redirect cap.
|
||||
pub const DEFAULT_MAX_REDIRECTS: u32 = 5;
|
||||
/// Hard ceiling on redirects.
|
||||
pub const MAX_REDIRECTS_CEILING: u32 = 10;
|
||||
/// 10 MB default body cap on both directions.
|
||||
const DEFAULT_BODY_LIMIT_BYTES: usize = 10 * 1024 * 1024;
|
||||
/// DNS + connect + TLS hard cap.
|
||||
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
|
||||
/// Outbound-HTTP tunables. Env-overridable following the same pattern
|
||||
/// as `TriggerConfig::from_env`.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct HttpConfig {
|
||||
/// Disables the SSRF deny-list entirely. Dev/test only — the binary
|
||||
/// logs a startup warning when this is set.
|
||||
pub allow_private: bool,
|
||||
pub max_request_body_bytes: usize,
|
||||
pub max_response_body_bytes: usize,
|
||||
}
|
||||
|
||||
impl HttpConfig {
|
||||
#[must_use]
|
||||
pub const fn conservative() -> Self {
|
||||
Self {
|
||||
allow_private: false,
|
||||
max_request_body_bytes: DEFAULT_BODY_LIMIT_BYTES,
|
||||
max_response_body_bytes: DEFAULT_BODY_LIMIT_BYTES,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn from_env() -> Self {
|
||||
let mut c = Self::conservative();
|
||||
if let Ok(v) = env::var("PICLOUD_HTTP_ALLOW_PRIVATE") {
|
||||
c.allow_private =
|
||||
matches!(v.trim().to_ascii_lowercase().as_str(), "1" | "true" | "yes");
|
||||
}
|
||||
load_usize(
|
||||
&mut c.max_request_body_bytes,
|
||||
"PICLOUD_HTTP_MAX_REQUEST_BODY_BYTES",
|
||||
);
|
||||
load_usize(
|
||||
&mut c.max_response_body_bytes,
|
||||
"PICLOUD_HTTP_MAX_RESPONSE_BODY_BYTES",
|
||||
);
|
||||
c
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for HttpConfig {
|
||||
fn default() -> Self {
|
||||
Self::conservative()
|
||||
}
|
||||
}
|
||||
|
||||
fn load_usize(dst: &mut usize, key: &str) {
|
||||
if let Ok(v) = env::var(key) {
|
||||
match v.parse::<usize>() {
|
||||
Ok(n) => *dst = n,
|
||||
Err(e) => {
|
||||
tracing::warn!(env = key, error = %e, "ignoring invalid http-config value");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct HttpServiceImpl {
|
||||
client: Client,
|
||||
authz: Arc<dyn AuthzRepo>,
|
||||
config: HttpConfig,
|
||||
/// Same policy wired into the DNS resolver. Held here too because
|
||||
/// reqwest only routes *hostnames* through the custom resolver — a
|
||||
/// URL with a **literal IP** host bypasses it, so literal IPs are
|
||||
/// checked directly at URL-validation time.
|
||||
policy: SsrfPolicy,
|
||||
}
|
||||
|
||||
impl HttpServiceImpl {
|
||||
/// Build the service, constructing the reqwest client with the SSRF
|
||||
/// resolver. Redirects are followed manually (so per-request limits
|
||||
/// are honored and every hop re-resolves through the SSRF
|
||||
/// resolver), hence `redirect(Policy::none())`.
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// Panics if the reqwest client fails to build — this is a
|
||||
/// startup-time invariant, not a runtime path.
|
||||
#[must_use]
|
||||
pub fn new(config: HttpConfig, authz: Arc<dyn AuthzRepo>) -> Self {
|
||||
let policy = SsrfPolicy::new(config.allow_private);
|
||||
let client = Client::builder()
|
||||
.dns_resolver(ssrf::resolver(policy))
|
||||
.connect_timeout(CONNECT_TIMEOUT)
|
||||
.redirect(reqwest::redirect::Policy::none())
|
||||
.build()
|
||||
.expect("build outbound http client");
|
||||
Self {
|
||||
client,
|
||||
authz,
|
||||
config,
|
||||
policy,
|
||||
}
|
||||
}
|
||||
|
||||
async fn check_request(&self, cx: &SdkCallCx) -> Result<(), HttpError> {
|
||||
if let Some(ref principal) = cx.principal {
|
||||
authz::require(
|
||||
&*self.authz,
|
||||
principal,
|
||||
Capability::AppHttpRequest(cx.app_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| HttpError::Forbidden)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl HttpService for HttpServiceImpl {
|
||||
async fn request(&self, cx: &SdkCallCx, req: HttpRequest) -> Result<HttpResponse, HttpError> {
|
||||
self.check_request(cx).await?;
|
||||
|
||||
// Request body cap.
|
||||
if let Some(ref body) = req.body {
|
||||
if body.len() > self.config.max_request_body_bytes {
|
||||
return Err(HttpError::BodyTooLarge("request"));
|
||||
}
|
||||
}
|
||||
|
||||
let timeout = Duration::from_millis(u64::from(req.timeout_ms.min(MAX_TIMEOUT_MS)));
|
||||
let started = std::time::Instant::now();
|
||||
let url_for_log = req.url.clone();
|
||||
|
||||
// Whole-request budget (DNS + connect + TLS + all redirect hops
|
||||
// + body read). Connect alone is further bounded by the
|
||||
// client's CONNECT_TIMEOUT.
|
||||
let outcome = match tokio::time::timeout(timeout, self.run(req)).await {
|
||||
Ok(r) => r,
|
||||
Err(_) => Err(HttpError::Timeout),
|
||||
};
|
||||
|
||||
let duration_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
|
||||
match &outcome {
|
||||
Ok(resp) => tracing::debug!(
|
||||
url = %url_for_log,
|
||||
status = resp.status,
|
||||
duration_ms,
|
||||
"outbound http"
|
||||
),
|
||||
Err(err) => tracing::debug!(
|
||||
url = %url_for_log,
|
||||
error = %err,
|
||||
duration_ms,
|
||||
"outbound http failed"
|
||||
),
|
||||
}
|
||||
outcome
|
||||
}
|
||||
}
|
||||
|
||||
impl HttpServiceImpl {
|
||||
/// Core request path: validate, build headers, follow redirects
|
||||
/// manually, read the response body with a cap.
|
||||
async fn run(&self, req: HttpRequest) -> Result<HttpResponse, HttpError> {
|
||||
let method = Method::from_bytes(req.method.as_bytes())
|
||||
.map_err(|_| HttpError::Backend(format!("invalid method: {}", req.method)))?;
|
||||
|
||||
let mut current = url::Url::parse(&req.url)
|
||||
.map_err(|e| HttpError::InvalidUrl(format!("{}: {e}", req.url)))?;
|
||||
validate_url(¤t, self.policy)?;
|
||||
|
||||
let mut header_map = build_headers(&req, ¤t)?;
|
||||
let mut method = method;
|
||||
let mut body = req.body.clone();
|
||||
let mut redirects: u32 = 0;
|
||||
let max_redirects = req.max_redirects.min(MAX_REDIRECTS_CEILING);
|
||||
|
||||
loop {
|
||||
// Re-validate scheme/port (and literal-IP SSRF) on each hop.
|
||||
// Hostname IP filtering is the resolver's job and runs
|
||||
// automatically at connect time.
|
||||
validate_url(¤t, self.policy)?;
|
||||
|
||||
let mut rb = self.client.request(method.clone(), current.clone());
|
||||
rb = rb.headers(header_map.clone());
|
||||
if let Some(ref b) = body {
|
||||
rb = rb.body(b.clone());
|
||||
}
|
||||
let resp = rb.send().await.map_err(map_reqwest_err)?;
|
||||
let status = resp.status();
|
||||
|
||||
if req.follow_redirects && is_redirect(status) {
|
||||
if let Some(loc) = resp.headers().get(LOCATION) {
|
||||
if redirects >= max_redirects {
|
||||
return Err(HttpError::Backend(format!(
|
||||
"too many redirects (max {max_redirects})"
|
||||
)));
|
||||
}
|
||||
redirects += 1;
|
||||
let loc_str = loc.to_str().map_err(|_| {
|
||||
HttpError::Backend("redirect Location not valid UTF-8".into())
|
||||
})?;
|
||||
current = current
|
||||
.join(loc_str)
|
||||
.map_err(|e| HttpError::InvalidUrl(format!("redirect target: {e}")))?;
|
||||
|
||||
// 303 always → GET; 301/302 historically downgrade
|
||||
// POST→GET (matches browsers). 307/308 preserve.
|
||||
if matches!(status.as_u16(), 301..=303) {
|
||||
method = Method::GET;
|
||||
body = None;
|
||||
header_map.remove(CONTENT_TYPE);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
return self.read_capped(resp).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn read_capped(&self, resp: reqwest::Response) -> Result<HttpResponse, HttpError> {
|
||||
let status = resp.status().as_u16();
|
||||
let mut headers = BTreeMap::new();
|
||||
for (name, value) in resp.headers() {
|
||||
// Header names lowercased per the documented response shape.
|
||||
headers.insert(
|
||||
name.as_str().to_ascii_lowercase(),
|
||||
value.to_str().unwrap_or("").to_string(),
|
||||
);
|
||||
}
|
||||
|
||||
let cap = self.config.max_response_body_bytes;
|
||||
if let Some(len) = resp.content_length() {
|
||||
if len > cap as u64 {
|
||||
return Err(HttpError::BodyTooLarge("response"));
|
||||
}
|
||||
}
|
||||
|
||||
let mut buf: Vec<u8> = Vec::new();
|
||||
let mut resp = resp;
|
||||
while let Some(chunk) = resp.chunk().await.map_err(map_reqwest_err)? {
|
||||
if buf.len() + chunk.len() > cap {
|
||||
return Err(HttpError::BodyTooLarge("response"));
|
||||
}
|
||||
buf.extend_from_slice(&chunk);
|
||||
}
|
||||
let body_raw = String::from_utf8_lossy(&buf).into_owned();
|
||||
Ok(HttpResponse {
|
||||
status,
|
||||
headers,
|
||||
body_raw,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// http/https only; block the SSH + SMTP ports; apply the SSRF policy
|
||||
/// to **literal-IP** hosts (hostnames are filtered by the DNS resolver
|
||||
/// at connect time, but literal IPs never reach the resolver).
|
||||
fn validate_url(url: &url::Url, policy: SsrfPolicy) -> Result<(), HttpError> {
|
||||
match url.scheme() {
|
||||
"http" | "https" => {}
|
||||
other => return Err(HttpError::BlockedScheme(other.to_string())),
|
||||
}
|
||||
match url.host() {
|
||||
None => return Err(HttpError::InvalidUrl("missing host".into())),
|
||||
Some(url::Host::Ipv4(ip)) => {
|
||||
policy
|
||||
.check(std::net::IpAddr::V4(ip))
|
||||
.map_err(|reason| HttpError::Ssrf(reason.to_string()))?;
|
||||
}
|
||||
Some(url::Host::Ipv6(ip)) => {
|
||||
policy
|
||||
.check(std::net::IpAddr::V6(ip))
|
||||
.map_err(|reason| HttpError::Ssrf(reason.to_string()))?;
|
||||
}
|
||||
Some(url::Host::Domain(_)) => {}
|
||||
}
|
||||
let port = url
|
||||
.port_or_known_default()
|
||||
.unwrap_or(if url.scheme() == "https" { 443 } else { 80 });
|
||||
if matches!(port, 22 | 25 | 465 | 587) {
|
||||
return Err(HttpError::BlockedPort(port));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Build the request header map: merge caller headers, then apply the
|
||||
/// default `User-Agent` (unless overridden) and the bridge-chosen
|
||||
/// `Content-Type` (unless overridden).
|
||||
fn build_headers(req: &HttpRequest, _url: &url::Url) -> Result<HeaderMap, HttpError> {
|
||||
let mut map = HeaderMap::new();
|
||||
let mut has_user_agent = false;
|
||||
let mut has_content_type = false;
|
||||
for (k, v) in &req.headers {
|
||||
let name = HeaderName::from_bytes(k.as_bytes())
|
||||
.map_err(|_| HttpError::Backend(format!("invalid header name: {k}")))?;
|
||||
let value = HeaderValue::from_str(v)
|
||||
.map_err(|_| HttpError::Backend(format!("invalid header value for {k}")))?;
|
||||
if name == USER_AGENT {
|
||||
has_user_agent = true;
|
||||
}
|
||||
if name == CONTENT_TYPE {
|
||||
has_content_type = true;
|
||||
}
|
||||
map.append(name, value);
|
||||
}
|
||||
|
||||
if !has_user_agent {
|
||||
let script = req.script_id.as_deref().unwrap_or("unknown");
|
||||
let ua = format!(
|
||||
"picloud/{} (script:{})",
|
||||
picloud_shared::PRODUCT_VERSION,
|
||||
script
|
||||
);
|
||||
if let Ok(value) = HeaderValue::from_str(&ua) {
|
||||
map.insert(USER_AGENT, value);
|
||||
}
|
||||
}
|
||||
|
||||
if !has_content_type {
|
||||
if let Some(ref ct) = req.content_type {
|
||||
if let Ok(value) = HeaderValue::from_str(ct) {
|
||||
map.insert(CONTENT_TYPE, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(map)
|
||||
}
|
||||
|
||||
const fn is_redirect(status: StatusCode) -> bool {
|
||||
matches!(status.as_u16(), 301..=303 | 307 | 308)
|
||||
}
|
||||
|
||||
/// Map a reqwest error to an `HttpError`, never leaking the resolved
|
||||
/// IP. SSRF blocks are detected by scanning the error source chain for
|
||||
/// the resolver's marker prefix.
|
||||
fn map_reqwest_err(err: reqwest::Error) -> HttpError {
|
||||
if let Some(reason) = ssrf_reason(&err) {
|
||||
return HttpError::Ssrf(reason);
|
||||
}
|
||||
if err.is_timeout() {
|
||||
return HttpError::Timeout;
|
||||
}
|
||||
if err.is_connect() {
|
||||
return HttpError::Network("connection failed".into());
|
||||
}
|
||||
if err.is_request() {
|
||||
return HttpError::Network("request failed".into());
|
||||
}
|
||||
HttpError::Network("network error".into())
|
||||
}
|
||||
|
||||
/// Walk the error source chain looking for the SSRF marker the resolver
|
||||
/// embeds. Returns the category reason (no IP) when found.
|
||||
fn ssrf_reason(err: &reqwest::Error) -> Option<String> {
|
||||
let mut src: Option<&(dyn std::error::Error + 'static)> = Some(err);
|
||||
while let Some(e) = src {
|
||||
let s = e.to_string();
|
||||
if let Some(idx) = s.find(SSRF_BLOCK_PREFIX) {
|
||||
return Some(s[idx + SSRF_BLOCK_PREFIX.len()..].to_string());
|
||||
}
|
||||
src = e.source();
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::authz::AuthzError;
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, Principal, RequestId, ScriptId,
|
||||
UserId,
|
||||
};
|
||||
use std::collections::BTreeMap;
|
||||
use std::io::Write as _;
|
||||
use std::net::SocketAddr;
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
struct AllowAuthz;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for AllowAuthz {
|
||||
async fn membership(&self, _u: UserId, _a: AppId) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(Some(AppRole::Editor))
|
||||
}
|
||||
}
|
||||
struct DenyAuthz;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for DenyAuthz {
|
||||
async fn membership(&self, _u: UserId, _a: AppId) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
fn dev_service(authz: Arc<dyn AuthzRepo>) -> HttpServiceImpl {
|
||||
// allow_private so the test TcpListener on 127.0.0.1 is reachable.
|
||||
let mut config = HttpConfig::conservative();
|
||||
config.allow_private = true;
|
||||
HttpServiceImpl::new(config, authz)
|
||||
}
|
||||
|
||||
fn anon_cx() -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id: AppId::new(),
|
||||
script_id: ScriptId::new(),
|
||||
principal: None,
|
||||
execution_id: ExecutionId::new(),
|
||||
request_id: RequestId::new(),
|
||||
trigger_depth: 0,
|
||||
root_execution_id: ExecutionId::new(),
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn member_cx() -> SdkCallCx {
|
||||
let mut cx = anon_cx();
|
||||
cx.principal = Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
scopes: None,
|
||||
app_binding: None,
|
||||
});
|
||||
cx
|
||||
}
|
||||
|
||||
fn req(method: &str, url: String) -> HttpRequest {
|
||||
HttpRequest {
|
||||
method: method.into(),
|
||||
url,
|
||||
headers: BTreeMap::new(),
|
||||
body: None,
|
||||
content_type: None,
|
||||
timeout_ms: 5000,
|
||||
follow_redirects: true,
|
||||
max_redirects: 5,
|
||||
script_id: Some("test-script".into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Minimal single-shot HTTP/1.1 server. Reads the request, runs
|
||||
/// `handler` to produce the raw response bytes, writes them, closes.
|
||||
/// Returns the bound address.
|
||||
async fn spawn_server<F>(handler: F) -> SocketAddr
|
||||
where
|
||||
F: Fn(String) -> Vec<u8> + Send + Sync + 'static,
|
||||
{
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
let Ok((mut sock, _)) = listener.accept().await else {
|
||||
break;
|
||||
};
|
||||
let mut buf = vec![0u8; 65536];
|
||||
let n = sock.read(&mut buf).await.unwrap_or(0);
|
||||
let request = String::from_utf8_lossy(&buf[..n]).to_string();
|
||||
let response = handler(request);
|
||||
let _ = sock.write_all(&response).await;
|
||||
let _ = sock.flush().await;
|
||||
}
|
||||
});
|
||||
addr
|
||||
}
|
||||
|
||||
fn ok_response(body: &str, content_type: &str) -> Vec<u8> {
|
||||
let mut v = Vec::new();
|
||||
write!(
|
||||
v,
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
)
|
||||
.unwrap();
|
||||
v
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn get_round_trip() {
|
||||
let addr = spawn_server(|_req| ok_response("hello", "text/plain")).await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let resp = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status, 200);
|
||||
assert_eq!(resp.body_raw, "hello");
|
||||
assert_eq!(
|
||||
resp.headers.get("content-type").map(String::as_str),
|
||||
Some("text/plain")
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn post_sends_body_and_default_user_agent() {
|
||||
let addr = spawn_server(|request| {
|
||||
// Echo back whether the body + default UA were present.
|
||||
let has_ua = request.to_lowercase().contains("user-agent: picloud/");
|
||||
let has_body = request.contains("xyzzy");
|
||||
ok_response(&format!("ua={has_ua},body={has_body}"), "text/plain")
|
||||
})
|
||||
.await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let mut r = req("POST", format!("http://{addr}/"));
|
||||
r.body = Some(b"xyzzy".to_vec());
|
||||
r.content_type = Some("text/plain".into());
|
||||
let resp = svc.request(&anon_cx(), r).await.unwrap();
|
||||
assert_eq!(resp.body_raw, "ua=true,body=true");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn custom_user_agent_overrides_default() {
|
||||
let addr = spawn_server(|request| {
|
||||
let has_custom = request.to_lowercase().contains("user-agent: my-agent");
|
||||
let has_default = request.to_lowercase().contains("picloud/");
|
||||
ok_response(
|
||||
&format!("custom={has_custom},default={has_default}"),
|
||||
"text/plain",
|
||||
)
|
||||
})
|
||||
.await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let mut r = req("GET", format!("http://{addr}/"));
|
||||
r.headers.insert("User-Agent".into(), "my-agent".into());
|
||||
let resp = svc.request(&anon_cx(), r).await.unwrap();
|
||||
assert_eq!(resp.body_raw, "custom=true,default=false");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_body_response() {
|
||||
let addr = spawn_server(|_r| {
|
||||
b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".to_vec()
|
||||
})
|
||||
.await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let resp = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status, 204);
|
||||
assert_eq!(resp.body_raw, "");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn non_2xx_does_not_error() {
|
||||
let addr = spawn_server(|_r| {
|
||||
b"HTTP/1.1 500 Internal Server Error\r\nContent-Length: 3\r\nConnection: close\r\n\r\nerr".to_vec()
|
||||
})
|
||||
.await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let resp = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status, 500);
|
||||
assert_eq!(resp.body_raw, "err");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn response_over_content_length_cap_rejected() {
|
||||
let addr = spawn_server(|_r| ok_response("0123456789", "text/plain")).await;
|
||||
let mut config = HttpConfig::conservative();
|
||||
config.allow_private = true;
|
||||
config.max_response_body_bytes = 5; // body is 10 bytes
|
||||
let svc = HttpServiceImpl::new(config, Arc::new(AllowAuthz));
|
||||
let err = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, HttpError::BodyTooLarge("response")));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn response_over_cap_without_content_length_rejected_mid_stream() {
|
||||
// No Content-Length header → must be caught while streaming.
|
||||
let addr = spawn_server(|_r| {
|
||||
b"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n0123456789ABCDEF".to_vec()
|
||||
})
|
||||
.await;
|
||||
let mut config = HttpConfig::conservative();
|
||||
config.allow_private = true;
|
||||
config.max_response_body_bytes = 4;
|
||||
let svc = HttpServiceImpl::new(config, Arc::new(AllowAuthz));
|
||||
let err = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, HttpError::BodyTooLarge("response")));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn request_body_over_cap_rejected_before_send() {
|
||||
let mut config = HttpConfig::conservative();
|
||||
config.allow_private = true;
|
||||
config.max_request_body_bytes = 3;
|
||||
let svc = HttpServiceImpl::new(config, Arc::new(AllowAuthz));
|
||||
let mut r = req("POST", "http://127.0.0.1:1/".into());
|
||||
r.body = Some(b"too long".to_vec());
|
||||
let err = svc.request(&anon_cx(), r).await.unwrap_err();
|
||||
assert!(matches!(err, HttpError::BodyTooLarge("request")));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn redirect_followed_up_to_then_throws_beyond_max() {
|
||||
// Server always 302s to itself → unbounded redirect loop,
|
||||
// bounded by max_redirects.
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
let Ok((mut sock, _)) = listener.accept().await else {
|
||||
break;
|
||||
};
|
||||
let mut buf = vec![0u8; 4096];
|
||||
let _ = sock.read(&mut buf).await;
|
||||
let body = format!(
|
||||
"HTTP/1.1 302 Found\r\nLocation: http://{addr}/next\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
);
|
||||
let _ = sock.write_all(body.as_bytes()).await;
|
||||
}
|
||||
});
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let mut r = req("GET", format!("http://{addr}/"));
|
||||
r.max_redirects = 2;
|
||||
let err = svc.request(&anon_cx(), r).await.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, HttpError::Backend(ref m) if m.contains("too many redirects")),
|
||||
"expected too-many-redirects, got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn scheme_rejected() {
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
for url in ["file:///etc/passwd", "ftp://host/x", "gopher://host/"] {
|
||||
let err = svc
|
||||
.request(&anon_cx(), req("GET", url.into()))
|
||||
.await
|
||||
.unwrap_err();
|
||||
match err {
|
||||
HttpError::BlockedScheme(s) => {
|
||||
assert!(url.starts_with(&s), "scheme {s} not in url {url}");
|
||||
}
|
||||
other => panic!("expected BlockedScheme for {url}, got {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ports_rejected() {
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
for port in [22u16, 25, 465, 587] {
|
||||
let err = svc
|
||||
.request(
|
||||
&anon_cx(),
|
||||
req("GET", format!("http://example.com:{port}/")),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, HttpError::BlockedPort(p) if p == port),
|
||||
"port {port} should be blocked, got {err:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ssrf_blocks_loopback_without_allow_private() {
|
||||
// Default config (deny-list ON). A request to a loopback host
|
||||
// must surface as Ssrf, not a generic network error.
|
||||
let svc = HttpServiceImpl::new(HttpConfig::conservative(), Arc::new(AllowAuthz));
|
||||
let err = svc
|
||||
.request(&anon_cx(), req("GET", "http://127.0.0.1:9/".into()))
|
||||
.await
|
||||
.unwrap_err();
|
||||
match err {
|
||||
HttpError::Ssrf(reason) => {
|
||||
assert_eq!(reason, "loopback");
|
||||
assert!(!reason.contains("127.0.0.1"), "reason must not leak the IP");
|
||||
}
|
||||
other => panic!("expected Ssrf, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ssrf_blocks_hostname_resolving_to_loopback() {
|
||||
// `localhost` resolves to 127.0.0.1 / ::1 — all denied. This
|
||||
// exercises the DNS-resolver path (vs the literal-IP path) and
|
||||
// must surface as Ssrf, not a generic DNS error.
|
||||
let svc = HttpServiceImpl::new(HttpConfig::conservative(), Arc::new(AllowAuthz));
|
||||
let err = svc
|
||||
.request(&anon_cx(), req("GET", "http://localhost:9/".into()))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, HttpError::Ssrf(_)),
|
||||
"expected Ssrf for localhost, got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn timeout_throws() {
|
||||
// Server that accepts then never responds.
|
||||
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = listener.local_addr().unwrap();
|
||||
tokio::spawn(async move {
|
||||
if let Ok((sock, _)) = listener.accept().await {
|
||||
// Hold the socket open without replying.
|
||||
tokio::time::sleep(Duration::from_secs(30)).await;
|
||||
drop(sock);
|
||||
}
|
||||
});
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let mut r = req("GET", format!("http://{addr}/"));
|
||||
r.timeout_ms = 300;
|
||||
let err = svc.request(&anon_cx(), r).await.unwrap_err();
|
||||
assert!(matches!(err, HttpError::Timeout), "got {err:?}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn anon_skips_authz_member_without_scope_forbidden() {
|
||||
let addr = spawn_server(|_r| ok_response("ok", "text/plain")).await;
|
||||
// Anonymous principal → authz skipped even with DenyAuthz.
|
||||
let svc = dev_service(Arc::new(DenyAuthz));
|
||||
let ok = svc
|
||||
.request(&anon_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await;
|
||||
assert!(ok.is_ok());
|
||||
// Authenticated member with no role → Forbidden.
|
||||
let err = svc
|
||||
.request(&member_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, HttpError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn member_with_role_allowed() {
|
||||
let addr = spawn_server(|_r| ok_response("ok", "text/plain")).await;
|
||||
let svc = dev_service(Arc::new(AllowAuthz));
|
||||
let resp = svc
|
||||
.request(&member_cx(), req("GET", format!("http://{addr}/")))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(resp.status, 200);
|
||||
}
|
||||
}
|
||||
@@ -188,7 +188,7 @@ mod tests {
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, NoopEventEmitter, Principal,
|
||||
RequestId, UserId,
|
||||
RequestId, ScriptId, UserId,
|
||||
};
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use tokio::sync::Mutex;
|
||||
@@ -301,6 +301,7 @@ mod tests {
|
||||
fn anon_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: None,
|
||||
execution_id: ExecutionId::new(),
|
||||
request_id: RequestId::new(),
|
||||
@@ -314,6 +315,7 @@ mod tests {
|
||||
fn owner_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Owner,
|
||||
@@ -332,6 +334,7 @@ mod tests {
|
||||
fn member_no_role_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
|
||||
@@ -16,12 +16,14 @@ pub mod app_domain_repo;
|
||||
pub mod app_members_api;
|
||||
pub mod app_members_repo;
|
||||
pub mod app_repo;
|
||||
pub mod app_secrets_repo;
|
||||
pub mod apps_api;
|
||||
pub mod auth;
|
||||
pub mod auth_api;
|
||||
pub mod auth_bootstrap;
|
||||
pub mod auth_middleware;
|
||||
pub mod authz;
|
||||
pub mod cron_scheduler;
|
||||
pub mod dead_letter_repo;
|
||||
pub mod dead_letter_service;
|
||||
pub mod dead_letters_api;
|
||||
@@ -29,7 +31,12 @@ pub mod dispatcher;
|
||||
pub mod docs_filter;
|
||||
pub mod docs_repo;
|
||||
pub mod docs_service;
|
||||
pub mod files_api;
|
||||
pub mod files_repo;
|
||||
pub mod files_service;
|
||||
pub mod files_sweep;
|
||||
pub mod gc;
|
||||
pub mod http_service;
|
||||
pub mod kv_repo;
|
||||
pub mod kv_service;
|
||||
pub mod log_sink;
|
||||
@@ -38,11 +45,17 @@ pub mod module_source;
|
||||
pub mod outbox_event_emitter;
|
||||
pub mod outbox_repo;
|
||||
pub mod principal_resolver;
|
||||
pub mod pubsub_repo;
|
||||
pub mod pubsub_service;
|
||||
pub mod realtime_authority;
|
||||
pub mod repo;
|
||||
pub mod route_admin;
|
||||
pub mod route_repo;
|
||||
pub mod sandbox;
|
||||
pub mod scheduler;
|
||||
pub mod ssrf;
|
||||
pub mod topic_repo;
|
||||
pub mod topics_api;
|
||||
pub mod trigger_config;
|
||||
pub mod trigger_repo;
|
||||
pub mod triggers_api;
|
||||
@@ -73,6 +86,9 @@ pub use app_members_repo::{
|
||||
PostgresAppMembersRepository,
|
||||
};
|
||||
pub use app_repo::{resolve_app, AppLookup, AppRepository, PostgresAppRepository};
|
||||
pub use app_secrets_repo::{
|
||||
AppSecretsRepo, AppSecretsRepoError, PostgresAppSecretsRepo, SIGNING_KEY_LEN,
|
||||
};
|
||||
pub use apps_api::{apps_router, AppsState};
|
||||
pub use auth_api::auth_router;
|
||||
pub use auth_bootstrap::{
|
||||
@@ -84,6 +100,7 @@ pub use auth_middleware::{
|
||||
API_KEY_PREFIX, API_KEY_PREFIX_LEN, SESSION_COOKIE,
|
||||
};
|
||||
pub use authz::{can, require, AuthzDenied, AuthzError, AuthzRepo, Capability, Decision};
|
||||
pub use cron_scheduler::spawn_cron_scheduler;
|
||||
pub use dead_letter_repo::{
|
||||
DeadLetterRepo, DeadLetterRepoError, DeadLetterRow, NewDeadLetter, PostgresDeadLetterRepo,
|
||||
};
|
||||
@@ -92,7 +109,12 @@ pub use dead_letters_api::{dead_letters_router, DeadLettersApiError, DeadLetters
|
||||
pub use dispatcher::{compute_backoff, Dispatcher, DispatcherError};
|
||||
pub use docs_repo::{DocsRepo, DocsRepoError, PostgresDocsRepo};
|
||||
pub use docs_service::DocsServiceImpl;
|
||||
pub use files_api::{files_admin_router, FilesAdminState};
|
||||
pub use files_repo::{FilesConfig, FilesRepo, FilesRepoError, FsFilesRepo};
|
||||
pub use files_service::FilesServiceImpl;
|
||||
pub use files_sweep::{spawn_files_orphan_sweep, sweep_orphan_tmp_files, SweepStats};
|
||||
pub use gc::{spawn_abandoned_gc, spawn_dead_letter_gc};
|
||||
pub use http_service::{HttpConfig, HttpServiceImpl};
|
||||
pub use kv_repo::{KvRepo, KvRepoError, PostgresKvRepo};
|
||||
pub use kv_service::KvServiceImpl;
|
||||
pub use log_sink::PostgresExecutionLogSink;
|
||||
@@ -102,6 +124,9 @@ pub use outbox_repo::{
|
||||
NewOutboxRow, OutboxRepo, OutboxRepoError, OutboxRow, OutboxSourceKind, PostgresOutboxRepo,
|
||||
};
|
||||
pub use principal_resolver::{AdminPrincipalResolver, PrincipalResolver, PrincipalResolverError};
|
||||
pub use pubsub_repo::{PostgresPubsubRepo, PublishCtx, PubsubRepo, PubsubRepoError};
|
||||
pub use pubsub_service::{PubsubServiceImpl, SubscriberTokenConfig};
|
||||
pub use realtime_authority::RealtimeAuthorityImpl;
|
||||
pub use repo::{
|
||||
ExecutionLogRepository, NewScript, PostgresExecutionLogRepository, PostgresScriptRepository,
|
||||
RepoResolver, ScriptPatch, ScriptRepository, ScriptRepositoryError,
|
||||
@@ -109,10 +134,13 @@ pub use repo::{
|
||||
pub use route_admin::{compile_routes, route_admin_router, RouteAdminState};
|
||||
pub use route_repo::{NewRoute, PostgresRouteRepository, RouteRepository};
|
||||
pub use sandbox::{CeilingError, SandboxCeiling};
|
||||
pub use topic_repo::{PostgresTopicRepo, Topic, TopicAuthMode, TopicRepo, TopicRepoError};
|
||||
pub use topics_api::{topics_router, TopicsApiError, TopicsState};
|
||||
pub use trigger_config::{BackoffShape, TriggerConfig};
|
||||
pub use trigger_repo::{
|
||||
collection_matches, CreateDeadLetterTrigger, CreateDocsTrigger, CreateKvTrigger,
|
||||
DeadLetterTriggerMatch, DocsTriggerMatch, KvTriggerMatch, PostgresTriggerRepo, Trigger,
|
||||
TriggerDetails, TriggerDispatchMode, TriggerKind, TriggerRepo, TriggerRepoError,
|
||||
collection_matches, CreateDeadLetterTrigger, CreateDocsTrigger, CreateFilesTrigger,
|
||||
CreateKvTrigger, CreatePubsubTrigger, DeadLetterTriggerMatch, DocsTriggerMatch,
|
||||
FilesTriggerMatch, KvTriggerMatch, PostgresTriggerRepo, Trigger, TriggerDetails,
|
||||
TriggerDispatchMode, TriggerKind, TriggerRepo, TriggerRepoError,
|
||||
};
|
||||
pub use triggers_api::{triggers_router, TriggersApiError, TriggersState};
|
||||
|
||||
@@ -19,7 +19,8 @@ use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
DocsEventOp, EmitError, KvEventOp, SdkCallCx, ServiceEvent, ServiceEventEmitter, TriggerEvent,
|
||||
DocsEventOp, EmitError, FileMeta, FilesEventOp, KvEventOp, SdkCallCx, ServiceEvent,
|
||||
ServiceEventEmitter, TriggerEvent,
|
||||
};
|
||||
|
||||
use crate::outbox_repo::{NewOutboxRow, OutboxRepo, OutboxSourceKind};
|
||||
@@ -43,6 +44,7 @@ impl ServiceEventEmitter for OutboxEventEmitter {
|
||||
match event.source {
|
||||
"kv" => self.emit_kv(cx, event).await,
|
||||
"docs" => self.emit_docs(cx, event).await,
|
||||
"files" => self.emit_files(cx, event).await,
|
||||
// Future sources land here. For now, silently drop — the
|
||||
// SDK calls `events.emit(...)` unconditionally for forward
|
||||
// compat, so swallowing without an error is correct.
|
||||
@@ -154,4 +156,68 @@ impl OutboxEventEmitter {
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// v1.1.5. Fan out a files mutation across matching files triggers.
|
||||
/// The `ServiceEvent.payload` is the file **metadata** (never the
|
||||
/// blob bytes); `old_payload` is the prior metadata (the deleted
|
||||
/// row's metadata on delete). The `TriggerEvent::Files` carries the
|
||||
/// metadata fields explicitly + `prev` for the change-data-capture
|
||||
/// surface.
|
||||
async fn emit_files(&self, cx: &SdkCallCx, event: ServiceEvent) -> Result<(), EmitError> {
|
||||
let Some(op) = FilesEventOp::from_wire(event.op) else {
|
||||
return Ok(());
|
||||
};
|
||||
let Some(collection) = event.collection.clone() else {
|
||||
return Ok(());
|
||||
};
|
||||
// The payload is the FileMeta JSON the FilesServiceImpl emitted.
|
||||
let Some(meta) = event
|
||||
.payload
|
||||
.clone()
|
||||
.and_then(|v| serde_json::from_value::<FileMeta>(v).ok())
|
||||
else {
|
||||
return Ok(());
|
||||
};
|
||||
|
||||
let matches = self
|
||||
.triggers
|
||||
.list_matching_files(cx.app_id, &collection, op)
|
||||
.await
|
||||
.map_err(|e| EmitError::Unavailable(format!("trigger lookup: {e}")))?;
|
||||
|
||||
if matches.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let trigger_event = TriggerEvent::Files {
|
||||
op,
|
||||
collection,
|
||||
id: meta.id.to_string(),
|
||||
name: meta.name,
|
||||
content_type: meta.content_type,
|
||||
size: meta.size,
|
||||
checksum: meta.checksum,
|
||||
prev: event.old_payload.clone(),
|
||||
};
|
||||
let payload = serde_json::to_value(&trigger_event)
|
||||
.map_err(|e| EmitError::Rejected(format!("event serialize: {e}")))?;
|
||||
|
||||
for m in matches {
|
||||
self.outbox
|
||||
.insert(NewOutboxRow {
|
||||
app_id: cx.app_id,
|
||||
source_kind: OutboxSourceKind::Files,
|
||||
trigger_id: Some(m.trigger_id),
|
||||
script_id: Some(m.script_id),
|
||||
reply_to: None,
|
||||
payload: payload.clone(),
|
||||
origin_principal: cx.principal.as_ref().map(|p| p.user_id),
|
||||
trigger_depth: cx.trigger_depth.saturating_add(1),
|
||||
root_execution_id: Some(cx.root_execution_id),
|
||||
})
|
||||
.await
|
||||
.map_err(|e| EmitError::Unavailable(format!("outbox insert: {e}")))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,6 +25,12 @@ pub enum OutboxSourceKind {
|
||||
/// v1.1.2.
|
||||
Docs,
|
||||
DeadLetter,
|
||||
/// v1.1.4.
|
||||
Cron,
|
||||
/// v1.1.5.
|
||||
Files,
|
||||
/// v1.1.5.
|
||||
Pubsub,
|
||||
}
|
||||
|
||||
impl OutboxSourceKind {
|
||||
@@ -35,6 +41,9 @@ impl OutboxSourceKind {
|
||||
Self::Kv => "kv",
|
||||
Self::Docs => "docs",
|
||||
Self::DeadLetter => "dead_letter",
|
||||
Self::Cron => "cron",
|
||||
Self::Files => "files",
|
||||
Self::Pubsub => "pubsub",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -45,6 +54,9 @@ impl OutboxSourceKind {
|
||||
"kv" => Some(Self::Kv),
|
||||
"docs" => Some(Self::Docs),
|
||||
"dead_letter" => Some(Self::DeadLetter),
|
||||
"cron" => Some(Self::Cron),
|
||||
"files" => Some(Self::Files),
|
||||
"pubsub" => Some(Self::Pubsub),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
118
crates/manager-core/src/pubsub_repo.rs
Normal file
118
crates/manager-core/src/pubsub_repo.rs
Normal file
@@ -0,0 +1,118 @@
|
||||
//! `PubsubRepo` — publish-time fan-out for the v1.1.5 `pubsub::*` SDK.
|
||||
//!
|
||||
//! `publish_durable` writes one outbox row per matching enabled `pubsub`
|
||||
//! trigger, all inside a single transaction so a partial fan-out (some
|
||||
//! subscribers got rows, others didn't, then a crash) can't happen.
|
||||
//! Each delivery row then retries / dead-letters independently through
|
||||
//! the existing dispatcher — no pub/sub-specific dispatch branching.
|
||||
//!
|
||||
//! Topic pattern matching runs in Rust (`picloud_shared::topic_matches`)
|
||||
//! against the small set of the app's enabled pubsub triggers, keeping
|
||||
//! the SELECT trivial. v1.2 can add a topic-trie index if fan-out
|
||||
//! becomes a hot path.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{topic_matches, AdminUserId, AppId, ExecutionId};
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum PubsubRepoError {
|
||||
#[error("database error: {0}")]
|
||||
Db(#[from] sqlx::Error),
|
||||
}
|
||||
|
||||
/// The execution-context bits a fan-out needs to stamp onto each outbox
|
||||
/// row. Derived from the publishing script's `SdkCallCx`.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct PublishCtx {
|
||||
pub app_id: AppId,
|
||||
pub origin_principal: Option<AdminUserId>,
|
||||
pub trigger_depth: u32,
|
||||
pub root_execution_id: ExecutionId,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait PubsubRepo: Send + Sync {
|
||||
/// Fan out a publish to every matching enabled pubsub trigger in
|
||||
/// `ctx.app_id`, inserting one outbox row each in a SINGLE
|
||||
/// transaction. `event_payload` is the serialized
|
||||
/// `TriggerEvent::Pubsub`. Returns the number of delivery rows
|
||||
/// written (0 when no trigger matched — the publish still succeeds).
|
||||
async fn fan_out_publish(
|
||||
&self,
|
||||
ctx: PublishCtx,
|
||||
topic: &str,
|
||||
event_payload: serde_json::Value,
|
||||
) -> Result<u32, PubsubRepoError>;
|
||||
}
|
||||
|
||||
pub struct PostgresPubsubRepo {
|
||||
pool: PgPool,
|
||||
}
|
||||
|
||||
impl PostgresPubsubRepo {
|
||||
#[must_use]
|
||||
pub fn new(pool: PgPool) -> Self {
|
||||
Self { pool }
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct PubsubTriggerRow {
|
||||
id: Uuid,
|
||||
script_id: Uuid,
|
||||
topic_pattern: String,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubRepo for PostgresPubsubRepo {
|
||||
async fn fan_out_publish(
|
||||
&self,
|
||||
ctx: PublishCtx,
|
||||
topic: &str,
|
||||
event_payload: serde_json::Value,
|
||||
) -> Result<u32, PubsubRepoError> {
|
||||
let mut tx = self.pool.begin().await?;
|
||||
|
||||
// Load all enabled pubsub triggers for the app; filter by topic
|
||||
// pattern in Rust (keeps the query simple, honours the
|
||||
// empty/`*`/prefix semantics without teaching SQL about globs).
|
||||
let rows: Vec<PubsubTriggerRow> = sqlx::query_as(
|
||||
"SELECT t.id, t.script_id, d.topic_pattern \
|
||||
FROM triggers t \
|
||||
JOIN pubsub_trigger_details d ON d.trigger_id = t.id \
|
||||
WHERE t.app_id = $1 AND t.kind = 'pubsub' AND t.enabled = TRUE",
|
||||
)
|
||||
.bind(ctx.app_id.into_inner())
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
|
||||
let mut written: u32 = 0;
|
||||
for r in rows {
|
||||
if !topic_matches(&r.topic_pattern, topic) {
|
||||
continue;
|
||||
}
|
||||
sqlx::query(
|
||||
"INSERT INTO outbox ( \
|
||||
app_id, source_kind, trigger_id, script_id, reply_to, \
|
||||
payload, origin_principal, trigger_depth, root_execution_id \
|
||||
) VALUES ($1, 'pubsub', $2, $3, NULL, $4, $5, $6, $7)",
|
||||
)
|
||||
.bind(ctx.app_id.into_inner())
|
||||
.bind(r.id)
|
||||
.bind(r.script_id)
|
||||
.bind(&event_payload)
|
||||
.bind(ctx.origin_principal.map(AdminUserId::into_inner))
|
||||
.bind(i32::try_from(ctx.trigger_depth.saturating_add(1)).unwrap_or(1))
|
||||
.bind(ctx.root_execution_id.into_inner())
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
written += 1;
|
||||
}
|
||||
|
||||
// Commit once — all rows or none.
|
||||
tx.commit().await?;
|
||||
Ok(written)
|
||||
}
|
||||
}
|
||||
726
crates/manager-core/src/pubsub_service.rs
Normal file
726
crates/manager-core/src/pubsub_service.rs
Normal file
@@ -0,0 +1,726 @@
|
||||
//! `PubsubServiceImpl` — wires `PubsubRepo` underneath the
|
||||
//! `picloud_shared::PubsubService` trait scripts see via the Rhai
|
||||
//! bridge.
|
||||
//!
|
||||
//! Mirrors the other stateful services: script-as-gate authz
|
||||
//! (`AppPubsubPublish`, skipped when `cx.principal` is `None`), with the
|
||||
//! backend doing a publish-time outbox fan-out instead of a row write.
|
||||
//! No `ServiceEventEmitter` here — pub/sub publishes directly to the
|
||||
//! outbox; it doesn't mutate local data that other triggers observe.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::subscriber_token::{self, TokenClaims};
|
||||
use picloud_shared::{
|
||||
PubsubError, PubsubService, RealtimeBroadcaster, RealtimeEvent, SdkCallCx, TriggerEvent,
|
||||
};
|
||||
|
||||
use crate::app_secrets_repo::AppSecretsRepo;
|
||||
use crate::authz::{self, AuthzRepo, Capability};
|
||||
use crate::pubsub_repo::{PublishCtx, PubsubRepo, PubsubRepoError};
|
||||
use crate::topic_repo::TopicRepo;
|
||||
|
||||
/// TTL bounds (seconds) for `pubsub::subscriber_token`. Env-overridable
|
||||
/// via `PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`.
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct SubscriberTokenConfig {
|
||||
pub min_ttl: i64,
|
||||
pub max_ttl: i64,
|
||||
pub default_ttl: i64,
|
||||
}
|
||||
|
||||
impl SubscriberTokenConfig {
|
||||
#[must_use]
|
||||
pub const fn conservative() -> Self {
|
||||
Self {
|
||||
min_ttl: 10,
|
||||
max_ttl: 86_400,
|
||||
default_ttl: 3_600,
|
||||
}
|
||||
}
|
||||
|
||||
/// Load from env, falling back to the conservative defaults for any
|
||||
/// missing / invalid value.
|
||||
#[must_use]
|
||||
pub fn from_env() -> Self {
|
||||
let mut c = Self::conservative();
|
||||
load_i64(&mut c.min_ttl, "PICLOUD_SUBSCRIBER_TOKEN_TTL_MIN_SEC");
|
||||
load_i64(&mut c.max_ttl, "PICLOUD_SUBSCRIBER_TOKEN_TTL_MAX_SEC");
|
||||
load_i64(
|
||||
&mut c.default_ttl,
|
||||
"PICLOUD_SUBSCRIBER_TOKEN_TTL_DEFAULT_SEC",
|
||||
);
|
||||
c
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for SubscriberTokenConfig {
|
||||
fn default() -> Self {
|
||||
Self::conservative()
|
||||
}
|
||||
}
|
||||
|
||||
fn load_i64(dst: &mut i64, key: &str) {
|
||||
if let Ok(v) = std::env::var(key) {
|
||||
match v.parse::<i64>() {
|
||||
Ok(n) if n > 0 => *dst = n,
|
||||
_ => tracing::warn!(env = key, value = %v, "ignoring invalid token-ttl value"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct PubsubServiceImpl {
|
||||
repo: Arc<dyn PubsubRepo>,
|
||||
authz: Arc<dyn AuthzRepo>,
|
||||
// Realtime extras (v1.1.6) — optional so the existing two-arg
|
||||
// constructor (and its unit tests) keep working without them. The
|
||||
// production binary attaches them via `with_realtime`.
|
||||
realtime: Option<Arc<dyn RealtimeBroadcaster>>,
|
||||
topics: Option<Arc<dyn TopicRepo>>,
|
||||
secrets: Option<Arc<dyn AppSecretsRepo>>,
|
||||
token_config: SubscriberTokenConfig,
|
||||
}
|
||||
|
||||
impl PubsubServiceImpl {
|
||||
#[must_use]
|
||||
pub fn new(repo: Arc<dyn PubsubRepo>, authz: Arc<dyn AuthzRepo>) -> Self {
|
||||
Self {
|
||||
repo,
|
||||
authz,
|
||||
realtime: None,
|
||||
topics: None,
|
||||
secrets: None,
|
||||
token_config: SubscriberTokenConfig::conservative(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Attach the v1.1.6 realtime surface: the in-process broadcaster
|
||||
/// (publish fan-out to SSE subscribers), the topic registry +
|
||||
/// app-secrets repo (subscriber-token minting), and the TTL config.
|
||||
#[must_use]
|
||||
pub fn with_realtime(
|
||||
mut self,
|
||||
broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
topics: Arc<dyn TopicRepo>,
|
||||
secrets: Arc<dyn AppSecretsRepo>,
|
||||
token_config: SubscriberTokenConfig,
|
||||
) -> Self {
|
||||
self.realtime = Some(broadcaster);
|
||||
self.topics = Some(topics);
|
||||
self.secrets = Some(secrets);
|
||||
self.token_config = token_config;
|
||||
self
|
||||
}
|
||||
|
||||
async fn check_publish(&self, cx: &SdkCallCx) -> Result<(), PubsubError> {
|
||||
if let Some(ref principal) = cx.principal {
|
||||
authz::require(
|
||||
&*self.authz,
|
||||
principal,
|
||||
Capability::AppPubsubPublish(cx.app_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| PubsubError::Forbidden)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl From<PubsubRepoError> for PubsubError {
|
||||
fn from(e: PubsubRepoError) -> Self {
|
||||
Self::Unavailable(e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubService for PubsubServiceImpl {
|
||||
async fn publish_durable(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
topic: &str,
|
||||
message: serde_json::Value,
|
||||
) -> Result<(), PubsubError> {
|
||||
if topic.trim().is_empty() {
|
||||
return Err(PubsubError::EmptyTopic);
|
||||
}
|
||||
self.check_publish(cx).await?;
|
||||
|
||||
// `published_at` is stamped once on the manager side so every
|
||||
// delivery path — durable triggers AND the realtime broadcast —
|
||||
// agrees on one instant. The message is cloned into the trigger
|
||||
// event so the realtime path can reuse the original.
|
||||
let published_at = chrono::Utc::now();
|
||||
let event = TriggerEvent::Pubsub {
|
||||
topic: topic.to_string(),
|
||||
message: message.clone(),
|
||||
published_at,
|
||||
};
|
||||
let payload = serde_json::to_value(&event)
|
||||
.map_err(|e| PubsubError::Rejected(format!("event serialize: {e}")))?;
|
||||
|
||||
let publish_ctx = PublishCtx {
|
||||
app_id: cx.app_id,
|
||||
origin_principal: cx.principal.as_ref().map(|p| p.user_id),
|
||||
trigger_depth: cx.trigger_depth,
|
||||
root_execution_id: cx.root_execution_id,
|
||||
};
|
||||
// Order (design notes §8): transactional outbox fan-out + commit
|
||||
// FIRST; only then the best-effort realtime broadcast. If the
|
||||
// fan-out fails, the publish throws and no broadcast happens. If
|
||||
// the broadcast fails after a committed fan-out, trigger
|
||||
// deliveries still happen and only SSE subscribers miss this
|
||||
// event (no replay in v1.1.6).
|
||||
//
|
||||
// No matching triggers → 0 rows written, publish still succeeds.
|
||||
self.repo
|
||||
.fan_out_publish(publish_ctx, topic, payload)
|
||||
.await?;
|
||||
|
||||
// Non-transactional, best-effort fan-out to in-process SSE
|
||||
// subscribers. Run on a child task so a panicking broadcaster
|
||||
// (or a future cluster-mode resolver fault) becomes a warn log,
|
||||
// never a failed publish — the durable deliveries already
|
||||
// committed above.
|
||||
if let Some(realtime) = self.realtime.clone() {
|
||||
let app_id = cx.app_id;
|
||||
let topic_owned = topic.to_string();
|
||||
let realtime_event = RealtimeEvent {
|
||||
topic: topic_owned.clone(),
|
||||
message,
|
||||
published_at,
|
||||
};
|
||||
let handle = tokio::spawn(async move {
|
||||
realtime.publish(app_id, &topic_owned, realtime_event).await;
|
||||
});
|
||||
if let Err(e) = handle.await {
|
||||
tracing::warn!(error = %e, "realtime broadcast failed; publish unaffected");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn mint_subscriber_token(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
topics: Vec<String>,
|
||||
ttl_seconds: Option<i64>,
|
||||
) -> Result<String, PubsubError> {
|
||||
// Anonymous (public-HTTP) scripts can't mint — that would bypass
|
||||
// the token-minting authz boundary.
|
||||
let Some(principal) = cx.principal.as_ref() else {
|
||||
return Err(PubsubError::SubscriberToken(
|
||||
"pubsub::subscriber_token: requires an authenticated principal \
|
||||
(a script on a public route cannot mint tokens)"
|
||||
.into(),
|
||||
));
|
||||
};
|
||||
// Minting reuses the existing pub/sub publish capability (no new
|
||||
// scope — the seven-scope commitment holds).
|
||||
authz::require(
|
||||
&*self.authz,
|
||||
principal,
|
||||
Capability::AppPubsubPublish(cx.app_id),
|
||||
)
|
||||
.await
|
||||
.map_err(|_| PubsubError::Forbidden)?;
|
||||
|
||||
let (Some(topic_repo), Some(secrets)) = (self.topics.as_ref(), self.secrets.as_ref())
|
||||
else {
|
||||
return Err(PubsubError::Unavailable(
|
||||
"subscriber tokens are not wired in".into(),
|
||||
));
|
||||
};
|
||||
|
||||
if topics.is_empty() {
|
||||
return Err(PubsubError::SubscriberToken(
|
||||
"pubsub::subscriber_token: topics list must not be empty".into(),
|
||||
));
|
||||
}
|
||||
|
||||
let ttl = ttl_seconds.unwrap_or(self.token_config.default_ttl);
|
||||
if ttl < self.token_config.min_ttl || ttl > self.token_config.max_ttl {
|
||||
return Err(PubsubError::SubscriberToken(format!(
|
||||
"pubsub::subscriber_token: ttl_seconds must be between {} and {}",
|
||||
self.token_config.min_ttl, self.token_config.max_ttl
|
||||
)));
|
||||
}
|
||||
|
||||
// Every requested topic must be registered as externally
|
||||
// subscribable in this app — fail fast rather than mint a token
|
||||
// that won't work.
|
||||
for name in &topics {
|
||||
let registered = topic_repo
|
||||
.get(cx.app_id, name)
|
||||
.await
|
||||
.map_err(|e| PubsubError::Unavailable(e.to_string()))?;
|
||||
if !registered.map(|t| t.external_subscribable).unwrap_or(false) {
|
||||
return Err(PubsubError::SubscriberToken(format!(
|
||||
"pubsub::subscriber_token: topic {name} is not externally subscribable"
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
let key = secrets
|
||||
.get_or_create_signing_key(cx.app_id)
|
||||
.await
|
||||
.map_err(|e| PubsubError::Unavailable(e.to_string()))?;
|
||||
let now = chrono::Utc::now().timestamp();
|
||||
let claims = TokenClaims {
|
||||
app_id: cx.app_id,
|
||||
topics,
|
||||
exp: now.saturating_add(ttl),
|
||||
iat: now,
|
||||
};
|
||||
Ok(subscriber_token::sign(&key, &claims))
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// Tests — in-memory PubsubRepo so unit tests don't need Postgres. The
|
||||
// real transactional fan-out is covered against a live DB by the
|
||||
// integration suite; the in-memory fake models the all-or-nothing
|
||||
// commit so the rollback semantics can be asserted without a DB.
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::authz::{AuthzError, AuthzRepo};
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{
|
||||
topic_matches, AdminUserId, AppId, AppRole, ExecutionId, InstanceRole, Principal,
|
||||
RequestId, ScriptId, UserId,
|
||||
};
|
||||
use std::sync::Mutex;
|
||||
|
||||
/// In-memory pubsub repo. Holds a set of `(app, pattern)`
|
||||
/// subscriptions and records the outbox rows a publish would write.
|
||||
/// `fail_at` simulates a mid-fan-out INSERT failure: when set to
|
||||
/// `Some(n)`, the n-th (1-indexed) matching row errors and NOTHING
|
||||
/// is recorded — modelling the single-transaction rollback.
|
||||
struct InMemoryPubsubRepo {
|
||||
subs: Vec<(AppId, String)>,
|
||||
written: Mutex<Vec<(AppId, String)>>,
|
||||
fail_at: Option<usize>,
|
||||
}
|
||||
|
||||
impl InMemoryPubsubRepo {
|
||||
fn new(subs: Vec<(AppId, String)>) -> Self {
|
||||
Self {
|
||||
subs,
|
||||
written: Mutex::new(Vec::new()),
|
||||
fail_at: None,
|
||||
}
|
||||
}
|
||||
fn written_count(&self) -> usize {
|
||||
self.written.lock().unwrap().len()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubRepo for InMemoryPubsubRepo {
|
||||
async fn fan_out_publish(
|
||||
&self,
|
||||
ctx: PublishCtx,
|
||||
topic: &str,
|
||||
_event_payload: serde_json::Value,
|
||||
) -> Result<u32, PubsubRepoError> {
|
||||
let matches: Vec<&(AppId, String)> = self
|
||||
.subs
|
||||
.iter()
|
||||
.filter(|(a, pat)| *a == ctx.app_id && topic_matches(pat, topic))
|
||||
.collect();
|
||||
let mut staged = Vec::new();
|
||||
for (i, _) in matches.iter().enumerate() {
|
||||
if self.fail_at == Some(i + 1) {
|
||||
// Rollback: nothing was committed.
|
||||
return Err(PubsubRepoError::Db(sqlx::Error::Protocol(
|
||||
"simulated insert failure".into(),
|
||||
)));
|
||||
}
|
||||
staged.push((ctx.app_id, topic.to_string()));
|
||||
}
|
||||
let n = staged.len();
|
||||
self.written.lock().unwrap().extend(staged);
|
||||
Ok(u32::try_from(n).unwrap_or(u32::MAX))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct DenyingAuthzRepo;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for DenyingAuthzRepo {
|
||||
async fn membership(
|
||||
&self,
|
||||
_user_id: UserId,
|
||||
_app_id: AppId,
|
||||
) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct EditorAuthzRepo;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for EditorAuthzRepo {
|
||||
async fn membership(
|
||||
&self,
|
||||
_user_id: UserId,
|
||||
_app_id: AppId,
|
||||
) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(Some(AppRole::Editor))
|
||||
}
|
||||
}
|
||||
|
||||
fn anon_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
app_id,
|
||||
script_id: ScriptId::new(),
|
||||
principal: None,
|
||||
execution_id: ExecutionId::new(),
|
||||
request_id: RequestId::new(),
|
||||
trigger_depth: 0,
|
||||
root_execution_id: ExecutionId::new(),
|
||||
is_dead_letter_handler: false,
|
||||
event: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn member_cx(app_id: AppId) -> SdkCallCx {
|
||||
SdkCallCx {
|
||||
principal: Some(Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
scopes: None,
|
||||
app_binding: None,
|
||||
}),
|
||||
..anon_cx(app_id)
|
||||
}
|
||||
}
|
||||
|
||||
fn svc(repo: Arc<dyn PubsubRepo>, authz: Arc<dyn AuthzRepo>) -> PubsubServiceImpl {
|
||||
PubsubServiceImpl::new(repo, authz)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn publish_writes_one_row_per_matching_trigger() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![
|
||||
(app, "user.*".into()),
|
||||
(app, "user.created".into()),
|
||||
(app, "order.*".into()), // does not match
|
||||
]));
|
||||
let files = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
|
||||
files
|
||||
.publish_durable(&anon_cx(app), "user.created", serde_json::json!({"id": 1}))
|
||||
.await
|
||||
.unwrap();
|
||||
// Two of the three subscriptions match "user.created".
|
||||
assert_eq!(repo.written_count(), 2);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_matching_trigger_succeeds_silently() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![(app, "order.*".into())]));
|
||||
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
|
||||
svc.publish_durable(&anon_cx(app), "user.created", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(repo.written_count(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_topic_rejected() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
|
||||
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
|
||||
let err = svc
|
||||
.publish_durable(&anon_cx(app), " ", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, PubsubError::EmptyTopic));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cross_app_isolation() {
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
// The only subscription belongs to app B.
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![(app_b, "*".into())]));
|
||||
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
|
||||
// App A publishes — app B's trigger must NOT fire.
|
||||
svc.publish_durable(&anon_cx(app_a), "user.created", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(repo.written_count(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn fan_out_is_transactional_all_or_nothing() {
|
||||
let app = AppId::new();
|
||||
let mut repo = InMemoryPubsubRepo::new(vec![
|
||||
(app, "*".into()),
|
||||
(app, "user.*".into()),
|
||||
(app, "user.created".into()),
|
||||
]);
|
||||
repo.fail_at = Some(3); // fail on the 3rd matching insert
|
||||
let repo = Arc::new(repo);
|
||||
let svc = svc(repo.clone(), Arc::new(DenyingAuthzRepo));
|
||||
let err = svc
|
||||
.publish_durable(&anon_cx(app), "user.created", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, PubsubError::Unavailable(_)));
|
||||
// Rollback: no partial fan-out survived.
|
||||
assert_eq!(repo.written_count(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn anonymous_cx_skips_authz() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
|
||||
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
|
||||
// No principal → no authz check even with a denying repo.
|
||||
svc.publish_durable(&anon_cx(app), "t", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn member_without_role_is_forbidden() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
|
||||
let svc = svc(repo, Arc::new(DenyingAuthzRepo));
|
||||
let err = svc
|
||||
.publish_durable(&member_cx(app), "t", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, PubsubError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn member_with_editor_role_allowed() {
|
||||
let app = AppId::new();
|
||||
let repo = Arc::new(InMemoryPubsubRepo::new(vec![]));
|
||||
let svc = svc(repo, Arc::new(EditorAuthzRepo));
|
||||
svc.publish_durable(&member_cx(app), "t", serde_json::json!(1))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------
|
||||
// v1.1.6 realtime broadcast + subscriber-token minting
|
||||
// ------------------------------------------------------------------
|
||||
|
||||
use crate::app_secrets_repo::{AppSecretsRepo, AppSecretsRepoError};
|
||||
use crate::topic_repo::{Topic, TopicAuthMode, TopicRepo, TopicRepoError};
|
||||
use picloud_orchestrator_core::InProcessBroadcaster;
|
||||
use picloud_shared::{RealtimeBroadcaster, RealtimeEvent};
|
||||
|
||||
/// Topic repo fake: returns the configured topics as registered +
|
||||
/// externally-subscribable (unless absent).
|
||||
struct FakeTopicRepo(Vec<String>);
|
||||
#[async_trait]
|
||||
impl TopicRepo for FakeTopicRepo {
|
||||
async fn create(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: bool,
|
||||
_: TopicAuthMode,
|
||||
) -> Result<Topic, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn list(&self, _: AppId) -> Result<Vec<Topic>, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn get(&self, _: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
|
||||
Ok(self.0.iter().any(|t| t == name).then(|| Topic {
|
||||
name: name.to_string(),
|
||||
external_subscribable: true,
|
||||
auth_mode: TopicAuthMode::Token,
|
||||
created_at: chrono::Utc::now(),
|
||||
updated_at: chrono::Utc::now(),
|
||||
}))
|
||||
}
|
||||
async fn update(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: Option<bool>,
|
||||
_: Option<TopicAuthMode>,
|
||||
) -> Result<Option<Topic>, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn delete(&self, _: AppId, _: &str) -> Result<bool, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct FakeSecrets;
|
||||
#[async_trait]
|
||||
impl AppSecretsRepo for FakeSecrets {
|
||||
async fn get_or_create_signing_key(
|
||||
&self,
|
||||
_: AppId,
|
||||
) -> Result<Vec<u8>, AppSecretsRepoError> {
|
||||
Ok(vec![42u8; 32])
|
||||
}
|
||||
async fn signing_key(&self, _: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
|
||||
Ok(Some(vec![42u8; 32]))
|
||||
}
|
||||
}
|
||||
|
||||
/// Broadcaster that panics on publish — proves a broadcast fault
|
||||
/// can't fail the publish.
|
||||
struct PanicBroadcaster;
|
||||
#[async_trait]
|
||||
impl RealtimeBroadcaster for PanicBroadcaster {
|
||||
async fn subscribe(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
) -> Result<tokio::sync::broadcast::Receiver<RealtimeEvent>, picloud_shared::BroadcasterError>
|
||||
{
|
||||
unimplemented!()
|
||||
}
|
||||
async fn publish(&self, _: AppId, _: &str, _: RealtimeEvent) {
|
||||
panic!("boom");
|
||||
}
|
||||
async fn drop_topic(&self, _: AppId, _: &str) {}
|
||||
}
|
||||
|
||||
fn realtime_svc(
|
||||
repo: Arc<dyn PubsubRepo>,
|
||||
broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
topics: Vec<String>,
|
||||
) -> PubsubServiceImpl {
|
||||
PubsubServiceImpl::new(repo, Arc::new(EditorAuthzRepo)).with_realtime(
|
||||
broadcaster,
|
||||
Arc::new(FakeTopicRepo(topics)),
|
||||
Arc::new(FakeSecrets),
|
||||
SubscriberTokenConfig::conservative(),
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn publish_broadcasts_to_in_process_subscribers() {
|
||||
let app = AppId::new();
|
||||
let broadcaster = Arc::new(InProcessBroadcaster::new(16));
|
||||
let mut rx = broadcaster.subscribe(app, "chat").await.unwrap();
|
||||
let svc = realtime_svc(
|
||||
Arc::new(InMemoryPubsubRepo::new(vec![])),
|
||||
broadcaster,
|
||||
vec![],
|
||||
);
|
||||
svc.publish_durable(&anon_cx(app), "chat", serde_json::json!({ "hi": 1 }))
|
||||
.await
|
||||
.unwrap();
|
||||
let ev = rx.recv().await.unwrap();
|
||||
assert_eq!(ev.topic, "chat");
|
||||
assert_eq!(ev.message, serde_json::json!({ "hi": 1 }));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn panicking_broadcaster_does_not_fail_publish() {
|
||||
let app = AppId::new();
|
||||
let svc = realtime_svc(
|
||||
Arc::new(InMemoryPubsubRepo::new(vec![])),
|
||||
Arc::new(PanicBroadcaster),
|
||||
vec![],
|
||||
);
|
||||
// The outbox fan-out committed; the broadcast panic is swallowed.
|
||||
svc.publish_durable(&anon_cx(app), "chat", serde_json::json!(1))
|
||||
.await
|
||||
.expect("publish must succeed despite broadcast panic");
|
||||
}
|
||||
|
||||
fn mint_svc(topics: Vec<String>) -> PubsubServiceImpl {
|
||||
realtime_svc(
|
||||
Arc::new(InMemoryPubsubRepo::new(vec![])),
|
||||
Arc::new(picloud_shared::NoopRealtimeBroadcaster),
|
||||
topics,
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mint_returns_token_scoped_to_topics() {
|
||||
let app = AppId::new();
|
||||
let svc = mint_svc(vec!["chat".into(), "notify".into()]);
|
||||
let token = svc
|
||||
.mint_subscriber_token(
|
||||
&member_cx(app),
|
||||
vec!["chat".into(), "notify".into()],
|
||||
Some(120),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
// Verify with the fake key; claims carry the topics + expiry.
|
||||
let claims = subscriber_token::verify(&[42u8; 32], &token, chrono::Utc::now().timestamp())
|
||||
.expect("token verifies");
|
||||
assert_eq!(claims.app_id, app);
|
||||
assert!(claims.allows_topic("chat") && claims.allows_topic("notify"));
|
||||
assert!(claims.exp > claims.iat);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mint_anonymous_principal_throws() {
|
||||
let app = AppId::new();
|
||||
let svc = mint_svc(vec!["chat".into()]);
|
||||
let err = svc
|
||||
.mint_subscriber_token(&anon_cx(app), vec!["chat".into()], None)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, PubsubError::SubscriberToken(_)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mint_empty_topics_throws() {
|
||||
let app = AppId::new();
|
||||
let svc = mint_svc(vec!["chat".into()]);
|
||||
let err = svc
|
||||
.mint_subscriber_token(&member_cx(app), vec![], None)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, PubsubError::SubscriberToken(_)));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mint_ttl_below_min_and_above_max_throw() {
|
||||
let app = AppId::new();
|
||||
let svc = mint_svc(vec!["chat".into()]);
|
||||
for bad in [Some(5), Some(90_000)] {
|
||||
let err = svc
|
||||
.mint_subscriber_token(&member_cx(app), vec!["chat".into()], bad)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
matches!(err, PubsubError::SubscriberToken(_)),
|
||||
"ttl {bad:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mint_unregistered_topic_throws_with_message() {
|
||||
let app = AppId::new();
|
||||
// "chat" registered; "secret" is not.
|
||||
let svc = mint_svc(vec!["chat".into()]);
|
||||
let err = svc
|
||||
.mint_subscriber_token(&member_cx(app), vec!["chat".into(), "secret".into()], None)
|
||||
.await
|
||||
.unwrap_err();
|
||||
match err {
|
||||
PubsubError::SubscriberToken(msg) => {
|
||||
assert!(
|
||||
msg.contains("topic secret is not externally subscribable"),
|
||||
"got: {msg}"
|
||||
);
|
||||
}
|
||||
other => panic!("expected SubscriberToken, got {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
338
crates/manager-core/src/realtime_authority.rs
Normal file
338
crates/manager-core/src/realtime_authority.rs
Normal file
@@ -0,0 +1,338 @@
|
||||
//! `RealtimeAuthorityImpl` — the DB-backed SSE subscribe gate (v1.1.6).
|
||||
//!
|
||||
//! Backs the [`picloud_shared::RealtimeAuthority`] trait the SSE handler
|
||||
//! in orchestrator-core calls. All `topics`-table reads and signing-key
|
||||
//! material stay inside this impl so the data-plane crate never touches
|
||||
//! the key.
|
||||
//!
|
||||
//! Verdict mapping (see [`SubscribeDenied`]):
|
||||
//! * topic missing OR not externally subscribable → `NotFound` (404).
|
||||
//! Both collapse to 404 so the endpoint can't probe internal topics.
|
||||
//! * `auth_mode = 'public'` → allow.
|
||||
//! * `auth_mode = 'token'` → verify the HMAC token (present, signed by
|
||||
//! this app's key, unexpired, scoped to this topic) → allow, else
|
||||
//! `Unauthorized` (401, generic — never says which check failed).
|
||||
//!
|
||||
//! Signing keys never change in v1.1.6 (no rotation API), so a small
|
||||
//! in-memory cache avoids a per-subscribe DB read once an app's key has
|
||||
//! been seen. The cache is purely an optimization — a cold miss reads
|
||||
//! the row.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{subscriber_token, AppId, RealtimeAuthority, SubscribeDenied};
|
||||
|
||||
use crate::app_secrets_repo::AppSecretsRepo;
|
||||
use crate::topic_repo::{TopicAuthMode, TopicRepo};
|
||||
|
||||
pub struct RealtimeAuthorityImpl {
|
||||
topics: Arc<dyn TopicRepo>,
|
||||
secrets: Arc<dyn AppSecretsRepo>,
|
||||
key_cache: Mutex<HashMap<AppId, Vec<u8>>>,
|
||||
}
|
||||
|
||||
impl RealtimeAuthorityImpl {
|
||||
#[must_use]
|
||||
pub fn new(topics: Arc<dyn TopicRepo>, secrets: Arc<dyn AppSecretsRepo>) -> Self {
|
||||
Self {
|
||||
topics,
|
||||
secrets,
|
||||
key_cache: Mutex::new(HashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetch the app's signing key, consulting the cache first. Returns
|
||||
/// `None` when the app has no key (no token ever minted) — which the
|
||||
/// caller maps to `Unauthorized`.
|
||||
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, SubscribeDenied> {
|
||||
if let Ok(cache) = self.key_cache.lock() {
|
||||
if let Some(k) = cache.get(&app_id) {
|
||||
return Ok(Some(k.clone()));
|
||||
}
|
||||
}
|
||||
let key = self
|
||||
.secrets
|
||||
.signing_key(app_id)
|
||||
.await
|
||||
.map_err(|e| SubscribeDenied::Backend(e.to_string()))?;
|
||||
if let Some(ref k) = key {
|
||||
if let Ok(mut cache) = self.key_cache.lock() {
|
||||
cache.insert(app_id, k.clone());
|
||||
}
|
||||
}
|
||||
Ok(key)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl RealtimeAuthority for RealtimeAuthorityImpl {
|
||||
async fn authorize_subscribe(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
topic: &str,
|
||||
token: Option<&str>,
|
||||
) -> Result<(), SubscribeDenied> {
|
||||
let registered = self
|
||||
.topics
|
||||
.get(app_id, topic)
|
||||
.await
|
||||
.map_err(|e| SubscribeDenied::Backend(e.to_string()))?;
|
||||
|
||||
// Missing topic AND internal-only topic both 404 — don't leak
|
||||
// which internal topics exist.
|
||||
let Some(t) = registered.filter(|t| t.external_subscribable) else {
|
||||
return Err(SubscribeDenied::NotFound);
|
||||
};
|
||||
|
||||
match t.auth_mode {
|
||||
TopicAuthMode::Public => Ok(()),
|
||||
TopicAuthMode::Token => {
|
||||
let token = token.ok_or(SubscribeDenied::Unauthorized)?;
|
||||
let key = self
|
||||
.signing_key(app_id)
|
||||
.await?
|
||||
.ok_or(SubscribeDenied::Unauthorized)?;
|
||||
let now = chrono::Utc::now().timestamp();
|
||||
let claims = subscriber_token::verify(&key, token, now)
|
||||
.map_err(|_| SubscribeDenied::Unauthorized)?;
|
||||
// Per-app key already makes a cross-app token fail the
|
||||
// signature check; this is belt-and-suspenders.
|
||||
if claims.app_id != app_id || !claims.allows_topic(topic) {
|
||||
return Err(SubscribeDenied::Unauthorized);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::app_secrets_repo::AppSecretsRepoError;
|
||||
use crate::topic_repo::{Topic, TopicRepoError};
|
||||
use chrono::Utc;
|
||||
use picloud_shared::subscriber_token::{sign, TokenClaims};
|
||||
|
||||
struct FakeTopics(Vec<(AppId, Topic)>);
|
||||
#[async_trait]
|
||||
impl TopicRepo for FakeTopics {
|
||||
async fn create(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: bool,
|
||||
_: TopicAuthMode,
|
||||
) -> Result<Topic, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn list(&self, _: AppId) -> Result<Vec<Topic>, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
|
||||
Ok(self
|
||||
.0
|
||||
.iter()
|
||||
.find(|(a, t)| *a == app_id && t.name == name)
|
||||
.map(|(_, t)| t.clone()))
|
||||
}
|
||||
async fn update(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: Option<bool>,
|
||||
_: Option<TopicAuthMode>,
|
||||
) -> Result<Option<Topic>, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn delete(&self, _: AppId, _: &str) -> Result<bool, TopicRepoError> {
|
||||
unimplemented!()
|
||||
}
|
||||
}
|
||||
|
||||
struct FakeSecrets(AppId, Vec<u8>);
|
||||
#[async_trait]
|
||||
impl AppSecretsRepo for FakeSecrets {
|
||||
async fn get_or_create_signing_key(
|
||||
&self,
|
||||
_: AppId,
|
||||
) -> Result<Vec<u8>, AppSecretsRepoError> {
|
||||
Ok(self.1.clone())
|
||||
}
|
||||
async fn signing_key(&self, app_id: AppId) -> Result<Option<Vec<u8>>, AppSecretsRepoError> {
|
||||
Ok((app_id == self.0).then(|| self.1.clone()))
|
||||
}
|
||||
}
|
||||
|
||||
fn topic(name: &str, external: bool, mode: TopicAuthMode) -> Topic {
|
||||
Topic {
|
||||
name: name.to_string(),
|
||||
external_subscribable: external,
|
||||
auth_mode: mode,
|
||||
created_at: Utc::now(),
|
||||
updated_at: Utc::now(),
|
||||
}
|
||||
}
|
||||
|
||||
fn authority(
|
||||
topics: Vec<(AppId, Topic)>,
|
||||
key_app: AppId,
|
||||
key: Vec<u8>,
|
||||
) -> RealtimeAuthorityImpl {
|
||||
RealtimeAuthorityImpl::new(
|
||||
Arc::new(FakeTopics(topics)),
|
||||
Arc::new(FakeSecrets(key_app, key)),
|
||||
)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn missing_topic_is_not_found() {
|
||||
let app = AppId::new();
|
||||
let a = authority(vec![], app, vec![0u8; 32]);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app, "ghost", None).await,
|
||||
Err(SubscribeDenied::NotFound)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn internal_only_topic_is_not_found() {
|
||||
let app = AppId::new();
|
||||
let a = authority(
|
||||
vec![(app, topic("internal", false, TopicAuthMode::Public))],
|
||||
app,
|
||||
vec![0u8; 32],
|
||||
);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app, "internal", None).await,
|
||||
Err(SubscribeDenied::NotFound)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn public_topic_allows_without_token() {
|
||||
let app = AppId::new();
|
||||
let a = authority(
|
||||
vec![(app, topic("news", true, TopicAuthMode::Public))],
|
||||
app,
|
||||
vec![0u8; 32],
|
||||
);
|
||||
assert!(a.authorize_subscribe(app, "news", None).await.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn token_topic_without_token_is_unauthorized() {
|
||||
let app = AppId::new();
|
||||
let a = authority(
|
||||
vec![(app, topic("chat", true, TopicAuthMode::Token))],
|
||||
app,
|
||||
vec![7u8; 32],
|
||||
);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app, "chat", None).await,
|
||||
Err(SubscribeDenied::Unauthorized)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn token_topic_with_valid_token_allows() {
|
||||
let app = AppId::new();
|
||||
let key = vec![9u8; 32];
|
||||
let a = authority(
|
||||
vec![(app, topic("chat", true, TopicAuthMode::Token))],
|
||||
app,
|
||||
key.clone(),
|
||||
);
|
||||
let token = sign(
|
||||
&key,
|
||||
&TokenClaims {
|
||||
app_id: app,
|
||||
topics: vec!["chat".into()],
|
||||
iat: Utc::now().timestamp(),
|
||||
exp: Utc::now().timestamp() + 60,
|
||||
},
|
||||
);
|
||||
assert!(a
|
||||
.authorize_subscribe(app, "chat", Some(&token))
|
||||
.await
|
||||
.is_ok());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn token_for_other_topic_is_unauthorized() {
|
||||
let app = AppId::new();
|
||||
let key = vec![9u8; 32];
|
||||
let a = authority(
|
||||
vec![(app, topic("chat", true, TopicAuthMode::Token))],
|
||||
app,
|
||||
key.clone(),
|
||||
);
|
||||
let token = sign(
|
||||
&key,
|
||||
&TokenClaims {
|
||||
app_id: app,
|
||||
topics: vec!["other".into()],
|
||||
iat: Utc::now().timestamp(),
|
||||
exp: Utc::now().timestamp() + 60,
|
||||
},
|
||||
);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app, "chat", Some(&token)).await,
|
||||
Err(SubscribeDenied::Unauthorized)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn expired_token_is_unauthorized() {
|
||||
let app = AppId::new();
|
||||
let key = vec![9u8; 32];
|
||||
let a = authority(
|
||||
vec![(app, topic("chat", true, TopicAuthMode::Token))],
|
||||
app,
|
||||
key.clone(),
|
||||
);
|
||||
let token = sign(
|
||||
&key,
|
||||
&TokenClaims {
|
||||
app_id: app,
|
||||
topics: vec!["chat".into()],
|
||||
iat: Utc::now().timestamp() - 120,
|
||||
exp: Utc::now().timestamp() - 60,
|
||||
},
|
||||
);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app, "chat", Some(&token)).await,
|
||||
Err(SubscribeDenied::Unauthorized)
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn token_signed_by_other_app_key_is_unauthorized() {
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let key_a = vec![1u8; 32];
|
||||
let key_b = vec![2u8; 32];
|
||||
// Authority for app B; its key is key_b.
|
||||
let a = authority(
|
||||
vec![(app_b, topic("chat", true, TopicAuthMode::Token))],
|
||||
app_b,
|
||||
key_b,
|
||||
);
|
||||
// Token signed by app A's key, claiming app A.
|
||||
let token = sign(
|
||||
&key_a,
|
||||
&TokenClaims {
|
||||
app_id: app_a,
|
||||
topics: vec!["chat".into()],
|
||||
iat: Utc::now().timestamp(),
|
||||
exp: Utc::now().timestamp() + 60,
|
||||
},
|
||||
);
|
||||
assert_eq!(
|
||||
a.authorize_subscribe(app_b, "chat", Some(&token)).await,
|
||||
Err(SubscribeDenied::Unauthorized)
|
||||
);
|
||||
}
|
||||
}
|
||||
457
crates/manager-core/src/ssrf.rs
Normal file
457
crates/manager-core/src/ssrf.rs
Normal file
@@ -0,0 +1,457 @@
|
||||
//! SSRF deny-list — the load-bearing security mechanism behind the
|
||||
//! v1.1.4 `http::*` SDK.
|
||||
//!
|
||||
//! The policy is applied to the **resolved IP address**, not the
|
||||
//! hostname. That is the DNS-rebinding defense: a hostname that
|
||||
//! resolves to a public IP at lookup time and a private IP at connect
|
||||
//! time is not exploitable, because reqwest re-runs every connection
|
||||
//! (including post-redirect hops) through [`SsrfResolver`], which
|
||||
//! filters the address list before the socket is opened.
|
||||
//!
|
||||
//! [`SsrfPolicy::check`] returns a CIDR-*category* reason on denial
|
||||
//! (e.g. `"loopback"`, `"private"`) — never the IP itself, so the
|
||||
//! script-visible error can't be used to map the internal network.
|
||||
//!
|
||||
//! `PICLOUD_HTTP_ALLOW_PRIVATE=true` flips `allow_private`, which
|
||||
//! short-circuits every check to allow. That is dev/test-only and the
|
||||
//! binary logs a startup warning when it's set.
|
||||
|
||||
use std::future::Future;
|
||||
use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr};
|
||||
use std::pin::Pin;
|
||||
use std::sync::Arc;
|
||||
|
||||
use reqwest::dns::{Addrs, Name, Resolve, Resolving};
|
||||
|
||||
/// Decision policy for a single resolved IP. Cheap to clone (one bool).
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct SsrfPolicy {
|
||||
/// When true, every address is allowed — the entire deny-list is
|
||||
/// disabled. Set from `PICLOUD_HTTP_ALLOW_PRIVATE`. Dev/test only.
|
||||
pub allow_private: bool,
|
||||
}
|
||||
|
||||
impl SsrfPolicy {
|
||||
#[must_use]
|
||||
pub const fn new(allow_private: bool) -> Self {
|
||||
Self { allow_private }
|
||||
}
|
||||
|
||||
/// `Ok(())` if the IP may be connected to; `Err(reason)` with a
|
||||
/// CIDR-category label otherwise. The reason is safe to surface to
|
||||
/// a script — it never contains the address.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns the deny reason when `ip` falls in a blocked range and
|
||||
/// `allow_private` is false.
|
||||
pub fn check(&self, ip: IpAddr) -> Result<(), &'static str> {
|
||||
if self.allow_private {
|
||||
return Ok(());
|
||||
}
|
||||
match ip {
|
||||
IpAddr::V4(v4) => check_v4(v4),
|
||||
IpAddr::V6(v6) => check_v6(v6),
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn is_allowed(&self, ip: IpAddr) -> bool {
|
||||
self.check(ip).is_ok()
|
||||
}
|
||||
}
|
||||
|
||||
/// IPv4 deny-list. Order doesn't matter (ranges are disjoint by
|
||||
/// construction); first match wins for the reason label.
|
||||
// Several arms share a reason ("private") for distinct CIDRs — keeping
|
||||
// them separate documents each blocked range explicitly.
|
||||
#[allow(clippy::match_same_arms)]
|
||||
fn check_v4(ip: Ipv4Addr) -> Result<(), &'static str> {
|
||||
let o = ip.octets();
|
||||
match o {
|
||||
[127, ..] => Err("loopback"),
|
||||
[0, ..] => Err("unspecified"), // 0.0.0.0/8 "this network"
|
||||
[10, ..] => Err("private"),
|
||||
[172, b, ..] if (16..=31).contains(&b) => Err("private"),
|
||||
[192, 168, ..] => Err("private"),
|
||||
[169, 254, ..] => Err("link-local"), // includes cloud metadata 169.254.169.254
|
||||
[100, b, ..] if (64..=127).contains(&b) => Err("carrier-grade-nat"),
|
||||
[224..=239, ..] => Err("multicast"),
|
||||
[240..=255, ..] => Err("reserved"),
|
||||
_ => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
/// IPv6 deny-list. IPv4-mapped addresses (`::ffff:0:0/96`) re-run the
|
||||
/// v4 deny-list against the embedded address.
|
||||
fn check_v6(ip: Ipv6Addr) -> Result<(), &'static str> {
|
||||
// IPv4-mapped (::ffff:a.b.c.d) — re-check the embedded v4 address
|
||||
// so a mapped private/loopback address can't sneak through.
|
||||
if let Some(v4) = ip.to_ipv4_mapped() {
|
||||
return check_v4(v4);
|
||||
}
|
||||
if ip == Ipv6Addr::LOCALHOST {
|
||||
return Err("loopback");
|
||||
}
|
||||
if ip == Ipv6Addr::UNSPECIFIED {
|
||||
return Err("unspecified");
|
||||
}
|
||||
let seg0 = ip.segments()[0];
|
||||
if seg0 & 0xffc0 == 0xfe80 {
|
||||
return Err("link-local"); // fe80::/10
|
||||
}
|
||||
if seg0 & 0xfe00 == 0xfc00 {
|
||||
return Err("unique-local"); // fc00::/7
|
||||
}
|
||||
if seg0 & 0xff00 == 0xff00 {
|
||||
return Err("multicast"); // ff00::/8
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Marker error returned by the resolver when *every* resolved address
|
||||
/// for a host was denied. reqwest wraps this into a connect error; the
|
||||
/// `http_service` impl walks the source chain for the
|
||||
/// `"blocked by SSRF policy:"` prefix to surface a clean
|
||||
/// [`crate::http_service::HttpError::Ssrf`] instead of a generic DNS
|
||||
/// failure. Keeping the reason a category label means no IP leaks.
|
||||
#[derive(Debug)]
|
||||
struct SsrfBlocked {
|
||||
reason: &'static str,
|
||||
}
|
||||
|
||||
impl std::fmt::Display for SsrfBlocked {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
write!(f, "blocked by SSRF policy: {}", self.reason)
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for SsrfBlocked {}
|
||||
|
||||
/// Prefix the resolver embeds in its error and the impl scans for.
|
||||
pub const SSRF_BLOCK_PREFIX: &str = "blocked by SSRF policy: ";
|
||||
|
||||
/// Pluggable host→addresses lookup. Production uses the system
|
||||
/// resolver; tests inject a closure (e.g. to simulate DNS rebinding —
|
||||
/// a different address on a later call).
|
||||
pub type LookupFn = Arc<
|
||||
dyn Fn(String) -> Pin<Box<dyn Future<Output = std::io::Result<Vec<SocketAddr>>> + Send>>
|
||||
+ Send
|
||||
+ Sync,
|
||||
>;
|
||||
|
||||
fn system_lookup(
|
||||
host: String,
|
||||
) -> Pin<Box<dyn Future<Output = std::io::Result<Vec<SocketAddr>>> + Send>> {
|
||||
Box::pin(async move {
|
||||
// Port 0 — reqwest overrides it with the real target port.
|
||||
Ok(tokio::net::lookup_host((host.as_str(), 0u16))
|
||||
.await?
|
||||
.collect())
|
||||
})
|
||||
}
|
||||
|
||||
/// reqwest DNS resolver that delegates to the system resolver, then
|
||||
/// filters the address list through [`SsrfPolicy`]. Plugged in via
|
||||
/// `ClientBuilder::dns_resolver`, so it runs at the actual connection
|
||||
/// point — including for every redirect hop. This is the DNS-rebinding
|
||||
/// defense: filtering happens at connect time, not at URL-parse time.
|
||||
#[derive(Clone)]
|
||||
pub struct SsrfResolver {
|
||||
policy: SsrfPolicy,
|
||||
lookup: LookupFn,
|
||||
}
|
||||
|
||||
impl SsrfResolver {
|
||||
#[must_use]
|
||||
pub fn new(policy: SsrfPolicy) -> Self {
|
||||
Self {
|
||||
policy,
|
||||
lookup: Arc::new(system_lookup),
|
||||
}
|
||||
}
|
||||
|
||||
/// Construct with an injected lookup (tests only).
|
||||
#[must_use]
|
||||
pub fn with_lookup(policy: SsrfPolicy, lookup: LookupFn) -> Self {
|
||||
Self { policy, lookup }
|
||||
}
|
||||
}
|
||||
|
||||
impl Resolve for SsrfResolver {
|
||||
fn resolve(&self, name: Name) -> Resolving {
|
||||
let policy = self.policy;
|
||||
let lookup = self.lookup.clone();
|
||||
let host = name.as_str().to_string();
|
||||
Box::pin(async move {
|
||||
let resolved: Vec<SocketAddr> = lookup(host)
|
||||
.await
|
||||
.map_err(|e| -> Box<dyn std::error::Error + Send + Sync> { Box::new(e) })?;
|
||||
|
||||
// Empty resolution → genuine DNS miss; let reqwest surface
|
||||
// it as a normal "no addresses" error.
|
||||
if resolved.is_empty() {
|
||||
let addrs: Addrs = Box::new(std::iter::empty());
|
||||
return Ok(addrs);
|
||||
}
|
||||
|
||||
let mut allowed: Vec<SocketAddr> = Vec::with_capacity(resolved.len());
|
||||
let mut last_reason: &'static str = "denied";
|
||||
for sa in resolved {
|
||||
match policy.check(sa.ip()) {
|
||||
Ok(()) => allowed.push(sa),
|
||||
Err(reason) => last_reason = reason,
|
||||
}
|
||||
}
|
||||
|
||||
// Resolution returned addresses but the policy denied them
|
||||
// all → fail with the SSRF marker so the impl can report a
|
||||
// policy block (not a generic DNS error).
|
||||
if allowed.is_empty() {
|
||||
let err: Box<dyn std::error::Error + Send + Sync> = Box::new(SsrfBlocked {
|
||||
reason: last_reason,
|
||||
});
|
||||
return Err(err);
|
||||
}
|
||||
|
||||
let addrs: Addrs = Box::new(allowed.into_iter());
|
||||
Ok(addrs)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the resolver. reqwest's `dns_resolver` is generic over a
|
||||
/// concrete `R: Resolve` (it stores `Arc<R>`), so this returns the
|
||||
/// concrete `Arc<SsrfResolver>` rather than a trait object.
|
||||
#[must_use]
|
||||
pub fn resolver(policy: SsrfPolicy) -> Arc<SsrfResolver> {
|
||||
Arc::new(SsrfResolver::new(policy))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use std::str::FromStr;
|
||||
|
||||
fn denied(ip: &str) -> &'static str {
|
||||
let policy = SsrfPolicy::new(false);
|
||||
policy
|
||||
.check(IpAddr::from_str(ip).unwrap())
|
||||
.expect_err(&format!("{ip} should be denied"))
|
||||
}
|
||||
|
||||
fn allowed(ip: &str) {
|
||||
let policy = SsrfPolicy::new(false);
|
||||
policy
|
||||
.check(IpAddr::from_str(ip).unwrap())
|
||||
.unwrap_or_else(|r| panic!("{ip} should be allowed, denied as {r}"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv4_loopback() {
|
||||
assert_eq!(denied("127.0.0.1"), "loopback");
|
||||
assert_eq!(denied("127.1.2.3"), "loopback");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv4_unspecified() {
|
||||
assert_eq!(denied("0.0.0.0"), "unspecified");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_rfc1918_private() {
|
||||
assert_eq!(denied("10.0.0.1"), "private");
|
||||
assert_eq!(denied("10.255.255.255"), "private");
|
||||
assert_eq!(denied("172.16.0.1"), "private");
|
||||
assert_eq!(denied("172.31.255.255"), "private");
|
||||
assert_eq!(denied("192.168.0.1"), "private");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allows_172_outside_private_range() {
|
||||
// 172.15.x and 172.32.x are public — only 172.16.0.0/12 is private.
|
||||
allowed("172.15.0.1");
|
||||
allowed("172.32.0.1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_link_local_and_cloud_metadata() {
|
||||
assert_eq!(denied("169.254.0.1"), "link-local");
|
||||
// The cloud metadata endpoint is the canonical SSRF target.
|
||||
assert_eq!(denied("169.254.169.254"), "link-local");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_carrier_grade_nat() {
|
||||
assert_eq!(denied("100.64.0.1"), "carrier-grade-nat");
|
||||
assert_eq!(denied("100.127.255.255"), "carrier-grade-nat");
|
||||
// 100.63.x and 100.128.x are outside 100.64.0.0/10.
|
||||
allowed("100.63.0.1");
|
||||
allowed("100.128.0.1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_multicast_and_reserved() {
|
||||
assert_eq!(denied("224.0.0.1"), "multicast");
|
||||
assert_eq!(denied("239.255.255.255"), "multicast");
|
||||
assert_eq!(denied("240.0.0.1"), "reserved");
|
||||
assert_eq!(denied("255.255.255.255"), "reserved");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allows_public_ipv4() {
|
||||
allowed("1.1.1.1");
|
||||
allowed("8.8.8.8");
|
||||
allowed("93.184.216.34"); // example.com
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv6_loopback() {
|
||||
assert_eq!(denied("::1"), "loopback");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv6_unspecified() {
|
||||
assert_eq!(denied("::"), "unspecified");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv6_link_local() {
|
||||
assert_eq!(denied("fe80::1"), "link-local");
|
||||
assert_eq!(denied("febf:ffff::1"), "link-local");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv6_unique_local() {
|
||||
assert_eq!(denied("fc00::1"), "unique-local");
|
||||
assert_eq!(denied("fd12:3456::1"), "unique-local");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn denies_ipv6_multicast() {
|
||||
assert_eq!(denied("ff00::1"), "multicast");
|
||||
assert_eq!(denied("ff02::1"), "multicast");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allows_public_ipv6() {
|
||||
allowed("2606:4700:4700::1111"); // cloudflare
|
||||
allowed("2001:4860:4860::8888"); // google
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ipv4_mapped_ipv6_rechecks_embedded_address() {
|
||||
// ::ffff:127.0.0.1 must be denied via the embedded-v4 re-check.
|
||||
assert_eq!(denied("::ffff:127.0.0.1"), "loopback");
|
||||
assert_eq!(denied("::ffff:10.0.0.1"), "private");
|
||||
assert_eq!(denied("::ffff:169.254.169.254"), "link-local");
|
||||
// A mapped *public* address stays allowed.
|
||||
allowed("::ffff:1.1.1.1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn allow_private_disables_all_denials() {
|
||||
let policy = SsrfPolicy::new(true);
|
||||
for ip in ["127.0.0.1", "10.0.0.1", "169.254.169.254", "::1", "fe80::1"] {
|
||||
assert!(policy.is_allowed(IpAddr::from_str(ip).unwrap()));
|
||||
}
|
||||
}
|
||||
|
||||
// --- resolver-path tests (the connect-time filter) ---
|
||||
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
|
||||
fn name(s: &str) -> Name {
|
||||
Name::from_str(s).unwrap()
|
||||
}
|
||||
|
||||
fn fixed_lookup(addrs: Vec<SocketAddr>) -> LookupFn {
|
||||
Arc::new(move |_host| {
|
||||
let addrs = addrs.clone();
|
||||
Box::pin(async move { Ok(addrs) })
|
||||
})
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn resolver_returns_only_allowed_addresses() {
|
||||
// A host resolving to one public + one private IP yields only
|
||||
// the public one to reqwest.
|
||||
let public: SocketAddr = "1.1.1.1:0".parse().unwrap();
|
||||
let private: SocketAddr = "10.0.0.1:0".parse().unwrap();
|
||||
let resolver =
|
||||
SsrfResolver::with_lookup(SsrfPolicy::new(false), fixed_lookup(vec![public, private]));
|
||||
let got: Vec<SocketAddr> = resolver
|
||||
.resolve(name("mixed.example"))
|
||||
.await
|
||||
.unwrap()
|
||||
.collect();
|
||||
assert_eq!(got, vec![public]);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn resolver_all_denied_fails_with_ssrf_marker() {
|
||||
// A host resolving to ONLY private IPs fails with the SSRF
|
||||
// marker (not a generic empty/DNS result).
|
||||
let resolver = SsrfResolver::with_lookup(
|
||||
SsrfPolicy::new(false),
|
||||
fixed_lookup(vec![
|
||||
"10.0.0.1:0".parse().unwrap(),
|
||||
"127.0.0.1:0".parse().unwrap(),
|
||||
]),
|
||||
);
|
||||
let Err(err) = resolver.resolve(name("internal.example")).await else {
|
||||
panic!("all-denied resolution should error");
|
||||
};
|
||||
assert!(
|
||||
err.to_string().starts_with(SSRF_BLOCK_PREFIX),
|
||||
"expected SSRF marker, got: {err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn resolver_dns_rebinding_second_resolution_denied() {
|
||||
// Simulate rebinding: public IP on the first lookup, private on
|
||||
// the second. The connect-time filter denies the second.
|
||||
let calls = Arc::new(AtomicUsize::new(0));
|
||||
let calls2 = calls.clone();
|
||||
let lookup: LookupFn = Arc::new(move |_host| {
|
||||
let n = calls2.fetch_add(1, Ordering::SeqCst);
|
||||
Box::pin(async move {
|
||||
let addr: SocketAddr = if n == 0 {
|
||||
"1.1.1.1:0".parse().unwrap()
|
||||
} else {
|
||||
"127.0.0.1:0".parse().unwrap()
|
||||
};
|
||||
Ok(vec![addr])
|
||||
})
|
||||
});
|
||||
let resolver = SsrfResolver::with_lookup(SsrfPolicy::new(false), lookup);
|
||||
|
||||
// First resolution: public → allowed.
|
||||
let first: Vec<SocketAddr> = resolver
|
||||
.resolve(name("rebind.example"))
|
||||
.await
|
||||
.unwrap()
|
||||
.collect();
|
||||
assert_eq!(first, vec!["1.1.1.1:0".parse::<SocketAddr>().unwrap()]);
|
||||
|
||||
// Second resolution: rebinding returns loopback → denied.
|
||||
let Err(err) = resolver.resolve(name("rebind.example")).await else {
|
||||
panic!("rebound private address must be denied");
|
||||
};
|
||||
assert!(err.to_string().contains("loopback"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn resolver_empty_resolution_is_not_ssrf() {
|
||||
// Genuine DNS miss (no addresses) returns an empty iterator,
|
||||
// NOT the SSRF marker — reqwest surfaces a normal DNS error.
|
||||
let resolver = SsrfResolver::with_lookup(SsrfPolicy::new(false), fixed_lookup(vec![]));
|
||||
let got: Vec<SocketAddr> = resolver
|
||||
.resolve(name("nxdomain.example"))
|
||||
.await
|
||||
.unwrap()
|
||||
.collect();
|
||||
assert!(got.is_empty());
|
||||
}
|
||||
}
|
||||
212
crates/manager-core/src/topic_repo.rs
Normal file
212
crates/manager-core/src/topic_repo.rs
Normal file
@@ -0,0 +1,212 @@
|
||||
//! `TopicRepo` — CRUD for the `topics` table (v1.1.6).
|
||||
//!
|
||||
//! This table holds ONLY topics that have been explicitly externalized
|
||||
//! for SSE subscription (design notes §5). Internal-only pub/sub topics
|
||||
//! stay implicit — they never get a row here, and the publish path never
|
||||
//! consults this table. The two readers are the topic admin endpoints
|
||||
//! ([`crate::topics_api`]) and the SSE subscribe authorization
|
||||
//! ([`crate::realtime_authority`]).
|
||||
|
||||
use async_trait::async_trait;
|
||||
use chrono::{DateTime, Utc};
|
||||
use picloud_shared::AppId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::PgPool;
|
||||
|
||||
/// External-subscriber auth gate for a topic. `'public'` + `'token'` in
|
||||
/// v1.1.6; `'session'` (v1.1.8) and `'script'` (v1.2) extend the DB
|
||||
/// CHECK constraint and this enum later.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum TopicAuthMode {
|
||||
Public,
|
||||
Token,
|
||||
}
|
||||
|
||||
impl TopicAuthMode {
|
||||
#[must_use]
|
||||
pub const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Public => "public",
|
||||
Self::Token => "token",
|
||||
}
|
||||
}
|
||||
|
||||
fn from_db(s: &str) -> Result<Self, TopicRepoError> {
|
||||
match s {
|
||||
"public" => Ok(Self::Public),
|
||||
"token" => Ok(Self::Token),
|
||||
other => Err(TopicRepoError::Backend(format!(
|
||||
"unknown auth_mode in DB: {other}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A registered, externally-subscribable topic row.
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub struct Topic {
|
||||
pub name: String,
|
||||
pub external_subscribable: bool,
|
||||
pub auth_mode: TopicAuthMode,
|
||||
pub created_at: DateTime<Utc>,
|
||||
pub updated_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TopicRepoError {
|
||||
#[error("a topic named {0:?} already exists in this app")]
|
||||
AlreadyExists(String),
|
||||
#[error("database error: {0}")]
|
||||
Db(#[from] sqlx::Error),
|
||||
#[error("topic backend error: {0}")]
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait TopicRepo: Send + Sync {
|
||||
/// Register a topic. Errors `AlreadyExists` on PK conflict.
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: bool,
|
||||
auth_mode: TopicAuthMode,
|
||||
) -> Result<Topic, TopicRepoError>;
|
||||
|
||||
/// List every registered topic in the app, ordered by name.
|
||||
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError>;
|
||||
|
||||
/// Fetch one topic by name, `None` if not registered.
|
||||
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError>;
|
||||
|
||||
/// Update `external_subscribable` and/or `auth_mode` (each `None`
|
||||
/// leaves the column unchanged). `None` return = no such topic.
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: Option<bool>,
|
||||
auth_mode: Option<TopicAuthMode>,
|
||||
) -> Result<Option<Topic>, TopicRepoError>;
|
||||
|
||||
/// Unregister a topic. Returns `true` if a row was removed.
|
||||
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError>;
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct TopicRow {
|
||||
name: String,
|
||||
external_subscribable: bool,
|
||||
auth_mode: String,
|
||||
created_at: DateTime<Utc>,
|
||||
updated_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
impl TopicRow {
|
||||
fn into_topic(self) -> Result<Topic, TopicRepoError> {
|
||||
Ok(Topic {
|
||||
auth_mode: TopicAuthMode::from_db(&self.auth_mode)?,
|
||||
name: self.name,
|
||||
external_subscribable: self.external_subscribable,
|
||||
created_at: self.created_at,
|
||||
updated_at: self.updated_at,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const SELECT_COLS: &str = "name, external_subscribable, auth_mode, created_at, updated_at";
|
||||
|
||||
pub struct PostgresTopicRepo {
|
||||
pool: PgPool,
|
||||
}
|
||||
|
||||
impl PostgresTopicRepo {
|
||||
#[must_use]
|
||||
pub fn new(pool: PgPool) -> Self {
|
||||
Self { pool }
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl TopicRepo for PostgresTopicRepo {
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: bool,
|
||||
auth_mode: TopicAuthMode,
|
||||
) -> Result<Topic, TopicRepoError> {
|
||||
let row: Option<TopicRow> = sqlx::query_as(&format!(
|
||||
"INSERT INTO topics (app_id, name, external_subscribable, auth_mode) \
|
||||
VALUES ($1, $2, $3, $4) \
|
||||
ON CONFLICT (app_id, name) DO NOTHING \
|
||||
RETURNING {SELECT_COLS}"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(name)
|
||||
.bind(external_subscribable)
|
||||
.bind(auth_mode.as_str())
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
match row {
|
||||
Some(r) => r.into_topic(),
|
||||
None => Err(TopicRepoError::AlreadyExists(name.to_string())),
|
||||
}
|
||||
}
|
||||
|
||||
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError> {
|
||||
let rows: Vec<TopicRow> = sqlx::query_as(&format!(
|
||||
"SELECT {SELECT_COLS} FROM topics WHERE app_id = $1 ORDER BY name"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.fetch_all(&self.pool)
|
||||
.await?;
|
||||
rows.into_iter().map(TopicRow::into_topic).collect()
|
||||
}
|
||||
|
||||
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
|
||||
let row: Option<TopicRow> = sqlx::query_as(&format!(
|
||||
"SELECT {SELECT_COLS} FROM topics WHERE app_id = $1 AND name = $2"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(name)
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
row.map(TopicRow::into_topic).transpose()
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: Option<bool>,
|
||||
auth_mode: Option<TopicAuthMode>,
|
||||
) -> Result<Option<Topic>, TopicRepoError> {
|
||||
// COALESCE leaves a column untouched when its bind is NULL.
|
||||
let row: Option<TopicRow> = sqlx::query_as(&format!(
|
||||
"UPDATE topics SET \
|
||||
external_subscribable = COALESCE($3, external_subscribable), \
|
||||
auth_mode = COALESCE($4, auth_mode), \
|
||||
updated_at = NOW() \
|
||||
WHERE app_id = $1 AND name = $2 \
|
||||
RETURNING {SELECT_COLS}"
|
||||
))
|
||||
.bind(app_id.into_inner())
|
||||
.bind(name)
|
||||
.bind(external_subscribable)
|
||||
.bind(auth_mode.map(|m| m.as_str()))
|
||||
.fetch_optional(&self.pool)
|
||||
.await?;
|
||||
row.map(TopicRow::into_topic).transpose()
|
||||
}
|
||||
|
||||
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError> {
|
||||
let res = sqlx::query("DELETE FROM topics WHERE app_id = $1 AND name = $2")
|
||||
.bind(app_id.into_inner())
|
||||
.bind(name)
|
||||
.execute(&self.pool)
|
||||
.await?;
|
||||
Ok(res.rows_affected() > 0)
|
||||
}
|
||||
}
|
||||
629
crates/manager-core/src/topics_api.rs
Normal file
629
crates/manager-core/src/topics_api.rs
Normal file
@@ -0,0 +1,629 @@
|
||||
//! `/api/v1/admin/apps/{id}/topics*` — topic registration admin
|
||||
//! endpoints (v1.1.6).
|
||||
//!
|
||||
//! These manage the `topics` table: the explicit registry of which
|
||||
//! pub/sub topics are externally subscribable over SSE (design notes
|
||||
//! §5). Internal-only topics never appear here.
|
||||
//!
|
||||
//! * `POST /apps/{id}/topics` — register a topic.
|
||||
//! * `GET /apps/{id}/topics` — list registered topics.
|
||||
//! * `PATCH /apps/{id}/topics/{name}` — flip external/auth_mode.
|
||||
//! * `DELETE /apps/{id}/topics/{name}` — unregister + disconnect.
|
||||
//!
|
||||
//! The PATCH endpoint is deliberately its OWN surface (not folded into a
|
||||
//! generic topic update) so every change to externally-subscribable
|
||||
//! status is a discrete, watchable/auditable API call (§5 commitment).
|
||||
//!
|
||||
//! Create / update / delete are gated by `AppTopicManage` (→ `app:admin`
|
||||
//! scope); list is gated by the existing `AppRead`. DELETE also drops
|
||||
//! the topic's in-process broadcast channel so live SSE subscribers
|
||||
//! disconnect cleanly.
|
||||
|
||||
use std::sync::Arc;
|
||||
|
||||
use axum::extract::{Path, State};
|
||||
use axum::http::StatusCode;
|
||||
use axum::response::{IntoResponse, Json, Response};
|
||||
use axum::routing::{get, patch};
|
||||
use axum::{Extension, Router};
|
||||
use picloud_shared::{AppId, Principal, RealtimeBroadcaster};
|
||||
use serde::Deserialize;
|
||||
use serde_json::json;
|
||||
|
||||
use crate::app_repo::AppRepository;
|
||||
use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
|
||||
use crate::topic_repo::{Topic, TopicAuthMode, TopicRepo, TopicRepoError};
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct TopicsState {
|
||||
pub topics: Arc<dyn TopicRepo>,
|
||||
pub apps: Arc<dyn AppRepository>,
|
||||
pub authz: Arc<dyn AuthzRepo>,
|
||||
pub broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
}
|
||||
|
||||
pub fn topics_router(state: TopicsState) -> Router {
|
||||
Router::new()
|
||||
.route("/apps/{app_id}/topics", get(list_topics).post(create_topic))
|
||||
.route(
|
||||
"/apps/{app_id}/topics/{name}",
|
||||
patch(update_topic).delete(delete_topic),
|
||||
)
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreateTopicRequest {
|
||||
pub name: String,
|
||||
#[serde(default)]
|
||||
pub external_subscribable: bool,
|
||||
#[serde(default = "default_auth_mode")]
|
||||
pub auth_mode: TopicAuthMode,
|
||||
}
|
||||
|
||||
const fn default_auth_mode() -> TopicAuthMode {
|
||||
TopicAuthMode::Public
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct UpdateTopicRequest {
|
||||
#[serde(default)]
|
||||
pub external_subscribable: Option<bool>,
|
||||
#[serde(default)]
|
||||
pub auth_mode: Option<TopicAuthMode>,
|
||||
}
|
||||
|
||||
/// Topic names are concrete (external pattern subscription is v1.2), so
|
||||
/// reject empties and `*` wildcards at registration.
|
||||
fn validate_topic_name(name: &str) -> Result<(), TopicsApiError> {
|
||||
if name.trim().is_empty() {
|
||||
return Err(TopicsApiError::Invalid(
|
||||
"topic name must not be empty".into(),
|
||||
));
|
||||
}
|
||||
if name.contains('*') {
|
||||
return Err(TopicsApiError::Invalid(
|
||||
"topic name must be a concrete topic, not a pattern (no '*')".into(),
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn create_topic(
|
||||
State(s): State<TopicsState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
Json(input): Json<CreateTopicRequest>,
|
||||
) -> Result<(StatusCode, Json<Topic>), TopicsApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppTopicManage(app_id),
|
||||
)
|
||||
.await?;
|
||||
validate_topic_name(&input.name)?;
|
||||
let topic = s
|
||||
.topics
|
||||
.create(
|
||||
app_id,
|
||||
input.name.trim(),
|
||||
input.external_subscribable,
|
||||
input.auth_mode,
|
||||
)
|
||||
.await?;
|
||||
Ok((StatusCode::CREATED, Json(topic)))
|
||||
}
|
||||
|
||||
#[derive(Debug, serde::Serialize)]
|
||||
struct ListTopicsResponse {
|
||||
topics: Vec<Topic>,
|
||||
}
|
||||
|
||||
async fn list_topics(
|
||||
State(s): State<TopicsState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
) -> Result<Json<ListTopicsResponse>, TopicsApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(s.authz.as_ref(), &principal, Capability::AppRead(app_id)).await?;
|
||||
let topics = s.topics.list(app_id).await?;
|
||||
Ok(Json(ListTopicsResponse { topics }))
|
||||
}
|
||||
|
||||
async fn update_topic(
|
||||
State(s): State<TopicsState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path((app_id, name)): Path<(AppId, String)>,
|
||||
Json(input): Json<UpdateTopicRequest>,
|
||||
) -> Result<Json<Topic>, TopicsApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppTopicManage(app_id),
|
||||
)
|
||||
.await?;
|
||||
let topic = s
|
||||
.topics
|
||||
.update(app_id, &name, input.external_subscribable, input.auth_mode)
|
||||
.await?
|
||||
.ok_or(TopicsApiError::NotFound)?;
|
||||
Ok(Json(topic))
|
||||
}
|
||||
|
||||
async fn delete_topic(
|
||||
State(s): State<TopicsState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path((app_id, name)): Path<(AppId, String)>,
|
||||
) -> Result<StatusCode, TopicsApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppTopicManage(app_id),
|
||||
)
|
||||
.await?;
|
||||
if !s.topics.delete(app_id, &name).await? {
|
||||
return Err(TopicsApiError::NotFound);
|
||||
}
|
||||
// Disconnect any live SSE subscribers for the now-unregistered topic.
|
||||
s.broadcaster.drop_topic(app_id, &name).await;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
||||
async fn ensure_app_exists(apps: &dyn AppRepository, app_id: AppId) -> Result<(), TopicsApiError> {
|
||||
apps.get_by_id(app_id)
|
||||
.await
|
||||
.map_err(|e| TopicsApiError::Backend(e.to_string()))?
|
||||
.ok_or(TopicsApiError::AppNotFound)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum TopicsApiError {
|
||||
#[error("app not found")]
|
||||
AppNotFound,
|
||||
#[error("topic not found")]
|
||||
NotFound,
|
||||
#[error("{0}")]
|
||||
AlreadyExists(String),
|
||||
#[error("invalid request: {0}")]
|
||||
Invalid(String),
|
||||
#[error("forbidden")]
|
||||
Forbidden,
|
||||
#[error("authorization repo error: {0}")]
|
||||
AuthzRepo(String),
|
||||
#[error("topics backend: {0}")]
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
impl From<AuthzDenied> for TopicsApiError {
|
||||
fn from(d: AuthzDenied) -> Self {
|
||||
match d {
|
||||
AuthzDenied::Denied => Self::Forbidden,
|
||||
AuthzDenied::Repo(e) => Self::AuthzRepo(e.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<AuthzError> for TopicsApiError {
|
||||
fn from(e: AuthzError) -> Self {
|
||||
Self::AuthzRepo(e.to_string())
|
||||
}
|
||||
}
|
||||
|
||||
impl From<TopicRepoError> for TopicsApiError {
|
||||
fn from(e: TopicRepoError) -> Self {
|
||||
match e {
|
||||
TopicRepoError::AlreadyExists(name) => {
|
||||
Self::AlreadyExists(format!("a topic named {name:?} already exists in this app"))
|
||||
}
|
||||
other => Self::Backend(other.to_string()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl IntoResponse for TopicsApiError {
|
||||
fn into_response(self) -> Response {
|
||||
let (status, body) = match &self {
|
||||
Self::AppNotFound | Self::NotFound => {
|
||||
(StatusCode::NOT_FOUND, json!({ "error": self.to_string() }))
|
||||
}
|
||||
Self::AlreadyExists(_) => (StatusCode::CONFLICT, json!({ "error": self.to_string() })),
|
||||
Self::Invalid(_) => (
|
||||
StatusCode::UNPROCESSABLE_ENTITY,
|
||||
json!({ "error": self.to_string() }),
|
||||
),
|
||||
Self::Forbidden => (StatusCode::FORBIDDEN, json!({ "error": self.to_string() })),
|
||||
Self::AuthzRepo(e) => {
|
||||
tracing::error!(error = %e, "topics admin authz repo error");
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
json!({ "error": "internal error" }),
|
||||
)
|
||||
}
|
||||
Self::Backend(e) => {
|
||||
tracing::error!(error = %e, "topics admin backend error");
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
json!({ "error": "internal error" }),
|
||||
)
|
||||
}
|
||||
};
|
||||
(status, Json(body)).into_response()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
//! In-memory handler tests: capability enforcement, the
|
||||
//! `external_subscribable` default, the flip being its own endpoint,
|
||||
//! cross-app isolation, and DELETE disconnecting subscribers. The
|
||||
//! Postgres repo is exercised by the schema + integration suites.
|
||||
|
||||
use super::*;
|
||||
use crate::repo::ScriptRepositoryError;
|
||||
use async_trait::async_trait;
|
||||
use chrono::Utc;
|
||||
use picloud_shared::{
|
||||
AdminUserId, App, AppRole, BroadcasterError, InstanceRole, RealtimeEvent, UserId,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Mutex as StdMutex;
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
#[derive(Default)]
|
||||
struct InMemoryTopicRepo {
|
||||
inner: Mutex<HashMap<(AppId, String), Topic>>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl TopicRepo for InMemoryTopicRepo {
|
||||
async fn create(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: bool,
|
||||
auth_mode: TopicAuthMode,
|
||||
) -> Result<Topic, TopicRepoError> {
|
||||
let mut g = self.inner.lock().await;
|
||||
if g.contains_key(&(app_id, name.to_string())) {
|
||||
return Err(TopicRepoError::AlreadyExists(name.to_string()));
|
||||
}
|
||||
let now = Utc::now();
|
||||
let t = Topic {
|
||||
name: name.to_string(),
|
||||
external_subscribable,
|
||||
auth_mode,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
};
|
||||
g.insert((app_id, name.to_string()), t.clone());
|
||||
Ok(t)
|
||||
}
|
||||
async fn list(&self, app_id: AppId) -> Result<Vec<Topic>, TopicRepoError> {
|
||||
let g = self.inner.lock().await;
|
||||
let mut v: Vec<Topic> = g
|
||||
.iter()
|
||||
.filter(|((a, _), _)| *a == app_id)
|
||||
.map(|(_, t)| t.clone())
|
||||
.collect();
|
||||
v.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok(v)
|
||||
}
|
||||
async fn get(&self, app_id: AppId, name: &str) -> Result<Option<Topic>, TopicRepoError> {
|
||||
Ok(self
|
||||
.inner
|
||||
.lock()
|
||||
.await
|
||||
.get(&(app_id, name.to_string()))
|
||||
.cloned())
|
||||
}
|
||||
async fn update(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
name: &str,
|
||||
external_subscribable: Option<bool>,
|
||||
auth_mode: Option<TopicAuthMode>,
|
||||
) -> Result<Option<Topic>, TopicRepoError> {
|
||||
let mut g = self.inner.lock().await;
|
||||
let Some(t) = g.get_mut(&(app_id, name.to_string())) else {
|
||||
return Ok(None);
|
||||
};
|
||||
if let Some(e) = external_subscribable {
|
||||
t.external_subscribable = e;
|
||||
}
|
||||
if let Some(m) = auth_mode {
|
||||
t.auth_mode = m;
|
||||
}
|
||||
t.updated_at = Utc::now();
|
||||
Ok(Some(t.clone()))
|
||||
}
|
||||
async fn delete(&self, app_id: AppId, name: &str) -> Result<bool, TopicRepoError> {
|
||||
Ok(self
|
||||
.inner
|
||||
.lock()
|
||||
.await
|
||||
.remove(&(app_id, name.to_string()))
|
||||
.is_some())
|
||||
}
|
||||
}
|
||||
|
||||
struct InMemoryAppRepo(AppId);
|
||||
#[async_trait]
|
||||
impl AppRepository for InMemoryAppRepo {
|
||||
async fn list(&self) -> Result<Vec<App>, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn list_for_user(&self, _: AdminUserId) -> Result<Vec<App>, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn get_by_id(&self, id: AppId) -> Result<Option<App>, ScriptRepositoryError> {
|
||||
if id != self.0 {
|
||||
return Ok(None);
|
||||
}
|
||||
let now = Utc::now();
|
||||
Ok(Some(App {
|
||||
id,
|
||||
slug: "test".into(),
|
||||
name: "test".into(),
|
||||
description: None,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
}))
|
||||
}
|
||||
async fn get_by_slug(&self, _: &str) -> Result<Option<App>, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn get_by_slug_or_history(
|
||||
&self,
|
||||
_: &str,
|
||||
) -> Result<Option<crate::app_repo::AppLookup>, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn slug_in_history(&self, _: &str) -> Result<Option<App>, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn create(
|
||||
&self,
|
||||
_: &str,
|
||||
_: &str,
|
||||
_: Option<&str>,
|
||||
) -> Result<App, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn create_with_takeover(
|
||||
&self,
|
||||
_: &str,
|
||||
_: &str,
|
||||
_: Option<&str>,
|
||||
) -> Result<App, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn update(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: Option<&str>,
|
||||
_: Option<Option<&str>>,
|
||||
) -> Result<App, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn rename_slug(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: bool,
|
||||
) -> Result<App, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn delete(&self, _: AppId) -> Result<(), ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn delete_cascade(&self, _: AppId) -> Result<(), ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn count_scripts_in_app(&self, _: AppId) -> Result<i64, ScriptRepositoryError> {
|
||||
unimplemented!()
|
||||
}
|
||||
}
|
||||
|
||||
/// Grants `AppAdmin` only for `granted_app`; denies elsewhere — used
|
||||
/// for the cross-app isolation test.
|
||||
struct PerAppAuthzRepo {
|
||||
granted_app: AppId,
|
||||
}
|
||||
#[async_trait]
|
||||
impl AuthzRepo for PerAppAuthzRepo {
|
||||
async fn membership(
|
||||
&self,
|
||||
_: UserId,
|
||||
app_id: AppId,
|
||||
) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok((app_id == self.granted_app).then_some(AppRole::AppAdmin))
|
||||
}
|
||||
}
|
||||
|
||||
struct DenyAuthzRepo;
|
||||
#[async_trait]
|
||||
impl AuthzRepo for DenyAuthzRepo {
|
||||
async fn membership(&self, _: UserId, _: AppId) -> Result<Option<AppRole>, AuthzError> {
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
struct RecordingBroadcaster {
|
||||
dropped: StdMutex<Vec<(AppId, String)>>,
|
||||
}
|
||||
#[async_trait]
|
||||
impl RealtimeBroadcaster for RecordingBroadcaster {
|
||||
async fn subscribe(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
) -> Result<tokio::sync::broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
|
||||
unimplemented!()
|
||||
}
|
||||
async fn publish(&self, _: AppId, _: &str, _: RealtimeEvent) {}
|
||||
async fn drop_topic(&self, app_id: AppId, topic: &str) {
|
||||
self.dropped
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push((app_id, topic.to_string()));
|
||||
}
|
||||
}
|
||||
|
||||
fn member() -> Principal {
|
||||
Principal {
|
||||
user_id: AdminUserId::new(),
|
||||
instance_role: InstanceRole::Member,
|
||||
scopes: None,
|
||||
app_binding: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn state(app_id: AppId, authz: Arc<dyn AuthzRepo>) -> (TopicsState, Arc<RecordingBroadcaster>) {
|
||||
let bc = Arc::new(RecordingBroadcaster::default());
|
||||
let state = TopicsState {
|
||||
topics: Arc::new(InMemoryTopicRepo::default()),
|
||||
apps: Arc::new(InMemoryAppRepo(app_id)),
|
||||
authz,
|
||||
broadcaster: bc.clone(),
|
||||
};
|
||||
(state, bc)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn register_defaults_external_subscribable_false() {
|
||||
let app = AppId::new();
|
||||
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
|
||||
let (status, Json(topic)) = create_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path(app),
|
||||
Json(CreateTopicRequest {
|
||||
name: "chat".into(),
|
||||
external_subscribable: false,
|
||||
auth_mode: TopicAuthMode::Public,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(status, StatusCode::CREATED);
|
||||
assert!(!topic.external_subscribable);
|
||||
assert_eq!(topic.name, "chat");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn flip_requires_app_admin_role() {
|
||||
let app = AppId::new();
|
||||
// Topic exists; the caller has no role → PATCH is forbidden.
|
||||
let (s, _) = state(app, Arc::new(DenyAuthzRepo));
|
||||
s.topics
|
||||
.create(app, "chat", false, TopicAuthMode::Public)
|
||||
.await
|
||||
.unwrap();
|
||||
let err = update_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path((app, "chat".to_string())),
|
||||
Json(UpdateTopicRequest {
|
||||
external_subscribable: Some(true),
|
||||
auth_mode: None,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, TopicsApiError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn flip_is_its_own_endpoint_and_toggles_external() {
|
||||
// The PATCH handler is a distinct surface from create; flipping
|
||||
// external_subscribable false→true is a single discrete call.
|
||||
let app = AppId::new();
|
||||
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
|
||||
s.topics
|
||||
.create(app, "chat", false, TopicAuthMode::Public)
|
||||
.await
|
||||
.unwrap();
|
||||
let Json(updated) = update_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path((app, "chat".to_string())),
|
||||
Json(UpdateTopicRequest {
|
||||
external_subscribable: Some(true),
|
||||
auth_mode: Some(TopicAuthMode::Token),
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(updated.external_subscribable);
|
||||
assert_eq!(updated.auth_mode, TopicAuthMode::Token);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn delete_disconnects_subscribers() {
|
||||
let app = AppId::new();
|
||||
let (s, bc) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
|
||||
s.topics
|
||||
.create(app, "chat", true, TopicAuthMode::Public)
|
||||
.await
|
||||
.unwrap();
|
||||
let status = delete_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path((app, "chat".to_string())),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(status, StatusCode::NO_CONTENT);
|
||||
assert_eq!(
|
||||
bc.dropped.lock().unwrap().as_slice(),
|
||||
&[(app, "chat".to_string())]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cross_app_admin_cannot_manage_other_app() {
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
// Caller is admin of app A only; both apps exist via separate state.
|
||||
let authz = Arc::new(PerAppAuthzRepo { granted_app: app_a });
|
||||
// App-B-scoped state, but the caller only has A's grant.
|
||||
let (s, _) = state(app_b, authz);
|
||||
let err = create_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path(app_b),
|
||||
Json(CreateTopicRequest {
|
||||
name: "chat".into(),
|
||||
external_subscribable: true,
|
||||
auth_mode: TopicAuthMode::Public,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, TopicsApiError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pattern_name_rejected() {
|
||||
let app = AppId::new();
|
||||
let (s, _) = state(app, Arc::new(PerAppAuthzRepo { granted_app: app }));
|
||||
let err = create_topic(
|
||||
State(s),
|
||||
Extension(member()),
|
||||
Path(app),
|
||||
Json(CreateTopicRequest {
|
||||
name: "user.*".into(),
|
||||
external_subscribable: true,
|
||||
auth_mode: TopicAuthMode::Public,
|
||||
}),
|
||||
)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(err, TopicsApiError::Invalid(_)));
|
||||
}
|
||||
}
|
||||
@@ -56,6 +56,11 @@ pub struct TriggerConfig {
|
||||
pub dead_letter_retention_days: u32,
|
||||
/// abandoned-execution retention before GC, in days. Default 7.
|
||||
pub abandoned_retention_days: u32,
|
||||
|
||||
/// Cron scheduler poll cadence, in ms (v1.1.4). Default 30 000 —
|
||||
/// real-world cron precision is per-minute, so a 30s tick is fine.
|
||||
/// Floored at 1s by the scheduler.
|
||||
pub cron_tick_interval_ms: u32,
|
||||
}
|
||||
|
||||
impl TriggerConfig {
|
||||
@@ -69,6 +74,7 @@ impl TriggerConfig {
|
||||
retry_jitter_pct: 20,
|
||||
dead_letter_retention_days: 30,
|
||||
abandoned_retention_days: 7,
|
||||
cron_tick_interval_ms: 30_000,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,6 +97,10 @@ impl TriggerConfig {
|
||||
&mut c.abandoned_retention_days,
|
||||
"PICLOUD_ABANDONED_EXECUTIONS_RETENTION_DAYS",
|
||||
);
|
||||
load_u32(
|
||||
&mut c.cron_tick_interval_ms,
|
||||
"PICLOUD_CRON_TICK_INTERVAL_MS",
|
||||
);
|
||||
c
|
||||
}
|
||||
}
|
||||
@@ -141,6 +151,7 @@ mod tests {
|
||||
assert_eq!(c.retry_jitter_pct, 20);
|
||||
assert_eq!(c.dead_letter_retention_days, 30);
|
||||
assert_eq!(c.abandoned_retention_days, 7);
|
||||
assert_eq!(c.cron_tick_interval_ms, 30_000);
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -5,7 +5,9 @@
|
||||
|
||||
use async_trait::async_trait;
|
||||
use chrono::{DateTime, Utc};
|
||||
use picloud_shared::{AdminUserId, AppId, DocsEventOp, KvEventOp, ScriptId, TriggerId};
|
||||
use picloud_shared::{
|
||||
AdminUserId, AppId, DocsEventOp, FilesEventOp, KvEventOp, ScriptId, TriggerId,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
@@ -49,6 +51,12 @@ pub enum TriggerKind {
|
||||
Kv,
|
||||
Docs,
|
||||
DeadLetter,
|
||||
/// v1.1.4.
|
||||
Cron,
|
||||
/// v1.1.5.
|
||||
Files,
|
||||
/// v1.1.5.
|
||||
Pubsub,
|
||||
}
|
||||
|
||||
impl TriggerKind {
|
||||
@@ -58,6 +66,9 @@ impl TriggerKind {
|
||||
Self::Kv => "kv",
|
||||
Self::Docs => "docs",
|
||||
Self::DeadLetter => "dead_letter",
|
||||
Self::Cron => "cron",
|
||||
Self::Files => "files",
|
||||
Self::Pubsub => "pubsub",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -67,6 +78,9 @@ impl TriggerKind {
|
||||
"kv" => Some(Self::Kv),
|
||||
"docs" => Some(Self::Docs),
|
||||
"dead_letter" => Some(Self::DeadLetter),
|
||||
"cron" => Some(Self::Cron),
|
||||
"files" => Some(Self::Files),
|
||||
"pubsub" => Some(Self::Pubsub),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
@@ -108,6 +122,21 @@ pub enum TriggerDetails {
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
script_id_filter: Option<ScriptId>,
|
||||
},
|
||||
/// v1.1.4. The 6-field cron schedule + IANA timezone the trigger
|
||||
/// fires on, plus the last enqueue time (for dashboard display).
|
||||
Cron {
|
||||
schedule: String,
|
||||
timezone: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
last_fired_at: Option<DateTime<Utc>>,
|
||||
},
|
||||
/// v1.1.5. Same shape as KV/docs: a collection glob + op subset.
|
||||
Files {
|
||||
collection_glob: String,
|
||||
ops: Vec<FilesEventOp>,
|
||||
},
|
||||
/// v1.1.5. A topic pattern: exact, `<prefix>.*`, or `*`.
|
||||
Pubsub { topic_pattern: String },
|
||||
}
|
||||
|
||||
/// Create payload for a KV trigger. Defaults applied at the admin
|
||||
@@ -148,6 +177,61 @@ pub struct CreateDeadLetterTrigger {
|
||||
pub registered_by_principal: AdminUserId,
|
||||
}
|
||||
|
||||
/// Create payload for a cron trigger (v1.1.4). `schedule` is a 6-field
|
||||
/// cron expression and `timezone` an IANA name; both are validated
|
||||
/// (by the admin endpoint and defensively by the repo) before insert.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CreateCronTrigger {
|
||||
pub script_id: ScriptId,
|
||||
pub schedule: String,
|
||||
pub timezone: String,
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
pub retry_max_attempts: u32,
|
||||
pub retry_backoff: BackoffShape,
|
||||
pub retry_base_ms: u32,
|
||||
pub registered_by_principal: AdminUserId,
|
||||
}
|
||||
|
||||
/// Create payload for a files trigger (v1.1.5). Same shape as KV with
|
||||
/// `FilesEventOp` ops.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CreateFilesTrigger {
|
||||
pub script_id: ScriptId,
|
||||
pub collection_glob: String,
|
||||
pub ops: Vec<FilesEventOp>,
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
pub retry_max_attempts: u32,
|
||||
pub retry_backoff: BackoffShape,
|
||||
pub retry_base_ms: u32,
|
||||
pub registered_by_principal: AdminUserId,
|
||||
}
|
||||
|
||||
/// One match for the dispatcher's files trigger fan-out lookup
|
||||
/// (v1.1.5). Same shape as `KvTriggerMatch`.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FilesTriggerMatch {
|
||||
pub trigger_id: TriggerId,
|
||||
pub script_id: ScriptId,
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
pub retry_max_attempts: u32,
|
||||
pub retry_backoff: BackoffShape,
|
||||
pub retry_base_ms: u32,
|
||||
pub registered_by_principal: AdminUserId,
|
||||
}
|
||||
|
||||
/// Create payload for a pubsub trigger (v1.1.5). `topic_pattern` is
|
||||
/// validated (exact / `<prefix>.*` / `*`) before insert.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CreatePubsubTrigger {
|
||||
pub script_id: ScriptId,
|
||||
pub topic_pattern: String,
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
pub retry_max_attempts: u32,
|
||||
pub retry_backoff: BackoffShape,
|
||||
pub retry_base_ms: u32,
|
||||
pub registered_by_principal: AdminUserId,
|
||||
}
|
||||
|
||||
/// One match for the dispatcher's "which KV triggers fire on this
|
||||
/// event" lookup. Carries everything the dispatcher needs to construct
|
||||
/// the outbox row.
|
||||
@@ -206,6 +290,29 @@ pub trait TriggerRepo: Send + Sync {
|
||||
req: CreateDeadLetterTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError>;
|
||||
|
||||
/// v1.1.4. `schedule` + `timezone` are validated before insert; an
|
||||
/// invalid expression or unknown IANA name returns
|
||||
/// `TriggerRepoError::Invalid`.
|
||||
async fn create_cron_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateCronTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError>;
|
||||
|
||||
/// v1.1.5.
|
||||
async fn create_files_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateFilesTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError>;
|
||||
|
||||
/// v1.1.5. `topic_pattern` is validated before insert.
|
||||
async fn create_pubsub_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreatePubsubTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError>;
|
||||
|
||||
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError>;
|
||||
|
||||
async fn get(&self, id: TriggerId) -> Result<Option<Trigger>, TriggerRepoError>;
|
||||
@@ -233,6 +340,16 @@ pub trait TriggerRepo: Send + Sync {
|
||||
op: DocsEventOp,
|
||||
) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError>;
|
||||
|
||||
/// Dispatcher hot path for files fan-out (v1.1.5). Mirrors the KV
|
||||
/// fan-out logic: pull every enabled files trigger, filter glob +
|
||||
/// ops in Rust (empty ops array means "any op").
|
||||
async fn list_matching_files(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
op: FilesEventOp,
|
||||
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError>;
|
||||
|
||||
/// Dispatcher hot path for dead-letter fan-out. Filters: source
|
||||
/// (or any-source), originating trigger_id (or any), originating
|
||||
/// script_id (or any). Each filter is "match OR is_null".
|
||||
@@ -453,6 +570,197 @@ impl TriggerRepo for PostgresTriggerRepo {
|
||||
})
|
||||
}
|
||||
|
||||
async fn create_cron_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateCronTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
// Defense-in-depth validation (the admin endpoint validates too).
|
||||
crate::cron_scheduler::validate_schedule(&req.schedule)
|
||||
.map_err(|e| TriggerRepoError::Invalid(format!("invalid cron schedule: {e}")))?;
|
||||
crate::cron_scheduler::validate_timezone(&req.timezone)
|
||||
.map_err(|e| TriggerRepoError::Invalid(format!("invalid timezone: {e}")))?;
|
||||
|
||||
let mut tx = self.pool.begin().await?;
|
||||
let parent: TriggerRow = sqlx::query_as(
|
||||
"INSERT INTO triggers ( \
|
||||
app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal \
|
||||
) VALUES ($1, $2, 'cron', TRUE, $3, $4, $5, $6, $7) \
|
||||
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal, created_at, updated_at",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(req.script_id.into_inner())
|
||||
.bind(req.dispatch_mode.as_str())
|
||||
.bind(i32::try_from(req.retry_max_attempts).unwrap_or(3))
|
||||
.bind(req.retry_backoff.as_str())
|
||||
.bind(i32::try_from(req.retry_base_ms).unwrap_or(1000))
|
||||
.bind(req.registered_by_principal.into_inner())
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO cron_trigger_details (trigger_id, schedule, timezone) \
|
||||
VALUES ($1, $2, $3)",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.bind(&req.schedule)
|
||||
.bind(&req.timezone)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(Trigger {
|
||||
id: parent.id.into(),
|
||||
app_id: parent.app_id.into(),
|
||||
script_id: parent.script_id.into(),
|
||||
kind: TriggerKind::Cron,
|
||||
enabled: parent.enabled,
|
||||
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
|
||||
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
|
||||
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
|
||||
.unwrap_or(BackoffShape::Exponential),
|
||||
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
|
||||
registered_by_principal: parent.registered_by_principal.into(),
|
||||
created_at: parent.created_at,
|
||||
updated_at: parent.updated_at,
|
||||
details: TriggerDetails::Cron {
|
||||
schedule: req.schedule,
|
||||
timezone: req.timezone,
|
||||
last_fired_at: None,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
async fn create_files_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateFilesTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
if req.collection_glob.is_empty() {
|
||||
return Err(TriggerRepoError::Invalid(
|
||||
"collection_glob must not be empty".into(),
|
||||
));
|
||||
}
|
||||
let mut tx = self.pool.begin().await?;
|
||||
let parent: TriggerRow = sqlx::query_as(
|
||||
"INSERT INTO triggers ( \
|
||||
app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal \
|
||||
) VALUES ($1, $2, 'files', TRUE, $3, $4, $5, $6, $7) \
|
||||
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal, created_at, updated_at",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(req.script_id.into_inner())
|
||||
.bind(req.dispatch_mode.as_str())
|
||||
.bind(i32::try_from(req.retry_max_attempts).unwrap_or(3))
|
||||
.bind(req.retry_backoff.as_str())
|
||||
.bind(i32::try_from(req.retry_base_ms).unwrap_or(1000))
|
||||
.bind(req.registered_by_principal.into_inner())
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
|
||||
let ops_str: Vec<String> = req.ops.iter().map(|o| o.as_str().to_string()).collect();
|
||||
sqlx::query(
|
||||
"INSERT INTO files_trigger_details (trigger_id, collection_glob, ops) \
|
||||
VALUES ($1, $2, $3)",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.bind(&req.collection_glob)
|
||||
.bind(&ops_str)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(Trigger {
|
||||
id: parent.id.into(),
|
||||
app_id: parent.app_id.into(),
|
||||
script_id: parent.script_id.into(),
|
||||
kind: TriggerKind::Files,
|
||||
enabled: parent.enabled,
|
||||
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
|
||||
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
|
||||
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
|
||||
.unwrap_or(BackoffShape::Exponential),
|
||||
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
|
||||
registered_by_principal: parent.registered_by_principal.into(),
|
||||
created_at: parent.created_at,
|
||||
updated_at: parent.updated_at,
|
||||
details: TriggerDetails::Files {
|
||||
collection_glob: req.collection_glob,
|
||||
ops: req.ops,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
async fn create_pubsub_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreatePubsubTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
// Defense-in-depth validation (the admin endpoint validates too).
|
||||
picloud_shared::validate_topic_pattern(&req.topic_pattern)
|
||||
.map_err(TriggerRepoError::Invalid)?;
|
||||
|
||||
let mut tx = self.pool.begin().await?;
|
||||
let parent: TriggerRow = sqlx::query_as(
|
||||
"INSERT INTO triggers ( \
|
||||
app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal \
|
||||
) VALUES ($1, $2, 'pubsub', TRUE, $3, $4, $5, $6, $7) \
|
||||
RETURNING id, app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
retry_max_attempts, retry_backoff, retry_base_ms, \
|
||||
registered_by_principal, created_at, updated_at",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.bind(req.script_id.into_inner())
|
||||
.bind(req.dispatch_mode.as_str())
|
||||
.bind(i32::try_from(req.retry_max_attempts).unwrap_or(3))
|
||||
.bind(req.retry_backoff.as_str())
|
||||
.bind(i32::try_from(req.retry_base_ms).unwrap_or(1000))
|
||||
.bind(req.registered_by_principal.into_inner())
|
||||
.fetch_one(&mut *tx)
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO pubsub_trigger_details (trigger_id, topic_pattern) VALUES ($1, $2)",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.bind(&req.topic_pattern)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(Trigger {
|
||||
id: parent.id.into(),
|
||||
app_id: parent.app_id.into(),
|
||||
script_id: parent.script_id.into(),
|
||||
kind: TriggerKind::Pubsub,
|
||||
enabled: parent.enabled,
|
||||
dispatch_mode: dispatch_from_str(&parent.dispatch_mode),
|
||||
retry_max_attempts: u32::try_from(parent.retry_max_attempts).unwrap_or(3),
|
||||
retry_backoff: BackoffShape::from_wire(&parent.retry_backoff)
|
||||
.unwrap_or(BackoffShape::Exponential),
|
||||
retry_base_ms: u32::try_from(parent.retry_base_ms).unwrap_or(1000),
|
||||
registered_by_principal: parent.registered_by_principal.into(),
|
||||
created_at: parent.created_at,
|
||||
updated_at: parent.updated_at,
|
||||
details: TriggerDetails::Pubsub {
|
||||
topic_pattern: req.topic_pattern,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> {
|
||||
let parents: Vec<TriggerRow> = sqlx::query_as(
|
||||
"SELECT id, app_id, script_id, kind, enabled, dispatch_mode, \
|
||||
@@ -591,6 +899,51 @@ impl TriggerRepo for PostgresTriggerRepo {
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
async fn list_matching_files(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
collection: &str,
|
||||
op: FilesEventOp,
|
||||
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError> {
|
||||
// Mirrors list_matching_kv: pull every enabled files trigger,
|
||||
// filter glob + ops in Rust (empty ops array means "any op").
|
||||
let rows: Vec<KvMatchRow> = sqlx::query_as(
|
||||
"SELECT t.id, t.script_id, t.dispatch_mode, \
|
||||
t.retry_max_attempts, t.retry_backoff, t.retry_base_ms, \
|
||||
t.registered_by_principal, \
|
||||
d.collection_glob, d.ops \
|
||||
FROM triggers t \
|
||||
JOIN files_trigger_details d ON d.trigger_id = t.id \
|
||||
WHERE t.app_id = $1 AND t.kind = 'files' AND t.enabled = TRUE",
|
||||
)
|
||||
.bind(app_id.into_inner())
|
||||
.fetch_all(&self.pool)
|
||||
.await?;
|
||||
|
||||
let op_str = op.as_str();
|
||||
let mut out = Vec::new();
|
||||
for r in rows {
|
||||
if !collection_matches(&r.collection_glob, collection) {
|
||||
continue;
|
||||
}
|
||||
let any_op = r.ops.is_empty();
|
||||
if !any_op && !r.ops.iter().any(|o| o == op_str) {
|
||||
continue;
|
||||
}
|
||||
out.push(FilesTriggerMatch {
|
||||
trigger_id: r.id.into(),
|
||||
script_id: r.script_id.into(),
|
||||
dispatch_mode: dispatch_from_str(&r.dispatch_mode),
|
||||
retry_max_attempts: u32::try_from(r.retry_max_attempts).unwrap_or(3),
|
||||
retry_backoff: BackoffShape::from_wire(&r.retry_backoff)
|
||||
.unwrap_or(BackoffShape::Exponential),
|
||||
retry_base_ms: u32::try_from(r.retry_base_ms).unwrap_or(1000),
|
||||
registered_by_principal: r.registered_by_principal.into(),
|
||||
});
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
async fn list_matching_dead_letter(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
@@ -627,6 +980,7 @@ impl TriggerRepo for PostgresTriggerRepo {
|
||||
}
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_lines)]
|
||||
async fn hydrate_one(pool: &PgPool, parent: TriggerRow) -> Result<Trigger, TriggerRepoError> {
|
||||
let kind = TriggerKind::from_wire(&parent.kind).ok_or_else(|| {
|
||||
TriggerRepoError::Invalid(format!("unknown trigger kind {}", parent.kind))
|
||||
@@ -681,6 +1035,48 @@ async fn hydrate_one(pool: &PgPool, parent: TriggerRow) -> Result<Trigger, Trigg
|
||||
script_id_filter: row.script_id_filter.map(Into::into),
|
||||
}
|
||||
}
|
||||
TriggerKind::Cron => {
|
||||
let row: CronDetailRow = sqlx::query_as(
|
||||
"SELECT schedule, timezone, last_fired_at \
|
||||
FROM cron_trigger_details WHERE trigger_id = $1",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
TriggerDetails::Cron {
|
||||
schedule: row.schedule,
|
||||
timezone: row.timezone,
|
||||
last_fired_at: row.last_fired_at,
|
||||
}
|
||||
}
|
||||
TriggerKind::Files => {
|
||||
let row: KvDetailRow = sqlx::query_as(
|
||||
"SELECT collection_glob, ops FROM files_trigger_details WHERE trigger_id = $1",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
let ops = row
|
||||
.ops
|
||||
.iter()
|
||||
.filter_map(|s| FilesEventOp::from_wire(s))
|
||||
.collect();
|
||||
TriggerDetails::Files {
|
||||
collection_glob: row.collection_glob,
|
||||
ops,
|
||||
}
|
||||
}
|
||||
TriggerKind::Pubsub => {
|
||||
let row: PubsubDetailRow = sqlx::query_as(
|
||||
"SELECT topic_pattern FROM pubsub_trigger_details WHERE trigger_id = $1",
|
||||
)
|
||||
.bind(parent.id)
|
||||
.fetch_one(pool)
|
||||
.await?;
|
||||
TriggerDetails::Pubsub {
|
||||
topic_pattern: row.topic_pattern,
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
Ok(Trigger {
|
||||
@@ -746,6 +1142,18 @@ struct KvDetailRow {
|
||||
ops: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct CronDetailRow {
|
||||
schedule: String,
|
||||
timezone: String,
|
||||
last_fired_at: Option<DateTime<Utc>>,
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct PubsubDetailRow {
|
||||
topic_pattern: String,
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
#[allow(clippy::struct_field_names)]
|
||||
struct DlDetailRow {
|
||||
|
||||
@@ -16,7 +16,9 @@ use axum::http::StatusCode;
|
||||
use axum::response::{IntoResponse, Json, Response};
|
||||
use axum::routing::{delete, get, post};
|
||||
use axum::{Extension, Router};
|
||||
use picloud_shared::{AppId, DocsEventOp, KvEventOp, Principal, ScriptId, ScriptKind, TriggerId};
|
||||
use picloud_shared::{
|
||||
AppId, DocsEventOp, FilesEventOp, KvEventOp, Principal, ScriptId, ScriptKind, TriggerId,
|
||||
};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::json;
|
||||
|
||||
@@ -25,8 +27,9 @@ use crate::authz::{require, AuthzDenied, AuthzError, AuthzRepo, Capability};
|
||||
use crate::repo::{ScriptRepository, ScriptRepositoryError};
|
||||
use crate::trigger_config::{BackoffShape, TriggerConfig};
|
||||
use crate::trigger_repo::{
|
||||
CreateDeadLetterTrigger, CreateDocsTrigger, CreateKvTrigger, Trigger, TriggerDispatchMode,
|
||||
TriggerRepo, TriggerRepoError,
|
||||
CreateCronTrigger, CreateDeadLetterTrigger, CreateDocsTrigger, CreateFilesTrigger,
|
||||
CreateKvTrigger, CreatePubsubTrigger, Trigger, TriggerDispatchMode, TriggerRepo,
|
||||
TriggerRepoError,
|
||||
};
|
||||
|
||||
#[derive(Clone)]
|
||||
@@ -53,6 +56,12 @@ pub fn triggers_router(state: TriggersState) -> Router {
|
||||
)
|
||||
.route("/apps/{app_id}/triggers/kv", post(create_kv_trigger))
|
||||
.route("/apps/{app_id}/triggers/docs", post(create_docs_trigger))
|
||||
.route("/apps/{app_id}/triggers/cron", post(create_cron_trigger))
|
||||
.route("/apps/{app_id}/triggers/files", post(create_files_trigger))
|
||||
.route(
|
||||
"/apps/{app_id}/triggers/pubsub",
|
||||
post(create_pubsub_trigger),
|
||||
)
|
||||
.route(
|
||||
"/apps/{app_id}/triggers/dead_letter",
|
||||
post(create_dl_trigger),
|
||||
@@ -116,6 +125,46 @@ pub struct CreateDocsTriggerRequest {
|
||||
pub retry_base_ms: Option<u32>,
|
||||
}
|
||||
|
||||
/// v1.1.4 cron trigger. `schedule` is a 6-field cron expression (with
|
||||
/// seconds); `timezone` is an IANA name (defaults to UTC if omitted).
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreateCronTriggerRequest {
|
||||
pub script_id: ScriptId,
|
||||
pub schedule: String,
|
||||
#[serde(default = "default_timezone")]
|
||||
pub timezone: String,
|
||||
#[serde(default = "default_dispatch")]
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
#[serde(default)]
|
||||
pub retry_max_attempts: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub retry_backoff: Option<BackoffShape>,
|
||||
#[serde(default)]
|
||||
pub retry_base_ms: Option<u32>,
|
||||
}
|
||||
|
||||
fn default_timezone() -> String {
|
||||
"UTC".to_string()
|
||||
}
|
||||
|
||||
/// v1.1.5 files trigger. Mirrors `CreateKvTriggerRequest`; `ops` uses
|
||||
/// `FilesEventOp` (`create` / `update` / `delete`).
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreateFilesTriggerRequest {
|
||||
pub script_id: ScriptId,
|
||||
pub collection_glob: String,
|
||||
#[serde(default)]
|
||||
pub ops: Vec<FilesEventOp>,
|
||||
#[serde(default = "default_dispatch")]
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
#[serde(default)]
|
||||
pub retry_max_attempts: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub retry_backoff: Option<BackoffShape>,
|
||||
#[serde(default)]
|
||||
pub retry_base_ms: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreateDeadLetterTriggerRequest {
|
||||
pub script_id: ScriptId,
|
||||
@@ -264,6 +313,135 @@ async fn create_docs_trigger(
|
||||
Ok((StatusCode::CREATED, Json(created)))
|
||||
}
|
||||
|
||||
async fn create_cron_trigger(
|
||||
State(s): State<TriggersState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
Json(input): Json<CreateCronTriggerRequest>,
|
||||
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppManageTriggers(app_id),
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Validate the schedule + timezone before touching the script repo
|
||||
// so a bad expression fails fast with a clear 422.
|
||||
crate::cron_scheduler::validate_schedule(&input.schedule)
|
||||
.map_err(|e| TriggersApiError::Invalid(format!("invalid cron schedule: {e}")))?;
|
||||
crate::cron_scheduler::validate_timezone(&input.timezone)
|
||||
.map_err(|e| TriggersApiError::Invalid(format!("invalid timezone: {e}")))?;
|
||||
|
||||
// v1.1.3 check: target script exists, lives in this app, is an
|
||||
// endpoint (not a module).
|
||||
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
|
||||
|
||||
let req = CreateCronTrigger {
|
||||
script_id: input.script_id,
|
||||
schedule: input.schedule,
|
||||
timezone: input.timezone,
|
||||
dispatch_mode: input.dispatch_mode,
|
||||
retry_max_attempts: input
|
||||
.retry_max_attempts
|
||||
.unwrap_or(s.config.retry_max_attempts),
|
||||
retry_backoff: input.retry_backoff.unwrap_or(s.config.retry_backoff),
|
||||
retry_base_ms: input.retry_base_ms.unwrap_or(s.config.retry_base_ms),
|
||||
registered_by_principal: principal.user_id,
|
||||
};
|
||||
let created = s.triggers.create_cron_trigger(app_id, req).await?;
|
||||
Ok((StatusCode::CREATED, Json(created)))
|
||||
}
|
||||
|
||||
/// v1.1.5 pubsub trigger. `topic_pattern` is validated to be exact /
|
||||
/// `<prefix>.*` / `*`.
|
||||
#[derive(Debug, Deserialize)]
|
||||
pub struct CreatePubsubTriggerRequest {
|
||||
pub script_id: ScriptId,
|
||||
pub topic_pattern: String,
|
||||
#[serde(default = "default_dispatch")]
|
||||
pub dispatch_mode: TriggerDispatchMode,
|
||||
#[serde(default)]
|
||||
pub retry_max_attempts: Option<u32>,
|
||||
#[serde(default)]
|
||||
pub retry_backoff: Option<BackoffShape>,
|
||||
#[serde(default)]
|
||||
pub retry_base_ms: Option<u32>,
|
||||
}
|
||||
|
||||
async fn create_pubsub_trigger(
|
||||
State(s): State<TriggersState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
Json(input): Json<CreatePubsubTriggerRequest>,
|
||||
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppManageTriggers(app_id),
|
||||
)
|
||||
.await?;
|
||||
|
||||
// Validate the topic pattern before touching the script repo so a
|
||||
// bad pattern fails fast with a clear 422.
|
||||
picloud_shared::validate_topic_pattern(&input.topic_pattern)
|
||||
.map_err(TriggersApiError::Invalid)?;
|
||||
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
|
||||
|
||||
let req = CreatePubsubTrigger {
|
||||
script_id: input.script_id,
|
||||
topic_pattern: input.topic_pattern,
|
||||
dispatch_mode: input.dispatch_mode,
|
||||
retry_max_attempts: input
|
||||
.retry_max_attempts
|
||||
.unwrap_or(s.config.retry_max_attempts),
|
||||
retry_backoff: input.retry_backoff.unwrap_or(s.config.retry_backoff),
|
||||
retry_base_ms: input.retry_base_ms.unwrap_or(s.config.retry_base_ms),
|
||||
registered_by_principal: principal.user_id,
|
||||
};
|
||||
let created = s.triggers.create_pubsub_trigger(app_id, req).await?;
|
||||
Ok((StatusCode::CREATED, Json(created)))
|
||||
}
|
||||
|
||||
async fn create_files_trigger(
|
||||
State(s): State<TriggersState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
Path(app_id): Path<AppId>,
|
||||
Json(input): Json<CreateFilesTriggerRequest>,
|
||||
) -> Result<(StatusCode, Json<Trigger>), TriggersApiError> {
|
||||
ensure_app_exists(&*s.apps, app_id).await?;
|
||||
require(
|
||||
s.authz.as_ref(),
|
||||
&principal,
|
||||
Capability::AppManageTriggers(app_id),
|
||||
)
|
||||
.await?;
|
||||
|
||||
if input.collection_glob.trim().is_empty() {
|
||||
return Err(TriggersApiError::Invalid(
|
||||
"collection_glob must not be empty".into(),
|
||||
));
|
||||
}
|
||||
validate_trigger_target(&*s.scripts, app_id, input.script_id).await?;
|
||||
|
||||
let req = CreateFilesTrigger {
|
||||
script_id: input.script_id,
|
||||
collection_glob: input.collection_glob,
|
||||
ops: input.ops,
|
||||
dispatch_mode: input.dispatch_mode,
|
||||
retry_max_attempts: input
|
||||
.retry_max_attempts
|
||||
.unwrap_or(s.config.retry_max_attempts),
|
||||
retry_backoff: input.retry_backoff.unwrap_or(s.config.retry_backoff),
|
||||
retry_base_ms: input.retry_base_ms.unwrap_or(s.config.retry_base_ms),
|
||||
registered_by_principal: principal.user_id,
|
||||
};
|
||||
let created = s.triggers.create_files_trigger(app_id, req).await?;
|
||||
Ok((StatusCode::CREATED, Json(created)))
|
||||
}
|
||||
|
||||
async fn create_dl_trigger(
|
||||
State(s): State<TriggersState>,
|
||||
Extension(principal): Extension<Principal>,
|
||||
@@ -420,13 +598,15 @@ mod tests {
|
||||
use super::*;
|
||||
use crate::app_repo::{AppLookup, AppRepository};
|
||||
use crate::trigger_repo::{
|
||||
DeadLetterTriggerMatch, DocsTriggerMatch, KvTriggerMatch, Trigger, TriggerDetails,
|
||||
TriggerRepo, TriggerRepoError,
|
||||
CreateCronTrigger, CreateFilesTrigger, CreatePubsubTrigger, DeadLetterTriggerMatch,
|
||||
DocsTriggerMatch, FilesTriggerMatch, KvTriggerMatch, Trigger, TriggerDetails, TriggerRepo,
|
||||
TriggerRepoError,
|
||||
};
|
||||
use async_trait::async_trait;
|
||||
use chrono::Utc;
|
||||
use picloud_shared::{
|
||||
AdminUserId, App, AppRole, DocsEventOp, KvEventOp, ScriptId, TriggerId, UserId,
|
||||
AdminUserId, App, AppRole, DocsEventOp, FilesEventOp, KvEventOp, ScriptId, TriggerId,
|
||||
UserId,
|
||||
};
|
||||
use std::collections::HashMap;
|
||||
use tokio::sync::Mutex;
|
||||
@@ -523,6 +703,90 @@ mod tests {
|
||||
self.inner.lock().await.insert(id, trigger.clone());
|
||||
Ok(trigger)
|
||||
}
|
||||
async fn create_cron_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateCronTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
let now = Utc::now();
|
||||
let id = TriggerId::new();
|
||||
let trigger = Trigger {
|
||||
id,
|
||||
app_id,
|
||||
script_id: req.script_id,
|
||||
kind: crate::trigger_repo::TriggerKind::Cron,
|
||||
enabled: true,
|
||||
dispatch_mode: req.dispatch_mode,
|
||||
retry_max_attempts: req.retry_max_attempts,
|
||||
retry_backoff: req.retry_backoff,
|
||||
retry_base_ms: req.retry_base_ms,
|
||||
registered_by_principal: req.registered_by_principal,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
details: TriggerDetails::Cron {
|
||||
schedule: req.schedule,
|
||||
timezone: req.timezone,
|
||||
last_fired_at: None,
|
||||
},
|
||||
};
|
||||
self.inner.lock().await.insert(id, trigger.clone());
|
||||
Ok(trigger)
|
||||
}
|
||||
async fn create_files_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreateFilesTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
let now = Utc::now();
|
||||
let id = TriggerId::new();
|
||||
let trigger = Trigger {
|
||||
id,
|
||||
app_id,
|
||||
script_id: req.script_id,
|
||||
kind: crate::trigger_repo::TriggerKind::Files,
|
||||
enabled: true,
|
||||
dispatch_mode: req.dispatch_mode,
|
||||
retry_max_attempts: req.retry_max_attempts,
|
||||
retry_backoff: req.retry_backoff,
|
||||
retry_base_ms: req.retry_base_ms,
|
||||
registered_by_principal: req.registered_by_principal,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
details: TriggerDetails::Files {
|
||||
collection_glob: req.collection_glob,
|
||||
ops: req.ops,
|
||||
},
|
||||
};
|
||||
self.inner.lock().await.insert(id, trigger.clone());
|
||||
Ok(trigger)
|
||||
}
|
||||
async fn create_pubsub_trigger(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
req: CreatePubsubTrigger,
|
||||
) -> Result<Trigger, TriggerRepoError> {
|
||||
let now = Utc::now();
|
||||
let id = TriggerId::new();
|
||||
let trigger = Trigger {
|
||||
id,
|
||||
app_id,
|
||||
script_id: req.script_id,
|
||||
kind: crate::trigger_repo::TriggerKind::Pubsub,
|
||||
enabled: true,
|
||||
dispatch_mode: req.dispatch_mode,
|
||||
retry_max_attempts: req.retry_max_attempts,
|
||||
retry_backoff: req.retry_backoff,
|
||||
retry_base_ms: req.retry_base_ms,
|
||||
registered_by_principal: req.registered_by_principal,
|
||||
created_at: now,
|
||||
updated_at: now,
|
||||
details: TriggerDetails::Pubsub {
|
||||
topic_pattern: req.topic_pattern,
|
||||
},
|
||||
};
|
||||
self.inner.lock().await.insert(id, trigger.clone());
|
||||
Ok(trigger)
|
||||
}
|
||||
async fn list_for_app(&self, app_id: AppId) -> Result<Vec<Trigger>, TriggerRepoError> {
|
||||
Ok(self
|
||||
.inner
|
||||
@@ -555,6 +819,14 @@ mod tests {
|
||||
) -> Result<Vec<DocsTriggerMatch>, TriggerRepoError> {
|
||||
Ok(vec![])
|
||||
}
|
||||
async fn list_matching_files(
|
||||
&self,
|
||||
_app_id: AppId,
|
||||
_collection: &str,
|
||||
_op: FilesEventOp,
|
||||
) -> Result<Vec<FilesTriggerMatch>, TriggerRepoError> {
|
||||
Ok(vec![])
|
||||
}
|
||||
async fn list_matching_dead_letter(
|
||||
&self,
|
||||
_app_id: AppId,
|
||||
@@ -1281,6 +1553,169 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// v1.1.4: cron trigger create.
|
||||
// ----------------------------------------------------------------
|
||||
|
||||
fn cron_req(script_id: ScriptId, schedule: &str, timezone: &str) -> CreateCronTriggerRequest {
|
||||
CreateCronTriggerRequest {
|
||||
script_id,
|
||||
schedule: schedule.into(),
|
||||
timezone: timezone.into(),
|
||||
dispatch_mode: TriggerDispatchMode::Async,
|
||||
retry_max_attempts: None,
|
||||
retry_backoff: None,
|
||||
retry_base_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_create_succeeds() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
let (status, Json(trigger)) = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(cron_req(
|
||||
script_id,
|
||||
"0 0 9 * * MON-FRI",
|
||||
"America/Los_Angeles",
|
||||
)),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(status, StatusCode::CREATED);
|
||||
assert!(matches!(
|
||||
trigger.kind,
|
||||
crate::trigger_repo::TriggerKind::Cron
|
||||
));
|
||||
match trigger.details {
|
||||
TriggerDetails::Cron {
|
||||
schedule,
|
||||
timezone,
|
||||
last_fired_at,
|
||||
} => {
|
||||
assert_eq!(schedule, "0 0 9 * * MON-FRI");
|
||||
assert_eq!(timezone, "America/Los_Angeles");
|
||||
assert!(last_fired_at.is_none());
|
||||
}
|
||||
other => panic!("expected Cron details, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_rejects_invalid_schedule() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
let res = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
// 5-field expression — not the 6-field format we accept.
|
||||
Json(cron_req(script_id, "* * * * *", "UTC")),
|
||||
)
|
||||
.await;
|
||||
let err = res.expect_err("invalid schedule should reject");
|
||||
let msg = match err {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("schedule"), "got {msg}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_rejects_unknown_timezone() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
let res = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(cron_req(script_id, "0 * * * * *", "Mars/Phobos")),
|
||||
)
|
||||
.await;
|
||||
let err = res.expect_err("unknown timezone should reject");
|
||||
let msg = match err {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("timezone"), "got {msg}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_rejects_module_target() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_id),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_module(app_id, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(cron_req(script_id, "0 * * * * *", "UTC")),
|
||||
)
|
||||
.await;
|
||||
let err = res.expect_err("module script should be rejected as cron target");
|
||||
let msg = match err {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("module"), "got {msg}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_rejects_cross_app_script() {
|
||||
// v1.1.3 isolation gap regression: app A cannot target app B's
|
||||
// script via a cron trigger.
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_a),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_a),
|
||||
Json(cron_req(script_id, "0 * * * * *", "UTC")),
|
||||
)
|
||||
.await;
|
||||
let err = res.expect_err("cross-app cron target should reject");
|
||||
let msg = match err {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("does not belong"), "got {msg}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cron_trigger_member_without_role_is_forbidden() {
|
||||
let app_id = AppId::new();
|
||||
let state = state_with(Arc::new(AlwaysDenyAuthzRepo), app_id);
|
||||
let res = create_cron_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(cron_req(ScriptId::new(), "0 * * * * *", "UTC")),
|
||||
)
|
||||
.await;
|
||||
let err = res.expect_err("member without role should be forbidden");
|
||||
assert!(matches!(err, TriggersApiError::Forbidden));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn kv_trigger_accepts_endpoint_target() {
|
||||
let app_id = AppId::new();
|
||||
@@ -1304,4 +1739,258 @@ mod tests {
|
||||
.expect("endpoint target should succeed");
|
||||
assert_eq!(status, StatusCode::CREATED);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------
|
||||
// v1.1.5: files + pubsub trigger create (Layout-E reject coverage).
|
||||
// ----------------------------------------------------------------
|
||||
|
||||
fn files_req(script_id: ScriptId, glob: &str) -> CreateFilesTriggerRequest {
|
||||
CreateFilesTriggerRequest {
|
||||
script_id,
|
||||
collection_glob: glob.into(),
|
||||
ops: vec![FilesEventOp::Create],
|
||||
dispatch_mode: TriggerDispatchMode::Async,
|
||||
retry_max_attempts: None,
|
||||
retry_backoff: None,
|
||||
retry_base_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn files_trigger_create_succeeds() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
let (status, Json(trigger)) = create_files_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(files_req(script_id, "avatars")),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(status, StatusCode::CREATED);
|
||||
assert!(matches!(
|
||||
trigger.kind,
|
||||
crate::trigger_repo::TriggerKind::Files
|
||||
));
|
||||
match trigger.details {
|
||||
TriggerDetails::Files {
|
||||
collection_glob,
|
||||
ops,
|
||||
} => {
|
||||
assert_eq!(collection_glob, "avatars");
|
||||
assert_eq!(ops, vec![FilesEventOp::Create]);
|
||||
}
|
||||
other => panic!("expected Files details, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn files_trigger_empty_glob_rejected() {
|
||||
let app_id = AppId::new();
|
||||
let state = state_with(Arc::new(AlwaysAllowAuthzRepo), app_id);
|
||||
let res = create_files_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(files_req(ScriptId::new(), " ")),
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(
|
||||
res.expect_err("empty glob"),
|
||||
TriggersApiError::Invalid(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn files_trigger_rejects_module_target() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_id),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_module(app_id, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_files_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(files_req(script_id, "avatars")),
|
||||
)
|
||||
.await;
|
||||
let msg = match res.expect_err("module rejected") {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("module"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn files_trigger_rejects_cross_app_script() {
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_a),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_files_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_a),
|
||||
Json(files_req(script_id, "avatars")),
|
||||
)
|
||||
.await;
|
||||
let msg = match res.expect_err("cross-app rejected") {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("does not belong"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn files_trigger_member_without_role_is_forbidden() {
|
||||
let app_id = AppId::new();
|
||||
let state = state_with(Arc::new(AlwaysDenyAuthzRepo), app_id);
|
||||
let res = create_files_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(files_req(ScriptId::new(), "avatars")),
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(
|
||||
res.expect_err("forbidden"),
|
||||
TriggersApiError::Forbidden
|
||||
));
|
||||
}
|
||||
|
||||
fn pubsub_req(script_id: ScriptId, pattern: &str) -> CreatePubsubTriggerRequest {
|
||||
CreatePubsubTriggerRequest {
|
||||
script_id,
|
||||
topic_pattern: pattern.into(),
|
||||
dispatch_mode: TriggerDispatchMode::Async,
|
||||
retry_max_attempts: None,
|
||||
retry_backoff: None,
|
||||
retry_base_ms: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pubsub_trigger_create_succeeds() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
let (status, Json(trigger)) = create_pubsub_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(pubsub_req(script_id, "user.*")),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(status, StatusCode::CREATED);
|
||||
match trigger.details {
|
||||
TriggerDetails::Pubsub { topic_pattern } => assert_eq!(topic_pattern, "user.*"),
|
||||
other => panic!("expected Pubsub details, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pubsub_trigger_rejects_bad_pattern() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = state_with_endpoint(Arc::new(AlwaysAllowAuthzRepo), app_id, script_id);
|
||||
for bad in ["*.created", "a.*.b", "**"] {
|
||||
let res = create_pubsub_trigger(
|
||||
State(state.clone()),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(pubsub_req(script_id, bad)),
|
||||
)
|
||||
.await;
|
||||
let msg = match res.expect_err("bad pattern") {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(
|
||||
msg.contains("unsupported pubsub topic pattern"),
|
||||
"got {msg} for {bad}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pubsub_trigger_rejects_module_target() {
|
||||
let app_id = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_id),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_module(app_id, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_pubsub_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(pubsub_req(script_id, "user.*")),
|
||||
)
|
||||
.await;
|
||||
let msg = match res.expect_err("module rejected") {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("module"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pubsub_trigger_rejects_cross_app_script() {
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let script_id = ScriptId::new();
|
||||
let state = TriggersState {
|
||||
triggers: Arc::new(InMemoryTriggerRepo::default()),
|
||||
apps: InMemoryAppRepo::with(app_a),
|
||||
authz: Arc::new(AlwaysAllowAuthzRepo),
|
||||
scripts: InMemoryScriptRepo::with_endpoint(app_b, script_id),
|
||||
config: TriggerConfig::conservative(),
|
||||
};
|
||||
let res = create_pubsub_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_a),
|
||||
Json(pubsub_req(script_id, "user.*")),
|
||||
)
|
||||
.await;
|
||||
let msg = match res.expect_err("cross-app rejected") {
|
||||
TriggersApiError::Invalid(m) => m,
|
||||
other => panic!("expected Invalid, got {other:?}"),
|
||||
};
|
||||
assert!(msg.to_lowercase().contains("does not belong"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pubsub_trigger_member_without_role_is_forbidden() {
|
||||
let app_id = AppId::new();
|
||||
let state = state_with(Arc::new(AlwaysDenyAuthzRepo), app_id);
|
||||
let res = create_pubsub_trigger(
|
||||
State(state),
|
||||
Extension(member_principal()),
|
||||
Path(app_id),
|
||||
Json(pubsub_req(ScriptId::new(), "user.*")),
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(
|
||||
res.expect_err("forbidden"),
|
||||
TriggersApiError::Forbidden
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,6 +3,16 @@
|
||||
|
||||
## tables
|
||||
|
||||
table: abandoned_executions
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
app_id: uuid NOT NULL
|
||||
outbox_id: uuid NOT NULL
|
||||
script_id: uuid NULL
|
||||
inbox_id: uuid NOT NULL
|
||||
status_code: integer NOT NULL
|
||||
result_summary: text NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: admin_sessions
|
||||
token_hash: text NOT NULL
|
||||
user_id: uuid NOT NULL
|
||||
@@ -48,6 +58,12 @@ table: app_members
|
||||
role: text NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: app_secrets
|
||||
app_id: uuid NOT NULL
|
||||
realtime_signing_key: bytea NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: app_slug_history
|
||||
slug: text NOT NULL
|
||||
current_app_id: uuid NOT NULL
|
||||
@@ -61,6 +77,48 @@ table: apps
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: cron_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
schedule: text NOT NULL
|
||||
timezone: text NOT NULL default='UTC'::text
|
||||
last_fired_at: timestamp with time zone NULL
|
||||
|
||||
table: dead_letter_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
source_filter: text NULL
|
||||
trigger_id_filter: uuid NULL
|
||||
script_id_filter: uuid NULL
|
||||
|
||||
table: dead_letters
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
app_id: uuid NOT NULL
|
||||
original_event_id: uuid NOT NULL
|
||||
source: text NOT NULL
|
||||
op: text NOT NULL
|
||||
trigger_id: uuid NULL
|
||||
script_id: uuid NULL
|
||||
payload: jsonb NOT NULL
|
||||
attempt_count: integer NOT NULL
|
||||
first_attempt_at: timestamp with time zone NOT NULL
|
||||
last_attempt_at: timestamp with time zone NOT NULL
|
||||
last_error: text NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
resolved_at: timestamp with time zone NULL
|
||||
resolution: text NULL
|
||||
|
||||
table: docs
|
||||
app_id: uuid NOT NULL
|
||||
collection: text NOT NULL
|
||||
id: uuid NOT NULL
|
||||
data: jsonb NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: docs_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
collection_glob: text NOT NULL
|
||||
ops: ARRAY NOT NULL
|
||||
|
||||
table: execution_logs
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
script_id: uuid NOT NULL
|
||||
@@ -76,6 +134,56 @@ table: execution_logs
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
app_id: uuid NOT NULL
|
||||
|
||||
table: files
|
||||
app_id: uuid NOT NULL
|
||||
collection: text NOT NULL
|
||||
id: uuid NOT NULL
|
||||
name: text NOT NULL
|
||||
content_type: text NOT NULL
|
||||
size_bytes: bigint NOT NULL
|
||||
checksum_sha256: text NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: files_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
collection_glob: text NOT NULL
|
||||
ops: ARRAY NOT NULL
|
||||
|
||||
table: kv_entries
|
||||
app_id: uuid NOT NULL
|
||||
collection: text NOT NULL
|
||||
key: text NOT NULL
|
||||
value: jsonb NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: kv_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
collection_glob: text NOT NULL
|
||||
ops: ARRAY NOT NULL
|
||||
|
||||
table: outbox
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
app_id: uuid NOT NULL
|
||||
source_kind: text NOT NULL
|
||||
trigger_id: uuid NULL
|
||||
script_id: uuid NULL
|
||||
reply_to: uuid NULL
|
||||
payload: jsonb NOT NULL
|
||||
origin_principal: uuid NULL
|
||||
trigger_depth: integer NOT NULL default=0
|
||||
root_execution_id: uuid NULL
|
||||
attempt_count: integer NOT NULL default=0
|
||||
next_attempt_at: timestamp with time zone NOT NULL default=now()
|
||||
claimed_at: timestamp with time zone NULL
|
||||
claimed_by: text NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: pubsub_trigger_details
|
||||
trigger_id: uuid NOT NULL
|
||||
topic_pattern: text NOT NULL
|
||||
|
||||
table: routes
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
script_id: uuid NOT NULL
|
||||
@@ -87,6 +195,13 @@ table: routes
|
||||
method: text NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
app_id: uuid NOT NULL
|
||||
dispatch_mode: text NOT NULL default='sync'::text
|
||||
|
||||
table: script_imports
|
||||
app_id: uuid NOT NULL
|
||||
importer_script_id: uuid NOT NULL
|
||||
imported_script_id: uuid NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: scripts
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
@@ -100,9 +215,36 @@ table: scripts
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
sandbox: jsonb NOT NULL default='{}'::jsonb
|
||||
app_id: uuid NOT NULL
|
||||
kind: text NOT NULL default='endpoint'::text
|
||||
|
||||
table: topics
|
||||
app_id: uuid NOT NULL
|
||||
name: text NOT NULL
|
||||
external_subscribable: boolean NOT NULL default=false
|
||||
auth_mode: text NOT NULL default='public'::text
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
table: triggers
|
||||
id: uuid NOT NULL default=gen_random_uuid()
|
||||
app_id: uuid NOT NULL
|
||||
script_id: uuid NOT NULL
|
||||
kind: text NOT NULL
|
||||
enabled: boolean NOT NULL default=true
|
||||
dispatch_mode: text NOT NULL default='async'::text
|
||||
retry_max_attempts: integer NOT NULL
|
||||
retry_backoff: text NOT NULL
|
||||
retry_base_ms: integer NOT NULL
|
||||
registered_by_principal: uuid NOT NULL
|
||||
created_at: timestamp with time zone NOT NULL default=now()
|
||||
updated_at: timestamp with time zone NOT NULL default=now()
|
||||
|
||||
## indexes
|
||||
|
||||
indexes on abandoned_executions:
|
||||
abandoned_executions_pkey: public.abandoned_executions USING btree (id)
|
||||
idx_abandoned_executions_gc: public.abandoned_executions USING btree (created_at)
|
||||
|
||||
indexes on admin_sessions:
|
||||
admin_sessions_expiry_idx: public.admin_sessions USING btree (expires_at)
|
||||
admin_sessions_pkey: public.admin_sessions USING btree (token_hash)
|
||||
@@ -128,6 +270,9 @@ indexes on app_members:
|
||||
app_members_pkey: public.app_members USING btree (app_id, user_id)
|
||||
app_members_user_id_idx: public.app_members USING btree (user_id)
|
||||
|
||||
indexes on app_secrets:
|
||||
app_secrets_pkey: public.app_secrets USING btree (app_id)
|
||||
|
||||
indexes on app_slug_history:
|
||||
app_slug_history_pkey: public.app_slug_history USING btree (slug)
|
||||
|
||||
@@ -135,11 +280,53 @@ indexes on apps:
|
||||
apps_pkey: public.apps USING btree (id)
|
||||
apps_slug_key: public.apps USING btree (slug)
|
||||
|
||||
indexes on cron_trigger_details:
|
||||
cron_trigger_details_pkey: public.cron_trigger_details USING btree (trigger_id)
|
||||
idx_cron_triggers_due: public.cron_trigger_details USING btree (last_fired_at)
|
||||
|
||||
indexes on dead_letter_trigger_details:
|
||||
dead_letter_trigger_details_pkey: public.dead_letter_trigger_details USING btree (trigger_id)
|
||||
|
||||
indexes on dead_letters:
|
||||
dead_letters_pkey: public.dead_letters USING btree (id)
|
||||
idx_dead_letters_app_unresolved: public.dead_letters USING btree (app_id) WHERE (resolved_at IS NULL)
|
||||
idx_dead_letters_gc: public.dead_letters USING btree (created_at)
|
||||
|
||||
indexes on docs:
|
||||
docs_pkey: public.docs USING btree (app_id, collection, id)
|
||||
idx_docs_app_collection: public.docs USING btree (app_id, collection)
|
||||
idx_docs_data_gin: public.docs USING gin (data jsonb_path_ops)
|
||||
|
||||
indexes on docs_trigger_details:
|
||||
docs_trigger_details_pkey: public.docs_trigger_details USING btree (trigger_id)
|
||||
|
||||
indexes on execution_logs:
|
||||
execution_logs_app_id_created_at_idx: public.execution_logs USING btree (app_id, created_at DESC)
|
||||
execution_logs_pkey: public.execution_logs USING btree (id)
|
||||
execution_logs_script_id_created_at_idx: public.execution_logs USING btree (script_id, created_at DESC)
|
||||
|
||||
indexes on files:
|
||||
files_pkey: public.files USING btree (app_id, collection, id)
|
||||
idx_files_app_collection: public.files USING btree (app_id, collection)
|
||||
|
||||
indexes on files_trigger_details:
|
||||
files_trigger_details_pkey: public.files_trigger_details USING btree (trigger_id)
|
||||
|
||||
indexes on kv_entries:
|
||||
idx_kv_entries_app_collection: public.kv_entries USING btree (app_id, collection)
|
||||
kv_entries_pkey: public.kv_entries USING btree (app_id, collection, key)
|
||||
|
||||
indexes on kv_trigger_details:
|
||||
kv_trigger_details_pkey: public.kv_trigger_details USING btree (trigger_id)
|
||||
|
||||
indexes on outbox:
|
||||
idx_outbox_app: public.outbox USING btree (app_id)
|
||||
idx_outbox_due: public.outbox USING btree (next_attempt_at) WHERE (claimed_at IS NULL)
|
||||
outbox_pkey: public.outbox USING btree (id)
|
||||
|
||||
indexes on pubsub_trigger_details:
|
||||
pubsub_trigger_details_pkey: public.pubsub_trigger_details USING btree (trigger_id)
|
||||
|
||||
indexes on routes:
|
||||
routes_app_id_idx: public.routes USING btree (app_id)
|
||||
routes_lookup_idx: public.routes USING btree (host_kind, host)
|
||||
@@ -147,13 +334,31 @@ indexes on routes:
|
||||
routes_script_id_idx: public.routes USING btree (script_id)
|
||||
routes_unique_binding_idx: public.routes USING btree (app_id, host_kind, host, path_kind, path, COALESCE(method, ''::text))
|
||||
|
||||
indexes on script_imports:
|
||||
idx_script_imports_app: public.script_imports USING btree (app_id)
|
||||
idx_script_imports_imported: public.script_imports USING btree (imported_script_id)
|
||||
script_imports_pkey: public.script_imports USING btree (importer_script_id, imported_script_id)
|
||||
|
||||
indexes on scripts:
|
||||
idx_scripts_app_kind: public.scripts USING btree (app_id, kind)
|
||||
scripts_app_id_idx: public.scripts USING btree (app_id)
|
||||
scripts_name_uidx: public.scripts USING btree (app_id, lower(name))
|
||||
scripts_pkey: public.scripts USING btree (id)
|
||||
|
||||
indexes on topics:
|
||||
topics_pkey: public.topics USING btree (app_id, name)
|
||||
|
||||
indexes on triggers:
|
||||
idx_triggers_app_kind_enabled: public.triggers USING btree (app_id, kind) WHERE (enabled = true)
|
||||
idx_triggers_app_pubsub_enabled: public.triggers USING btree (app_id, kind) WHERE ((enabled = true) AND (kind = 'pubsub'::text))
|
||||
triggers_pkey: public.triggers USING btree (id)
|
||||
|
||||
## constraints
|
||||
|
||||
constraints on abandoned_executions:
|
||||
[FOREIGN KEY] abandoned_executions_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] abandoned_executions_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on admin_sessions:
|
||||
[FOREIGN KEY] admin_sessions_user_id_fkey: FOREIGN KEY (user_id) REFERENCES admin_users(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] admin_sessions_pkey: PRIMARY KEY (token_hash)
|
||||
@@ -181,6 +386,10 @@ constraints on app_members:
|
||||
[FOREIGN KEY] app_members_user_id_fkey: FOREIGN KEY (user_id) REFERENCES admin_users(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] app_members_pkey: PRIMARY KEY (app_id, user_id)
|
||||
|
||||
constraints on app_secrets:
|
||||
[FOREIGN KEY] app_secrets_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] app_secrets_pkey: PRIMARY KEY (app_id)
|
||||
|
||||
constraints on app_slug_history:
|
||||
[FOREIGN KEY] app_slug_history_current_app_id_fkey: FOREIGN KEY (current_app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] app_slug_history_pkey: PRIMARY KEY (slug)
|
||||
@@ -189,25 +398,94 @@ constraints on apps:
|
||||
[PRIMARY KEY] apps_pkey: PRIMARY KEY (id)
|
||||
[UNIQUE] apps_slug_key: UNIQUE (slug)
|
||||
|
||||
constraints on cron_trigger_details:
|
||||
[FOREIGN KEY] cron_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] cron_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on dead_letter_trigger_details:
|
||||
[FOREIGN KEY] dead_letter_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] dead_letter_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on dead_letters:
|
||||
[CHECK] dead_letters_resolution_check: CHECK ((resolution = ANY (ARRAY['replayed'::text, 'ignored'::text, 'handled_by_script'::text, 'handler_failed'::text])))
|
||||
[FOREIGN KEY] dead_letters_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] dead_letters_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on docs:
|
||||
[FOREIGN KEY] docs_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] docs_pkey: PRIMARY KEY (app_id, collection, id)
|
||||
|
||||
constraints on docs_trigger_details:
|
||||
[FOREIGN KEY] docs_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] docs_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on execution_logs:
|
||||
[CHECK] execution_logs_status_check: CHECK ((status = ANY (ARRAY['success'::text, 'error'::text, 'timeout'::text, 'budget_exceeded'::text])))
|
||||
[FOREIGN KEY] execution_logs_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] execution_logs_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] execution_logs_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on files:
|
||||
[FOREIGN KEY] files_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] files_pkey: PRIMARY KEY (app_id, collection, id)
|
||||
|
||||
constraints on files_trigger_details:
|
||||
[FOREIGN KEY] files_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] files_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on kv_entries:
|
||||
[FOREIGN KEY] kv_entries_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] kv_entries_pkey: PRIMARY KEY (app_id, collection, key)
|
||||
|
||||
constraints on kv_trigger_details:
|
||||
[FOREIGN KEY] kv_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] kv_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on outbox:
|
||||
[CHECK] outbox_source_kind_check: CHECK ((source_kind = ANY (ARRAY['http'::text, 'kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text, 'files'::text, 'pubsub'::text])))
|
||||
[FOREIGN KEY] outbox_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] outbox_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on pubsub_trigger_details:
|
||||
[FOREIGN KEY] pubsub_trigger_details_trigger_id_fkey: FOREIGN KEY (trigger_id) REFERENCES triggers(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] pubsub_trigger_details_pkey: PRIMARY KEY (trigger_id)
|
||||
|
||||
constraints on routes:
|
||||
[CHECK] routes_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text])))
|
||||
[CHECK] routes_host_kind_check: CHECK ((host_kind = ANY (ARRAY['any'::text, 'strict'::text, 'wildcard'::text])))
|
||||
[CHECK] routes_path_kind_check: CHECK ((path_kind = ANY (ARRAY['exact'::text, 'prefix'::text, 'param'::text])))
|
||||
[FOREIGN KEY] routes_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] routes_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] routes_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on script_imports:
|
||||
[FOREIGN KEY] script_imports_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] script_imports_imported_script_id_fkey: FOREIGN KEY (imported_script_id) REFERENCES scripts(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] script_imports_importer_script_id_fkey: FOREIGN KEY (importer_script_id) REFERENCES scripts(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] script_imports_pkey: PRIMARY KEY (importer_script_id, imported_script_id)
|
||||
|
||||
constraints on scripts:
|
||||
[CHECK] scripts_kind_check: CHECK ((kind = ANY (ARRAY['endpoint'::text, 'module'::text])))
|
||||
[CHECK] scripts_memory_limit_mb_check: CHECK (((memory_limit_mb > 0) AND (memory_limit_mb <= 2048)))
|
||||
[CHECK] scripts_module_name_shape: CHECK (((kind <> 'module'::text) OR (name ~ '^[a-zA-Z_][a-zA-Z0-9_]{0,63}$'::text)))
|
||||
[CHECK] scripts_timeout_seconds_check: CHECK (((timeout_seconds > 0) AND (timeout_seconds <= 300)))
|
||||
[FOREIGN KEY] scripts_app_id_fk: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE RESTRICT
|
||||
[PRIMARY KEY] scripts_pkey: PRIMARY KEY (id)
|
||||
|
||||
constraints on topics:
|
||||
[CHECK] topics_auth_mode_check: CHECK ((auth_mode = ANY (ARRAY['public'::text, 'token'::text])))
|
||||
[FOREIGN KEY] topics_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] topics_pkey: PRIMARY KEY (app_id, name)
|
||||
|
||||
constraints on triggers:
|
||||
[CHECK] triggers_dispatch_mode_check: CHECK ((dispatch_mode = ANY (ARRAY['sync'::text, 'async'::text])))
|
||||
[CHECK] triggers_kind_check: CHECK ((kind = ANY (ARRAY['kv'::text, 'dead_letter'::text, 'docs'::text, 'cron'::text, 'files'::text, 'pubsub'::text])))
|
||||
[CHECK] triggers_retry_backoff_check: CHECK ((retry_backoff = ANY (ARRAY['exponential'::text, 'linear'::text, 'constant'::text])))
|
||||
[FOREIGN KEY] triggers_app_id_fkey: FOREIGN KEY (app_id) REFERENCES apps(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] triggers_registered_by_principal_fkey: FOREIGN KEY (registered_by_principal) REFERENCES admin_users(id) ON DELETE CASCADE
|
||||
[FOREIGN KEY] triggers_script_id_fkey: FOREIGN KEY (script_id) REFERENCES scripts(id) ON DELETE CASCADE
|
||||
[PRIMARY KEY] triggers_pkey: PRIMARY KEY (id)
|
||||
|
||||
## applied migrations
|
||||
0001: init
|
||||
0002: sandbox
|
||||
@@ -215,3 +493,19 @@ constraints on scripts:
|
||||
0004: admin auth
|
||||
0005: apps
|
||||
0006: users authz
|
||||
0007: kv
|
||||
0008: triggers
|
||||
0009: outbox
|
||||
0010: dead letters
|
||||
0011: abandoned executions
|
||||
0012: routes dispatch mode
|
||||
0013: docs
|
||||
0014: docs triggers
|
||||
0015: scripts kind
|
||||
0016: script imports
|
||||
0017: cron triggers
|
||||
0018: files
|
||||
0019: files triggers
|
||||
0020: pubsub triggers
|
||||
0021: topics
|
||||
0022: app secrets
|
||||
|
||||
@@ -25,22 +25,46 @@
|
||||
//!
|
||||
//! Review the resulting diff in the same PR as the new migration.
|
||||
//!
|
||||
//! Like the orchestrator integration tests, this is `#[ignore]`'d by
|
||||
//! default so plain `cargo test --workspace` stays green without
|
||||
//! infrastructure.
|
||||
//! v1.1.5: this test is no longer `#[ignore]`'d. It runs whenever
|
||||
//! `DATABASE_URL` is set (CI wires a `postgres:15` service) and **skips
|
||||
//! cleanly** when it's absent, so plain `cargo test --workspace` stays
|
||||
//! green on machines without Postgres. Unlike the previous
|
||||
//! `#[sqlx::test]` form (which spun up an isolated throwaway database),
|
||||
//! it now applies the migrations against the `DATABASE_URL` database
|
||||
//! directly — migrations are forward-only and idempotent, and CI's
|
||||
//! Postgres is fresh, so the structural dump is identical either way.
|
||||
|
||||
use std::fmt::Write as _;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use sqlx::{PgPool, Row};
|
||||
|
||||
const SCHEMA: &str = "public";
|
||||
|
||||
const SNAPSHOT_PATH: &str = "tests/expected_schema.txt";
|
||||
|
||||
#[ignore = "needs DATABASE_URL pointing at a running Postgres"]
|
||||
#[sqlx::test(migrations = "./migrations")]
|
||||
async fn schema_after_replay_matches_snapshot(pool: PgPool) {
|
||||
#[tokio::test]
|
||||
async fn schema_after_replay_matches_snapshot() {
|
||||
// Skip cleanly when DATABASE_URL is unset so `cargo test --workspace`
|
||||
// stays green without Postgres. CI sets it (postgres:15 service).
|
||||
let Ok(url) = std::env::var("DATABASE_URL") else {
|
||||
eprintln!(
|
||||
"schema_snapshot: DATABASE_URL unset — skipping. Set it (e.g. \
|
||||
postgres://picloud:picloud@localhost:5432/picloud) to run this guardrail."
|
||||
);
|
||||
return;
|
||||
};
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(1)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect to DATABASE_URL");
|
||||
sqlx::migrate!("./migrations")
|
||||
.run(&pool)
|
||||
.await
|
||||
.expect("apply migrations");
|
||||
|
||||
let actual = dump_schema(&pool).await;
|
||||
|
||||
let snapshot_file = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join(SNAPSHOT_PATH);
|
||||
|
||||
@@ -23,8 +23,13 @@ chrono.workspace = true
|
||||
reqwest.workspace = true
|
||||
rhai.workspace = true
|
||||
tokio.workspace = true
|
||||
tokio-stream.workspace = true
|
||||
urlencoding.workspace = true
|
||||
|
||||
# v1.1.3 — top-level script AST cache lives in orchestrator-core's
|
||||
# LocalExecutorClient; key is ScriptId, value is `(updated_at, Arc<rhai::AST>)`.
|
||||
lru.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
# `ServiceExt::oneshot` for driving the SSE router in unit tests.
|
||||
tower.workspace = true
|
||||
|
||||
@@ -12,6 +12,8 @@ pub mod api;
|
||||
pub mod client;
|
||||
pub mod gate;
|
||||
pub mod inbox;
|
||||
pub mod realtime;
|
||||
pub mod realtime_api;
|
||||
pub mod resolver;
|
||||
pub mod routing;
|
||||
|
||||
@@ -19,4 +21,8 @@ pub use api::{data_plane_router, user_routes_router, DataPlaneState};
|
||||
pub use client::{ExecutorClient, LocalExecutorClient, RemoteExecutorClient, ScriptIdentity};
|
||||
pub use gate::{AcquireError, ExecutionGate};
|
||||
pub use inbox::InboxRegistry;
|
||||
pub use realtime::{spawn_realtime_gc, InProcessBroadcaster, DEFAULT_BROADCAST_CAPACITY};
|
||||
pub use realtime_api::{
|
||||
heartbeat_secs_from_env, realtime_router, RealtimeState, DEFAULT_HEARTBEAT_SECS,
|
||||
};
|
||||
pub use resolver::{ResolverError, ScriptResolver};
|
||||
|
||||
242
crates/orchestrator-core/src/realtime.rs
Normal file
242
crates/orchestrator-core/src/realtime.rs
Normal file
@@ -0,0 +1,242 @@
|
||||
//! In-process `RealtimeBroadcaster` — the SSE fan-out registry (v1.1.6).
|
||||
//!
|
||||
//! Sibling of [`crate::inbox::InboxRegistry`], but multi-receiver and
|
||||
//! repeated-event: a `Mutex<HashMap<(AppId, topic), broadcast::Sender>>`
|
||||
//! over `tokio::sync::broadcast` instead of a oneshot map. The publish
|
||||
//! side ([`PubsubServiceImpl`]) and the SSE subscribe side both hold one
|
||||
//! shared `Arc<InProcessBroadcaster>`.
|
||||
//!
|
||||
//! Delivery is best-effort: each channel has a bounded buffer
|
||||
//! (`PICLOUD_REALTIME_BROADCAST_CAPACITY`, default 64); a slow consumer
|
||||
//! that falls behind sees the oldest events dropped (standard
|
||||
//! `broadcast` lag semantics — the receiver gets `RecvError::Lagged`).
|
||||
//! SSE's transport-layer auto-reconnect is the recovery path; there's no
|
||||
//! server-side replay in v1.1.6.
|
||||
//!
|
||||
//! Channels are created lazily on first subscribe. A periodic GC task
|
||||
//! ([`spawn_realtime_gc`]) drops senders whose receiver count has fallen
|
||||
//! to zero so one-shot subscribers don't grow the map unboundedly.
|
||||
//!
|
||||
//! Cluster mode (v1.3+) swaps this for a Postgres `LISTEN/NOTIFY`-backed
|
||||
//! resolver behind the same `RealtimeBroadcaster` trait.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use picloud_shared::{AppId, BroadcasterError, RealtimeBroadcaster, RealtimeEvent};
|
||||
use tokio::sync::broadcast;
|
||||
|
||||
/// Default per-channel broadcast buffer depth.
|
||||
pub const DEFAULT_BROADCAST_CAPACITY: usize = 64;
|
||||
const ENV_CAPACITY: &str = "PICLOUD_REALTIME_BROADCAST_CAPACITY";
|
||||
|
||||
/// Default GC sweep interval for empty channels.
|
||||
pub const DEFAULT_GC_INTERVAL_SECS: u64 = 60;
|
||||
|
||||
pub struct InProcessBroadcaster {
|
||||
inner: Mutex<HashMap<(AppId, String), broadcast::Sender<RealtimeEvent>>>,
|
||||
capacity: usize,
|
||||
}
|
||||
|
||||
impl InProcessBroadcaster {
|
||||
#[must_use]
|
||||
pub fn new(capacity: usize) -> Self {
|
||||
Self {
|
||||
inner: Mutex::new(HashMap::new()),
|
||||
capacity: capacity.max(1),
|
||||
}
|
||||
}
|
||||
|
||||
/// Build from `PICLOUD_REALTIME_BROADCAST_CAPACITY` (default 64).
|
||||
#[must_use]
|
||||
pub fn from_env() -> Self {
|
||||
let capacity = match std::env::var(ENV_CAPACITY) {
|
||||
Err(_) => DEFAULT_BROADCAST_CAPACITY,
|
||||
Ok(v) => match v.parse::<usize>() {
|
||||
Ok(n) if n > 0 => n,
|
||||
Ok(_) => {
|
||||
tracing::warn!(env = ENV_CAPACITY, value = %v, "must be > 0; using default");
|
||||
DEFAULT_BROADCAST_CAPACITY
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(env = ENV_CAPACITY, value = %v, error = %e, "invalid; using default");
|
||||
DEFAULT_BROADCAST_CAPACITY
|
||||
}
|
||||
},
|
||||
};
|
||||
Self::new(capacity)
|
||||
}
|
||||
|
||||
/// Number of live channels in the map (test/observability helper).
|
||||
#[must_use]
|
||||
pub fn channel_count(&self) -> usize {
|
||||
self.inner.lock().map(|g| g.len()).unwrap_or(0)
|
||||
}
|
||||
|
||||
/// Drop senders with zero receivers. Returns how many were removed.
|
||||
/// Called periodically by [`spawn_realtime_gc`].
|
||||
pub fn gc(&self) -> usize {
|
||||
let Ok(mut g) = self.inner.lock() else {
|
||||
return 0;
|
||||
};
|
||||
let before = g.len();
|
||||
g.retain(|_, tx| tx.receiver_count() > 0);
|
||||
before - g.len()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl RealtimeBroadcaster for InProcessBroadcaster {
|
||||
async fn subscribe(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
topic: &str,
|
||||
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
|
||||
let mut g = self
|
||||
.inner
|
||||
.lock()
|
||||
.map_err(|_| BroadcasterError::Unavailable("broadcaster map poisoned".into()))?;
|
||||
let tx = g
|
||||
.entry((app_id, topic.to_string()))
|
||||
.or_insert_with(|| broadcast::channel(self.capacity).0);
|
||||
Ok(tx.subscribe())
|
||||
}
|
||||
|
||||
async fn publish(&self, app_id: AppId, topic: &str, event: RealtimeEvent) {
|
||||
let Ok(g) = self.inner.lock() else {
|
||||
return;
|
||||
};
|
||||
// Only fan out to an existing channel: a topic with no live
|
||||
// subscribers has no sender (publish never creates one). `send`
|
||||
// returns Err iff every receiver has dropped — a benign no-op.
|
||||
if let Some(tx) = g.get(&(app_id, topic.to_string())) {
|
||||
let _ = tx.send(event);
|
||||
}
|
||||
}
|
||||
|
||||
async fn drop_topic(&self, app_id: AppId, topic: &str) {
|
||||
if let Ok(mut g) = self.inner.lock() {
|
||||
// Removing the sender closes the channel; existing receivers
|
||||
// observe `RecvError::Closed` and disconnect cleanly.
|
||||
g.remove(&(app_id, topic.to_string()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Spawn the background GC sweep that drops empty channels every
|
||||
/// `interval_secs` (default [`DEFAULT_GC_INTERVAL_SECS`]). Spawned at
|
||||
/// startup alongside the other housekeeping tasks.
|
||||
pub fn spawn_realtime_gc(broadcaster: Arc<InProcessBroadcaster>, interval_secs: u64) {
|
||||
let period = Duration::from_secs(interval_secs.max(1));
|
||||
tokio::spawn(async move {
|
||||
let mut ticker = tokio::time::interval(period);
|
||||
ticker.tick().await; // skip the immediate first fire
|
||||
loop {
|
||||
ticker.tick().await;
|
||||
let removed = broadcaster.gc();
|
||||
if removed > 0 {
|
||||
tracing::debug!(removed, "realtime broadcaster GC dropped empty channels");
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use chrono::Utc;
|
||||
use serde_json::json;
|
||||
|
||||
fn event(topic: &str, n: i64) -> RealtimeEvent {
|
||||
RealtimeEvent {
|
||||
topic: topic.to_string(),
|
||||
message: json!({ "n": n }),
|
||||
published_at: Utc::now(),
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn multiple_subscribers_each_receive_each_event() {
|
||||
let b = InProcessBroadcaster::new(16);
|
||||
let app = AppId::new();
|
||||
let mut rx1 = b.subscribe(app, "chat").await.unwrap();
|
||||
let mut rx2 = b.subscribe(app, "chat").await.unwrap();
|
||||
|
||||
b.publish(app, "chat", event("chat", 1)).await;
|
||||
b.publish(app, "chat", event("chat", 2)).await;
|
||||
|
||||
for rx in [&mut rx1, &mut rx2] {
|
||||
assert_eq!(rx.recv().await.unwrap().message, json!({ "n": 1 }));
|
||||
assert_eq!(rx.recv().await.unwrap().message, json!({ "n": 2 }));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dropped_subscriber_does_not_leak_after_gc() {
|
||||
let b = InProcessBroadcaster::new(16);
|
||||
let app = AppId::new();
|
||||
let rx = b.subscribe(app, "t").await.unwrap();
|
||||
assert_eq!(b.channel_count(), 1);
|
||||
drop(rx);
|
||||
// GC reclaims the now-empty channel.
|
||||
assert_eq!(b.gc(), 1);
|
||||
assert_eq!(b.channel_count(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn drop_topic_disconnects_existing_subscribers() {
|
||||
let b = InProcessBroadcaster::new(16);
|
||||
let app = AppId::new();
|
||||
let mut rx = b.subscribe(app, "t").await.unwrap();
|
||||
b.drop_topic(app, "t").await;
|
||||
// Sender gone → receiver observes a closed channel.
|
||||
assert!(rx.recv().await.is_err());
|
||||
assert_eq!(b.channel_count(), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn slow_consumer_loses_oldest_events() {
|
||||
// Capacity 2: a consumer that never drains sees the oldest
|
||||
// events dropped (broadcast Lagged semantics).
|
||||
let b = InProcessBroadcaster::new(2);
|
||||
let app = AppId::new();
|
||||
let mut rx = b.subscribe(app, "t").await.unwrap();
|
||||
for i in 0..5 {
|
||||
b.publish(app, "t", event("t", i)).await;
|
||||
}
|
||||
// First recv reports the lag rather than event 0.
|
||||
let first = rx.recv().await;
|
||||
assert!(
|
||||
matches!(first, Err(broadcast::error::RecvError::Lagged(_))),
|
||||
"expected Lagged, got {first:?}"
|
||||
);
|
||||
// Subsequent recvs return the most recent buffered events.
|
||||
let next = rx.recv().await.unwrap();
|
||||
assert_eq!(next.message, json!({ "n": 3 }));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn cross_app_isolation() {
|
||||
let b = InProcessBroadcaster::new(16);
|
||||
let app_a = AppId::new();
|
||||
let app_b = AppId::new();
|
||||
let mut rx_a = b.subscribe(app_a, "shared").await.unwrap();
|
||||
let mut rx_b = b.subscribe(app_b, "shared").await.unwrap();
|
||||
|
||||
b.publish(app_a, "shared", event("shared", 1)).await;
|
||||
// App B's subscriber must not see app A's publish.
|
||||
assert_eq!(rx_a.recv().await.unwrap().message, json!({ "n": 1 }));
|
||||
assert!(rx_b.try_recv().is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn publish_with_no_subscribers_is_noop() {
|
||||
let b = InProcessBroadcaster::new(16);
|
||||
let app = AppId::new();
|
||||
// No subscriber → no sender created → no panic, nothing fanned out.
|
||||
b.publish(app, "ghost", event("ghost", 1)).await;
|
||||
assert_eq!(b.channel_count(), 0);
|
||||
}
|
||||
}
|
||||
408
crates/orchestrator-core/src/realtime_api.rs
Normal file
408
crates/orchestrator-core/src/realtime_api.rs
Normal file
@@ -0,0 +1,408 @@
|
||||
//! SSE realtime endpoint — `GET /realtime/topics/{topic}` (v1.1.6).
|
||||
//!
|
||||
//! This is a data-plane surface, deliberately NOT under `/api/`
|
||||
//! (realtime is its own versioning surface per the path scheme). It is
|
||||
//! merged at the router root by the `picloud` binary alongside
|
||||
//! `/healthz`, `/version`, and the user-route fallback.
|
||||
//!
|
||||
//! Handshake:
|
||||
//! 1. Resolve `Host` → `app_id` (two-phase dispatch). No app → 404.
|
||||
//! 2. Extract the token from `Authorization: Bearer <t>` OR `?token=<t>`
|
||||
//! (EventSource can't set custom headers, so the query form is the
|
||||
//! browser-compatible path).
|
||||
//! 3. Ask the injected [`RealtimeAuthority`]: missing/internal topic →
|
||||
//! 404, bad/absent token on a token-gated topic → 401, otherwise OK.
|
||||
//! 4. Acquire a `broadcast::Receiver` and stream events as SSE until
|
||||
//! the client disconnects (dropping the receiver — the broadcaster
|
||||
//! cleans up on its own).
|
||||
//!
|
||||
//! Heartbeats (`:` comment lines) keep idle proxies from closing the
|
||||
//! connection; interval is `PICLOUD_REALTIME_HEARTBEAT_SEC` (default 30).
|
||||
|
||||
use std::sync::Arc;
|
||||
use std::time::Duration;
|
||||
|
||||
use axum::extract::{Path, Query, State};
|
||||
use axum::http::{HeaderMap, StatusCode};
|
||||
use axum::response::sse::{Event, KeepAlive, Sse};
|
||||
use axum::response::{IntoResponse, Response};
|
||||
use axum::routing::get;
|
||||
use axum::Router;
|
||||
use picloud_shared::{RealtimeAuthority, RealtimeBroadcaster, SubscribeDenied};
|
||||
use serde::Deserialize;
|
||||
use tokio_stream::wrappers::BroadcastStream;
|
||||
use tokio_stream::{Stream, StreamExt};
|
||||
|
||||
use crate::routing::AppDomainTable;
|
||||
|
||||
/// Default heartbeat interval (seconds) for idle SSE connections.
|
||||
pub const DEFAULT_HEARTBEAT_SECS: u64 = 30;
|
||||
const ENV_HEARTBEAT: &str = "PICLOUD_REALTIME_HEARTBEAT_SEC";
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct RealtimeState {
|
||||
/// Host → app_id resolver (shared with the rest of the data plane).
|
||||
pub app_domains: Arc<AppDomainTable>,
|
||||
pub broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
pub authority: Arc<dyn RealtimeAuthority>,
|
||||
pub heartbeat: Duration,
|
||||
}
|
||||
|
||||
impl RealtimeState {
|
||||
#[must_use]
|
||||
pub fn new(
|
||||
app_domains: Arc<AppDomainTable>,
|
||||
broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
authority: Arc<dyn RealtimeAuthority>,
|
||||
) -> Self {
|
||||
Self {
|
||||
app_domains,
|
||||
broadcaster,
|
||||
authority,
|
||||
heartbeat: Duration::from_secs(heartbeat_secs_from_env()),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read `PICLOUD_REALTIME_HEARTBEAT_SEC` (default 30, must be > 0).
|
||||
#[must_use]
|
||||
pub fn heartbeat_secs_from_env() -> u64 {
|
||||
match std::env::var(ENV_HEARTBEAT) {
|
||||
Err(_) => DEFAULT_HEARTBEAT_SECS,
|
||||
Ok(v) => match v.parse::<u64>() {
|
||||
Ok(n) if n > 0 => n,
|
||||
_ => {
|
||||
tracing::warn!(env = ENV_HEARTBEAT, value = %v, "invalid; using default");
|
||||
DEFAULT_HEARTBEAT_SECS
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/// Router for the realtime SSE surface. Merged at the router root.
|
||||
#[must_use]
|
||||
pub fn realtime_router(state: RealtimeState) -> Router {
|
||||
Router::new()
|
||||
.route("/realtime/topics/{topic}", get(sse_topic))
|
||||
.with_state(state)
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct TokenQuery {
|
||||
token: Option<String>,
|
||||
}
|
||||
|
||||
async fn sse_topic(
|
||||
State(state): State<RealtimeState>,
|
||||
Path(topic): Path<String>,
|
||||
Query(q): Query<TokenQuery>,
|
||||
headers: HeaderMap,
|
||||
) -> Response {
|
||||
// 1. Host → app.
|
||||
let host = headers
|
||||
.get("host")
|
||||
.and_then(|h| h.to_str().ok())
|
||||
.unwrap_or("");
|
||||
let Some(app_id) = state.app_domains.resolve_app(host) else {
|
||||
return not_found("no app claims this host");
|
||||
};
|
||||
|
||||
// 2. Token: Authorization: Bearer <t> takes precedence, else ?token=.
|
||||
let token = bearer_token(&headers).or(q.token);
|
||||
|
||||
// 3. Authorize.
|
||||
match state
|
||||
.authority
|
||||
.authorize_subscribe(app_id, &topic, token.as_deref())
|
||||
.await
|
||||
{
|
||||
Ok(()) => {}
|
||||
Err(SubscribeDenied::NotFound) => return not_found("topic not found"),
|
||||
Err(SubscribeDenied::Unauthorized) => return unauthorized(),
|
||||
Err(SubscribeDenied::Backend(e)) => {
|
||||
tracing::error!(error = %e, "realtime authority backend error");
|
||||
return (
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
axum::Json(serde_json::json!({ "error": "internal error" })),
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Subscribe + stream.
|
||||
let rx = match state.broadcaster.subscribe(app_id, &topic).await {
|
||||
Ok(rx) => rx,
|
||||
Err(e) => {
|
||||
tracing::error!(error = %e, "failed to acquire realtime subscription");
|
||||
return (
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
axum::Json(serde_json::json!({ "error": "internal error" })),
|
||||
)
|
||||
.into_response();
|
||||
}
|
||||
};
|
||||
|
||||
let stream = event_stream(rx);
|
||||
let sse =
|
||||
Sse::new(stream).keep_alive(KeepAlive::new().interval(state.heartbeat).text("heartbeat"));
|
||||
|
||||
// Sse sets Content-Type: text/event-stream + Cache-Control: no-cache.
|
||||
// Add X-Accel-Buffering: no so an intermediate nginx doesn't buffer
|
||||
// the stream (ignored by other proxies). Connection management is
|
||||
// hyper's concern (and is hop-by-hop on HTTP/1.1, server-managed on
|
||||
// HTTP/2), so we don't set Connection ourselves.
|
||||
let mut resp = sse.into_response();
|
||||
resp.headers_mut().insert(
|
||||
"X-Accel-Buffering",
|
||||
axum::http::HeaderValue::from_static("no"),
|
||||
);
|
||||
resp
|
||||
}
|
||||
|
||||
/// Map the broadcast receiver into a stream of SSE events. Lagged
|
||||
/// notifications (slow consumer) are skipped; a closed channel
|
||||
/// (`drop_topic`, or all senders gone) ends the stream and the SSE
|
||||
/// connection closes cleanly.
|
||||
fn event_stream(
|
||||
rx: tokio::sync::broadcast::Receiver<picloud_shared::RealtimeEvent>,
|
||||
) -> impl Stream<Item = Result<Event, std::convert::Infallible>> {
|
||||
BroadcastStream::new(rx).filter_map(|item| {
|
||||
let ev = item.ok()?; // drop Lagged errors
|
||||
let payload = serde_json::json!({
|
||||
"topic": ev.topic,
|
||||
"message": ev.message,
|
||||
"published_at": ev.published_at.to_rfc3339(),
|
||||
});
|
||||
Some(Ok(Event::default().data(payload.to_string())))
|
||||
})
|
||||
}
|
||||
|
||||
fn bearer_token(headers: &HeaderMap) -> Option<String> {
|
||||
let raw = headers
|
||||
.get(axum::http::header::AUTHORIZATION)?
|
||||
.to_str()
|
||||
.ok()?;
|
||||
raw.strip_prefix("Bearer ")
|
||||
.map(|t| t.trim().to_string())
|
||||
.filter(|t| !t.is_empty())
|
||||
}
|
||||
|
||||
fn not_found(msg: &str) -> Response {
|
||||
(
|
||||
StatusCode::NOT_FOUND,
|
||||
axum::Json(serde_json::json!({ "error": msg })),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
fn unauthorized() -> Response {
|
||||
// Generic — never leaks which check failed.
|
||||
(
|
||||
StatusCode::UNAUTHORIZED,
|
||||
axum::Json(serde_json::json!({ "error": "unauthorized" })),
|
||||
)
|
||||
.into_response()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::realtime::InProcessBroadcaster;
|
||||
use crate::routing::AppDomainTable;
|
||||
use async_trait::async_trait;
|
||||
use axum::body::Body;
|
||||
use axum::http::Request;
|
||||
use picloud_shared::{AppId, RealtimeEvent};
|
||||
use tower::ServiceExt; // oneshot
|
||||
|
||||
/// Authority stub returning a fixed verdict.
|
||||
struct StubAuthority(Result<(), SubscribeDenied>);
|
||||
#[async_trait]
|
||||
impl RealtimeAuthority for StubAuthority {
|
||||
async fn authorize_subscribe(
|
||||
&self,
|
||||
_: AppId,
|
||||
_: &str,
|
||||
_: Option<&str>,
|
||||
) -> Result<(), SubscribeDenied> {
|
||||
self.0.clone()
|
||||
}
|
||||
}
|
||||
|
||||
/// App-domain table that maps a fixed host to a fixed app.
|
||||
fn domains(host: &str, app: AppId) -> Arc<AppDomainTable> {
|
||||
use crate::routing::{parse_app_domain, CompiledAppDomain};
|
||||
let d = parse_app_domain(host).unwrap();
|
||||
let table = AppDomainTable::new();
|
||||
table.replace(vec![CompiledAppDomain {
|
||||
app_id: app,
|
||||
pattern: d.pattern,
|
||||
shape_key: d.shape_key,
|
||||
}]);
|
||||
Arc::new(table)
|
||||
}
|
||||
|
||||
fn state(
|
||||
app: AppId,
|
||||
host: &str,
|
||||
verdict: Result<(), SubscribeDenied>,
|
||||
broadcaster: Arc<dyn RealtimeBroadcaster>,
|
||||
) -> RealtimeState {
|
||||
RealtimeState {
|
||||
app_domains: domains(host, app),
|
||||
broadcaster,
|
||||
authority: Arc::new(StubAuthority(verdict)),
|
||||
heartbeat: Duration::from_millis(100),
|
||||
}
|
||||
}
|
||||
|
||||
async fn get_status(state: RealtimeState, host: &str, topic: &str) -> StatusCode {
|
||||
let app = realtime_router(state);
|
||||
let req = Request::builder()
|
||||
.uri(format!("/realtime/topics/{topic}"))
|
||||
.header("host", host)
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
app.oneshot(req).await.unwrap().status()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unknown_host_is_404() {
|
||||
let app = AppId::new();
|
||||
let st = state(
|
||||
app,
|
||||
"app.example.com",
|
||||
Ok(()),
|
||||
Arc::new(InProcessBroadcaster::new(8)),
|
||||
);
|
||||
// Request a different host → no app claims it.
|
||||
assert_eq!(
|
||||
get_status(st, "other.example.com", "chat").await,
|
||||
StatusCode::NOT_FOUND
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn not_found_topic_is_404() {
|
||||
let app = AppId::new();
|
||||
let st = state(
|
||||
app,
|
||||
"app.example.com",
|
||||
Err(SubscribeDenied::NotFound),
|
||||
Arc::new(InProcessBroadcaster::new(8)),
|
||||
);
|
||||
assert_eq!(
|
||||
get_status(st, "app.example.com", "ghost").await,
|
||||
StatusCode::NOT_FOUND
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unauthorized_token_is_401() {
|
||||
let app = AppId::new();
|
||||
let st = state(
|
||||
app,
|
||||
"app.example.com",
|
||||
Err(SubscribeDenied::Unauthorized),
|
||||
Arc::new(InProcessBroadcaster::new(8)),
|
||||
);
|
||||
assert_eq!(
|
||||
get_status(st, "app.example.com", "chat").await,
|
||||
StatusCode::UNAUTHORIZED
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn public_topic_returns_event_stream() {
|
||||
let app = AppId::new();
|
||||
let st = state(
|
||||
app,
|
||||
"app.example.com",
|
||||
Ok(()),
|
||||
Arc::new(InProcessBroadcaster::new(8)),
|
||||
);
|
||||
let appr = realtime_router(st);
|
||||
let req = Request::builder()
|
||||
.uri("/realtime/topics/chat")
|
||||
.header("host", "app.example.com")
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
let resp = appr.oneshot(req).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
let ct = resp
|
||||
.headers()
|
||||
.get(axum::http::header::CONTENT_TYPE)
|
||||
.unwrap()
|
||||
.to_str()
|
||||
.unwrap();
|
||||
assert!(ct.starts_with("text/event-stream"));
|
||||
assert_eq!(resp.headers().get("x-accel-buffering").unwrap(), "no");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn subscribe_receives_published_event() {
|
||||
let app = AppId::new();
|
||||
let broadcaster = Arc::new(InProcessBroadcaster::new(8));
|
||||
let st = state(app, "app.example.com", Ok(()), broadcaster.clone());
|
||||
let appr = realtime_router(st);
|
||||
let req = Request::builder()
|
||||
.uri("/realtime/topics/chat")
|
||||
.header("host", "app.example.com")
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
let resp = appr.oneshot(req).await.unwrap();
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
|
||||
// The handler has subscribed; publish and read the first chunk.
|
||||
// Give the streaming task a beat to register its receiver.
|
||||
let mut body = resp.into_body().into_data_stream();
|
||||
tokio::time::sleep(Duration::from_millis(50)).await;
|
||||
broadcaster
|
||||
.publish(
|
||||
app,
|
||||
"chat",
|
||||
RealtimeEvent {
|
||||
topic: "chat".into(),
|
||||
message: serde_json::json!({ "hi": 1 }),
|
||||
published_at: chrono::Utc::now(),
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
let chunk = tokio::time::timeout(Duration::from_secs(2), body.next())
|
||||
.await
|
||||
.expect("a chunk within timeout")
|
||||
.expect("stream item")
|
||||
.expect("chunk ok");
|
||||
let text = String::from_utf8_lossy(&chunk);
|
||||
assert!(text.contains("data:"), "got: {text}");
|
||||
assert!(text.contains("\"hi\":1"), "got: {text}");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn heartbeat_fires_on_idle_connection() {
|
||||
let app = AppId::new();
|
||||
let broadcaster = Arc::new(InProcessBroadcaster::new(8));
|
||||
// Hold a clone so the channel's sender outlives the router (which
|
||||
// oneshot consumes) — otherwise the stream closes immediately.
|
||||
let _keepalive = broadcaster.clone();
|
||||
let st = state(app, "app.example.com", Ok(()), broadcaster);
|
||||
let appr = realtime_router(st);
|
||||
let req = Request::builder()
|
||||
.uri("/realtime/topics/chat")
|
||||
.header("host", "app.example.com")
|
||||
.body(Body::empty())
|
||||
.unwrap();
|
||||
let resp = appr.oneshot(req).await.unwrap();
|
||||
let mut body = resp.into_body().into_data_stream();
|
||||
// No publish — with a 100ms heartbeat, a keep-alive comment must
|
||||
// arrive well within a second.
|
||||
let chunk = tokio::time::timeout(Duration::from_secs(1), body.next())
|
||||
.await
|
||||
.expect("heartbeat within timeout")
|
||||
.expect("stream item")
|
||||
.expect("chunk ok");
|
||||
let text = String::from_utf8_lossy(&chunk);
|
||||
assert!(text.starts_with(':'), "expected SSE comment, got: {text}");
|
||||
}
|
||||
}
|
||||
@@ -11,27 +11,32 @@ use axum::{routing::get, Json, Router};
|
||||
use picloud_executor_core::{Engine, Limits};
|
||||
use picloud_manager_core::{
|
||||
admin_router, admins_router, api_keys_router, app_members_router, apps_api, apps_router,
|
||||
attach_principal_if_present, auth_router, compile_routes, dead_letters_router, migrations,
|
||||
require_authenticated, route_admin_router, triggers_router, AbandonedRepo,
|
||||
AdminPrincipalResolver, AdminSessionRepository, AdminState, AdminUserRepository, AdminsState,
|
||||
ApiKeyRepository, ApiKeysState, AppDomainRepository, AppMembersRepository, AppMembersState,
|
||||
AppRepository, AppsState, AuthState, AuthzRepo, DeadLetterRepo, DeadLettersState, Dispatcher,
|
||||
DocsServiceImpl, KvServiceImpl, OutboxEventEmitter, OutboxRepo, PostgresAbandonedRepo,
|
||||
PostgresAdminSessionRepository, PostgresAdminUserRepository, PostgresApiKeyRepository,
|
||||
PostgresAppDomainRepository, PostgresAppMembersRepository, PostgresAppRepository,
|
||||
PostgresDeadLetterRepo, PostgresDeadLetterService, PostgresDocsRepo,
|
||||
PostgresExecutionLogRepository, PostgresExecutionLogSink, PostgresKvRepo, PostgresOutboxRepo,
|
||||
PostgresRouteRepository, PostgresScriptRepository, PostgresTriggerRepo, PrincipalResolver,
|
||||
RepoResolver, RouteAdminState, RouteRepository, SandboxCeiling, ScriptRepository,
|
||||
TriggerConfig, TriggerRepo, TriggersState,
|
||||
attach_principal_if_present, auth_router, compile_routes, dead_letters_router,
|
||||
files_admin_router, migrations, require_authenticated, route_admin_router, topics_router,
|
||||
triggers_router, AbandonedRepo, AdminPrincipalResolver, AdminSessionRepository, AdminState,
|
||||
AdminUserRepository, AdminsState, ApiKeyRepository, ApiKeysState, AppDomainRepository,
|
||||
AppMembersRepository, AppMembersState, AppRepository, AppsState, AuthState, AuthzRepo,
|
||||
DeadLetterRepo, DeadLettersState, Dispatcher, DocsServiceImpl, FilesAdminState, FilesConfig,
|
||||
FilesServiceImpl, FsFilesRepo, HttpConfig, HttpServiceImpl, KvServiceImpl, OutboxEventEmitter,
|
||||
OutboxRepo, PostgresAbandonedRepo, PostgresAdminSessionRepository, PostgresAdminUserRepository,
|
||||
PostgresApiKeyRepository, PostgresAppDomainRepository, PostgresAppMembersRepository,
|
||||
PostgresAppRepository, PostgresAppSecretsRepo, PostgresDeadLetterRepo,
|
||||
PostgresDeadLetterService, PostgresDocsRepo, PostgresExecutionLogRepository,
|
||||
PostgresExecutionLogSink, PostgresKvRepo, PostgresOutboxRepo, PostgresPubsubRepo,
|
||||
PostgresRouteRepository, PostgresScriptRepository, PostgresTopicRepo, PostgresTriggerRepo,
|
||||
PrincipalResolver, PubsubServiceImpl, RealtimeAuthorityImpl, RepoResolver, RouteAdminState,
|
||||
RouteRepository, SandboxCeiling, ScriptRepository, SubscriberTokenConfig, TopicRepo,
|
||||
TopicsState, TriggerConfig, TriggerRepo, TriggersState,
|
||||
};
|
||||
use picloud_orchestrator_core::realtime::DEFAULT_GC_INTERVAL_SECS;
|
||||
use picloud_orchestrator_core::routing::{AppDomainTable, RouteTable};
|
||||
use picloud_orchestrator_core::{
|
||||
data_plane_router, user_routes_router, DataPlaneState, ExecutionGate, InboxRegistry,
|
||||
LocalExecutorClient,
|
||||
data_plane_router, realtime_router, spawn_realtime_gc, user_routes_router, DataPlaneState,
|
||||
ExecutionGate, InProcessBroadcaster, InboxRegistry, LocalExecutorClient, RealtimeState,
|
||||
};
|
||||
use picloud_shared::{
|
||||
DeadLetterService, DocsService, ExecutionLogSink, InboxResolver, KvService, OutboxWriter,
|
||||
DeadLetterService, DocsService, ExecutionLogSink, FilesService, HttpService, InboxResolver,
|
||||
KvService, OutboxWriter, PubsubService, RealtimeAuthority, RealtimeBroadcaster,
|
||||
ScriptValidator, ServiceEventEmitter, Services, API_VERSION, PRODUCT_VERSION, SDK_VERSION,
|
||||
WIRE_VERSION,
|
||||
};
|
||||
@@ -143,9 +148,71 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
|
||||
outbox_repo.clone(),
|
||||
authz.clone(),
|
||||
));
|
||||
let modules: Arc<dyn picloud_shared::ModuleSource> =
|
||||
Arc::new(picloud_manager_core::PostgresModuleSource::new(pool));
|
||||
let services = Services::new(kv, docs, dl_service.clone(), events, modules);
|
||||
let modules: Arc<dyn picloud_shared::ModuleSource> = Arc::new(
|
||||
picloud_manager_core::PostgresModuleSource::new(pool.clone()),
|
||||
);
|
||||
// v1.1.4 outbound HTTP. The reqwest client is built once here with
|
||||
// the SSRF deny-list resolver. `PICLOUD_HTTP_ALLOW_PRIVATE=true`
|
||||
// disables the deny-list entirely — dev/test only, so warn loudly.
|
||||
let http_config = HttpConfig::from_env();
|
||||
if http_config.allow_private {
|
||||
tracing::warn!(
|
||||
"PICLOUD_HTTP_ALLOW_PRIVATE is set — the outbound-HTTP SSRF deny-list is DISABLED. \
|
||||
Scripts can reach loopback/private/link-local addresses. Do NOT use in production."
|
||||
);
|
||||
}
|
||||
let http: Arc<dyn HttpService> = Arc::new(HttpServiceImpl::new(http_config, authz.clone()));
|
||||
// v1.1.5 filesystem-backed blob storage. Metadata lives in Postgres;
|
||||
// the bytes live on disk under `PICLOUD_FILES_ROOT` (default ./data).
|
||||
let files_config = FilesConfig::from_env();
|
||||
let files_max_size = files_config.max_file_size_bytes;
|
||||
// Kept for the v1.1.6 orphan sweeper (cleans stale `*.tmp.*` files).
|
||||
let files_root = files_config.root.clone();
|
||||
let files_repo = Arc::new(FsFilesRepo::new(pool.clone(), files_config));
|
||||
let files: Arc<dyn FilesService> = Arc::new(FilesServiceImpl::new(
|
||||
files_repo.clone(),
|
||||
authz.clone(),
|
||||
events.clone(),
|
||||
files_max_size,
|
||||
));
|
||||
// v1.1.6 realtime: the in-process broadcaster is shared between the
|
||||
// publish path (PubsubServiceImpl fans out to SSE subscribers after
|
||||
// the durable outbox fan-out) and the SSE endpoint (subscribe side).
|
||||
// The topic registry + app-secrets repo back the subscriber-token
|
||||
// mint + SSE subscribe-authorization.
|
||||
let broadcaster_concrete = Arc::new(InProcessBroadcaster::from_env());
|
||||
let broadcaster: Arc<dyn RealtimeBroadcaster> = broadcaster_concrete.clone();
|
||||
let topic_repo: Arc<dyn TopicRepo> = Arc::new(PostgresTopicRepo::new(pool.clone()));
|
||||
let app_secrets_repo = Arc::new(PostgresAppSecretsRepo::new(pool.clone()));
|
||||
let realtime_authority: Arc<dyn RealtimeAuthority> = Arc::new(RealtimeAuthorityImpl::new(
|
||||
topic_repo.clone(),
|
||||
app_secrets_repo.clone(),
|
||||
));
|
||||
|
||||
// v1.1.5 durable pub/sub, extended in v1.1.6 with the realtime
|
||||
// broadcast + subscriber-token mint. Publishes fan out to matching
|
||||
// pubsub triggers at publish time (one outbox row each, delivered by
|
||||
// the same dispatcher as every other async trigger) AND, best-effort,
|
||||
// to in-process SSE subscribers.
|
||||
let pubsub_repo = Arc::new(PostgresPubsubRepo::new(pool.clone()));
|
||||
let pubsub: Arc<dyn PubsubService> = Arc::new(
|
||||
PubsubServiceImpl::new(pubsub_repo, authz.clone()).with_realtime(
|
||||
broadcaster.clone(),
|
||||
topic_repo.clone(),
|
||||
app_secrets_repo,
|
||||
SubscriberTokenConfig::from_env(),
|
||||
),
|
||||
);
|
||||
let services = Services::new(
|
||||
kv,
|
||||
docs,
|
||||
dl_service.clone(),
|
||||
events,
|
||||
modules,
|
||||
http,
|
||||
files,
|
||||
pubsub,
|
||||
);
|
||||
let engine = Arc::new(Engine::new(Limits::default(), services));
|
||||
|
||||
// Compile the routes table once at startup; admin writes refresh it.
|
||||
@@ -241,6 +308,14 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
|
||||
abandoned_repo.clone(),
|
||||
trigger_config.abandoned_retention_days,
|
||||
);
|
||||
// v1.1.4: cron scheduler. Polls cron_trigger_details on a tick and
|
||||
// enqueues due triggers into the outbox; the dispatcher above
|
||||
// delivers them like any other async trigger.
|
||||
picloud_manager_core::spawn_cron_scheduler(pool, trigger_config.cron_tick_interval_ms);
|
||||
// v1.1.6: GC empty realtime broadcast channels (one-shot subscribers)
|
||||
// and sweep orphaned `*.tmp.*` blobs left by crashed file writes.
|
||||
spawn_realtime_gc(broadcaster_concrete, DEFAULT_GC_INTERVAL_SECS);
|
||||
picloud_manager_core::spawn_files_orphan_sweep(files_root);
|
||||
let triggers_state = TriggersState {
|
||||
triggers: trigger_repo,
|
||||
apps: apps_repo.clone(),
|
||||
@@ -254,11 +329,22 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
|
||||
apps: apps_repo.clone(),
|
||||
authz: authz.clone(),
|
||||
};
|
||||
let files_admin_state = FilesAdminState {
|
||||
files: files_repo,
|
||||
apps: apps_repo.clone(),
|
||||
authz: authz.clone(),
|
||||
};
|
||||
let topics_state = TopicsState {
|
||||
topics: topic_repo,
|
||||
apps: apps_repo.clone(),
|
||||
authz: authz.clone(),
|
||||
broadcaster: broadcaster.clone(),
|
||||
};
|
||||
let apps_state = AppsState {
|
||||
apps: apps_repo,
|
||||
domains: domains_repo,
|
||||
routes: route_repo,
|
||||
domain_table: app_domain_table,
|
||||
domain_table: app_domain_table.clone(),
|
||||
authz: authz.clone(),
|
||||
};
|
||||
|
||||
@@ -296,6 +382,8 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
|
||||
.merge(app_members_router(app_members_state))
|
||||
.merge(api_keys_router(api_keys_state))
|
||||
.merge(triggers_router(triggers_state))
|
||||
.merge(files_admin_router(files_admin_state))
|
||||
.merge(topics_router(topics_state))
|
||||
.merge(dead_letters_router(dead_letters_state))
|
||||
.layer(from_fn_with_state(
|
||||
auth_state.clone(),
|
||||
@@ -326,10 +414,21 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
|
||||
.nest("/admin", guarded_admin)
|
||||
.merge(data_plane_routed);
|
||||
|
||||
// v1.1.6 SSE realtime surface, merged at the root (deliberately NOT
|
||||
// under /api/ — realtime is its own versioning surface). Public auth
|
||||
// is per-topic; no principal middleware (token verification is the
|
||||
// gate, handled inside the authority).
|
||||
let realtime = realtime_router(RealtimeState::new(
|
||||
app_domain_table,
|
||||
broadcaster,
|
||||
realtime_authority,
|
||||
));
|
||||
|
||||
Ok(Router::new()
|
||||
.route("/healthz", get(healthz))
|
||||
.route("/version", get(version))
|
||||
.nest(&format!("/api/v{API_VERSION}"), api_v1)
|
||||
.merge(realtime)
|
||||
.merge(user_routes)
|
||||
.layer(TraceLayer::new_for_http()))
|
||||
}
|
||||
|
||||
353
crates/picloud/tests/dispatcher_e2e.rs
Normal file
353
crates/picloud/tests/dispatcher_e2e.rs
Normal file
@@ -0,0 +1,353 @@
|
||||
//! End-to-end dispatcher tests — one per trigger kind (v1.1.5 follow-up,
|
||||
//! landed in v1.1.6). Each test wires the full all-in-one app via
|
||||
//! `build_app` (which spawns the real dispatcher + cron scheduler +
|
||||
//! executor), creates an app + a logging handler script + a trigger,
|
||||
//! causes the originating event, and polls for the handler's side effect.
|
||||
//!
|
||||
//! ## Gating
|
||||
//!
|
||||
//! These need a Postgres reachable via `DATABASE_URL`. They follow the
|
||||
//! `schema_snapshot` pattern (NOT `#[ignore]`): when `DATABASE_URL` is
|
||||
//! unset the test prints a notice and returns early, so plain
|
||||
//! `cargo test` stays green locally while CI (which sets `DATABASE_URL`)
|
||||
//! runs them.
|
||||
//!
|
||||
//! ## How "the handler fired" is observed
|
||||
//!
|
||||
//! The dispatcher does not write `execution_log` rows for trigger
|
||||
//! handlers, so each handler instead records its `ctx.event` into a KV
|
||||
//! marker (`collection = "e2e_markers"`, which no trigger watches — no
|
||||
//! recursion). The test polls `kv_entries` for that marker and asserts
|
||||
//! the event shape. See HANDBACK §deviations for why this lives in
|
||||
//! `picloud/tests/` rather than `manager-core/tests/` (build_app lives in
|
||||
//! the `picloud` crate) and for the `dead_letter` reinterpretation.
|
||||
|
||||
#![allow(clippy::needless_pass_by_value)]
|
||||
|
||||
use std::time::Duration;
|
||||
|
||||
use axum_test::TestServer;
|
||||
use serde_json::{json, Value};
|
||||
use sqlx::postgres::PgPoolOptions;
|
||||
use sqlx::PgPool;
|
||||
use uuid::Uuid;
|
||||
|
||||
/// Connect + migrate, or return `None` (printing a skip notice) when
|
||||
/// `DATABASE_URL` is unset — mirrors `schema_snapshot.rs`.
|
||||
async fn pool_or_skip() -> Option<PgPool> {
|
||||
let Ok(url) = std::env::var("DATABASE_URL") else {
|
||||
eprintln!("dispatcher_e2e: DATABASE_URL unset — skipping");
|
||||
return None;
|
||||
};
|
||||
let pool = PgPoolOptions::new()
|
||||
.max_connections(5)
|
||||
.connect(&url)
|
||||
.await
|
||||
.expect("connect to DATABASE_URL");
|
||||
sqlx::migrate!("../manager-core/migrations")
|
||||
.run(&pool)
|
||||
.await
|
||||
.expect("apply migrations");
|
||||
Some(pool)
|
||||
}
|
||||
|
||||
/// Build the app over the shared pool with a uniquely-named owner admin,
|
||||
/// log in, and create a fresh app. `suffix` must be unique per test (the
|
||||
/// pool is shared, so names must not collide).
|
||||
async fn server_for(pool: PgPool, suffix: &str) -> (TestServer, String) {
|
||||
use picloud_manager_core::auth::hash_password;
|
||||
use picloud_shared::InstanceRole;
|
||||
|
||||
let unique = format!("{suffix}-{}", Uuid::new_v4().simple());
|
||||
let auth = picloud::AuthDeps::from_pool(pool.clone());
|
||||
let username = format!("e2e-{unique}");
|
||||
let hash = hash_password("pw").expect("hash");
|
||||
auth.users
|
||||
.create(&username, &hash, InstanceRole::Owner, None)
|
||||
.await
|
||||
.expect("seed admin");
|
||||
|
||||
let app = picloud::build_app(pool, auth).await.expect("build_app");
|
||||
let mut server = TestServer::new(app).expect("TestServer");
|
||||
let resp = server
|
||||
.post("/api/v1/admin/auth/login")
|
||||
.json(&json!({ "username": username, "password": "pw" }))
|
||||
.await;
|
||||
resp.assert_status_ok();
|
||||
let token = resp.json::<Value>()["token"]
|
||||
.as_str()
|
||||
.expect("login token")
|
||||
.to_string();
|
||||
server.add_header("authorization", format!("Bearer {token}"));
|
||||
|
||||
// A fresh app keeps each test's KV / events isolated from siblings.
|
||||
let slug = format!("e2e-{unique}");
|
||||
let created: Value = server
|
||||
.post("/api/v1/admin/apps")
|
||||
.json(&json!({ "slug": slug, "name": slug }))
|
||||
.await
|
||||
.json();
|
||||
let app_id = created["id"].as_str().expect("app id").to_string();
|
||||
(server, app_id)
|
||||
}
|
||||
|
||||
async fn create_script(server: &TestServer, app_id: &str, name: &str, source: &str) -> String {
|
||||
let created: Value = server
|
||||
.post("/api/v1/admin/scripts")
|
||||
.json(&json!({ "app_id": app_id, "name": name, "source": source }))
|
||||
.await
|
||||
.json();
|
||||
created["id"].as_str().expect("script id").to_string()
|
||||
}
|
||||
|
||||
/// A handler that records its `ctx.event` into a KV marker the test can
|
||||
/// observe. The marker collection is watched by no trigger.
|
||||
const MARKER_HANDLER: &str = r#"
|
||||
let e = ctx.event;
|
||||
kv::collection("e2e_markers").set("marker", e);
|
||||
#{ ok: true }
|
||||
"#;
|
||||
|
||||
/// Poll the marker KV key until present (or ~10s timeout).
|
||||
async fn poll_marker(pool: &PgPool, app_id: &str) -> Option<Value> {
|
||||
poll_marker_n(pool, app_id, 100).await
|
||||
}
|
||||
|
||||
/// Poll the marker KV key for `iters` × 100ms.
|
||||
async fn poll_marker_n(pool: &PgPool, app_id: &str, iters: u32) -> Option<Value> {
|
||||
let app_uuid = Uuid::parse_str(app_id).expect("app uuid");
|
||||
for _ in 0..iters {
|
||||
let row: Option<(Value,)> = sqlx::query_as(
|
||||
"SELECT value FROM kv_entries \
|
||||
WHERE app_id = $1 AND collection = 'e2e_markers' AND key = 'marker'",
|
||||
)
|
||||
.bind(app_uuid)
|
||||
.fetch_optional(pool)
|
||||
.await
|
||||
.expect("query marker");
|
||||
if let Some((value,)) = row {
|
||||
return Some(value);
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
async fn execute(server: &TestServer, script_id: &str) {
|
||||
server
|
||||
.post(&format!("/api/v1/execute/{script_id}"))
|
||||
.json(&json!({}))
|
||||
.await
|
||||
.assert_status_ok();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_kv_to_handler() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "kv").await;
|
||||
|
||||
let handler = create_script(&server, &app_id, "kv-handler", MARKER_HANDLER).await;
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
|
||||
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
let source = create_script(
|
||||
&server,
|
||||
&app_id,
|
||||
"kv-source",
|
||||
r#"kv::collection("src").set("k", 42); #{ ok: true }"#,
|
||||
)
|
||||
.await;
|
||||
execute(&server, &source).await;
|
||||
|
||||
let event = poll_marker(&pool, &app_id).await.expect("kv handler fired");
|
||||
assert_eq!(event["source"], "kv");
|
||||
assert_eq!(event["op"], "insert");
|
||||
assert_eq!(event["kv"]["collection"], "src");
|
||||
assert_eq!(event["kv"]["key"], "k");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_docs_to_handler() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "docs").await;
|
||||
|
||||
let handler = create_script(&server, &app_id, "docs-handler", MARKER_HANDLER).await;
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/docs"))
|
||||
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
let source = create_script(
|
||||
&server,
|
||||
&app_id,
|
||||
"docs-source",
|
||||
r#"docs::collection("src").create(#{ x: 1 }); #{ ok: true }"#,
|
||||
)
|
||||
.await;
|
||||
execute(&server, &source).await;
|
||||
|
||||
let event = poll_marker(&pool, &app_id)
|
||||
.await
|
||||
.expect("docs handler fired");
|
||||
assert_eq!(event["source"], "docs");
|
||||
assert_eq!(event["op"], "create");
|
||||
assert_eq!(event["docs"]["collection"], "src");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_cron_to_handler() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "cron").await;
|
||||
|
||||
let handler = create_script(&server, &app_id, "cron-handler", MARKER_HANDLER).await;
|
||||
// Fire every second (6-field cron, seconds-resolution).
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/cron"))
|
||||
.json(&json!({ "script_id": handler, "schedule": "* * * * * *", "timezone": "UTC" }))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
// No source — the scheduler enqueues the due tick on its own. The
|
||||
// scheduler skips its first tick and then ticks every
|
||||
// PICLOUD_CRON_TICK_INTERVAL_MS (default 30s), so poll past that
|
||||
// (set the env var lower to speed CI up if desired).
|
||||
let event = poll_marker_n(&pool, &app_id, 450)
|
||||
.await
|
||||
.expect("cron handler fired");
|
||||
assert_eq!(event["source"], "cron");
|
||||
assert_eq!(event["op"], "tick");
|
||||
assert_eq!(event["cron"]["timezone"], "UTC");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_files_to_handler() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "files").await;
|
||||
|
||||
let handler = create_script(&server, &app_id, "files-handler", MARKER_HANDLER).await;
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/files"))
|
||||
.json(&json!({ "script_id": handler, "collection_glob": "src" }))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
let source = create_script(
|
||||
&server,
|
||||
&app_id,
|
||||
"files-source",
|
||||
r#"
|
||||
let data = base64::decode("aGk=");
|
||||
files::collection("src").create(#{ name: "f.txt", content_type: "text/plain", data: data });
|
||||
#{ ok: true }
|
||||
"#,
|
||||
)
|
||||
.await;
|
||||
execute(&server, &source).await;
|
||||
|
||||
let event = poll_marker(&pool, &app_id)
|
||||
.await
|
||||
.expect("files handler fired");
|
||||
assert_eq!(event["source"], "files");
|
||||
assert_eq!(event["op"], "create");
|
||||
assert_eq!(event["files"]["collection"], "src");
|
||||
assert_eq!(event["files"]["name"], "f.txt");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_pubsub_to_handler() {
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "pubsub").await;
|
||||
|
||||
let handler = create_script(&server, &app_id, "pubsub-handler", MARKER_HANDLER).await;
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/pubsub"))
|
||||
.json(&json!({ "script_id": handler, "topic_pattern": "e2e.topic" }))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
let source = create_script(
|
||||
&server,
|
||||
&app_id,
|
||||
"pubsub-source",
|
||||
r#"pubsub::publish_durable("e2e.topic", #{ hello: 1 }); #{ ok: true }"#,
|
||||
)
|
||||
.await;
|
||||
execute(&server, &source).await;
|
||||
|
||||
let event = poll_marker(&pool, &app_id)
|
||||
.await
|
||||
.expect("pubsub handler fired");
|
||||
assert_eq!(event["source"], "pubsub");
|
||||
assert_eq!(event["op"], "publish");
|
||||
assert_eq!(event["pubsub"]["topic"], "e2e.topic");
|
||||
assert_eq!(event["pubsub"]["message"]["hello"], 1);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn dispatcher_delivers_dead_letter_to_handler() {
|
||||
// NOTE: the dead-letter creation path (`dispatcher::handle_failure` →
|
||||
// `DeadLetterRepo::insert`) writes the `dead_letters` row but does not
|
||||
// appear to enqueue deliveries for `dead_letter`-kind triggers
|
||||
// (`TriggerRepo::list_matching_dead_letter` has no production caller —
|
||||
// see HANDBACK latent-findings). So this test asserts the wired
|
||||
// behavior: a failing handler that exhausts its (single) attempt
|
||||
// produces a dead-letter row. If/when DL→handler fan-out lands, this
|
||||
// can be upgraded to assert the handler marker like the others.
|
||||
let Some(pool) = pool_or_skip().await else {
|
||||
return;
|
||||
};
|
||||
let (server, app_id) = server_for(pool.clone(), "dl").await;
|
||||
|
||||
// A handler that always throws, with a single attempt so it
|
||||
// dead-letters immediately (no retry backoff).
|
||||
let failing = create_script(&server, &app_id, "dl-failing", r#"throw "boom";"#).await;
|
||||
server
|
||||
.post(&format!("/api/v1/admin/apps/{app_id}/triggers/kv"))
|
||||
.json(&json!({
|
||||
"script_id": failing,
|
||||
"collection_glob": "dlsrc",
|
||||
"retry_max_attempts": 1,
|
||||
"retry_base_ms": 0
|
||||
}))
|
||||
.await
|
||||
.assert_status(axum::http::StatusCode::CREATED);
|
||||
|
||||
let source = create_script(
|
||||
&server,
|
||||
&app_id,
|
||||
"dl-source",
|
||||
r#"kv::collection("dlsrc").set("k", 1); #{ ok: true }"#,
|
||||
)
|
||||
.await;
|
||||
execute(&server, &source).await;
|
||||
|
||||
// Poll the dead_letters table for this app.
|
||||
let app_uuid = Uuid::parse_str(&app_id).unwrap();
|
||||
let mut count: i64 = 0;
|
||||
for _ in 0..100 {
|
||||
count = sqlx::query_scalar("SELECT COUNT(*) FROM dead_letters WHERE app_id = $1")
|
||||
.bind(app_uuid)
|
||||
.fetch_one(&pool)
|
||||
.await
|
||||
.expect("count dead_letters");
|
||||
if count > 0 {
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(100)).await;
|
||||
}
|
||||
assert!(count > 0, "a dead-letter row should have been produced");
|
||||
}
|
||||
@@ -15,3 +15,12 @@ serde_json.workspace = true
|
||||
thiserror.workspace = true
|
||||
uuid.workspace = true
|
||||
chrono.workspace = true
|
||||
# Realtime broadcaster trait returns a broadcast::Receiver; subscriber
|
||||
# tokens are HMAC-SHA256 over a base64url payload (v1.1.6).
|
||||
tokio = { workspace = true, features = ["sync"] }
|
||||
hmac.workspace = true
|
||||
sha2.workspace = true
|
||||
base64.workspace = true
|
||||
|
||||
[dev-dependencies]
|
||||
tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread", "time", "sync"] }
|
||||
|
||||
339
crates/shared/src/files.rs
Normal file
339
crates/shared/src/files.rs
Normal file
@@ -0,0 +1,339 @@
|
||||
//! `FilesService` — the v1.1.5 filesystem-backed blob store contract.
|
||||
//!
|
||||
//! Lives in `picloud-shared` (not `executor-core`) so the Rhai bridge,
|
||||
//! the manager-core filesystem+Postgres impl, and any in-memory test
|
||||
//! impl can all depend on the same trait without dragging
|
||||
//! `executor-core` into a Postgres or filesystem dependency.
|
||||
//!
|
||||
//! Implementations MUST derive every storage `app_id` from `cx.app_id`
|
||||
//! — never from a script-passed argument. That is the cross-app
|
||||
//! isolation boundary; see `docs/sdk-shape.md`.
|
||||
//!
|
||||
//! `FilesService` is collection-scoped: scripts get a handle via
|
||||
//! `files::collection(name)` and call
|
||||
//! `create`/`head`/`get`/`update`/`delete`/`list` on it. The blob bytes
|
||||
//! never travel through Postgres or through trigger payloads — the row
|
||||
//! is metadata + a SHA-256 checksum; the bytes live on the filesystem.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use chrono::{DateTime, Utc};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use thiserror::Error;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::SdkCallCx;
|
||||
|
||||
/// POSIX-portable filename cap (255 bytes).
|
||||
pub const MAX_FILE_NAME_BYTES: usize = 255;
|
||||
/// RFC 6838 puts a reasonable media-type ceiling around 127 chars.
|
||||
pub const MAX_CONTENT_TYPE_BYTES: usize = 127;
|
||||
|
||||
/// Payload for `create` — a brand-new blob. The id is server-generated
|
||||
/// (a UUID); scripts never supply it.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct NewFile {
|
||||
pub name: String,
|
||||
pub content_type: String,
|
||||
pub data: Vec<u8>,
|
||||
}
|
||||
|
||||
/// Payload for `update` — replacement bytes plus optional metadata. If
|
||||
/// `name` / `content_type` are `None` the prior values are kept.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FileUpdate {
|
||||
pub data: Vec<u8>,
|
||||
pub name: Option<String>,
|
||||
pub content_type: Option<String>,
|
||||
}
|
||||
|
||||
/// File metadata as scripts and triggers see it. Serialized into
|
||||
/// `ServiceEvent.payload` (the blob bytes are NOT included — files are
|
||||
/// too big to ship through trigger payloads), and surfaced to Rhai by
|
||||
/// `head` / `list`.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct FileMeta {
|
||||
pub id: Uuid,
|
||||
pub collection: String,
|
||||
pub name: String,
|
||||
pub content_type: String,
|
||||
pub size: u64,
|
||||
/// Lowercase hex SHA-256 of the content.
|
||||
pub checksum: String,
|
||||
pub created_at: DateTime<Utc>,
|
||||
pub updated_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
/// One page of file metadata from `FilesService::list`. `next_cursor`
|
||||
/// is `Some` when more pages exist, `None` when exhausted.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct FilesListPage {
|
||||
pub files: Vec<FileMeta>,
|
||||
pub next_cursor: Option<String>,
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait FilesService: Send + Sync {
|
||||
/// Create a new blob; returns its server-generated id. Throws on a
|
||||
/// missing required field, an over-limit blob, or an invalid
|
||||
/// collection name.
|
||||
async fn create(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
new: NewFile,
|
||||
) -> Result<Uuid, FilesError>;
|
||||
|
||||
/// Metadata only — no body read. `None` if the file is missing.
|
||||
async fn head(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<FileMeta>, FilesError>;
|
||||
|
||||
/// Full content. `None` if missing. Verifies the stored checksum
|
||||
/// against the bytes on disk and returns `FilesError::Corrupted`
|
||||
/// when they diverge.
|
||||
async fn get(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
) -> Result<Option<Vec<u8>>, FilesError>;
|
||||
|
||||
/// Replace content (and optionally metadata). Throws `NotFound`
|
||||
/// when the file doesn't exist.
|
||||
async fn update(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
id: &str,
|
||||
upd: FileUpdate,
|
||||
) -> Result<(), FilesError>;
|
||||
|
||||
/// Delete by id; returns whether the file was present.
|
||||
async fn delete(&self, cx: &SdkCallCx, collection: &str, id: &str) -> Result<bool, FilesError>;
|
||||
|
||||
/// Cursor-paginated metadata listing (same shape as KV's list).
|
||||
async fn list(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
collection: &str,
|
||||
cursor: Option<&str>,
|
||||
limit: u32,
|
||||
) -> Result<FilesListPage, FilesError>;
|
||||
}
|
||||
|
||||
/// Failure modes surfaced to the Rhai bridge. The bridge converts each
|
||||
/// to a Rhai runtime error string; the discriminants exist so internal
|
||||
/// callers (admin endpoints, tests) can react more precisely.
|
||||
#[derive(Debug, Error)]
|
||||
pub enum FilesError {
|
||||
/// Empty collection name, or one containing a path separator / `..`
|
||||
/// / NUL — rejected at the SDK boundary per `docs/sdk-shape.md`.
|
||||
#[error("invalid collection name: {0}")]
|
||||
InvalidCollection(String),
|
||||
|
||||
/// A required field on `create` was missing or empty. The string
|
||||
/// names the field (`name` / `content_type` / `data`).
|
||||
#[error("missing required field: {0}")]
|
||||
MissingField(&'static str),
|
||||
|
||||
/// Blob exceeds the per-file size cap (default 100 MB,
|
||||
/// `PICLOUD_FILES_MAX_FILE_SIZE_BYTES`).
|
||||
#[error("file too large: {size} bytes exceeds limit of {limit} bytes")]
|
||||
TooLarge { size: usize, limit: usize },
|
||||
|
||||
/// Filename exceeds `MAX_FILE_NAME_BYTES`.
|
||||
#[error("file name too long: {0} bytes exceeds 255")]
|
||||
NameTooLong(usize),
|
||||
|
||||
/// Content-type exceeds `MAX_CONTENT_TYPE_BYTES`.
|
||||
#[error("content_type too long: {0} bytes exceeds 127")]
|
||||
ContentTypeTooLong(usize),
|
||||
|
||||
/// `update` on a non-existent file.
|
||||
#[error("file not found")]
|
||||
NotFound,
|
||||
|
||||
/// The bytes on disk no longer match the stored checksum — the
|
||||
/// filesystem corrupted or a backup was misconfigured. The operator
|
||||
/// decides what to do with the metadata-vs-bytes mismatch; the repo
|
||||
/// does NOT auto-delete.
|
||||
#[error("file content corrupted (checksum mismatch)")]
|
||||
Corrupted,
|
||||
|
||||
/// Caller principal lacked the required capability. Only raised when
|
||||
/// `cx.principal.is_some()` — scripts running with `principal: None`
|
||||
/// (public HTTP) operate under script-as-gate semantics and skip
|
||||
/// the capability check.
|
||||
#[error("forbidden")]
|
||||
Forbidden,
|
||||
|
||||
/// Anything else — Postgres unavailable, filesystem I/O error, etc.
|
||||
#[error("files backend error: {0}")]
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
impl NewFile {
|
||||
/// Validate required fields + length caps at the SDK boundary.
|
||||
///
|
||||
/// Empty `data` is **accepted** as a valid stored state (v1.1.6
|
||||
/// relaxed the v1.1.5 rejection — empty files are a legitimate use
|
||||
/// case: sentinels, placeholders, zero-byte uploads. See HANDBACK
|
||||
/// §7). `name` and `content_type` are still required.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns the field-specific [`FilesError`] for the first failing
|
||||
/// check.
|
||||
pub fn validate(&self, max_size: usize) -> Result<(), FilesError> {
|
||||
if self.name.trim().is_empty() {
|
||||
return Err(FilesError::MissingField("name"));
|
||||
}
|
||||
if self.content_type.trim().is_empty() {
|
||||
return Err(FilesError::MissingField("content_type"));
|
||||
}
|
||||
if self.name.len() > MAX_FILE_NAME_BYTES {
|
||||
return Err(FilesError::NameTooLong(self.name.len()));
|
||||
}
|
||||
if self.content_type.len() > MAX_CONTENT_TYPE_BYTES {
|
||||
return Err(FilesError::ContentTypeTooLong(self.content_type.len()));
|
||||
}
|
||||
if self.data.len() > max_size {
|
||||
return Err(FilesError::TooLarge {
|
||||
size: self.data.len(),
|
||||
limit: max_size,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl FileUpdate {
|
||||
/// Validate the replacement bytes + any supplied metadata.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns the field-specific [`FilesError`] for the first failing
|
||||
/// check.
|
||||
pub fn validate(&self, max_size: usize) -> Result<(), FilesError> {
|
||||
// Empty replacement bytes are accepted (v1.1.6 relaxation —
|
||||
// consistent with NewFile::validate; updating a file to zero
|
||||
// bytes is as legitimate as creating one).
|
||||
if let Some(name) = &self.name {
|
||||
if name.trim().is_empty() {
|
||||
return Err(FilesError::MissingField("name"));
|
||||
}
|
||||
if name.len() > MAX_FILE_NAME_BYTES {
|
||||
return Err(FilesError::NameTooLong(name.len()));
|
||||
}
|
||||
}
|
||||
if let Some(ct) = &self.content_type {
|
||||
if ct.trim().is_empty() {
|
||||
return Err(FilesError::MissingField("content_type"));
|
||||
}
|
||||
if ct.len() > MAX_CONTENT_TYPE_BYTES {
|
||||
return Err(FilesError::ContentTypeTooLong(ct.len()));
|
||||
}
|
||||
}
|
||||
if self.data.len() > max_size {
|
||||
return Err(FilesError::TooLarge {
|
||||
size: self.data.len(),
|
||||
limit: max_size,
|
||||
});
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Reject a collection name that is empty or could escape the per-app
|
||||
/// files tree. UUID-shaped ids never produce traversal paths, but
|
||||
/// collection names come from scripts so they're validated defensively
|
||||
/// at both the SDK boundary and the repo.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns [`FilesError::InvalidCollection`] when the name is empty or
|
||||
/// contains `/`, `\`, `..`, or a NUL byte.
|
||||
pub fn validate_collection(collection: &str) -> Result<(), FilesError> {
|
||||
if collection.is_empty() {
|
||||
return Err(FilesError::InvalidCollection("must not be empty".into()));
|
||||
}
|
||||
if collection.contains('/')
|
||||
|| collection.contains('\\')
|
||||
|| collection.contains("..")
|
||||
|| collection.contains('\0')
|
||||
{
|
||||
return Err(FilesError::InvalidCollection(format!(
|
||||
"collection {collection:?} must not contain '/', '\\', '..', or NUL"
|
||||
)));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Stub used by the test harness so executor-core integration tests
|
||||
/// (which don't touch files) can construct a `Services` bundle without
|
||||
/// a filesystem or Postgres. Every call returns
|
||||
/// `FilesError::Backend("...")` so accidental use surfaces clearly.
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct NoopFilesService;
|
||||
|
||||
#[async_trait]
|
||||
impl FilesService for NoopFilesService {
|
||||
async fn create(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_new: NewFile,
|
||||
) -> Result<Uuid, FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
|
||||
async fn head(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_id: &str,
|
||||
) -> Result<Option<FileMeta>, FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
|
||||
async fn get(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_id: &str,
|
||||
) -> Result<Option<Vec<u8>>, FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
|
||||
async fn update(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_id: &str,
|
||||
_upd: FileUpdate,
|
||||
) -> Result<(), FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
|
||||
async fn delete(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_id: &str,
|
||||
) -> Result<bool, FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
|
||||
async fn list(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_collection: &str,
|
||||
_cursor: Option<&str>,
|
||||
_limit: u32,
|
||||
) -> Result<FilesListPage, FilesError> {
|
||||
Err(FilesError::Backend("files is not wired in".into()))
|
||||
}
|
||||
}
|
||||
137
crates/shared/src/http.rs
Normal file
137
crates/shared/src/http.rs
Normal file
@@ -0,0 +1,137 @@
|
||||
//! `HttpService` — the v1.1.4 outbound-HTTP contract.
|
||||
//!
|
||||
//! Lives in `picloud-shared` (not `executor-core` or `manager-core`)
|
||||
//! so the Rhai bridge and the manager-core reqwest-backed impl can both
|
||||
//! depend on the same trait without dragging `executor-core` into
|
||||
//! `manager-core`'s dep graph — mirrors [`crate::kv`].
|
||||
//!
|
||||
//! Unlike KV/docs, `http::*` has no app-scoped data, so there is no
|
||||
//! cross-app isolation boundary to enforce here. `cx.app_id` is still
|
||||
//! forwarded for audit-log attribution and (future, v1.2) per-app rate
|
||||
//! limits. The load-bearing security mechanism is the SSRF deny-list
|
||||
//! applied to the *resolved IP* — that lives in the manager-core impl,
|
||||
//! not in this contract.
|
||||
//!
|
||||
//! Body encoding + per-method dispatch happen in the Rhai bridge before
|
||||
//! the request reaches this trait: the service receives an already-
|
||||
//! encoded body plus a `content_type`, so the impl stays a thin
|
||||
//! transport layer.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use async_trait::async_trait;
|
||||
use thiserror::Error;
|
||||
|
||||
use crate::SdkCallCx;
|
||||
|
||||
/// A fully-resolved outbound request. The bridge builds this from the
|
||||
/// script-facing `(url, body, opts)` arguments; the service backend
|
||||
/// turns it into a real network call.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct HttpRequest {
|
||||
/// Uppercased HTTP method (`GET`, `POST`, …). The escape-hatch
|
||||
/// `http::request(method, …)` lets scripts pass arbitrary methods,
|
||||
/// so the impl validates this rather than the bridge.
|
||||
pub method: String,
|
||||
pub url: String,
|
||||
/// Caller-supplied headers, merged into the request. Header names
|
||||
/// are case-insensitive on the wire; stored verbatim here.
|
||||
pub headers: BTreeMap<String, String>,
|
||||
/// Already-encoded body. `None` means no body (GET/HEAD, or an
|
||||
/// explicit `()` body).
|
||||
pub body: Option<Vec<u8>>,
|
||||
/// Content-Type the bridge chose for `body` (e.g.
|
||||
/// `application/json`). Ignored when the caller set their own
|
||||
/// `Content-Type` header. `None` when there is no body.
|
||||
pub content_type: Option<String>,
|
||||
/// Total request budget in ms (already clamped to the 60s ceiling
|
||||
/// by the bridge).
|
||||
pub timeout_ms: u32,
|
||||
pub follow_redirects: bool,
|
||||
/// Max redirects to follow (already clamped to 10 by the bridge).
|
||||
pub max_redirects: u32,
|
||||
/// Script id for the default `User-Agent` and audit attribution.
|
||||
/// `None` when unavailable (the bridge always sets it from
|
||||
/// `cx`-adjacent context, but the field stays optional so the
|
||||
/// trait isn't coupled to how the id is sourced).
|
||||
pub script_id: Option<String>,
|
||||
}
|
||||
|
||||
/// The response shape the bridge turns into a Rhai map. JSON parsing of
|
||||
/// `body_raw` happens in the bridge (it needs the Rhai value types), so
|
||||
/// the service returns only the raw string + lowercased headers.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct HttpResponse {
|
||||
pub status: u16,
|
||||
/// Header names lowercased (per the documented response shape).
|
||||
pub headers: BTreeMap<String, String>,
|
||||
pub body_raw: String,
|
||||
}
|
||||
|
||||
/// Failure modes surfaced to the Rhai bridge. The bridge prefixes each
|
||||
/// `Display` string with `"http: "`. **None of these may leak the
|
||||
/// resolved IP** — the SSRF reason is a CIDR-category label only.
|
||||
#[derive(Debug, Error)]
|
||||
pub enum HttpError {
|
||||
/// Caller principal lacked `AppHttpRequest`. Only raised when
|
||||
/// `cx.principal.is_some()`; public-HTTP scripts skip the check.
|
||||
#[error("forbidden")]
|
||||
Forbidden,
|
||||
|
||||
/// URL failed to parse, or carried no host.
|
||||
#[error("invalid url: {0}")]
|
||||
InvalidUrl(String),
|
||||
|
||||
/// Scheme other than http/https (file, ftp, gopher, …).
|
||||
#[error("scheme not allowed: {0}")]
|
||||
BlockedScheme(String),
|
||||
|
||||
/// Destination port is on the explicit block list (22, 25, 465, 587).
|
||||
#[error("port not allowed: {0}")]
|
||||
BlockedPort(u16),
|
||||
|
||||
/// Resolved IP hit the SSRF deny-list. `reason` is a CIDR-category
|
||||
/// label (e.g. "loopback", "private", "link-local") — never the IP.
|
||||
#[error("blocked by SSRF policy: {0}")]
|
||||
Ssrf(String),
|
||||
|
||||
/// The request exceeded the wall-clock budget.
|
||||
#[error("request timed out")]
|
||||
Timeout,
|
||||
|
||||
/// Request or response body exceeded the configured size cap.
|
||||
/// `which` is `"request"` or `"response"`.
|
||||
#[error("{0} body exceeds size limit")]
|
||||
BodyTooLarge(&'static str),
|
||||
|
||||
/// DNS / connect / TLS failure. The message is generic and MUST NOT
|
||||
/// contain the resolved IP.
|
||||
#[error("{0}")]
|
||||
Network(String),
|
||||
|
||||
/// Anything else the impl wants to surface (still safe to show a
|
||||
/// script).
|
||||
#[error("{0}")]
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
/// Stub used by the executor-core test harness so engine integration
|
||||
/// tests (which don't make real network calls) can construct a
|
||||
/// `Services` bundle. Every call errors so accidental use surfaces.
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct NoopHttpService;
|
||||
|
||||
#[async_trait]
|
||||
impl HttpService for NoopHttpService {
|
||||
async fn request(&self, _cx: &SdkCallCx, _req: HttpRequest) -> Result<HttpResponse, HttpError> {
|
||||
Err(HttpError::Network("http is not wired in".into()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Outbound-HTTP contract. A single generic `request` method funnels
|
||||
/// every verb (`get`/`post`/…/`request`); the bridge maps the
|
||||
/// script-facing surface onto it.
|
||||
#[async_trait]
|
||||
pub trait HttpService: Send + Sync {
|
||||
async fn request(&self, cx: &SdkCallCx, req: HttpRequest) -> Result<HttpResponse, HttpError>;
|
||||
}
|
||||
@@ -12,17 +12,23 @@ pub mod error;
|
||||
pub mod events;
|
||||
pub mod exec_summary;
|
||||
pub mod execution_log;
|
||||
pub mod files;
|
||||
pub mod http;
|
||||
pub mod ids;
|
||||
pub mod inbox;
|
||||
pub mod kv;
|
||||
pub mod log_sink;
|
||||
pub mod modules;
|
||||
pub mod outbox_writer;
|
||||
pub mod pubsub;
|
||||
pub mod realtime;
|
||||
pub mod realtime_authority;
|
||||
pub mod route;
|
||||
pub mod sandbox;
|
||||
pub mod script;
|
||||
pub mod sdk_cx;
|
||||
pub mod services;
|
||||
pub mod subscriber_token;
|
||||
pub mod trigger_event;
|
||||
pub mod validator;
|
||||
pub mod version;
|
||||
@@ -35,6 +41,11 @@ pub use error::Error;
|
||||
pub use events::{EmitError, NoopEventEmitter, ServiceEvent, ServiceEventEmitter};
|
||||
pub use exec_summary::ExecResponseSummary;
|
||||
pub use execution_log::{ExecutionLog, ExecutionStatus};
|
||||
pub use files::{
|
||||
validate_collection as validate_files_collection, FileMeta, FileUpdate, FilesError,
|
||||
FilesListPage, FilesService, NewFile, NoopFilesService,
|
||||
};
|
||||
pub use http::{HttpError, HttpRequest, HttpResponse, HttpService, NoopHttpService};
|
||||
pub use ids::{AdminUserId, ApiKeyId, AppId, ExecutionId, RequestId, ScriptId, TriggerId};
|
||||
pub use inbox::{
|
||||
InboxDeliveryOutcome, InboxFailureKind, InboxResolver, InboxResult, NoopInboxResolver,
|
||||
@@ -43,11 +54,18 @@ pub use kv::{KvError, KvListPage, KvService, NoopKvService};
|
||||
pub use log_sink::{ExecutionLogSink, LogSinkError};
|
||||
pub use modules::{ModuleScript, ModuleSource, ModuleSourceError, NoopModuleSource};
|
||||
pub use outbox_writer::{HttpDispatchPayload, NewHttpOutbox, OutboxWriter, OutboxWriterError};
|
||||
pub use pubsub::{
|
||||
topic_matches, validate_topic_pattern, NoopPubsubService, PubsubError, PubsubService,
|
||||
};
|
||||
pub use realtime::{BroadcasterError, NoopRealtimeBroadcaster, RealtimeBroadcaster, RealtimeEvent};
|
||||
pub use realtime_authority::{DenyAllRealtimeAuthority, RealtimeAuthority, SubscribeDenied};
|
||||
pub use route::{DispatchMode, HostKind, PathKind, Route};
|
||||
pub use sandbox::ScriptSandbox;
|
||||
pub use script::{Script, ScriptKind};
|
||||
pub use sdk_cx::SdkCallCx;
|
||||
pub use services::Services;
|
||||
pub use trigger_event::{DeadLetterEventDetail, DocsEventOp, KvEventOp, TriggerEvent};
|
||||
pub use trigger_event::{
|
||||
DeadLetterEventDetail, DocsEventOp, FilesEventOp, KvEventOp, TriggerEvent,
|
||||
};
|
||||
pub use validator::{ScriptValidator, ValidatedScript, ValidationError};
|
||||
pub use version::{API_VERSION, PRODUCT_VERSION, SDK_VERSION, WIRE_VERSION};
|
||||
|
||||
194
crates/shared/src/pubsub.rs
Normal file
194
crates/shared/src/pubsub.rs
Normal file
@@ -0,0 +1,194 @@
|
||||
//! `PubsubService` — the v1.1.5 durable pub/sub contract.
|
||||
//!
|
||||
//! `pubsub::publish_durable(topic, message)` writes to the universal
|
||||
//! outbox; the publish-time fan-out inserts one delivery row per
|
||||
//! matching `pubsub` trigger, and each delivery retries / dead-letters
|
||||
//! independently (the dispatcher already handles one-row-equals-one-
|
||||
//! dispatch — no dispatcher changes for pub/sub).
|
||||
//!
|
||||
//! `publish_ephemeral` is committed as a v1.2 addition — the suffix
|
||||
//! naming exists now so users learn "durable by default" from day one.
|
||||
//!
|
||||
//! Topic pattern matching runs in Rust (not SQL) so the trigger-select
|
||||
//! query stays simple. The matcher + validator live here in
|
||||
//! `picloud-shared` so the manager-core publish path, the admin trigger
|
||||
//! endpoint, and tests all agree on the rules.
|
||||
|
||||
use async_trait::async_trait;
|
||||
use thiserror::Error;
|
||||
|
||||
use crate::SdkCallCx;
|
||||
|
||||
#[async_trait]
|
||||
pub trait PubsubService: Send + Sync {
|
||||
/// Durable publish: writes the message to the outbox, fanned out to
|
||||
/// every matching enabled `pubsub` trigger in `cx.app_id`. Succeeds
|
||||
/// silently (zero rows written) when no trigger matches the topic.
|
||||
async fn publish_durable(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
topic: &str,
|
||||
message: serde_json::Value,
|
||||
) -> Result<(), PubsubError>;
|
||||
|
||||
/// Mint an HMAC-signed realtime subscriber token (v1.1.6). Backs the
|
||||
/// `pubsub::subscriber_token(topics, ttl)` Rhai SDK call. The minted
|
||||
/// token authorizes an external SSE client to subscribe to the given
|
||||
/// `topics` for `ttl_seconds` (clamped to the configured bounds; the
|
||||
/// configured default applies when `ttl_seconds` is `None`).
|
||||
///
|
||||
/// Every topic must already be registered as externally subscribable
|
||||
/// in `cx.app_id`; `cx.principal` must be `Some` (anonymous
|
||||
/// public-HTTP scripts can't mint). See [`PubsubError::SubscriberToken`]
|
||||
/// for the rejection messages.
|
||||
///
|
||||
/// The default impl errors `Unavailable` so test fakes and the
|
||||
/// `NoopPubsubService` keep compiling; the real minting lives in
|
||||
/// manager-core's `PubsubServiceImpl`.
|
||||
async fn mint_subscriber_token(
|
||||
&self,
|
||||
cx: &SdkCallCx,
|
||||
topics: Vec<String>,
|
||||
ttl_seconds: Option<i64>,
|
||||
) -> Result<String, PubsubError> {
|
||||
let _ = (cx, topics, ttl_seconds);
|
||||
Err(PubsubError::Unavailable(
|
||||
"subscriber tokens are not wired in".into(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum PubsubError {
|
||||
/// Empty topic; rejected at the SDK boundary.
|
||||
#[error("topic must not be empty")]
|
||||
EmptyTopic,
|
||||
|
||||
/// Caller principal lacked the required capability. Only raised when
|
||||
/// `cx.principal.is_some()` (script-as-gate; public HTTP skips it).
|
||||
#[error("forbidden")]
|
||||
Forbidden,
|
||||
|
||||
/// Serialization / validation failure on the message.
|
||||
#[error("pubsub rejected: {0}")]
|
||||
Rejected(String),
|
||||
|
||||
/// A `pubsub::subscriber_token` mint was rejected (empty topics,
|
||||
/// unregistered topic, ttl out of range, anonymous caller). The
|
||||
/// string is the full user-facing message; the SDK surfaces it
|
||||
/// verbatim so scripts see the documented wording.
|
||||
#[error("{0}")]
|
||||
SubscriberToken(String),
|
||||
|
||||
/// Anything else — Postgres unavailable, etc.
|
||||
#[error("pubsub backend error: {0}")]
|
||||
Unavailable(String),
|
||||
}
|
||||
|
||||
/// Match a stored `topic_pattern` against a published `topic`.
|
||||
///
|
||||
/// - `"*"` matches every topic.
|
||||
/// - `"<prefix>.*"` matches any topic starting with `"<prefix>."`.
|
||||
/// - anything else is an exact match.
|
||||
///
|
||||
/// Mid-pattern wildcards (`*.created`, `a.*.b`) are NOT supported — they
|
||||
/// are rejected at trigger creation by [`validate_topic_pattern`], so
|
||||
/// the only patterns reaching this matcher are exact / prefix / `*`.
|
||||
#[must_use]
|
||||
pub fn topic_matches(pattern: &str, topic: &str) -> bool {
|
||||
if pattern == "*" {
|
||||
return true;
|
||||
}
|
||||
if let Some(prefix) = pattern.strip_suffix('*') {
|
||||
// `prefix` retains the trailing '.', e.g. "user." for "user.*".
|
||||
return topic.starts_with(prefix);
|
||||
}
|
||||
pattern == topic
|
||||
}
|
||||
|
||||
/// Validate a subscription topic pattern. Accepts exactly: `"*"`
|
||||
/// (universal), `"<prefix>.*"` (prefix wildcard, single trailing star),
|
||||
/// or a literal with no `*` (exact). Everything else — mid-pattern
|
||||
/// wildcards, multiple stars, a star not at the end — is rejected.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns `Err(message)` with `"unsupported pubsub topic pattern: …"`
|
||||
/// for any unsupported shape (or an empty pattern).
|
||||
pub fn validate_topic_pattern(pattern: &str) -> Result<(), String> {
|
||||
if pattern.is_empty() {
|
||||
return Err("unsupported pubsub topic pattern: <empty>".to_string());
|
||||
}
|
||||
if pattern == "*" {
|
||||
return Ok(());
|
||||
}
|
||||
let stars = pattern.matches('*').count();
|
||||
if stars == 0 {
|
||||
return Ok(()); // exact
|
||||
}
|
||||
if stars == 1 && pattern.ends_with(".*") {
|
||||
return Ok(()); // prefix wildcard
|
||||
}
|
||||
Err(format!("unsupported pubsub topic pattern: {pattern}"))
|
||||
}
|
||||
|
||||
/// Stub for the test harness so executor-core integration tests can
|
||||
/// build a `Services` bundle without a database. Every call errors.
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct NoopPubsubService;
|
||||
|
||||
#[async_trait]
|
||||
impl PubsubService for NoopPubsubService {
|
||||
async fn publish_durable(
|
||||
&self,
|
||||
_cx: &SdkCallCx,
|
||||
_topic: &str,
|
||||
_message: serde_json::Value,
|
||||
) -> Result<(), PubsubError> {
|
||||
Err(PubsubError::Unavailable("pubsub is not wired in".into()))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn exact_match() {
|
||||
assert!(topic_matches("user.created", "user.created"));
|
||||
assert!(!topic_matches("user.created", "user.deleted"));
|
||||
assert!(!topic_matches("user.created", "user.created.x"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prefix_wildcard() {
|
||||
assert!(topic_matches("user.*", "user.created"));
|
||||
assert!(topic_matches("user.*", "user.deleted"));
|
||||
assert!(!topic_matches("user.*", "users.created"));
|
||||
assert!(!topic_matches("user.*", "order.created"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn universal() {
|
||||
assert!(topic_matches("*", "anything"));
|
||||
assert!(topic_matches("*", "a.b.c"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validation_accepts_supported_shapes() {
|
||||
assert!(validate_topic_pattern("*").is_ok());
|
||||
assert!(validate_topic_pattern("user.created").is_ok());
|
||||
assert!(validate_topic_pattern("user.*").is_ok());
|
||||
assert!(validate_topic_pattern("a.b.c").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validation_rejects_unsupported_shapes() {
|
||||
for bad in ["*.created", "**", "a.*.b", "user.*x", "*user", ""] {
|
||||
assert!(
|
||||
validate_topic_pattern(bad).is_err(),
|
||||
"expected {bad:?} to be rejected"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
86
crates/shared/src/realtime.rs
Normal file
86
crates/shared/src/realtime.rs
Normal file
@@ -0,0 +1,86 @@
|
||||
//! `RealtimeBroadcaster` — the in-process fan-out seam for SSE realtime
|
||||
//! delivery (v1.1.6).
|
||||
//!
|
||||
//! Structurally a sibling of [`crate::inbox::InboxResolver`]: the trait
|
||||
//! lives here in `picloud-shared` because the publish side
|
||||
//! (`PubsubServiceImpl` in manager-core) and the subscribe side (the SSE
|
||||
//! handler in orchestrator-core) live in different crates and both need
|
||||
//! one shared instance. The in-process impl lives in orchestrator-core
|
||||
//! (`Mutex<HashMap<(AppId, topic), broadcast::Sender>>`); cluster mode
|
||||
//! (v1.3+) swaps it for a Postgres `LISTEN/NOTIFY`-backed resolver behind
|
||||
//! the same trait without touching either caller.
|
||||
//!
|
||||
//! Delivery is **best-effort, at-most-once**: this is the realtime path,
|
||||
//! NOT the durable one. Durable trigger fan-out (retry / dead-letter)
|
||||
//! goes through the outbox and is the publish caller's separate concern.
|
||||
//! A slow SSE consumer loses the oldest events (bounded broadcast
|
||||
//! buffer); SSE's own transport-layer auto-reconnect is the recovery
|
||||
//! mechanism (no server-side replay in v1.1.6).
|
||||
|
||||
use async_trait::async_trait;
|
||||
use chrono::{DateTime, Utc};
|
||||
use thiserror::Error;
|
||||
use tokio::sync::broadcast;
|
||||
|
||||
use crate::AppId;
|
||||
|
||||
/// A single realtime event delivered to in-process SSE subscribers. The
|
||||
/// SSE handler serializes this to `data: {...}\n\n` on the wire.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct RealtimeEvent {
|
||||
pub topic: String,
|
||||
pub message: serde_json::Value,
|
||||
pub published_at: DateTime<Utc>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Error)]
|
||||
pub enum BroadcasterError {
|
||||
/// Reserved for backends that can fail to register a subscriber
|
||||
/// (e.g. the cluster-mode `LISTEN/NOTIFY` resolver). The in-process
|
||||
/// impl never returns this.
|
||||
#[error("realtime broadcaster unavailable: {0}")]
|
||||
Unavailable(String),
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait RealtimeBroadcaster: Send + Sync {
|
||||
/// Subscribe to events on `(app_id, topic)`. Returns a receiver that
|
||||
/// yields events until dropped. Channels are created lazily on first
|
||||
/// subscribe.
|
||||
async fn subscribe(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
topic: &str,
|
||||
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError>;
|
||||
|
||||
/// Publish an event to in-process subscribers. NOT durable — the
|
||||
/// outbox-backed durable fan-out is the publish caller's separate
|
||||
/// concern. A publish with no live subscribers is a silent no-op.
|
||||
async fn publish(&self, app_id: AppId, topic: &str, event: RealtimeEvent);
|
||||
|
||||
/// Drop every subscriber for a topic (called on topic DELETE). Live
|
||||
/// receivers observe a closed channel and disconnect cleanly.
|
||||
async fn drop_topic(&self, app_id: AppId, topic: &str);
|
||||
}
|
||||
|
||||
/// Bootstrap / test impl: subscribe yields a receiver on a throwaway
|
||||
/// channel, publish is a no-op. Lets a `Services`-style bundle build
|
||||
/// without the real registry wired in.
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct NoopRealtimeBroadcaster;
|
||||
|
||||
#[async_trait]
|
||||
impl RealtimeBroadcaster for NoopRealtimeBroadcaster {
|
||||
async fn subscribe(
|
||||
&self,
|
||||
_app_id: AppId,
|
||||
_topic: &str,
|
||||
) -> Result<broadcast::Receiver<RealtimeEvent>, BroadcasterError> {
|
||||
let (_tx, rx) = broadcast::channel(1);
|
||||
Ok(rx)
|
||||
}
|
||||
|
||||
async fn publish(&self, _app_id: AppId, _topic: &str, _event: RealtimeEvent) {}
|
||||
|
||||
async fn drop_topic(&self, _app_id: AppId, _topic: &str) {}
|
||||
}
|
||||
70
crates/shared/src/realtime_authority.rs
Normal file
70
crates/shared/src/realtime_authority.rs
Normal file
@@ -0,0 +1,70 @@
|
||||
//! `RealtimeAuthority` — the SSE subscribe authorization seam (v1.1.6).
|
||||
//!
|
||||
//! The SSE endpoint (`GET /realtime/topics/{topic}`) is a data-plane
|
||||
//! surface in orchestrator-core, but deciding whether a subscribe is
|
||||
//! allowed needs a `topics` table read plus (for token-gated topics) an
|
||||
//! HMAC verify against the app's signing key — both of which require DB
|
||||
//! access and the signing-key material that must NOT leak into the
|
||||
//! data-plane crate. This trait keeps all of that inside the manager-core
|
||||
//! impl: orchestrator-core only ever sees the three-way verdict below.
|
||||
//!
|
||||
//! `NotFound` is deliberately returned for *both* "no such topic" and
|
||||
//! "topic exists but isn't externally subscribable" so the endpoint
|
||||
//! can't be used to probe which internal topics exist (design notes §5).
|
||||
|
||||
use async_trait::async_trait;
|
||||
|
||||
use crate::AppId;
|
||||
|
||||
/// Why a subscribe attempt was refused. The SSE handler maps these to
|
||||
/// HTTP status codes.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum SubscribeDenied {
|
||||
/// No externally-subscribable topic by that name in this app → 404.
|
||||
/// Used for genuinely-missing topics AND internal-only ones, so the
|
||||
/// endpoint doesn't leak which internal topics exist.
|
||||
NotFound,
|
||||
/// The topic is token-gated and the presented token was missing,
|
||||
/// malformed, badly signed, expired, or not scoped to this topic →
|
||||
/// 401 (generic; never says which check failed).
|
||||
Unauthorized,
|
||||
/// Backend failure (DB unavailable, etc.) → 500.
|
||||
Backend(String),
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
pub trait RealtimeAuthority: Send + Sync {
|
||||
/// Decide whether an external client may subscribe to
|
||||
/// `(app_id, topic)`. `token` is the bearer/query token if the
|
||||
/// client presented one (`None` otherwise).
|
||||
///
|
||||
/// Returns `Ok(())` when the subscribe is permitted (public topic,
|
||||
/// or token-gated topic with a valid token scoped to it).
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// [`SubscribeDenied`] — see the variants for the status mapping.
|
||||
async fn authorize_subscribe(
|
||||
&self,
|
||||
app_id: AppId,
|
||||
topic: &str,
|
||||
token: Option<&str>,
|
||||
) -> Result<(), SubscribeDenied>;
|
||||
}
|
||||
|
||||
/// Bootstrap impl: denies everything as `NotFound`. Replaced in
|
||||
/// `build_app` with the manager-core DB-backed authority.
|
||||
#[derive(Debug, Default, Clone, Copy)]
|
||||
pub struct DenyAllRealtimeAuthority;
|
||||
|
||||
#[async_trait]
|
||||
impl RealtimeAuthority for DenyAllRealtimeAuthority {
|
||||
async fn authorize_subscribe(
|
||||
&self,
|
||||
_app_id: AppId,
|
||||
_topic: &str,
|
||||
_token: Option<&str>,
|
||||
) -> Result<(), SubscribeDenied> {
|
||||
Err(SubscribeDenied::NotFound)
|
||||
}
|
||||
}
|
||||
@@ -12,7 +12,7 @@
|
||||
//! the cx in is shared by both sides. Pure value type — no handles, no
|
||||
//! DB pool references, no allocations beyond what's in `Principal`.
|
||||
|
||||
use crate::{AppId, ExecutionId, Principal, RequestId, TriggerEvent};
|
||||
use crate::{AppId, ExecutionId, Principal, RequestId, ScriptId, TriggerEvent};
|
||||
|
||||
/// Per-invocation context for every stateful SDK service call.
|
||||
///
|
||||
@@ -27,6 +27,11 @@ pub struct SdkCallCx {
|
||||
/// every `(app_id, …)` storage lookup the script makes.
|
||||
pub app_id: AppId,
|
||||
|
||||
/// The script being executed. Used for audit-log attribution and
|
||||
/// the default outbound-HTTP `User-Agent` (`picloud/<v>
|
||||
/// (script:<id>)`). Added in v1.1.4 for the `http::*` SDK.
|
||||
pub script_id: ScriptId,
|
||||
|
||||
/// Caller identity, when authenticated. `None` for unauthenticated
|
||||
/// data-plane HTTP requests (the common case for public endpoints);
|
||||
/// `Some` when the call came in via the dashboard, an API key, or a
|
||||
|
||||
@@ -20,8 +20,9 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
use crate::{
|
||||
DeadLetterService, DocsService, KvService, ModuleSource, NoopDeadLetterService,
|
||||
NoopDocsService, NoopEventEmitter, NoopKvService, NoopModuleSource, ServiceEventEmitter,
|
||||
DeadLetterService, DocsService, FilesService, HttpService, KvService, ModuleSource,
|
||||
NoopDeadLetterService, NoopDocsService, NoopEventEmitter, NoopFilesService, NoopHttpService,
|
||||
NoopKvService, NoopModuleSource, NoopPubsubService, PubsubService, ServiceEventEmitter,
|
||||
};
|
||||
|
||||
/// SDK service bundle. See module docs for the lifecycle and the v1.1.x
|
||||
@@ -53,6 +54,25 @@ pub struct Services {
|
||||
/// `import`. Backed by Postgres in the picloud binary; in-memory
|
||||
/// fakes in resolver tests.
|
||||
pub modules: Arc<dyn ModuleSource>,
|
||||
|
||||
/// Outbound HTTP (v1.1.4). Scripts get `http::{get,post,…}`.
|
||||
/// Backed by a reqwest client with the SSRF deny-list resolver in
|
||||
/// the picloud binary; `NoopHttpService` in tests that don't make
|
||||
/// network calls.
|
||||
pub http: Arc<dyn HttpService>,
|
||||
|
||||
/// Filesystem-backed blob storage (v1.1.5). Scripts get
|
||||
/// `files::collection(name).{create,head,get,update,delete,list}`.
|
||||
/// Backed by a Postgres-metadata + on-disk-bytes repo in the
|
||||
/// picloud binary; `NoopFilesService` in tests that don't touch
|
||||
/// files.
|
||||
pub files: Arc<dyn FilesService>,
|
||||
|
||||
/// Durable pub/sub (v1.1.5). Scripts get
|
||||
/// `pubsub::publish_durable(topic, message)`. Backed by a
|
||||
/// publish-time outbox fan-out in the picloud binary;
|
||||
/// `NoopPubsubService` in tests that don't publish.
|
||||
pub pubsub: Arc<dyn PubsubService>,
|
||||
}
|
||||
|
||||
impl Services {
|
||||
@@ -60,12 +80,16 @@ impl Services {
|
||||
/// The picloud binary's `main` wires this up after the DB pool is
|
||||
/// open; tests build it from in-memory fakes.
|
||||
#[must_use]
|
||||
#[allow(clippy::too_many_arguments)] // one Arc per stateful service; a builder would just move the noise
|
||||
pub fn new(
|
||||
kv: Arc<dyn KvService>,
|
||||
docs: Arc<dyn DocsService>,
|
||||
dead_letters: Arc<dyn DeadLetterService>,
|
||||
events: Arc<dyn ServiceEventEmitter>,
|
||||
modules: Arc<dyn ModuleSource>,
|
||||
http: Arc<dyn HttpService>,
|
||||
files: Arc<dyn FilesService>,
|
||||
pubsub: Arc<dyn PubsubService>,
|
||||
) -> Self {
|
||||
Self {
|
||||
kv,
|
||||
@@ -73,6 +97,9 @@ impl Services {
|
||||
dead_letters,
|
||||
events,
|
||||
modules,
|
||||
http,
|
||||
files,
|
||||
pubsub,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,6 +116,9 @@ impl Services {
|
||||
Arc::new(NoopDeadLetterService),
|
||||
Arc::new(NoopEventEmitter),
|
||||
Arc::new(NoopModuleSource),
|
||||
Arc::new(NoopHttpService),
|
||||
Arc::new(NoopFilesService),
|
||||
Arc::new(NoopPubsubService),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
200
crates/shared/src/subscriber_token.rs
Normal file
200
crates/shared/src/subscriber_token.rs
Normal file
@@ -0,0 +1,200 @@
|
||||
//! HMAC-signed realtime subscriber tokens (v1.1.6, design notes §5).
|
||||
//!
|
||||
//! A token is a compact, URL-safe, two-part string:
|
||||
//!
|
||||
//! ```text
|
||||
//! <base64url(payload)>.<base64url(signature)>
|
||||
//! ```
|
||||
//!
|
||||
//! where `payload` is the JSON [`TokenClaims`] and `signature` is
|
||||
//! `HMAC-SHA256(app_signing_key, base64url(payload))`. Tokens are minted
|
||||
//! by scripts via `pubsub::subscriber_token` (the minter lives in
|
||||
//! manager-core's `PubsubServiceImpl`) and verified by the SSE subscribe
|
||||
//! path (the verifier lives in manager-core's `RealtimeAuthority` impl).
|
||||
//! Both sides depend on this module so the byte-for-byte contract has a
|
||||
//! single home.
|
||||
//!
|
||||
//! There is no per-token revocation in v1.1.6 by design: HMAC bearers
|
||||
//! can't be individually revoked. Rotating an app's signing key
|
||||
//! invalidates every token for that app wholesale; short TTLs are the
|
||||
//! safety mechanism.
|
||||
|
||||
use base64::engine::general_purpose::URL_SAFE_NO_PAD;
|
||||
use base64::Engine as _;
|
||||
use hmac::{Hmac, Mac};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::Sha256;
|
||||
use thiserror::Error;
|
||||
|
||||
use crate::AppId;
|
||||
|
||||
type HmacSha256 = Hmac<Sha256>;
|
||||
|
||||
/// The signed payload. `exp` / `iat` are Unix seconds.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
pub struct TokenClaims {
|
||||
pub app_id: AppId,
|
||||
pub topics: Vec<String>,
|
||||
pub exp: i64,
|
||||
pub iat: i64,
|
||||
}
|
||||
|
||||
impl TokenClaims {
|
||||
/// Does this token grant access to `topic`?
|
||||
#[must_use]
|
||||
pub fn allows_topic(&self, topic: &str) -> bool {
|
||||
self.topics.iter().any(|t| t == topic)
|
||||
}
|
||||
|
||||
/// Is the token expired relative to `now_unix` (Unix seconds)?
|
||||
#[must_use]
|
||||
pub fn is_expired(&self, now_unix: i64) -> bool {
|
||||
now_unix >= self.exp
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Error, PartialEq, Eq)]
|
||||
pub enum TokenError {
|
||||
#[error("token is malformed")]
|
||||
Malformed,
|
||||
#[error("token signature is invalid")]
|
||||
BadSignature,
|
||||
#[error("token has expired")]
|
||||
Expired,
|
||||
}
|
||||
|
||||
/// Sign `claims` with `key`, producing the `payload.signature` string.
|
||||
#[must_use]
|
||||
pub fn sign(key: &[u8], claims: &TokenClaims) -> String {
|
||||
// `serde_json` on a fixed-field struct never fails to serialize.
|
||||
let payload_json = serde_json::to_vec(claims).expect("TokenClaims serialize");
|
||||
let payload_b64 = URL_SAFE_NO_PAD.encode(&payload_json);
|
||||
let sig = mac_sign(key, payload_b64.as_bytes());
|
||||
let sig_b64 = URL_SAFE_NO_PAD.encode(sig);
|
||||
format!("{payload_b64}.{sig_b64}")
|
||||
}
|
||||
|
||||
/// Verify `token` against `key` and check expiry against `now_unix`
|
||||
/// (Unix seconds). Returns the decoded [`TokenClaims`] on success.
|
||||
///
|
||||
/// Topic-scope checking (is the requested topic in the token's list?)
|
||||
/// is the caller's responsibility via [`TokenClaims::allows_topic`] —
|
||||
/// this function proves authenticity + liveness only.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// [`TokenError::Malformed`] if the shape / base64 / JSON is wrong,
|
||||
/// [`TokenError::BadSignature`] if the HMAC doesn't match, or
|
||||
/// [`TokenError::Expired`] if `now_unix >= exp`.
|
||||
pub fn verify(key: &[u8], token: &str, now_unix: i64) -> Result<TokenClaims, TokenError> {
|
||||
let (payload_b64, sig_b64) = token.split_once('.').ok_or(TokenError::Malformed)?;
|
||||
|
||||
let provided_sig = URL_SAFE_NO_PAD
|
||||
.decode(sig_b64)
|
||||
.map_err(|_| TokenError::Malformed)?;
|
||||
|
||||
// Constant-time verify of the MAC over the exact payload bytes.
|
||||
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC accepts any key length");
|
||||
mac.update(payload_b64.as_bytes());
|
||||
mac.verify_slice(&provided_sig)
|
||||
.map_err(|_| TokenError::BadSignature)?;
|
||||
|
||||
// Signature good → decode the claims and check expiry.
|
||||
let payload_json = URL_SAFE_NO_PAD
|
||||
.decode(payload_b64)
|
||||
.map_err(|_| TokenError::Malformed)?;
|
||||
let claims: TokenClaims =
|
||||
serde_json::from_slice(&payload_json).map_err(|_| TokenError::Malformed)?;
|
||||
|
||||
if claims.is_expired(now_unix) {
|
||||
return Err(TokenError::Expired);
|
||||
}
|
||||
Ok(claims)
|
||||
}
|
||||
|
||||
fn mac_sign(key: &[u8], data: &[u8]) -> Vec<u8> {
|
||||
let mut mac = HmacSha256::new_from_slice(key).expect("HMAC accepts any key length");
|
||||
mac.update(data);
|
||||
mac.finalize().into_bytes().to_vec()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn claims(app: AppId, topics: &[&str], exp: i64) -> TokenClaims {
|
||||
TokenClaims {
|
||||
app_id: app,
|
||||
topics: topics.iter().map(|s| (*s).to_string()).collect(),
|
||||
iat: 1000,
|
||||
exp,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn round_trip_verifies() {
|
||||
let key = b"super-secret-key-bytes-0123456789";
|
||||
let app = AppId::new();
|
||||
let c = claims(app, &["chat.room.1", "user.notify"], 5000);
|
||||
let token = sign(key, &c);
|
||||
let got = verify(key, &token, 2000).expect("valid token verifies");
|
||||
assert_eq!(got, c);
|
||||
assert!(got.allows_topic("chat.room.1"));
|
||||
assert!(!got.allows_topic("chat.room.2"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tampered_payload_fails() {
|
||||
let key = b"super-secret-key-bytes-0123456789";
|
||||
let app = AppId::new();
|
||||
let token = sign(key, &claims(app, &["t"], 5000));
|
||||
// Flip a character in the payload half.
|
||||
let (payload, sig) = token.split_once('.').unwrap();
|
||||
let mut bytes = payload.as_bytes().to_vec();
|
||||
bytes[0] ^= 0x01;
|
||||
let tampered = format!("{}.{sig}", String::from_utf8_lossy(&bytes));
|
||||
assert_eq!(verify(key, &tampered, 2000), Err(TokenError::BadSignature));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tampered_signature_fails() {
|
||||
let key = b"super-secret-key-bytes-0123456789";
|
||||
let app = AppId::new();
|
||||
let token = sign(key, &claims(app, &["t"], 5000));
|
||||
let (payload, _sig) = token.split_once('.').unwrap();
|
||||
// A valid-base64 but wrong signature.
|
||||
let bogus = URL_SAFE_NO_PAD.encode([0u8; 32]);
|
||||
let tampered = format!("{payload}.{bogus}");
|
||||
assert_eq!(verify(key, &tampered, 2000), Err(TokenError::BadSignature));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn different_key_fails() {
|
||||
let app = AppId::new();
|
||||
let token = sign(
|
||||
b"key-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
&claims(app, &["t"], 5000),
|
||||
);
|
||||
assert_eq!(
|
||||
verify(b"key-bbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", &token, 2000),
|
||||
Err(TokenError::BadSignature)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn expired_token_fails_at_expiry_check() {
|
||||
let key = b"super-secret-key-bytes-0123456789";
|
||||
let app = AppId::new();
|
||||
let token = sign(key, &claims(app, &["t"], 5000));
|
||||
// now == exp → expired (>= boundary).
|
||||
assert_eq!(verify(key, &token, 5000), Err(TokenError::Expired));
|
||||
assert_eq!(verify(key, &token, 9999), Err(TokenError::Expired));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_token_fails() {
|
||||
let key = b"super-secret-key-bytes-0123456789";
|
||||
assert_eq!(verify(key, "no-dot-here", 0), Err(TokenError::Malformed));
|
||||
assert_eq!(verify(key, "a.b.c", 0), Err(TokenError::Malformed));
|
||||
}
|
||||
}
|
||||
@@ -78,6 +78,39 @@ impl DocsEventOp {
|
||||
}
|
||||
}
|
||||
|
||||
/// Operations a files trigger can fire on. v1.1.5. Stored as a
|
||||
/// lowercase string in `files_trigger_details.ops` (Postgres `text[]`).
|
||||
/// CRUD verbs (`create`) mirror `DocsEventOp`, distinct from KV's
|
||||
/// set/upsert flavour.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum FilesEventOp {
|
||||
Create,
|
||||
Update,
|
||||
Delete,
|
||||
}
|
||||
|
||||
impl FilesEventOp {
|
||||
#[must_use]
|
||||
pub const fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Self::Create => "create",
|
||||
Self::Update => "update",
|
||||
Self::Delete => "delete",
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn from_wire(s: &str) -> Option<Self> {
|
||||
match s {
|
||||
"create" => Some(Self::Create),
|
||||
"update" => Some(Self::Update),
|
||||
"delete" => Some(Self::Delete),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Discriminated description of a triggering event. Lifted from the
|
||||
/// outbox row's payload at dispatch time. Each variant carries the
|
||||
/// fields the corresponding `ctx.event` shape exposes to the script.
|
||||
@@ -111,6 +144,48 @@ pub enum TriggerEvent {
|
||||
prev_data: Option<serde_json::Value>,
|
||||
},
|
||||
|
||||
/// A cron schedule fired this handler. v1.1.4. Carries the
|
||||
/// schedule + timezone the trigger was configured with, the
|
||||
/// canonical cron moment (`scheduled_at`, the instant the
|
||||
/// expression *meant*), and when the scheduler actually enqueued
|
||||
/// the fire (`fired_at`). Surfaced to scripts as `ctx.event.cron`.
|
||||
Cron {
|
||||
schedule: String,
|
||||
timezone: String,
|
||||
scheduled_at: DateTime<Utc>,
|
||||
fired_at: DateTime<Utc>,
|
||||
},
|
||||
|
||||
/// A files create / update / delete fired this handler. v1.1.5.
|
||||
/// Carries the affected file's **metadata only** — never the blob
|
||||
/// bytes (files are too big to ship through trigger payloads). A
|
||||
/// handler that wants the bytes calls
|
||||
/// `files::collection(c).get(id)` itself. `prev` is the prior
|
||||
/// metadata for update (and the deleted-row metadata for delete);
|
||||
/// absent on create. Surfaced to scripts as `ctx.event.files`.
|
||||
Files {
|
||||
op: FilesEventOp,
|
||||
collection: String,
|
||||
/// UUID as string — Rhai sees it as a string.
|
||||
id: String,
|
||||
name: String,
|
||||
content_type: String,
|
||||
size: u64,
|
||||
/// Lowercase hex SHA-256.
|
||||
checksum: String,
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
prev: Option<serde_json::Value>,
|
||||
},
|
||||
|
||||
/// A durable pub/sub publish fired this handler. v1.1.5. Carries
|
||||
/// the topic, the JSON-decoded message, and the publish instant.
|
||||
/// Surfaced to scripts as `ctx.event.pubsub`.
|
||||
Pubsub {
|
||||
topic: String,
|
||||
message: serde_json::Value,
|
||||
published_at: DateTime<Utc>,
|
||||
},
|
||||
|
||||
/// A dead-letter row fired this handler. The original event is
|
||||
/// nested verbatim plus the dead-letter metadata the design notes
|
||||
/// §4 require.
|
||||
@@ -135,6 +210,9 @@ impl TriggerEvent {
|
||||
match self {
|
||||
Self::Kv { .. } => "kv",
|
||||
Self::Docs { .. } => "docs",
|
||||
Self::Cron { .. } => "cron",
|
||||
Self::Files { .. } => "files",
|
||||
Self::Pubsub { .. } => "pubsub",
|
||||
Self::DeadLetter { .. } => "dead_letter",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,7 +33,32 @@ pub const PRODUCT_VERSION: &str = env!("CARGO_PKG_VERSION");
|
||||
/// app. Cross-app imports are unreachable (the `name` argument carries
|
||||
/// no `app_id`). Modules expose `fn`/`const` declarations only;
|
||||
/// top-level statements are rejected at create-time.
|
||||
pub const SDK_VERSION: &str = "1.4";
|
||||
///
|
||||
/// 1.5 additions (v1.1.4): `http::{get,post,put,patch,delete,head,
|
||||
/// post_form,request}` for outbound HTTP from scripts (guarded by an
|
||||
/// SSRF deny-list on the resolved IP); `ctx.event.cron` for cron-trigger
|
||||
/// handlers (carries `schedule`, `timezone`, `scheduled_at`, `fired_at`).
|
||||
/// The `Services` bundle gains `http: Arc<dyn HttpService>`.
|
||||
///
|
||||
/// 1.6 additions (v1.1.5):
|
||||
/// `files::collection(name).{create,head,get,update,delete,list}` —
|
||||
/// filesystem-backed blob storage (blobs in/out; metadata maps;
|
||||
/// checksum-verified reads) with `ctx.event.files` for files-trigger
|
||||
/// handlers (metadata only, never the bytes); and
|
||||
/// `pubsub::publish_durable(topic, message)` — durable pub/sub with
|
||||
/// publish-time fan-out and `ctx.event.pubsub` for pubsub-trigger
|
||||
/// handlers. The `Services` bundle gains `files: Arc<dyn FilesService>`
|
||||
/// and `pubsub: Arc<dyn PubsubService>`.
|
||||
///
|
||||
/// 1.7 additions (v1.1.6): `pubsub::subscriber_token(topics, ttl)` —
|
||||
/// mints an HMAC-signed realtime subscriber token for externally-
|
||||
/// subscribable topics (requires an authenticated principal). This is
|
||||
/// the only new script-visible surface; the rest of the release is
|
||||
/// server-side (the SSE `/realtime/topics/{topic}` endpoint; the
|
||||
/// `RealtimeBroadcaster` / `RealtimeEvent` / `RealtimeAuthority` traits;
|
||||
/// the `topics` registry + admin endpoints; the `@picloud/client`
|
||||
/// TypeScript package).
|
||||
pub const SDK_VERSION: &str = "1.7";
|
||||
|
||||
/// HTTP API major version. Appears in URL paths as `/api/v{N}/...`.
|
||||
/// Bump (new integer + new URL prefix) when the request/response
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "picloud-dashboard",
|
||||
"version": "0.9.0",
|
||||
"version": "0.12.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
|
||||
@@ -211,6 +211,87 @@ export interface DeadLetterRow {
|
||||
resolution: 'replayed' | 'ignored' | 'handled_by_script' | 'handler_failed' | null;
|
||||
}
|
||||
|
||||
export type TriggerKind = 'kv' | 'docs' | 'dead_letter' | 'cron' | 'files' | 'pubsub';
|
||||
export type TriggerDispatchMode = 'sync' | 'async';
|
||||
|
||||
/// Per-kind detail, tagged by `kind` to match the Rust serde shape.
|
||||
export type TriggerDetails =
|
||||
| { kind: 'kv'; collection_glob: string; ops: string[] }
|
||||
| { kind: 'docs'; collection_glob: string; ops: string[] }
|
||||
| { kind: 'dead_letter'; source_filter?: string; trigger_id_filter?: string; script_id_filter?: string }
|
||||
| { kind: 'cron'; schedule: string; timezone: string; last_fired_at?: string | null }
|
||||
| { kind: 'files'; collection_glob: string; ops: string[] }
|
||||
| { kind: 'pubsub'; topic_pattern: string };
|
||||
|
||||
/// v1.1.5 file metadata as the admin files endpoint returns it.
|
||||
export interface FileMeta {
|
||||
id: string;
|
||||
collection: string;
|
||||
name: string;
|
||||
content_type: string;
|
||||
size: number;
|
||||
checksum: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface Trigger {
|
||||
id: string;
|
||||
app_id: string;
|
||||
script_id: string;
|
||||
kind: TriggerKind;
|
||||
enabled: boolean;
|
||||
dispatch_mode: TriggerDispatchMode;
|
||||
retry_max_attempts: number;
|
||||
retry_backoff: 'exponential' | 'linear' | 'constant';
|
||||
retry_base_ms: number;
|
||||
registered_by_principal: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
details: TriggerDetails;
|
||||
}
|
||||
|
||||
export interface CreateCronTriggerInput {
|
||||
script_id: string;
|
||||
schedule: string;
|
||||
timezone: string;
|
||||
dispatch_mode?: TriggerDispatchMode;
|
||||
retry_max_attempts?: number;
|
||||
retry_backoff?: 'exponential' | 'linear' | 'constant';
|
||||
retry_base_ms?: number;
|
||||
}
|
||||
|
||||
export interface CreatePubsubTriggerInput {
|
||||
script_id: string;
|
||||
topic_pattern: string;
|
||||
dispatch_mode?: TriggerDispatchMode;
|
||||
retry_max_attempts?: number;
|
||||
retry_backoff?: 'exponential' | 'linear' | 'constant';
|
||||
retry_base_ms?: number;
|
||||
}
|
||||
|
||||
// v1.1.6 — externally-subscribable realtime topics.
|
||||
export type TopicAuthMode = 'public' | 'token';
|
||||
|
||||
export interface Topic {
|
||||
name: string;
|
||||
external_subscribable: boolean;
|
||||
auth_mode: TopicAuthMode;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
}
|
||||
|
||||
export interface CreateTopicInput {
|
||||
name: string;
|
||||
external_subscribable: boolean;
|
||||
auth_mode: TopicAuthMode;
|
||||
}
|
||||
|
||||
export interface UpdateTopicInput {
|
||||
external_subscribable?: boolean;
|
||||
auth_mode?: TopicAuthMode;
|
||||
}
|
||||
|
||||
export interface ExecutionResult {
|
||||
status: number;
|
||||
headers: Record<string, string>;
|
||||
@@ -572,6 +653,67 @@ export const api = {
|
||||
)
|
||||
},
|
||||
|
||||
triggers: {
|
||||
list: (idOrSlug: string) =>
|
||||
adminRequest<{ triggers: Trigger[] }>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/triggers`
|
||||
),
|
||||
createCron: (idOrSlug: string, input: CreateCronTriggerInput) =>
|
||||
adminRequest<Trigger>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/triggers/cron`,
|
||||
{ method: 'POST', body: JSON.stringify(input) }
|
||||
),
|
||||
createPubsub: (idOrSlug: string, input: CreatePubsubTriggerInput) =>
|
||||
adminRequest<Trigger>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/triggers/pubsub`,
|
||||
{ method: 'POST', body: JSON.stringify(input) }
|
||||
),
|
||||
remove: (idOrSlug: string, triggerId: string) =>
|
||||
adminRequest<null>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/triggers/${triggerId}`,
|
||||
{ method: 'DELETE' }
|
||||
)
|
||||
},
|
||||
|
||||
topics: {
|
||||
list: (idOrSlug: string) =>
|
||||
adminRequest<{ topics: Topic[] }>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/topics`
|
||||
),
|
||||
create: (idOrSlug: string, input: CreateTopicInput) =>
|
||||
adminRequest<Topic>(`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/topics`, {
|
||||
method: 'POST',
|
||||
body: JSON.stringify(input)
|
||||
}),
|
||||
update: (idOrSlug: string, name: string, input: UpdateTopicInput) =>
|
||||
adminRequest<Topic>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/topics/${encodeURIComponent(name)}`,
|
||||
{ method: 'PATCH', body: JSON.stringify(input) }
|
||||
),
|
||||
remove: (idOrSlug: string, name: string) =>
|
||||
adminRequest<null>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/topics/${encodeURIComponent(name)}`,
|
||||
{ method: 'DELETE' }
|
||||
)
|
||||
},
|
||||
|
||||
files: {
|
||||
list: (idOrSlug: string, collection: string, opts: { cursor?: string; limit?: number } = {}) => {
|
||||
const params = new URLSearchParams();
|
||||
params.set('collection', collection);
|
||||
if (opts.cursor) params.set('cursor', opts.cursor);
|
||||
if (opts.limit !== undefined) params.set('limit', String(opts.limit));
|
||||
return adminRequest<{ files: FileMeta[]; next_cursor: string | null }>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/files?${params.toString()}`
|
||||
);
|
||||
},
|
||||
remove: (idOrSlug: string, collection: string, fileId: string) =>
|
||||
adminRequest<null>(
|
||||
`/api/v1/admin/apps/${encodeURIComponent(idOrSlug)}/files/${encodeURIComponent(collection)}/${fileId}`,
|
||||
{ method: 'DELETE' }
|
||||
)
|
||||
},
|
||||
|
||||
execute: async (
|
||||
id: string,
|
||||
body: unknown,
|
||||
|
||||
@@ -10,7 +10,10 @@
|
||||
type AppDomain,
|
||||
type AppMemberDto,
|
||||
type AppRole,
|
||||
type Script
|
||||
type Script,
|
||||
type Trigger,
|
||||
type Topic,
|
||||
type TopicAuthMode
|
||||
} from '$lib/api';
|
||||
import CodeEditor from '$lib/CodeEditor.svelte';
|
||||
import ConfirmModal from '$lib/ConfirmModal.svelte';
|
||||
@@ -24,7 +27,26 @@
|
||||
const SAMPLE_SOURCE =
|
||||
'#{\n statusCode: 200,\n body: #{ ok: true, echo: ctx.request.body }\n}';
|
||||
|
||||
type Tab = 'scripts' | 'domains' | 'members' | 'settings';
|
||||
type Tab = 'scripts' | 'domains' | 'members' | 'settings' | 'triggers' | 'topics';
|
||||
|
||||
// Common IANA timezones offered in the cron form dropdown. Not
|
||||
// exhaustive — the backend validates any IANA name via chrono-tz.
|
||||
const COMMON_TIMEZONES = [
|
||||
'UTC',
|
||||
'America/Los_Angeles',
|
||||
'America/Denver',
|
||||
'America/Chicago',
|
||||
'America/New_York',
|
||||
'America/Sao_Paulo',
|
||||
'Europe/London',
|
||||
'Europe/Berlin',
|
||||
'Europe/Paris',
|
||||
'Europe/Moscow',
|
||||
'Asia/Kolkata',
|
||||
'Asia/Shanghai',
|
||||
'Asia/Tokyo',
|
||||
'Australia/Sydney'
|
||||
];
|
||||
|
||||
let slug = $derived(page.params.slug ?? '');
|
||||
let app = $state<App | null>(null);
|
||||
@@ -91,6 +113,183 @@
|
||||
let removingDomain = $state(false);
|
||||
let removeDomainError = $state<string | null>(null);
|
||||
|
||||
// Triggers tab (v1.1.4 — cron triggers). Admin-gated, like Members.
|
||||
let triggers = $state<Trigger[]>([]);
|
||||
let createCronScriptId = $state('');
|
||||
let createCronSchedule = $state('0 0 9 * * MON-FRI');
|
||||
let createCronTimezone = $state('UTC');
|
||||
let creatingCron = $state(false);
|
||||
let createCronError = $state<string | null>(null);
|
||||
// Pub/Sub triggers (v1.1.5).
|
||||
let createPubsubScriptId = $state('');
|
||||
let createPubsubTopic = $state('');
|
||||
let creatingPubsub = $state(false);
|
||||
let createPubsubError = $state<string | null>(null);
|
||||
let triggerToRemove = $state<Trigger | null>(null);
|
||||
let removingTrigger = $state(false);
|
||||
// Endpoint scripts only — modules can't be trigger targets.
|
||||
const endpointScripts = $derived(scripts.filter((s) => s.kind === 'endpoint'));
|
||||
|
||||
async function loadTriggers(idOrSlug: string) {
|
||||
try {
|
||||
const r = await api.triggers.list(idOrSlug);
|
||||
triggers = r.triggers;
|
||||
} catch {
|
||||
triggers = [];
|
||||
}
|
||||
}
|
||||
|
||||
async function submitCreateCron(e: SubmitEvent) {
|
||||
e.preventDefault();
|
||||
if (!app) return;
|
||||
creatingCron = true;
|
||||
createCronError = null;
|
||||
try {
|
||||
await api.triggers.createCron(app.id, {
|
||||
script_id: createCronScriptId,
|
||||
schedule: createCronSchedule.trim(),
|
||||
timezone: createCronTimezone
|
||||
});
|
||||
createCronScriptId = '';
|
||||
await loadTriggers(app.id);
|
||||
} catch (err) {
|
||||
createCronError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
creatingCron = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function submitCreatePubsub(e: SubmitEvent) {
|
||||
e.preventDefault();
|
||||
if (!app) return;
|
||||
creatingPubsub = true;
|
||||
createPubsubError = null;
|
||||
try {
|
||||
await api.triggers.createPubsub(app.id, {
|
||||
script_id: createPubsubScriptId,
|
||||
topic_pattern: createPubsubTopic.trim()
|
||||
});
|
||||
createPubsubScriptId = '';
|
||||
createPubsubTopic = '';
|
||||
await loadTriggers(app.id);
|
||||
} catch (err) {
|
||||
createPubsubError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
creatingPubsub = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function confirmRemoveTrigger() {
|
||||
if (!app || !triggerToRemove) return;
|
||||
removingTrigger = true;
|
||||
try {
|
||||
await api.triggers.remove(app.id, triggerToRemove.id);
|
||||
triggerToRemove = null;
|
||||
await loadTriggers(app.id);
|
||||
} catch (err) {
|
||||
createCronError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
removingTrigger = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Topics tab (v1.1.6 — externally-subscribable realtime topics). Admin-gated.
|
||||
let topics = $state<Topic[]>([]);
|
||||
let createTopicName = $state('');
|
||||
let createTopicExternal = $state(false);
|
||||
let createTopicAuthMode = $state<TopicAuthMode>('public');
|
||||
let creatingTopic = $state(false);
|
||||
let createTopicError = $state<string | null>(null);
|
||||
// Edit modal.
|
||||
let topicToEdit = $state<Topic | null>(null);
|
||||
let editTopicExternal = $state(false);
|
||||
let editTopicAuthMode = $state<TopicAuthMode>('public');
|
||||
let savingTopic = $state(false);
|
||||
let editTopicError = $state<string | null>(null);
|
||||
// Flipping internal → external is the security-sensitive change.
|
||||
const editFlipToExternal = $derived(
|
||||
!!topicToEdit && !topicToEdit.external_subscribable && editTopicExternal
|
||||
);
|
||||
// Delete confirm.
|
||||
let topicToRemove = $state<Topic | null>(null);
|
||||
let removingTopic = $state(false);
|
||||
|
||||
async function loadTopics(idOrSlug: string) {
|
||||
try {
|
||||
const r = await api.topics.list(idOrSlug);
|
||||
topics = r.topics;
|
||||
} catch {
|
||||
topics = [];
|
||||
}
|
||||
}
|
||||
|
||||
async function submitCreateTopic(e: SubmitEvent) {
|
||||
e.preventDefault();
|
||||
if (!app) return;
|
||||
creatingTopic = true;
|
||||
createTopicError = null;
|
||||
try {
|
||||
await api.topics.create(app.id, {
|
||||
name: createTopicName.trim(),
|
||||
external_subscribable: createTopicExternal,
|
||||
auth_mode: createTopicAuthMode
|
||||
});
|
||||
createTopicName = '';
|
||||
createTopicExternal = false;
|
||||
createTopicAuthMode = 'public';
|
||||
await loadTopics(app.id);
|
||||
} catch (err) {
|
||||
createTopicError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
creatingTopic = false;
|
||||
}
|
||||
}
|
||||
|
||||
function openEditTopic(t: Topic) {
|
||||
topicToEdit = t;
|
||||
editTopicExternal = t.external_subscribable;
|
||||
editTopicAuthMode = t.auth_mode;
|
||||
editTopicError = null;
|
||||
}
|
||||
|
||||
async function confirmEditTopic() {
|
||||
if (!app || !topicToEdit) return;
|
||||
savingTopic = true;
|
||||
editTopicError = null;
|
||||
try {
|
||||
await api.topics.update(app.id, topicToEdit.name, {
|
||||
external_subscribable: editTopicExternal,
|
||||
auth_mode: editTopicAuthMode
|
||||
});
|
||||
topicToEdit = null;
|
||||
await loadTopics(app.id);
|
||||
} catch (err) {
|
||||
editTopicError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
savingTopic = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function confirmRemoveTopic() {
|
||||
if (!app || !topicToRemove) return;
|
||||
removingTopic = true;
|
||||
try {
|
||||
await api.topics.remove(app.id, topicToRemove.name);
|
||||
topicToRemove = null;
|
||||
await loadTopics(app.id);
|
||||
} catch (err) {
|
||||
createTopicError =
|
||||
err instanceof ApiError ? err.message : err instanceof Error ? err.message : String(err);
|
||||
} finally {
|
||||
removingTopic = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Members tab
|
||||
let eligibleUsers = $state<AdminDto[]>([]);
|
||||
let eligibleLoadError = $state<string | null>(null);
|
||||
@@ -131,7 +330,12 @@
|
||||
loadDeadLetterCount(app.id)
|
||||
];
|
||||
if (canAdmin) {
|
||||
loaders.push(loadMembers(app.id), loadEligibleUsers());
|
||||
loaders.push(
|
||||
loadMembers(app.id),
|
||||
loadEligibleUsers(),
|
||||
loadTriggers(app.id),
|
||||
loadTopics(app.id)
|
||||
);
|
||||
}
|
||||
await Promise.all(loaders);
|
||||
} catch (e) {
|
||||
@@ -398,7 +602,13 @@
|
||||
// backend still 403s the underlying calls, but no point showing an
|
||||
// empty tab.
|
||||
$effect(() => {
|
||||
if (!canAdmin && (activeTab === 'settings' || activeTab === 'members')) {
|
||||
if (
|
||||
!canAdmin &&
|
||||
(activeTab === 'settings' ||
|
||||
activeTab === 'members' ||
|
||||
activeTab === 'triggers' ||
|
||||
activeTab === 'topics')
|
||||
) {
|
||||
activeTab = 'scripts';
|
||||
}
|
||||
});
|
||||
@@ -440,11 +650,28 @@
|
||||
class:active={activeTab === 'members'}
|
||||
onclick={() => (activeTab = 'members')}>Members ({members.length})</button
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
class:active={activeTab === 'triggers'}
|
||||
onclick={() => (activeTab = 'triggers')}>Triggers ({triggers.length})</button
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
class:active={activeTab === 'topics'}
|
||||
onclick={() => (activeTab = 'topics')}>Topics ({topics.length})</button
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
class:active={activeTab === 'settings'}
|
||||
onclick={() => (activeTab = 'settings')}>Settings</button
|
||||
>
|
||||
<a
|
||||
class="tab-link"
|
||||
href="{base}/apps/{slug}/files"
|
||||
title="Files — browse and delete stored blobs by collection"
|
||||
>
|
||||
Files
|
||||
</a>
|
||||
<a
|
||||
class="tab-link"
|
||||
href="{base}/apps/{slug}/dead-letters"
|
||||
@@ -698,6 +925,212 @@
|
||||
</div>
|
||||
{/if}
|
||||
</section>
|
||||
{:else if activeTab === 'triggers' && canAdmin}
|
||||
<section>
|
||||
<h2>Cron triggers</h2>
|
||||
<p class="muted">
|
||||
Run an endpoint script on a schedule. Schedules are 6-field cron
|
||||
expressions (with seconds): <code>sec min hour day-of-month month day-of-week</code>.
|
||||
The timezone disambiguates schedules like "every weekday at 9am".
|
||||
</p>
|
||||
|
||||
<form class="create-form" onsubmit={submitCreateCron}>
|
||||
<div class="row">
|
||||
<label>
|
||||
<span>Target script</span>
|
||||
<select bind:value={createCronScriptId} required>
|
||||
<option value="" disabled>Select an endpoint script…</option>
|
||||
{#each endpointScripts as s (s.id)}
|
||||
<option value={s.id}>{s.name}</option>
|
||||
{/each}
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Schedule</span>
|
||||
<input
|
||||
bind:value={createCronSchedule}
|
||||
required
|
||||
placeholder="0 0 9 * * MON-FRI"
|
||||
/>
|
||||
</label>
|
||||
<label>
|
||||
<span>Timezone</span>
|
||||
<select bind:value={createCronTimezone}>
|
||||
{#each COMMON_TIMEZONES as tz (tz)}
|
||||
<option value={tz}>{tz}</option>
|
||||
{/each}
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
{#if endpointScripts.length === 0}
|
||||
<p class="muted small">
|
||||
This app has no endpoint scripts yet — create one first (modules
|
||||
can't be trigger targets).
|
||||
</p>
|
||||
{/if}
|
||||
{#if createCronError}
|
||||
<div class="error">{createCronError}</div>
|
||||
{/if}
|
||||
<div class="actions">
|
||||
<button type="submit" disabled={creatingCron || !createCronScriptId}>
|
||||
{creatingCron ? 'Creating…' : 'Create cron trigger'}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
<h2>Pub/Sub triggers</h2>
|
||||
<p class="muted">
|
||||
Subscribe an endpoint script to durable pub/sub messages. Topic
|
||||
patterns are an exact topic (<code>user.created</code>), a prefix
|
||||
wildcard (<code>user.*</code>), or <code>*</code> for every topic.
|
||||
</p>
|
||||
|
||||
<form class="create-form" onsubmit={submitCreatePubsub}>
|
||||
<div class="row">
|
||||
<label>
|
||||
<span>Target script</span>
|
||||
<select bind:value={createPubsubScriptId} required>
|
||||
<option value="" disabled>Select an endpoint script…</option>
|
||||
{#each endpointScripts as s (s.id)}
|
||||
<option value={s.id}>{s.name}</option>
|
||||
{/each}
|
||||
</select>
|
||||
</label>
|
||||
<label>
|
||||
<span>Topic pattern</span>
|
||||
<input bind:value={createPubsubTopic} required placeholder="user.*" />
|
||||
</label>
|
||||
</div>
|
||||
{#if createPubsubError}
|
||||
<div class="error">{createPubsubError}</div>
|
||||
{/if}
|
||||
<div class="actions">
|
||||
<button
|
||||
type="submit"
|
||||
disabled={creatingPubsub || !createPubsubScriptId || !createPubsubTopic.trim()}
|
||||
>
|
||||
{creatingPubsub ? 'Creating…' : 'Create pub/sub trigger'}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
{#if triggers.length === 0}
|
||||
<p class="muted">No triggers in this app yet.</p>
|
||||
{:else}
|
||||
<ul class="list">
|
||||
{#each triggers as t (t.id)}
|
||||
<li class="domain-row">
|
||||
<div>
|
||||
<span class="kind-badge">{t.kind}</span>
|
||||
{#if t.details.kind === 'cron'}
|
||||
<code>{t.details.schedule}</code>
|
||||
<span class="muted">— {t.details.timezone}</span>
|
||||
<span class="muted small">
|
||||
last fired: {t.details.last_fired_at ?? 'never'}
|
||||
</span>
|
||||
{:else if t.details.kind === 'kv' || t.details.kind === 'docs' || t.details.kind === 'files'}
|
||||
<code>{t.details.collection_glob}</code>
|
||||
<span class="muted">— {t.details.ops.join(', ') || 'any op'}</span>
|
||||
{:else if t.details.kind === 'pubsub'}
|
||||
<code>{t.details.topic_pattern}</code>
|
||||
{/if}
|
||||
<span class="muted small">→ {t.script_id}</span>
|
||||
</div>
|
||||
<button
|
||||
type="button"
|
||||
class="secondary danger"
|
||||
onclick={() => (triggerToRemove = t)}
|
||||
>
|
||||
Delete
|
||||
</button>
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
{/if}
|
||||
</section>
|
||||
{:else if activeTab === 'topics' && canAdmin}
|
||||
<section>
|
||||
<h2>Realtime topics</h2>
|
||||
<p class="muted">
|
||||
Pub/sub topics are <strong>internal-only</strong> by default — scripts
|
||||
subscribe via triggers, browsers can't. Register a topic here and mark it
|
||||
<strong>externally subscribable</strong> to let frontend clients connect over
|
||||
SSE at <code>/realtime/topics/<name></code>. <code>public</code> topics
|
||||
need no auth; <code>token</code> topics require a subscriber token minted by a
|
||||
script via <code>pubsub::subscriber_token</code>.
|
||||
</p>
|
||||
|
||||
<form class="create-form" onsubmit={submitCreateTopic}>
|
||||
<div class="row">
|
||||
<label class="grow">
|
||||
<span>Topic name</span>
|
||||
<input bind:value={createTopicName} required placeholder="chat-room-updates" />
|
||||
</label>
|
||||
</div>
|
||||
<label class="checkbox-row">
|
||||
<input type="checkbox" bind:checked={createTopicExternal} />
|
||||
<span>Externally subscribable (allow browser SSE clients to subscribe)</span>
|
||||
</label>
|
||||
{#if createTopicExternal}
|
||||
<fieldset class="auth-mode">
|
||||
<legend>Auth mode</legend>
|
||||
<label class="radio-row">
|
||||
<input type="radio" value="public" bind:group={createTopicAuthMode} />
|
||||
<span><strong>public</strong> — anyone with the URL can subscribe</span>
|
||||
</label>
|
||||
<label class="radio-row">
|
||||
<input type="radio" value="token" bind:group={createTopicAuthMode} />
|
||||
<span><strong>token</strong> — requires a valid subscriber token</span>
|
||||
</label>
|
||||
</fieldset>
|
||||
{/if}
|
||||
{#if createTopicError}
|
||||
<div class="error">{createTopicError}</div>
|
||||
{/if}
|
||||
<div class="actions">
|
||||
<button type="submit" disabled={creatingTopic || !createTopicName.trim()}>
|
||||
{creatingTopic ? 'Creating…' : 'Register topic'}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
|
||||
{#if topics.length === 0}
|
||||
<p class="muted">No registered topics in this app yet.</p>
|
||||
{:else}
|
||||
<ul class="list">
|
||||
{#each topics as t (t.name)}
|
||||
<li class="domain-row">
|
||||
<div>
|
||||
<code>{t.name}</code>
|
||||
{#if t.external_subscribable}
|
||||
<span class="badge badge-external" title="Browser clients can subscribe over SSE">
|
||||
external
|
||||
</span>
|
||||
<span class="badge badge-auth">{t.auth_mode}</span>
|
||||
{:else}
|
||||
<span class="badge badge-internal" title="Internal-only: scripts subscribe via triggers">
|
||||
internal
|
||||
</span>
|
||||
{/if}
|
||||
<span class="muted small">· {shortDate(t.created_at)}</span>
|
||||
</div>
|
||||
<div class="topic-actions">
|
||||
<button type="button" class="secondary" onclick={() => openEditTopic(t)}>
|
||||
Edit
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
class="secondary danger"
|
||||
onclick={() => (topicToRemove = t)}
|
||||
>
|
||||
Delete
|
||||
</button>
|
||||
</div>
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
{/if}
|
||||
</section>
|
||||
{:else if activeTab === 'settings' && canAdmin}
|
||||
<section>
|
||||
<h2>Settings</h2>
|
||||
@@ -855,6 +1288,82 @@
|
||||
{/if}
|
||||
</ConfirmModal>
|
||||
{/if}
|
||||
|
||||
{#if triggerToRemove}
|
||||
<ConfirmModal
|
||||
title="Delete trigger"
|
||||
variant="danger"
|
||||
confirmLabel="Delete trigger"
|
||||
busyLabel="Deleting…"
|
||||
busy={removingTrigger}
|
||||
onConfirm={confirmRemoveTrigger}
|
||||
onCancel={() => (triggerToRemove = null)}
|
||||
>
|
||||
<p>
|
||||
This {triggerToRemove.kind} trigger will stop firing. The target
|
||||
script is not affected.
|
||||
</p>
|
||||
</ConfirmModal>
|
||||
{/if}
|
||||
|
||||
{#if topicToEdit}
|
||||
<ConfirmModal
|
||||
title="Edit topic “{topicToEdit.name}”"
|
||||
confirmLabel="Save changes"
|
||||
busyLabel="Saving…"
|
||||
busy={savingTopic}
|
||||
onConfirm={confirmEditTopic}
|
||||
onCancel={() => (topicToEdit = null)}
|
||||
>
|
||||
<label class="checkbox-row">
|
||||
<input type="checkbox" bind:checked={editTopicExternal} />
|
||||
<span>Externally subscribable</span>
|
||||
</label>
|
||||
{#if editTopicExternal}
|
||||
<fieldset class="auth-mode">
|
||||
<legend>Auth mode</legend>
|
||||
<label class="radio-row">
|
||||
<input type="radio" value="public" bind:group={editTopicAuthMode} />
|
||||
<span><strong>public</strong> — anyone with the URL can subscribe</span>
|
||||
</label>
|
||||
<label class="radio-row">
|
||||
<input type="radio" value="token" bind:group={editTopicAuthMode} />
|
||||
<span><strong>token</strong> — requires a valid subscriber token</span>
|
||||
</label>
|
||||
</fieldset>
|
||||
{/if}
|
||||
{#if editFlipToExternal}
|
||||
<div class="warning">
|
||||
Marking <strong>{topicToEdit.name}</strong> externally-subscribable means
|
||||
anyone with the URL can subscribe to this topic (if auth_mode is
|
||||
<code>public</code>) or anyone with a valid token can subscribe (if
|
||||
auth_mode is <code>token</code>). Are you sure?
|
||||
</div>
|
||||
{/if}
|
||||
{#if editTopicError}
|
||||
<p class="modal-error">{editTopicError}</p>
|
||||
{/if}
|
||||
</ConfirmModal>
|
||||
{/if}
|
||||
|
||||
{#if topicToRemove}
|
||||
<ConfirmModal
|
||||
title="Delete topic “{topicToRemove.name}”"
|
||||
variant="danger"
|
||||
confirmLabel="Delete topic"
|
||||
busyLabel="Deleting…"
|
||||
busy={removingTopic}
|
||||
onConfirm={confirmRemoveTopic}
|
||||
onCancel={() => (topicToRemove = null)}
|
||||
>
|
||||
<p>
|
||||
Unregistering <code>{topicToRemove.name}</code> disconnects any live SSE
|
||||
subscribers immediately. Scripts can still <code>publish_durable</code> to
|
||||
it (internal triggers keep working) — it just won't be externally
|
||||
subscribable.
|
||||
</p>
|
||||
</ConfirmModal>
|
||||
{/if}
|
||||
{/if}
|
||||
|
||||
<style>
|
||||
@@ -1205,4 +1714,64 @@
|
||||
.small {
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
|
||||
.checkbox-row,
|
||||
.radio-row {
|
||||
flex-direction: row;
|
||||
align-items: center;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.checkbox-row input,
|
||||
.radio-row input {
|
||||
flex: none;
|
||||
width: auto;
|
||||
}
|
||||
|
||||
.auth-mode {
|
||||
border: 1px solid #334155;
|
||||
border-radius: 0.375rem;
|
||||
padding: 0.5rem 0.75rem;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.4rem;
|
||||
}
|
||||
|
||||
.auth-mode legend {
|
||||
font-size: 0.8rem;
|
||||
color: #94a3b8;
|
||||
padding: 0 0.3rem;
|
||||
}
|
||||
|
||||
.topic-actions {
|
||||
display: flex;
|
||||
gap: 0.5rem;
|
||||
}
|
||||
|
||||
.badge {
|
||||
display: inline-block;
|
||||
padding: 0 0.45rem;
|
||||
margin-left: 0.4rem;
|
||||
border-radius: 0.25rem;
|
||||
font-size: 0.7rem;
|
||||
font-weight: 600;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.05em;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.badge-external {
|
||||
background: #064e3b;
|
||||
color: #6ee7b7;
|
||||
}
|
||||
|
||||
.badge-internal {
|
||||
background: #334155;
|
||||
color: #cbd5e1;
|
||||
}
|
||||
|
||||
.badge-auth {
|
||||
background: #1e3a5f;
|
||||
color: #93c5fd;
|
||||
}
|
||||
</style>
|
||||
|
||||
229
dashboard/src/routes/apps/[slug]/files/+page.svelte
Normal file
229
dashboard/src/routes/apps/[slug]/files/+page.svelte
Normal file
@@ -0,0 +1,229 @@
|
||||
<script lang="ts">
|
||||
import { base } from '$app/paths';
|
||||
import { page } from '$app/state';
|
||||
import { api, ApiError, type App, type FileMeta } from '$lib/api';
|
||||
import ConfirmModal from '$lib/ConfirmModal.svelte';
|
||||
|
||||
let slug = $derived(page.params.slug ?? '');
|
||||
let app = $state<App | null>(null);
|
||||
let collection = $state('');
|
||||
let activeCollection = $state('');
|
||||
let files = $state<FileMeta[]>([]);
|
||||
let nextCursor = $state<string | null>(null);
|
||||
let loading = $state(false);
|
||||
let error = $state<string | null>(null);
|
||||
let fileToRemove = $state<FileMeta | null>(null);
|
||||
let removing = $state(false);
|
||||
|
||||
async function loadApp() {
|
||||
try {
|
||||
app = await api.apps.get(slug);
|
||||
} catch (e) {
|
||||
error = e instanceof ApiError ? e.message : String(e);
|
||||
}
|
||||
}
|
||||
|
||||
$effect(() => {
|
||||
void slug;
|
||||
void loadApp();
|
||||
});
|
||||
|
||||
async function loadFiles(cursor?: string) {
|
||||
const c = collection.trim();
|
||||
if (!c) {
|
||||
error = 'Enter a collection name to list its files.';
|
||||
return;
|
||||
}
|
||||
loading = true;
|
||||
error = null;
|
||||
try {
|
||||
const res = await api.files.list(slug, c, { cursor, limit: 100 });
|
||||
if (cursor) {
|
||||
files = [...files, ...res.files];
|
||||
} else {
|
||||
files = res.files;
|
||||
activeCollection = c;
|
||||
}
|
||||
nextCursor = res.next_cursor;
|
||||
} catch (e) {
|
||||
error = e instanceof ApiError ? e.message : String(e);
|
||||
} finally {
|
||||
loading = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function confirmRemove() {
|
||||
if (!fileToRemove) return;
|
||||
removing = true;
|
||||
try {
|
||||
await api.files.remove(slug, fileToRemove.collection, fileToRemove.id);
|
||||
files = files.filter((f) => f.id !== fileToRemove!.id);
|
||||
fileToRemove = null;
|
||||
} catch (e) {
|
||||
error = e instanceof ApiError ? e.message : String(e);
|
||||
} finally {
|
||||
removing = false;
|
||||
}
|
||||
}
|
||||
|
||||
function fmtTime(iso: string): string {
|
||||
return new Date(iso).toLocaleString();
|
||||
}
|
||||
|
||||
function fmtSize(bytes: number): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
|
||||
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
|
||||
}
|
||||
</script>
|
||||
|
||||
<svelte:head>
|
||||
<title>Files · {slug} · PiCloud</title>
|
||||
</svelte:head>
|
||||
|
||||
<div class="container">
|
||||
<header>
|
||||
<div>
|
||||
<a href="{base}/apps/{slug}" class="back">← back to {app?.name ?? slug}</a>
|
||||
<h1>Files</h1>
|
||||
<p class="subtitle">
|
||||
Browse and delete stored blobs by collection. Uploads happen from scripts via
|
||||
<code>files::collection(c).create(…)</code>.
|
||||
</p>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<form
|
||||
class="collection-form"
|
||||
onsubmit={(e) => {
|
||||
e.preventDefault();
|
||||
void loadFiles();
|
||||
}}
|
||||
>
|
||||
<label>
|
||||
<span>Collection</span>
|
||||
<input bind:value={collection} placeholder="avatars" required />
|
||||
</label>
|
||||
<button type="submit" disabled={loading || !collection.trim()}>
|
||||
{loading ? 'Loading…' : 'List files'}
|
||||
</button>
|
||||
</form>
|
||||
|
||||
{#if error}
|
||||
<div class="error">{error}</div>
|
||||
{/if}
|
||||
|
||||
{#if activeCollection}
|
||||
{#if files.length === 0 && !loading}
|
||||
<p class="muted">No files in collection <code>{activeCollection}</code>.</p>
|
||||
{:else}
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Name</th>
|
||||
<th>Content type</th>
|
||||
<th>Size</th>
|
||||
<th>Created</th>
|
||||
<th>ID</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{#each files as f (f.id)}
|
||||
<tr>
|
||||
<td>{f.name}</td>
|
||||
<td><code>{f.content_type}</code></td>
|
||||
<td>{fmtSize(f.size)}</td>
|
||||
<td>{fmtTime(f.created_at)}</td>
|
||||
<td class="mono small">{f.id}</td>
|
||||
<td>
|
||||
<button type="button" class="danger" onclick={() => (fileToRemove = f)}>
|
||||
Delete
|
||||
</button>
|
||||
</td>
|
||||
</tr>
|
||||
{/each}
|
||||
</tbody>
|
||||
</table>
|
||||
{#if nextCursor}
|
||||
<button type="button" class="secondary" onclick={() => loadFiles(nextCursor ?? undefined)}>
|
||||
Load more
|
||||
</button>
|
||||
{/if}
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
{#if fileToRemove}
|
||||
<ConfirmModal
|
||||
title="Delete file"
|
||||
variant="danger"
|
||||
confirmLabel="Delete file"
|
||||
onConfirm={confirmRemove}
|
||||
onCancel={() => (fileToRemove = null)}
|
||||
>
|
||||
<p>
|
||||
Delete <strong>{fileToRemove.name}</strong> ({fmtSize(fileToRemove.size)}) from collection
|
||||
<code>{fileToRemove.collection}</code>? This removes both the metadata row and the bytes on
|
||||
disk and cannot be undone.
|
||||
</p>
|
||||
{#if removing}<p class="muted">Deleting…</p>{/if}
|
||||
</ConfirmModal>
|
||||
{/if}
|
||||
|
||||
<style>
|
||||
.container {
|
||||
max-width: 60rem;
|
||||
margin: 0 auto;
|
||||
padding: 1.5rem;
|
||||
}
|
||||
header {
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
.back {
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
.subtitle {
|
||||
color: var(--muted, #666);
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
.collection-form {
|
||||
display: flex;
|
||||
align-items: flex-end;
|
||||
gap: 0.75rem;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
.collection-form label {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 0.25rem;
|
||||
font-size: 0.85rem;
|
||||
}
|
||||
table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
font-size: 0.9rem;
|
||||
}
|
||||
th,
|
||||
td {
|
||||
text-align: left;
|
||||
padding: 0.4rem 0.6rem;
|
||||
border-bottom: 1px solid var(--border, #e2e2e2);
|
||||
}
|
||||
.mono {
|
||||
font-family: monospace;
|
||||
}
|
||||
.small {
|
||||
font-size: 0.78rem;
|
||||
}
|
||||
.muted {
|
||||
color: var(--muted, #666);
|
||||
}
|
||||
.error {
|
||||
color: #b00020;
|
||||
margin: 0.5rem 0;
|
||||
}
|
||||
button.danger {
|
||||
color: #b00020;
|
||||
}
|
||||
</style>
|
||||
Reference in New Issue
Block a user