CHANGELOG.md: new v1.1.9 entry above v1.1.8 — covers queue::* SDK, queue:receive trigger kind, dispatcher queue arm + visibility-timeout reclaim, invoke() + invoke_async(), retry:: SDK (including the retry::run rename deviation), admin HTTP endpoints, dashboard 0.15.0, Services::new + Limits + Engine + register_all signature changes, migrations 0034/0035, F1-F5 follow-ups, env vars + SDK schema bump. No upgrade-order constraint (pure additive). HANDBACK.md: replaces v1.1.8's at repo root. 12 sections per the brief: 1. Scope coverage table — 21 line items, all ✅ 2. Queue dispatcher design (claim SQL, ack/nack/dead-letter, reclaim) 3. invoke() re-entrancy (Engine self_weak, fresh SdkCallCx, cross-app guard, depth bound) 4. retry::* (closure passing, sleep mechanics, clamping, jitter) 5. Dashboard notes 6. F1-F5 implementation per follow-up 7. Deviations beyond the brief — 7 numbered: D1: retry::with → retry::run (both with/call are Rhai reserved) D2: Trait move skipped (Engine in scope, AST cache loss deferred) D3: Retry columns on parent only (avoids duplicating source of truth) D4: Limits::trigger_depth_max mirrored from TriggerConfig D5: One-consumer-per-queue via pg_advisory_xact_lock D6: invoke() exposed globally (not invoke::*) D7: invoke_async runs once 8. Verification with LITERAL output (fmt + clippy + test totals) 9. Open questions for the reviewer (4) 10. Latent findings (lints surfaced + fixed; no carry-forward from main) 11. Deferred items (nothing slipped; standing v1.2+ list) 12. Known limitations (closures across invoke, in-process only, etc.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1013 lines
51 KiB
Markdown
1013 lines
51 KiB
Markdown
# PiCloud Changelog
|
||
|
||
## v1.1.9 — Durable Queues & Function Composition (unreleased)
|
||
|
||
Per-app durable queues + same-app function composition + caller-
|
||
controlled retries. The final v1.1.x release before the v1.2 phase
|
||
milestone. **No upgrade-order constraint** — pure additive surface,
|
||
no destructive migrations.
|
||
|
||
### Added — `queue::*` SDK (durable per-app named queues)
|
||
|
||
- **`queue::enqueue(name, message)` / `queue::enqueue(name, message, opts)`**
|
||
— write one message onto a per-app named queue. `opts` is a Map
|
||
with `delay_ms` (defer delivery until `NOW() + delay`) and
|
||
`max_attempts` (clamped to `[1, 20]`; default 3). Message is any
|
||
JSON-serializable value — Maps, Arrays, scalars, and Blobs (which
|
||
base64-encode at any depth, mirroring `pubsub::publish_durable`).
|
||
FnPtr / closures are rejected at the bridge.
|
||
- **`queue::depth(name)` / `queue::depth_pending(name)`** — read-only
|
||
inspection. `depth` is `COUNT(*)`; `depth_pending` filters to
|
||
`claim_token IS NULL AND (deliver_after IS NULL OR deliver_after <= NOW())`.
|
||
- Identity tuple is `(app_id, queue_name)`. Queue names are implicit
|
||
— no queue registry table, no separate "create queue" ceremony. A
|
||
queue exists once the first message is enqueued under that name.
|
||
- No `peek` / `dequeue` / `purge` script-side surface. Consumers are
|
||
triggers; exposing manual dequeue would force the platform to
|
||
surface visibility-timeout semantics into script-land.
|
||
- New capability `AppQueueEnqueue(AppId)` (script:write scope, editor+).
|
||
|
||
### Added — `queue:receive` trigger kind
|
||
|
||
- **Exactly one consumer per `(app_id, queue_name)`** — enforced via
|
||
`pg_advisory_xact_lock(hash(app_id || queue_name))` + a SELECT-then-
|
||
INSERT in one transaction (a partial unique index can't span the
|
||
parent's `app_id` from the detail table). The second create returns
|
||
`422 Invalid` with the message `queue 'X' already has a consumer
|
||
trigger; remove the existing one first`.
|
||
- Per-trigger `visibility_timeout_secs` (clamped `[5, 3600]`; default
|
||
via `PICLOUD_QUEUE_DEFAULT_VISIBILITY_TIMEOUT_SECS`). The dispatcher
|
||
treats this as the lease window; a handler that holds the claim
|
||
past this point loses it to the periodic reclaim task.
|
||
- Retry policy lives on the parent `triggers` row (the same
|
||
`retry_max_attempts` / `retry_backoff` / `retry_base_ms` every
|
||
other trigger kind reads); no duplication on the detail row.
|
||
- Surfaced to handler scripts as `ctx.event.queue` —
|
||
`{ queue_name, message, enqueued_at, attempt, message_id }` — plus
|
||
`ctx.event.source = "queue"` and `ctx.event.op = "receive"`.
|
||
- Trigger executes as the principal that **registered** it (matches
|
||
design notes §4 — every trigger kind does this).
|
||
|
||
### Added — Dispatcher queue arm + visibility-timeout reclaim task
|
||
|
||
- The queue table IS the outbox for queue semantics. The dispatcher
|
||
doesn't double-buffer through `outbox`; it claims directly from
|
||
`queue_messages` via the single-round-trip
|
||
`UPDATE … WHERE id = (SELECT id … FOR UPDATE SKIP LOCKED LIMIT 1)
|
||
RETURNING …`. `SKIP LOCKED` keeps concurrent dispatchers (cluster
|
||
mode v1.3+) safe.
|
||
- Per tick (every 100ms), interleaved with the existing outbox arm:
|
||
list every enabled `queue:receive` consumer and try one claim each.
|
||
Bounded by the registered-consumer count, so one busy queue can't
|
||
starve the rest.
|
||
- **Auto-ack**: handler returns successfully → `DELETE FROM queue_messages
|
||
WHERE id = $1 AND claim_token = $2`. The token in the WHERE is the
|
||
leasing guarantee.
|
||
- **Auto-nack**: handler throws → if `attempt < max_attempts`, clear
|
||
the claim and set `deliver_after = NOW() + compute_backoff(...)`;
|
||
else, write a `dead_letters` row (the existing `fan_out_dead_letter`
|
||
path then fires registered `dead_letter` triggers off it) and delete
|
||
the queue row, in one transaction.
|
||
- **Visibility-timeout reclaim**: a separate `tokio::spawn` task ticks
|
||
every `PICLOUD_QUEUE_RECLAIM_INTERVAL_MS` (default 30000) and clears
|
||
claims older than the per-queue `visibility_timeout_secs`. A crashed
|
||
consumer (or one whose handler hung past the visibility window) thus
|
||
loses its lease and the message becomes claimable again.
|
||
|
||
### Added — `invoke()` + `invoke_async()` (same-app function composition)
|
||
|
||
- **`invoke(target, args)`** — sync, returns the callee's response
|
||
body as a Rhai Dynamic. `target` is a string (path → route trie,
|
||
UUID → `ScriptId`, else → `(app_id, name)` lookup). Same engine
|
||
instance and `Services` are reused; only the per-call `SdkCallCx`
|
||
changes (callee gets `trigger_depth + 1` and a fresh `execution_id`,
|
||
inherits the caller's principal and `root_execution_id`).
|
||
- **`invoke_async(target, args)`** — fire-and-forget. Writes an
|
||
`OutboxSourceKind::Invoke` row the dispatcher fires once (no retry
|
||
loop — callers who want retries wrap in `retry::run`). Returns the
|
||
new `ExecutionId` as a string for caller tracking.
|
||
- **Cross-app invokes are rejected** at the `InvokeService::resolve`
|
||
layer with `InvokeError::CrossApp` ("invoke: target script belongs
|
||
to a different app"). v1.1.x maintains strict isolation; cross-app
|
||
sharing arrives in v1.3+.
|
||
- **Depth-bound**: `cx.trigger_depth + 1 > limits.trigger_depth_max`
|
||
throws "invoke: depth limit exceeded (max N)" — shared with the
|
||
trigger fan-out cap so a misbehaving recursive script can't blow
|
||
the stack.
|
||
- FnPtr / closures in `args` are rejected (closures don't survive
|
||
re-entry into a fresh `SdkCallCx`).
|
||
|
||
### Added — `retry::*` SDK (caller-controlled retry)
|
||
|
||
- **`retry::policy(opts)`** — constructs a `Policy` value (custom Rhai
|
||
type). Validates + clamps: `max_attempts ∈ [1, 20]`,
|
||
`base_ms ∈ [1, 60_000]`, `jitter_pct ∈ [0, 100]`,
|
||
`backoff ∈ {"exponential" | "linear" | "constant"}`.
|
||
- **`retry::run(policy, closure)`** — calls `closure()`; on throw,
|
||
sleeps per the policy and re-invokes; after `max_attempts`
|
||
exhausted, throws the last error. Sleeps via `tokio::time::sleep`
|
||
through `TokioHandle::block_on` — the bridge is already inside the
|
||
caller's `spawn_blocking` thread.
|
||
- **`retry::on_codes(policy, codes_array)`** — returns a new policy
|
||
with an error-string filter. When non-empty, only throws containing
|
||
one of the codes retry; others surface immediately.
|
||
- **NOTE on the brief deviation**: the brief asked for `retry::with`,
|
||
but both `with` AND `call` are Rhai reserved keywords (rejected at
|
||
the parser, before registration could matter). Shipped as
|
||
`retry::run(policy, closure)` instead. Documented in HANDBACK §7.
|
||
|
||
### Added — Admin HTTP endpoints
|
||
|
||
- `POST /api/v1/admin/apps/{id}/triggers/queue` — create a
|
||
`queue:receive` trigger. Body:
|
||
`{ script_id, queue_name, visibility_timeout_secs?, dispatch_mode?,
|
||
retry_max_attempts?, retry_backoff?, retry_base_ms? }`.
|
||
Capability: `AppManageTriggers`.
|
||
- `GET /api/v1/admin/apps/{id}/queues` — list every distinct queue
|
||
name in the app with `(total, pending, claimed)` counts.
|
||
Capability: `AppLogRead`.
|
||
- `GET /api/v1/admin/apps/{id}/queues/{queue_name}` — drill-down:
|
||
stats + registered consumer trigger (if any). Capability:
|
||
`AppLogRead`.
|
||
- No mutating queue endpoints (purge / requeue / delete-message stays
|
||
at v1.2).
|
||
|
||
### Added — Dashboard (v0.15.0)
|
||
|
||
- New `/apps/{slug}/queues` read-only list view (queue name + counts +
|
||
drilldown link) and `/apps/{slug}/queues/{name}` drill-down (depth +
|
||
registered consumer + visibility timeout + last_fired_at).
|
||
- "Queues" link added to the app-level nav between Files and Dead
|
||
letters.
|
||
- Triggers tab gains a "Queue:receive trigger (v1.1.9)" form alongside
|
||
cron / pubsub / email. Trigger list renders queue triggers with the
|
||
queue name + visibility timeout + last_fired_at.
|
||
- TypeScript types: `TriggerKind` gains `'queue'`; `TriggerDetails`
|
||
gains the `Queue` variant. `CreateQueueTriggerInput`,
|
||
`QueueSummary`, `QueueConsumer`, `QueueDetail` exported.
|
||
|
||
### Changed
|
||
|
||
- `Services::new` signature gains `queue: Arc<dyn QueueService>` and
|
||
`invoke: Arc<dyn InvokeService>` positionally after `users` (mirrors
|
||
v1.1.8's positional append of `users`). `with_noop_services` updated.
|
||
- `Limits` (`crates/executor-core/src/sandbox.rs`) gains
|
||
`trigger_depth_max: u32` (default 8). The picloud binary syncs it
|
||
from `TriggerConfig::from_env().max_trigger_depth` so the
|
||
dispatcher's trigger-depth cap and the invoke depth-bound stay
|
||
aligned (env var: `PICLOUD_MAX_TRIGGER_DEPTH`).
|
||
- `Engine` gains `set_self_weak(Weak<Engine>)` + `self_arc()`. The
|
||
picloud binary calls `engine.set_self_weak(Arc::downgrade(&engine))`
|
||
right after construction so the invoke bridge can re-enter the
|
||
engine synchronously. `Weak` so the Arc-cycle stays loose.
|
||
- `register_all` gains two parameters: `limits: Limits` and
|
||
`self_engine: Option<Arc<Engine>>`. The invoke bridge surfaces a
|
||
clear error if `self_engine` is `None` (only in harnesses that don't
|
||
wire it).
|
||
- `OutboxSourceKind` gains `Invoke`; `TriggerKind` gains `Queue`;
|
||
`TriggerDetails` gains the `Queue` variant; `TriggerEvent` gains the
|
||
`Queue` variant with `source = "queue"` discriminant.
|
||
- `ScriptRepository` gains `get_by_name(app_id, name)` for the
|
||
`invoke()` name resolution path.
|
||
- SDK schema 1.9 → 1.10.
|
||
|
||
### Migrations
|
||
|
||
- `0034_queue_messages.sql` — the `queue_messages` table with three
|
||
indexes: a partial `(app_id, queue_name, enqueued_at) WHERE
|
||
claim_token IS NULL` for the dispatch hot path, a `(app_id,
|
||
queue_name)` for the depth queries, and a partial `(claimed_at)
|
||
WHERE claim_token IS NOT NULL` for the reclaim scan.
|
||
- `0035_queue_triggers.sql` — widens `triggers.kind` to admit
|
||
`'queue'`, widens `outbox.source_kind` to admit `'invoke'`, adds the
|
||
`queue_trigger_details` table.
|
||
|
||
### F1–F5 follow-ups (carry-forward from v1.1.8)
|
||
|
||
- **F1 — integration test density**: four new DB-gated test binaries
|
||
(`crates/picloud/tests/queue_e2e.rs` ×4, `…/invoke_e2e.rs` ×4,
|
||
`…/retry_e2e.rs` ×3, `crates/manager-core/tests/migration_queue_messages.rs`
|
||
×4) + 50+ net new inline unit tests. All skip cleanly when
|
||
`DATABASE_URL` is unset; all pass against the docker-compose
|
||
postgres on `localhost:15432`.
|
||
- **F2 — schema snapshot re-blessed** via
|
||
`BLESS=1 cargo test -p picloud-manager-core --test schema_snapshot
|
||
-- --include-ignored`. Delta matches the plan exactly (no unrelated
|
||
drift).
|
||
- **F3 — fmt + clippy attestation as literal output**: see HANDBACK §8.
|
||
- **F4 — `TIMING_FLAT_DUMMY_HASH` dedup**: inline copy in
|
||
`crates/manager-core/src/auth_api.rs::login` now references the
|
||
shared `crate::auth::TIMING_FLAT_DUMMY_HASH` const. `grep -rn` for
|
||
the PHC literal returns exactly one hit.
|
||
- **F5 — clippy without cold cache**: incremental cache used; HANDBACK
|
||
§8 notes the explicit choice.
|
||
|
||
### Notes
|
||
|
||
- New env vars:
|
||
- `PICLOUD_QUEUE_RECLAIM_INTERVAL_MS` (default 30000) — visibility-
|
||
timeout reclaim cadence.
|
||
- `PICLOUD_QUEUE_DEFAULT_VISIBILITY_TIMEOUT_SECS` (default 30) —
|
||
visibility timeout fallback when the trigger create request omits
|
||
one. Per-trigger column overrides this.
|
||
- SDK schema bumped to 1.10.
|
||
- `@picloud/client` unchanged — no client-library work this release.
|
||
|
||
---
|
||
|
||
## v1.1.8 — User Management (unreleased)
|
||
|
||
Per-app data-plane user management — a `users::*` SDK for scripts
|
||
plus an admin HTTP surface and dashboard tab — together with three
|
||
load-bearing follow-ups from v1.1.7 (dropping the plaintext realtime
|
||
signing-key column, `--all-targets` clippy discipline, and adding a
|
||
session-based realtime SSE auth mode).
|
||
|
||
**UPGRADE PATH IS LOAD-BEARING.** v1.1.8 requires v1.1.7 to have
|
||
been applied first. Operators upgrading directly from v1.1.6 or
|
||
earlier MUST apply v1.1.7 (and let its startup encryption sweep
|
||
complete) before applying v1.1.8 — migration 0032's guard refuses
|
||
to drop the plaintext `realtime_signing_key` column while any row
|
||
still needs encryption.
|
||
|
||
### Added — `users::*` SDK (data-plane user management)
|
||
|
||
- **`users::create(#{...})` / `users::get(id)` / `users::find_by_email(...)`
|
||
/ `users::update(id, #{...})` / `users::delete(id)` /
|
||
`users::list(#{ "$limit", cursor })`** — CRUD on per-app end-users
|
||
(distinct from the control-plane `admin_users` table). Identity tuple
|
||
is `(app_id, user_id)`; uniqueness is on `(app_id, lower(email))` so
|
||
the same email can exist across two apps but not twice within one.
|
||
Password hash is Argon2id PHC (same algorithm as `admin_users`);
|
||
the public AppUser record never carries the hash.
|
||
- **`users::login(email, password)`** returns a session token string,
|
||
or `()` on bad credentials. The bad-email and bad-password branches
|
||
share wall-clock cost via a timing-flat dummy Argon2id verify on
|
||
email miss. **Required for security**, not polish.
|
||
- **`users::verify(token)`** returns the user on success (bumping the
|
||
sliding TTL), `()` on missing / expired / revoked / cross-app.
|
||
- **`users::logout(token)`** revokes the session row.
|
||
- `migrations/0026_app_users.sql` + `0027_app_user_sessions.sql`.
|
||
|
||
### Added — Sessions
|
||
|
||
- `app_user_sessions` table with SHA-256 token hash, sliding TTL
|
||
(`PICLOUD_APP_USER_SESSION_TTL_HOURS`, default 24h), hard cap
|
||
(`PICLOUD_APP_USER_SESSION_ABSOLUTE_HOURS`, default 720h = 30d),
|
||
and explicit `revoked_at` for logout / admin revoke-all / password-
|
||
reset side-effects. Lookups reject revoked/expired rows
|
||
immediately, before the weekly GC sweep runs.
|
||
|
||
### Added — Email verification
|
||
|
||
- **`users::send_verification_email(user_id, #{ link_base, from,
|
||
subject, body_template })`** mints a 32-byte token, stores its
|
||
SHA-256 in `app_user_email_verifications`, and dispatches via
|
||
`email::send` with `{link}` in the body substituted for
|
||
`link_base?token=<raw>`. `EmailError::NotConfigured` propagates as
|
||
`UsersError::EmailNotConfigured` so scripts already handling the
|
||
v1.1.7 email-disabled mode don't need new branches.
|
||
- **`users::verify_email(token)`** atomically consumes the one-shot
|
||
token, sets `email_verified_at = NOW()`, emits
|
||
`users::email_verified`.
|
||
- `migrations/0028_app_user_email_verifications.sql`.
|
||
|
||
### Added — Password reset
|
||
|
||
- **`users::request_password_reset(email, #{ template })`** returns
|
||
`Ok(())` regardless of whether the email matched — no
|
||
existence-leak signal in script-land. Email-not-configured does
|
||
surface (operators need to know); other email failures are silently
|
||
swallowed and logged server-side.
|
||
- **`users::complete_password_reset(token, new_password)`** atomically
|
||
consumes the token, updates the Argon2id hash, and **revokes every
|
||
active session** for the user (stale tokens shouldn't ride out the
|
||
reset). TTL `PICLOUD_APP_USER_PASSWORD_RESET_TTL_HOURS` (default 1h).
|
||
- `migrations/0029_app_user_password_resets.sql`.
|
||
|
||
### Added — Invitations
|
||
|
||
- **`users::invite(email, #{ template?, display_name?, roles? })`**
|
||
gated on `AppUsersAdmin`. Omitting the template skips the email
|
||
send so an admin can issue invitations for out-of-band delivery.
|
||
- **`users::accept_invite(token, password, display_name?)`** atomically
|
||
consumes the invitation, creates the user with pre-staged roles
|
||
applied, and mints a fresh session — caller can return both the
|
||
user and the session token in one round trip. TTL
|
||
`PICLOUD_APP_USER_INVITATION_TTL_DAYS` (default 7d).
|
||
- `migrations/0030_app_user_invitations.sql`.
|
||
|
||
### Added — Per-app roles
|
||
|
||
- **`users::add_role(id, role)` / `users::remove_role(id, role)` /
|
||
`users::has_role(id, role)`** — string-tagged, per-app. The script
|
||
app decides what `"admin"` / `"editor"` / `"viewer"` mean. Stored
|
||
in `app_user_roles` (composite PK so `add` is idempotent).
|
||
- The full `AppUser` record returned by every `users::*` getter now
|
||
carries a `roles` array.
|
||
- `migrations/0031_app_user_roles.sql`.
|
||
- **Role permission matrices / hierarchy / role registry are
|
||
explicitly v1.2 work** — v1.1.8 does not pre-commit to a model
|
||
that would lock in a wrong API.
|
||
|
||
### Added — Admin HTTP surface
|
||
|
||
- `GET / POST / PATCH / DELETE /api/v1/admin/apps/{id}/users` and
|
||
`/users/{user_id}` — list, get, create, update, delete.
|
||
- `POST /users/{user_id}/reset-password` — returns a one-shot reset
|
||
token in the response body (TTL 1h). No email is sent — that's
|
||
the SDK's `users::request_password_reset`.
|
||
- `POST /users/{user_id}/revoke-sessions` — bulk revoke for that
|
||
user.
|
||
- `GET / POST / DELETE /api/v1/admin/apps/{id}/invitations` and
|
||
`/invitations/{invite_id}` — list, create, revoke pending.
|
||
- Capability gating: `AppUsersRead` for reads, `AppUsersWrite` for
|
||
CRUD, `AppUsersAdmin` for admin-mediated verbs + invitations. All
|
||
three map to existing scopes (`script:read` / `script:write`) —
|
||
**no new scope** (seven-scope commitment).
|
||
- DTOs never include password hashes, session tokens, or unrotated
|
||
reset tokens.
|
||
|
||
### Added — Dashboard Users tab
|
||
|
||
- `apps/[slug]/users/+page.svelte` — list, create form, edit modal,
|
||
revoke-sessions, reset-password (one-shot token displayed once),
|
||
delete. Sub-route `users/invitations/+page.svelte` for pending
|
||
invitations (list, create with optional inline email template,
|
||
revoke).
|
||
- The main app-detail tab strip gains a Users entry above Files.
|
||
- `@picloud/dashboard` 0.13.0 → 0.14.0.
|
||
|
||
### Changed — Realtime: drop plaintext signing-key column (F1)
|
||
|
||
- `migrations/0032_drop_plaintext_realtime_signing_key.sql` —
|
||
guard query + `ALTER TABLE app_secrets DROP COLUMN IF EXISTS
|
||
realtime_signing_key`. The guard refuses to apply if any row still
|
||
has plaintext but no encrypted counterpart.
|
||
- v1.1.7's `migrate_plaintext_keys` startup task and its call from
|
||
`build_app` are deleted; the read path now consults only the
|
||
encrypted columns.
|
||
|
||
### Changed — Realtime: `auth_mode = 'session'` (F3)
|
||
|
||
- `migrations/0033_topics_auth_mode_session.sql` widens the
|
||
`topics_auth_mode_check` CHECK to allow `'session'` alongside
|
||
`'public'` and `'token'`.
|
||
- `RealtimeAuthorityImpl` gains a `UsersService` dependency; the
|
||
Session branch of `authorize_subscribe` delegates to
|
||
`verify_session_for_realtime(app_id, token)` (same DB lookup +
|
||
sliding-TTL bump as the SDK's `users::verify`, but app_id is
|
||
taken from the topic row, not a script).
|
||
- Dashboard Topics tab gains `session` as a third radio option.
|
||
- `token` (HMAC) validator is unchanged; both coexist per topic.
|
||
|
||
### Notes
|
||
|
||
- New env vars (all with documented defaults):
|
||
- `PICLOUD_APP_USER_SESSION_TTL_HOURS` (24)
|
||
- `PICLOUD_APP_USER_SESSION_ABSOLUTE_HOURS` (720 = 30d)
|
||
- `PICLOUD_APP_USER_VERIFICATION_TTL_HOURS` (48)
|
||
- `PICLOUD_APP_USER_PASSWORD_RESET_TTL_HOURS` (1)
|
||
- `PICLOUD_APP_USER_INVITATION_TTL_DAYS` (7)
|
||
- SDK schema 1.8 → 1.9 (additive surface: `users::*` + `session`
|
||
topic auth mode).
|
||
- Workspace version 1.1.7 → 1.1.8.
|
||
- `@picloud/client` does NOT bump in v1.1.8 — its `auth.login` /
|
||
`auth.logout` helpers already call dev-defined endpoints;
|
||
nothing to change in the client.
|
||
- New weekly GC sweep (`spawn_app_user_token_gc`) prunes expired /
|
||
revoked / consumed rows across the four new token tables.
|
||
|
||
## v1.1.7 — Configuration & Email (unreleased)
|
||
|
||
The operational-config layer: **encrypted per-app secrets**, **outbound
|
||
email**, and an **inbound email trigger** — plus the long-missing
|
||
**dead-letter handler wiring** and **at-rest encryption of the realtime
|
||
signing key**. All at-rest encryption uses a single process master key
|
||
(AES-256-GCM); key rotation is deferred to v1.2.
|
||
|
||
### Added — Encryption infrastructure
|
||
|
||
- **Process master key** from `PICLOUD_SECRET_KEY` (base64 of exactly 32
|
||
bytes). REQUIRED at startup — an unset or malformed key is fatal.
|
||
Generate one with `openssl rand -base64 32`. A deterministic in-memory
|
||
dev key is used ONLY when `PICLOUD_SECRET_KEY` is unset AND
|
||
`PICLOUD_DEV_MODE=true` (with a prominent startup warning); there is no
|
||
quiet unencrypted mode.
|
||
- **`picloud_shared::crypto`** — `encrypt`/`decrypt` envelope:
|
||
`Aes256Gcm`, 96-bit CSPRNG nonce, 128-bit auth tag appended to the
|
||
ciphertext (RustCrypto `Aead` layout). Both ciphertext and nonce are
|
||
stored.
|
||
- **Key rotation is out of scope.** Changing `PICLOUD_SECRET_KEY` between
|
||
deploys renders all existing ciphertext undecryptable. v1.2+ adds
|
||
key-version columns + a re-encryption pass.
|
||
|
||
### Added — Encrypted per-app secrets
|
||
|
||
- **`secrets::{get,set,delete,list}(name)`** SDK — collection-less,
|
||
per-app. `set` accepts a String/Map/Array (JSON-encoded then encrypted);
|
||
`get` returns the same Rhai type back; missing → `()`. 64 KB plaintext
|
||
cap (`PICLOUD_SECRET_MAX_VALUE_BYTES`). `migrations/0023_secrets.sql`.
|
||
- **Admin API** `GET/POST/DELETE /api/v1/admin/apps/{id}/secrets` — list
|
||
returns names + `updated_at` only, **never values**.
|
||
- **Dashboard Secrets tab** — list names + last-modified, create/update
|
||
(masked value with a confirm-gated reveal), delete with confirm.
|
||
- `Capability::AppSecretsRead`/`Write` (→ `script:read` / `script:write`).
|
||
No new Scope variants (seven-scope commitment). Secret writes
|
||
deliberately do **not** emit trigger events.
|
||
|
||
### Added — Outbound email
|
||
|
||
- **`email::send` / `email::send_html`** SDK over an SMTP relay
|
||
(`lettre`). Config from `PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/TLS/
|
||
TIMEOUT_SECS`; if HOST/USER/PASSWORD aren't all set the service runs in
|
||
**disabled mode** (every send throws `NotConfigured`, warned at
|
||
startup). Required `to`/`from`/`subject` + one of `text`/`html`;
|
||
RFC 5322-ish address validation; 25 MB per-message cap
|
||
(`PICLOUD_EMAIL_MAX_MESSAGE_BYTES`); `reply_to` defaults to `from`.
|
||
Per-call connection (pooling deferred to v1.2); per-app `from`
|
||
validation / SPF / DKIM are the operator's SMTP-relay concern.
|
||
- `Capability::AppEmailSend` (→ `script:write`).
|
||
|
||
### Added — Inbound email (`email:receive` trigger)
|
||
|
||
- **Webhook receiver** `POST /api/v1/email-inbound/{app_id}/{trigger_id}`
|
||
— a provider (Mailgun / Postmark / SendGrid / SES) POSTs the generic
|
||
JSON shape `{from,to[],cc[],subject,text,html,message_id}`; the
|
||
receiver verifies the optional HMAC signature, normalizes to
|
||
`TriggerEvent::Email`, and enqueues an outbox row. 202 accepted, 401
|
||
bad/missing signature, 404 missing/wrong-kind/cross-app, 422 malformed.
|
||
Handlers see `ctx.event.email`. `migrations/0024_email_triggers.sql`.
|
||
- **Admin** `POST /api/v1/admin/apps/{id}/triggers/email` +
|
||
dashboard form (with the webhook URL + expected payload). The HMAC
|
||
`inbound_secret` is stored **encrypted** via the master key (deviation
|
||
from the original plaintext design — see HANDBACK §7).
|
||
- Provider-specific payload unmarshallers + inbound attachments → v1.2.
|
||
Native SMTP listener → v1.3+.
|
||
|
||
### Security/correctness fix (retroactive) — dead_letter handlers
|
||
|
||
The `dead_letter` trigger kind has been registerable since v1.1.1 but,
|
||
due to missing dispatcher wiring (`list_matching_dead_letter` had no
|
||
production caller), handlers have **never fired**. Any deploy running
|
||
v1.1.1 through v1.1.6 with `dead_letter` triggers configured has had
|
||
silently non-functional handlers. v1.1.7 fixes the wiring; existing
|
||
`dead_letters` rows remain (no migration needed) but only NEW
|
||
dead-letter events (post-v1.1.7) trigger handlers. To process older
|
||
rows, use the existing admin replay surface to re-enqueue them.
|
||
|
||
### Changed — Realtime signing key encrypted at rest (two-phase)
|
||
|
||
`app_secrets.realtime_signing_key` was stored as 32 plaintext bytes. It
|
||
is now encrypted with the master key. `migrations/0025_encrypt_realtime_keys.sql`
|
||
adds NULL-able encrypted columns and drops `NOT NULL` on the plaintext
|
||
column; a startup task encrypts pre-existing rows; the read path prefers
|
||
the encrypted columns and falls back to plaintext during the compat
|
||
window. **v1.1.8 will drop the plaintext `realtime_signing_key`
|
||
column** — operators should upgrade through v1.1.7 (which performs the
|
||
encryption) before v1.1.8.
|
||
|
||
### Notes
|
||
|
||
- **New deps:** `aes-gcm` (RustCrypto AEAD), `lettre` (SMTP).
|
||
- **New env vars:** `PICLOUD_SECRET_KEY` (required), `PICLOUD_DEV_MODE`,
|
||
`PICLOUD_SECRET_MAX_VALUE_BYTES`, `PICLOUD_SMTP_HOST/PORT/USER/PASSWORD/
|
||
TLS/TIMEOUT_SECS`, `PICLOUD_EMAIL_MAX_MESSAGE_BYTES`.
|
||
- **SDK schema** 1.7 → 1.8; **dashboard** 0.12.0 → 0.13.0.
|
||
|
||
## v1.1.6 — Realtime Channels & Client Library (unreleased)
|
||
|
||
The first **external realtime surface** and the first **frontend
|
||
library**, co-shipped per the §5/§6 design-notes decisions. Browser
|
||
clients can subscribe over SSE to per-app pub/sub topics that have been
|
||
explicitly externalized; everything else stays internal-only. The
|
||
`@picloud/client` TypeScript package wraps typed HTTP, SSE, auth, and
|
||
React/Svelte hooks. Plus three v1.1.5 follow-ups.
|
||
|
||
### Added — Realtime
|
||
|
||
- **`topics` registry** (`migrations/0021_topics.sql`) — pub/sub topics
|
||
are internal-only by default; a `topics` row with
|
||
`external_subscribable = true` opts one into external SSE subscription.
|
||
`auth_mode` is `'public'` or `'token'`.
|
||
- **Topic admin endpoints** under `/api/v1/admin/apps/{id}/topics` —
|
||
`POST` (register), `GET` (list), `PATCH /{name}` (flip
|
||
external/auth_mode — its own audited surface), `DELETE /{name}`
|
||
(unregister + disconnect live subscribers). Gated by the new
|
||
`Capability::AppTopicManage` → `app:admin` scope (no new scope; the
|
||
seven-scope commitment holds).
|
||
- **SSE endpoint `GET /realtime/topics/{topic}`** — data-plane surface
|
||
(deliberately not under `/api/`). Resolves `Host` → app, authorizes
|
||
via the `RealtimeAuthority` (404 for missing/internal topics, 401 for
|
||
bad/absent tokens), then streams `data: {topic,message,published_at}`
|
||
events with a configurable heartbeat (`PICLOUD_REALTIME_HEARTBEAT_SEC`,
|
||
default 30). Token via `Authorization: Bearer` or `?token=`.
|
||
- **`RealtimeBroadcaster` + `RealtimeEvent` + `RealtimeAuthority`**
|
||
traits (`picloud-shared`); in-process `InProcessBroadcaster`
|
||
(`tokio::sync::broadcast`, per-channel capacity
|
||
`PICLOUD_REALTIME_BROADCAST_CAPACITY` default 64, periodic empty-channel
|
||
GC) and the DB-backed `RealtimeAuthorityImpl` (orchestrator-core /
|
||
manager-core respectively). The publish path now also fans out to
|
||
in-process SSE subscribers, best-effort, after the durable outbox
|
||
fan-out commits — a broadcast failure never fails the publish.
|
||
- **`pubsub::subscriber_token(topics, ttl)`** Rhai SDK (SDK schema
|
||
1.6 → 1.7) — mints an HMAC-SHA256 subscriber token (URL-safe
|
||
`payload.signature`) scoped to externally-subscribable topics.
|
||
Requires an authenticated principal + the pub/sub publish capability.
|
||
TTL clamped to `[10s, 24h]` (default 1h), env-overridable via
|
||
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`. Per-app signing
|
||
keys persist in the new `app_secrets` table
|
||
(`migrations/0022_app_secrets.sql`), created lazily on first mint. No
|
||
per-token revocation (rotation invalidates wholesale; short TTL is the
|
||
safety mechanism).
|
||
- **Dashboard Topics tab** — register/list/edit/delete topics with a
|
||
prominent external/internal badge, auth-mode radio (conditional on
|
||
external), and a confirmation when flipping a topic external.
|
||
|
||
### Added — `@picloud/client` (TypeScript, v1.0.0)
|
||
|
||
- New top-level package `clients/typescript/` (tsup dual ESM+CJS +
|
||
`.d.ts`, vitest). Typed HTTP via `endpoint<Req,Res>(path).get()/.post()`
|
||
with auth-token injection and structured errors; SSE `subscribe(topic,
|
||
cb, {token, onTokenExpired})` with exponential-backoff reconnect,
|
||
401 token-refresh, and `Last-Event-ID` resume; `auth.login/logout/token`
|
||
over dev-defined endpoints; React (`useTopic`/`useEndpoint` +
|
||
`PicloudProvider`) and Svelte (`topicStore`/`endpointStore`) subpath
|
||
exports. Optional zod/valibot runtime validation via a `{ parse }`
|
||
adapter (no hard dep). Hybrid model: no direct service access from the
|
||
browser.
|
||
|
||
### Changed / Fixed — v1.1.5 follow-ups
|
||
|
||
- **Empty blobs accepted** — `NewFile::validate` / `FileUpdate::validate`
|
||
no longer reject zero-length `data`; empty files are a valid stored
|
||
state (sentinels, placeholders). Non-breaking.
|
||
- **Orphan `*.tmp.*` sweeper** — a startup tokio task
|
||
(`spawn_files_orphan_sweep`) walks the files root every
|
||
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC` (default 6h) and unlinks temp
|
||
blobs older than `PICLOUD_FILES_ORPHAN_TMP_TTL_SEC` (default 1h). No DB
|
||
cross-check (that full reconciler is v1.3+).
|
||
- **Dispatcher end-to-end tests** — `crates/picloud/tests/dispatcher_e2e.rs`,
|
||
one per trigger kind (kv/docs/cron/files/pubsub/dead_letter),
|
||
DATABASE_URL-gated (skip cleanly when unset).
|
||
|
||
### Notes
|
||
|
||
- New deps: `hmac` (token signing, picloud-shared), `tokio-stream` (SSE
|
||
body stream, orchestrator-core).
|
||
- New env vars: `PICLOUD_REALTIME_HEARTBEAT_SEC`,
|
||
`PICLOUD_REALTIME_BROADCAST_CAPACITY`,
|
||
`PICLOUD_SUBSCRIBER_TOKEN_TTL_{MIN,MAX,DEFAULT}_SEC`,
|
||
`PICLOUD_FILES_ORPHAN_SWEEP_INTERVAL_SEC`,
|
||
`PICLOUD_FILES_ORPHAN_TMP_TTL_SEC`.
|
||
|
||
## v1.1.5 — Files & Pub/Sub (unreleased)
|
||
|
||
Two stateful services + two trigger kinds. **`files::*`** is
|
||
filesystem-backed blob storage (atomic writes, path-sharded layout,
|
||
single-pass SHA-256 with checksum-verified reads); the metadata row
|
||
lives in Postgres, the bytes on disk. **`pubsub::publish_durable`** is
|
||
durable pub/sub through the universal outbox, fanning out one delivery
|
||
row per matching subscriber **at publish time** inside a single
|
||
transaction. Both ride the v1.1.1 trigger framework as the fifth and
|
||
sixth concrete kinds via the established Layout-E extension pattern.
|
||
|
||
### Added
|
||
|
||
- **`files::collection(name).{create,head,get,update,delete,list}`** —
|
||
blob storage SDK. `create`/`update` take a Rhai `Blob`; `get` returns
|
||
a `Blob` (or `()` if missing); `head`/`list` return metadata maps
|
||
(`id, name, content_type, size, checksum, created_at, updated_at`).
|
||
`create`/`update`/`delete` throw on failure; `get`/`head` return `()`
|
||
for a missing file; `delete` returns a was-present bool. Missing
|
||
required field on `create` throws naming the field.
|
||
- **Atomic writes** — temp file → fsync → rename → fsync parent dir →
|
||
DB row, so a crash never leaves a readable half-written file. SHA-256
|
||
is computed in a single pass during the write; `get` re-verifies it
|
||
and surfaces `FilesError::Corrupted` (logged with the path, never
|
||
auto-deleted) on a mismatch. Shard dirs are created `0o700`.
|
||
- **`files:*` trigger kind** — `ctx.event.files` carries the metadata
|
||
only (never the bytes; a handler that wants them calls
|
||
`files::collection(c).get(id)`). `prev` is `()` on create, the prior
|
||
metadata on update, the deleted metadata on delete.
|
||
- **`pubsub::publish_durable(topic, message)`** — durable publish.
|
||
Message is any JSON-serializable Rhai value; Blobs encode as base64
|
||
(at any nesting depth). No matching subscriber → the publish succeeds
|
||
silently with zero outbox rows.
|
||
- **`pubsub:*` trigger kind** — topic patterns are exact, `<prefix>.*`,
|
||
or `*`; mid-pattern wildcards are rejected at trigger creation.
|
||
`ctx.event.pubsub` carries `topic`, `message`, `published_at`.
|
||
- **`FilesService` + `PubsubService` traits** (`picloud-shared`) +
|
||
`FsFilesRepo`/`FilesServiceImpl` and `PostgresPubsubRepo`/
|
||
`PubsubServiceImpl` (manager-core). Wired into the `Services` bundle
|
||
as `files` and `pubsub`.
|
||
- **Capabilities** `AppFilesRead`/`AppFilesWrite` → `script:read`/
|
||
`script:write`, `AppPubsubPublish` → `script:write`. No new `Scope`
|
||
variant — the seven-scope commitment holds. Script-as-gate: skipped
|
||
when the script runs unauthenticated.
|
||
- **Admin files API** (`GET`/`DELETE /apps/{id}/files`) + dashboard
|
||
Files view per app; **Pub/Sub trigger form** on the Triggers tab.
|
||
- **CI** — first `.github/workflows/ci.yml` (Postgres service, fmt +
|
||
clippy + `cargo test --workspace`); the schema-snapshot guardrail now
|
||
runs instead of being `#[ignore]`'d.
|
||
|
||
### Changed
|
||
|
||
- Workspace version: 1.1.4 → 1.1.5
|
||
- Rhai SDK version: 1.5 → 1.6
|
||
- Dashboard version: 0.10.0 → 0.11.0
|
||
- `schema_snapshot` test: no longer `#[ignore]`'d — runs against
|
||
`DATABASE_URL` when set, skips cleanly when absent.
|
||
|
||
### Migrations
|
||
|
||
- 0018_files.sql — `files` metadata table (bytes live on disk).
|
||
- 0019_files_triggers.sql — widen kind/source_kind CHECKs + add
|
||
`files_trigger_details`.
|
||
- 0020_pubsub_triggers.sql — widen kind/source_kind CHECKs + add
|
||
`pubsub_trigger_details` + partial index.
|
||
|
||
### New environment variables
|
||
|
||
- `PICLOUD_FILES_ROOT` (default `./data`)
|
||
- `PICLOUD_FILES_MAX_FILE_SIZE_BYTES` (default 100 MB)
|
||
|
||
## v1.1.4 — Outbound HTTP & Cron triggers (unreleased)
|
||
|
||
Two surfaces. **`http::*`** lets Rhai scripts make outbound HTTP
|
||
requests (Slack webhooks, Stripe, third-party REST) fronted by an SSRF
|
||
deny-list applied to the *resolved IP* (DNS-rebinding defense), with
|
||
scheme/port restrictions, request/response body caps, and a layered
|
||
timeout. **Cron triggers** add the fourth concrete kind on the v1.1.1
|
||
trigger framework: a scheduler task enqueues due triggers into the same
|
||
universal outbox the dispatcher already drains.
|
||
|
||
### Added
|
||
|
||
- **`http::{get,post,put,patch,delete,head,post_form,request}`** — outbound
|
||
HTTP SDK. Body and options are separate positional args
|
||
(`verb(url, body, opts)`); `opts` is
|
||
`{headers, timeout_ms, follow_redirects, max_redirects}` (unknown keys
|
||
throw). Body dispatch by type: Map/Array → JSON, String → text/plain,
|
||
`()` → none. Response is `#{ status, headers, body, body_raw }` with
|
||
`body` auto-parsed when the response is `application/json`. Non-2xx
|
||
does NOT throw (fetch-style); network/timeout/SSRF/size errors throw
|
||
with an `"http: …"` prefix.
|
||
- **SSRF deny-list** — applied to the resolved IP via a custom reqwest
|
||
`dns_resolver` (so it covers every redirect hop and defeats DNS
|
||
rebinding), plus a literal-IP check at URL-parse time. Blocks
|
||
loopback, RFC1918 private, link-local (incl. `169.254.169.254`),
|
||
carrier-grade NAT, multicast, reserved, IPv6 ULA/link-local/loopback,
|
||
and IPv4-mapped IPv6 (re-checked against the embedded v4 address).
|
||
The script-visible error carries a CIDR-category reason, never the IP.
|
||
`PICLOUD_HTTP_ALLOW_PRIVATE=true` disables it (dev-only; logs a startup
|
||
warning).
|
||
- **`HttpService` trait** (`picloud-shared`) + `HttpServiceImpl`
|
||
(manager-core, reqwest-backed). Wired into the `Services` bundle as
|
||
`http: Arc<dyn HttpService>`.
|
||
- **`Capability::AppHttpRequest(AppId)`** — maps to the existing
|
||
`script:write` scope (any outbound request can exfiltrate data, so the
|
||
conservative write mapping is used). No new `Scope` variant — the
|
||
seven-scope commitment holds. Script-as-gate: skipped when the script
|
||
runs unauthenticated.
|
||
- **Cron triggers** — `POST /api/v1/admin/apps/{id}/triggers/cron`
|
||
(`script_id`, `schedule`, `timezone`, optional retry overrides).
|
||
6-field cron expressions (with seconds) validated by the `cron` crate;
|
||
IANA timezones validated by `chrono-tz`. A scheduler task
|
||
(`spawn_cron_scheduler`, poll cadence `PICLOUD_CRON_TICK_INTERVAL_MS`,
|
||
default 30s) enqueues due triggers into the outbox; the existing
|
||
dispatcher delivers them. Catch-up policy: a trigger that missed N
|
||
windows fires exactly **once** on the next tick, not N times.
|
||
- **`ctx.event.cron`** — `{ schedule, timezone, scheduled_at, fired_at }`
|
||
for cron-trigger handlers (`ctx.event.source == "cron"`,
|
||
`ctx.event.op == "tick"`).
|
||
- **Dashboard Triggers tab** — admin-gated cron trigger create form
|
||
(target endpoint script, schedule, timezone dropdown) + triggers list
|
||
showing schedule / timezone / last-fired.
|
||
|
||
### Changed
|
||
|
||
- **Workspace version**: `1.1.3` → `1.1.4`.
|
||
- **Rhai SDK version**: `1.4` → `1.5` (additive — `http::*` SDK +
|
||
`ctx.event.cron`). The `Services` bundle constructor becomes
|
||
`Services::new(kv, docs, dead_letters, events, modules, http)`.
|
||
- **Dashboard version**: `0.9.0` → `0.10.0`.
|
||
- **`SdkCallCx`** — gains a `script_id` field (audit attribution + the
|
||
default outbound `User-Agent`, `picloud/<version> (script:<id>)`).
|
||
- **Rhai pin tightened** — workspace dep `rhai = "1.19"` → `rhai = "=1.24"`
|
||
so future bumps of the non-semver-stable `internals` surface are
|
||
deliberate.
|
||
- **Module backend errors redacted** — `PicloudModuleResolver` now
|
||
surfaces a stable generic (`"module backend unavailable; check server
|
||
logs"`) to scripts and logs the original at error level, instead of
|
||
leaking the backend error verbatim (see v1.1.3 follow-up).
|
||
|
||
### Migrations
|
||
|
||
- `0017_cron_triggers.sql` — widens `triggers.kind` and
|
||
`outbox.source_kind` CHECK constraints to include `'cron'`; adds
|
||
`cron_trigger_details (trigger_id, schedule, timezone, last_fired_at)`
|
||
with a `last_fired_at` index. Additive — applies cleanly on a fresh DB
|
||
and on top of the v1.1.3 schema.
|
||
|
||
### New environment variables
|
||
|
||
- `PICLOUD_HTTP_ALLOW_PRIVATE` (default false; dev-only) — disable the
|
||
SSRF deny-list.
|
||
- `PICLOUD_HTTP_MAX_REQUEST_BODY_BYTES` / `PICLOUD_HTTP_MAX_RESPONSE_BODY_BYTES`
|
||
(default 10 MB each).
|
||
- `PICLOUD_CRON_TICK_INTERVAL_MS` (default 30000) — cron scheduler poll
|
||
cadence (floored at 1s).
|
||
|
||
## v1.1.3 — Modules (unreleased)
|
||
|
||
Real per-app Rhai module system. Scripts can `import "<name>" as
|
||
<alias>;` other scripts in the same app as reusable libraries. The
|
||
v1.0 placeholder `DummyModuleResolver` is replaced by a per-call
|
||
`PicloudModuleResolver` that loads `kind = 'module'` scripts via a
|
||
new `ModuleSource` trait, compiles them into Rhai modules, caches
|
||
the compiled output, and enforces cross-app isolation, circular-
|
||
import detection, and an import-depth limit. Two LRU AST caches
|
||
(top-level script + per-module compiled module) eliminate the
|
||
per-invocation compile cost; both invalidate on `updated_at` change.
|
||
|
||
### Added
|
||
|
||
- **`scripts.kind` column** — `'endpoint' | 'module'`, default
|
||
`'endpoint'`. Endpoints handle HTTP routes / trigger events;
|
||
modules are libraries imported by other scripts. The dashboard
|
||
scripts list + script detail page surface the distinction as a
|
||
colored badge.
|
||
- **`script_imports` dep-graph table** — populated at script save-
|
||
time from the literal-path `import "<name>"` declarations in the
|
||
source. FK-CASCADE on both columns. No admin surface in v1.1.3
|
||
(drives a v1.2+ "Used by" dashboard panel and v1.3+ cluster-mode
|
||
eager invalidation).
|
||
- **`ModuleSource` trait** — `lookup(&SdkCallCx, name)`. Postgres
|
||
impl `PostgresModuleSource` in manager-core. `app_id` derived from
|
||
`cx.app_id` (cross-app isolation boundary, mirrors KV / docs).
|
||
- **`PicloudModuleResolver`** — implements `rhai::ModuleResolver`.
|
||
Per-call instance owns `Arc<SdkCallCx>`, the in-progress imports
|
||
stack, the depth counter. Bridges sync `resolve()` to async
|
||
`lookup()` via `Handle::block_on` (safe under the executor's
|
||
`spawn_blocking` wrap). Replaces `DummyModuleResolver` at line 139
|
||
of `executor-core::engine::build_engine`.
|
||
- **Module-shape validation** — `kind = 'module'` source must contain
|
||
only `fn` declarations, `const` declarations, and `import`
|
||
statements at top level (no executable expressions). Walks
|
||
`ast.statements()` via `rhai/internals`. Admin endpoint is the
|
||
primary gate; the resolver re-runs the check at load time for
|
||
defense in depth against DB-direct inserts.
|
||
- **Per-module compiled-Module cache** — `LruCache<(AppId, name),
|
||
(updated_at, Arc<rhai::Module>)>` owned by `Engine`. Invalidated
|
||
lazily on `updated_at` mismatch. Size via
|
||
`PICLOUD_MODULE_CACHE_SIZE` (default 512).
|
||
- **Top-level script AST cache** — `LruCache<ScriptId, (updated_at,
|
||
Arc<rhai::AST>)>` owned by `LocalExecutorClient`. Same staleness
|
||
semantics. Size via `PICLOUD_SCRIPT_CACHE_SIZE` (default 256).
|
||
- **`ScriptIdentity` + `ExecutorClient::execute_with_identity`** —
|
||
new method on the trait; default impl forwards to `execute` so
|
||
`RemoteExecutorClient` (and future transports) keep working.
|
||
`LocalExecutorClient` overrides it to consult the script cache and
|
||
pass the resulting `Arc<rhai::AST>` to `Engine::execute_ast`.
|
||
- **`Engine::execute_ast`** — companion to `execute` that takes a
|
||
pre-compiled AST so callers (the orchestrator) can reuse one
|
||
compile across many invocations.
|
||
- **Import depth limit** — `Limits::module_import_depth_max`
|
||
(default 8). Not script-overridable.
|
||
- **Reserved module names** — module-kind scripts cannot be named
|
||
`log`, `regex`, `random`, `time`, `json`, `base64`, `hex`, `url`,
|
||
`kv`, `docs`, `dead_letters`, `http`, `files`, `pubsub`, `secrets`,
|
||
`email`, `users`, `queue`. Defense against author confusion with
|
||
stdlib namespaces.
|
||
|
||
### Changed
|
||
|
||
- **Workspace version**: `1.1.2` → `1.1.3`.
|
||
- **Rhai SDK version**: `1.3` → `1.4` (additive — every v1.3 script
|
||
still runs unchanged; new surface: `import "<name>" as <alias>;`
|
||
for endpoint scripts that consume modules in the same app).
|
||
- **Dashboard version**: `0.8.0` → `0.9.0`. Adds kind dropdown on
|
||
script create + kind badges on the scripts list and detail page.
|
||
- **`Services` bundle** — grows a `modules: Arc<dyn ModuleSource>`
|
||
field. Constructor signature becomes
|
||
`Services::new(kv, docs, dead_letters, events, modules)`.
|
||
- **`ScriptValidator` trait** — `validate` now returns
|
||
`ValidatedScript { imports: Vec<String> }` so the repo can write
|
||
dep-graph edges in the same transaction as the script row. New
|
||
`validate_module` method enforces module-shape rules.
|
||
- **Trigger creation tightening** — `POST /api/v1/admin/apps/{id}/triggers/{kv,docs,dead_letter}`
|
||
now load the target script and reject when (1) it doesn't exist,
|
||
(2) it belongs to a different app (latent v1.1.1/v1.1.2 gap —
|
||
closed in v1.1.3), or (3) it is `kind = 'module'`.
|
||
- **Route creation** — `POST /api/v1/admin/scripts/{id}/routes`
|
||
returns 400 when the target script is `kind = 'module'`.
|
||
|
||
### Security fix
|
||
|
||
- **Cross-app trigger target (CVE-class: broken access control).** In
|
||
v1.1.1 and v1.1.2, `POST /api/v1/admin/apps/{id}/triggers/{kv,docs,dead_letter}`
|
||
validated only that the caller could manage triggers on `{id}` — it
|
||
did **not** verify that the target `script_id` belonged to that same
|
||
app. A member with trigger-management rights on app A could therefore
|
||
register a trigger in A pointing at a script owned by app B, causing
|
||
B's script to execute on A's events (a cross-app isolation break).
|
||
v1.1.3 closes this: every trigger-create handler now loads the target
|
||
script and rejects it unless `script.app_id == path app_id` (and it is
|
||
not a module). **Upgrade recommendation:** anyone running a pre-v1.1.3
|
||
multi-tenant deploy should upgrade and audit existing `triggers` rows
|
||
for any whose `script_id` resolves to a script in a different `app_id`.
|
||
|
||
### Migrations
|
||
|
||
- `0015_scripts_kind.sql` — adds `scripts.kind` with CHECK
|
||
`IN ('endpoint','module')`, composite index `(app_id, kind)`, and
|
||
a module-name shape CHECK (`^[a-zA-Z_][a-zA-Z0-9_]{0,63}$`).
|
||
- `0016_script_imports.sql` — adds the dep-graph table with FK
|
||
CASCADE on both columns, PK `(importer, imported)`, and a
|
||
reverse-edge index on `imported_script_id`.
|
||
|
||
### Downgrade caveats
|
||
|
||
Rolling back v1.1.3 → v1.1.2 with module-kind scripts present
|
||
strands them (no `kind` column means everything looks like an
|
||
endpoint; modules will then succeed as route targets and immediately
|
||
fail to execute meaningfully). Migration `0016_script_imports.sql`
|
||
is safe to drop (the table is auxiliary). `0015_scripts_kind.sql`
|
||
must be reversed by `DROP COLUMN kind` only after manually re-homing
|
||
or deleting module-kind rows.
|
||
|
||
## v1.1.2 — Documents (unreleased)
|
||
|
||
`docs::*` SDK — schemaless JSONB document storage with a first-cut
|
||
query DSL — plus `docs:*` triggers as the second concrete kind on the
|
||
v1.1.1 triggers framework. Sets the precedent for the v1.2 query DSL
|
||
expansion and `dead_letters::list`.
|
||
|
||
### Added
|
||
|
||
- **Docs store** — `docs` table keyed `(app_id, collection, id)` with
|
||
JSONB values and a GIN-on-`jsonb_path_ops` index. Rhai SDK exposes
|
||
the handle pattern:
|
||
`docs::collection(name).{create,get,find,find_one,update,delete,list}`.
|
||
Cursor-style pagination on `list`. Cross-app isolation enforced via
|
||
`cx.app_id` (never script-passed). Document envelope shape returned
|
||
by reads: `#{ id, data: #{...}, created_at, updated_at }` — explicit
|
||
metadata + user-data separation (sets precedent for v1.2
|
||
`dead_letters::list`).
|
||
- **Query DSL (v1.1.2 subset)** — implicit equality at top level
|
||
(`#{ tier: "gold" }`), operator-object form
|
||
(`#{ created_at: #{ "$gt": "..." } }`), dotted field paths up to 5
|
||
levels (`"user.email"`), and operators `$eq`/`$ne`/`$gt`/`$gte`/
|
||
`$lt`/`$lte`/`$in`. Filter modifiers `$sort` (single field) and
|
||
`$limit`. Unsupported operators (`$or`, `$regex`, etc.) reject with
|
||
a clear v1.2-pointer error.
|
||
- **Docs triggers (`docs:*`)** — `docs_trigger_details` table mirrors
|
||
`kv_trigger_details`. Admin endpoint
|
||
`POST /api/v1/admin/apps/{id}/triggers/docs` accepts the same DTO
|
||
shape as the KV endpoint with `ops` of `DocsEventOp` (create /
|
||
update / delete). Dispatcher routes `OutboxSourceKind::Docs` through
|
||
the same generic path as KV + dead-letter.
|
||
- **`ctx.event.docs.prev_data`** — change-data-capture surface for
|
||
docs trigger handlers. `prev_data` carries the document state prior
|
||
to the mutation (`None` for create), letting handlers see what
|
||
changed. The repo reads the old row in the same SQL statement as
|
||
the write so the trigger event has the prior value.
|
||
- **`Capability::AppDocsRead(AppId)`** + `AppDocsWrite(AppId)` —
|
||
granted to Viewer / Editor respectively in the per-app role table.
|
||
Same trust shape as KV's `AppKvRead` / `AppKvWrite`.
|
||
|
||
### Changed
|
||
|
||
- **Workspace version**: `1.1.1` → `1.1.2`.
|
||
- **Rhai SDK version**: `1.2` → `1.3` (additive — every v1.2 script
|
||
still runs unchanged; new surfaces: `docs::collection(name).{...}`,
|
||
`ctx.event.docs` for triggered handlers).
|
||
- **Dashboard version**: `0.7.0` → `0.8.0`. Workspace alignment; no
|
||
docs-specific UI in v1.1.2 (the dashboard's Rhai-mode hints don't
|
||
list KV completions either — focused UX pass is a separate task).
|
||
- **`Services` bundle** — grows a `docs: Arc<dyn DocsService>` field.
|
||
Constructor signature becomes
|
||
`Services::new(kv, docs, dead_letters, events)`.
|
||
- **Scope mapping**: API keys with `script:read` scope can call
|
||
`docs::find` / `get` / `list`; `script:write` can call
|
||
`docs::create` / `update` / `delete`. Same trust shape as KV —
|
||
honors the seven-scope commitment from v1.1.0.
|
||
|
||
### Migrations
|
||
|
||
- `0013_docs.sql` — `docs` table + per-`(app_id, collection)` index +
|
||
GIN-on-`jsonb_path_ops` index.
|
||
- `0014_docs_triggers.sql` — extends `triggers.kind` and
|
||
`outbox.source_kind` CHECK constraints to include `'docs'`; adds
|
||
`docs_trigger_details` table.
|
||
|
||
### Downgrade caveats
|
||
|
||
Rolling a deployment back from v1.1.2 → v1.1.1 with `docs`-source
|
||
outbox rows still queued will cause the v1.1.1 dispatcher to fail
|
||
deserialising `TriggerEvent::Docs` (`#[serde(tag = "source")]`
|
||
rejects unknown variants). Drain or delete
|
||
`outbox WHERE source_kind = 'docs'` before downgrading. Trunk-only
|
||
deployments don't hit this.
|
||
|
||
### Known limitations
|
||
|
||
- Text-lex comparison for `$gt` / `$gte` / `$lt` / `$lte` is
|
||
incorrect for unpadded numbers crossing digit-count boundaries
|
||
(`'10' < '9'` is TRUE under any text collation). Workaround:
|
||
zero-pad numeric strings. v1.2's advanced query expansion adds
|
||
numeric-aware operators.
|
||
- Concurrent `update()`s on the same doc may both emit the
|
||
pre-update `prev_data` (last-writer-wins). Inherited from KV's
|
||
`set` pattern; documented for forensic-trace use cases.
|
||
- v1.1.2 has no partial-update DSL — scripts that want partial
|
||
update do `get + modify + update`. Planned for v1.2.
|
||
|
||
## v1.1.1 — Storage & Events (unreleased)
|
||
|
||
The triggers framework — KV store + universal outbox + dispatcher +
|
||
NATS-style sync HTTP + per-route async dispatch + dead-letter
|
||
handling + dashboard surface. Every subsequent v1.1.x service module
|
||
(docs, files, pubsub, …) hangs off the dispatcher built here.
|
||
|
||
### Added
|
||
|
||
- **KV store** — `kv_entries` table keyed `(app_id, collection, key)`
|
||
with JSONB values. Rhai SDK exposes the handle pattern:
|
||
`kv::collection(name).{get,set,has,delete,list}`. Cursor-style
|
||
pagination with opaque base64 cursors. Cross-app isolation
|
||
enforced via `cx.app_id` (never script-passed).
|
||
- **Triggers framework (Layout E)** — parent `triggers` table +
|
||
per-kind detail tables (`kv_trigger_details`,
|
||
`dead_letter_trigger_details`). Trigger CRUD admin endpoints
|
||
(`/api/v1/admin/apps/{id}/triggers/{kv,dead_letter}`) +
|
||
`Capability::AppManageTriggers(AppId)`.
|
||
- **Universal outbox + dispatcher** — single tokio task that polls
|
||
the outbox via `FOR UPDATE SKIP LOCKED`, routes due rows to the
|
||
executor through the shared `ExecutionGate`. Retry with
|
||
exponential backoff + ±jitter; on exhaustion, dead-letter.
|
||
- **NATS-style sync HTTP via outbox** — `InboxRegistry` (in-process
|
||
oneshot map) lets the orchestrator await dispatcher delivery on
|
||
every sync HTTP request. Cluster mode (v1.3+) swaps this for
|
||
`LISTEN/NOTIFY` behind the same `InboxResolver` trait.
|
||
- **`dispatch_mode: async` on routes** — `POST` to a route with
|
||
`dispatch_mode = 'async'` returns `202 Accepted` immediately;
|
||
the script runs via the dispatcher (with retries / dead-letter).
|
||
- **Dead-letter handling** — separate `dead_letters` table per
|
||
design notes §4. `dead_letters::{replay,resolve}` Rhai SDK +
|
||
admin endpoints + `Capability::AppDeadLetterManage(AppId)`.
|
||
Recursion-stop rule: dead-letter handler failures annotate the
|
||
original row as `resolution = 'handler_failed'` and never produce
|
||
a new dead-letter or retry.
|
||
- **Dashboard surface for dead letters** — unresolved-count red
|
||
badge on the apps list + per-app page; per-app dead-letters list
|
||
view at `/admin/apps/{slug}/dead-letters` with Replay + Mark
|
||
resolved per-row actions and expandable payload detail.
|
||
- **`abandoned_executions` table** — forensic row written by the
|
||
dispatcher when it tries to resolve an inbox the orchestrator
|
||
already abandoned (timed out). Counter metric path reserved.
|
||
- **Trigger-depth limit** — `cx.trigger_depth > max_trigger_depth`
|
||
(default 8) skips execution + logs; does NOT dead-letter
|
||
(depth-exceeded means "you built a loop").
|
||
- **GC sweepers** — weekly retention sweeps for `dead_letters`
|
||
(30 days) and `abandoned_executions` (7 days), both with
|
||
`FOR UPDATE SKIP LOCKED` for cluster-mode safety.
|
||
- **Env-overridable trigger config** — `TriggerConfig::from_env`
|
||
reads `PICLOUD_MAX_TRIGGER_DEPTH`, `PICLOUD_TRIGGER_RETRY_*`,
|
||
`PICLOUD_DEAD_LETTER_RETENTION_DAYS`,
|
||
`PICLOUD_ABANDONED_EXECUTIONS_RETENTION_DAYS`.
|
||
|
||
### Changed
|
||
|
||
- **Workspace version**: `1.1.0` → `1.1.1`.
|
||
- **Rhai SDK version**: `1.1` → `1.2` (additive — every v1.1 script
|
||
still runs unchanged; new surfaces: `kv::*`, `dead_letters::*`,
|
||
`ctx.event` for triggered handlers).
|
||
- **Dashboard version**: `0.6.0` → `0.7.0` for the dead-letters UI.
|
||
- **`Services` bundle** — replaces v1.1.0's no-arg `Services::new()`
|
||
with explicit `Services::new(kv, dead_letters, events)`. Tests
|
||
use `Services::default()` for an all-noop bundle.
|
||
- **`SdkCallCx`** grows `is_dead_letter_handler: bool` and
|
||
`event: Option<TriggerEvent>` fields.
|
||
- **`ExecRequest`** mirrors the new `SdkCallCx` fields and grows
|
||
`event` for serializable trigger payload transport.
|
||
- **Routes table** grows `dispatch_mode TEXT NOT NULL DEFAULT 'sync'`
|
||
(CHECK in {sync, async}).
|
||
- **Schema version**: 6 → 12 (migrations 0007 through 0012).
|
||
|
||
### Migrations
|
||
|
||
- `0007_kv.sql` — `kv_entries` table + index
|
||
- `0008_triggers.sql` — `triggers` + `kv_trigger_details` +
|
||
`dead_letter_trigger_details`
|
||
- `0009_outbox.sql` — universal `outbox` table + due-row partial index
|
||
- `0010_dead_letters.sql` — `dead_letters` table + unresolved partial
|
||
index + GC index
|
||
- `0011_abandoned_executions.sql` — forensic table + GC index
|
||
- `0012_routes_dispatch_mode.sql` — `routes.dispatch_mode` column
|
||
|
||
## v1.1.0 — Foundation & Standard Library
|
||
|
||
See `docs/v1.1.x-design-notes.md` §7 for the full v1.1.x roadmap.
|