docs(v1.1.9): CHANGELOG + HANDBACK.md
CHANGELOG.md: new v1.1.9 entry above v1.1.8 — covers queue::* SDK, queue:receive trigger kind, dispatcher queue arm + visibility-timeout reclaim, invoke() + invoke_async(), retry:: SDK (including the retry::run rename deviation), admin HTTP endpoints, dashboard 0.15.0, Services::new + Limits + Engine + register_all signature changes, migrations 0034/0035, F1-F5 follow-ups, env vars + SDK schema bump. No upgrade-order constraint (pure additive). HANDBACK.md: replaces v1.1.8's at repo root. 12 sections per the brief: 1. Scope coverage table — 21 line items, all ✅ 2. Queue dispatcher design (claim SQL, ack/nack/dead-letter, reclaim) 3. invoke() re-entrancy (Engine self_weak, fresh SdkCallCx, cross-app guard, depth bound) 4. retry::* (closure passing, sleep mechanics, clamping, jitter) 5. Dashboard notes 6. F1-F5 implementation per follow-up 7. Deviations beyond the brief — 7 numbered: D1: retry::with → retry::run (both with/call are Rhai reserved) D2: Trait move skipped (Engine in scope, AST cache loss deferred) D3: Retry columns on parent only (avoids duplicating source of truth) D4: Limits::trigger_depth_max mirrored from TriggerConfig D5: One-consumer-per-queue via pg_advisory_xact_lock D6: invoke() exposed globally (not invoke::*) D7: invoke_async runs once 8. Verification with LITERAL output (fmt + clippy + test totals) 9. Open questions for the reviewer (4) 10. Latent findings (lints surfaced + fixed; no carry-forward from main) 11. Deferred items (nothing slipped; standing v1.2+ list) 12. Known limitations (closures across invoke, in-process only, etc.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
215
CHANGELOG.md
215
CHANGELOG.md
@@ -1,5 +1,220 @@
|
||||
# PiCloud Changelog
|
||||
|
||||
## v1.1.9 — Durable Queues & Function Composition (unreleased)
|
||||
|
||||
Per-app durable queues + same-app function composition + caller-
|
||||
controlled retries. The final v1.1.x release before the v1.2 phase
|
||||
milestone. **No upgrade-order constraint** — pure additive surface,
|
||||
no destructive migrations.
|
||||
|
||||
### Added — `queue::*` SDK (durable per-app named queues)
|
||||
|
||||
- **`queue::enqueue(name, message)` / `queue::enqueue(name, message, opts)`**
|
||||
— write one message onto a per-app named queue. `opts` is a Map
|
||||
with `delay_ms` (defer delivery until `NOW() + delay`) and
|
||||
`max_attempts` (clamped to `[1, 20]`; default 3). Message is any
|
||||
JSON-serializable value — Maps, Arrays, scalars, and Blobs (which
|
||||
base64-encode at any depth, mirroring `pubsub::publish_durable`).
|
||||
FnPtr / closures are rejected at the bridge.
|
||||
- **`queue::depth(name)` / `queue::depth_pending(name)`** — read-only
|
||||
inspection. `depth` is `COUNT(*)`; `depth_pending` filters to
|
||||
`claim_token IS NULL AND (deliver_after IS NULL OR deliver_after <= NOW())`.
|
||||
- Identity tuple is `(app_id, queue_name)`. Queue names are implicit
|
||||
— no queue registry table, no separate "create queue" ceremony. A
|
||||
queue exists once the first message is enqueued under that name.
|
||||
- No `peek` / `dequeue` / `purge` script-side surface. Consumers are
|
||||
triggers; exposing manual dequeue would force the platform to
|
||||
surface visibility-timeout semantics into script-land.
|
||||
- New capability `AppQueueEnqueue(AppId)` (script:write scope, editor+).
|
||||
|
||||
### Added — `queue:receive` trigger kind
|
||||
|
||||
- **Exactly one consumer per `(app_id, queue_name)`** — enforced via
|
||||
`pg_advisory_xact_lock(hash(app_id || queue_name))` + a SELECT-then-
|
||||
INSERT in one transaction (a partial unique index can't span the
|
||||
parent's `app_id` from the detail table). The second create returns
|
||||
`422 Invalid` with the message `queue 'X' already has a consumer
|
||||
trigger; remove the existing one first`.
|
||||
- Per-trigger `visibility_timeout_secs` (clamped `[5, 3600]`; default
|
||||
via `PICLOUD_QUEUE_DEFAULT_VISIBILITY_TIMEOUT_SECS`). The dispatcher
|
||||
treats this as the lease window; a handler that holds the claim
|
||||
past this point loses it to the periodic reclaim task.
|
||||
- Retry policy lives on the parent `triggers` row (the same
|
||||
`retry_max_attempts` / `retry_backoff` / `retry_base_ms` every
|
||||
other trigger kind reads); no duplication on the detail row.
|
||||
- Surfaced to handler scripts as `ctx.event.queue` —
|
||||
`{ queue_name, message, enqueued_at, attempt, message_id }` — plus
|
||||
`ctx.event.source = "queue"` and `ctx.event.op = "receive"`.
|
||||
- Trigger executes as the principal that **registered** it (matches
|
||||
design notes §4 — every trigger kind does this).
|
||||
|
||||
### Added — Dispatcher queue arm + visibility-timeout reclaim task
|
||||
|
||||
- The queue table IS the outbox for queue semantics. The dispatcher
|
||||
doesn't double-buffer through `outbox`; it claims directly from
|
||||
`queue_messages` via the single-round-trip
|
||||
`UPDATE … WHERE id = (SELECT id … FOR UPDATE SKIP LOCKED LIMIT 1)
|
||||
RETURNING …`. `SKIP LOCKED` keeps concurrent dispatchers (cluster
|
||||
mode v1.3+) safe.
|
||||
- Per tick (every 100ms), interleaved with the existing outbox arm:
|
||||
list every enabled `queue:receive` consumer and try one claim each.
|
||||
Bounded by the registered-consumer count, so one busy queue can't
|
||||
starve the rest.
|
||||
- **Auto-ack**: handler returns successfully → `DELETE FROM queue_messages
|
||||
WHERE id = $1 AND claim_token = $2`. The token in the WHERE is the
|
||||
leasing guarantee.
|
||||
- **Auto-nack**: handler throws → if `attempt < max_attempts`, clear
|
||||
the claim and set `deliver_after = NOW() + compute_backoff(...)`;
|
||||
else, write a `dead_letters` row (the existing `fan_out_dead_letter`
|
||||
path then fires registered `dead_letter` triggers off it) and delete
|
||||
the queue row, in one transaction.
|
||||
- **Visibility-timeout reclaim**: a separate `tokio::spawn` task ticks
|
||||
every `PICLOUD_QUEUE_RECLAIM_INTERVAL_MS` (default 30000) and clears
|
||||
claims older than the per-queue `visibility_timeout_secs`. A crashed
|
||||
consumer (or one whose handler hung past the visibility window) thus
|
||||
loses its lease and the message becomes claimable again.
|
||||
|
||||
### Added — `invoke()` + `invoke_async()` (same-app function composition)
|
||||
|
||||
- **`invoke(target, args)`** — sync, returns the callee's response
|
||||
body as a Rhai Dynamic. `target` is a string (path → route trie,
|
||||
UUID → `ScriptId`, else → `(app_id, name)` lookup). Same engine
|
||||
instance and `Services` are reused; only the per-call `SdkCallCx`
|
||||
changes (callee gets `trigger_depth + 1` and a fresh `execution_id`,
|
||||
inherits the caller's principal and `root_execution_id`).
|
||||
- **`invoke_async(target, args)`** — fire-and-forget. Writes an
|
||||
`OutboxSourceKind::Invoke` row the dispatcher fires once (no retry
|
||||
loop — callers who want retries wrap in `retry::run`). Returns the
|
||||
new `ExecutionId` as a string for caller tracking.
|
||||
- **Cross-app invokes are rejected** at the `InvokeService::resolve`
|
||||
layer with `InvokeError::CrossApp` ("invoke: target script belongs
|
||||
to a different app"). v1.1.x maintains strict isolation; cross-app
|
||||
sharing arrives in v1.3+.
|
||||
- **Depth-bound**: `cx.trigger_depth + 1 > limits.trigger_depth_max`
|
||||
throws "invoke: depth limit exceeded (max N)" — shared with the
|
||||
trigger fan-out cap so a misbehaving recursive script can't blow
|
||||
the stack.
|
||||
- FnPtr / closures in `args` are rejected (closures don't survive
|
||||
re-entry into a fresh `SdkCallCx`).
|
||||
|
||||
### Added — `retry::*` SDK (caller-controlled retry)
|
||||
|
||||
- **`retry::policy(opts)`** — constructs a `Policy` value (custom Rhai
|
||||
type). Validates + clamps: `max_attempts ∈ [1, 20]`,
|
||||
`base_ms ∈ [1, 60_000]`, `jitter_pct ∈ [0, 100]`,
|
||||
`backoff ∈ {"exponential" | "linear" | "constant"}`.
|
||||
- **`retry::run(policy, closure)`** — calls `closure()`; on throw,
|
||||
sleeps per the policy and re-invokes; after `max_attempts`
|
||||
exhausted, throws the last error. Sleeps via `tokio::time::sleep`
|
||||
through `TokioHandle::block_on` — the bridge is already inside the
|
||||
caller's `spawn_blocking` thread.
|
||||
- **`retry::on_codes(policy, codes_array)`** — returns a new policy
|
||||
with an error-string filter. When non-empty, only throws containing
|
||||
one of the codes retry; others surface immediately.
|
||||
- **NOTE on the brief deviation**: the brief asked for `retry::with`,
|
||||
but both `with` AND `call` are Rhai reserved keywords (rejected at
|
||||
the parser, before registration could matter). Shipped as
|
||||
`retry::run(policy, closure)` instead. Documented in HANDBACK §7.
|
||||
|
||||
### Added — Admin HTTP endpoints
|
||||
|
||||
- `POST /api/v1/admin/apps/{id}/triggers/queue` — create a
|
||||
`queue:receive` trigger. Body:
|
||||
`{ script_id, queue_name, visibility_timeout_secs?, dispatch_mode?,
|
||||
retry_max_attempts?, retry_backoff?, retry_base_ms? }`.
|
||||
Capability: `AppManageTriggers`.
|
||||
- `GET /api/v1/admin/apps/{id}/queues` — list every distinct queue
|
||||
name in the app with `(total, pending, claimed)` counts.
|
||||
Capability: `AppLogRead`.
|
||||
- `GET /api/v1/admin/apps/{id}/queues/{queue_name}` — drill-down:
|
||||
stats + registered consumer trigger (if any). Capability:
|
||||
`AppLogRead`.
|
||||
- No mutating queue endpoints (purge / requeue / delete-message stays
|
||||
at v1.2).
|
||||
|
||||
### Added — Dashboard (v0.15.0)
|
||||
|
||||
- New `/apps/{slug}/queues` read-only list view (queue name + counts +
|
||||
drilldown link) and `/apps/{slug}/queues/{name}` drill-down (depth +
|
||||
registered consumer + visibility timeout + last_fired_at).
|
||||
- "Queues" link added to the app-level nav between Files and Dead
|
||||
letters.
|
||||
- Triggers tab gains a "Queue:receive trigger (v1.1.9)" form alongside
|
||||
cron / pubsub / email. Trigger list renders queue triggers with the
|
||||
queue name + visibility timeout + last_fired_at.
|
||||
- TypeScript types: `TriggerKind` gains `'queue'`; `TriggerDetails`
|
||||
gains the `Queue` variant. `CreateQueueTriggerInput`,
|
||||
`QueueSummary`, `QueueConsumer`, `QueueDetail` exported.
|
||||
|
||||
### Changed
|
||||
|
||||
- `Services::new` signature gains `queue: Arc<dyn QueueService>` and
|
||||
`invoke: Arc<dyn InvokeService>` positionally after `users` (mirrors
|
||||
v1.1.8's positional append of `users`). `with_noop_services` updated.
|
||||
- `Limits` (`crates/executor-core/src/sandbox.rs`) gains
|
||||
`trigger_depth_max: u32` (default 8). The picloud binary syncs it
|
||||
from `TriggerConfig::from_env().max_trigger_depth` so the
|
||||
dispatcher's trigger-depth cap and the invoke depth-bound stay
|
||||
aligned (env var: `PICLOUD_MAX_TRIGGER_DEPTH`).
|
||||
- `Engine` gains `set_self_weak(Weak<Engine>)` + `self_arc()`. The
|
||||
picloud binary calls `engine.set_self_weak(Arc::downgrade(&engine))`
|
||||
right after construction so the invoke bridge can re-enter the
|
||||
engine synchronously. `Weak` so the Arc-cycle stays loose.
|
||||
- `register_all` gains two parameters: `limits: Limits` and
|
||||
`self_engine: Option<Arc<Engine>>`. The invoke bridge surfaces a
|
||||
clear error if `self_engine` is `None` (only in harnesses that don't
|
||||
wire it).
|
||||
- `OutboxSourceKind` gains `Invoke`; `TriggerKind` gains `Queue`;
|
||||
`TriggerDetails` gains the `Queue` variant; `TriggerEvent` gains the
|
||||
`Queue` variant with `source = "queue"` discriminant.
|
||||
- `ScriptRepository` gains `get_by_name(app_id, name)` for the
|
||||
`invoke()` name resolution path.
|
||||
- SDK schema 1.9 → 1.10.
|
||||
|
||||
### Migrations
|
||||
|
||||
- `0034_queue_messages.sql` — the `queue_messages` table with three
|
||||
indexes: a partial `(app_id, queue_name, enqueued_at) WHERE
|
||||
claim_token IS NULL` for the dispatch hot path, a `(app_id,
|
||||
queue_name)` for the depth queries, and a partial `(claimed_at)
|
||||
WHERE claim_token IS NOT NULL` for the reclaim scan.
|
||||
- `0035_queue_triggers.sql` — widens `triggers.kind` to admit
|
||||
`'queue'`, widens `outbox.source_kind` to admit `'invoke'`, adds the
|
||||
`queue_trigger_details` table.
|
||||
|
||||
### F1–F5 follow-ups (carry-forward from v1.1.8)
|
||||
|
||||
- **F1 — integration test density**: four new DB-gated test binaries
|
||||
(`crates/picloud/tests/queue_e2e.rs` ×4, `…/invoke_e2e.rs` ×4,
|
||||
`…/retry_e2e.rs` ×3, `crates/manager-core/tests/migration_queue_messages.rs`
|
||||
×4) + 50+ net new inline unit tests. All skip cleanly when
|
||||
`DATABASE_URL` is unset; all pass against the docker-compose
|
||||
postgres on `localhost:15432`.
|
||||
- **F2 — schema snapshot re-blessed** via
|
||||
`BLESS=1 cargo test -p picloud-manager-core --test schema_snapshot
|
||||
-- --include-ignored`. Delta matches the plan exactly (no unrelated
|
||||
drift).
|
||||
- **F3 — fmt + clippy attestation as literal output**: see HANDBACK §8.
|
||||
- **F4 — `TIMING_FLAT_DUMMY_HASH` dedup**: inline copy in
|
||||
`crates/manager-core/src/auth_api.rs::login` now references the
|
||||
shared `crate::auth::TIMING_FLAT_DUMMY_HASH` const. `grep -rn` for
|
||||
the PHC literal returns exactly one hit.
|
||||
- **F5 — clippy without cold cache**: incremental cache used; HANDBACK
|
||||
§8 notes the explicit choice.
|
||||
|
||||
### Notes
|
||||
|
||||
- New env vars:
|
||||
- `PICLOUD_QUEUE_RECLAIM_INTERVAL_MS` (default 30000) — visibility-
|
||||
timeout reclaim cadence.
|
||||
- `PICLOUD_QUEUE_DEFAULT_VISIBILITY_TIMEOUT_SECS` (default 30) —
|
||||
visibility timeout fallback when the trigger create request omits
|
||||
one. Per-trigger column overrides this.
|
||||
- SDK schema bumped to 1.10.
|
||||
- `@picloud/client` unchanged — no client-library work this release.
|
||||
|
||||
---
|
||||
|
||||
## v1.1.8 — User Management (unreleased)
|
||||
|
||||
Per-app data-plane user management — a `users::*` SDK for scripts
|
||||
|
||||
Reference in New Issue
Block a user