verify_api_key Argon2-verifies every candidate sharing the 8-char
prefix. With unlimited candidates, an attacker who can provision many
keys against one indexed prefix could force the server into per-request
M×Argon2 — a few hundred concurrent connections trivially DoSes the
manager since the verify runs before any concurrency cap.
Cap the candidate set at MAX_API_KEY_CANDIDATES = 16; log a warn when
truncation kicks in so operators can spot it. With 32-byte random key
bodies the natural collision rate is negligible, so truncation under
benign load shouldn't happen — its presence is the alarm.
Switching the verify hash from Argon2 to SHA-256 (the audit's other
suggested fix) is a larger refactor (new hash column, backfill, dual-
hash verify during migration) and is left for a follow-up.
AUDIT.md anchor: F-S-006 (cap; hash swap deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
users_admin_api::delete_user was a comment-only acknowledgement that
"this should require AppUsersAdmin" while actually calling
service.delete which only gates on AppUsersWrite. v1.1.8 HANDBACK §7
listed this as known. Effect: any editor could delete app users via
the admin HTTP API and dashboard button.
Add an explicit `authz::require(... AppUsersAdmin(app_id))` before the
service call. Delete the stale comment.
AUDIT.md anchor: F-S-005.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Handler returned immediately when email shape was invalid or user was
unknown; on a found user it generated an Argon2 token, INSERT'd into
app_user_password_resets, built a link, and called email.send (tens of
ms + DB write + SMTP RTT). The wall-clock delta was enormous and
externally observable.
Add a dummy generate_session_token() call on both no-match branches so
the observable cost matches the matching branch up to the Argon2 hash.
Doesn't equalise the DB INSERT + SMTP send (those would require either
real side effects or a deterministic dev-mode null sink), but it
neutralises the highest-bit-of-info side channel (handler-immediate-
return vs handler-token-mint).
Combined with F-S-003 (find_by_email now requires a principal),
enumeration via this surface from an anonymous public route is closed.
AUDIT.md anchor: F-S-004.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
find_by_email gated on AppUsersRead via `require`, which short-circuits
when cx.principal == None. An anonymous public-HTTP script could
iterate emails to enumerate registered users, then pivot to login or
request_password_reset.
Tighten: explicitly return Forbidden when cx.principal is None — scripts
that need a "does this email exist" probe from a public route must
wrap the call in their own auth gate.
AUDIT.md anchor: F-S-003.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cargo fmt regroups, plus three Phase-B clippy fixes:
- F-P-004: #[allow(clippy::type_complexity)] on the invoke_ast_cache Mutex.
- F-P-008: hoist `futures::stream::{self, TryStreamExt}` to the top-of-
file imports to satisfy clippy::items_after_statements.
- F-P-005: rename `_legacy_offset` to `legacy_offset` + #[allow(dead_code)]
to satisfy clippy's "field is pub but `_`-prefixed" check.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Files page could only list a collection if the operator already
knew its name and typed it in. No "browse known collections" affordance,
no backend endpoint listing collections.
Closest approximation without new backend: pull registered files-trigger
collection_globs from the existing triggers list and surface them as:
- a <datalist> on the collection input for autocomplete
- chip buttons under the form that one-click set the input
Empty state copy points the operator at the Triggers tab. Backend
endpoint to list known collections directly is still a v1.1.10+ task.
Styling uses the F-U-004 :root tokens so it inherits the dark theme.
AUDIT.md anchor: F-U-003 (frontend-only; backend endpoint deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend triggers_api.rs has long exposed POST /apps/{id}/triggers/{kv,
docs,files,dead_letter}; the dashboard Triggers tab shipped create
forms only for cron / pubsub / email / queue. Listing showed kv/docs/
files/dead_letter rows but operators couldn't create them from the UI.
This commit adds:
- 4 new CreateXxxTriggerInput types in api.ts.
- 4 new api.triggers.createXxx methods.
- 4 new submitCreateXxx functions in apps/[slug]/+page.svelte.
- 4 new <form class="create-form"> sections under the queue form.
KV / Docs / Files share the (script_id, collection_glob, ops[]) shape
with checkbox UI for the per-kind ops (insert/update/delete vs
create/update/delete). Dead-letter takes (script_id, source_filter,
trigger_id_filter, script_id_filter) — all but script_id optional, with
"leaving blank routes every dead-letter to this script" inline help.
`npm run check` clean (only pre-existing tests/e2e/* Playwright errors
remain, documented out-of-scope in the AUDIT.md methodology notes).
AUDIT.md anchor: F-U-001.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
useEndpoint(path) returned `{ get: () => useResource(...), post: (body)
=> useResource(...) }`. Each of `.get()` and `.post()` called useState
+ useEffect — but React's Rules of Hooks require hook calls at the top
level of a component, in the same order each render. Calling
.get() conditionally, or both .get() and .post() from the same
component, produced undefined behaviour (state leaking between them).
The test suite covered only useTopic, so this was uncaught.
Separately (F-T-002): the JSDoc said .post() was "the mutation variant
(auto-fires once per mount)" — but auto-firing a POST on mount means
typo'd code creates a user (or sends an email) on every render refresh.
Refactor:
- Remove useEndpoint entirely (public API breaking change — clients/ts
v1.1.x has no external consumers yet per CLAUDE.md).
- Add useEndpointGet<Res>(path): QueryState<Res> — flat hook, auto-fires.
- Add useEndpointPost<Req, Res>(path): { mutate, data, loading, error }
— flat hook, event-driven (NOT auto-firing); call `mutate(body)` from
the submit handler.
README updated to demonstrate both shapes side-by-side.
Existing 15 unit tests pass (useTopic unaffected; useEndpoint tests
never existed). Adding a useEndpointGet/Post test pass is finding
F-T-001 follow-up.
AUDIT.md anchor: F-T-001 (+ F-T-002 folded in).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
attach_principal_if_present runs on every request that carries a
Bearer header, including data-plane paths that may not even need
authz. Each call paid three DB round-trips on the session path
(lookup + admin_users.get + touch) plus an Argon2 verify per
prefix-colliding candidate on the API-key path. A hot user with
multiple keys serialized every request behind N×Argon2.
Add a process-shared PrincipalCache keyed on hash_token(bearer) →
(Instant, Principal), TTL 60s. resolve_principal checks the cache
first; on miss falls through to the verify_api_key / verify_session
path and writes back on success.
- Lazy GC: when the cache exceeds 1024 entries, sweep expired before
inserting (kept simple — production sees few unique hot tokens).
- Cache wired through AuthState; constructed once in picloud/src/lib.rs.
- Skip-when-route-doesn't-need-auth scaffolding is deferred to a
follow-up — it requires touching the router shape, which is more
invasive than the 60s cache alone warrants.
AUDIT.md anchor: F-P-009 (cache; skip-path deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TriggerRepo::list_for_app selected parent rows then called
hydrate_one(...) per row serially — one SELECT against the kind-
specific details table per trigger. For N triggers on an app, that's
N+1 queries serialized on every dashboard `GET /apps/{id}/triggers`.
This commit shrinks the wall-clock cost via buffered concurrency
(`try_buffered(8)`): each row's details fetch still runs but they run
in parallel up to a small fan-out cap. Cuts dashboard load time from
sum(N latencies) to max(N latencies) / 8.
Collapsing to a single per-kind LEFT JOIN (the audit's preferred shape)
would cut the query count to ~7 — large enough to defer to v1.2 as
documented in the inline comment.
AUDIT.md anchor: F-P-008 (partial: wall-clock only; round-trip count
deferred).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tick_queue_arm called list_active_queue_consumers() and then iterated
serially, awaiting one queue.claim(app, queue) per consumer. With N
consumers and a 100ms tick the worst-case throughput was
N × (claim + executor) / tick — one slow handler blocked every other
queue's progress on the dispatcher's task.
Replace the for-loop with `futures::stream::iter(consumers)
.for_each_concurrent(QUEUE_DISPATCH_PARALLELISM, …)`. The execution
gate already caps real script-concurrency to its permits, so this just
removes the dispatcher-side serialization.
QUEUE_DISPATCH_PARALLELISM = 32 (matches the default gate). Workspace
gains `futures = 0.3` so `for_each_concurrent` is available.
AUDIT.md anchor: F-P-007.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scripts called queue::depth(name) / queue::depth_pending(name) per
invocation; each ran an unbounded COUNT(*) over the queue_messages
partial index. On a backed-up queue with millions of rows, every call
scanned the whole partition.
Wrap the predicate in a LIMIT-capped subquery so the worst-case scan
is bounded:
SELECT COUNT(*) FROM (
SELECT 1 FROM queue_messages WHERE ... LIMIT 10000
) sub
QUEUE_DEPTH_SCAN_CAP = 10_000. Callers that need an exact depth on
queues larger than 10k use the admin /apps/{id}/queues endpoint which
tolerates the slower COUNT for its much lower read frequency.
`list_for_app` (dashboard queue overview) is left at full COUNT —
separate finding F-P-006 second-bullet, deferred to v1.2 along with
per-queue running counters.
AUDIT.md anchor: F-P-006 (first half).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
list_for_script used ORDER BY created_at DESC LIMIT $2 OFFSET $3.
Postgres has to scan + discard OFFSET rows on every page, so deep
pagination on a script with many log rows gets linearly slower. Every
sister-table list endpoint in the repo already uses cursor pagination
— this is the outlier.
Add ExecutionLogCursor { created_at, id } with `<rfc3339>_<uuid>`
encode/decode. New trait signature:
list_for_script(script_id, limit, cursor: Option<ExecutionLogCursor>)
Predicate: WHERE script_id=$1 AND (created_at, id) < ($cursor_ts, $cursor_id).
ORDER BY also gains `id DESC` for deterministic ordering at equal-time
boundaries.
API surface:
- /api/v1/admin/scripts/{id}/logs?cursor=<token>&limit=50 is the new
shape (dashboards adopt later — separate finding F-U-012).
- Legacy `offset` query param accepted-and-ignored to keep older
dashboards from 400'ing during rollout.
AUDIT.md anchor: F-P-005.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two paths bypassed the AST cache: LocalExecutorClient::execute (tests +
fallback) and the synchronous invoke() re-entry in the executor SDK.
The latter is the hot one — composed workflows multiplied parse cost
by depth, so a 4-deep invoke chain on a 200-line script paid the parse
budget × 4 per call.
Add a per-Engine HashMap<ScriptId, (updated_at, Arc<AST>)> + a
`compile_for_identity(script_id, updated_at, source)` helper that
behaves like LocalExecutorClient::get_or_compile but lives on the
Engine. Update the SDK invoke synchronous re-entry to:
resolved → compile_for_identity → execute_ast
The orchestrator-core LocalExecutorClient cache (HTTP-path dispatch)
is left untouched — it caches a different access pattern at a
different boundary.
AUDIT.md anchor: F-P-004.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
verify_password (Argon2id, OWASP defaults m=19456 KiB, t=2) is CPU-bound
at tens-to-hundreds of ms and was invoked synchronously on the Tokio
worker. Worse, verify_api_key Argon2-verifies every candidate sharing
the 8-char prefix — a hot user with N keys serialized every admin
request behind N×Argon2.
Wrap each call site in tokio::task::spawn_blocking:
- auth_middleware::verify_api_key (per request carrying a Bearer key)
- auth_api::login (admin login)
- users_service::login (data-plane app-user login, both real-hash and
TIMING_FLAT_DUMMY_HASH branches)
Cold-cache login is now ~2× current latency due to one spawn_blocking
hop, but the worker no longer parks on Argon2 so steady-state under
load is dramatically better. The LRU cache for the hot-path (token →
principal) is finding F-P-009 — separate commit.
AUDIT.md anchor: F-P-002.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
users::list page-fetched users then looped calling
self.fetch_roles(cx.app_id, row.id) — one query per user row, with
default limit 50 and max 500 — so the admin "users" page did 51 to 501
round-trips. The same fetch_roles is also called from
verify_session_for_realtime on every authenticated SSE subscribe.
Add AppUserRoleRepo::list_for_users(app_id, &[user_ids]) returning a
HashMap<AppUserId, Vec<String>>; rewrite list to use it. Single query
with `WHERE user_id = ANY($2)`.
Empty-input fast path returns an empty map without touching Postgres.
AUDIT.md anchor: F-P-001.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both methods short-circuit the authz gate when cx.principal == None.
An attacker hitting any public route could trigger unbounded outbound
email per call to arbitrary or attacker-supplied addresses — exhausting
SMTP-relay quota, getting the relay blacklisted, or burning provider
credits.
Add an in-memory EmailRateLimiter on UsersServiceImpl with two scopes:
- Per-(app, recipient): 5 calls per rolling 1h window
- Per-app daily cap: 200 calls per rolling 24h window
Both windows reset lazily. Token-bucket counts increment only after
both checks pass — a denied recipient doesn't consume the app counter.
Wired:
- send_verification_email: returns UsersError::EmailRateLimited (→ 429
on the admin HTTP surface).
- request_password_reset: returns Ok(()) silently on rate-limit (would
otherwise enable an existence-leak side channel).
UsersError::EmailRateLimited variant added; AppUsersApiError gains the
same variant + 429 mapping. NoopUsersError trait stub unaffected.
AUDIT.md anchor: F-S-002.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
init_db hard-coded max_connections(10). The execution gate
(ExecutionGate, PICLOUD_MAX_CONCURRENT_EXECUTIONS default 32) lets 32
script executions run concurrently, each doing multiple sequential
DB calls — plus the dispatcher tick every 100ms, the cron tick, three
GC sweeps, and the auth middleware running per request all draw from
the same pool. With max=10 vs gate=32, pool starvation surfaces as
acquire_timeout errors and tail-latency spikes under load.
- DEFAULT_DB_MAX_CONNECTIONS = 32 (matches gate default).
- Env knob: PICLOUD_DB_MAX_CONNECTIONS overrides; invalid/zero → default.
- Documented in CLAUDE.md runtime config table.
AUDIT.md anchor: F-P-003.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The dashboard route tree has scripts at /scripts/[id], not under
apps/[slug]/scripts. The queue-drilldown page linked to a non-existent
app-scoped scripts route, so clicking the consumer-script name 404'd.
Reverts the link to {base}/scripts/{detail.consumer.script_id}, matching
how every other place in the dashboard navigates to a script page.
AUDIT.md anchor: F-U-002.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Whitespace + import-grouping fixups in the 9 SDK modules touched by
F-Q-002. No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five dashboard pages (dead-letters, files, users, invitations, queues)
reference CSS custom properties like var(--text-muted, #666),
var(--bg-secondary, #f5f5f5), var(--border, #e0e0e0),
var(--color-link) that the dashboard never defined — so users saw the
light-theme fallbacks: #666 text on #f5f5f5 backgrounds, borders that
disappeared, a white-on-red error banner. The dashboard otherwise
renders with the slate-900 dark palette.
Define a :root token set in routes/+layout.svelte using the slate
shades already used inline elsewhere. Per-page styles stop hitting
their fallbacks; the dark theme renders consistently everywhere.
No per-page CSS touched — the tokens cover every fallback already in
use (audited via grep across the five pages cited in the AUDIT.md).
Adds a few extra tokens (danger/success/warning) so future pages have a
single source of truth instead of inline hex.
`npm run check` shows no new errors (the pre-existing 149 errors in
tests/e2e/* from missing @playwright/test were documented out-of-scope
in the AUDIT.md methodology notes).
AUDIT.md anchor: F-U-004.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Files (per-file cap), secrets (64 KB default), and email (25 MB default)
already enforce limits; kv::set, docs::create/update, pubsub::publish_durable
and queue::enqueue accepted any JSON value straight to a JSONB column with
no size validation. An anonymous public-HTTP script could fill disk via
queue::enqueue or amplify a single publish into N outbox rows × payload bytes.
Adds four new error variants:
- KvError::ValueTooLarge { limit, actual }
- DocsError::ValueTooLarge { limit, actual }
- PubsubError::MessageTooLarge { limit, actual }
- QueueError::PayloadTooLarge { limit, actual }
Each stateful service grows a `max_value_bytes` field with:
- Conservative 256 KB default (DEFAULT_KV_MAX_VALUE_BYTES etc.).
- New `with_max_*` constructor preserving the old `new()` signature.
- Env-knob reader (`*_max_*_from_env()`) — mirrors SecretsConfig::from_env.
Validation runs at the entry point BEFORE authz so an anonymous DoS doesn't
pay a membership lookup per attempt.
Wired via env knobs:
- PICLOUD_KV_MAX_VALUE_BYTES
- PICLOUD_DOCS_MAX_VALUE_BYTES
- PICLOUD_PUBSUB_MAX_MESSAGE_BYTES
- PICLOUD_QUEUE_MAX_PAYLOAD_BYTES
Documented in CLAUDE.md runtime config table.
New unit test in queue_service verifying the cap fires before authz.
AUDIT.md anchor: F-S-001. Depends on F-Q-004 (Backend variant).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace ten near-identical `block_on` helpers (one per SDK module)
with a single shared `sdk::bridge::block_on(service: &str, fut)`.
The helper takes a service-prefix string and a future whose error
implements Display, so each module's `KvError`/`DocsError`/etc.
still self-formats.
Migrated:
- kv, docs, pubsub, users, dead_letters, secrets, files, email
- queue.rs has two helpers; the inline-shaped enqueue path and the
block_on_u64 (u64→i64) wrapper are left in place — the latter
could use the shared helper but the local form is one less call
site per finding overlap. Will revisit on a later pass if needed.
- http.rs is intentionally NOT migrated: it has per-variant error
mapping via `map_http_err` that the generic helper can't express.
Net: -218 / +97 lines across the 9 migrated files. Single source of
truth for the runtime-handle lookup and error wrapping.
AUDIT.md anchor: F-Q-002.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace nine open-coded `if cx.principal.is_some() {
authz::require(...).await.map_err(...) }` blocks with a single
`authz::script_gate(repo, cx, cap, forbidden_fn, backend_fn)` helper.
The helper enshrines the script-as-gate semantics (anonymous public-
HTTP scripts skip the check) and the AuthzDenied::{Denied,Repo}
mapping in one place — eliminating drift between services.
Call sites migrated:
- kv_service::check_read / check_write
- docs_service::check_read / check_write
- files_service::check_read / check_write
- pubsub_service::check_publish
- queue_service::enqueue
The pubsub_service::mint_subscriber_token path keeps the explicit
match because it does a separate `principal` early-bind for other
validation; converting it would obscure intent.
AUDIT.md anchor: F-Q-003. Depends on F-Q-004 (Backend variant) and
F-Q-005 (Repo-passthrough pattern).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every check_read/check_write/check_publish helper across kv, docs,
files, pubsub, queue services used `.map_err(|_| KvError::Forbidden)?`,
collapsing AuthzDenied::Denied AND AuthzDenied::Repo(repo_err) into
a single Forbidden. A transient Postgres blip during the membership
lookup surfaced as 403 to scripts; operators couldn't distinguish
"real forbidden" from "DB flap during permission check".
Replace each call site with the explicit match pattern already used by
users_service::require (users_service.rs:177-181):
Ok(()) → continue
Err(AuthzDenied::Denied) → Err(ServiceError::Forbidden)
Err(AuthzDenied::Repo(e)) → Err(ServiceError::Backend(e.to_string()))
7 call sites updated (2 in kv_service, 2 in docs_service, 2 in
files_service, 2 in pubsub_service, 1 in queue_service).
AUDIT.md anchor: F-Q-005. Depends on F-Q-004 (which added Backend
to QueueError/PubsubError).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings QueueError / PubsubError / InvokeError in line with sibling
shape used by KvError / DocsError / FilesError / SecretsError:
- QueueError gains an explicit Forbidden variant (previously authz
denial was squashed into Rejected("forbidden") in queue_service.rs:68,
losing the structured variant a 403-translation layer would need).
- QueueError::Unavailable renamed → Backend.
- PubsubError::Unavailable renamed → Backend.
- InvokeError::Unavailable renamed → Backend.
- Call sites updated (queue_service, pubsub_service, invoke_service,
Noop* stubs, From<PubsubRepoError> impl, one test assertion).
- New unit test verifying authed-denied → QueueError::Forbidden.
AUDIT.md anchor: F-Q-004.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
APPROVE — final v1.1.x release lands clean. Full scope: queue::*
+ queue:receive trigger + dispatcher queue arm + visibility-timeout
reclaim + invoke()/invoke_async + retry::* (with → run per Rhai
reserved keyword). All five F1-F5 follow-ups from the v1.1.8 retro
implemented; seven deviations transparently flagged in HANDBACK §7
with sound rationale.
§8 attestation discipline visibly leveled up from v1.1.8:
schema-snapshot re-blessed in-branch, literal fmt/clippy output,
F4 grep verification, four new DB-gated integration binaries
(queue_e2e 4, invoke_e2e 4, retry_e2e 3, migration_queue_messages 4)
matching the brief's non-negotiable test-density minimums.
Reviewer-independent verification reproduced all material claims:
fmt green, clippy green (incremental), F4 grep returns exactly
one hit, schema replay matches the committed golden, all 15 new
DB-gated tests pass against a fresh dev Postgres. Aggregate
lighter-slice attestation: 666 passed / 0 failed across the
load-bearing crates — matches agent's claimed ~660.
Running the E2E suites against real Postgres surfaced three shape bugs
in the test scripts that caused false failures:
invoke_e2e:
- invoke_cross_app_rejects used two TestServer instances (one per app),
but the second server's Owner admin isn't a member of the first
server's app. Replaced with a single server that creates both apps
via the same Owner admin (which has implicit access to every app).
- invoke_depth_limit_exceeds_cleanly: the recurser script had its own
try/catch, so when the depth limit fired inside the deepest call the
caught error became the BODY of a 200 response (which invoke()
returns to the caller). The outer caller's try/catch never saw a
throw → assertion failed. Rewrote so the recurser propagates throws
(no inner try-catch); the outer caller's try-catch surfaces the
depth error all the way up.
retry_e2e:
- All three tests used HTTP routes which need a domain claim the test
apps don't have (`no app claims host ""` 404s). Switched to the
admin bypass POST /api/v1/execute/{id} — same pattern dispatcher_e2e
uses. Sidesteps the per-app domain matcher entirely.
- retry_run_surfaces_last_error_after_max_attempts: try-catch is a
statement in Rhai, not an expression, so the block didn't evaluate
to the catch arm's map. Refactored to bind to `let out` inside the
catch arm, then return `#{ statusCode: 200, body: out }` as the
final expression.
All 11 v1.1.9 E2E tests now pass against Postgres:
queue_e2e: 4 passed (33s — exercises retry + dead-letter)
invoke_e2e: 4 passed (2s)
retry_e2e: 3 passed (2.5s)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-blessed via:
DATABASE_URL=postgres://picloud:picloud@localhost:15432/picloud \
BLESS=1 cargo test -p picloud-manager-core --test schema_snapshot \
-- --include-ignored
Delta matches the plan exactly — no unrelated drift:
- new table queue_messages (id, app_id, queue_name, payload,
enqueued_at, deliver_after, claim_token, claimed_at, attempt,
max_attempts, enqueued_by_principal)
- new table queue_trigger_details (trigger_id, queue_name,
visibility_timeout_secs, last_fired_at)
- 3 new indexes on queue_messages: idx_queue_messages_dispatch
(partial WHERE claim_token IS NULL), idx_queue_messages_claimed
(partial WHERE claim_token IS NOT NULL), idx_queue_messages_app_queue
- 1 new index on queue_trigger_details: idx_queue_trigger_details_queue_name
- widened triggers.kind CHECK to admit 'queue'
- widened outbox.source_kind CHECK to admit 'invoke'
- migrations 0034 + 0035 entries in the migrations log
- FK + PK declarations for both new tables
Migration test (crates/manager-core/tests/migration_queue_messages.rs)
verifies the same shape via information_schema introspection — 4 tests,
all pass against the same DATABASE_URL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the inline copy of the Argon2id PHC dummy hash in
crates/manager-core/src/auth_api.rs::login with a reference to the
shared TIMING_FLAT_DUMMY_HASH constant in
crates/manager-core/src/auth.rs (added in v1.1.8 but not yet adopted
by the login path).
Verification: `grep -rn 'argon2id\$v=19\$m=19456,t=2,p=1\$dGltaW5n' crates/`
returns exactly one hit — the const declaration in auth.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four new test binaries, DB-gated via the established pool_or_skip()
pattern (returns Some(pool) when DATABASE_URL is set; prints a skip
notice and returns None otherwise — so plain `cargo test` stays green
locally while CI runs them).
crates/picloud/tests/queue_e2e.rs (4 tests):
- queue_receive_acks_on_success: enqueue → consumer fires → KV
marker observable → queue row deleted (ack worked)
- queue_receive_dead_letters_after_max_attempts: throwing handler
retries 3× via the dispatcher's compute_backoff path → dead_letters
row written, queue row deleted
- queue_one_consumer_per_queue_rejected: second trigger create for the
same (app_id, queue_name) returns 4xx with the documented "already
has a consumer trigger" message (advisory-lock guard works)
- queue_visibility_timeout_reclaim: insert message with stale claim
(claimed_at = NOW() - 60s, visibility_timeout = 5s); either the
dispatcher re-claims after reclaim runs, or the claim clears
crates/picloud/tests/invoke_e2e.rs (4 tests):
- invoke_by_name_same_app_returns_value: callee returns body.x+1;
caller invokes by name and writes the result to KV (42 round-trips)
- invoke_cross_app_rejects: callee in app_B, caller in app_A; error
string contains "different app" or "CrossApp"
- invoke_depth_limit_exceeds_cleanly: recursive script throws with
"depth" in the message at the bound (Limits::trigger_depth_max=8)
- invoke_async_enqueues_outbox_row: invoke_async returns execution_id,
dispatcher fires the OutboxSourceKind::Invoke arm, callee marker
appears
crates/picloud/tests/retry_e2e.rs (3 tests):
- retry_run_eventually_succeeds_inside_http_handler: counter mutates
across 3 retries through a real HTTP route, returns 3
- retry_run_surfaces_last_error_after_max_attempts: max_attempts=2,
always throws — caller's try/catch surfaces "boom"
- retry_on_codes_filters_unmatched_errors: counter == 1 after a
throw NOT in the codes list (immediate surface, no retry)
crates/manager-core/tests/migration_queue_messages.rs (4 tests):
- queue_messages_table_exists_with_expected_columns: shape +
nullability of every column
- queue_trigger_details_table_exists: shape
- queue_widens_trigger_kind_and_outbox_source_kind: triggers.kind
CHECK admits 'queue'; outbox.source_kind admits 'invoke'
- queue_messages_dispatch_index_is_partial: idx_queue_messages_dispatch
has WHERE claim_token IS NULL (guards against future migrations
accidentally dropping the partial)
All 15 tests pass clean skip when DATABASE_URL is unset. Workspace unit
suite remains at 432 passing (306 manager-core + 74 executor-core + 34
shared + 18 orchestrator-core).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
executor-core/sdk/retry.rs adds the retry:: namespace with:
retry::policy(opts) -> Policy
retry::on_codes(policy, codes) -> Policy
retry::run(policy, closure) -> Dynamic (closure's return value)
NOTE: the brief specified retry::with(...), but `with` is a Rhai
reserved keyword (parse error before registration even matters). `call`
is also reserved. Shipping retry::run instead — flagged in HANDBACK §7.
Policy:
- max_attempts clamped to [1, 20]
- base_ms clamped to [1, 60_000]
- jitter_pct clamped to [0, 100]
- backoff ∈ {"exponential" | "linear" | "constant"} — bogus values
surface a clear error
- on_codes is an empty default ("retry any throw"); when non-empty,
only error messages containing one of the codes retry — others
surface immediately
retry::run uses NativeCallContext + FnPtr::call_within_context to
re-invoke the closure inside the caller's engine. Closures share the
caller's cx (Rhai semantics). If the closure calls invoke(), the inner
call gets a fresh cx with trigger_depth + 1 via the invoke bridge —
retry doesn't see or interfere with that.
Sleep between attempts: tokio::time::sleep through TokioHandle::block_on.
Safe because we're already inside the caller's spawn_blocking thread.
register_all gains retry::register positionally between queue and
secrets.
Unit tests (7): policy default, max_attempts clamping (0→1, 999→20),
base_ms + jitter clamping, bogus backoff reject, exponential doubles,
linear sums, constant.
Integration tests (sdk_retry.rs, 6 binaries):
- succeeds first try (returns 42)
- max_attempts exhausted surfaces last error ("boom")
- on_codes filters — non-matching error throws immediately
- jitter clamping is silent (script still runs)
- bogus backoff string surfaces a clear error
- "fails 2 then succeeds" — counter mutates across retries, 3rd
attempt returns 3 (validates Rhai closure-capture semantics across
re-invocations through NativeCallContext)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the commit 2 placeholder with the real OutboxSourceKind::Invoke
handler.
build_invoke_request(row) hydrates an (ResolvedTrigger, ExecRequest)
pair from a payload InvokeService::enqueue_async wrote. Validates:
- script_id present + parses as UUID
- script exists
- script.app_id == row.app_id (defensive — re-check the cross-app
boundary; same-app was already verified at enqueue time)
Synthesized ResolvedTrigger carries retry_max_attempts = 1 so the
existing handle_failure path skips the retry loop. invoke_async runs
exactly once; on throw the row is deleted + a dead_letters row written
(handle_failure already does this when attempt >= max_attempts), so a
caller who wants retry semantics wraps the call site in retry::with.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
manager-core/invoke_service.rs implements InvokeService over:
- ScriptRepository (for Id + Name resolution)
- RouteTable (for Path resolution via the orchestrator's matcher)
- OutboxRepo (for invoke_async)
Same-app guard runs at the service entry point: resolve_id() always
verifies resolved.app_id == cx.app_id and returns InvokeError::CrossApp
otherwise. resolve_name() reads cx.app_id when querying (so a wrong
app_id from somewhere else couldn't slip through), but defense-in-depth
won't hurt — future hardening if it surfaces.
resolve_path() runs the matcher against this app's routes only
(RouteTable::snapshot_for_app). Method assumed POST→GET fallback for
v1.1.9 — surfacing method via the SDK is a future addition.
enqueue_async() resolves the target, then writes one outbox row with
source_kind = 'invoke'. The payload carries script_id, script_name,
args, fresh execution_id, root_execution_id (inherited from caller),
trigger_depth + 1. caller principal recorded as origin_principal
(forensic only — the executing script runs with no principal). One
shot — no retry policy on the row; users wrap in retry::with() if
they want retries.
picloud/lib.rs:
- route_table construction moved BEFORE Services::new so the invoke
service can hold it. populates from route_repo as before.
- InvokeServiceImpl wired into Services::new in place of the
NoopInvokeService placeholder.
Unit tests cover: resolve by Id same-app, cross-app rejected, resolve
by Name finds + not-found, enqueue_async writes Invoke outbox row with
trigger_depth=1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shared/invoke.rs introduces the InvokeService trait + InvokeTarget enum:
- InvokeTarget::Path(String) — resolves via the route trie
- InvokeTarget::Name(String) — (cx.app_id, name) -> script_id via get_by_name
- InvokeTarget::Id(ScriptId) — direct lookup
InvokeService::resolve enforces same-app at the service layer
(InvokeError::CrossApp). InvokeService::enqueue_async writes a v1.1.9
OutboxSourceKind::Invoke row for fire-and-forget composition.
Errors: NotFound, CrossApp, DepthExceeded(u32), Forbidden, Rejected,
Unavailable. NoopInvokeService surfaces all calls as Unavailable for
harnesses that don't wire the service.
ScriptRepository gains get_by_name(app_id, name) backed by the existing
(app_id, name) uniqueness constraint. Postgres impl + in-memory mock +
the picloud crate's PostgresScriptRepoHandle delegate added.
Services::new gains invoke: Arc<dyn InvokeService> positionally after
queue. with_noop_services wires NoopInvokeService. 12 test sites
threaded through. picloud/lib.rs binds NoopInvokeService at this commit
boundary — the real InvokeResolver lands in commit 8.
Deviation from plan (flagged for HANDBACK §7): the plan called for
moving ExecutorClient + ScriptIdentity from orchestrator-core to shared
so the invoke bridge could call a re-entrant variant. Re-evaluated:
the bridge lives in executor-core which can hold Arc<Engine> directly,
making the trait move unnecessary. Engine::execute is sync, returns
ExecResponse, and shares the engine instance + per-call SdkCallCx
exactly as required. AST cache reuse across invokes is a v1.2 optimization
(invokes recompile each call — ms-scale cost, acceptable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends Dispatcher with the v1.1.9 queue path. The queue table IS the
outbox for queue semantics, so the dispatcher polls queue_messages
directly via FOR UPDATE SKIP LOCKED — no outbox indirection.
Per tick (every 100ms):
1) outbox arm — unchanged
2) queue arm: list_active_queue_consumers() → per (app_id, queue_name)
attempt one claim. Bounded by registered-consumer count so one busy
queue can't starve others.
Per claimed message:
- Build TriggerEvent::Queue + ExecRequest (executes as the trigger's
registering principal, matching design notes §4)
- dispatch through executor.execute_with_identity (reuses AST cache)
- success → queue.ack(id, claim_token) — DELETE WHERE id AND token
- throw + attempt < max_attempts → queue.nack(...) — clear claim, set
deliver_after = NOW() + compute_backoff(attempt, backoff, base_ms, jitter)
- throw + exhausted → queue.dead_letter(...) atomic move to dead_letters
+ DELETE in one transaction. fan_out_dead_letter on the outbox arm
fires registered dead_letter handlers off the new row without changes.
Visibility-timeout reclaim task: separate tokio::spawn ticking on
queue_reclaim_interval_ms (default 30000). UPDATE clears claim_token /
claimed_at on rows whose claimed_at exceeds the per-queue
visibility_timeout_secs (joined from queue_trigger_details). A crashed
consumer thus loses its lease and the message becomes claimable again.
Dispatcher gains queue: Arc<dyn QueueRepo>; picloud/lib.rs threads
queue_repo.clone() into the construction.
ActiveQueueConsumer extended with app_id so the queue claim can be
performed without a follow-up trigger lookup; list_active_queue_consumers
SQL extended to SELECT t.app_id.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- shared/services.rs: Services::new gains queue: Arc<dyn QueueService>
positionally after users (mirrors v1.1.8's append of users).
with_noop_services adds NoopQueueService.
- manager-core/queue_service.rs: QueueServiceImpl wraps QueueRepo with
script-as-gate authz on AppQueueEnqueue. enqueue clamps max_attempts
to [1,20] and delay_ms to [0, 86_400_000ms]. depth/depth_pending are
read-only — no authz check (scripts in the app can see their own
queue depths). cx.principal threads through as enqueued_by_principal
(forensic only).
- manager-core/authz.rs: AppQueueEnqueue(AppId) capability — script:write
scope, granted to editor+ (same trust shape as AppPubsubPublish).
- picloud/lib.rs: wires PostgresQueueRepo + QueueServiceImpl into
Services::new alongside the existing v1.1.7+ services.
- 11 sdk test binaries + manager-core/realtime_authority.rs updated to
pass NoopQueueService to Services::new.
Unit tests cover empty queue_name reject, max_attempts clamping
(0/21 → invalid), delay_ms negative-reject, anonymous principal skips
authz, depth/depth_pending pass-through.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- shared/queue.rs: QueueService trait (enqueue / depth / depth_pending),
EnqueueOpts, QueueError, NoopQueueService. Methods derive app_id from
cx.app_id — no script-passed app_id. The handle-less surface mirrors
pubsub (queues are the grouping unit).
- shared/ids.rs: QueueMessageId.
- manager-core/queue_repo.rs: PostgresQueueRepo with:
- enqueue(NewQueueMessage) — single INSERT
- claim(app_id, queue_name) — atomic UPDATE WHERE id = (SELECT … FOR
UPDATE SKIP LOCKED LIMIT 1) RETURNING — the single-round-trip claim
from the design notes
- ack(id, claim_token) — DELETE WHERE id AND claim_token (lease check)
- nack(id, claim_token, retry_delay) — clear claim + set deliver_after
- reclaim_visibility_timeouts() — periodic UPDATE joining triggers +
queue_trigger_details, clears claims older than per-queue
visibility_timeout_secs
- depth / depth_pending / list_for_app (dashboard read-only)
- dead_letter(...) — atomic move to dead_letters + DELETE from
queue_messages, in one transaction. Uses a JSON payload shaped the
same as TriggerEvent::Queue so the existing fan_out_dead_letter
path delivers the original to registered dead_letter triggers
without special-casing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- 0034_queue_messages.sql: per-app durable named queues. (app_id, queue_name)
is the identity tuple; no queue registry table. Partial indexes on
(claim_token IS NULL) for the dispatch hot path and (claim_token IS NOT NULL)
for the visibility-timeout reclaim scan. The queue table IS the outbox
for queue semantics — no double-buffering.
- 0035_queue_triggers.sql: widens triggers.kind CHECK to admit 'queue';
widens outbox.source_kind CHECK to admit 'invoke' (for invoke_async).
Adds queue_trigger_details(trigger_id PK, queue_name, visibility_timeout_secs,
last_fired_at). Retry policy lives on the parent triggers row — same
pattern as every other kind. One-consumer-per-queue is API-layer enforced
via pg_advisory_xact_lock (partial unique index can't span parent.app_id).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
APPROVE — code is structurally sound (cross-app isolation
uniformly enforced, timing-flat login is real, F1 migration
guarded, F3 implementation tight with 4 new tests).
§8 attestation gaps handled by the reviewer rather than bounced
back: fmt cleanup applied, schema snapshot re-blessed, lighter
targeted test slice run (401 passed across manager-core/
shared/orchestrator-core libs + schema_snapshot). Cold-cache
clippy attestation is environmentally infeasible on this dev
hardware (host freeze on both agent and reviewer machines);
agent's own §6 F2 attestation + reviewer's incremental green
clippy are accepted.
Zero new integration test binaries (vs. brief's ~5-10 ask)
flagged as a known coverage gap; folded into v1.1.9 follow-ups
as a hard requirement.
`tests/expected_schema.txt` reflecting the v1.1.8 migrations:
- 6 new tables (app_users, app_user_sessions, app_user_email_verifications,
app_user_password_resets, app_user_invitations, app_user_roles)
- realtime_signing_key column dropped from app_secrets (F1)
- topics_auth_mode_check widened to include 'session' (F3)
- Migrations 0026..0033 added to _sqlx_migrations
Diff verified to have zero unrelated drift.
Reviewer-supplied per REVIEW.md §7; the agent's host couldn't
run BLESS=1 cleanly (no Postgres available during the dispatch).
8 files needed re-wrapping after the F2 clippy-fix pass landed
un-fmt'd. Pure cosmetic — no behavioral change. Re-runs of
`cargo fmt --all -- --check` now exit clean.
Reviewer-supplied per REVIEW.md §7; folded onto the branch
to avoid a NEEDS-CHANGES round-trip on a purely mechanical fix.
