# Security audit — DoS, rate limiting & resource exhaustion **Audit date:** 2026-06-11 **Scope:** PiCloud at `main` (post-v1.1.9, pre-v1.2). Single-node MVP. **Reviewer:** Subagent 08 of 10 (parallel security audit). **Threat model:** Solo-dev / Pi-class hardware. A single anonymous attacker or one noisy authenticated tenant should not OOM the host, fill the disk, or starve the executor for other tenants in minutes. ## Counts by severity | Severity | Count | |---|---| | Critical | 0 | | High | 5 | | Medium | 7 | | Low | 6 | | Info | 3 | ## Verified / no-finding These were investigated and judged adequate for the threat model: - **Concurrent script execution cap.** `ExecutionGate` uses a single `tokio::sync::Semaphore` with `try_acquire_owned()` and translates `NoPermits`/`Closed` into `AcquireError::Overloaded { retry_after_secs: 1 }`, which `IntoResponse` for `ApiError` maps to HTTP 503 with `Retry-After: 1s` (`crates/orchestrator-core/src/api.rs:712-721`). Default 32, env-tunable via `PICLOUD_MAX_CONCURRENT_EXECUTIONS`. Both sync HTTP and the dispatcher share the same gate so the dispatcher arms back off correctly when sync HTTP is saturated. **Note on fairness:** `try_acquire` is non-blocking so "fairness" is irrelevant — there is no queue, callers get 503 immediately. - **Postgres pool default size.** `init_db` in `crates/picloud/src/lib.rs:603-619` now defaults to `DEFAULT_DB_MAX_CONNECTIONS = 32` and is env-tunable via `PICLOUD_DB_MAX_CONNECTIONS`, matching the gate. (F-P-003 from the prior audit was remediated.) Acquire timeout 5s. - **Per-message SDK service caps.** `kv_service`, `docs_service`, `pubsub_service`, `queue_service`, `files_service`, `secrets_service` all enforce a per-message JSON-encoded byte cap (256 KB default, except files at 100 MB) **before** authz. `kv_service.rs:141`, `docs_service.rs:93`, `pubsub_service.rs:178`, `queue_service.rs:82`. (F-S-001 from the prior audit was remediated.) - **Outbound HTTP defaults (SDK http::request).** 30s default timeout, 60s hard ceiling; 10s connect timeout; 10 MB request/response body cap; 5 redirect default, 10 ceiling. SSRF deny-list on by default. See `crates/manager-core/src/http_service.rs:41-53`. - **Email outbound rate limit.** `EmailRateLimiter` (token bucket + per-(app, recipient) burst + 200/day per-app cap) gates `users::*` email surfaces (`crates/manager-core/src/users_service.rs:139-187`). F-S-002 remediated. - **Outbox claim shape.** Dispatcher claims at most `CLAIM_BATCH = 8` rows per tick (100 ms default) with `FOR UPDATE SKIP LOCKED`, gates every dispatch through the same `ExecutionGate`, and reschedules on permit refusal — bounded by design. See `crates/manager-core/src/dispatcher.rs:69,193-216,482-491`. - **Outbox row deletion.** Outbox rows are deleted on success/dead-letter inline (`OutboxRepo::delete`), so the table doesn't accumulate completed rows under normal load. Bloat risk surveyed below as a Medium (stuck rows). --- ## High ### H-1 — No Axum default body limit; the only inbound 503-path is the data-plane user route — all admin/email/webhook handlers fall under Axum's default 2 MB and the `/api/v1/execute/{id}` handler reads `body: Bytes` with no explicit override. - **Severity:** High - **Location:** `crates/orchestrator-core/src/api.rs:106-123` (execute_by_id, `body: Bytes`, no `DefaultBodyLimit`); `crates/manager-core/src/email_inbound_api.rs:131-136` (inbound webhook, `body: Bytes`); `crates/picloud/src/lib.rs:587-593` (router composition — no `RequestBodyLimitLayer`); admin JSON handlers (`Json<…>` extractors at `auth_api.rs:74`, `admin_users_api.rs`, `users_admin_api.rs`, etc.). - **Summary:** Axum's `Bytes`/`Json` extractors default to 2 MB. `Caddy` has no `request_body { max_size … }` directive in either `Caddyfile` or `Caddyfile.prod` (just unrestricted `reverse_proxy`), so any request up to **10 GB** (Caddy's hard ceiling) can hit picloud. Only the data-plane user-route handler (`user_route_handler` in `api.rs:219`) explicitly bounds the body at 10 MiB; everything else rides Axum's default. - For `/email-inbound/{app_id}/{trigger_id}` the body is `Bytes` with no override — so the 2 MB ceiling is the only thing standing between an anonymous attacker and a 2 MB JSON-parse fork-bomb per request. With no IP rate limit, one client can sustain hundreds of MB/s of CPU-bound `serde_json::from_slice` and HMAC verification work, saturating one execution worker indefinitely. This is the canonical "no rate limit on a public unauthenticated endpoint" finding — see also H-3. - For `POST /api/v1/execute/{id}` Axum's 2 MB default applies, but the request lands at the executor regardless of script size, meaning a script with `body_limit_kb = 32` still pays the serialization cost for a 2 MB body. - **Fix shape:** - Add a global `axum::extract::DefaultBodyLimit::max(2 * 1024 * 1024)` layer at the router root (already implicit, but should be explicit + lower the default to ~1 MB). - Add per-route higher limits where intentional (email inbound is typically tens of KB; lower it to 1 MB explicitly). - Caddy: `request_body { max_size 5MB }` in both Caddyfiles. ### H-2 — Login endpoint has no rate limit; Argon2id is the work multiplier - **Severity:** High - **Location:** `crates/manager-core/src/auth_api.rs:74-167`. - **Summary:** `POST /api/v1/admin/auth/login` is reachable unauthenticated, and on every request it does a username lookup + Argon2id verify (m=19456 KiB, t=2 default Argon2id params) via `spawn_blocking`. There is no per-IP, per-username, or process-wide cap on requests / failed attempts. Argon2 is ~50–150 ms per attempt on Pi-class hardware; an anonymous attacker with 32 in-flight HTTP requests can keep every blocking-thread worker pinned on Argon2 forever, **and** can attempt ≈600 password guesses/sec process-wide. Username enumeration is correctly defended (dummy-hash path) so the same Argon2 cost is paid on bad usernames — which means the cost inflicts on the defender for every miss. - **Threat:** Combined with H-1 (no body limit on `Json`) this is the cheapest CPU-DoS in the system. A laptop attacker can fully saturate a Raspberry Pi's executor in seconds. - **Fix shape:** add a sliding-window login rate limiter keyed on `(client_ip, username)` (e.g. 5 attempts / 60s, lockout for 5 min after 20), and a global `Semaphore` cap (e.g. 4 concurrent in-flight Argon2 verifies) so admin auth can't starve the executor's `spawn_blocking` pool. Memory only — Postgres-backed counters add a round-trip. The existing `EmailRateLimiter` pattern in `users_service.rs:139` is the shape to copy. ### H-3 — Email-inbound webhook has no IP rate limit; HMAC is optional - **Severity:** High - **Location:** `crates/manager-core/src/email_inbound_api.rs:131-191`, mounted **outside** `require_authenticated` at `crates/picloud/src/lib.rs:571-575`. - **Summary:** `POST /api/v1/email-inbound/{app_id}/{trigger_id}` is public. When the trigger has an `inbound_secret` configured, HMAC verification is correct (timestamp-bound, ±5 min, in-memory replay dedup). But: 1. If the operator did NOT configure an `inbound_secret` for the trigger, **no signature is required at all** (see the `if let (Some(ct), Some(nonce)) = …` guard at line 150). An anonymous attacker who knows the trigger UUID can enqueue an unbounded number of outbox rows + executor invocations per second, bypassing every other rate limit. 2. With a secret configured, attackers still pay no cost for sending garbage — the receiver must decrypt the secret from the secret store (one DB call + master-key open) and HMAC-verify per request **before** rejecting. No `(app_id, trigger_id)` invalid-signature bucket. So an attacker who learns the URL but not the secret can still pin a worker thread on secret-decrypt + HMAC. 3. The body is `Bytes` without a `DefaultBodyLimit` override — falls under Axum's 2 MB cap (see H-1). - **Threat:** A single anonymous HTTP client knowing only an `app_id` + a `trigger_id` (often leaked through public docs / TLS cert transparency logs / mis-configured provider dashboards) can enqueue enough outbox rows to fill the dispatcher batch every tick. With retries, one row → up to `retry_max_attempts` (default 3) executions. Outbox grows unbounded; disk fills. - **Fix shape:** require HMAC at the trigger-row level (DB CHECK that email triggers MUST have a secret); add per-`(app_id, trigger_id)` token-bucket gating both `inbound_secret_missing` (drop fast) and `bad_signature` paths; add a `DefaultBodyLimit::max(1 MB)` to the router merge in `lib.rs`. ### H-4 — No per-app cap on trigger / route / cron registration - **Severity:** High - **Location:** `crates/manager-core/src/triggers_api.rs:127-140` (create endpoints), `crates/manager-core/src/route_admin.rs:160` (create_route); no counterpart `MAX_TRIGGERS_PER_APP` / `MAX_ROUTES_PER_APP` constant in the codebase. - **Summary:** An authenticated app developer can register an unbounded number of triggers (KV, docs, files, pubsub, cron, email, queue, dead-letter) and routes. Two amplifications follow: 1. **Cron storm.** `cron_scheduler::tick` SELECTs **all** enabled cron rows in one transaction (`fetch_all`, no `LIMIT`), iterates in Rust, and inserts one outbox row per due trigger (`cron_scheduler.rs:117-167`). 10 000 cron triggers all set to `*/1 * * * * *` (every second) → 10 000 outbox rows per second, each becoming a 100ms dispatcher claim of 8 rows + 32 concurrent executions. The dispatcher backs up; outbox bloats; Postgres `next_attempt_at` index degrades. 2. **Pubsub fan-out.** `pubsub_repo::fan_out_publish` SELECTs all enabled `pubsub` triggers for the app, filters in Rust on `topic_matches`, inserts one outbox row per match in a single transaction (`pubsub_repo.rs:81-115`). N triggers subscribed to the same topic → N writes per publish. 3. **KV / docs / files fan-out.** Same pattern in `outbox_event_emitter.rs:88-103` — one outbox row per matching trigger per mutation. - **Threat:** A single tenant with `AppTriggerCreate` capability can register thousands of triggers and bring the whole node down by pubsub-publishing one message; or fill disk by cron-firing. Cross-tenant impact via shared dispatcher. - **Fix shape:** per-app trigger / route caps (e.g. 100 each, env- tunable) enforced in the *_api create handlers; per-publish fan-out cap (e.g. 100 matched triggers per publish, then dead-letter the excess with a clear error); cron-scheduler tick LIMIT clause (`LIMIT 1000`) so a misbehaving app can't ride the whole tick budget. ### H-5 — Cron-scheduler tick is unbounded `fetch_all`; one slow row blocks all - **Severity:** High - **Location:** `crates/manager-core/src/cron_scheduler.rs:115-173`. - **Summary:** Every cron tick (default 30s) opens a transaction, SELECTs every enabled cron row with `FOR UPDATE OF d SKIP LOCKED`, computes `next_due` in Rust, and INSERTs an outbox row + UPDATEs `last_fired_at` per due row — **all serially inside one transaction**. Three concrete failure modes: 1. **Long-running tx.** With N triggers and the per-row cost ~1ms, a few thousand cron rows make the tick exceed the tick interval, pile up overlapping transactions, and hold row-locks on `cron_trigger_details` for seconds — blocking dashboard reads of the same table. 2. **No LIMIT.** A malicious or buggy tenant registering 100 000 cron triggers turns every tick into a multi-second scan + INSERT-bomb. 3. **All-or-nothing commit.** One serialization failure (very possible on Pg under load) rolls back every fire in the tick. - **Fix shape:** add `LIMIT 1000` to the cron `fetch_all`; break the per-tick work into smaller transactions; require a positive `tick_interval_ms` floor *above* 1s (already floored, but the per-row work is what's unbounded, not the tick). --- ## Medium ### M-1 — Outbox has no time-based retention sweep - **Severity:** Medium - **Location:** `crates/manager-core/src/outbox_repo.rs` (no `prune_expired` / `delete_older_than`); `crates/manager-core/src/gc.rs` (sweeps dead_letters + abandoned + app-user tokens, but NOT outbox). - **Summary:** Outbox rows are deleted inline on success or when exceeded; but stuck rows (`claimed_at` set but never released after a process crash mid-dispatch, or rows whose `trigger_depth > max_trigger_depth` slip past) accumulate. With no GC sweep, a bug-induced leak silently bloats the outbox table. On Pi-class Postgres this becomes apparent after weeks, not hours, so Medium. - **Fix shape:** `spawn_outbox_gc` mirroring the dead_letter pattern, pruning rows with `claimed_at < NOW() - INTERVAL '1 hour'` (orphaned claims) and rows past a 30-day retention. ### M-2 — No SSE subscriber cap per IP / app - **Severity:** Medium - **Location:** `crates/orchestrator-core/src/realtime_api.rs:85, 94-159`. - **Summary:** `GET /realtime/topics/{topic}` is gated by topic-token auth, but once authorized the subscriber stays connected indefinitely (heartbeat keepalive). No process-wide / per-IP / per-app cap on active subscriptions. A leaked token plus N connections from one client lets the attacker pin N hyper tasks + N `broadcast::Receiver` slots, each holding a `tokio::sync::broadcast` channel reference. Per topic the broadcast channel buffers `capacity` items; with many topics that's many MB of resident state. - **Fix shape:** per-IP cap (e.g. 4 active SSE connections per remote address), process-wide cap (e.g. 1024), and per-app cap (e.g. 64). 503 on overflow. ### M-3 — No global cap on in-flight HTTP connections / slowloris exposure - **Severity:** Medium - **Location:** `crates/picloud/src/main.rs:79-83` (`TcpListener::bind` then `axum::serve` with default config). - **Summary:** No `set_keepalive`, no `set_nodelay`, no read/write header timeout, no concurrent-connection cap. Hyper defaults are permissive — slowloris-style attackers can hold thousands of half-open TLS handshakes and read-byte-per-second connections. Caddy fronts in prod and mitigates somewhat, but the dev/single- node deployment has no defense. - **Fix shape:** wrap the `TcpListener` with a per-connection `tower::limit::ConcurrencyLimit` (e.g. 256 concurrent connections); set `tcp_keepalive` + `http2_keep_alive_interval` on the hyper builder; set `header_read_timeout = 10s`. Also document a Caddy `servers { timeouts { read_header 10s read 30s write 30s } }` block in both Caddyfiles. ### M-4 — No per-app KV / docs / files row-count or byte quota - **Severity:** Medium - **Location:** repo-side; no `app_quotas` table or check in any `*_repo::insert`. - **Summary:** Per-row caps exist (256 KB / 100 MB) but per-app totals are unbounded. One tenant can store millions of KV rows or fill `PICLOUD_FILES_ROOT` to disk-full. Defense in depth: the CLAUDE.md note ("per-app quotas deferred to v1.2") explicitly acknowledges this; downgrading the severity reflects "documented v1.2 work" but it is still a real DoS today on a shared instance. - **Fix shape:** `app_quotas` table with `bytes_used` columns per service, updated transactionally on insert/update, checked at service-entry. Or a soft cap (warn + log) for v1.1.x; hard quota for v1.2. ### M-5 — Dispatcher retry storm: HTTP-async retries up to `retry_max_attempts` with no per-target circuit breaker - **Severity:** Medium - **Location:** `crates/manager-core/src/dispatcher.rs:885-905`, `trigger_config.rs:83`. - **Summary:** Async HTTP outbox rows retry per `TriggerConfig` default `retry_max_attempts = 3` with exponential backoff. No circuit breaker per `(app_id, script_id)` or per outbound host — a script that always fails fires 3 invocations per event for as long as events arrive. With pubsub fan-out from H-4, one publish → N triggers × 3 retries = `3N` executor invocations. - **Fix shape:** per-`(app_id, script_id)` failure counter with a configurable circuit-break threshold (e.g. 50% failures over 60s → pause for 5 min, write a `dead_letters` row). ### M-6 — Queue depth is unbounded; per-app queue count is unbounded - **Severity:** Medium - **Location:** `crates/manager-core/src/queue_service.rs:67-118` (enqueue), `crates/manager-core/src/queue_repo.rs:264-322` (depth uses `COUNT(*)`). - **Summary:** `queue::enqueue` has a per-message 256 KB cap but no per-queue depth cap. A loop in a public-HTTP script can fill a queue with millions of messages × 256 KB = GB of JSONB; combined with `queue::depth` being a full scan (F-P-006, still open per AUDIT.md remediation state), every depth call magnifies the attack. - **Fix shape:** `PICLOUD_QUEUE_MAX_DEPTH_PER_QUEUE` (e.g. 10 000) + `PICLOUD_QUEUE_MAX_QUEUES_PER_APP` (e.g. 100). ### M-7 — Trigger fan-out from KV/docs/files mutations writes outbox row per matching trigger inline (sync), with no per-event cap - **Severity:** Medium - **Location:** `crates/manager-core/src/outbox_event_emitter.rs:88-103, 142-160, 200-…` (similar pattern). - **Summary:** Each `kv::set` matching N triggers inserts N outbox rows inside the calling request's transaction. No per-event cap on N. Combined with H-4 (unbounded trigger registration), a single `kv::set` can stall a request for seconds while writing thousands of rows. Pubsub fan-out (`pubsub_repo.rs:81-117`) is the same shape. - **Fix shape:** cap `matches.len()` at e.g. 100; reject with a `PolicyTooManyMatches` if exceeded so the developer notices. --- ## Low ### L-1 — Login attempts are not logged for forensics - **Severity:** Low - **Location:** `crates/manager-core/src/auth_api.rs:74-167`. - **Summary:** Failed login attempts are swallowed by the `invalid_credentials()` return path with no structured log line, no counter, no `auth_audit` row. After H-2 is fixed, the rate-limit decisions will need observability hooks; preempt that. - **Fix shape:** `tracing::warn!(client_ip, username, "admin login failed")` (with username **before** the lookup so it can't leak whether the user exists via what's logged). ### L-2 — `tracing::info!` JSON output is unlimited and unrotated - **Severity:** Low - **Location:** `crates/picloud/src/main.rs:222-227`. - **Summary:** `init_tracing` writes to stdout in JSON. Operators running outside docker (where journald handles rotation) won't have automatic rotation; logs eat the disk over weeks. No `tracing_appender` rolling file appender wired in. - **Fix shape:** document the journald / `docker logs --max-size` expectation, or wire `tracing_appender::rolling::daily` behind an env var so the no-supervisor case is also safe. ### L-3 — `attach_principal_if_present` runs Argon2 verify on every API-key prefix candidate on every authed request - **Severity:** Low (already flagged at High in F-P-002; downgraded here because pretty much an authed-user-only problem). - **Location:** `crates/manager-core/src/auth_middleware.rs:192-229`. - **Summary:** AUDIT.md flagged this; including here because the DoS angle is real: a hot user with N API keys serializes every admin request behind N Argon2 verifies. A noisy authenticated tenant can stall the admin plane. ### L-4 — Pubsub `mint_subscriber_token` has no per-app rate limit - **Severity:** Low - **Location:** `crates/manager-core/src/pubsub_service.rs:240-…`. - **Summary:** Token minting requires authz, but an authenticated caller can mint unbounded tokens — each is a tiny HMAC, so no storage DoS, but it makes the topic-secret rotation surface noisier. Worth a sliding-window cap. ### L-5 — `outbox.insert` failures are not visible to the script caller in a non-200 way for fan-out paths - **Severity:** Low (correctness-adjacent) - **Location:** `outbox_event_emitter.rs:101-103` etc. - **Summary:** If an `INSERT INTO outbox` fails (Postgres full disk, deadlock), the SDK call returns the error to the script. Good. But the script is free to retry-in-a-loop — there's no script-side circuit breaker for "the outbox is unavailable". A script could busy-loop hitting outbox-insert failures indefinitely. - **Fix shape:** none beyond M-5's circuit breaker. ### L-6 — `inbox_router` / future SMTP ingress will need its own caps - **Severity:** Low / Info - **Location:** not yet implemented. - **Summary:** Heads-up for v1.2 — SMTP ingress and queue triggers with external receivers will need the same connection-cap + IP rate limit pattern as H-3. --- ## Info ### I-1 — `pubsub::publish_durable` realtime broadcast does `tokio::spawn` + `handle.await` - **Location:** `crates/manager-core/src/pubsub_service.rs:222-236`. - **Note:** the `tokio::spawn(...)` immediately followed by `handle.await` defeats the comment ("Run on a child task so a panicking broadcaster becomes a warn log") — a panic in the spawned task surfaces as a `JoinError` in the awaiting frame and the publish succeeds (because the durable fan-out already committed). The catch is correct, but the comment is slightly misleading. Not a DoS finding. ### I-2 — `DEFAULT_TICK_INTERVAL = 100 ms` is reasonable for Pi - **Location:** `crates/manager-core/src/dispatcher.rs:101`. - **Note:** 100ms tick × 8 rows/tick = 80 dispatches/sec ceiling per process, before the gate. Adequate for a single-node MVP. The env-override (`PICLOUD_DISPATCHER_TICK_INTERVAL_MS`) lets operators trade latency for CPU. ### I-3 — Caddy in dev runs at `:80` with `auto_https off` - **Location:** `caddy/Caddyfile:24-26`. - **Note:** Prod Caddyfile enables auto-HTTPS via ACME. No HTTPS for the dev port is fine for development; mentioned only because every rate-limit story benefits from terminating TLS at Caddy. --- ## Recommendations, prioritized 1. **Login rate limit + global Argon2 semaphore** (H-2). Highest leverage: closes the cheapest-CPU-DoS in the system. 2. **Require HMAC on email-inbound + per-IP bucket** (H-3). One of two anonymous-reachable endpoints; the other is the script data plane, which already has a 503-gate. 3. **Per-app caps on triggers / routes / cron, plus cron tick LIMIT** (H-4 + H-5). Closes the cross-tenant amplification path. Quotas land naturally in the v1.2 work the blueprint already plans. 4. **Explicit `DefaultBodyLimit` at the router root** (H-1). One-line fix; deep value. 5. **Caddy `request_body { max_size … }` and `servers { timeouts …}`** in both Caddyfiles. (Closes H-1 / M-3 at the proxy.) 6. **Outbox GC + per-app KV/docs row caps** (M-1 / M-4). Mechanically easy; defends the long tail. ## Methodology Read AUDIT.md to map prior findings and remediation state, then read: `crates/orchestrator-core/src/gate.rs`, `client.rs`, `api.rs`, `realtime.rs`, `realtime_api.rs`; `crates/manager-core/src/dispatcher.rs`, `cron_scheduler.rs`, `auth_api.rs`, `email_inbound_api.rs`, `pubsub_service.rs`, `pubsub_repo.rs`, `queue_service.rs`, `kv_service.rs`, `docs_service.rs`, `files_service.rs`, `http_service.rs`, `outbox_event_emitter.rs`, `outbox_repo.rs`, `gc.rs`, `triggers_api.rs`, `route_admin.rs`, `trigger_config.rs`, `users_service.rs` (rate-limiter shape); `crates/picloud/src/main.rs`, `lib.rs`; both Caddyfiles. Grepped for `DefaultBodyLimit`, `RequestBodyLimit`, `rate_limit`, `Semaphore`, `tracing_appender`, `set_keepalive`, `count_for_app`, `MAX_TRIGGERS`, `MAX_ROUTES`.