PiCloud/security_audit/08_dos_resource.md

Security audit — DoS, rate limiting & resource exhaustion

Audit date: 2026-06-11 Scope: PiCloud at main (post-v1.1.9, pre-v1.2). Single-node MVP. Reviewer: Subagent 08 of 10 (parallel security audit). Threat model: Solo-dev / Pi-class hardware. A single anonymous attacker or one noisy authenticated tenant should not OOM the host, fill the disk, or starve the executor for other tenants in minutes.

Counts by severity

Severity	Count
Critical	0
High	5
Medium	7
Low	6
Info	3

Verified / no-finding

These were investigated and judged adequate for the threat model:

• Concurrent script execution cap. ExecutionGate uses a single tokio::sync::Semaphore with try_acquire_owned() and translates NoPermits/Closed into AcquireError::Overloaded { retry_after_secs: 1 }, which IntoResponse for ApiError maps to HTTP 503 with Retry-After: 1s (crates/orchestrator-core/src/api.rs:712-721). Default 32, env-tunable via PICLOUD_MAX_CONCURRENT_EXECUTIONS. Both sync HTTP and the dispatcher share the same gate so the dispatcher arms back off correctly when sync HTTP is saturated. Note on fairness: try_acquire is non-blocking so "fairness" is irrelevant — there is no queue, callers get 503 immediately.
• Postgres pool default size. init_db in crates/picloud/src/lib.rs:603-619 now defaults to DEFAULT_DB_MAX_CONNECTIONS = 32 and is env-tunable via PICLOUD_DB_MAX_CONNECTIONS, matching the gate. (F-P-003 from the prior audit was remediated.) Acquire timeout 5s.
• Per-message SDK service caps. kv_service, docs_service, pubsub_service, queue_service, files_service, secrets_service all enforce a per-message JSON-encoded byte cap (256 KB default, except files at 100 MB) before authz. kv_service.rs:141, docs_service.rs:93, pubsub_service.rs:178, queue_service.rs:82. (F-S-001 from the prior audit was remediated.)
• Outbound HTTP defaults (SDK http::request). 30s default timeout, 60s hard ceiling; 10s connect timeout; 10 MB request/response body cap; 5 redirect default, 10 ceiling. SSRF deny-list on by default. See crates/manager-core/src/http_service.rs:41-53.
• Email outbound rate limit. EmailRateLimiter (token bucket + per-(app, recipient) burst + 200/day per-app cap) gates users::* email surfaces (crates/manager-core/src/users_service.rs:139-187). F-S-002 remediated.
• Outbox claim shape. Dispatcher claims at most CLAIM_BATCH = 8 rows per tick (100 ms default) with FOR UPDATE SKIP LOCKED, gates every dispatch through the same ExecutionGate, and reschedules on permit refusal — bounded by design. See crates/manager-core/src/dispatcher.rs:69,193-216,482-491.
• Outbox row deletion. Outbox rows are deleted on success/dead-letter inline (OutboxRepo::delete), so the table doesn't accumulate completed rows under normal load. Bloat risk surveyed below as a Medium (stuck rows).

---

High

H-1 — No Axum default body limit; the only inbound 503-path is the

data-plane user route — all admin/email/webhook handlers fall under Axum's default 2 MB and the /api/v1/execute/{id} handler reads body: Bytes with no explicit override.

• Severity: High
• Location: crates/orchestrator-core/src/api.rs:106-123 (execute_by_id, body: Bytes, no DefaultBodyLimit); crates/manager-core/src/email_inbound_api.rs:131-136 (inbound webhook, body: Bytes); crates/picloud/src/lib.rs:587-593 (router composition — no RequestBodyLimitLayer); admin JSON handlers (Json<…> extractors at auth_api.rs:74, admin_users_api.rs, users_admin_api.rs, etc.).
• Summary: Axum's Bytes/Json extractors default to 2 MB. Caddy has no request_body { max_size … } directive in either Caddyfile or Caddyfile.prod (just unrestricted reverse_proxy), so any request up to 10 GB (Caddy's hard ceiling) can hit picloud. Only the data-plane user-route handler (user_route_handler in api.rs:219) explicitly bounds the body at 10 MiB; everything else rides Axum's default.
  • For /email-inbound/{app_id}/{trigger_id} the body is Bytes with no override — so the 2 MB ceiling is the only thing standing between an anonymous attacker and a 2 MB JSON-parse fork-bomb per request. With no IP rate limit, one client can sustain hundreds of MB/s of CPU-bound serde_json::from_slice and HMAC verification work, saturating one execution worker indefinitely. This is the canonical "no rate limit on a public unauthenticated endpoint" finding — see also H-3.
  • For POST /api/v1/execute/{id} Axum's 2 MB default applies, but the request lands at the executor regardless of script size, meaning a script with body_limit_kb = 32 still pays the serialization cost for a 2 MB body.
• Fix shape:
  • Add a global axum::extract::DefaultBodyLimit::max(2 * 1024 * 1024) layer at the router root (already implicit, but should be explicit
    • lower the default to ~1 MB).
  • Add per-route higher limits where intentional (email inbound is typically tens of KB; lower it to 1 MB explicitly).
  • Caddy: request_body { max_size 5MB } in both Caddyfiles.

H-2 — Login endpoint has no rate limit; Argon2id is the work multiplier

• Severity: High
• Location: crates/manager-core/src/auth_api.rs:74-167.
• Summary: POST /api/v1/admin/auth/login is reachable unauthenticated, and on every request it does a username lookup + Argon2id verify (m=19456 KiB, t=2 default Argon2id params) via spawn_blocking. There is no per-IP, per-username, or process-wide cap on requests / failed attempts. Argon2 is ~50–150 ms per attempt on Pi-class hardware; an anonymous attacker with 32 in-flight HTTP requests can keep every blocking-thread worker pinned on Argon2 forever, and can attempt ≈600 password guesses/sec process-wide. Username enumeration is correctly defended (dummy-hash path) so the same Argon2 cost is paid on bad usernames — which means the cost inflicts on the defender for every miss.
• Threat: Combined with H-1 (no body limit on Json<LoginRequest>) this is the cheapest CPU-DoS in the system. A laptop attacker can fully saturate a Raspberry Pi's executor in seconds.
• Fix shape: add a sliding-window login rate limiter keyed on (client_ip, username) (e.g. 5 attempts / 60s, lockout for 5 min after 20), and a global Semaphore cap (e.g. 4 concurrent in-flight Argon2 verifies) so admin auth can't starve the executor's spawn_blocking pool. Memory only — Postgres-backed counters add a round-trip. The existing EmailRateLimiter pattern in users_service.rs:139 is the shape to copy.

H-3 — Email-inbound webhook has no IP rate limit; HMAC is optional

• Severity: High
• Location: crates/manager-core/src/email_inbound_api.rs:131-191, mounted outside require_authenticated at crates/picloud/src/lib.rs:571-575.
• Summary: POST /api/v1/email-inbound/{app_id}/{trigger_id} is public. When the trigger has an inbound_secret configured, HMAC verification is correct (timestamp-bound, ±5 min, in-memory replay dedup). But:
  1. If the operator did NOT configure an inbound_secret for the trigger, no signature is required at all (see the if let (Some(ct), Some(nonce)) = … guard at line 150). An anonymous attacker who knows the trigger UUID can enqueue an unbounded number of outbox rows + executor invocations per second, bypassing every other rate limit.
  2. With a secret configured, attackers still pay no cost for sending garbage — the receiver must decrypt the secret from the secret store (one DB call + master-key open) and HMAC-verify per request before rejecting. No (app_id, trigger_id) invalid-signature bucket. So an attacker who learns the URL but not the secret can still pin a worker thread on secret-decrypt + HMAC.
  3. The body is Bytes without a DefaultBodyLimit override — falls under Axum's 2 MB cap (see H-1).
• Threat: A single anonymous HTTP client knowing only an app_id
  • a trigger_id (often leaked through public docs / TLS cert transparency logs / mis-configured provider dashboards) can enqueue enough outbox rows to fill the dispatcher batch every tick. With retries, one row → up to retry_max_attempts (default 3) executions. Outbox grows unbounded; disk fills.
• Fix shape: require HMAC at the trigger-row level (DB CHECK that email triggers MUST have a secret); add per-(app_id, trigger_id) token-bucket gating both inbound_secret_missing (drop fast) and bad_signature paths; add a DefaultBodyLimit::max(1 MB) to the router merge in lib.rs.

H-4 — No per-app cap on trigger / route / cron registration

• Severity: High
• Location: crates/manager-core/src/triggers_api.rs:127-140 (create endpoints), crates/manager-core/src/route_admin.rs:160 (create_route); no counterpart MAX_TRIGGERS_PER_APP / MAX_ROUTES_PER_APP constant in the codebase.
• Summary: An authenticated app developer can register an unbounded number of triggers (KV, docs, files, pubsub, cron, email, queue, dead-letter) and routes. Two amplifications follow:
  1. Cron storm. cron_scheduler::tick SELECTs all enabled cron rows in one transaction (fetch_all, no LIMIT), iterates in Rust, and inserts one outbox row per due trigger (cron_scheduler.rs:117-167). 10 000 cron triggers all set to */1 * * * * * (every second) → 10 000 outbox rows per second, each becoming a 100ms dispatcher claim of 8 rows + 32 concurrent executions. The dispatcher backs up; outbox bloats; Postgres next_attempt_at index degrades.
  2. Pubsub fan-out. pubsub_repo::fan_out_publish SELECTs all enabled pubsub triggers for the app, filters in Rust on topic_matches, inserts one outbox row per match in a single transaction (pubsub_repo.rs:81-115). N triggers subscribed to the same topic → N writes per publish.
  3. KV / docs / files fan-out. Same pattern in outbox_event_emitter.rs:88-103 — one outbox row per matching trigger per mutation.
• Threat: A single tenant with AppTriggerCreate capability can register thousands of triggers and bring the whole node down by pubsub-publishing one message; or fill disk by cron-firing. Cross-tenant impact via shared dispatcher.
• Fix shape: per-app trigger / route caps (e.g. 100 each, env- tunable) enforced in the *_api create handlers; per-publish fan-out cap (e.g. 100 matched triggers per publish, then dead-letter the excess with a clear error); cron-scheduler tick LIMIT clause (LIMIT 1000) so a misbehaving app can't ride the whole tick budget.

H-5 — Cron-scheduler tick is unbounded fetch_all; one slow row blocks all

• Severity: High
• Location: crates/manager-core/src/cron_scheduler.rs:115-173.
• Summary: Every cron tick (default 30s) opens a transaction, SELECTs every enabled cron row with FOR UPDATE OF d SKIP LOCKED, computes next_due in Rust, and INSERTs an outbox row + UPDATEs last_fired_at per due row — all serially inside one transaction. Three concrete failure modes:
  1. Long-running tx. With N triggers and the per-row cost ~1ms, a few thousand cron rows make the tick exceed the tick interval, pile up overlapping transactions, and hold row-locks on cron_trigger_details for seconds — blocking dashboard reads of the same table.
  2. No LIMIT. A malicious or buggy tenant registering 100 000 cron triggers turns every tick into a multi-second scan + INSERT-bomb.
  3. All-or-nothing commit. One serialization failure (very possible on Pg under load) rolls back every fire in the tick.
• Fix shape: add LIMIT 1000 to the cron fetch_all; break the per-tick work into smaller transactions; require a positive tick_interval_ms floor above 1s (already floored, but the per-row work is what's unbounded, not the tick).

---

Medium

M-1 — Outbox has no time-based retention sweep

• Severity: Medium
• Location: crates/manager-core/src/outbox_repo.rs (no prune_expired / delete_older_than); crates/manager-core/src/gc.rs (sweeps dead_letters + abandoned + app-user tokens, but NOT outbox).
• Summary: Outbox rows are deleted inline on success or when exceeded; but stuck rows (claimed_at set but never released after a process crash mid-dispatch, or rows whose trigger_depth > max_trigger_depth slip past) accumulate. With no GC sweep, a bug-induced leak silently bloats the outbox table. On Pi-class Postgres this becomes apparent after weeks, not hours, so Medium.
• Fix shape: spawn_outbox_gc mirroring the dead_letter pattern, pruning rows with claimed_at < NOW() - INTERVAL '1 hour' (orphaned claims) and rows past a 30-day retention.

M-2 — No SSE subscriber cap per IP / app

• Severity: Medium
• Location: crates/orchestrator-core/src/realtime_api.rs:85, 94-159.
• Summary: GET /realtime/topics/{topic} is gated by topic-token auth, but once authorized the subscriber stays connected indefinitely (heartbeat keepalive). No process-wide / per-IP / per-app cap on active subscriptions. A leaked token plus N connections from one client lets the attacker pin N hyper tasks + N broadcast::Receiver slots, each holding a tokio::sync::broadcast channel reference. Per topic the broadcast channel buffers capacity items; with many topics that's many MB of resident state.
• Fix shape: per-IP cap (e.g. 4 active SSE connections per remote address), process-wide cap (e.g. 1024), and per-app cap (e.g. 64). 503 on overflow.

M-3 — No global cap on in-flight HTTP connections / slowloris exposure

• Severity: Medium
• Location: crates/picloud/src/main.rs:79-83 (TcpListener::bind then axum::serve with default config).
• Summary: No set_keepalive, no set_nodelay, no read/write header timeout, no concurrent-connection cap. Hyper defaults are permissive — slowloris-style attackers can hold thousands of half-open TLS handshakes and read-byte-per-second connections. Caddy fronts in prod and mitigates somewhat, but the dev/single- node deployment has no defense.
• Fix shape: wrap the TcpListener with a per-connection tower::limit::ConcurrencyLimit (e.g. 256 concurrent connections); set tcp_keepalive + http2_keep_alive_interval on the hyper builder; set header_read_timeout = 10s. Also document a Caddy servers { timeouts { read_header 10s read 30s write 30s } } block in both Caddyfiles.

M-4 — No per-app KV / docs / files row-count or byte quota

• Severity: Medium
• Location: repo-side; no app_quotas table or check in any *_repo::insert.
• Summary: Per-row caps exist (256 KB / 100 MB) but per-app totals are unbounded. One tenant can store millions of KV rows or fill PICLOUD_FILES_ROOT to disk-full. Defense in depth: the CLAUDE.md note ("per-app quotas deferred to v1.2") explicitly acknowledges this; downgrading the severity reflects "documented v1.2 work" but it is still a real DoS today on a shared instance.
• Fix shape: app_quotas table with bytes_used columns per service, updated transactionally on insert/update, checked at service-entry. Or a soft cap (warn + log) for v1.1.x; hard quota for v1.2.

M-5 — Dispatcher retry storm: HTTP-async retries up to retry_max_attempts with no per-target circuit breaker

• Severity: Medium
• Location: crates/manager-core/src/dispatcher.rs:885-905, trigger_config.rs:83.
• Summary: Async HTTP outbox rows retry per TriggerConfig default retry_max_attempts = 3 with exponential backoff. No circuit breaker per (app_id, script_id) or per outbound host — a script that always fails fires 3 invocations per event for as long as events arrive. With pubsub fan-out from H-4, one publish → N triggers × 3 retries = 3N executor invocations.
• Fix shape: per-(app_id, script_id) failure counter with a configurable circuit-break threshold (e.g. 50% failures over 60s → pause for 5 min, write a dead_letters row).

M-6 — Queue depth is unbounded; per-app queue count is unbounded

• Severity: Medium
• Location: crates/manager-core/src/queue_service.rs:67-118 (enqueue), crates/manager-core/src/queue_repo.rs:264-322 (depth uses COUNT(*)).
• Summary: queue::enqueue has a per-message 256 KB cap but no per-queue depth cap. A loop in a public-HTTP script can fill a queue with millions of messages × 256 KB = GB of JSONB; combined with queue::depth being a full scan (F-P-006, still open per AUDIT.md remediation state), every depth call magnifies the attack.
• Fix shape: PICLOUD_QUEUE_MAX_DEPTH_PER_QUEUE (e.g. 10 000) + PICLOUD_QUEUE_MAX_QUEUES_PER_APP (e.g. 100).

M-7 — Trigger fan-out from KV/docs/files mutations writes outbox row per matching trigger inline (sync), with no per-event cap

• Severity: Medium
• Location: crates/manager-core/src/outbox_event_emitter.rs:88-103, 142-160, 200-… (similar pattern).
• Summary: Each kv::set matching N triggers inserts N outbox rows inside the calling request's transaction. No per-event cap on N. Combined with H-4 (unbounded trigger registration), a single kv::set can stall a request for seconds while writing thousands of rows. Pubsub fan-out (pubsub_repo.rs:81-117) is the same shape.
• Fix shape: cap matches.len() at e.g. 100; reject with a PolicyTooManyMatches if exceeded so the developer notices.

---

Low

L-1 — Login attempts are not logged for forensics

• Severity: Low
• Location: crates/manager-core/src/auth_api.rs:74-167.
• Summary: Failed login attempts are swallowed by the invalid_credentials() return path with no structured log line, no counter, no auth_audit row. After H-2 is fixed, the rate-limit decisions will need observability hooks; preempt that.
• Fix shape: tracing::warn!(client_ip, username, "admin login failed") (with username before the lookup so it can't leak whether the user exists via what's logged).

L-2 — tracing::info! JSON output is unlimited and unrotated

• Severity: Low
• Location: crates/picloud/src/main.rs:222-227.
• Summary: init_tracing writes to stdout in JSON. Operators running outside docker (where journald handles rotation) won't have automatic rotation; logs eat the disk over weeks. No tracing_appender rolling file appender wired in.
• Fix shape: document the journald / docker logs --max-size expectation, or wire tracing_appender::rolling::daily behind an env var so the no-supervisor case is also safe.

L-3 — attach_principal_if_present runs Argon2 verify on every API-key prefix candidate on every authed request

• Severity: Low (already flagged at High in F-P-002; downgraded here because pretty much an authed-user-only problem).
• Location: crates/manager-core/src/auth_middleware.rs:192-229.
• Summary: AUDIT.md flagged this; including here because the DoS angle is real: a hot user with N API keys serializes every admin request behind N Argon2 verifies. A noisy authenticated tenant can stall the admin plane.

L-4 — Pubsub mint_subscriber_token has no per-app rate limit

• Severity: Low
• Location: crates/manager-core/src/pubsub_service.rs:240-….
• Summary: Token minting requires authz, but an authenticated caller can mint unbounded tokens — each is a tiny HMAC, so no storage DoS, but it makes the topic-secret rotation surface noisier. Worth a sliding-window cap.

L-5 — outbox.insert failures are not visible to the script caller in a non-200 way for fan-out paths

• Severity: Low (correctness-adjacent)
• Location: outbox_event_emitter.rs:101-103 etc.
• Summary: If an INSERT INTO outbox fails (Postgres full disk, deadlock), the SDK call returns the error to the script. Good. But the script is free to retry-in-a-loop — there's no script-side circuit breaker for "the outbox is unavailable". A script could busy-loop hitting outbox-insert failures indefinitely.
• Fix shape: none beyond M-5's circuit breaker.

L-6 — inbox_router / future SMTP ingress will need its own caps

• Severity: Low / Info
• Location: not yet implemented.
• Summary: Heads-up for v1.2 — SMTP ingress and queue triggers with external receivers will need the same connection-cap + IP rate limit pattern as H-3.

---

Info

I-1 — pubsub::publish_durable realtime broadcast does tokio::spawn + handle.await

• Location: crates/manager-core/src/pubsub_service.rs:222-236.
• Note: the tokio::spawn(...) immediately followed by handle.await defeats the comment ("Run on a child task so a panicking broadcaster becomes a warn log") — a panic in the spawned task surfaces as a JoinError in the awaiting frame and the publish succeeds (because the durable fan-out already committed). The catch is correct, but the comment is slightly misleading. Not a DoS finding.

I-2 — DEFAULT_TICK_INTERVAL = 100 ms is reasonable for Pi

• Location: crates/manager-core/src/dispatcher.rs:101.
• Note: 100ms tick × 8 rows/tick = 80 dispatches/sec ceiling per process, before the gate. Adequate for a single-node MVP. The env-override (PICLOUD_DISPATCHER_TICK_INTERVAL_MS) lets operators trade latency for CPU.

I-3 — Caddy in dev runs at :80 with auto_https off

• Location: caddy/Caddyfile:24-26.
• Note: Prod Caddyfile enables auto-HTTPS via ACME. No HTTPS for the dev port is fine for development; mentioned only because every rate-limit story benefits from terminating TLS at Caddy.

---

Recommendations, prioritized

1. Login rate limit + global Argon2 semaphore (H-2). Highest leverage: closes the cheapest-CPU-DoS in the system.
2. Require HMAC on email-inbound + per-IP bucket (H-3). One of two anonymous-reachable endpoints; the other is the script data plane, which already has a 503-gate.
3. Per-app caps on triggers / routes / cron, plus cron tick LIMIT (H-4 + H-5). Closes the cross-tenant amplification path. Quotas land naturally in the v1.2 work the blueprint already plans.
4. Explicit DefaultBodyLimit at the router root (H-1). One-line fix; deep value.
5. Caddy request_body { max_size … } and servers { timeouts …} in both Caddyfiles. (Closes H-1 / M-3 at the proxy.)
6. Outbox GC + per-app KV/docs row caps (M-1 / M-4). Mechanically easy; defends the long tail.

Methodology

Read AUDIT.md to map prior findings and remediation state, then read: crates/orchestrator-core/src/gate.rs, client.rs, api.rs, realtime.rs, realtime_api.rs; crates/manager-core/src/dispatcher.rs, cron_scheduler.rs, auth_api.rs, email_inbound_api.rs, pubsub_service.rs, pubsub_repo.rs, queue_service.rs, kv_service.rs, docs_service.rs, files_service.rs, http_service.rs, outbox_event_emitter.rs, outbox_repo.rs, gc.rs, triggers_api.rs, route_admin.rs, trigger_config.rs, users_service.rs (rate-limiter shape); crates/picloud/src/main.rs, lib.rs; both Caddyfiles. Grepped for DefaultBodyLimit, RequestBodyLimit, rate_limit, Semaphore, tracing_appender, set_keepalive, count_for_app, MAX_TRIGGERS, MAX_ROUTES.