style: cargo fmt across audit-2026-06-11 tier-3 changes

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-12 18:38:28 +02:00
parent e8cc3afa07
commit ec4a2aa24a
15 changed files with 1838 additions and 5 deletions

View File

@@ -0,0 +1,454 @@
# Security audit — DoS, rate limiting & resource exhaustion
**Audit date:** 2026-06-11
**Scope:** PiCloud at `main` (post-v1.1.9, pre-v1.2). Single-node MVP.
**Reviewer:** Subagent 08 of 10 (parallel security audit).
**Threat model:** Solo-dev / Pi-class hardware. A single anonymous attacker
or one noisy authenticated tenant should not OOM the host, fill the disk,
or starve the executor for other tenants in minutes.
## Counts by severity
| Severity | Count |
|---|---|
| Critical | 0 |
| High | 5 |
| Medium | 7 |
| Low | 6 |
| Info | 3 |
## Verified / no-finding
These were investigated and judged adequate for the threat model:
- **Concurrent script execution cap.** `ExecutionGate` uses a single
`tokio::sync::Semaphore` with `try_acquire_owned()` and translates
`NoPermits`/`Closed` into `AcquireError::Overloaded { retry_after_secs: 1 }`,
which `IntoResponse` for `ApiError` maps to HTTP 503 with `Retry-After: 1s`
(`crates/orchestrator-core/src/api.rs:712-721`). Default 32, env-tunable
via `PICLOUD_MAX_CONCURRENT_EXECUTIONS`. Both sync HTTP and the dispatcher
share the same gate so the dispatcher arms back off correctly when sync
HTTP is saturated. **Note on fairness:** `try_acquire` is non-blocking so
"fairness" is irrelevant — there is no queue, callers get 503 immediately.
- **Postgres pool default size.** `init_db` in
`crates/picloud/src/lib.rs:603-619` now defaults to
`DEFAULT_DB_MAX_CONNECTIONS = 32` and is env-tunable via
`PICLOUD_DB_MAX_CONNECTIONS`, matching the gate. (F-P-003 from the prior
audit was remediated.) Acquire timeout 5s.
- **Per-message SDK service caps.** `kv_service`, `docs_service`,
`pubsub_service`, `queue_service`, `files_service`, `secrets_service`
all enforce a per-message JSON-encoded byte cap (256 KB default, except
files at 100 MB) **before** authz. `kv_service.rs:141`, `docs_service.rs:93`,
`pubsub_service.rs:178`, `queue_service.rs:82`. (F-S-001 from the prior
audit was remediated.)
- **Outbound HTTP defaults (SDK http::request).** 30s default timeout,
60s hard ceiling; 10s connect timeout; 10 MB request/response body cap;
5 redirect default, 10 ceiling. SSRF deny-list on by default. See
`crates/manager-core/src/http_service.rs:41-53`.
- **Email outbound rate limit.** `EmailRateLimiter` (token bucket +
per-(app, recipient) burst + 200/day per-app cap) gates `users::*` email
surfaces (`crates/manager-core/src/users_service.rs:139-187`). F-S-002
remediated.
- **Outbox claim shape.** Dispatcher claims at most `CLAIM_BATCH = 8`
rows per tick (100 ms default) with `FOR UPDATE SKIP LOCKED`, gates
every dispatch through the same `ExecutionGate`, and reschedules on
permit refusal — bounded by design. See
`crates/manager-core/src/dispatcher.rs:69,193-216,482-491`.
- **Outbox row deletion.** Outbox rows are deleted on success/dead-letter
inline (`OutboxRepo::delete`), so the table doesn't accumulate
completed rows under normal load. Bloat risk surveyed below as a
Medium (stuck rows).
---
## High
### H-1 — No Axum default body limit; the only inbound 503-path is the
data-plane user route — all admin/email/webhook handlers fall under Axum's
default 2 MB and the `/api/v1/execute/{id}` handler reads `body: Bytes`
with no explicit override.
- **Severity:** High
- **Location:**
`crates/orchestrator-core/src/api.rs:106-123` (execute_by_id, `body: Bytes`,
no `DefaultBodyLimit`);
`crates/manager-core/src/email_inbound_api.rs:131-136` (inbound webhook,
`body: Bytes`);
`crates/picloud/src/lib.rs:587-593` (router composition — no
`RequestBodyLimitLayer`); admin JSON handlers (`Json<…>` extractors at
`auth_api.rs:74`, `admin_users_api.rs`, `users_admin_api.rs`, etc.).
- **Summary:** Axum's `Bytes`/`Json` extractors default to 2 MB. `Caddy`
has no `request_body { max_size … }` directive in either `Caddyfile`
or `Caddyfile.prod` (just unrestricted `reverse_proxy`), so any
request up to **10 GB** (Caddy's hard ceiling) can hit picloud. Only
the data-plane user-route handler (`user_route_handler` in
`api.rs:219`) explicitly bounds the body at 10 MiB; everything else
rides Axum's default.
- For `/email-inbound/{app_id}/{trigger_id}` the body is `Bytes`
with no override — so the 2 MB ceiling is the only thing standing
between an anonymous attacker and a 2 MB JSON-parse fork-bomb per
request. With no IP rate limit, one client can sustain hundreds of
MB/s of CPU-bound `serde_json::from_slice` and HMAC verification
work, saturating one execution worker indefinitely. This is the
canonical "no rate limit on a public unauthenticated endpoint"
finding — see also H-3.
- For `POST /api/v1/execute/{id}` Axum's 2 MB default applies, but
the request lands at the executor regardless of script size,
meaning a script with `body_limit_kb = 32` still pays the
serialization cost for a 2 MB body.
- **Fix shape:**
- Add a global `axum::extract::DefaultBodyLimit::max(2 * 1024 * 1024)`
layer at the router root (already implicit, but should be explicit
+ lower the default to ~1 MB).
- Add per-route higher limits where intentional (email inbound is
typically tens of KB; lower it to 1 MB explicitly).
- Caddy: `request_body { max_size 5MB }` in both Caddyfiles.
### H-2 — Login endpoint has no rate limit; Argon2id is the work multiplier
- **Severity:** High
- **Location:** `crates/manager-core/src/auth_api.rs:74-167`.
- **Summary:** `POST /api/v1/admin/auth/login` is reachable
unauthenticated, and on every request it does a username lookup +
Argon2id verify (m=19456 KiB, t=2 default Argon2id params) via
`spawn_blocking`. There is no per-IP, per-username, or process-wide
cap on requests / failed attempts. Argon2 is ~50150 ms per attempt
on Pi-class hardware; an anonymous attacker with 32 in-flight HTTP
requests can keep every blocking-thread worker pinned on Argon2
forever, **and** can attempt ≈600 password guesses/sec process-wide.
Username enumeration is correctly defended (dummy-hash path) so the
same Argon2 cost is paid on bad usernames — which means the cost
inflicts on the defender for every miss.
- **Threat:** Combined with H-1 (no body limit on `Json<LoginRequest>`)
this is the cheapest CPU-DoS in the system. A laptop attacker can
fully saturate a Raspberry Pi's executor in seconds.
- **Fix shape:** add a sliding-window login rate limiter keyed on
`(client_ip, username)` (e.g. 5 attempts / 60s, lockout for 5 min
after 20), and a global `Semaphore` cap (e.g. 4 concurrent in-flight
Argon2 verifies) so admin auth can't starve the executor's
`spawn_blocking` pool. Memory only — Postgres-backed counters add a
round-trip. The existing `EmailRateLimiter` pattern in
`users_service.rs:139` is the shape to copy.
### H-3 — Email-inbound webhook has no IP rate limit; HMAC is optional
- **Severity:** High
- **Location:** `crates/manager-core/src/email_inbound_api.rs:131-191`,
mounted **outside** `require_authenticated` at
`crates/picloud/src/lib.rs:571-575`.
- **Summary:** `POST /api/v1/email-inbound/{app_id}/{trigger_id}` is
public. When the trigger has an `inbound_secret` configured, HMAC
verification is correct (timestamp-bound, ±5 min, in-memory replay
dedup). But:
1. If the operator did NOT configure an `inbound_secret` for the
trigger, **no signature is required at all** (see the
`if let (Some(ct), Some(nonce)) = …` guard at line 150). An
anonymous attacker who knows the trigger UUID can enqueue an
unbounded number of outbox rows + executor invocations per
second, bypassing every other rate limit.
2. With a secret configured, attackers still pay no cost for sending
garbage — the receiver must decrypt the secret from the secret
store (one DB call + master-key open) and HMAC-verify per request
**before** rejecting. No `(app_id, trigger_id)` invalid-signature
bucket. So an attacker who learns the URL but not the secret can
still pin a worker thread on secret-decrypt + HMAC.
3. The body is `Bytes` without a `DefaultBodyLimit` override — falls
under Axum's 2 MB cap (see H-1).
- **Threat:** A single anonymous HTTP client knowing only an `app_id`
+ a `trigger_id` (often leaked through public docs / TLS cert
transparency logs / mis-configured provider dashboards) can enqueue
enough outbox rows to fill the dispatcher batch every tick. With
retries, one row → up to `retry_max_attempts` (default 3) executions.
Outbox grows unbounded; disk fills.
- **Fix shape:** require HMAC at the trigger-row level (DB CHECK that
email triggers MUST have a secret); add per-`(app_id, trigger_id)`
token-bucket gating both `inbound_secret_missing` (drop fast) and
`bad_signature` paths; add a `DefaultBodyLimit::max(1 MB)` to the
router merge in `lib.rs`.
### H-4 — No per-app cap on trigger / route / cron registration
- **Severity:** High
- **Location:**
`crates/manager-core/src/triggers_api.rs:127-140` (create endpoints),
`crates/manager-core/src/route_admin.rs:160` (create_route);
no counterpart `MAX_TRIGGERS_PER_APP` / `MAX_ROUTES_PER_APP` constant
in the codebase.
- **Summary:** An authenticated app developer can register an
unbounded number of triggers (KV, docs, files, pubsub, cron, email,
queue, dead-letter) and routes. Two amplifications follow:
1. **Cron storm.** `cron_scheduler::tick` SELECTs **all** enabled
cron rows in one transaction (`fetch_all`, no `LIMIT`), iterates
in Rust, and inserts one outbox row per due trigger
(`cron_scheduler.rs:117-167`). 10 000 cron triggers all set to
`*/1 * * * * *` (every second) → 10 000 outbox rows per second,
each becoming a 100ms dispatcher claim of 8 rows + 32 concurrent
executions. The dispatcher backs up; outbox bloats; Postgres
`next_attempt_at` index degrades.
2. **Pubsub fan-out.** `pubsub_repo::fan_out_publish` SELECTs all
enabled `pubsub` triggers for the app, filters in Rust on
`topic_matches`, inserts one outbox row per match in a single
transaction (`pubsub_repo.rs:81-115`). N triggers subscribed to
the same topic → N writes per publish.
3. **KV / docs / files fan-out.** Same pattern in
`outbox_event_emitter.rs:88-103` — one outbox row per matching
trigger per mutation.
- **Threat:** A single tenant with `AppTriggerCreate` capability can
register thousands of triggers and bring the whole node down by
pubsub-publishing one message; or fill disk by cron-firing.
Cross-tenant impact via shared dispatcher.
- **Fix shape:** per-app trigger / route caps (e.g. 100 each, env-
tunable) enforced in the *_api create handlers; per-publish fan-out
cap (e.g. 100 matched triggers per publish, then dead-letter the
excess with a clear error); cron-scheduler tick LIMIT clause
(`LIMIT 1000`) so a misbehaving app can't ride the whole tick budget.
### H-5 — Cron-scheduler tick is unbounded `fetch_all`; one slow row blocks all
- **Severity:** High
- **Location:** `crates/manager-core/src/cron_scheduler.rs:115-173`.
- **Summary:** Every cron tick (default 30s) opens a transaction,
SELECTs every enabled cron row with `FOR UPDATE OF d SKIP LOCKED`,
computes `next_due` in Rust, and INSERTs an outbox row + UPDATEs
`last_fired_at` per due row — **all serially inside one transaction**.
Three concrete failure modes:
1. **Long-running tx.** With N triggers and the per-row cost ~1ms,
a few thousand cron rows make the tick exceed the tick interval,
pile up overlapping transactions, and hold row-locks on
`cron_trigger_details` for seconds — blocking dashboard reads
of the same table.
2. **No LIMIT.** A malicious or buggy tenant registering 100 000
cron triggers turns every tick into a multi-second scan +
INSERT-bomb.
3. **All-or-nothing commit.** One serialization failure (very
possible on Pg under load) rolls back every fire in the tick.
- **Fix shape:** add `LIMIT 1000` to the cron `fetch_all`; break the
per-tick work into smaller transactions; require a positive
`tick_interval_ms` floor *above* 1s (already floored, but the
per-row work is what's unbounded, not the tick).
---
## Medium
### M-1 — Outbox has no time-based retention sweep
- **Severity:** Medium
- **Location:** `crates/manager-core/src/outbox_repo.rs` (no
`prune_expired` / `delete_older_than`);
`crates/manager-core/src/gc.rs` (sweeps dead_letters + abandoned +
app-user tokens, but NOT outbox).
- **Summary:** Outbox rows are deleted inline on success or when
exceeded; but stuck rows (`claimed_at` set but never released after
a process crash mid-dispatch, or rows whose `trigger_depth >
max_trigger_depth` slip past) accumulate. With no GC sweep, a
bug-induced leak silently bloats the outbox table. On Pi-class
Postgres this becomes apparent after weeks, not hours, so Medium.
- **Fix shape:** `spawn_outbox_gc` mirroring the dead_letter pattern,
pruning rows with `claimed_at < NOW() - INTERVAL '1 hour'`
(orphaned claims) and rows past a 30-day retention.
### M-2 — No SSE subscriber cap per IP / app
- **Severity:** Medium
- **Location:** `crates/orchestrator-core/src/realtime_api.rs:85,
94-159`.
- **Summary:** `GET /realtime/topics/{topic}` is gated by topic-token
auth, but once authorized the subscriber stays connected indefinitely
(heartbeat keepalive). No process-wide / per-IP / per-app cap on
active subscriptions. A leaked token plus N connections from one
client lets the attacker pin N hyper tasks + N `broadcast::Receiver`
slots, each holding a `tokio::sync::broadcast` channel reference.
Per topic the broadcast channel buffers `capacity` items; with many
topics that's many MB of resident state.
- **Fix shape:** per-IP cap (e.g. 4 active SSE connections per remote
address), process-wide cap (e.g. 1024), and per-app cap
(e.g. 64). 503 on overflow.
### M-3 — No global cap on in-flight HTTP connections / slowloris exposure
- **Severity:** Medium
- **Location:** `crates/picloud/src/main.rs:79-83`
(`TcpListener::bind` then `axum::serve` with default config).
- **Summary:** No `set_keepalive`, no `set_nodelay`, no read/write
header timeout, no concurrent-connection cap. Hyper defaults are
permissive — slowloris-style attackers can hold thousands of
half-open TLS handshakes and read-byte-per-second connections.
Caddy fronts in prod and mitigates somewhat, but the dev/single-
node deployment has no defense.
- **Fix shape:** wrap the `TcpListener` with a per-connection
`tower::limit::ConcurrencyLimit` (e.g. 256 concurrent connections);
set `tcp_keepalive` + `http2_keep_alive_interval` on the hyper
builder; set `header_read_timeout = 10s`. Also document a Caddy
`servers { timeouts { read_header 10s read 30s write 30s } }` block
in both Caddyfiles.
### M-4 — No per-app KV / docs / files row-count or byte quota
- **Severity:** Medium
- **Location:** repo-side; no `app_quotas` table or check in any
`*_repo::insert`.
- **Summary:** Per-row caps exist (256 KB / 100 MB) but per-app
totals are unbounded. One tenant can store millions of KV rows or
fill `PICLOUD_FILES_ROOT` to disk-full. Defense in depth: the
CLAUDE.md note ("per-app quotas deferred to v1.2") explicitly
acknowledges this; downgrading the severity reflects "documented
v1.2 work" but it is still a real DoS today on a shared instance.
- **Fix shape:** `app_quotas` table with `bytes_used` columns per
service, updated transactionally on insert/update, checked at
service-entry. Or a soft cap (warn + log) for v1.1.x; hard quota
for v1.2.
### M-5 — Dispatcher retry storm: HTTP-async retries up to `retry_max_attempts` with no per-target circuit breaker
- **Severity:** Medium
- **Location:** `crates/manager-core/src/dispatcher.rs:885-905`,
`trigger_config.rs:83`.
- **Summary:** Async HTTP outbox rows retry per `TriggerConfig`
default `retry_max_attempts = 3` with exponential backoff. No
circuit breaker per `(app_id, script_id)` or per outbound host —
a script that always fails fires 3 invocations per event for as
long as events arrive. With pubsub fan-out from H-4, one
publish → N triggers × 3 retries = `3N` executor invocations.
- **Fix shape:** per-`(app_id, script_id)` failure counter with a
configurable circuit-break threshold (e.g. 50% failures over 60s →
pause for 5 min, write a `dead_letters` row).
### M-6 — Queue depth is unbounded; per-app queue count is unbounded
- **Severity:** Medium
- **Location:** `crates/manager-core/src/queue_service.rs:67-118`
(enqueue), `crates/manager-core/src/queue_repo.rs:264-322`
(depth uses `COUNT(*)`).
- **Summary:** `queue::enqueue` has a per-message 256 KB cap but no
per-queue depth cap. A loop in a public-HTTP script can fill a
queue with millions of messages × 256 KB = GB of JSONB; combined
with `queue::depth` being a full scan (F-P-006, still open per
AUDIT.md remediation state), every depth call magnifies the
attack.
- **Fix shape:** `PICLOUD_QUEUE_MAX_DEPTH_PER_QUEUE` (e.g. 10 000) +
`PICLOUD_QUEUE_MAX_QUEUES_PER_APP` (e.g. 100).
### M-7 — Trigger fan-out from KV/docs/files mutations writes outbox row per matching trigger inline (sync), with no per-event cap
- **Severity:** Medium
- **Location:**
`crates/manager-core/src/outbox_event_emitter.rs:88-103,
142-160, 200-…` (similar pattern).
- **Summary:** Each `kv::set` matching N triggers inserts N outbox
rows inside the calling request's transaction. No per-event cap
on N. Combined with H-4 (unbounded trigger registration), a
single `kv::set` can stall a request for seconds while writing
thousands of rows. Pubsub fan-out (`pubsub_repo.rs:81-117`) is
the same shape.
- **Fix shape:** cap `matches.len()` at e.g. 100; reject with a
`PolicyTooManyMatches` if exceeded so the developer notices.
---
## Low
### L-1 — Login attempts are not logged for forensics
- **Severity:** Low
- **Location:** `crates/manager-core/src/auth_api.rs:74-167`.
- **Summary:** Failed login attempts are swallowed by the
`invalid_credentials()` return path with no structured log line, no
counter, no `auth_audit` row. After H-2 is fixed, the rate-limit
decisions will need observability hooks; preempt that.
- **Fix shape:** `tracing::warn!(client_ip, username, "admin login
failed")` (with username **before** the lookup so it can't leak
whether the user exists via what's logged).
### L-2 — `tracing::info!` JSON output is unlimited and unrotated
- **Severity:** Low
- **Location:** `crates/picloud/src/main.rs:222-227`.
- **Summary:** `init_tracing` writes to stdout in JSON. Operators
running outside docker (where journald handles rotation) won't have
automatic rotation; logs eat the disk over weeks. No `tracing_appender`
rolling file appender wired in.
- **Fix shape:** document the journald / `docker logs --max-size`
expectation, or wire `tracing_appender::rolling::daily` behind an env
var so the no-supervisor case is also safe.
### L-3 — `attach_principal_if_present` runs Argon2 verify on every API-key prefix candidate on every authed request
- **Severity:** Low (already flagged at High in F-P-002; downgraded
here because pretty much an authed-user-only problem).
- **Location:** `crates/manager-core/src/auth_middleware.rs:192-229`.
- **Summary:** AUDIT.md flagged this; including here because the
DoS angle is real: a hot user with N API keys serializes every
admin request behind N Argon2 verifies. A noisy authenticated
tenant can stall the admin plane.
### L-4 — Pubsub `mint_subscriber_token` has no per-app rate limit
- **Severity:** Low
- **Location:** `crates/manager-core/src/pubsub_service.rs:240-…`.
- **Summary:** Token minting requires authz, but an authenticated
caller can mint unbounded tokens — each is a tiny HMAC, so no
storage DoS, but it makes the topic-secret rotation surface
noisier. Worth a sliding-window cap.
### L-5 — `outbox.insert` failures are not visible to the script caller in a non-200 way for fan-out paths
- **Severity:** Low (correctness-adjacent)
- **Location:** `outbox_event_emitter.rs:101-103` etc.
- **Summary:** If an `INSERT INTO outbox` fails (Postgres full disk,
deadlock), the SDK call returns the error to the script. Good. But
the script is free to retry-in-a-loop — there's no script-side
circuit breaker for "the outbox is unavailable". A script could
busy-loop hitting outbox-insert failures indefinitely.
- **Fix shape:** none beyond M-5's circuit breaker.
### L-6 — `inbox_router` / future SMTP ingress will need its own caps
- **Severity:** Low / Info
- **Location:** not yet implemented.
- **Summary:** Heads-up for v1.2 — SMTP ingress and queue triggers
with external receivers will need the same connection-cap + IP rate
limit pattern as H-3.
---
## Info
### I-1 — `pubsub::publish_durable` realtime broadcast does `tokio::spawn` + `handle.await`
- **Location:** `crates/manager-core/src/pubsub_service.rs:222-236`.
- **Note:** the `tokio::spawn(...)` immediately followed by
`handle.await` defeats the comment ("Run on a child task so a
panicking broadcaster becomes a warn log") — a panic in the spawned
task surfaces as a `JoinError` in the awaiting frame and the publish
succeeds (because the durable fan-out already committed). The catch
is correct, but the comment is slightly misleading. Not a DoS finding.
### I-2 — `DEFAULT_TICK_INTERVAL = 100 ms` is reasonable for Pi
- **Location:** `crates/manager-core/src/dispatcher.rs:101`.
- **Note:** 100ms tick × 8 rows/tick = 80 dispatches/sec ceiling per
process, before the gate. Adequate for a single-node MVP. The
env-override (`PICLOUD_DISPATCHER_TICK_INTERVAL_MS`) lets operators
trade latency for CPU.
### I-3 — Caddy in dev runs at `:80` with `auto_https off`
- **Location:** `caddy/Caddyfile:24-26`.
- **Note:** Prod Caddyfile enables auto-HTTPS via ACME. No HTTPS for
the dev port is fine for development; mentioned only because every
rate-limit story benefits from terminating TLS at Caddy.
---
## Recommendations, prioritized
1. **Login rate limit + global Argon2 semaphore** (H-2). Highest
leverage: closes the cheapest-CPU-DoS in the system.
2. **Require HMAC on email-inbound + per-IP bucket** (H-3). One of two
anonymous-reachable endpoints; the other is the script data plane,
which already has a 503-gate.
3. **Per-app caps on triggers / routes / cron, plus cron tick LIMIT**
(H-4 + H-5). Closes the cross-tenant amplification path. Quotas
land naturally in the v1.2 work the blueprint already plans.
4. **Explicit `DefaultBodyLimit` at the router root** (H-1). One-line
fix; deep value.
5. **Caddy `request_body { max_size … }` and `servers { timeouts …}`**
in both Caddyfiles. (Closes H-1 / M-3 at the proxy.)
6. **Outbox GC + per-app KV/docs row caps** (M-1 / M-4). Mechanically
easy; defends the long tail.
## Methodology
Read AUDIT.md to map prior findings and remediation state, then read:
`crates/orchestrator-core/src/gate.rs`, `client.rs`, `api.rs`,
`realtime.rs`, `realtime_api.rs`; `crates/manager-core/src/dispatcher.rs`,
`cron_scheduler.rs`, `auth_api.rs`, `email_inbound_api.rs`,
`pubsub_service.rs`, `pubsub_repo.rs`, `queue_service.rs`,
`kv_service.rs`, `docs_service.rs`, `files_service.rs`, `http_service.rs`,
`outbox_event_emitter.rs`, `outbox_repo.rs`, `gc.rs`,
`triggers_api.rs`, `route_admin.rs`, `trigger_config.rs`,
`users_service.rs` (rate-limiter shape); `crates/picloud/src/main.rs`,
`lib.rs`; both Caddyfiles. Grepped for `DefaultBodyLimit`,
`RequestBodyLimit`, `rate_limit`, `Semaphore`, `tracing_appender`,
`set_keepalive`, `count_for_app`, `MAX_TRIGGERS`, `MAX_ROUTES`.