Files
PiCloud/security_audit/08_dos_resource.md
MechaCat02 ec4a2aa24a style: cargo fmt across audit-2026-06-11 tier-3 changes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-12 18:38:28 +02:00

455 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Security audit — DoS, rate limiting & resource exhaustion
**Audit date:** 2026-06-11
**Scope:** PiCloud at `main` (post-v1.1.9, pre-v1.2). Single-node MVP.
**Reviewer:** Subagent 08 of 10 (parallel security audit).
**Threat model:** Solo-dev / Pi-class hardware. A single anonymous attacker
or one noisy authenticated tenant should not OOM the host, fill the disk,
or starve the executor for other tenants in minutes.
## Counts by severity
| Severity | Count |
|---|---|
| Critical | 0 |
| High | 5 |
| Medium | 7 |
| Low | 6 |
| Info | 3 |
## Verified / no-finding
These were investigated and judged adequate for the threat model:
- **Concurrent script execution cap.** `ExecutionGate` uses a single
`tokio::sync::Semaphore` with `try_acquire_owned()` and translates
`NoPermits`/`Closed` into `AcquireError::Overloaded { retry_after_secs: 1 }`,
which `IntoResponse` for `ApiError` maps to HTTP 503 with `Retry-After: 1s`
(`crates/orchestrator-core/src/api.rs:712-721`). Default 32, env-tunable
via `PICLOUD_MAX_CONCURRENT_EXECUTIONS`. Both sync HTTP and the dispatcher
share the same gate so the dispatcher arms back off correctly when sync
HTTP is saturated. **Note on fairness:** `try_acquire` is non-blocking so
"fairness" is irrelevant — there is no queue, callers get 503 immediately.
- **Postgres pool default size.** `init_db` in
`crates/picloud/src/lib.rs:603-619` now defaults to
`DEFAULT_DB_MAX_CONNECTIONS = 32` and is env-tunable via
`PICLOUD_DB_MAX_CONNECTIONS`, matching the gate. (F-P-003 from the prior
audit was remediated.) Acquire timeout 5s.
- **Per-message SDK service caps.** `kv_service`, `docs_service`,
`pubsub_service`, `queue_service`, `files_service`, `secrets_service`
all enforce a per-message JSON-encoded byte cap (256 KB default, except
files at 100 MB) **before** authz. `kv_service.rs:141`, `docs_service.rs:93`,
`pubsub_service.rs:178`, `queue_service.rs:82`. (F-S-001 from the prior
audit was remediated.)
- **Outbound HTTP defaults (SDK http::request).** 30s default timeout,
60s hard ceiling; 10s connect timeout; 10 MB request/response body cap;
5 redirect default, 10 ceiling. SSRF deny-list on by default. See
`crates/manager-core/src/http_service.rs:41-53`.
- **Email outbound rate limit.** `EmailRateLimiter` (token bucket +
per-(app, recipient) burst + 200/day per-app cap) gates `users::*` email
surfaces (`crates/manager-core/src/users_service.rs:139-187`). F-S-002
remediated.
- **Outbox claim shape.** Dispatcher claims at most `CLAIM_BATCH = 8`
rows per tick (100 ms default) with `FOR UPDATE SKIP LOCKED`, gates
every dispatch through the same `ExecutionGate`, and reschedules on
permit refusal — bounded by design. See
`crates/manager-core/src/dispatcher.rs:69,193-216,482-491`.
- **Outbox row deletion.** Outbox rows are deleted on success/dead-letter
inline (`OutboxRepo::delete`), so the table doesn't accumulate
completed rows under normal load. Bloat risk surveyed below as a
Medium (stuck rows).
---
## High
### H-1 — No Axum default body limit; the only inbound 503-path is the
data-plane user route — all admin/email/webhook handlers fall under Axum's
default 2 MB and the `/api/v1/execute/{id}` handler reads `body: Bytes`
with no explicit override.
- **Severity:** High
- **Location:**
`crates/orchestrator-core/src/api.rs:106-123` (execute_by_id, `body: Bytes`,
no `DefaultBodyLimit`);
`crates/manager-core/src/email_inbound_api.rs:131-136` (inbound webhook,
`body: Bytes`);
`crates/picloud/src/lib.rs:587-593` (router composition — no
`RequestBodyLimitLayer`); admin JSON handlers (`Json<…>` extractors at
`auth_api.rs:74`, `admin_users_api.rs`, `users_admin_api.rs`, etc.).
- **Summary:** Axum's `Bytes`/`Json` extractors default to 2 MB. `Caddy`
has no `request_body { max_size … }` directive in either `Caddyfile`
or `Caddyfile.prod` (just unrestricted `reverse_proxy`), so any
request up to **10 GB** (Caddy's hard ceiling) can hit picloud. Only
the data-plane user-route handler (`user_route_handler` in
`api.rs:219`) explicitly bounds the body at 10 MiB; everything else
rides Axum's default.
- For `/email-inbound/{app_id}/{trigger_id}` the body is `Bytes`
with no override — so the 2 MB ceiling is the only thing standing
between an anonymous attacker and a 2 MB JSON-parse fork-bomb per
request. With no IP rate limit, one client can sustain hundreds of
MB/s of CPU-bound `serde_json::from_slice` and HMAC verification
work, saturating one execution worker indefinitely. This is the
canonical "no rate limit on a public unauthenticated endpoint"
finding — see also H-3.
- For `POST /api/v1/execute/{id}` Axum's 2 MB default applies, but
the request lands at the executor regardless of script size,
meaning a script with `body_limit_kb = 32` still pays the
serialization cost for a 2 MB body.
- **Fix shape:**
- Add a global `axum::extract::DefaultBodyLimit::max(2 * 1024 * 1024)`
layer at the router root (already implicit, but should be explicit
+ lower the default to ~1 MB).
- Add per-route higher limits where intentional (email inbound is
typically tens of KB; lower it to 1 MB explicitly).
- Caddy: `request_body { max_size 5MB }` in both Caddyfiles.
### H-2 — Login endpoint has no rate limit; Argon2id is the work multiplier
- **Severity:** High
- **Location:** `crates/manager-core/src/auth_api.rs:74-167`.
- **Summary:** `POST /api/v1/admin/auth/login` is reachable
unauthenticated, and on every request it does a username lookup +
Argon2id verify (m=19456 KiB, t=2 default Argon2id params) via
`spawn_blocking`. There is no per-IP, per-username, or process-wide
cap on requests / failed attempts. Argon2 is ~50150 ms per attempt
on Pi-class hardware; an anonymous attacker with 32 in-flight HTTP
requests can keep every blocking-thread worker pinned on Argon2
forever, **and** can attempt ≈600 password guesses/sec process-wide.
Username enumeration is correctly defended (dummy-hash path) so the
same Argon2 cost is paid on bad usernames — which means the cost
inflicts on the defender for every miss.
- **Threat:** Combined with H-1 (no body limit on `Json<LoginRequest>`)
this is the cheapest CPU-DoS in the system. A laptop attacker can
fully saturate a Raspberry Pi's executor in seconds.
- **Fix shape:** add a sliding-window login rate limiter keyed on
`(client_ip, username)` (e.g. 5 attempts / 60s, lockout for 5 min
after 20), and a global `Semaphore` cap (e.g. 4 concurrent in-flight
Argon2 verifies) so admin auth can't starve the executor's
`spawn_blocking` pool. Memory only — Postgres-backed counters add a
round-trip. The existing `EmailRateLimiter` pattern in
`users_service.rs:139` is the shape to copy.
### H-3 — Email-inbound webhook has no IP rate limit; HMAC is optional
- **Severity:** High
- **Location:** `crates/manager-core/src/email_inbound_api.rs:131-191`,
mounted **outside** `require_authenticated` at
`crates/picloud/src/lib.rs:571-575`.
- **Summary:** `POST /api/v1/email-inbound/{app_id}/{trigger_id}` is
public. When the trigger has an `inbound_secret` configured, HMAC
verification is correct (timestamp-bound, ±5 min, in-memory replay
dedup). But:
1. If the operator did NOT configure an `inbound_secret` for the
trigger, **no signature is required at all** (see the
`if let (Some(ct), Some(nonce)) = …` guard at line 150). An
anonymous attacker who knows the trigger UUID can enqueue an
unbounded number of outbox rows + executor invocations per
second, bypassing every other rate limit.
2. With a secret configured, attackers still pay no cost for sending
garbage — the receiver must decrypt the secret from the secret
store (one DB call + master-key open) and HMAC-verify per request
**before** rejecting. No `(app_id, trigger_id)` invalid-signature
bucket. So an attacker who learns the URL but not the secret can
still pin a worker thread on secret-decrypt + HMAC.
3. The body is `Bytes` without a `DefaultBodyLimit` override — falls
under Axum's 2 MB cap (see H-1).
- **Threat:** A single anonymous HTTP client knowing only an `app_id`
+ a `trigger_id` (often leaked through public docs / TLS cert
transparency logs / mis-configured provider dashboards) can enqueue
enough outbox rows to fill the dispatcher batch every tick. With
retries, one row → up to `retry_max_attempts` (default 3) executions.
Outbox grows unbounded; disk fills.
- **Fix shape:** require HMAC at the trigger-row level (DB CHECK that
email triggers MUST have a secret); add per-`(app_id, trigger_id)`
token-bucket gating both `inbound_secret_missing` (drop fast) and
`bad_signature` paths; add a `DefaultBodyLimit::max(1 MB)` to the
router merge in `lib.rs`.
### H-4 — No per-app cap on trigger / route / cron registration
- **Severity:** High
- **Location:**
`crates/manager-core/src/triggers_api.rs:127-140` (create endpoints),
`crates/manager-core/src/route_admin.rs:160` (create_route);
no counterpart `MAX_TRIGGERS_PER_APP` / `MAX_ROUTES_PER_APP` constant
in the codebase.
- **Summary:** An authenticated app developer can register an
unbounded number of triggers (KV, docs, files, pubsub, cron, email,
queue, dead-letter) and routes. Two amplifications follow:
1. **Cron storm.** `cron_scheduler::tick` SELECTs **all** enabled
cron rows in one transaction (`fetch_all`, no `LIMIT`), iterates
in Rust, and inserts one outbox row per due trigger
(`cron_scheduler.rs:117-167`). 10 000 cron triggers all set to
`*/1 * * * * *` (every second) → 10 000 outbox rows per second,
each becoming a 100ms dispatcher claim of 8 rows + 32 concurrent
executions. The dispatcher backs up; outbox bloats; Postgres
`next_attempt_at` index degrades.
2. **Pubsub fan-out.** `pubsub_repo::fan_out_publish` SELECTs all
enabled `pubsub` triggers for the app, filters in Rust on
`topic_matches`, inserts one outbox row per match in a single
transaction (`pubsub_repo.rs:81-115`). N triggers subscribed to
the same topic → N writes per publish.
3. **KV / docs / files fan-out.** Same pattern in
`outbox_event_emitter.rs:88-103` — one outbox row per matching
trigger per mutation.
- **Threat:** A single tenant with `AppTriggerCreate` capability can
register thousands of triggers and bring the whole node down by
pubsub-publishing one message; or fill disk by cron-firing.
Cross-tenant impact via shared dispatcher.
- **Fix shape:** per-app trigger / route caps (e.g. 100 each, env-
tunable) enforced in the *_api create handlers; per-publish fan-out
cap (e.g. 100 matched triggers per publish, then dead-letter the
excess with a clear error); cron-scheduler tick LIMIT clause
(`LIMIT 1000`) so a misbehaving app can't ride the whole tick budget.
### H-5 — Cron-scheduler tick is unbounded `fetch_all`; one slow row blocks all
- **Severity:** High
- **Location:** `crates/manager-core/src/cron_scheduler.rs:115-173`.
- **Summary:** Every cron tick (default 30s) opens a transaction,
SELECTs every enabled cron row with `FOR UPDATE OF d SKIP LOCKED`,
computes `next_due` in Rust, and INSERTs an outbox row + UPDATEs
`last_fired_at` per due row — **all serially inside one transaction**.
Three concrete failure modes:
1. **Long-running tx.** With N triggers and the per-row cost ~1ms,
a few thousand cron rows make the tick exceed the tick interval,
pile up overlapping transactions, and hold row-locks on
`cron_trigger_details` for seconds — blocking dashboard reads
of the same table.
2. **No LIMIT.** A malicious or buggy tenant registering 100 000
cron triggers turns every tick into a multi-second scan +
INSERT-bomb.
3. **All-or-nothing commit.** One serialization failure (very
possible on Pg under load) rolls back every fire in the tick.
- **Fix shape:** add `LIMIT 1000` to the cron `fetch_all`; break the
per-tick work into smaller transactions; require a positive
`tick_interval_ms` floor *above* 1s (already floored, but the
per-row work is what's unbounded, not the tick).
---
## Medium
### M-1 — Outbox has no time-based retention sweep
- **Severity:** Medium
- **Location:** `crates/manager-core/src/outbox_repo.rs` (no
`prune_expired` / `delete_older_than`);
`crates/manager-core/src/gc.rs` (sweeps dead_letters + abandoned +
app-user tokens, but NOT outbox).
- **Summary:** Outbox rows are deleted inline on success or when
exceeded; but stuck rows (`claimed_at` set but never released after
a process crash mid-dispatch, or rows whose `trigger_depth >
max_trigger_depth` slip past) accumulate. With no GC sweep, a
bug-induced leak silently bloats the outbox table. On Pi-class
Postgres this becomes apparent after weeks, not hours, so Medium.
- **Fix shape:** `spawn_outbox_gc` mirroring the dead_letter pattern,
pruning rows with `claimed_at < NOW() - INTERVAL '1 hour'`
(orphaned claims) and rows past a 30-day retention.
### M-2 — No SSE subscriber cap per IP / app
- **Severity:** Medium
- **Location:** `crates/orchestrator-core/src/realtime_api.rs:85,
94-159`.
- **Summary:** `GET /realtime/topics/{topic}` is gated by topic-token
auth, but once authorized the subscriber stays connected indefinitely
(heartbeat keepalive). No process-wide / per-IP / per-app cap on
active subscriptions. A leaked token plus N connections from one
client lets the attacker pin N hyper tasks + N `broadcast::Receiver`
slots, each holding a `tokio::sync::broadcast` channel reference.
Per topic the broadcast channel buffers `capacity` items; with many
topics that's many MB of resident state.
- **Fix shape:** per-IP cap (e.g. 4 active SSE connections per remote
address), process-wide cap (e.g. 1024), and per-app cap
(e.g. 64). 503 on overflow.
### M-3 — No global cap on in-flight HTTP connections / slowloris exposure
- **Severity:** Medium
- **Location:** `crates/picloud/src/main.rs:79-83`
(`TcpListener::bind` then `axum::serve` with default config).
- **Summary:** No `set_keepalive`, no `set_nodelay`, no read/write
header timeout, no concurrent-connection cap. Hyper defaults are
permissive — slowloris-style attackers can hold thousands of
half-open TLS handshakes and read-byte-per-second connections.
Caddy fronts in prod and mitigates somewhat, but the dev/single-
node deployment has no defense.
- **Fix shape:** wrap the `TcpListener` with a per-connection
`tower::limit::ConcurrencyLimit` (e.g. 256 concurrent connections);
set `tcp_keepalive` + `http2_keep_alive_interval` on the hyper
builder; set `header_read_timeout = 10s`. Also document a Caddy
`servers { timeouts { read_header 10s read 30s write 30s } }` block
in both Caddyfiles.
### M-4 — No per-app KV / docs / files row-count or byte quota
- **Severity:** Medium
- **Location:** repo-side; no `app_quotas` table or check in any
`*_repo::insert`.
- **Summary:** Per-row caps exist (256 KB / 100 MB) but per-app
totals are unbounded. One tenant can store millions of KV rows or
fill `PICLOUD_FILES_ROOT` to disk-full. Defense in depth: the
CLAUDE.md note ("per-app quotas deferred to v1.2") explicitly
acknowledges this; downgrading the severity reflects "documented
v1.2 work" but it is still a real DoS today on a shared instance.
- **Fix shape:** `app_quotas` table with `bytes_used` columns per
service, updated transactionally on insert/update, checked at
service-entry. Or a soft cap (warn + log) for v1.1.x; hard quota
for v1.2.
### M-5 — Dispatcher retry storm: HTTP-async retries up to `retry_max_attempts` with no per-target circuit breaker
- **Severity:** Medium
- **Location:** `crates/manager-core/src/dispatcher.rs:885-905`,
`trigger_config.rs:83`.
- **Summary:** Async HTTP outbox rows retry per `TriggerConfig`
default `retry_max_attempts = 3` with exponential backoff. No
circuit breaker per `(app_id, script_id)` or per outbound host —
a script that always fails fires 3 invocations per event for as
long as events arrive. With pubsub fan-out from H-4, one
publish → N triggers × 3 retries = `3N` executor invocations.
- **Fix shape:** per-`(app_id, script_id)` failure counter with a
configurable circuit-break threshold (e.g. 50% failures over 60s →
pause for 5 min, write a `dead_letters` row).
### M-6 — Queue depth is unbounded; per-app queue count is unbounded
- **Severity:** Medium
- **Location:** `crates/manager-core/src/queue_service.rs:67-118`
(enqueue), `crates/manager-core/src/queue_repo.rs:264-322`
(depth uses `COUNT(*)`).
- **Summary:** `queue::enqueue` has a per-message 256 KB cap but no
per-queue depth cap. A loop in a public-HTTP script can fill a
queue with millions of messages × 256 KB = GB of JSONB; combined
with `queue::depth` being a full scan (F-P-006, still open per
AUDIT.md remediation state), every depth call magnifies the
attack.
- **Fix shape:** `PICLOUD_QUEUE_MAX_DEPTH_PER_QUEUE` (e.g. 10 000) +
`PICLOUD_QUEUE_MAX_QUEUES_PER_APP` (e.g. 100).
### M-7 — Trigger fan-out from KV/docs/files mutations writes outbox row per matching trigger inline (sync), with no per-event cap
- **Severity:** Medium
- **Location:**
`crates/manager-core/src/outbox_event_emitter.rs:88-103,
142-160, 200-…` (similar pattern).
- **Summary:** Each `kv::set` matching N triggers inserts N outbox
rows inside the calling request's transaction. No per-event cap
on N. Combined with H-4 (unbounded trigger registration), a
single `kv::set` can stall a request for seconds while writing
thousands of rows. Pubsub fan-out (`pubsub_repo.rs:81-117`) is
the same shape.
- **Fix shape:** cap `matches.len()` at e.g. 100; reject with a
`PolicyTooManyMatches` if exceeded so the developer notices.
---
## Low
### L-1 — Login attempts are not logged for forensics
- **Severity:** Low
- **Location:** `crates/manager-core/src/auth_api.rs:74-167`.
- **Summary:** Failed login attempts are swallowed by the
`invalid_credentials()` return path with no structured log line, no
counter, no `auth_audit` row. After H-2 is fixed, the rate-limit
decisions will need observability hooks; preempt that.
- **Fix shape:** `tracing::warn!(client_ip, username, "admin login
failed")` (with username **before** the lookup so it can't leak
whether the user exists via what's logged).
### L-2 — `tracing::info!` JSON output is unlimited and unrotated
- **Severity:** Low
- **Location:** `crates/picloud/src/main.rs:222-227`.
- **Summary:** `init_tracing` writes to stdout in JSON. Operators
running outside docker (where journald handles rotation) won't have
automatic rotation; logs eat the disk over weeks. No `tracing_appender`
rolling file appender wired in.
- **Fix shape:** document the journald / `docker logs --max-size`
expectation, or wire `tracing_appender::rolling::daily` behind an env
var so the no-supervisor case is also safe.
### L-3 — `attach_principal_if_present` runs Argon2 verify on every API-key prefix candidate on every authed request
- **Severity:** Low (already flagged at High in F-P-002; downgraded
here because pretty much an authed-user-only problem).
- **Location:** `crates/manager-core/src/auth_middleware.rs:192-229`.
- **Summary:** AUDIT.md flagged this; including here because the
DoS angle is real: a hot user with N API keys serializes every
admin request behind N Argon2 verifies. A noisy authenticated
tenant can stall the admin plane.
### L-4 — Pubsub `mint_subscriber_token` has no per-app rate limit
- **Severity:** Low
- **Location:** `crates/manager-core/src/pubsub_service.rs:240-…`.
- **Summary:** Token minting requires authz, but an authenticated
caller can mint unbounded tokens — each is a tiny HMAC, so no
storage DoS, but it makes the topic-secret rotation surface
noisier. Worth a sliding-window cap.
### L-5 — `outbox.insert` failures are not visible to the script caller in a non-200 way for fan-out paths
- **Severity:** Low (correctness-adjacent)
- **Location:** `outbox_event_emitter.rs:101-103` etc.
- **Summary:** If an `INSERT INTO outbox` fails (Postgres full disk,
deadlock), the SDK call returns the error to the script. Good. But
the script is free to retry-in-a-loop — there's no script-side
circuit breaker for "the outbox is unavailable". A script could
busy-loop hitting outbox-insert failures indefinitely.
- **Fix shape:** none beyond M-5's circuit breaker.
### L-6 — `inbox_router` / future SMTP ingress will need its own caps
- **Severity:** Low / Info
- **Location:** not yet implemented.
- **Summary:** Heads-up for v1.2 — SMTP ingress and queue triggers
with external receivers will need the same connection-cap + IP rate
limit pattern as H-3.
---
## Info
### I-1 — `pubsub::publish_durable` realtime broadcast does `tokio::spawn` + `handle.await`
- **Location:** `crates/manager-core/src/pubsub_service.rs:222-236`.
- **Note:** the `tokio::spawn(...)` immediately followed by
`handle.await` defeats the comment ("Run on a child task so a
panicking broadcaster becomes a warn log") — a panic in the spawned
task surfaces as a `JoinError` in the awaiting frame and the publish
succeeds (because the durable fan-out already committed). The catch
is correct, but the comment is slightly misleading. Not a DoS finding.
### I-2 — `DEFAULT_TICK_INTERVAL = 100 ms` is reasonable for Pi
- **Location:** `crates/manager-core/src/dispatcher.rs:101`.
- **Note:** 100ms tick × 8 rows/tick = 80 dispatches/sec ceiling per
process, before the gate. Adequate for a single-node MVP. The
env-override (`PICLOUD_DISPATCHER_TICK_INTERVAL_MS`) lets operators
trade latency for CPU.
### I-3 — Caddy in dev runs at `:80` with `auto_https off`
- **Location:** `caddy/Caddyfile:24-26`.
- **Note:** Prod Caddyfile enables auto-HTTPS via ACME. No HTTPS for
the dev port is fine for development; mentioned only because every
rate-limit story benefits from terminating TLS at Caddy.
---
## Recommendations, prioritized
1. **Login rate limit + global Argon2 semaphore** (H-2). Highest
leverage: closes the cheapest-CPU-DoS in the system.
2. **Require HMAC on email-inbound + per-IP bucket** (H-3). One of two
anonymous-reachable endpoints; the other is the script data plane,
which already has a 503-gate.
3. **Per-app caps on triggers / routes / cron, plus cron tick LIMIT**
(H-4 + H-5). Closes the cross-tenant amplification path. Quotas
land naturally in the v1.2 work the blueprint already plans.
4. **Explicit `DefaultBodyLimit` at the router root** (H-1). One-line
fix; deep value.
5. **Caddy `request_body { max_size … }` and `servers { timeouts …}`**
in both Caddyfiles. (Closes H-1 / M-3 at the proxy.)
6. **Outbox GC + per-app KV/docs row caps** (M-1 / M-4). Mechanically
easy; defends the long tail.
## Methodology
Read AUDIT.md to map prior findings and remediation state, then read:
`crates/orchestrator-core/src/gate.rs`, `client.rs`, `api.rs`,
`realtime.rs`, `realtime_api.rs`; `crates/manager-core/src/dispatcher.rs`,
`cron_scheduler.rs`, `auth_api.rs`, `email_inbound_api.rs`,
`pubsub_service.rs`, `pubsub_repo.rs`, `queue_service.rs`,
`kv_service.rs`, `docs_service.rs`, `files_service.rs`, `http_service.rs`,
`outbox_event_emitter.rs`, `outbox_repo.rs`, `gc.rs`,
`triggers_api.rs`, `route_admin.rs`, `trigger_config.rs`,
`users_service.rs` (rate-limiter shape); `crates/picloud/src/main.rs`,
`lib.rs`; both Caddyfiles. Grepped for `DefaultBodyLimit`,
`RequestBodyLimit`, `rate_limit`, `Semaphore`, `tracing_appender`,
`set_keepalive`, `count_for_app`, `MAX_TRIGGERS`, `MAX_ROUTES`.