feat(orchestrator-core): ExecutionGate + 503/Retry-After on overflow

Adds a single global concurrency cap on the data-plane dispatch path:

  - orchestrator-core::gate::ExecutionGate wraps tokio::Semaphore.
    Non-blocking try_acquire — no queue. PICLOUD_MAX_CONCURRENT_EXECUTIONS
    env var (default 32) sets the cap.
  - LocalExecutorClient acquires a permit before spawn_blocking; the
    permit drops with the future so the slot returns automatically.
  - On refusal, ExecError::Overloaded { retry_after_secs: 1 } surfaces
    upward. ApiError::IntoResponse already maps that to 503 with a
    Retry-After header (landed in the previous commit alongside the
    variant itself).
  - picloud binary constructs the gate once at build_app and shares it
    with LocalExecutorClient.

The cap exists so a Rhai script storm can't drain the blocking-thread
pool — pushing back hard beats letting requests pile up against a
finite worker count. Per-app / per-script caps stay deferred until a
real workload demands them.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-05-30 18:50:44 +02:00
parent fe1dd90836
commit dea776b2a3
4 changed files with 182 additions and 4 deletions

View File

@@ -22,7 +22,7 @@ use picloud_manager_core::{
};
use picloud_orchestrator_core::routing::{AppDomainTable, RouteTable};
use picloud_orchestrator_core::{
data_plane_router, user_routes_router, DataPlaneState, LocalExecutorClient,
data_plane_router, user_routes_router, DataPlaneState, ExecutionGate, LocalExecutorClient,
};
use picloud_shared::{
ExecutionLogSink, ScriptValidator, Services, API_VERSION, PRODUCT_VERSION, SDK_VERSION,
@@ -129,7 +129,10 @@ pub async fn build_app(pool: PgPool, auth: AuthDeps) -> anyhow::Result<Router> {
let resolver = Arc::new(RepoResolver::new(PostgresScriptRepoHandle(
script_repo.clone(),
)));
let executor = Arc::new(LocalExecutorClient::new(engine.clone()));
// Single global gate — overflow is rejected with 503 + Retry-After.
// See `ExecutionGate` docs and `PICLOUD_MAX_CONCURRENT_EXECUTIONS`.
let gate = Arc::new(ExecutionGate::from_env());
let executor = Arc::new(LocalExecutorClient::new(engine.clone(), gate));
let admin = AdminState {
repo: Arc::new(PostgresScriptRepoHandle(script_repo.clone())),