fix(daemons): race in-flight work against cancellation on shutdown (0.87.4)

Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.

**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.

Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.

**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-06-22 21:18:18 +02:00
parent ee9f5c1a4d
commit 660184a048
7 changed files with 232 additions and 10 deletions

View File

@@ -255,9 +255,30 @@ impl WorkerContext {
let started = std::time::Instant::now();
let dispatch = AssertUnwindSafe(self.dispatcher.dispatch(page_id)).catch_unwind();
let outcome = tokio::time::timeout(self.job_timeout, dispatch).await;
// Race the vision call against shutdown. Without this, shutdown
// (SIGTERM / `Supervisors::reload_analysis` toggling analysis off)
// blocks for up to `job_timeout` (default 600s) on whichever call
// happens to be in flight. On cancel we drop the dispatch future
// and `jobs::release` the lease — next tick picks it up without
// burning an attempt.
let cancel_result = tokio::select! {
biased;
_ = self.cancel.cancelled() => None,
o = tokio::time::timeout(self.job_timeout, dispatch) => Some(o),
};
let elapsed_ms = started.elapsed().as_millis() as i64;
heartbeat.abort();
let Some(outcome) = cancel_result else {
tracing::info!(
worker = self.id,
lease_id = %lease.id,
"analysis worker: cancelled mid-dispatch — releasing lease"
);
let _ = jobs::release(&self.pool, lease.id).await;
// Don't publish Failed (the page didn't actually fail) and
// don't write a duration row — neither outcome is true.
return;
};
// Flatten timeout / panic / dispatch error into one Result + message.
let result: Result<(), String> = match outcome {
@@ -413,6 +434,32 @@ pub mod test_support {
}
}
/// A dispatcher that sleeps for `delay` before returning `Ok`. Lets a
/// test fire shutdown WHILE a dispatch is in flight, to exercise the
/// cancellation race in `process_lease`.
pub struct SlowDispatcher {
pub calls: AtomicUsize,
pub delay: Duration,
}
impl SlowDispatcher {
pub fn new(delay: Duration) -> Arc<Self> {
Arc::new(Self { calls: AtomicUsize::new(0), delay })
}
pub fn call_count(&self) -> usize {
self.calls.load(Ordering::Acquire)
}
}
#[async_trait]
impl AnalyzeDispatcher for SlowDispatcher {
async fn dispatch(&self, _page_id: Uuid) -> anyhow::Result<()> {
self.calls.fetch_add(1, Ordering::AcqRel);
tokio::time::sleep(self.delay).await;
Ok(())
}
}
#[async_trait]
impl AnalyzeDispatcher for CountingDispatcher {
async fn dispatch(&self, _page_id: Uuid) -> anyhow::Result<()> {