fix(daemons): race in-flight work against cancellation on shutdown (0.87.4)
Two high findings sharing one root cause: long-running daemon work that
ignored the shutdown signal until it finished naturally.
**Cron tick (crawler).** `CronContext::run_tick` ran the metadata-pass
body inside `AssertUnwindSafe(...).catch_unwind().await` with no
cancellation. A fresh-catalog walk (many minutes) wedged
`DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under the
supervisor lock), and SIGTERM responsiveness for the whole pass.
Wrap the body in `tokio::select! { biased; _ = self.cancel.cancelled() => ...; r = body => ... }`.
On cancel the body future drops, which cascades to dropping
`metadata.run()` — exactly what gives Chromium/lease teardown a chance
via the BrowserManager idle reaper. The advisory unlock + conn drop
still run unconditionally.
**Analysis dispatch.** `process_lease` wrapped the vision call only in
`tokio::time::timeout(self.job_timeout, …)` — default 600s. Toggling
analysis off in the dashboard (under the supervisor lock) or stopping
the container blocked on whichever call was in flight. Same fix:
`tokio::select!` against `self.cancel.cancelled()`. On cancel,
`jobs::release` returns the lease to the queue without burning an
attempt — next tick picks it up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -280,7 +280,8 @@ impl CronContext {
|
||||
// no future tick would ever run — workers would keep going but
|
||||
// no new metadata work would be scheduled until daemon restart.
|
||||
// The advisory unlock below runs unconditionally so a panicked
|
||||
// tick doesn't leave the lock held for another replica.
|
||||
// (or cancelled) tick doesn't leave the lock held for another
|
||||
// replica.
|
||||
let metadata = &self.metadata;
|
||||
let pool = &self.pool;
|
||||
let retention_days = self.retention_days;
|
||||
@@ -312,8 +313,23 @@ impl CronContext {
|
||||
tracing::warn!(?e, "cron: persist last_metadata_tick_at failed");
|
||||
}
|
||||
};
|
||||
if let Err(_panic) = AssertUnwindSafe(body).catch_unwind().await {
|
||||
tracing::error!("cron: tick body panicked — continuing");
|
||||
// Race the tick body against shutdown. Without this, a long
|
||||
// metadata pass on a fresh catalog (minutes) wedges
|
||||
// `DaemonHandle::shutdown`, `Supervisors::reload_crawler` (under
|
||||
// the supervisor lock), and SIGTERM responsiveness for the
|
||||
// entire pass. On cancel, dropping `body` cascades to dropping
|
||||
// `metadata.run()`, which is what gives Chromium/lease cleanup
|
||||
// their chance via the Browse Manager's idle reaper.
|
||||
tokio::select! {
|
||||
biased;
|
||||
_ = self.cancel.cancelled() => {
|
||||
tracing::info!("cron: cancelled mid-tick — abandoning to release advisory lock");
|
||||
}
|
||||
r = AssertUnwindSafe(body).catch_unwind() => {
|
||||
if r.is_err() {
|
||||
tracing::error!("cron: tick body panicked — continuing");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
|
||||
@@ -662,6 +678,35 @@ pub mod test_support {
|
||||
}
|
||||
}
|
||||
|
||||
/// `MetadataPass` that sleeps for `delay` before returning. Lets a
|
||||
/// test trigger shutdown WHILE a tick body is in flight, exercising
|
||||
/// the cancellation race in `CronContext::run_tick`.
|
||||
pub struct SlowMetadataPass {
|
||||
pub delay: std::time::Duration,
|
||||
pub started: AtomicUsize,
|
||||
}
|
||||
|
||||
impl SlowMetadataPass {
|
||||
pub fn new(delay: std::time::Duration) -> Arc<Self> {
|
||||
Arc::new(Self {
|
||||
delay,
|
||||
started: AtomicUsize::new(0),
|
||||
})
|
||||
}
|
||||
pub fn start_count(&self) -> usize {
|
||||
self.started.load(Ordering::Acquire)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl MetadataPass for SlowMetadataPass {
|
||||
async fn run(&self) -> anyhow::Result<pipeline::MetadataStats> {
|
||||
self.started.fetch_add(1, Ordering::AcqRel);
|
||||
tokio::time::sleep(self.delay).await;
|
||||
Ok(pipeline::MetadataStats::default())
|
||||
}
|
||||
}
|
||||
|
||||
pub type DispatchFn = Arc<
|
||||
dyn Fn(JobPayload) -> futures_util::future::BoxFuture<'static, anyhow::Result<SyncOutcome>>
|
||||
+ Send
|
||||
|
||||
Reference in New Issue
Block a user