[WIP] Audio/threading fixes + crash investigation; NEW ORACLE: crash is ours not the game
Some checks failed
Orchestrator / Commit Message Validation (push) Has been skipped
Orchestrator / Lint (push) Failing after 1m34s
Orchestrator / Windows (x86-64) (push) Has been skipped
Orchestrator / Linux (x86-64) (push) Has been skipped
Orchestrator / Create Release (push) Has been skipped
Some checks failed
Orchestrator / Commit Message Validation (push) Has been skipped
Orchestrator / Lint (push) Failing after 1m34s
Orchestrator / Windows (x86-64) (push) Has been skipped
Orchestrator / Linux (x86-64) (push) Has been skipped
Orchestrator / Create Release (push) Has been skipped
Snapshot for handoff. Contains the mission-audio + threading fixes and the
crash-investigation instrumentation (all diagnostic cvars default-OFF).
Fixes (behavioral):
- threading_posix.cc: reap-once guard on PosixCondition<Thread>::post_execution
(double pthread_join at mission teardown -> fault loop -> audio death + freeze).
- xma_decoder.cc: work_event_->Set() in Pause() so the idle XMA worker observes
paused_ and signals pause_fence_ (Pause() deadlock -> permanent audio death).
- audio_system / xma_context_master / xboxkrnl_audio / apu_flags / alsa: mission
audio keepalive + guest_audio_flags + watchdogs.
Instrumentation (additive, default-off): xboxkrnl_debug cache-throw diag +
guest-catch dispatcher, xex_module PE/PDATA/EH scans, kernel_state mem_watch
(NOTE: mem_watch DEFAULTS TRUE -- an always-on host poll thread; prime crash suspect).
NEW ORACLE (see HANDOFF-crash-oracle-2026-07-16.md): stock 6e5b8324f built with
our toolchain + zero custom code = NO crash, NO sound-stop, plays the Ready Room.
=> the Ready-Room out_of_range crash is introduced by THESE changes, not the game
and not the (LTO-broken) build chain. Bisection plan + suspect ranking in the note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
104
src/xenia/apu/audio_watchdog.h
Normal file
104
src/xenia/apu/audio_watchdog.h
Normal file
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
******************************************************************************
|
||||
* Xenia : Xbox 360 Emulator Research Project *
|
||||
******************************************************************************
|
||||
* Copyright 2026 Ben Vanik. All rights reserved. *
|
||||
* Released under the BSD license - see LICENSE in the root for more details. *
|
||||
******************************************************************************
|
||||
*/
|
||||
|
||||
#ifndef XENIA_APU_AUDIO_WATCHDOG_H_
|
||||
#define XENIA_APU_AUDIO_WATCHDOG_H_
|
||||
|
||||
#include <atomic>
|
||||
#include <cstdint>
|
||||
|
||||
// Read-only diagnostics for "audio stops mid-mission and never recovers".
|
||||
//
|
||||
// The audio pipeline has four stages, each of which can die independently:
|
||||
//
|
||||
// guest XAudio callback -> SubmitFrame -> ring buffer -> ALSA write
|
||||
// (guest_pumps) (frames_submitted) (alsa_writes)
|
||||
// XMA decoding feeds the guest (xma_works)
|
||||
//
|
||||
// Each stage bumps a counter here. The watchdog thread in audio_system.cc
|
||||
// samples them once a second and, when audio dies, reports which stage stopped
|
||||
// advancing -- turning "no sound" into a named culprit.
|
||||
//
|
||||
// Note the APU worker runs the guest's audio callback IN-LINE
|
||||
// (AudioSystem::WorkerThreadMain -> processor_->Execute). If that guest call
|
||||
// ever blocks, pumping stops forever and audio can never recover, so we also
|
||||
// track how long we have been inside guest code.
|
||||
//
|
||||
// Everything here is counters only: cvar-gated (--audio_watchdog), no
|
||||
// behaviour change.
|
||||
|
||||
namespace xe {
|
||||
namespace apu {
|
||||
namespace watchdog {
|
||||
|
||||
// Pumps of the guest XAudio callback (AudioSystem worker).
|
||||
inline std::atomic<uint64_t> guest_pumps{0};
|
||||
// steady_clock microseconds at which the worker entered guest code; 0 = not
|
||||
// currently inside the guest callback.
|
||||
inline std::atomic<uint64_t> in_guest_callback_since_us{0};
|
||||
// Frames the guest handed to the driver (AudioSystem::SubmitFrame).
|
||||
inline std::atomic<uint64_t> frames_submitted{0};
|
||||
// XMA decoder Work() calls that actually decoded something.
|
||||
inline std::atomic<uint64_t> xma_works{0};
|
||||
// ALSA: periods of real guest audio written, silence keepalive periods
|
||||
// written, and underruns recovered.
|
||||
inline std::atomic<uint64_t> alsa_writes{0};
|
||||
inline std::atomic<uint64_t> alsa_silence{0};
|
||||
inline std::atomic<uint64_t> alsa_xruns{0};
|
||||
// Last observed snd_pcm_state_t (-1 = unknown / driver not ALSA).
|
||||
inline std::atomic<int> alsa_state{-1};
|
||||
|
||||
// Why is the APU worker not pumping? Three very different causes look the same
|
||||
// from outside the process, so distinguish them here:
|
||||
// worker_loops == 0 && clients_in_use == 0 -> no client (guest unregistered)
|
||||
// worker_loops == 0 && clients_in_use > 0 -> parked despite a client (pacing)
|
||||
// worker_loops > 0 && pump_skips > 0 -> looping, but no free output slot
|
||||
inline std::atomic<uint64_t> worker_loops{0};
|
||||
inline std::atomic<uint64_t> pump_skips{0};
|
||||
inline std::atomic<int> clients_in_use{-1};
|
||||
inline std::atomic<uint64_t> register_calls{0};
|
||||
inline std::atomic<uint64_t> unregister_calls{0};
|
||||
|
||||
// Exactly where the APU worker thread is sitting. A parked worker and a worker
|
||||
// blocked on the kernel global lock are indistinguishable from outside the
|
||||
// process (both are futex waits with zero context switches), so record it.
|
||||
enum class WorkerPhase : int {
|
||||
kStart = 0,
|
||||
kAcquiringGlobalLock, // blocked here => a kernel lock is held elsewhere
|
||||
kParkedNoClient, // blocked here => guest has no audio client
|
||||
kPacingSleep, // normal: waiting for the next 5.33ms deadline
|
||||
kInGuestCallback, // normal: running the game's mixer
|
||||
kSubmitting,
|
||||
};
|
||||
inline std::atomic<WorkerPhase> worker_phase{WorkerPhase::kStart};
|
||||
|
||||
inline const char* WorkerPhaseName(WorkerPhase p) {
|
||||
switch (p) {
|
||||
case WorkerPhase::kStart:
|
||||
return "start";
|
||||
case WorkerPhase::kAcquiringGlobalLock:
|
||||
return "ACQUIRING-GLOBAL-LOCK";
|
||||
case WorkerPhase::kParkedNoClient:
|
||||
return "PARKED-NO-CLIENT";
|
||||
case WorkerPhase::kPacingSleep:
|
||||
return "pacing";
|
||||
case WorkerPhase::kInGuestCallback:
|
||||
return "in-guest-callback";
|
||||
case WorkerPhase::kSubmitting:
|
||||
return "submitting";
|
||||
default:
|
||||
return "?";
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace watchdog
|
||||
} // namespace apu
|
||||
} // namespace xe
|
||||
|
||||
#endif // XENIA_APU_AUDIO_WATCHDOG_H_
|
||||
Reference in New Issue
Block a user