[iterate-4D] multi-core Phase C: free-running executor (opt-in) + diagnosis

Removes the per-round phaser barrier that capped the coarse parallel path at
~parity. Workers now FREE-RUN their HW slots (pick slot's thread, run a
parallel-safe region on the extracted ctx, writeback+epilogue under the lock,
repeat — no waiting on peers); a coordinator thread runs the same housekeeping
(coord_pre_round tickers/timers, dispatch_graphics_interrupts, inline-GPU drain,
coord_idle_advance) only on a wall-clock cadence, quiescing the workers at a
7-party phaser (via a global `quiesce` AtomicBool, robust vs the earlier
epoch-diff which desynced a late-starting worker) so ctx-borrowing housekeeping
stays race-free.

New: run_execution_parallel_freerun, parallel_region_budget() (default 2048,
XENIA_PARALLEL_BUDGET; decoupled from lockstep's 128 so the golden is
untouched), scheduler slot_runnable()/any_runnable(), a per-thread `retired=`
field in the XENIA_DUMP_SLOTS diagnostic, and dropping the global
mmio_access_count region break in the parallel driver (that shared counter,
bumped by ANY worker, collapsed every region to one block).

STATUS — correct but NOT yet a win. Measured (n=2B, --gpu-inline, JIT):
- runs the full 2B and plays the video (4939 draws / 1361 swaps).
- BIMODAL: good runs ~21s (edges out lockstep-JIT's ~24s) but a
  kernel-mutex-contention / guest-spin-wait pathology intermittently latches
  and makes a run 2-3x slower (~57-87s).
Root cause: the single Arc<Mutex<KernelState>> serializes the 6 workers, so the
~4.3x thread-parallelism the workload exposes (measured: work spread across
~5-8 balanced guest threads, top only ~7%) collapses to ~parity. Region-tuning
levers (barrier granularity, MMIO break, GPU cadence, sync break) were each
measured and none crack it — the real win requires FINE-GRAINED kernel locking.

Gated OPT-IN behind XENIA_PARALLEL_FREERUN=1; default --parallel stays the
per-round barrier executor. Lockstep untouched (6-config golden byte-identical);
parallel_stress_short 20/20 ok under FREERUN=1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
MechaCat02
2026-07-05 18:30:40 +02:00
parent cce887d69b
commit 0b00b72292
2 changed files with 460 additions and 4 deletions

View File

@@ -3070,6 +3070,35 @@ fn superblock_budget() -> u64 {
})
}
/// Region budget for the **parallel** unlocked driver
/// (`run_superblock_parallel_unlocked`). Decoupled from [`superblock_budget`]
/// because that 128 is a lockstep scheduling-fidelity value (the boot present
/// loop starves above ~384) tied to the byte-identical golden — the parallel
/// executors are non-deterministic and want a MUCH larger region so each
/// kernel-lock acquisition amortizes over more work (less contention across
/// the 6 workers). Default 2048; `XENIA_PARALLEL_BUDGET` overrides. Never read
/// on the lockstep/golden path. If `XENIA_SUPERBLOCK_BUDGET` is set (the old
/// shared knob) it still wins, so prior A/B measurements remain reproducible.
const PARALLEL_REGION_BUDGET: u64 = 2048;
fn parallel_region_budget() -> u64 {
use std::sync::OnceLock;
static BUDGET: OnceLock<u64> = OnceLock::new();
*BUDGET.get_or_init(|| {
if let Some(v) = std::env::var("XENIA_SUPERBLOCK_BUDGET")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&v| v >= 1)
{
return v;
}
std::env::var("XENIA_PARALLEL_BUDGET")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&v| v >= 1)
.unwrap_or(PARALLEL_REGION_BUDGET)
})
}
/// Superblock runner (iterate-3AL). Executes a *chain* of basic blocks
/// for one slot-visit — following each block's terminating branch into
/// the next block — instead of a single block, amortizing the per-round
@@ -3353,7 +3382,7 @@ fn run_superblock_parallel_unlocked(
) {
use xenia_cpu::interpreter::{step_block, StepResult};
let budget = superblock_budget();
let budget = parallel_region_budget();
let chain_allowed = budget > 1;
let thunk_band = wc.thunk_band;
@@ -3362,7 +3391,6 @@ fn run_superblock_parallel_unlocked(
let mut total_executed: u64 = 0;
loop {
let mmio_before = mem.mmio_access_count();
// Raw-pointer deref (not a `wc` borrow), so the `wc.jit_cache` /
// `wc.block_cache` mutable borrows below don't conflict — same
// discipline as `run_superblock`. `block` is only read (step +
@@ -3384,10 +3412,22 @@ fn run_superblock_parallel_unlocked(
}
total_executed = total_executed.saturating_add(executed);
// NOTE: unlike the lockstep `run_superblock`, we do NOT break the region
// on `mem.mmio_access_count()` changing. That counter is a global
// atomic; with N workers, ANY peer's MMIO bumps it, so keying a break
// off it collapsed every worker's region to a single block. MMIO
// mid-region is thread-safe (guest stores hit the GPU's atomic
// mailboxes; the coordinator/GPU thread drains them), so the guest's
// WAIT_REG_MEM polls still progress.
//
// We DO still break on `sync_sensitive` (lwarx/stwcx/sync/…): measured
// — chaining past a contended guest spinlock made each run 23× SLOWER
// and consistently so (a spinner running a full budget of `lwarx/stwcx`
// per region worsened producer/consumer interleaving). Breaking here
// keeps the spin regions short so a lock-holder gets scheduled sooner.
if !chain_allowed
|| !matches!(result, StepResult::Continue)
|| block.sync_sensitive
|| mem.mmio_access_count() != mmio_before
|| total_executed >= budget
{
return (result, block_ptr, pc_before, total_executed);
@@ -4017,6 +4057,35 @@ fn run_execution_parallel(
.unwrap_or(false),
"--parallel mode is incompatible with XENIA_FORCE_PER_INSTR=1"
);
// Phase C (multi-core): free-running batched executor. Workers run many
// regions back-to-back and rendezvous with the coordinator only on a
// wall-clock cadence (not per round), removing the per-round phaser barrier.
// OPT-IN (XENIA_PARALLEL_FREERUN=1): measured correct (runs the full video)
// and it CAN edge out lockstep-JIT (~21s vs ~24s), but it is bimodal — a
// kernel-mutex-contention / spin-wait pathology intermittently latches and
// makes a run 23× slower (~5787s). Realizing the ~4.3× thread-parallelism
// the workload exposes needs fine-grained kernel locking (the single
// Arc<Mutex<KernelState>> serializes the 6 workers); until then the default
// --parallel path stays the per-round barrier executor below.
if std::env::var("XENIA_PARALLEL_FREERUN")
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false)
{
return run_execution_parallel_freerun(
mem,
kernel_arc,
debugger,
thunk_map,
db_writer,
max_instructions,
ips_limit,
quiet,
halt_on_deadlock,
shutdown_outer,
);
}
let _ = (debugger, db_writer.take()); // suppress 'unused mut' / 'unused' on the bound param
let halt_on_deadlock = halt_on_deadlock
@@ -4443,6 +4512,375 @@ fn run_execution_parallel(
stats_mtx.into_inner().expect("stats mutex poisoned")
}
/// Phase C (multi-core, free-running) — the executor that actually captures
/// the ~4.3× runnable-width parallelism the intro video exposes.
///
/// The per-round phaser barrier in `run_execution_parallel` rendezvoused all
/// six workers + coordinator on EVERY region, so the barrier + lock dance cost
/// dwarfed the region work and the coarse parallel path sat at ~parity (often
/// slightly slower) than lockstep-JIT despite the distributed work. Here the
/// workers **free-run their slots**: each loops picking its slot's runnable
/// thread and running a parallel-safe region (`run_superblock_parallel_unlocked`,
/// on an extracted ctx, unlocked) back-to-back, taking the kernel mutex only
/// for the short prologue / writeback+epilogue — never waiting on peers.
///
/// A dedicated coordinator (this thread) runs the SAME housekeeping as the
/// barrier executor (`coord_pre_round` tickers/timers, autosignals, timestamp
/// bundle, `dispatch_graphics_interrupts`, inline-GPU drain via
/// `coord_post_round`, `coord_idle_advance` deadlock/idle handling) — but only
/// on a wall-clock cadence (`COORD_TICK`), not once per region. To keep the
/// ctx-borrowing housekeeping (ISR victim borrow) race-free, the coordinator
/// bumps `batch_epoch` and rendezvouses the workers at a 7-party phaser so they
/// are fully quiesced (no extracted ctx in flight) for the housekeeping window,
/// then releases them. So the barrier still exists, but fires ~once per tick
/// instead of once per region — amortized ~1000× — while housekeeping stays as
/// safe as the barrier model.
///
/// Non-deterministic by design (thread interleaving); this is the opt-in perf
/// mode. Lockstep (`run_execution`) remains the byte-identical golden path.
#[allow(clippy::too_many_arguments)]
fn run_execution_parallel_freerun(
mem: &xenia_memory::GuestMemory,
kernel_arc: &std::sync::Arc<std::sync::Mutex<xenia_kernel::KernelState>>,
debugger: &mut xenia_debugger::Debugger,
thunk_map: &HashMap<u32, (ModuleId, u16, String)>,
mut db_writer: Option<&mut xenia_analysis::DbWriter>,
max_instructions: Option<u64>,
ips_limit: Option<u64>,
quiet: bool,
halt_on_deadlock: bool,
shutdown_outer: Option<std::sync::Arc<std::sync::atomic::AtomicBool>>,
) -> ExecStats {
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use xenia_cpu::{Phaser, PhaserOutcome};
let _ = quiet;
// Same incompatibilities as the barrier executor (asserted by the caller
// `run_execution_parallel` before it delegates here).
let _ = (&debugger, db_writer.take());
let halt_on_deadlock = halt_on_deadlock
|| std::env::var("XENIA_HALT_ON_DEADLOCK")
.map(|v| v == "1" || v.eq_ignore_ascii_case("true"))
.unwrap_or(false);
let throttle_start = Instant::now();
// How long workers free-run between coordinator housekeeping rendezvous.
// Small enough that vsync/timer/GPU cadence stays responsive; large enough
// that the quiesce barrier is deeply amortized over each batch's regions.
let coord_tick = Duration::from_micros(
std::env::var("XENIA_PARALLEL_TICK_US")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.filter(|&v| v >= 10)
.unwrap_or(200),
);
// Idle-slot backoff: a worker whose HW slot has no runnable thread parks
// this long, then re-checks (coarse parking — no precise cross-thread wake
// plumbing in this increment; the coordinator unparks on rendezvous).
let idle_park = Duration::from_micros(50);
const COORD_ID: u8 = xenia_cpu::scheduler::HW_THREAD_COUNT as u8; // = 6
const PARTY_COUNT: u32 = xenia_cpu::scheduler::HW_THREAD_COUNT as u32 + 1;
let phaser: Arc<Phaser> = Arc::new(Phaser::new(PARTY_COUNT));
let internal_shutdown: Arc<AtomicBool> = Arc::new(AtomicBool::new(false));
// Set by the coordinator (Release) to request that every worker wrap up its
// current region and rendezvous at the quiesce barrier; cleared by the
// coordinator after B1 (all workers acked) and before B2, so a worker's
// post-B2 read always sees `false` and it resumes free-running. A global
// flag — no per-worker epoch state that could initialize out of sync (the
// bug the epoch model had: a late-starting worker read the already-bumped
// epoch as "already acked" and never rendezvoused).
let quiesce: Arc<AtomicBool> = Arc::new(AtomicBool::new(false));
// Lock order: kernel mutex first, stats mutex second (never inverted).
let stats_mtx: Mutex<ExecStats> = Mutex::new(ExecStats::default());
std::thread::scope(|scope| {
let mut worker_threads = Vec::with_capacity(xenia_cpu::scheduler::HW_THREAD_COUNT);
for hw_id in 0..xenia_cpu::scheduler::HW_THREAD_COUNT as u8 {
let phaser_w = phaser.clone();
let kernel_w = kernel_arc.clone();
let shutdown_w = internal_shutdown.clone();
let quiesce_w = quiesce.clone();
let stats_ref: &Mutex<ExecStats> = &stats_mtx;
let mem_ref: &xenia_memory::GuestMemory = mem;
let thunk_map_ref = thunk_map;
let handle = scope.spawn(move || {
let mut wc = WorkerCtx::new(hw_id, /*force_per_instr=*/ false);
let mut local_debugger = xenia_debugger::Debugger::new();
local_debugger.paused = false;
local_debugger.step_mode = xenia_debugger::StepMode::Run;
local_debugger.trace_enabled = false;
let mut local_db_writer: Option<&mut xenia_analysis::DbWriter> = None;
'worker: loop {
if shutdown_w.load(Ordering::Acquire) {
break 'worker;
}
// Coordinator requested a quiesce → rendezvous at the
// barrier (B1), wait out its housekeeping (B2), resume. The
// coordinator clears `quiesce` between B1 and B2, so our
// post-B2 read below sees `false` and we free-run again.
if quiesce_w.load(Ordering::Acquire) {
match phaser_w.arrive_and_wait(hw_id) {
PhaserOutcome::Advanced => {}
PhaserOutcome::Shutdown => break 'worker,
PhaserOutcome::Timeout => {
shutdown_w.store(true, Ordering::Release);
phaser_w.shutdown();
break 'worker;
}
}
match phaser_w.arrive_and_wait(hw_id) {
PhaserOutcome::Advanced => {}
PhaserOutcome::Shutdown => break 'worker,
PhaserOutcome::Timeout => {
shutdown_w.store(true, Ordering::Release);
phaser_w.shutdown();
break 'worker;
}
}
continue 'worker;
}
// ── Run one region on this HW slot. ──
let prologue_outcome = {
let mut guard = kernel_w.lock().expect("kernel mutex poisoned");
if !guard.scheduler.slot_runnable(hw_id) {
// Idle slot — nothing to run this instant. Park
// briefly (coord unparks on quiesce) and re-check.
drop(guard);
std::thread::park_timeout(idle_park);
continue 'worker;
}
if wc.thunk_band.is_none() {
wc.thunk_band = guard.thunk_addr_band();
}
let mut s = stats_ref.lock().expect("stats mutex poisoned");
let r = worker_prologue(
&mut wc,
&mut *guard,
mem_ref,
&mut local_debugger,
&mut local_db_writer,
thunk_map_ref,
&mut *s,
);
drop(s);
(r, guard)
};
match prologue_outcome.0 {
PrologueOutcome::Continue => {
drop(prologue_outcome.1);
}
PrologueOutcome::BreakOuter => {
drop(prologue_outcome.1);
shutdown_w.store(true, Ordering::Release);
phaser_w.shutdown();
break 'worker;
}
PrologueOutcome::StepBlock {
tid,
thread_ref,
block_ptr,
pc_before,
} => {
let mut guard = prologue_outcome.1;
let mut ctx_taken = std::mem::replace(
guard.scheduler.ctx_mut_ref(thread_ref),
xenia_cpu::PpcContext::new(),
);
guard.scheduler.end_slot_visit();
drop(guard);
// ── unlocked window: a whole region ──
let (result, last_block_ptr, last_pc_before, executed) =
run_superblock_parallel_unlocked(
&mut wc,
mem_ref,
&mut ctx_taken,
thunk_map_ref,
block_ptr,
pc_before,
);
let mut guard =
kernel_w.lock().expect("kernel mutex poisoned");
let target_ref = tid
.and_then(|t| guard.scheduler.find_by_tid(t))
.unwrap_or(thread_ref);
*guard.scheduler.ctx_mut_ref(target_ref) = ctx_taken;
guard.scheduler.advance_global_clock(executed);
guard.scheduler.current = Some(target_ref);
let epilogue_outcome = {
let mut s =
stats_ref.lock().expect("stats mutex poisoned");
let r = worker_epilogue(
&mut wc,
&mut *guard,
&mut local_debugger,
&mut *s,
tid,
target_ref,
last_block_ptr,
last_pc_before,
result,
executed,
);
drop(s);
r
};
guard.scheduler.current = None;
drop(guard);
if matches!(epilogue_outcome, SlotOutcome::BreakOuter) {
shutdown_w.store(true, Ordering::Release);
phaser_w.shutdown();
break 'worker;
}
}
}
}
});
worker_threads.push(handle);
}
// ── Coordinator (this thread) ──
let mut isr_decode_cache = xenia_cpu::decoder::DecodeCache::new();
let mut last_instr: u64 = 0;
'coord: loop {
if internal_shutdown.load(Ordering::Acquire) {
phaser.shutdown();
break 'coord;
}
// Let the workers free-run their slots for one tick.
std::thread::sleep(coord_tick);
// Request a rendezvous and nudge any idle-parked workers so they
// notice promptly instead of waiting out their park timeout.
quiesce.store(true, Ordering::Release);
for h in &worker_threads {
h.thread().unpark();
}
// B1: wait for all six workers to reach the barrier. The coordinator
// holds no lock here, so workers can finish their in-flight region
// (writeback under the lock) before arriving.
match phaser.arrive_and_wait(COORD_ID) {
PhaserOutcome::Advanced => {}
PhaserOutcome::Shutdown => break 'coord,
PhaserOutcome::Timeout => {
tracing::warn!(
instr = last_instr,
"freerun coordinator: B1 timeout; shutting down"
);
internal_shutdown.store(true, Ordering::Release);
phaser.shutdown();
break 'coord;
}
}
// All workers are now blocked at B2 with no extracted ctx in flight.
// Clear the quiesce request BEFORE housekeeping / B2 so each worker's
// post-B2 read sees `false` and resumes free-running.
quiesce.store(false, Ordering::Release);
// ── Housekeeping (workers fully quiesced at B2 — no extracted ctx
// in flight, so ISR ctx-borrowing is race-free, exactly as in the
// barrier executor). ──
let mut done = false;
{
let mut guard = kernel_arc.lock().expect("kernel mutex poisoned");
// Guarantee a clean `scheduler.current` for the ctx-borrowing
// housekeeping (a worker's thunk-dispatch leaves it Some).
guard.scheduler.end_slot_visit();
let mut s = stats_mtx.lock().expect("stats mutex poisoned");
match coord_pre_round(
&mut *guard,
&*s,
max_instructions,
ips_limit,
throttle_start,
&shutdown_outer,
) {
RoundCtl::BreakOuter => done = true,
RoundCtl::Continue => {}
}
if !done {
guard.set_now_cycle_hint(s.instruction_count);
guard.fire_due_silph_autosignals(s.instruction_count);
{
let clock = guard.scheduler.global_clock();
guard.update_timestamp_bundle(mem, clock);
}
dispatch_graphics_interrupts(
&mut *guard,
mem,
&mut *s,
&mut isr_decode_cache,
thunk_map,
);
let start = last_instr;
last_instr = s.instruction_count;
if matches!(
coord_post_round(&mut *guard, mem, &*s, start),
RoundCtl::BreakOuter
) && !guard.scheduler.has_live_thread()
{
done = true;
}
// If nothing is runnable, advance time to the next deadline
// / handle deadlock (same as the barrier executor's idle
// path). Timer fires here re-arm runnable threads that the
// workers pick up next batch.
if !done && !guard.scheduler.any_runnable() {
match coord_idle_advance(
&mut *guard,
halt_on_deadlock,
&shutdown_outer,
&*s,
) {
RoundCtl::BreakOuter => done = true,
RoundCtl::Continue => {}
}
}
}
drop(s);
drop(guard);
}
if done {
internal_shutdown.store(true, Ordering::Release);
phaser.shutdown(); // releases workers waiting at B2
break 'coord;
}
// B2: release the workers for the next batch.
match phaser.arrive_and_wait(COORD_ID) {
PhaserOutcome::Advanced => {}
PhaserOutcome::Shutdown => break 'coord,
PhaserOutcome::Timeout => {
tracing::warn!("freerun coordinator: B2 timeout; shutting down");
internal_shutdown.store(true, Ordering::Release);
phaser.shutdown();
break 'coord;
}
}
}
}); // <- thread::scope joins all workers here.
stats_mtx.into_inner().expect("stats mutex poisoned")
}
/// Iterate-2.BE — host-driven synchronous dispatch of all queued
/// graphics interrupts. Mirrors canary's
/// [`EmulateCPInterruptDPC`](../../../../xenia-canary/src/xenia/kernel/kernel_state.cc#L1370)
@@ -4896,7 +5334,7 @@ fn dump_thread_diagnostic(
println!(" hw={} running_idx={:?} depth={}", hw_id, slot.running_idx, slot.runqueue.len());
for (idx, t) in slot.runqueue.iter().enumerate() {
println!(
" idx={} tid={} handle={:?} state={:?} suspend_count={} pri={} mask={:#04x} pc={:#010x}",
" idx={} tid={} handle={:?} state={:?} suspend_count={} pri={} mask={:#04x} pc={:#010x} retired={}",
idx,
t.tid,
t.thread_handle.map(|h| format!("{:#06x}", h)),
@@ -4905,6 +5343,7 @@ fn dump_thread_diagnostic(
t.priority,
t.affinity_mask,
t.ctx.pc,
t.ctx.cycle_count,
);
}
}

View File

@@ -586,6 +586,23 @@ impl Scheduler {
}
}
/// True if HW slot `hw_id` currently has at least one Ready/ServicingIrq
/// thread (from the cached `non_empty_runnable` bitmap — O(1), no scan).
/// Used by the free-running parallel executor so an idle-slot worker can
/// park instead of spinning through a full `worker_prologue`.
#[inline]
pub fn slot_runnable(&self, hw_id: u8) -> bool {
(self.non_empty_runnable & (1 << hw_id)) != 0
}
/// True if ANY HW slot has a runnable thread (O(1)). Lets the free-running
/// coordinator distinguish "all slots idle → advance time / check deadlock"
/// from "work in flight".
#[inline]
pub fn any_runnable(&self) -> bool {
self.non_empty_runnable != 0
}
// ----- Compat accessors (preserve the pre-Axis-1 hw_threads[i].ctx pattern) -----
/// Read-only context of the currently-running thread on `hw_id`.