xenia-rs/audit-runs/scheduler-determinism-plan/investigation.md

Investigation Notes — Scheduler-Determinism Plan (2026-05-18)

Source citations and probe results from the Phase-1 investigation. All claims here are verified against source or runtime data; speculation is flagged.

1. Canary threading & scheduling model

Verdict: 1-host-thread-per-XThread; scheduling delegated to host OS (Wine on Linux). No internal scheduler.

• Each guest XThread owns a host xe::threading::Thread (xenia-canary/src/xenia/kernel/xthread.h:476).
• POSIX backend: pthread per XThread (xenia-canary/src/xenia/base/threading_posix.cc).
• TLS bridge: thread_local XThread* current_xthread_tls_ (xthread.cc:105). XThread::TryGetCurrentThread() returns null when called outside a guest thread (C+15-α robustness fix for the boot-time emitter).
• Tid assignment: thread_id_(++next_xthread_id_) in ctor (xthread.cc:62).
• KPCR per XThread, allocated at pcr_address_ (xthread.h:506); contains scheduler-like state mirroring real Xenon KPRCB.
• CheckQuantumAndDecay() (xthread.h:437) fires ~20ms via KernelState's timer — simulates Xenon priority decay but does NOT preempt; runs on whichever host thread the host OS schedules.

No internal scheduler. No lockstep, deterministic, replay cvar (grep confirmed across xenia-canary/src/xenia/).

2. Canary clock infrastructure

Verdict: wallclock-driven (rdtsc or platform API). Optional scaling, no full deterministic mode.

• Canonical class xe::Clock (base/clock.h:30).
• Clock::QueryHostTickCount() (base/clock.cc:128): rdtsc on x64 if clock_source_raw=true, else platform API.
• Clock::QueryGuestSystemTime() (clock.h:82): host time adjusted by guest_time_scalar_.
• KeQuerySystemTime_entry (xboxkrnl_threading.cc:459): declared void, writes via OUT pointer; reads Clock::QueryGuestSystemTime(). (C+1 verified parity with ours's void-export framing.)
• KeWaitForSingleObject_entry (xboxkrnl_threading.cc:1003): reads *timeout_ptr as i64×100 → ns (C+23 verified ours computes the same value).
• Cvars: clock_no_scaling (base/clock.cc:24), clock_source_raw (base/clock.cc:28). Neither makes the clock deterministic across Wine runs — wallclock drift is irreducible.

3. Canary wait primitives

Verdict: xe::threading::Wait → pthread_cond_timedwait (POSIX) / WaitForMultipleObjects (Win32).

• xeKeWaitForSingleObject() (xboxkrnl_threading.cc:969) → XObject::Wait() → xe::threading::Wait() → host primitive.
• Whether contention happens is purely host-OS-scheduler-driven. Reading-error #32 from C+20 documents this: 3 fresh canary cold runs at tid=6 idx 104,606 showed different patterns (no wait.begin / wait.begin contended / offset-shifted).

4. Canary RtlEnterCriticalSection — spin-then-wait (DISCOVERED)

xenia-canary/src/xenia/kernel/xboxkrnl/xboxkrnl_rtl.cc:596-633 — RtlEnterCriticalSection_entry:

uint32_t spin_count = cs->header.absolute * 256;  // game-supplied spin count
if (cs->owning_thread == cur_thread) { recursion++; return; }
while (spin_count--) {
  if (xe::atomic_cas(-1, 0, &cs->lock_count)) { /* acquired via spin */ break; }
}
if (xe::atomic_inc(&cs->lock_count) != 0) {
  xeKeWaitForSingleObject(...);  // slow path
}
cs->owning_thread = cur_thread; cs->recursion_count = 1;

Implication: under low contention, spin succeeds and no wait.begin is emitted. Under high contention, spin fails and wait.begin fires. Whether spin succeeds depends on host-OS timing — non-deterministic across Wine runs.

5. Ours threading & scheduling

Verdict: single host thread; 6 cooperative HW slots; deterministic by construction.

• xenia-rs/crates/xenia-cpu/src/scheduler.rs:
  • OrderMode { Fixed, Seeded { seed } } (lines 230-258).
  • round_schedule() (lines 710-740): returns slot-id vector; advances rotation_cursor by 1.
  • park_current(BlockReason) (line 808).
  • wake_ref(ThreadRef) (line 831).
• M3 optional --parallel mode (6 workers + coordinator, 7-party phaser) exists but is not default.

Determinism foundation: 23 phases of stabilization invested in e1dfcb15… cold digest × 3 reproducible.

6. Ours RtlEnterCriticalSection — NO spin

xenia-rs/crates/xenia-kernel/src/exports.rs:2886-2946 — rtl_enter_critical_section:

let owner = mem.read_u32(cs_ptr + CS_OFFS_OWNING_THREAD);
let owner_is_live = owner != 0 && state.scheduler.find_by_tid(owner).is_some();
if owner == 0 || !owner_is_live {
  /* claim immediately — write owning_thread, lock_count=0, recursion=1 */
  return;
}
if owner == current_tid { /* recursive lock — increment counts */ return; }
// Truly contended against a live peer — park IMMEDIATELY (no spin).
state.cs_waiters.entry(cs_ptr).or_default().push(current_ref);
state.scheduler.park_current(BlockReason::CriticalSection(cs_ptr));

Asymmetry summary: canary spins ~256×N times before parking; ours parks immediately. Under the cooperative scheduler, ours's tid=1 runs monolithically until it parks — no other thread has a chance to acquire the CS first. Hence at 104,607, the CS is free when tid=1 tries, while in canary it was held by another thread that got scheduled in between.

7. Ours clock infrastructure

Verdict: fixed FILETIME constant. No wallclock dependency in the hot path.

• KeQuerySystemTime returns 132_500_000_000_000_000 (~2021) via OUT-ptr (exports.rs:628).
• KeQueryInterruptTime returns 0x0000_0001_0000_0000 (exports.rs:504).
• event_log.rs uses Instant::now() for the observability host_ns field — non-deterministic but not consumed by the matched-prefix metric.

8. Sylpheed workload profile (probe)

Ran on xenia-rs/audit-runs/phase-c22-rtl-enter-leave-control-flow/ours-cold.jsonl (121,569 events):

event	count	notes
RtlEnterCriticalSection (kernel.call)	19,494	≈80% of all kernel.calls
RtlLeaveCriticalSection (kernel.call)	19,492	matches Enter (off-by-2 from boot edge)
NtClose	160
NtCreateEvent	103
NtReleaseSemaphore	99
NtQueryInformationFile	93
NtWaitForMultipleObjectsEx	92
KeWaitForSingleObject	5
KeWaitForMultipleObjects	1
KeQuerySystemTime	2	clock-light workload
KeQueryPerformanceFrequency	6
KeQueryPerformanceCounter	0
KeQueryInterruptTime	0
KeDelayExecutionThread	0
NtYieldExecution	0
wait.begin events (all kinds)	34	most with timeout_ns=-1 (indefinite)

Implications:

• Sylpheed is CS-dominated. Stage-1 emitter on RtlEnterCS captures the dominant signal.
• Sylpheed barely touches the clock. Approach A (cycle clock in canary) addresses ≈2 events out of 121,569. Wrong target.
• Wait surface is small (34 events). Wait-side replay is low-value; scope to CS only.

9. The 104,607 divergence (re-verified)

From C+22 memory + jitter jsonl re-analysis:

sample	tid=6 events 104,604..104,615 (import.call only)
c21 archived	E E L L
canary jitter-1	E (wait.begin slow path) E L L
canary jitter-2	E E L L
canary jitter-3	(shifted) E E L L
fresh c22	E (wait.begin slow path) E L L

All canary samples have the EXTRA nested RtlEnterCriticalSection (second E before the final L L). Ours never does — it goes E L NtClose. Structural divergence post-absorber-engagement.

Shared dispatcher: canary's wait.begin handles_semantic_ids=['75ae880ec432eb36'] — this is the CS embedded Event dispatcher, lazy-wrapped by XObject::GetNativeObject. Same SID computed via C+18 shared-global recipe in both engines.

10. Cvar inventory (canary side)

Grep across xenia-canary/src/xenia/ for DEFINE_bool|DEFINE_int|DEFINE_uint|DEFINE_string:

• clock_no_scaling (base/clock.cc:24)
• clock_source_raw (base/clock.cc:28)
• ignore_thread_priorities (kernel/xthread.cc:30)
• ignore_thread_affinities (kernel/xthread.cc:33)
• stack_size_multiplier_hack (kernel/xthread.cc:37)
• main_xthread_stack_size_multiplier_hack (kernel/xthread.cc:39)
• phase_a_event_log_path (cpu/cpu_flags.cc:84) — Phase A trace gate
• phase_a_event_log_mem_writes (cpu/cpu_flags.cc:88) — reserved, not wired
• phase_b_snapshot_dir (cpu/cpu_flags.cc:94) — Phase B image snapshot
• phase_b_snapshot_and_exit (cpu/cpu_flags.cc:100)

No lockstep, deterministic, replay, single_thread, cooperative cvars exist. No built-in deterministic mode.

11. Diff-tool absorber state (post-C+21)

xenia-rs/tools/diff-events/diff_events.py (767 LOC):

• collect_shared_global_sids(): pre-pass union of (a) recipe-matching SIDs (C+18) and (b) cross-tid usage heuristic — any SID used by handle.create OR wait.begin on ≥2 distinct tids.
• is_shared_global_wait_begin(): classifies a wait.begin as floating if any handle_sid is in the shared-global set.
• diff_one_tid(): floating-absorbs handle.create (C+18) and wait.begin (C+21) on kind mismatches.
• SKIP_PAYLOAD_FIELDS_BY_KIND: skips engine-local fields per kind.

Reading-error #23 boundary: absorbing the post-wait Enter/Leave block (canary's extra E then L at 104,610-104,615) would be folding real guest behavior, not transient observation. The plan's Stage 3 instead makes ours produce the same observation by forcing ours into the same contended state.

12. Tid-chain mapping (stable per memory baseline)

canary	ours
6	1
4	11
7	2
12	7
14	9
15	10

This is a display convention for cross-engine alignment in diff reports. In the wire format, each engine emits its native tid. The manifest in Stage 2-3 keys on the source-side native tid — no translation needed since each side consumes events it produced.

13. Methodology rules in force

• Reading-error #28 (verify source first): applied — read both engines' RtlEnterCS implementations before designing.
• Reading-error #32 (canary non-deterministic in contention regions): characterized — 3 jitter samples documented.
• Reading-error #33 (canary cache lives in binary-dir under wine): not relevant here.
• Reading-error #34 (use .iso not loose .xex): apply in all validation runs.
• Cold-vs-cold protocol: canary --mute=true, ours XENIA_CACHE_WIPE=1.
• Stop hook rename: rename background binaries before any backgrounded run (e.g. xrs-verify-stage0, xrs-replay).

14. Confidence calibration

claim	source-verified	probe-verified	confidence
Canary spins, ours doesn't	yes (xboxkrnl_rtl.cc:613 + exports.rs:2927)	n/a (static)	high
Sylpheed clock-light	n/a	yes (kernel.call counts)	high
104,607 divergence is structural	yes (C+22 mech)	yes (5 canary samples consistent)	high
C+18 shared-global SID is cross-engine identical	yes (event_log.rs + event_log.cc)	implicit (matched in diff reports)	high
Canary has no deterministic mode cvar	yes (grep)	n/a	high
Stage-0 quantum spike may unblock	no (untested)	no	medium
Stage-3 manifest replay unblocks	no	no	medium-high (mechanism sound, integration risk)
Sister chain regression ≤5 acceptable	n/a	n/a	open question for user

Open unknowns (deferred to implementation)

1. The exact cs_ptr of the contended CS at canary tid=6 idx 104,608 is not directly emitted by the current schema (the wait.begin payload carries SID but not the raw pointer). Stage 1's cs_ptr field plugs this gap.
2. Does Sylpheed initialize the contended CS with RtlInitializeCriticalSectionAndSpinCount(spin_count > 0) or just RtlInitializeCriticalSection (zero spin)? Affects whether canary's spin path can succeed at this site. Probe by reading the cs's header.absolute field during a canary run.
3. The dispatcher Event's first-toucher tid differs across cold runs (canary tid=9 in one, others in others). Does this stable enough across cold runs of the SAME canary binary to be a reliable replay anchor? Stage 1 round-trip validation will reveal.
4. Does the M3 --parallel mode in ours reproduce the same divergence pattern? Untested. Out of scope for this plan but worth a future probe.