xenia-rs/audit-runs/phase-c20-rtl-enter-cs-wait/investigation.md

Phase C+20 investigation — RtlEnterCriticalSection wait.begin (2026-05-14)

Framing verification (reading-error #28 discipline)

Canary's RtlEnterCriticalSection — xboxkrnl_rtl.cc:596-633

void RtlEnterCriticalSection_entry(pointer_t<X_RTL_CRITICAL_SECTION> cs) {
  if (!cs.guest_address()) { ... return; }
  CriticalSectionPrefetchW(&cs->lock_count);
  uint32_t cur_thread = XThread::GetCurrentThread()->guest_object();
  uint32_t spin_count = cs->header.absolute * 256;

  if (cs->owning_thread == cur_thread) {           // RECURSIVE FAST PATH
    xe::atomic_inc(&cs->lock_count);
    cs->recursion_count++;
    return;
  }

  // Spin loop
  while (spin_count--) {
    if (xe::atomic_cas(-1, 0, &cs->lock_count)) {  // UNCONTENDED FAST PATH
      cs->owning_thread = cur_thread;
      cs->recursion_count = 1;
      return;
    }
  }

  if (xe::atomic_inc(&cs->lock_count) != 0) {      // CONTENDED SLOW PATH
    // Create a full waiter.
    xeKeWaitForSingleObject(reinterpret_cast<void*>(cs.host_address()), 8, 0, 0,
                            nullptr);
  }
  assert_true(cs->owning_thread == 0);
  cs->owning_thread = cur_thread;
  cs->recursion_count = 1;
}

Canary only emits wait.begin on the contended slow path (via the xeKeWaitForSingleObject call). The wait handle is the CS struct pointer; xeKeWaitForSingleObject resolves it via XObject::GetNativeObject which lazy-wraps the embedded DISPATCHER_HEADER (first 12 bytes of the CS struct) as an XEvent — the SID 75ae880ec432eb36 (object_type=1, raw_handle=0xf8000044) seen at canary tid=9 idx=295 IS this Event, synthesized on first contention.

xeKeWaitForSingleObject emit point — xboxkrnl_threading.cc:969-991

uint32_t xeKeWaitForSingleObject(void* object_ptr, uint32_t wait_reason, ...) {
  auto object = XObject::GetNativeObject<XObject>(kernel_state(), object_ptr);
  if (!object) { assert_always(); return X_STATUS_ABANDONED_WAIT_0; }

  if (phase_a::IsEnabled()) {
    uint64_t sid = 0;
    if (!object->handles().empty()) {
      sid = phase_a::LookupHandleSemanticId(object->handles()[0]);
    }
    int64_t timeout_ns = timeout_ptr ? (*timeout_ptr * 100) : -1;
    phase_a::EmitWaitBegin(&sid, 1, timeout_ns, alertable != 0, false);
  }

  X_STATUS result = object->Wait(...);
  ...
}

Confirms: wait.begin fires only when the slow path is taken.

Ours's rtl_enter_critical_section — exports.rs:2886-2946

Has three branches:

1. owner == 0 || !owner_is_live → claim uncontended.
2. owner == current_tid → recursive bump.
3. otherwise → park current thread on cs_waiters via state.scheduler.park_current(BlockReason::CriticalSection(cs_ptr)).

The park path does NOT emit wait.begin. Symmetric to canary's slow path semantically, but no schema event.

Divergent event observed (fresh canary cold + fresh ours cold)

[104604] ours+canary import.call RtlEnterCriticalSection
[104605] ours+canary kernel.call  RtlEnterCriticalSection
[104606] CANARY   wait.begin sid=75ae880ec432eb36 timeout=-1 wait_type=any
[104606] OURS     kernel.return RtlEnterCriticalSection rv=0
[104607] CANARY   kernel.return RtlEnterCriticalSection rv=0

Classification

This is a (B) Real contention difference, NOT (A) always-wait, NOT (C) emit gap.

Evidence:

1. Canary's RtlEnterCriticalSection source code provably only emits wait.begin in the contended branch. The earlier two RtlEnterCriticalSection sequences (canary tid=6 idx=104,598-600 and idx=104,608-610) BOTH fast-path (no wait.begin) — proving canary's path is conditional on contention.

2. SID 75ae880ec432eb36 appears 15 times in canary, on 4 different tids (tid=6/9/10/18). Always with object_type=1 (Event). All 15 are wait.begin (or 1 handle.create first-touch). This is a shared CS used across the title's thread pool.

3. At canary's idx 104,604, the CS is contended because tid=9 is simultaneously doing cache-file work (NtCreateFile cache:\69d8e45ce534ffea.tmp at canary tid=9 idx=305) that almost certainly enters the same CS first. Canary's host_ns gap between ours-idx 104,603 (RtlLeave) and 104,604 (RtlEnter) is 268.2 ms, during which thousands of other-tid events fire.

4. At ours's idx 104,604, only tid=1 and tid=5 are active in a 1ms window around the call. tid=5 is in MmFreePhysicalMemory — not touching this CS. Ours's gap between idx 104,603→104,604 is 7.6 μs. Effectively single-threaded.

5. Ours has no other live thread holding this CS — fast path is the correct semantic result for ours's scheduling.

Why this is scheduler determinism

The contention pattern emerges from the interleaving of multiple guest threads racing on a shared CS. To make ours produce the same event sequence as canary at this idx, we would need:

• tid=9 (or another holder) to be currently inside its critical section block when tid=1 reaches idx 104,604.
• That requires ours to schedule tid=9 ahead of (or concurrently with) tid=1's RtlEnter, exactly as canary's host scheduler did.
• Ours's deterministic single-stepping scheduler runs tid=1 near-monolithically through this region — tid=9 has no opportunity to claim the CS before tid=1 fast-paths through.

This is the canonical signature of cross-thread scheduling asymmetry. Fixing it requires either:

(i) Reworking ours's scheduler to interleave threads at finer granularity matching canary's preemption points — substantial refactor of xenia-cpu::scheduler.

(ii) Recording a "scheduling trace" from canary (which thread holds which CS at which guest_cycle) and replaying it in ours — new subsystem.

(iii) Forcing ours to spin-wait briefly at every RtlEnter so other tids get a chance to claim the CS — extremely fragile, no guarantee of matching canary's exact interleave.

None of these are scoped for a single phase-C iteration. The prompt's authorized scope explicitly says:

You may NOT refactor thread scheduling (escalation: scheduler determinism is a separate session).

Escalation: if classification is (B) and scheduler determinism is required, escalate cleanly — don't push through.

Decision: ESCALATE + diff-tool TODO

C+20 produces no engine change. The classification, supporting evidence, and recommended escalation path are recorded for a future "scheduler-determinism" milestone.

Additional diff-tool action (NOT executed in C+20 per scope): the diff tool should be taught to absorb cross-tid race-window wait.begin events on shared CS dispatchers (analog to C+18's shared-global SID floating-absorb for handle.create). The divergence at idx 104,606 is a strict sub-case of class #30 (scheduling-determinism observation artifact). A follow-up phase (C+20.5 or part of the scheduler-determinism track) should:

1. Detect wait.begin events with SID matching the canary jitter-1's 75ae880ec432eb36 pattern (multi-tid usage, type=1 Event, first-touched by GetNativeObject from an RtlEnter slow path).
2. Mark as "scheduling-jitter-window" and floating-absorb in the diff walk so matched-prefix doesn't anchor to it.

This would reveal the true next divergence beyond the jitter cloud.

Risk of "partial" fixes considered

Could we just always emit wait.begin in ours's rtl_enter_critical_section?

No — would produce phantom wait.begin events on the fast path where canary correctly emits none. Would regress at the very next RtlEnterCriticalSection that ours fast-paths (e.g., ours idx 104,598 where canary also fast-paths). Net effect: shifts the divergence elsewhere, doesn't fix it.

Could we wire wait.begin into ours's park_current(CriticalSection)?

Yes — this would be semantically symmetric to canary and is a small patch (~25 LOC). But it would NOT fix the divergence at idx 104,606, because ours doesn't park at this call site at all. The patch would be inert until a different test case exposes a path where ours does park on a CS. Useful prophylactic, but not the C+20 target.

Could we remove the owner_is_live shortcut?

The !owner_is_live heuristic in ours treats owner != 0 && find_by_tid(owner).is_none() as "free". At idx 104,604, this is not the triggered branch — the CS is genuinely uncontended (owner == 0 on the first probe), so removing it doesn't change behavior here.

Reading-error class #31 (documented per prompt) + #32 (NEW)

#31 Stale-canary-jsonl trap — always re-run canary fresh for cold-vs-cold measurements. The prompt established this.

#32 (NEW) Canary itself is non-deterministic across cold runs in contention-dependent regions. Cross-checking the 3 fresh canary jitter jsonls at tid=6 idx 104,595-104,612 confirms canary is structurally non-deterministic here:

jitter	idx 104,606 event
1	wait.begin sid=75ae880ec432eb36
2	kernel.return RtlEnterCriticalSection (fast path, no wait!)
3	kernel.call RtlLeaveCriticalSection (sequence shifted; the
	wait.begin shifted to idx 104,603 with sid=a25a16a4f6f547aa)

jitter-2's behavior at idx 104,606 is bit-identical to ours. jitter-3 has the wait.begin at a different idx with a different SID — proving the contention pattern is host-scheduler-dependent in canary itself.

This means:

1. The prompt's framing ("canary emits wait.begin, ours emits kernel.return") was based on ONE jitter sample (jitter-1). It is not a stable structural property of canary.
2. Matched-prefix as a cross-engine metric is unreliable in regions where canary's contention is host-scheduler-driven.
3. There is NO real engine bug to fix here. Ours's behavior matches canary jitter-2 at idx 104,606 verbatim.

Reading-error class #32: assuming canary determinism by sampling ONE cold run; need ≥2-3 cold samples to distinguish "real divergence" from "scheduler-driven jitter window".

Cascade outcome

• A=verify canary's RtlEnterCriticalSection impl: PASS.
• B=classify (A/B/C): PASS — (B), real contention.
• C=land fix (or clean escalation): ESCALATION (per prompt authorized scope).
• D=main matched-prefix > 104,606: N/A (no code change).

Recommendation for next session

C+20-escalation = open a parallel scheduler-determinism track:

1. Add a per-CS-pointer "expected contention" inference from canary logs.
2. Drive ours's scheduler to preempt tid=1 at each RtlEnter site where canary's matched call exhibits a wait.begin.
3. Verify diff-tool absorbs as a structured "scheduling-trace replay" event class.

In parallel, address D-NEW-2 (KeWaitForSingleObject timeout_ns sign/scale asymmetry on tid=12→7 idx=3) — a small ε-class encoding fix that's independent of scheduler determinism.

Also worth landing as a small prophylactic patch (NOT in C+20): wire wait.begin into ours's rtl_enter_critical_section park path so that whenever the slow path IS triggered, ours emits the schema event. Defer until first such case manifests.