xenia-rs/audit-runs/iterate-2T-wake-requested/writer-report.md

Iterate 2.T — wake.requested instrumentation in wake_eligible_waiters (writer report)

Date: 2026-05-28. LOC delta: engine ~95 (event_log.rs +55, exports.rs ~40 across helpers + 2 in-loop snapshot/emit blocks), tooling +1 (diff_events.py ENGINE_LOCAL_KINDS extension). All retained, cvar-gated default-off via existing event_log::is_enabled(). Tests: 227 kernel + 19 path + 149 cpu + 300 main + ~30 smaller = full suite PASS, 0 regressions. Cascade: N/A — observability class, no semantics changed.

Headline

C-2B-SCHEDULER-PICK-BUG-IDENTIFIED. The kernel-side wake primitive (wake_eligible_waiters) correctly transitions tid=6 from Blocked → Ready EVERY TIME a signal targets it. 5 wake.requested events fire for target_tid=6 on handle 0x000010b4 over [485, 764]ms, all with transitioned=true, new_state="Ready", target_cpu=5. After the 5th wake at 764.29 ms, tid=6 is Ready on CPU5 but never executes again — last tid=6 event remains 663.60 ms — even though the trace continues until 766.86 ms (tids 4, 5, 13 keep emitting events). The exit-thread-state confirms tid=6 finishes in Ready on hw_id=5, priority=0 — sharing CPU5 with tid=10 at priority=15 (and tid=12 at priority=0). The pick_runnable rule in xenia-cpu/src/scheduler.rs:211-218 is strict-priority within a slot: max_by_key(|(i, t)| (t.priority, -(*i as i64))). tid=10's priority=15 deterministically beats tid=6's priority=0. C-2a (kernel wake-call bug) is FALSIFIED; C-2b (scheduler-pick-skipping Ready tid on CPU) is CONFIRMED, with the specific mechanism being strict-priority starvation by a same-CPU peer (tid=10, priority=15).

Mode

Pure observability emitter. No semantic change to wake_eligible_waiters or any caller. Cvar-gated via existing event_log::is_enabled() and implicitly skipped when zero waiters are eligible (the wake loop's existing early-returns happen before the prior-state snapshot). ENGINE_LOCAL in the diff tool — does not affect matched-prefix.

Invocation (identical to 2.Q):

XENIA_CACHE_WIPE=1 timeout 600 ./target/release/xenia-rs exec \
  -n 50000000 --quiet \
  --phase-a-event-log audit-runs/iterate-2T-wake-requested/ours-cold.jsonl \
  "../Project Sylpheed - Arc of Deception (USA, Europe) (En,Ja).iso"

Exit code 0. Output: ours-cold.jsonl (28.7 MB, 121,641 events — 121,569 baseline + 36 signal.match + 36 wake.requested), exit-thread-state.json (9651 bytes, bit-identical to 2.M/2.N/2.Q/2.S), ours-cold.stderr.log (single 2.M emission-notice line), ours-cold.stdout.log (empty — quiet mode).

Patch summary

file	purpose	LOC added
crates/xenia-kernel/src/event_log.rs	new emit_wake_requested(signaling_tid, cycle, target_tid, target_handle, wait_kind, transitioned, new_state, target_cpu) — emits schema-v1 wake.requested event mirroring the signal.match shape from 2.Q	~55
crates/xenia-kernel/src/exports.rs	wake_classify_state(HwState) -> (wait_kind, state_name) + wake_signaling_ctx(state) -> (tid, cycle) + emit_wake_requested_for(state, signaler, cycle, target, handle, prior_kind, prior_name) shim (~40 LOC), plus 2 in-loop snapshot blocks inside wake_eligible_waiters (~14 LOC: manual-fan-out path + auto-winner path) capturing prior state BEFORE set_wake_status_for_waitany + wake_ref and emitting AFTER	~40
crates/xenia-kernel/src/exports.rs	let (signaling_tid, signaling_cycle) = wake_signaling_ctx(state); once at entry to wake_eligible_waiters (captures the signal-call caller's identity for the entire fan-out attribution)	1
tools/diff-events/diff_events.py	ENGINE_LOCAL_KINDS += {"wake.requested"} so the new kind consumes a per-tid idx slot on the emitter side without alignment cost (analog to 2.Q's signal.match addition)	1

Total engine ~95 LOC, tooling +1. Within the 80-150 target / 200 hard cap.

Schema-v1 event emitted:

{"schema_version":1,"engine":"ours","kind":"wake.requested",
 "tid":<signaling tid>,"tid_event_idx":<idx>,"guest_cycle":<cycle>,
 "host_ns":<ns>,"deterministic":true,
 "payload":{"target_tid":<woken tid>,"target_handle":"0x<8hex>",
            "wait_kind":"WaitAny"|"WaitAll"|"WaitSingle"|"Other",
            "transitioned":true|false,
            "new_state":"Ready"|"StillBlocked"|"AlreadyReady"|"Exited"|"ServicingIrq"|"Idle"|"Other",
            "target_cpu":<hw_id>|null}}

Emit policy: exactly ONE event per waiter the kernel touches (the wake loop's early-returns naturally skip zero-eligible-waiter cases per spec). Snapshot of prior state captured immediately BEFORE set_wake_status_for_waitany + wake_ref; post-state read AFTER. transitioned is the strict prior=Blocked && post=Ready predicate.

Test results

cargo build --release  -> OK (1 pre-existing dead_code warning unrelated)
cargo test --release   -> all suites PASS:
  xenia-kernel    227 passed, 0 failed
  xenia-cpu       149 passed, 0 failed
  xenia-app       300 passed, 0 failed
  xenia-path       19 passed, 0 failed
  + 30+ smaller suites, 0 failures total

Baseline event count UNCHANGED at 121,569 (vs 2.J/2.K/2.Q/2.S); signal.match count UNCHANGED at 36 (vs 2.Q/2.S); 36 NEW wake.requested events emitted. Total 121,641 = 121,569 + 36 + 36.

wake.requested summary (all 36 events)

field	distribution
transitioned	true: 36 — every wake successfully transitioned its target from Blocked → Ready
new_state	"Ready": 36 — no StillBlocked / AlreadyReady / etc.
wait_kind	"WaitAny": 36 — all single-WaitAny parks (consistent with NtWaitForSingleObjectEx routing through WaitAny{handles:[h]} per do_wait_single)
signaler tids	tid=5:19, tid=1:9, tid=6:3, tid=2:1, tid=11:1, tid=9:1, tid=8:1, tid=13:1
target tids	tid=5:11, tid=1:9, tid=4:7, tid=6:5, tid=8:2, tid=9:1, tid=10:1

Every signal that targeted a parked waiter produced a successful Blocked→Ready transition. The kernel wake primitive plumbing is sound for this trajectory.

tid=6 deep-dive — the C-2 case

Five wake.requested events for target_tid=6, paired 1:1 with the 5 signal.match events on handle 0x000010b4 (same timestamps to ms precision):

ns (ms)	signaler	handle	transitioned	new_state	target_cpu	tid=6 next event
485.03	tid=5	0x10b4	true	Ready	5	(tid=6 runs through ~520 ms)
485.14	tid=5	0x10b4	true	Ready	5	(back-to-back wake; coalesces)
510.64	tid=5	0x10b4	true	Ready	5	(tid=6 runs through ~660 ms)
659.00	tid=5	0x10b4	true	Ready	5	tid=6 last event @ 663.60 ms
764.29	tid=5	0x10b4	true	Ready	5	NEVER — tid=6 emits no further events; trace continues to 766.86 ms

The first 4 wakes are followed by tid=6 execution. The 5th wake (764.29 ms) transitions tid=6 Blocked→Ready on CPU5 — exactly as expected — but tid=6 never gets picked by the scheduler before the 50M-instruction budget cuts off ~2.6 ms later. Other tids on the trace continue emitting events past 764.29 ms:

tid	last event ns (ms)	vs tid=6 quiescence
1	760.12	predecessor
4	766.84	+103 ms after tid=6 last event; +2.55 ms after tid=6's final wake
5	766.86	+103 ms after; the signaler still alive
13	763.04	+99 ms after

So the trace is NOT truncated; tid=6 specifically is starved post-wake.

CPU5 contention — the C-2b mechanism

Exit-thread-state CPU5 (hw_id=5) Ready set:

tid	state	hw_id	affinity	priority	pc
6	Ready	5	0x20	0	0x824ab214
10	Ready	5	0x20	15	0x824d1404
12	Ready	5	0x20	0	0x824aa6a4
3	Blocked	5	0x20	0	—

Per xenia-cpu/src/scheduler.rs:211-218:

pub fn pick_runnable(&self) -> Option<usize> {
    self.runqueue.iter().enumerate()
        .filter(|(_, t)| matches!(t.state, HwState::Ready | HwState::ServicingIrq(_)))
        .max_by_key(|(i, t)| (t.priority, -(*i as i64)))
        .map(|(i, _)| i)
}

Strict-priority within a slot. tid=10 (priority=15) deterministically beats tid=6 (priority=0) whenever tid=10 is also Ready. tid=10's last trace event is 727.92 ms, but tid=10 is still Ready in the exit state — meaning tid=10 is executing past 727.92 ms in a path that doesn't cross kernel boundaries (no further kernel.call, import.call, etc.), almost certainly a CPU-bound loop. That loop starves tid=6 on CPU5 indefinitely.

This is the C-2b mechanism with concrete identification:

• NOT a wake-call bug (wake.requested fires 5× with transitioned=true, new_state=Ready on tid=6) — falsifies C-2a.
• IS a scheduler-pick-skip but specifically a strict-priority starvation: when a same-slot peer is at priority ≥1, lower-priority Ready threads on that slot never run.

Disambiguation result vs goal-spec

outcome	gate predicate	result	conclusion
C-2A-KERNEL-WAKE-BUG	wake.requested for tid=6 absent OR transitioned=false	NO (5 events, all transitioned)	FALSIFIED
C-2B-SCHEDULER-PICK-BUG	wake.requested fires for tid=6 with transitioned=true, new_state=Ready AND tid=6 still doesn't execute	YES (5 events, last at 764.29 ms; tid=6 last event 663.60 ms; never runs after the 5th wake)	CONFIRMED
NEITHER-CLEAN-NEW-HYPOTHESIS	neither sub-hypothesis matches	NO	NOT TRIGGERED
RUN-FAILED	non-zero exit / crash / hang / no wake.requested events	NO (EXIT=0, 36 events)	NOT TRIGGERED

Result: C-2b CONFIRMED with concrete mechanism — strict-priority starvation on CPU5 by tid=10 (priority=15) blocking tid=6 (priority=0).

Tripstone audit

• #28 (cross-engine tid stability): No cross-engine tid claims. All reported tids are ours-side scheduler tids; stable within this trajectory (verified per-tid counts intact vs 2.Q/2.S baseline).
• #39 (composite progression IS progression): NO progression claim. VdSwap=1, draws=0, render_targets=0 — bit-identical to 2.J/2.K/2.Q/2.N/2.S. The C-2b confirmation is a root-cause identification, not a progression event.
• #40 (single-keystone framing): Care taken. The headline names strict-priority starvation as the SPECIFIC mechanism behind C-2b (not just "the scheduler is buggy"). The data isolates the priority asymmetry (15 vs 0) and same-slot pinning (0x20=CPU5 only). Open follow-ups are not collapsed: why is tid=10 priority=15 (canary parity? game intent?), and why does the strict-priority scheduler not implement starvation-avoidance (intentional simplification or oversight)?
• #41 (categorized diff tags): wake.requested is ENGINE_LOCAL in the diff harness — doesn't affect matched-prefix.
• #42 (Phase-A blind to blocked-forever waits): Used the always-on exit-thread-state.json to confirm tid=6's final Ready state post-trace-end (Phase-A alone would have shown only the 5 wakes without the final unscheduled outcome).
• #43 (no budget-cap framing): 2.S already established the 50M budget reproduces the phenomenon; this iterate uses 50M and confirms the C-2b mechanism is structural and reproducible in the smaller budget. The 2.6 ms gap between the 5th wake (764.29) and tid=5/4's final events (~766.86) is sufficient to show the scheduler had ample opportunity to pick tid=6 and didn't.

Comparison: 2.K → 2.Q → 2.S → 2.T

gate	2.K (500M, baseline)	2.Q (50M + signal.match)	2.S (500M + signal.match)	2.T (50M + signal.match + wake.requested)
total events	121,569	121,605	121,605	121,641
baseline events	121,569	121,569	121,569	121,569
signal.match events	n/a	36	36	36
wake.requested events	n/a	n/a	n/a	36
wake.requested for tid=6 (transitioned=true)	n/a	n/a	n/a	5
exit-state size	9651	9651	9651	9651
tid=6 final state	Ready	Ready	Ready	Ready
Termination	budget hit	(50M)	budget hit (500M)	budget hit (50M)

Baseline + signal.match counts bit-identical to 2.Q/2.S. New 36 wake.requested events 1:1 with the 36 signal.match events (every successful signal that found a waiter resulted in a wake call that transitioned the waiter — sanity check of the kernel wake plumbing across all signaler tids and target handles).

Confidence

• HIGH that the patch is correct and observability-only: 0 test regressions; wake_eligible_waiters body unchanged except for the snapshot reads and post-call emits.
• HIGH that wake.requested events fire as designed: 36 events emitted, all with transitioned=true && new_state=Ready (which matches the expected branch in the wake primitive — every waker pulled off a non-empty waiter queue should transition Blocked→Ready).
• HIGH that tid=6 is woken 5× by tid=5 on handle 0x10b4 with successful transition — grepped exhaustively by exact handle string
  • target_tid filter.
• HIGH that tid=6 emits no events after 663.60 ms despite the 5th wake at 764.29 ms (per-tid scan).
• HIGH that exit-state thread geometry matches 2.M/2.N/2.Q/2.S bit-identically (9651 bytes; tid=6 final Ready/hw_id=5/affinity=0x20).
• HIGH that C-2a (kernel wake-call bug) is FALSIFIED: the wake-call IS issued AND IS transitioning tid=6 Blocked→Ready correctly.
• HIGH that C-2b (scheduler-pick-skip) is CONFIRMED with strict- priority starvation as the specific mechanism, given CPU5's exit- state Ready set has tid=10 at priority=15 (vs tid=6 at priority=0) pinned to affinity 0x20.
• MEDIUM-HIGH that tid=10 is in a CPU-bound non-kernel-crossing loop after 727.92 ms (inferred from "still Ready at exit" + "no trace events after 727.92 ms"). Could also be a tight kernel.call loop where the trace happens to not have a corresponding emit — but on CPU5 with priority advantage, tid=10 keeps the slot.

Next-iterate recommendation

Three complementary directions, priority order:

(1) 2.U — investigate why tid=10 is priority=15 / what tid=10 is doing post-727.92ms (~0 LOC observability OR ~50 LOC PC-trace + KeSetBasePriorityThread hook). Pull from existing audit DB (sylpheed.db) the basic block containing PC 0x824d1404 (tid=10's final PC) — find the loop body. Compare with canary's analog tid to see whether canary's same-PC thread also runs at priority=15. If canary runs the same code at a different priority OR has a yield/sleep in the loop, that's the divergence point. If canary is identical, then the problem is ours's scheduler lacks fair preemption that canary's host-OS-thread scheduler provides naturally (canary multiplexes guest threads onto OS threads, getting OS-level fairness for free; ours's cooperative-pick-runnable model needs explicit anti-starvation).

(2) 2.V — add starvation-avoidance to pick_runnable (~30-80 LOC engine, semantic change). E.g., dynamic priority aging or a quantum-expiration override that occasionally picks the highest-priority not-recently-run thread. Risk: changes scheduling semantics — matched-prefix and Phase-A trace counts will shift; needs careful parity check vs canary. NOT recommended as next without first establishing canary baseline (2.U).

(3) 2.W — canary-side wake.requested mirror (~30-60 LOC C++) for cross-engine parity diff. Lower urgency than 2.U because the ours-side data alone is sufficient to identify the scheduler-pick mechanism; canary parity becomes relevant only if 2.U finds tid=10 is intentionally high-priority and canary handles it via OS-level fairness.

Recommended: 2.U first (~30 min DB lookup + cross-engine check), then 2.V or 2.W depending on what 2.U reveals.

Artifacts

Under xenia-rs/audit-runs/iterate-2T-wake-requested/:

• ours-cold.jsonl (28.7 MB, 121,641 events = 121,569 baseline + 36 signal.match + 36 wake.requested)
• ours-cold.stdout.log (empty — quiet mode)
• ours-cold.stderr.log (single 2.M emission-notice line)
• exit-thread-state.json (9651 bytes; bit-identical to 2.M/2.N/2.Q/2.S — 13 threads + 10 wedge entries)
• writer-report.md (this file)

Engine HEAD e6d43a23ac393004d2e5adf2f0395fd0b5e6448b + uncommitted 2.Q signal.match patch + this iterate's 2.T wake.requested patch. xenia-canary UNCHANGED.