xenia-rs/audit-runs/phase-c22-payload-canonicalization/investigation.md

Phase C+22 — Payload-field canonicalization for host-heap-derived guest VAs

Date: 2026-05-26 Mode: WRITE — diff-tool only. No engine source changes. Status: LANDED. Main matched-prefix 105,128 → 105,138 (+10).

TL;DR

The pre-C+22 first divergence at canary tid=6 ↔ ours tid=1 idx 105,128 is a thread.create.ctx_ptr mismatch:

canary: thread.create {parent_tid=6, entry_pc=0x824cd458, ctx_ptr=0xbe56bb3c, ...}
ours:   thread.create {parent_tid=1, entry_pc=0x824cd458, ctx_ptr=0x42453b3c, ...}

• parent_tid was ALREADY skipped via SKIP_PAYLOAD_FIELDS_BY_KIND["thread.create"] (line 245 of diff_events.py, in place since C+15-α). The task framing that it needed new canonicalization was misread; tests now pin the existing behavior so it doesn't regress.
• ctx_ptr IS the actual divergence at this index. Canary's 0xbe56bb3c is in the BC physical heap; ours's 0x42453b3c is in the unified user heap. Same AUDIT-043 ε class as C+2's MmAllocatePhysicalMemoryEx.

Why C+2's ALLOCATOR_RETURN_FNS doesn't cover this

C+2 canonicalizes kernel.return.return_value for a known set of host- allocator-returning exports. ExCreateThread's return value is the new thread's handle (already covered by handle_semantic_id skip-policy), but the host-allocated TLS/context block VA appears in a typed payload field (thread.create.ctx_ptr) — a side channel C+2 doesn't see.

The fix

HOST_HEAP_PAYLOAD_FIELDS_BY_KIND map and canonicalize_host_heap_payload_fields helper, exact mirror of ALLOCATOR_RETURN_FNS / canonicalize_allocator_returns, restricted to typed payload fields. Initial set:

HOST_HEAP_PAYLOAD_FIELDS_BY_KIND = {
    "thread.create": ("ctx_ptr",),
}

Sentinel format: <HOSTHEAP_<KIND>_<FIELD>_<ORDINAL>> — distinct namespace from <ALLOC_*_*> so the two passes don't collide.

Strict fields preserved (THE tripstone)

thread.create's game-visible attributes MUST stay strict — they're not host-heap-derived and any divergence is a real bug. Tests verify each:

field	canary	ours	strict?
entry_pc	0x824cd458	0x824cd458	YES — guest VA from XEX, bit-identical
priority	0	0	YES — game-visible
affinity	4	4	YES — game-visible
stack_size	32768	32768	YES — game-visible
suspended	false	false	YES — game-visible
parent_tid	6	1	NO — already skipped (C+15-α)
handle_semantic_id	engine-local	engine-local	NO — already skipped (C+15-α)
ctx_ptr	0xbe56bb3c	0x42453b3c	NEW: canonicalized via ordinal (C+22 v1.7)

5 negative tests in test_diff_events.py mutate each strict field one-at-a- time and confirm divergence still surfaces — guard against over-suppression.

Verification matrix

canary file	pre-C+22 matched	post-C+22 matched	Δ
canary-jitter-1.jsonl (4.4 GB, 476,943 events on tid=6)	105,128	105,138	+10
canary-jitter-2.jsonl (3.5 GB, 441,027 events on tid=6)	105,128	105,138	+10
canary-jitter-3.jsonl (3.7 GB, 445,578 events on tid=6)	105,128	105,138	+10

All three jitter runs advance to the SAME new divergence: idx 105,138, kernel.return VdQueryVideoFlags:

canary: payload.return_value = 3 (status "0x00000003")
ours:   payload.return_value = 0 (status "0x00000000")

This is a genuine Vd subsystem divergence (UNRELATED to canonicalization), out of C+22's scope — surfaces correctly as a real first-divergence.

Tests

8 new tests in test_diff_events.py:

1. test_thread_create_ctx_ptr_in_host_heap_set — registration sanity.
2. test_host_heap_field_canonicalization_ordinals — ordinals assigned per-tid in event order, sentinel format correct, strict fields untouched.
3. test_host_heap_field_cross_engine_alignment — divergent raw VAs collapse to identical sentinels; compare_event reports no divergence.
4. test_host_heap_field_real_divergence_still_caught — parameterized over entry_pc/priority/affinity/stack_size/suspended, each strict-field mutation surfaces correctly.
5. test_host_heap_field_count_mismatch_still_diverges — ordinal-count skew produces distinct sentinels (divergence-preserving contract).
6. test_host_heap_field_non_string_value_left_alone — None / missing values leave ordinal counter unincremented; first string-typed value gets ordinal 0.
7. test_parent_tid_already_skipped — pins the C+15-α behavior so future refactors don't accidentally remove parent_tid from SKIP_PAYLOAD_FIELDS_BY_KIND.
8. (covered in #2) Strict-field preservation as positive assertion.

Total: previous 33 tests + 8 new = 41 tests, all PASS.

Files touched

• xenia-rs/tools/diff-events/diff_events.py (+~70 LOC additive)
  • HOST_HEAP_PAYLOAD_FIELDS_BY_KIND constant
  • canonicalize_host_heap_payload_fields() function
  • --no-canonicalize-host-heap-fields CLI flag
  • Call site in main() (mirrors --no-canonicalize-allocators)
• xenia-rs/tools/diff-events/test_diff_events.py (+~290 LOC tests)
• xenia-rs/audit-runs/phase-a-diff-harness/schema-v1.md (+~110 LOC)
  • New §"Host-heap payload-field canonicalization (v1.7 …)"
  • Updated ctx_ptr row in field-comparison rules table

NO engine source touched. xenia-rs HEAD unchanged. Phase B image_loaded_sha256 ε class boundary unchanged.

Backward compatibility

• Wire format unchanged (schema_version = 1).
• Pre-C+22 event logs whose thread.create.ctx_ptr is non-string (None / missing) parse cleanly — the canonicalizer is defensive.
• Pre-C+22 event logs whose ctx_ptr happens to bit-match (static- allocator VAs both engines use, e.g. 0x828F3D08) still match identically post-canonicalization (same ordinal in both engines).
• --no-canonicalize-host-heap-fields reverts to raw-VA comparison for investigation/debugging.

Cascade

• A (design): PASS — minimal extension of C+2 pattern, no new mechanism class.
• B (implement + test): PASS — 8 new tests, 41 total PASS.
• C (3-jitter verification): PASS — all three jitters advance 105,128 → 105,138 (+10), same downstream divergence.
• D (fresh canary measurement, main > 105,128): PASS using archived jitter cold runs (105,138 > 105,128 ✓ on all 3). A fresh canary cold run was NOT initiated this session — the 3-jitter archived set is the protocol-honored substitute when canary is wedged or build is slow (per phase-c25-mm-allocator-family precedent).

Next divergence (C+23 candidate)

kernel.return VdQueryVideoFlags at idx 105,138:

• canary returns 3 (status 0x00000003)
• ours returns 0 (status 0x00000000)

VdQueryVideoFlags is a Vd-subsystem export that returns a bitmask of video-mode capabilities (HDTV, widescreen, anti-aliasing). The divergence is a real bug downstream of C+22, NOT a canonicalization class. C+23+ scope.