The instrument behind the F1 menu-repeat measurement. It was uncommitted for
a day while the numbers it produced were already shipping in the port.
The file driver deliberately emits exactly one event per press -- repeat is
what makes scripted menu steps overshoot -- so it structurally could not show
whether the GAME repeats a held direction. Measured through it, a held (down)
moved the cursor once and never again, at any hold length. That is a property
of the driver, not of the game, and the page that recorded it said so.
`--pad_file_repeat` opts in, off by default, so no existing script changes.
It reuses the SDL driver's Waiting/Repeating state machine and its constants
VERBATIM (HID_SDL_REPEAT_DELAY=400, HID_SDL_REPEAT_RATE=100, guest-time ms)
rather than re-deriving them -- the point is to be a fair stand-in for a real
controller, and a re-derived constant would only measure our own arithmetic.
With it, the cursor cycled the whole 5-item menu for as long as the button
was held: 19 distinct positions, 12 frames to the first repeat, 4 frames
between the rest, at 29.87 fps guest.
⚠️ The 4-frame interval is NOT the 100ms constant that drives it (100ms is
~3 frames). The game consumes drained keystrokes at its own per-frame pace.
The 400ms delay, by contrast, comes back as 402ms -- that confirms the
instrument, not the game.
docs/re/f1-repeat-measured-via-driver-patch.md in the Sylpheed repo.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A base address cannot distinguish 'the guest decoded a new frame' from 'the
guest rotated to the next buffer of a triple-buffered set' -- a rotating
buffer visits the same three addresses either way. Reading a clean
one-base-change-per-present as one decode per present is how
docs/re/guest-frame-rate-measured.md reached a conclusion it had to
withdraw.
h=<FNV-1a over 4096 sampled bytes>, omitted rather than faked when the
address does not translate, so a missing hash cannot read as a matching
one. Sampled rather than full: a 1280x720 plane is 900 KB and hashing all
of it per draw would change the thing being measured.
The UI draw log recorded blend state, textures and geometry, which is
enough to say WHICH sprite a draw is and how it composites, and not
enough to answer whether a screen has more than one PASS. A second draw
into an off-screen target, or a resolve of the EDRAM into a texture that
a later full-screen quad samples, both look like just another quad in
the old format.
Three additions, all read straight out of the register file:
mode= RB_MODECONTROL.edram_mode. kCopy (6) is how this GPU issues
a RESOLVE, and it arrives through the same DRAW_INDX packet
as a sprite -- so without this field a resolve was
indistinguishable from a draw.
rt0=/pitch RB_COLOR_INFO's EDRAM tile and format, RB_SURFACE_INFO's
pitch and MSAA. A second target shows up as a different
tile; a half-resolution post-process shows up as a pitch
that is not the screen's.
RESOLVE on a kCopy draw, RB_COPY_CONTROL and RB_COPY_DEST_BASE.
That destination reappearing as a later tex[base=...] is
what 'resolve-and-resample' means stated in addresses,
rather than inferred from the picture.
ps_c[...] the pixel shader's float constants, taken off its OWN
float_bitmap -- the same one the backend uploads from -- so
this is the shader's declared dependency set rather than a
fixed window that could miss the one that matters. Pixel
constants live at SHADER_CONSTANT_256_X and the c# printed
is the index the shader's disassembly uses.
Without the constants, an alpha ramp driven by a shader constant and one
driven by per-vertex colour are the same picture.
8 vertices is TWO QUADS. A batched UI draw carries more: on Project Sylpheed's
EXTRAS screen one 24-index draw holds six sprites, and truncating at 8 reported
the first two while ptframe4, pteff21, pteff22 and pteff23 looked like elements
the game never draws at all. A cap that hides geometry is worse than a long
line, because the missing rows do not announce themselves.
CaptureUiDrawForRE now records RB_BLENDCONTROL0, RB_COLORCONTROL and
RB_COLOR_MASK per draw, raw alongside the decoded src/op/dst fields so a decode
bug here cannot quietly become the answer.
The question it answers: the Godot port composites every UI element with
straight alpha-over and four elements come out too dark against the capture,
with the shortfall correlating with the background. Nothing on the disc selects
a per-element mode, so this reads what the GPU was actually told. Result: the
title-side UI uses two states and one pixel shader -- 0x07010701 (src ONE, dst
1-SRC_ALPHA) for backgrounds, text and buttons, and 0x01010101 (src ONE, dst
ONE, ADDITIVE) for the frame sprites and the rotated sweep strips.
A SILENT semantic conflict from merging auto/re-kernel-pages-probe: git merged
every file cleanly and the result did not compile. The RE aid added by
043002a87 prints the dialog counters
xam_dialogs_shown_ / xam_nui_dialogs_shown_
which upstream has since collapsed into a single is_xam_dialog_present_ flag,
reachable only through IsUIActive(). The counts no longer exist, so report the
state that does -- which is what the message is actually about.
Worth noting for the next merge of this fork: a clean `git merge` is not
evidence that instrumentation still matches the API it reads. Build it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The v1 streak counter caught nothing during a real freeze: the healthy-run
baseline and not one new (thread, object) pair, while the process burned 1255
ticks per 10 s with two guest threads sitting in KeWaitForSingleObject. That
refuted "loops on timeouts against one object" and left two blind spots, and this
covers both.
A per-thread one-second window counts EVERY call, tracks how many DISTINCT
objects it saw, and logs the last object and the last RESULT when the rate passes
500/s. So a thread rotating over several handles (which resets a same-object
streak) and a thread whose waits SUCCEED rather than time out (which a timeout
counter cannot see) both show up now.
Self-selecting like v1: the healthy poller runs at ~33 calls/s on a 30 ms
timeout, so the 500/s floor stays silent on a good run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
Project Sylpheed freezes mid-mission about half the time. Under gdb every one of
the emulator's 79 threads is in a WAIT, yet the process still burns 1253 ticks
per 10 s - 280 of them in guest threads the backtrace shows blocked inside
KeWaitForSingleObject. So they are cycling: a timed wait that expires and is
re-entered on an object nobody signals. Naming that object is the next step.
Logging it the ordinary way is not possible. KeWaitForSingleObject is
kHighFrequency, so it is silent unless --log_high_frequency_kernel_calls=true,
and measured: that flag writes 175 MB and leaves the emulator seventeen minutes
into a boot with the screen still black.
So count CONSECUTIVE timeouts on the SAME object, per thread, in
xeKeWaitForSingleObject, and log at 100 and then every 500. It is self-selecting:
a wait that is being satisfied never builds a streak.
Measured on a healthy 25-minute Stage 02 run: 27 lines, all one thread
(F800004C) polling one Event (guest VA BE56BB5C) with a ~30 ms timeout - a
legitimate poller, and the baseline a frozen run has to be compared against. Not
"silent", as first drafted, but quiet enough to leave on.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PMRJjbxLqZtsb5Vb7KunPE
A thread created suspended publishes its state and its suspend count in TWO
separate lock scopes:
{ lock; state_ = kSuspended; notify_all(); } // lock released here
if (create_suspended) { lock; suspend_count_ = 1; wait(count == 0); }
and Resume() does WaitStarted() - which waits only for state_ != kUninitialized -
followed by `if (suspend_count_ == 0) return false;`. So a resumer can slip into
the gap: it sees the thread started, sees suspend_count_ still 0, drops the
resume and returns false. The new thread then sets the count to 1 and waits on it
forever. A textbook lost wakeup.
Measured in Project Sylpheed. Pressing (A) on the title makes the game do
XamUserGetXUID -> NtCreateEvent -> ExCreateThread(entry=821748F0,
CREATE_SUSPENDED) -> NtResumeThread, and the loader thread then never ran: zero
kernel calls of its own (it appeared in the log only as an argument) and 00:00:00
host CPU time, while the emulator sat at 546% CPU. Boots reached the main menu
1 time in 6.
Fixed by publishing state_ and suspend_count_ under one lock and waiting without
releasing it, so a resumer past WaitStarted() always observes 1.
On the first clean boot after the fix the same loader thread is the CALLER on 20
kernel-call lines and issues 4 ResolvePath asset reads. Every failed boot before
it had exactly zero of both.
Also logs when the host resume is refused. XThread::Resume's Linux path
discarded that bool - the Windows path turns it into X_STATUS_UNSUCCESSFUL - so a
dropped resume was invisible from both sides. Note the log is not by itself a
defect: resuming a thread that is not suspended legitimately returns false, and
it fires ~7 times in a normal boot.
(cherry picked from commit a60fe7d11c)
The pad looked correct and did nothing. Its own log showed A arriving, the
emulator sat on "PRESS (A) BUTTON", and the title never advanced.
Cause: 360 front-ends poll XamInputGetKeystrokeEx, not XamInputGetState. This
title imports both, and its menus use the keystroke path; the driver returned
X_ERROR_EMPTY there, so every scripted press went into the void while GetState
faithfully reported a button nobody asked about.
Implement it edge-triggered, one event per call: KEYUPs for everything released
first, then KEYDOWNs, matching the SDL driver's ordering (so a thumb transition
clears before it sets). Deliberately NO auto-repeat -- scripted input wants
exactly one event per press, and repeat is precisely what makes menu steps
overshoot. Bits without a virtual key (guide, unused) are swallowed rather than
re-offered forever.
Verified on the real game: title -> main menu -> EXTRAS driven entirely from the
pad file, with `[file-pad] keystroke vk=5800 down/up` in the log for each press.
(cherry picked from commit 15fe11d5d9)
Two things that only show up once you actually script this pad.
st_mtime is whole seconds. Combined with size it looked like enough and is not:
a script stepping a menu writes several same-length states per second
(`press=A` then `press=B`, both 8 bytes), and every one after the first was
silently dropped -- the emulator simply did not react, with nothing in any log
to say why. Compare st_mtim.tv_nsec as well, and track whether the file existed
at all so a delete is registered once rather than every frame.
Also log one line per state change (not per frame, so it stays quiet). Driving
the emulator headless means there is nothing to watch; this line is the only
proof that a scripted press was picked up, which turns "did my input land?" from
a guess into a grep.
(cherry picked from commit e3e17e4951)
The scripted-input tool this project uses for reverse engineering created its
pad through /dev/uinput. Input devices are NOT namespaced by the kernel, so a
uinput device created inside a container registers with the HOST's input stack:
every trigger hold and button press is delivered to whatever on the host reads
gamepads, not just to the emulator. That was noticed the hard way, and it made
every runtime experiment -- booting, menu navigation, unit harvesting, flight
measurement -- unusable from inside the box.
This driver takes the kernel out of the loop. Pad state lives in an ordinary
text file only the container can see; GetState re-reads it when it changes.
Nothing is registered with the host and no X server is involved. A bonus for RE:
analogue values are exact rather than whatever a virtual stick quantises to.
press=A,START buttons by name, comma separated
buttons=0x1010 or the raw XINPUT mask
lt=0 rt=255 triggers, 0..255
lx=0 ly=0 thumbs, -32768..32767
Absent keys are neutral, so `press=A` alone is a valid file, and a missing or
empty file means no input -- the safe default if it is deleted mid-run.
Selected with --hid=file, path from --pad_file (default /tmp/xenia_pad.txt).
Deliberately NOT part of "any": this pad has to be asked for. Header-only, so it
adds no build target and no cost to anyone not using it.
(cherry picked from commit d15c8cfab6)
A thread created suspended publishes its state and its suspend count in TWO
separate lock scopes:
{ lock; state_ = kSuspended; notify_all(); } // lock released here
if (create_suspended) { lock; suspend_count_ = 1; wait(count == 0); }
and Resume() does WaitStarted() - which waits only for state_ != kUninitialized -
followed by `if (suspend_count_ == 0) return false;`. So a resumer can slip into
the gap: it sees the thread started, sees suspend_count_ still 0, drops the
resume and returns false. The new thread then sets the count to 1 and waits on it
forever. A textbook lost wakeup.
Measured in Project Sylpheed. Pressing (A) on the title makes the game do
XamUserGetXUID -> NtCreateEvent -> ExCreateThread(entry=821748F0,
CREATE_SUSPENDED) -> NtResumeThread, and the loader thread then never ran: zero
kernel calls of its own (it appeared in the log only as an argument) and 00:00:00
host CPU time, while the emulator sat at 546% CPU. Boots reached the main menu
1 time in 6.
Fixed by publishing state_ and suspend_count_ under one lock and waiting without
releasing it, so a resumer past WaitStarted() always observes 1.
On the first clean boot after the fix the same loader thread is the CALLER on 20
kernel-call lines and issues 4 ResolvePath asset reads. Every failed boot before
it had exactly zero of both.
Also logs when the host resume is refused. XThread::Resume's Linux path
discarded that bool - the Windows path turns it into X_STATUS_UNSUCCESSFUL - so a
dropped resume was invisible from both sides. Note the log is not by itself a
defect: resuming a thread that is not suspended legitimately returns false, and
it fires ~7 times in a normal boot.
X_ERROR_EMPTY is 0x10D2 and XSUCCEEDED is ((s & 0xC0000000) == 0), so an empty
keystroke poll passes it. The first run of this instrumentation therefore logged
every empty poll: 6499 lines of vk=0000 burying the two that mattered.
The pre-existing assignment above shares the quirk - it writes user_index_ptr on
an empty poll too. Left alone: it is upstream's and harmless, since the struct
is zeroed first.
The measurement itself survives, and is worth stating: at the title screen the
game polls constantly and receives EXACTLY the two events sent - vk=5800
flags=0001 (KEYDOWN) and flags=0002 (KEYUP). Input delivery is not the problem.
The existing [RE-INPUT] line only fires when IsUIActive() swallows a keystroke,
so its silence is ambiguous: it means either 'the game is polling and getting
nothing' or 'the game is not polling at all'. That is exactly the fork the
title-screen investigation is stuck on - with the sign-in dialog fixed the
swallow count is 0 and the title still does not respond to (A).
Adds two logs on the other side: a rate-limited count of calls that get past
IsUIActive to a driver, and one line per keystroke actually handed to the guest
(rare by construction - one per physical press - so every one is logged, with
user index, virtual key and flags).
A UI quad carries a k_8_8_8_8 colour next to its position, and its alpha is the
fade the bundle's keyframes animate. Logging it costs one more read per vertex
and turns the capture from "where is this quad" into "where is it and how faded".
It was added to separate two elements that decode to the same 1133x280 and sit on
opposite sides of a third in the declaration table. It does NOT separate them —
both rest at 255 — but it does check the fade decode against the running game for
the first time: every static element draws at exactly the resting alpha the
bundle predicts, and the only two quads whose alpha changes between consecutive
frames are the rotating effect pair and the PRESS (A) glow, which are the two
things visibly animating.
The capture used to require --log_ui_draws at LAUNCH, which put an unexplained
variable into every navigation run: across 12 runs, launching with the flag
correlates with the title screen refusing (A) — 0 of 7 with it, 4 of 5 without.
No mechanism was found. The cvar is read in exactly one place, when F10 arms a
capture, and F10 was never pressed in those runs; the per-draw hook is a single
relaxed atomic load; the startup config dumps of the two arms are byte-identical.
An interleaved A/B also ruled out the obvious confound (boot duration): the
latest title of all, 268 s, ACCEPTED (A), while a 232 s title refused.
So rather than keep a variable nobody can explain in the path of every run, arm
on F10 the way the ship capture next to it already does. The cvar stays, marked
obsolete, so existing command lines still parse. Verified: F10 with no capture
flags at all writes xenia_re_ui_draws_01.log.
XamInputGetKeystrokeEx returns X_ERROR_SUCCESS with a zeroed keystroke whenever
any Xenia dialog is up, before consulting a driver. That is correct behaviour and
an invisible one: from outside, scripted input simply stops working, the pad
driver logs nothing because it is never asked, and the guest keeps polling. It is
easy to enter that state by accident — F10 is both the RE capture hotkey and the
toolkit's menu-bar key, and a dialog closed by its own [x] rather than by the
menu toggle leaves the counter incremented.
So log it, rate-limited to one line per 600 swallowed calls, with the dialog
counters. On the runs this was written to diagnose it stays silent, which is what
ruled the theory out — a diagnostic that is useful when it does not fire.
The title screen's (A) opens the Sign In dialog when no profile exists, and the
game goes no further. That dialog cannot be completed from a scripted container:
"Create Profile" asks for a gamertag in an ImGui text field, and synthetic X key
events do not reach it — tried with the window focused, via XTEST and via
--window, char by char, after clicking the field. Mouse clicks work; text entry
does not.
So: when no profile exists on disk and this cvar is set, create one at startup
and sign it in. It uses ProfileManager's fixed `default_xuid`
(B13EBABEBABEBABE), which is what makes it usable from a script — a later run
passes --logged_profile_slot_0_xuid=B13EBABEBABEBABE and is deterministic.
Ignored when a profile already exists, so it is safe to leave in a launcher.
Verified: "RE bootstrap profile 'SylphRE' -> created", then on the next boot
"Found 1 Profiles" / "Loaded SylphRE (GUID: B13EBABEBABEBABE) to slot 0".
This does NOT get the game past the title — see the Reborn repo's
docs/re/canary-scripted-input-traps.md for what does and does not, and for the
content-path crash that is the actual blocker.
Three changes, each paid for by a measurement that could not be made without it:
* `--ui_draw_capture_frames` / `--ui_draw_capture_max` replace the compiled-in
3-frame, 20k-draw bounds. A screen that redraws every frame needs 3; finding
the frames in which a screen is BUILT needs hundreds, and that answer (the
title screen never rebuilds — it submits the same 11 draws every frame) is
only reachable by turning the window up.
* the vertex dump prints all three floats. Attribute 0 of a UI quad is
k_32_32_32_FLOAT, so the stream carries a Z — the game's own layer key, if it
had one. It does not: every Z is 0.00000, which is what makes submission order
the whole of the paint order.
* positions print as floats rather than raw words, now that the format is known.
Neither existing RE hook can say in what order the game paints a UI screen:
`log_draws` de-dups by vertex-declaration fingerprint, so a screen's sprites —
which share a declaration — collapse into one record; and the F10 ship capture
returns early on any draw without an f32x3 position stream, which is every UI
quad.
So: `--log_ui_draws` arms (on F10, alongside the ship capture) a snapshot of the
next `--ui_draw_capture_frames` submitted frames, undeduplicated, writing every
draw with its primitive type, index count, VS/PS hashes, each bound texture's
base and dimensions, and the quad's vertex positions. The positions are the part
that matters: a screen's sprites all share one shader and sample big texture
pages, so the geometry is what names an element.
Frames are counted here rather than from `counter_`. `counter_` looks like a
frame number — the VdSwap packet increments it — but MarkVblank() increments it
too, from the vblank thread, so it advances between two draws of the SAME frame.
Bounded by it, the first capture ended after one draw having "covered 6 frames".
Measured on the title screen: 11 draws a frame, every frame.
The offset was being multiplied by the scale and divided by the host scaled size, which collapsed back to guest step, so the cvar changed nothing on Vulkan while D3D12 stepped host texels. Since the size is already in host texels, dividing the offset by it gives the proper step.
Unnormalized coordinates convert to host texels before the offset add so the offset isn't multiplied along with the coordinate. Folds in the fix for 5841095A.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Strips and fans normalize to triangle list host draws instead of being rejected. The conversion buffer is built at runtime when the backend did not bake one at init.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Co-authored-by: Reality <reality@xenios.jp>
They were picked whenever front faces were culled, even with both faces culled. Matches D3D12.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Words at or past the size in the fetch constant read as 0 now instead of whatever sits in shared memory, matching real hardware.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Guest sample 1 lives in host sample 3 when 2x-as-4x. This might have originally just been a typo or oversight.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Mainly for NaN preservation. GS discards primitives by checking positions for NaN, mostly for vertex kill, and without the controls drivers might fold those checks away.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Mips were always regenerated by blitting down from base, even when guest resolved real data into scaled memory.
The upload footprint is the guest mip reduced then scaled while the subresource is the base scaled then reduced, so the deepest mips of scaled textures can disagree by a row or column per axis. Compressed formats round up to the block and absorb it, uncompressed copies now clamp per axis so they never overrun the host image.
Matches D3D12 fix for the same problem.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Exp 31 is a large finite value up to 131,008 on the guest, not Inf or NaN.
Packing used to clamp colors at 65504 and unpacking returned Inf for extended encodings.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
The domain shader reads the patch index from the hull shader output instead of gl_PrimitiveID, which bypassed the endian swap, offset, wrap and clamp already applied upstream.
The tessellator winds clockwise now. Clip space Y is not flipped on Vulkan, so counterclockwise winding inverted the facing and guest backface culling removed whole surfaces.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
This is a combination of three different Edge commits.
Guest page access is resolved with system page granularity so that anything deciding on protection now takes permissive access of every guest page a system page covers, which matters when the host is larger than the guest.
Access violations: Write faults on pages with no watch armed are only reported handled if guest mapping allows the write.
Invalidation of unwatched ranges when made writable or freed: Decommit, Release and Protect-to-writable (including write-combine) raise invalidation callbacks even with no watch armed.
Co-authored-by: Herman S. <429230+has207@users.noreply.github.com>
Steps 2 and 3 (spirv-opt, spirv-dis) run subprocess without text=True, so
result.stderr is bytes -- and `sys.stderr.write(bytes)` raises TypeError. The
tool's real message is replaced by a Python traceback at exactly the moment you
need it.
Building in a clean container, the visible failure was:
ERROR: spirv-opt failed for guest_output_bilinear.ps.xesl
TypeError: write() argument must be str, not bytes
with the actual cause -- `Unknown flag '--canonicalize-ids'`, i.e. a SPIRV-Tools
too old -- never printed. Use .buffer.write, as compile_shader_dxbc.py already
does at the same site.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Taking phase_b_snapshot.{cc,h} as files from the snapshot branch left it
referencing three cvars that live in that branch's cpu_flags and were not
carried across, so the kernel library failed to compile:
phase_b_snapshot_dir / phase_b_snapshot_and_exit / phase_b_dump_section_content
Ported their DEFINE_/DECLARE_ pair verbatim. Also switched one
`std::filesystem::path base(...)` to brace init -- with a single named argument
that parses as a function declaration under a newer clang than the branch was
written against, and this tree builds with -Werror.
Verified on the linked binary: all fifteen instrumentation cvars from the three
merged lineages are present, and so is every output path the superset draw
logger emits (xenia_re_draws.log, xenia_re_shaders.log, xenia_re_files.log,
xenia_ship_capture_NN.log, with idx_raw / vsconst / psconst records).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
