Add OPCODE_NEGATED_MUL_ADD/OPCODE_NEGATED_MUL_SUB

Proper handling of nans for VMX max/min on x64 (minps/maxps has special behavior depending on the operand order that vmx does not have for vminfp/vmaxfp)
Add extremely unintrusive guest code profiler utilizing KUSER_SHARED systemtime. This profiler is disabled on platforms other than windows, and on windows is disabled by default by a cvar
Repurpose GUEST_SCRATCH64 stack offset to instead be for storing guest function profile times, define GUEST_SCRATCH as 0 instead, since thats already meant to be a scratch area
Fix xenia silently closing on config errors/other fatal errors by setting has_console_attached_'s default to false
Add alternative code path for guest clock that uses kusershared systemtime instead of QueryPerformanceCounter. This is way faster and I have tested it and found it to be working, but i have disabled it because i do not know how well it works on wine or on processors other than mine
Significantly reduce log spam by setting XELOGAPU and XELOGGPU to be LogLevel::Debug
Changed some LOGI to LOGD in places to reduce log spam
Mark VdSwap as kHighFrequency, it was spamming up logs
Make logging calls less intrusive for the caller by forcing the test of log level inline and moving the format/AppendLogLine stuff to an outlined cold function
Add swcache namespace for software cache operations like prefetches, streaming stores and streaming loads.
Add XE_MSVC_REORDER_BARRIER for preventing msvc from propagating a value too close to its store or from its load
Add xe_unlikely_mutex for locks we know have very little contention
add XE_HOST_CACHE_LINE_SIZE and XE_RESTRICT to platform.h
Microoptimization: Changed most uses of size_t to ring_size_t in RingBuffer, this reduces the size of the inlined ringbuffer operations slightly by eliminating rex prefixes, depending on register allocation
Add BeginPrefetchedRead to ringbuffer, which prefetches the second range if there is one according to the provided PrefetchTag
added inline_loadclock cvar, which will directly use the value of the guest clock from clock.cc in jitted guest code. off by default
change uses of GUEST_SCRATCH64 to GUEST_SCRATCH
Add fast vectorized xenos_half_to_float/xenos_float_to_half (currently resides in x64_seq_vector, move to gpu code maybe at some point)
Add fast x64 codegen for PackFloat16_4/UnpackFloat16_4. Same code can be used for Float16_2 in future commit. This should speed up some games that use these functions heavily
Remove cvar for toggling old float16 behavior
Add VRSAVE register, support mfspr/mtspr vrsave
Add cvar for toggling off codegen for trap instructions and set it to true by default.
Add specialized methods to CommandProcessor: WriteRegistersFromMem, WriteRegisterRangeFromRing, and WriteOneRegisterFromRing. These reduce the overall cost of WriteRegister
Use a fixed size vmem vector for upload ranges, realloc/memsetting on resize  in the inner loop of requestranges was showing up on the profiler (the search in requestranges itself needs work)
Rename fixed_vmem_vector to better fit xenia's naming convention
Only log unknown register writes in WriteRegister if DEBUG :/. We're stuck on MSVC with c++17 so we have no way of influencing the branch ordering for that function without profile guided optimization
Remove binding stride assert in shader_translator.cc, triangle told me its leftover ogl stuff
Mark xe::FatalError as noreturn
If a controller is not connected, delay by 1.1 seconds before checking if it has been reconnected. Asking Xinput about a controller slot that is unused is extremely slow, and XinputGetState/SetState were taking up
an enormous amount of time in profiles. this may have caused a bit of input lag
Protect accesses to input_system with a lock
Add proper handling for user_index>= 4 in XamInputGetState/SetState, properly return zeroed state in GetState
Add missing argument to NtQueryVirtualMemory_entry
Fixed RtlCompareMemoryUlong_entry, it actually does not care if the source is misaligned, and for length it aligns down
Fixed RtlUpperChar and RtlLowerChar, added a table that has their correct return values precomputed
This commit is contained in:
chss95cs@gmail.com
2022-08-20 11:40:19 -07:00
parent 7cc364dcb8
commit 457296850e
54 changed files with 1441 additions and 615 deletions

View File

@@ -10,7 +10,7 @@
#include "xenia/cpu/backend/x64/x64_backend.h"
#include <stddef.h>
#include <algorithm>
#include "third_party/capstone/include/capstone/capstone.h"
#include "third_party/capstone/include/capstone/x86.h"
@@ -50,6 +50,9 @@ DEFINE_bool(record_mmio_access_exceptions, true,
"for them. This info can then be used on a subsequent run to "
"instruct the recompiler to emit checks",
"CPU");
#if XE_X64_PROFILER_AVAILABLE == 1
DECLARE_bool(instrument_call_times);
#endif
namespace xe {
namespace cpu {
@@ -96,6 +99,68 @@ static void ForwardMMIOAccessForRecording(void* context, void* hostaddr) {
reinterpret_cast<X64Backend*>(context)
->RecordMMIOExceptionForGuestInstruction(hostaddr);
}
#if XE_X64_PROFILER_AVAILABLE == 1
// todo: better way of passing to atexit. maybe do in destructor instead?
// nope, destructor is never called
static GuestProfilerData* backend_profiler_data = nullptr;
static uint64_t nanosecond_lifetime_start = 0;
static void WriteGuestProfilerData() {
if (cvars::instrument_call_times) {
uint64_t end = Clock::QueryHostSystemTime();
uint64_t total = end - nanosecond_lifetime_start;
double totaltime_divisor = static_cast<double>(total);
FILE* output_file = nullptr;
std::vector<std::pair<uint32_t, uint64_t>> unsorted_profile{};
for (auto&& entry : *backend_profiler_data) {
if (entry.second) { // skip times of 0
unsorted_profile.emplace_back(entry.first, entry.second);
}
}
std::sort(unsorted_profile.begin(), unsorted_profile.end(),
[](auto& x, auto& y) { return x.second < y.second; });
fopen_s(&output_file, "profile_times.txt", "w");
FILE* idapy_file = nullptr;
fopen_s(&idapy_file, "profile_print_times.py", "w");
for (auto&& sorted_entry : unsorted_profile) {
// double time_in_seconds =
// static_cast<double>(sorted_entry.second) / 10000000.0;
double time_in_milliseconds =
static_cast<double>(sorted_entry.second) / (10000000.0 / 1000.0);
double slice = static_cast<double>(sorted_entry.second) /
static_cast<double>(totaltime_divisor);
fprintf(output_file,
"%X took %.20f milliseconds, totaltime slice percentage %.20f \n",
sorted_entry.first, time_in_milliseconds, slice);
fprintf(idapy_file,
"print(get_name(0x%X) + ' took %.20f ms, %.20f percent')\n",
sorted_entry.first, time_in_milliseconds, slice);
}
fclose(output_file);
fclose(idapy_file);
}
}
static void GuestProfilerUpdateThreadProc() {
nanosecond_lifetime_start = Clock::QueryHostSystemTime();
do {
xe::threading::Sleep(std::chrono::seconds(30));
WriteGuestProfilerData();
} while (true);
}
static std::unique_ptr<xe::threading::Thread> g_profiler_update_thread{};
#endif
bool X64Backend::Initialize(Processor* processor) {
if (!Backend::Initialize(processor)) {
@@ -159,6 +224,21 @@ bool X64Backend::Initialize(Processor* processor) {
processor->memory()->SetMMIOExceptionRecordingCallback(
ForwardMMIOAccessForRecording, (void*)this);
#if XE_X64_PROFILER_AVAILABLE == 1
if (cvars::instrument_call_times) {
backend_profiler_data = &profiler_data_;
xe::threading::Thread::CreationParameters slimparams;
slimparams.create_suspended = false;
slimparams.initial_priority = xe::threading::ThreadPriority::kLowest;
slimparams.stack_size = 65536 * 4;
g_profiler_update_thread = std::move(xe::threading::Thread::Create(
slimparams, GuestProfilerUpdateThreadProc));
}
#endif
return true;
}
@@ -734,6 +814,7 @@ void X64Backend::InitializeBackendContext(void* ctx) {
bctx->flags = 0;
// https://media.discordapp.net/attachments/440280035056943104/1000765256643125308/unknown.png
bctx->Ox1000 = 0x1000;
bctx->guest_tick_count = Clock::GetGuestTickCountPointer();
}
const uint32_t mxcsr_table[8] = {
0x1F80, 0x7F80, 0x5F80, 0x3F80, 0x9F80, 0xFF80, 0xDF80, 0xBF80,
@@ -747,6 +828,23 @@ void X64Backend::SetGuestRoundingMode(void* ctx, unsigned int mode) {
bctx->mxcsr_fpu = mxcsr_table[control];
((ppc::PPCContext*)ctx)->fpscr.bits.rn = control;
}
#if XE_X64_PROFILER_AVAILABLE == 1
uint64_t* X64Backend::GetProfilerRecordForFunction(uint32_t guest_address) {
// who knows, we might want to compile different versions of a function one
// day
auto entry = profiler_data_.find(guest_address);
if (entry != profiler_data_.end()) {
return &entry->second;
} else {
profiler_data_[guest_address] = 0;
return &profiler_data_[guest_address];
}
}
#endif
} // namespace x64
} // namespace backend
} // namespace cpu

View File

@@ -15,6 +15,14 @@
#include "xenia/base/cvar.h"
#include "xenia/cpu/backend/backend.h"
#if XE_PLATFORM_WIN32 == 1
// we use KUSER_SHARED's systemtime field, which is at a fixed address and
// obviously windows specific, to get the start/end time for a function using
// rdtsc would be too slow and skew the results by consuming extra cpu time, so
// we have lower time precision but better overall accuracy
#define XE_X64_PROFILER_AVAILABLE 1
#endif
DECLARE_int32(x64_extension_mask);
namespace xe {
@@ -24,6 +32,8 @@ namespace xe {
namespace cpu {
namespace backend {
namespace x64 {
// mapping of guest function addresses to total nanoseconds taken in the func
using GuestProfilerData = std::map<uint32_t, uint64_t>;
class X64CodeCache;
@@ -37,8 +47,10 @@ typedef void (*ResolveFunctionThunk)();
// negatively index the membase reg)
struct X64BackendContext {
void* ResolveFunction_Ptr; // cached pointer to resolvefunction
unsigned int mxcsr_fpu; // currently, the way we implement rounding mode
// affects both vmx and the fpu
uint64_t* guest_tick_count;
unsigned int mxcsr_fpu; // currently, the way we implement rounding mode
// affects both vmx and the fpu
unsigned int mxcsr_vmx;
unsigned int flags; // bit 0 = 0 if mxcsr is fpu, else it is vmx
unsigned int Ox1000; // constant 0x1000 so we can shrink each tail emitted
@@ -93,7 +105,9 @@ class X64Backend : public Backend {
virtual void SetGuestRoundingMode(void* ctx, unsigned int mode) override;
void RecordMMIOExceptionForGuestInstruction(void* host_address);
#if XE_X64_PROFILER_AVAILABLE == 1
uint64_t* GetProfilerRecordForFunction(uint32_t guest_address);
#endif
private:
static bool ExceptionCallbackThunk(Exception* ex, void* data);
bool ExceptionCallback(Exception* ex);
@@ -106,6 +120,10 @@ class X64Backend : public Backend {
HostToGuestThunk host_to_guest_thunk_;
GuestToHostThunk guest_to_host_thunk_;
ResolveFunctionThunk resolve_function_thunk_;
#if XE_X64_PROFILER_AVAILABLE == 1
GuestProfilerData profiler_data_;
#endif
};
} // namespace x64

View File

@@ -57,6 +57,12 @@ DEFINE_bool(enable_incorrect_roundingmode_behavior, false,
"code. The workaround may cause reduced CPU performance but is a "
"more accurate emulation",
"x64");
#if XE_X64_PROFILER_AVAILABLE == 1
DEFINE_bool(instrument_call_times, false,
"Compute time taken for functions, for profiling guest code",
"x64");
#endif
namespace xe {
namespace cpu {
namespace backend {
@@ -120,28 +126,37 @@ X64Emitter::X64Emitter(X64Backend* backend, XbyakAllocator* allocator)
*/
unsigned int data[4];
Xbyak::util::Cpu::getCpuid(0x80000001, data);
if (data[2] & (1U << 5)) {
unsigned amd_flags = data[2];
if (amd_flags & (1U << 5)) {
if ((cvars::x64_extension_mask & kX64EmitLZCNT) == kX64EmitLZCNT) {
feature_flags_ |= kX64EmitLZCNT;
}
}
// todo: although not reported by cpuid, zen 1 and zen+ also have fma4
if (amd_flags & (1U << 16)) {
if ((cvars::x64_extension_mask & kX64EmitFMA4) == kX64EmitFMA4) {
feature_flags_ |= kX64EmitFMA4;
}
}
if (amd_flags & (1U << 21)) {
if ((cvars::x64_extension_mask & kX64EmitTBM) == kX64EmitTBM) {
feature_flags_ |= kX64EmitTBM;
}
}
if (cpu_.has(Xbyak::util::Cpu::tAMD)) {
bool is_zennish = cpu_.displayFamily >= 0x17;
/*
chrispy: according to agner's tables, all amd architectures that
we support (ones with avx) have the same timings for
jrcxz/loop/loope/loopne as for other jmps
*/
feature_flags_ |= kX64FastJrcx;
feature_flags_ |= kX64FastLoop;
if (is_zennish) {
// ik that i heard somewhere that this is the case for zen, but i need to
// verify. cant find my original source for that.
// todo: ask agner?
feature_flags_ |= kX64FlagsIndependentVars;
feature_flags_ |= kX64FastJrcx;
if (cpu_.displayFamily > 0x17) {
feature_flags_ |= kX64FastLoop;
} else if (cpu_.displayFamily == 0x17 && cpu_.displayModel >= 0x31) {
feature_flags_ |= kX64FastLoop;
} // todo:figure out at model zen+ became zen2, this is just the model
// for my cpu, which is ripper90
}
}
may_use_membase32_as_zero_reg_ =
@@ -157,6 +172,7 @@ bool X64Emitter::Emit(GuestFunction* function, HIRBuilder* builder,
std::vector<SourceMapEntry>* out_source_map) {
SCOPE_profile_cpu_f("cpu");
guest_module_ = dynamic_cast<XexModule*>(function->module());
current_guest_function_ = function->address();
// Reset.
debug_info_ = debug_info;
debug_info_flags_ = debug_info_flags;
@@ -286,10 +302,19 @@ bool X64Emitter::Emit(HIRBuilder* builder, EmitFunctionInfo& func_info) {
* chrispy: removed this, it serves no purpose
mov(qword[rsp + StackLayout::GUEST_CTX_HOME], GetContextReg());
*/
mov(qword[rsp + StackLayout::GUEST_RET_ADDR], rcx);
mov(qword[rsp + StackLayout::GUEST_CALL_RET_ADDR], rax); // 0
#if XE_X64_PROFILER_AVAILABLE == 1
if (cvars::instrument_call_times) {
mov(rdx, 0x7ffe0014); // load pointer to kusershared systemtime
mov(rdx, qword[rdx]);
mov(qword[rsp + StackLayout::GUEST_PROFILER_START],
rdx); // save time for end of function
}
#endif
// Safe now to do some tracing.
if (debug_info_flags_ & DebugInfoFlags::kDebugInfoTraceFunctions) {
// We require 32-bit addresses.
@@ -363,6 +388,7 @@ bool X64Emitter::Emit(HIRBuilder* builder, EmitFunctionInfo& func_info) {
mov(GetContextReg(), qword[rsp + StackLayout::GUEST_CTX_HOME]);
*/
code_offsets.epilog = getSize();
EmitProfilerEpilogue();
add(rsp, (uint32_t)stack_size);
ret();
@@ -391,6 +417,27 @@ bool X64Emitter::Emit(HIRBuilder* builder, EmitFunctionInfo& func_info) {
return true;
}
// dont use rax, we do this in tail call handling
void X64Emitter::EmitProfilerEpilogue() {
#if XE_X64_PROFILER_AVAILABLE == 1
if (cvars::instrument_call_times) {
uint64_t* profiler_entry =
backend()->GetProfilerRecordForFunction(current_guest_function_);
mov(ecx, 0x7ffe0014);
mov(rdx, qword[rcx]);
mov(rbx, (uintptr_t)profiler_entry);
sub(rdx, qword[rsp + StackLayout::GUEST_PROFILER_START]);
// atomic add our time to the profiler entry
// this could be atomic free if we had per thread profile counts, and on a
// threads exit we lock and sum up to the global counts, which would make
// this a few cycles less intrusive, but its good enough for now
// actually... lets just try without atomics lol
// lock();
add(qword[rbx], rdx);
}
#endif
}
void X64Emitter::MarkSourceOffset(const Instr* i) {
auto entry = source_map_arena_.Alloc<SourceMapEntry>();
@@ -558,7 +605,7 @@ void X64Emitter::Call(const hir::Instr* instr, GuestFunction* function) {
if (instr->flags & hir::CALL_TAIL) {
// Since we skip the prolog we need to mark the return here.
EmitTraceUserCallReturn();
EmitProfilerEpilogue();
// Pass the callers return address over.
mov(rcx, qword[rsp + StackLayout::GUEST_RET_ADDR]);
@@ -602,7 +649,7 @@ void X64Emitter::CallIndirect(const hir::Instr* instr,
if (instr->flags & hir::CALL_TAIL) {
// Since we skip the prolog we need to mark the return here.
EmitTraceUserCallReturn();
EmitProfilerEpilogue();
// Pass the callers return address over.
mov(rcx, qword[rsp + StackLayout::GUEST_RET_ADDR]);
@@ -952,7 +999,34 @@ static const vec128_t xmm_consts[] = {
/*XMMVSRShlByteshuf*/
v128_setr_bytes(13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3, 0x80),
// XMMVSRMask
vec128b(1)};
vec128b(1),
/*
XMMF16UnpackLCPI2
*/
vec128i(0x38000000),
/*
XMMF16UnpackLCPI3
*/
vec128q(0x7fe000007fe000ULL),
/* XMMF16PackLCPI0*/
vec128i(0x8000000),
/*XMMF16PackLCPI2*/
vec128i(0x47ffe000),
/*XMMF16PackLCPI3*/
vec128i(0xc7800000),
/*XMMF16PackLCPI4
*/
vec128i(0xf7fdfff),
/*XMMF16PackLCPI5*/
vec128i(0x7fff),
/*
XMMF16PackLCPI6
*/
vec128i(0x8000)
};
void* X64Emitter::FindByteConstantOffset(unsigned bytevalue) {
for (auto& vec : xmm_consts) {

View File

@@ -159,7 +159,15 @@ enum XmmConst {
XMMThreeFloatMask, // for clearing the fourth float prior to DOT_PRODUCT_3
XMMXenosF16ExtRangeStart,
XMMVSRShlByteshuf,
XMMVSRMask
XMMVSRMask,
XMMF16UnpackLCPI2, // 0x38000000, 1/ 32768
XMMF16UnpackLCPI3, // 0x0x7fe000007fe000
XMMF16PackLCPI0,
XMMF16PackLCPI2,
XMMF16PackLCPI3,
XMMF16PackLCPI4,
XMMF16PackLCPI5,
XMMF16PackLCPI6
};
// X64Backend specific Instr->runtime_flags
enum : uint32_t {
@@ -177,7 +185,7 @@ class XbyakAllocator : public Xbyak::Allocator {
enum X64EmitterFeatureFlags {
kX64EmitAVX2 = 1 << 0,
kX64EmitFMA = 1 << 1,
kX64EmitLZCNT = 1 << 2,
kX64EmitLZCNT = 1 << 2, // this is actually ABM and includes popcount
kX64EmitBMI1 = 1 << 3,
kX64EmitBMI2 = 1 << 4,
kX64EmitF16C = 1 << 5,
@@ -201,7 +209,11 @@ enum X64EmitterFeatureFlags {
// inc/dec) do not introduce false dependencies on EFLAGS
// because the individual flags are treated as different vars by
// the processor. (this applies to zen)
kX64EmitPrefetchW = 1 << 16
kX64EmitPrefetchW = 1 << 16,
kX64EmitXOP = 1 << 17, // chrispy: xop maps really well to many vmx
// instructions, and FX users need the boost
kX64EmitFMA4 = 1 << 18, // todo: also use on zen1?
kX64EmitTBM = 1 << 19
};
class ResolvableGuestCall {
public:
@@ -337,6 +349,8 @@ class X64Emitter : public Xbyak::CodeGenerator {
XexModule* GuestModule() { return guest_module_; }
void EmitProfilerEpilogue();
protected:
void* Emplace(const EmitFunctionInfo& func_info,
GuestFunction* function = nullptr);
@@ -352,7 +366,7 @@ class X64Emitter : public Xbyak::CodeGenerator {
XexModule* guest_module_ = nullptr;
Xbyak::util::Cpu cpu_;
uint32_t feature_flags_ = 0;
uint32_t current_guest_function_ = 0;
Xbyak::Label* epilog_label_ = nullptr;
hir::Instr* current_instr_ = nullptr;

View File

@@ -19,10 +19,6 @@
#include "xenia/base/cvar.h"
#include "xenia/cpu/backend/x64/x64_stack_layout.h"
DEFINE_bool(use_extended_range_half, true,
"Emulate extended range half-precision, may be slower on games "
"that use it heavily",
"CPU");
namespace xe {
namespace cpu {
namespace backend {
@@ -1982,6 +1978,137 @@ struct PERMUTE_V128
};
EMITTER_OPCODE_TABLE(OPCODE_PERMUTE, PERMUTE_I32, PERMUTE_V128);
#define LCPI(name, quad1) const __m128i name = _mm_set1_epi32(quad1)
// xmm0 is precasted to int, but contains float
// chrispy: todo: make available to gpu code
static __m128i xenos_float4_to_float16_x4(__m128i xmm0) {
LCPI(LCPI0_0, 2147483647);
LCPI(LCPI0_1, 1207951360);
LCPI(LCPI0_2, 134217728);
LCPI(LCPI0_3, 3347054592);
LCPI(LCPI0_4, 260038655);
LCPI(LCPI0_5, 32767);
LCPI(LCPI0_6, 4294934528);
__m128i xmm1 = _mm_and_si128(xmm0, LCPI0_0);
__m128i xmm2 = LCPI0_1;
__m128i xmm3 = _mm_add_epi32(xmm0, LCPI0_2);
xmm2 = _mm_cmpgt_epi32(xmm2, xmm1);
xmm3 = _mm_srli_epi32(xmm3, 13);
xmm1 = _mm_add_epi32(xmm1, LCPI0_3);
__m128i xmm4 = _mm_min_epu32(xmm1, LCPI0_4);
xmm1 = _mm_cmpeq_epi32(xmm1, xmm4);
xmm4 = LCPI0_5;
xmm3 = _mm_and_si128(xmm3, xmm4);
xmm1 = _mm_and_si128(xmm1, xmm3);
xmm1 = _mm_castps_si128(_mm_blendv_ps(
_mm_castsi128_ps(xmm4), _mm_castsi128_ps(xmm1), _mm_castsi128_ps(xmm2)));
xmm0 = _mm_srli_epi32(xmm0, 16);
xmm0 = _mm_and_si128(xmm0, LCPI0_6);
xmm0 = _mm_or_si128(xmm1, xmm0);
xmm0 = _mm_packus_epi32(xmm0, _mm_setzero_si128());
return xmm0;
}
// returns floats, uncasted
// chrispy: todo, make this available to gpu code?
static __m128i xenos_halves_to_floats(__m128i xmm0) {
LCPI(LCPI3_0, 0x1f);
LCPI(LCPI3_1, 0x80000000);
LCPI(LCPI3_2, 0x38000000);
LCPI(LCPI3_3, 0x7fe000);
__m128i xmm1, xmm2, xmm3, xmm4;
xmm1 = _mm_cvtepu16_epi32(xmm0);
xmm2 = _mm_srli_epi32(xmm1, 10);
xmm2 = _mm_and_si128(xmm2, LCPI3_0);
xmm0 = _mm_cvtepi16_epi32(xmm0);
xmm0 = _mm_and_si128(xmm0, LCPI3_1);
xmm3 = _mm_setzero_si128();
xmm4 = _mm_slli_epi32(xmm2, 23);
xmm4 = _mm_add_epi32(xmm4, LCPI3_2);
xmm2 = _mm_cmpeq_epi32(xmm2, xmm3);
xmm1 = _mm_slli_epi32(xmm1, 13);
xmm1 = _mm_and_si128(xmm1, LCPI3_3);
xmm3 = _mm_andnot_si128(xmm2, xmm4);
xmm1 = _mm_andnot_si128(xmm2, xmm1);
xmm0 = _mm_or_si128(xmm1, xmm0);
xmm0 = _mm_or_si128(xmm0, xmm3);
return xmm0;
}
#undef LCPI
template <typename Inst>
static void emit_fast_f16_unpack(X64Emitter& e, const Inst& i,
XmmConst initial_shuffle) {
auto src1 = i.src1;
e.vpshufb(i.dest, src1, e.GetXmmConstPtr(initial_shuffle));
e.vpmovsxwd(e.xmm1, i.dest);
e.vpsrld(e.xmm2, e.xmm1, 10);
e.vpmovsxwd(e.xmm0, i.dest);
e.vpand(e.xmm0, e.xmm0, e.GetXmmConstPtr(XMMSignMaskPS));
e.vpand(e.xmm2, e.xmm2, e.GetXmmConstPtr(XMMPermuteByteMask));
e.vpslld(e.xmm3, e.xmm2, 23);
e.vpaddd(e.xmm3, e.xmm3, e.GetXmmConstPtr(XMMF16UnpackLCPI2));
e.vpcmpeqd(e.xmm2, e.xmm2, e.GetXmmConstPtr(XMMZero));
e.vpslld(e.xmm1, e.xmm1, 13);
e.vpandn(e.xmm1, e.xmm2, e.xmm1);
e.vpandn(e.xmm2, e.xmm2, e.xmm3);
e.vpand(e.xmm1, e.xmm1, e.GetXmmConstPtr(XMMF16UnpackLCPI3));
e.vpor(e.xmm0, e.xmm1, e.xmm0);
e.vpor(i.dest, e.xmm0, e.xmm2);
}
template <typename Inst>
static void emit_fast_f16_pack(X64Emitter& e, const Inst& i,
XmmConst final_shuffle) {
e.vpaddd(e.xmm1, i.src1, e.GetXmmConstPtr(XMMF16PackLCPI0));
e.vpand(e.xmm2, i.src1, e.GetXmmConstPtr(XMMAbsMaskPS));
e.vmovdqa(e.xmm3, e.GetXmmConstPtr(XMMF16PackLCPI2));
e.vpcmpgtd(e.xmm3, e.xmm3, e.xmm2);
e.vpsrld(e.xmm1, e.xmm1, 13);
e.vpaddd(e.xmm2, e.xmm2, e.GetXmmConstPtr(XMMF16PackLCPI3));
e.vpminud(e.xmm0, e.xmm2, e.GetXmmConstPtr(XMMF16PackLCPI4));
e.vpcmpeqd(e.xmm2, e.xmm2, e.xmm0);
e.vmovdqa(e.xmm0, e.GetXmmConstPtr(XMMF16PackLCPI5));
e.vpand(e.xmm1, e.xmm1, e.xmm0);
e.vpand(e.xmm1, e.xmm2, e.xmm1);
e.vpxor(e.xmm2, e.xmm2, e.xmm2);
e.vblendvps(e.xmm1, e.xmm0, e.xmm1, e.xmm3);
e.vpsrld(e.xmm0, i.src1, 16);
e.vpand(e.xmm0, e.xmm0, e.GetXmmConstPtr(XMMF16PackLCPI6));
e.vorps(e.xmm0, e.xmm1, e.xmm0);
e.vpackusdw(i.dest, e.xmm0, e.xmm2);
e.vpshufb(i.dest, i.dest, e.GetXmmConstPtr(final_shuffle));
}
// ============================================================================
// OPCODE_SWIZZLE
// ============================================================================
@@ -2081,14 +2208,9 @@ struct PACK : Sequence<PACK, I<OPCODE_PACK, V128Op, V128Op, V128Op>> {
alignas(16) uint16_t b[8];
_mm_store_ps(a, src1);
std::memset(b, 0, sizeof(b));
if (!cvars::use_extended_range_half) {
for (int i = 0; i < 2; i++) {
b[7 - i] = half_float::detail::float2half<std::round_toward_zero>(a[i]);
}
} else {
for (int i = 0; i < 2; i++) {
b[7 - i] = float_to_xenos_half(a[i]);
}
for (int i = 0; i < 2; i++) {
b[7 - i] = float_to_xenos_half(a[i]);
}
return _mm_load_si128(reinterpret_cast<__m128i*>(b));
@@ -2098,70 +2220,26 @@ struct PACK : Sequence<PACK, I<OPCODE_PACK, V128Op, V128Op, V128Op>> {
// http://blogs.msdn.com/b/chuckw/archive/2012/09/11/directxmath-f16c-and-fma.aspx
// dest = [(src1.x | src1.y), 0, 0, 0]
if (e.IsFeatureEnabled(kX64EmitF16C) && !cvars::use_extended_range_half) {
Xmm src;
if (i.src1.is_constant) {
src = i.dest;
e.LoadConstantXmm(src, i.src1.constant());
} else {
src = i.src1;
}
// 0|0|0|0|W|Z|Y|X
e.vcvtps2ph(i.dest, src, 0b00000011);
// Shuffle to X|Y|0|0|0|0|0|0
e.vpshufb(i.dest, i.dest, e.GetXmmConstPtr(XMMPackFLOAT16_2));
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_2));
e.vmovaps(i.dest, e.xmm0);
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_2));
e.vmovaps(i.dest, e.xmm0);
}
static __m128i EmulateFLOAT16_4(void*, __m128 src1) {
alignas(16) float a[4];
alignas(16) uint16_t b[8];
_mm_store_ps(a, src1);
std::memset(b, 0, sizeof(b));
if (!cvars::use_extended_range_half) {
for (int i = 0; i < 4; i++) {
b[7 - (i ^ 2)] =
half_float::detail::float2half<std::round_toward_zero>(a[i]);
}
} else {
for (int i = 0; i < 4; i++) {
b[7 - (i ^ 2)] = float_to_xenos_half(a[i]);
}
}
return _mm_load_si128(reinterpret_cast<__m128i*>(b));
}
static void EmitFLOAT16_4(X64Emitter& e, const EmitArgType& i) {
assert_true(i.src2.value->IsConstantZero());
// dest = [(src1.z | src1.w), (src1.x | src1.y), 0, 0]
if (e.IsFeatureEnabled(kX64EmitF16C) && !cvars::use_extended_range_half) {
Xmm src;
if (i.src1.is_constant) {
src = i.dest;
e.LoadConstantXmm(src, i.src1.constant());
} else {
src = i.src1;
}
// 0|0|0|0|W|Z|Y|X
e.vcvtps2ph(i.dest, src, 0b00000011);
// Shuffle to Z|W|X|Y|0|0|0|0
e.vpshufb(i.dest, i.dest, e.GetXmmConstPtr(XMMPackFLOAT16_4));
if (!i.src1.is_constant) {
emit_fast_f16_pack(e, i, XMMPackFLOAT16_4);
} else {
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
vec128_t result = vec128b(0);
for (unsigned idx = 0; idx < 4; ++idx) {
result.u16[(7 - (idx ^ 2))] =
float_to_xenos_half(i.src1.constant().f32[idx]);
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_4));
e.vmovaps(i.dest, e.xmm0);
e.LoadConstantXmm(i.dest, result);
}
}
static void EmitSHORT_2(X64Emitter& e, const EmitArgType& i) {
@@ -2508,15 +2586,10 @@ struct UNPACK : Sequence<UNPACK, I<OPCODE_UNPACK, V128Op, V128Op>> {
alignas(16) float b[4];
_mm_store_si128(reinterpret_cast<__m128i*>(a), src1);
if (!cvars::use_extended_range_half) {
for (int i = 0; i < 2; i++) {
b[i] = half_float::detail::half2float(a[VEC128_W(6 + i)]);
}
} else {
for (int i = 0; i < 2; i++) {
b[i] = xenos_half_to_float(a[VEC128_W(6 + i)]);
}
for (int i = 0; i < 2; i++) {
b[i] = xenos_half_to_float(a[VEC128_W(6 + i)]);
}
// Constants, or something
b[2] = 0.f;
b[3] = 1.f;
@@ -2536,74 +2609,28 @@ struct UNPACK : Sequence<UNPACK, I<OPCODE_UNPACK, V128Op, V128Op>> {
// Also zero out the high end.
// TODO(benvanik): special case constant unpacks that just get 0/1/etc.
if (e.IsFeatureEnabled(kX64EmitF16C) &&
!cvars::use_extended_range_half) { // todo: can use cvtph and bit logic
// to implement
Xmm src;
if (i.src1.is_constant) {
src = i.dest;
e.LoadConstantXmm(src, i.src1.constant());
} else {
src = i.src1;
}
// sx = src.iw >> 16;
// sy = src.iw & 0xFFFF;
// dest = { XMConvertHalfToFloat(sx),
// XMConvertHalfToFloat(sy),
// 0.0,
// 1.0 };
// Shuffle to 0|0|0|0|0|0|Y|X
e.vpshufb(i.dest, src, e.GetXmmConstPtr(XMMUnpackFLOAT16_2));
e.vcvtph2ps(i.dest, i.dest);
e.vpshufd(i.dest, i.dest, 0b10100100);
e.vpor(i.dest, e.GetXmmConstPtr(XMM0001));
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_2));
e.vmovaps(i.dest, e.xmm0);
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_2));
e.vmovaps(i.dest, e.xmm0);
}
static __m128 EmulateFLOAT16_4(void*, __m128i src1) {
alignas(16) uint16_t a[8];
alignas(16) float b[4];
_mm_store_si128(reinterpret_cast<__m128i*>(a), src1);
if (!cvars::use_extended_range_half) {
for (int i = 0; i < 4; i++) {
b[i] = half_float::detail::half2float(a[VEC128_W(4 + i)]);
}
} else {
for (int i = 0; i < 4; i++) {
b[i] = xenos_half_to_float(a[VEC128_W(4 + i)]);
}
}
return _mm_load_ps(b);
}
static void EmitFLOAT16_4(X64Emitter& e, const EmitArgType& i) {
// src = [(dest.x | dest.y), (dest.z | dest.w), 0, 0]
if (e.IsFeatureEnabled(kX64EmitF16C) && !cvars::use_extended_range_half) {
Xmm src;
if (i.src1.is_constant) {
src = i.dest;
e.LoadConstantXmm(src, i.src1.constant());
} else {
src = i.src1;
if (i.src1.is_constant) {
vec128_t result{};
for (int idx = 0; idx < 4; ++idx) {
result.f32[idx] =
xenos_half_to_float(i.src1.constant().u16[VEC128_W(4 + idx)]);
}
// Shuffle to 0|0|0|0|W|Z|Y|X
e.vpshufb(i.dest, src, e.GetXmmConstPtr(XMMUnpackFLOAT16_4));
e.vcvtph2ps(i.dest, i.dest);
e.LoadConstantXmm(i.dest, result);
} else {
if (i.src1.is_constant) {
e.lea(e.GetNativeParam(0), e.StashConstantXmm(0, i.src1.constant()));
} else {
e.lea(e.GetNativeParam(0), e.StashXmm(0, i.src1));
}
e.CallNativeSafe(reinterpret_cast<void*>(EmulateFLOAT16_4));
e.vmovaps(i.dest, e.xmm0);
emit_fast_f16_unpack(e, i, XMMUnpackFLOAT16_4);
}
}
static void EmitSHORT_2(X64Emitter& e, const EmitArgType& i) {

View File

@@ -50,6 +50,10 @@ DEFINE_bool(no_round_to_single, false,
"Not for users, breaks games. Skip rounding double values to "
"single precision and back",
"CPU");
DEFINE_bool(
inline_loadclock, false,
"Directly read cached guest clock without calling the LoadClock method (it gets repeatedly updated by calls from other threads)",
"CPU");
namespace xe {
namespace cpu {
namespace backend {
@@ -475,33 +479,39 @@ EMITTER_OPCODE_TABLE(OPCODE_ROUND, ROUND_F32, ROUND_F64, ROUND_V128);
// ============================================================================
struct LOAD_CLOCK : Sequence<LOAD_CLOCK, I<OPCODE_LOAD_CLOCK, I64Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
// When scaling is disabled and the raw clock source is selected, the code
// in the Clock class is actually just forwarding tick counts after one
// simple multiply and division. In that case we rather bake the scaling in
// here to cut extra function calls with CPU cache misses and stack frame
// overhead.
if (cvars::clock_no_scaling && cvars::clock_source_raw) {
auto ratio = Clock::guest_tick_ratio();
// The 360 CPU is an in-order CPU, AMD64 usually isn't. Without
// mfence/lfence magic the rdtsc instruction can be executed sooner or
// later in the cache window. Since it's resolution however is much higher
// than the 360's mftb instruction this can safely be ignored.
// Read time stamp in edx (high part) and eax (low part).
e.rdtsc();
// Make it a 64 bit number in rax.
e.shl(e.rdx, 32);
e.or_(e.rax, e.rdx);
// Apply tick frequency scaling.
e.mov(e.rcx, ratio.first);
e.mul(e.rcx);
// We actually now have a 128 bit number in rdx:rax.
e.mov(e.rcx, ratio.second);
e.div(e.rcx);
e.mov(i.dest, e.rax);
if (cvars::inline_loadclock) {
e.mov(e.rcx,
e.GetBackendCtxPtr(offsetof(X64BackendContext, guest_tick_count)));
e.mov(i.dest, e.qword[e.rcx]);
} else {
e.CallNative(LoadClock);
e.mov(i.dest, e.rax);
// When scaling is disabled and the raw clock source is selected, the code
// in the Clock class is actually just forwarding tick counts after one
// simple multiply and division. In that case we rather bake the scaling
// in here to cut extra function calls with CPU cache misses and stack
// frame overhead.
if (cvars::clock_no_scaling && cvars::clock_source_raw) {
auto ratio = Clock::guest_tick_ratio();
// The 360 CPU is an in-order CPU, AMD64 usually isn't. Without
// mfence/lfence magic the rdtsc instruction can be executed sooner or
// later in the cache window. Since it's resolution however is much
// higher than the 360's mftb instruction this can safely be ignored.
// Read time stamp in edx (high part) and eax (low part).
e.rdtsc();
// Make it a 64 bit number in rax.
e.shl(e.rdx, 32);
e.or_(e.rax, e.rdx);
// Apply tick frequency scaling.
e.mov(e.rcx, ratio.first);
e.mul(e.rcx);
// We actually now have a 128 bit number in rdx:rax.
e.mov(e.rcx, ratio.second);
e.div(e.rcx);
e.mov(i.dest, e.rax);
} else {
e.CallNative(LoadClock);
e.mov(i.dest, e.rax);
}
}
}
static uint64_t LoadClock(void* raw_context) {
@@ -539,10 +549,12 @@ struct MAX_F64 : Sequence<MAX_F64, I<OPCODE_MAX, F64Op, F64Op, F64Op>> {
struct MAX_V128 : Sequence<MAX_V128, I<OPCODE_MAX, V128Op, V128Op, V128Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Vmx);
EmitCommutativeBinaryXmmOp(e, i,
[](X64Emitter& e, Xmm dest, Xmm src1, Xmm src2) {
e.vmaxps(dest, src1, src2);
});
auto src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
auto src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
e.vmaxps(e.xmm2, src1, src2);
e.vmaxps(e.xmm3, src2, src1);
e.vorps(i.dest, e.xmm2, e.xmm3);
}
};
EMITTER_OPCODE_TABLE(OPCODE_MAX, MAX_F32, MAX_F64, MAX_V128);
@@ -597,10 +609,11 @@ struct MIN_F64 : Sequence<MIN_F64, I<OPCODE_MIN, F64Op, F64Op, F64Op>> {
struct MIN_V128 : Sequence<MIN_V128, I<OPCODE_MIN, V128Op, V128Op, V128Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Vmx);
EmitCommutativeBinaryXmmOp(e, i,
[](X64Emitter& e, Xmm dest, Xmm src1, Xmm src2) {
e.vminps(dest, src1, src2);
});
auto src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
auto src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
e.vminps(e.xmm2, src1, src2);
e.vminps(e.xmm3, src2, src1);
e.vorps(i.dest, e.xmm2, e.xmm3);
}
};
EMITTER_OPCODE_TABLE(OPCODE_MIN, MIN_I8, MIN_I16, MIN_I32, MIN_I64, MIN_F32,
@@ -768,6 +781,7 @@ struct SELECT_V128_V128
} else if (mayblend == PermittedBlend::Ps) {
e.vblendvps(i.dest, src2, src3, src1);
} else {
//ideally we would have an xop path here...
// src1 ? src2 : src3;
e.vpandn(e.xmm3, src1, src2);
e.vpand(i.dest, src1, src3);
@@ -1932,6 +1946,53 @@ struct MUL_ADD_V128
};
EMITTER_OPCODE_TABLE(OPCODE_MUL_ADD, MUL_ADD_F32, MUL_ADD_F64, MUL_ADD_V128);
struct NEGATED_MUL_ADD_F64
: Sequence<NEGATED_MUL_ADD_F64,
I<OPCODE_NEGATED_MUL_ADD, F64Op, F64Op, F64Op, F64Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Fpu);
Xmm src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
Xmm src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
Xmm src3 = GetInputRegOrConstant(e, i.src3, e.xmm2);
if (e.IsFeatureEnabled(kX64EmitFMA)) {
// todo: this is garbage
e.vmovapd(e.xmm3, src1);
e.vfmadd213sd(e.xmm3, src2, src3);
e.vxorpd(i.dest, e.xmm3, e.GetXmmConstPtr(XMMSignMaskPD));
} else {
// todo: might need to use x87 in this case...
e.vmulsd(e.xmm3, src1, src2);
e.vaddsd(i.dest, e.xmm3, src3);
e.vxorpd(i.dest, i.dest, e.GetXmmConstPtr(XMMSignMaskPD));
}
}
};
struct NEGATED_MUL_ADD_V128
: Sequence<NEGATED_MUL_ADD_V128,
I<OPCODE_NEGATED_MUL_ADD, V128Op, V128Op, V128Op, V128Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Vmx);
Xmm src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
Xmm src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
Xmm src3 = GetInputRegOrConstant(e, i.src3, e.xmm2);
if (e.IsFeatureEnabled(kX64EmitFMA)) {
// todo: this is garbage
e.vmovaps(e.xmm3, src1);
e.vfmadd213ps(e.xmm3, src2, src3);
e.vxorps(i.dest, e.xmm3, e.GetXmmConstPtr(XMMSignMaskPS));
} else {
// todo: might need to use x87 in this case...
e.vmulps(e.xmm3, src1, src2);
e.vaddps(i.dest, e.xmm3, src3);
e.vxorps(i.dest, i.dest, e.GetXmmConstPtr(XMMSignMaskPS));
}
}
};
EMITTER_OPCODE_TABLE(OPCODE_NEGATED_MUL_ADD, NEGATED_MUL_ADD_F64,
NEGATED_MUL_ADD_V128);
// ============================================================================
// OPCODE_MUL_SUB
// ============================================================================
@@ -1944,12 +2005,7 @@ EMITTER_OPCODE_TABLE(OPCODE_MUL_ADD, MUL_ADD_F32, MUL_ADD_F64, MUL_ADD_V128);
// - 132 -> $1 = $1 * $3 - $2
// - 213 -> $1 = $2 * $1 - $3
// - 231 -> $1 = $2 * $3 - $1
struct MUL_SUB_F32
: Sequence<MUL_SUB_F32, I<OPCODE_MUL_SUB, F32Op, F32Op, F32Op, F32Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
assert_impossible_sequence(MUL_SUB_F32);
}
};
struct MUL_SUB_F64
: Sequence<MUL_SUB_F64, I<OPCODE_MUL_SUB, F64Op, F64Op, F64Op, F64Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
@@ -1991,7 +2047,54 @@ struct MUL_SUB_V128
}
}
};
EMITTER_OPCODE_TABLE(OPCODE_MUL_SUB, MUL_SUB_F32, MUL_SUB_F64, MUL_SUB_V128);
EMITTER_OPCODE_TABLE(OPCODE_MUL_SUB, MUL_SUB_F64, MUL_SUB_V128);
struct NEGATED_MUL_SUB_F64
: Sequence<NEGATED_MUL_SUB_F64,
I<OPCODE_NEGATED_MUL_SUB, F64Op, F64Op, F64Op, F64Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Fpu);
Xmm src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
Xmm src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
Xmm src3 = GetInputRegOrConstant(e, i.src3, e.xmm2);
if (e.IsFeatureEnabled(kX64EmitFMA)) {
// todo: this is garbage
e.vmovapd(e.xmm3, src1);
e.vfmsub213sd(e.xmm3, src2, src3);
e.vxorpd(i.dest, e.xmm3, e.GetXmmConstPtr(XMMSignMaskPD));
} else {
// todo: might need to use x87 in this case...
e.vmulsd(e.xmm3, src1, src2);
e.vsubsd(i.dest, e.xmm3, src3);
e.vxorpd(i.dest, i.dest, e.GetXmmConstPtr(XMMSignMaskPD));
}
}
};
struct NEGATED_MUL_SUB_V128
: Sequence<NEGATED_MUL_SUB_V128,
I<OPCODE_NEGATED_MUL_SUB, V128Op, V128Op, V128Op, V128Op>> {
static void Emit(X64Emitter& e, const EmitArgType& i) {
e.ChangeMxcsrMode(MXCSRMode::Vmx);
Xmm src1 = GetInputRegOrConstant(e, i.src1, e.xmm0);
Xmm src2 = GetInputRegOrConstant(e, i.src2, e.xmm1);
Xmm src3 = GetInputRegOrConstant(e, i.src3, e.xmm2);
if (e.IsFeatureEnabled(kX64EmitFMA)) {
// todo: this is garbage
e.vmovaps(e.xmm3, src1);
e.vfmsub213ps(e.xmm3, src2, src3);
e.vxorps(i.dest, e.xmm3, e.GetXmmConstPtr(XMMSignMaskPS));
} else {
// todo: might need to use x87 in this case...
e.vmulps(e.xmm3, src1, src2);
e.vsubps(i.dest, e.xmm3, src3);
e.vxorps(i.dest, i.dest, e.GetXmmConstPtr(XMMSignMaskPS));
}
}
};
EMITTER_OPCODE_TABLE(OPCODE_NEGATED_MUL_SUB, NEGATED_MUL_SUB_F64,
NEGATED_MUL_SUB_V128);
// ============================================================================
// OPCODE_NEG
@@ -2264,7 +2367,7 @@ struct DOT_PRODUCT_3_V128
e.ChangeMxcsrMode(MXCSRMode::Vmx);
// todo: add fast_dot_product path that just checks for infinity instead of
// using mxcsr
auto mxcsr_storage = e.dword[e.rsp + StackLayout::GUEST_SCRATCH64];
auto mxcsr_storage = e.dword[e.rsp + StackLayout::GUEST_SCRATCH];
// this is going to hurt a bit...
/*
@@ -2380,7 +2483,7 @@ struct DOT_PRODUCT_4_V128
e.ChangeMxcsrMode(MXCSRMode::Vmx);
// todo: add fast_dot_product path that just checks for infinity instead of
// using mxcsr
auto mxcsr_storage = e.dword[e.rsp + StackLayout::GUEST_SCRATCH64];
auto mxcsr_storage = e.dword[e.rsp + StackLayout::GUEST_SCRATCH];
bool is_lensqr = i.instr->src1.value == i.instr->src2.value;
@@ -3162,9 +3265,9 @@ struct SET_ROUNDING_MODE_I32
// backends dont have to worry about it
if (i.src1.is_constant) {
e.mov(e.eax, mxcsr_table[i.src1.constant()]);
e.mov(e.dword[e.rsp + StackLayout::GUEST_SCRATCH64], e.eax);
e.mov(e.dword[e.rsp + StackLayout::GUEST_SCRATCH], e.eax);
e.mov(e.GetBackendCtxPtr(offsetof(X64BackendContext, mxcsr_fpu)), e.eax);
e.vldmxcsr(e.dword[e.rsp + StackLayout::GUEST_SCRATCH64]);
e.vldmxcsr(e.dword[e.rsp + StackLayout::GUEST_SCRATCH]);
} else {
e.mov(e.ecx, i.src1);

View File

@@ -123,7 +123,10 @@ class StackLayout {
*/
static const size_t GUEST_STACK_SIZE = 104;
//was GUEST_CTX_HOME, can't remove because that'd throw stack alignment off. instead, can be used as a temporary in sequences
static const size_t GUEST_SCRATCH64 = 80;
static const size_t GUEST_SCRATCH = 0;
//when profiling is on, this stores the nanosecond time at the start of the function
static const size_t GUEST_PROFILER_START = 80;
static const size_t GUEST_RET_ADDR = 88;
static const size_t GUEST_CALL_RET_ADDR = 96;
};

View File

@@ -1636,15 +1636,7 @@ Value* HIRBuilder::Div(Value* value1, Value* value2,
Value* HIRBuilder::MulAdd(Value* value1, Value* value2, Value* value3) {
ASSERT_TYPES_EQUAL(value1, value2);
ASSERT_TYPES_EQUAL(value1, value3);
#if 0
bool c1 = value1->IsConstant();
bool c2 = value2->IsConstant();
if (c1 && c2) {
Value* dest = CloneValue(value1);
dest->Mul(value2);
return Add(dest, value3);
}
#endif
Instr* i = AppendInstr(OPCODE_MUL_ADD_info, 0, AllocValue(value1->type));
i->set_src1(value1);
i->set_src2(value2);
@@ -1655,15 +1647,7 @@ Value* HIRBuilder::MulAdd(Value* value1, Value* value2, Value* value3) {
Value* HIRBuilder::MulSub(Value* value1, Value* value2, Value* value3) {
ASSERT_TYPES_EQUAL(value1, value2);
ASSERT_TYPES_EQUAL(value1, value3);
#if 0
bool c1 = value1->IsConstant();
bool c2 = value2->IsConstant();
if (c1 && c2) {
Value* dest = CloneValue(value1);
dest->Mul(value2);
return Sub(dest, value3);
}
#endif
Instr* i = AppendInstr(OPCODE_MUL_SUB_info, 0, AllocValue(value1->type));
i->set_src1(value1);
i->set_src2(value2);
@@ -1671,6 +1655,30 @@ Value* HIRBuilder::MulSub(Value* value1, Value* value2, Value* value3) {
return i->dest;
}
Value* HIRBuilder::NegatedMulAdd(Value* value1, Value* value2, Value* value3) {
ASSERT_TYPES_EQUAL(value1, value2);
ASSERT_TYPES_EQUAL(value1, value3);
Instr* i =
AppendInstr(OPCODE_NEGATED_MUL_ADD_info, 0, AllocValue(value1->type));
i->set_src1(value1);
i->set_src2(value2);
i->set_src3(value3);
return i->dest;
}
Value* HIRBuilder::NegatedMulSub(Value* value1, Value* value2, Value* value3) {
ASSERT_TYPES_EQUAL(value1, value2);
ASSERT_TYPES_EQUAL(value1, value3);
Instr* i =
AppendInstr(OPCODE_NEGATED_MUL_SUB_info, 0, AllocValue(value1->type));
i->set_src1(value1);
i->set_src2(value2);
i->set_src3(value3);
return i->dest;
}
Value* HIRBuilder::Neg(Value* value) {
Instr* i = AppendInstr(OPCODE_NEG_info, 0, AllocValue(value->type));
i->set_src1(value);

View File

@@ -214,6 +214,10 @@ class HIRBuilder {
Value* Div(Value* value1, Value* value2, uint32_t arithmetic_flags = 0);
Value* MulAdd(Value* value1, Value* value2, Value* value3); // (1 * 2) + 3
Value* MulSub(Value* value1, Value* value2, Value* value3); // (1 * 2) - 3
Value* NegatedMulAdd(Value* value1, Value* value2,
Value* value3); // -((1 * 2) + 3)
Value* NegatedMulSub(Value* value1, Value* value2,
Value* value3); // -((1 * 2) - 3)
Value* Neg(Value* value);
Value* Abs(Value* value);
Value* Sqrt(Value* value);
@@ -265,6 +269,7 @@ class HIRBuilder {
Value* AtomicAdd(Value* address, Value* value);
Value* AtomicSub(Value* address, Value* value);
void SetNJM(Value* value);
protected:
void DumpValue(StringBuffer* str, Value* value);
void DumpOp(StringBuffer* str, OpcodeSignatureType sig_type, Instr::Op* op);

View File

@@ -208,6 +208,12 @@ enum Opcode {
OPCODE_STORE_OFFSET,
OPCODE_LOAD,
OPCODE_STORE,
// chrispy: todo: implement, our current codegen for the unaligned loads is
// very bad
OPCODE_LVLX,
OPCODE_LVRX,
OPCODE_STVLX,
OPCODE_STVRX,
OPCODE_MEMSET,
OPCODE_CACHE_CONTROL,
OPCODE_MEMORY_BARRIER,
@@ -244,7 +250,9 @@ enum Opcode {
OPCODE_MUL_HI, // TODO(benvanik): remove this and add INT128 type.
OPCODE_DIV,
OPCODE_MUL_ADD,
OPCODE_NEGATED_MUL_ADD,
OPCODE_MUL_SUB,
OPCODE_NEGATED_MUL_SUB,
OPCODE_NEG,
OPCODE_ABS,
OPCODE_SQRT,
@@ -284,7 +292,8 @@ enum Opcode {
OPCODE_TO_SINGLE, // i could not find a decent name to assign to this opcode,
// as we already have OPCODE_ROUND. round double to float (
// ppc "single" fpu instruction result rounding behavior )
OPCODE_SET_NJM,
OPCODE_SET_NJM,
__OPCODE_MAX_VALUE, // Keep at end.
};

View File

@@ -464,6 +464,19 @@ DEFINE_OPCODE(
OPCODE_SIG_V_V_V_V,
OPCODE_FLAG_DISALLOW_CONSTANT_FOLDING)
DEFINE_OPCODE(
OPCODE_NEGATED_MUL_ADD,
"negated_mul_add",
OPCODE_SIG_V_V_V_V,
OPCODE_FLAG_DISALLOW_CONSTANT_FOLDING)
DEFINE_OPCODE(
OPCODE_NEGATED_MUL_SUB,
"negated_mul_sub",
OPCODE_SIG_V_V_V_V,
OPCODE_FLAG_DISALLOW_CONSTANT_FOLDING)
DEFINE_OPCODE(
OPCODE_NEG,
"neg",
@@ -692,4 +705,5 @@ DEFINE_OPCODE(
"set_njm",
OPCODE_SIG_X_V,
0
)
)

View File

@@ -613,10 +613,12 @@ class Value {
// returns true if every single use is as an operand to a single instruction
// (add var2, var1, var1)
bool AllUsesByOneInsn() const;
//the maybe is here because this includes vec128, which is untyped data that can be treated as float or int depending on the context
// the maybe is here because this includes vec128, which is untyped data that
// can be treated as float or int depending on the context
bool MaybeFloaty() const {
return type == FLOAT32_TYPE || type == FLOAT64_TYPE || type == VEC128_TYPE;
}
private:
static bool CompareInt8(Opcode opcode, Value* a, Value* b);
static bool CompareInt16(Opcode opcode, Value* a, Value* b);

View File

@@ -14,8 +14,8 @@
#include <mutex>
#include <string>
#include "xenia/base/vec128.h"
#include "xenia/base/mutex.h"
#include "xenia/base/vec128.h"
namespace xe {
namespace cpu {
class Processor;
@@ -390,9 +390,11 @@ typedef struct alignas(64) PPCContext_s {
// These are split to make it easier to do DCE on unused stores.
uint64_t cr() const;
void set_cr(uint64_t value);
// todo: remove, saturation should be represented by a vector
uint8_t vscr_sat;
uint32_t vrsave;
// uint32_t get_fprf() {
// return fpscr.value & 0x000F8000;
// }

View File

@@ -1197,7 +1197,7 @@ int InstrEmit_vnmsubfp_(PPCHIRBuilder& f, uint32_t vd, uint32_t va, uint32_t vb,
Value* b = f.VectorDenormFlush(f.LoadVR(vb));
Value* c = f.VectorDenormFlush(f.LoadVR(vc));
Value* v = f.Neg(f.MulSub(a, c, b));
Value* v = f.NegatedMulSub(a, c, b);
f.StoreVR(vd, v);
return 0;
}

View File

@@ -16,6 +16,12 @@
#include "xenia/cpu/ppc/ppc_hir_builder.h"
#include <stddef.h>
// chrispy: added this, we can have simpler control flow and do dce on the
// inputs
DEFINE_bool(ignore_trap_instructions, true,
"Generate no code for powerpc trap instructions, can result in "
"better performance in games that aggressively check with trap.",
"CPU");
namespace xe {
namespace cpu {
@@ -449,6 +455,9 @@ constexpr uint32_t TRAP_SLT = 1 << 4, TRAP_SGT = 1 << 3, TRAP_EQ = 1 << 2,
int InstrEmit_trap(PPCHIRBuilder& f, const InstrData& i, Value* va, Value* vb,
uint32_t TO) {
if (cvars::ignore_trap_instructions) {
return 0;
}
// if (a < b) & TO[0] then TRAP
// if (a > b) & TO[1] then TRAP
// if (a = b) & TO[2] then TRAP
@@ -521,6 +530,9 @@ int InstrEmit_trap(PPCHIRBuilder& f, const InstrData& i, Value* va, Value* vb,
}
int InstrEmit_td(PPCHIRBuilder& f, const InstrData& i) {
if (cvars::ignore_trap_instructions) {
return 0;
}
// a <- (RA)
// b <- (RB)
// if (a < b) & TO[0] then TRAP
@@ -534,6 +546,9 @@ int InstrEmit_td(PPCHIRBuilder& f, const InstrData& i) {
}
int InstrEmit_tdi(PPCHIRBuilder& f, const InstrData& i) {
if (cvars::ignore_trap_instructions) {
return 0;
}
// a <- (RA)
// if (a < EXTS(SI)) & TO[0] then TRAP
// if (a > EXTS(SI)) & TO[1] then TRAP
@@ -546,6 +561,9 @@ int InstrEmit_tdi(PPCHIRBuilder& f, const InstrData& i) {
}
int InstrEmit_tw(PPCHIRBuilder& f, const InstrData& i) {
if (cvars::ignore_trap_instructions) {
return 0;
}
// a <- EXTS((RA)[32:63])
// b <- EXTS((RB)[32:63])
// if (a < b) & TO[0] then TRAP
@@ -561,6 +579,9 @@ int InstrEmit_tw(PPCHIRBuilder& f, const InstrData& i) {
}
int InstrEmit_twi(PPCHIRBuilder& f, const InstrData& i) {
if (cvars::ignore_trap_instructions) {
return 0;
}
// a <- EXTS((RA)[32:63])
// if (a < EXTS(SI)) & TO[0] then TRAP
// if (a > EXTS(SI)) & TO[1] then TRAP
@@ -645,7 +666,9 @@ int InstrEmit_mfspr(PPCHIRBuilder& f, const InstrData& i) {
break;
case 256:
// VRSAVE
v = f.LoadZeroInt64();
v = f.ZeroExtend(f.LoadContext(offsetof(PPCContext, vrsave), INT32_TYPE),
INT64_TYPE);
break;
case 268:
// TB
@@ -749,6 +772,8 @@ int InstrEmit_mtspr(PPCHIRBuilder& f, const InstrData& i) {
f.StoreCTR(rt);
break;
case 256:
f.StoreContext(offsetof(PPCContext, vrsave), f.Truncate(rt, INT32_TYPE));
// VRSAVE
break;
default:
@@ -768,6 +793,7 @@ int InstrEmit_mfmsr(PPCHIRBuilder& f, const InstrData& i) {
// bit 48 = EE; interrupt enabled
// bit 62 = RI; recoverable interrupt
// return 8000h if unlocked (interrupts enabled), else 0
#if 0
f.MemoryBarrier();
if (cvars::disable_global_lock || true) {
f.StoreGPR(i.X.RT, f.LoadConstantUint64(0));
@@ -777,63 +803,23 @@ int InstrEmit_mfmsr(PPCHIRBuilder& f, const InstrData& i) {
f.StoreGPR(i.X.RT,
f.LoadContext(offsetof(PPCContext, scratch), INT64_TYPE));
}
#else
f.StoreGPR(i.X.RT, f.LoadConstantUint64(0));
#endif
return 0;
}
int InstrEmit_mtmsr(PPCHIRBuilder& f, const InstrData& i) {
if (i.X.RA & 0x01) {
// L = 1
// iff storing from r13
f.MemoryBarrier();
f.StoreContext(
offsetof(PPCContext, scratch),
f.ZeroExtend(f.ZeroExtend(f.LoadGPR(i.X.RT), INT64_TYPE), INT64_TYPE));
#if 0
if (i.X.RT == 13) {
// iff storing from r13 we are taking a lock (disable interrupts).
if (!cvars::disable_global_lock) {
f.CallExtern(f.builtins()->enter_global_lock);
}
} else {
// Otherwise we are restoring interrupts (probably).
if (!cvars::disable_global_lock) {
f.CallExtern(f.builtins()->leave_global_lock);
}
}
#endif
return 0;
} else {
// L = 0
XEINSTRNOTIMPLEMENTED();
return 1;
}
f.StoreContext(
offsetof(PPCContext, scratch),
f.ZeroExtend(f.ZeroExtend(f.LoadGPR(i.X.RT), INT64_TYPE), INT64_TYPE));
return 0;
}
int InstrEmit_mtmsrd(PPCHIRBuilder& f, const InstrData& i) {
if (i.X.RA & 0x01) {
// L = 1
f.MemoryBarrier();
f.StoreContext(offsetof(PPCContext, scratch),
f.ZeroExtend(f.LoadGPR(i.X.RT), INT64_TYPE));
#if 0
if (i.X.RT == 13) {
// iff storing from r13 we are taking a lock (disable interrupts).
if (!cvars::disable_global_lock) {
f.CallExtern(f.builtins()->enter_global_lock);
}
} else {
// Otherwise we are restoring interrupts (probably).
if (!cvars::disable_global_lock) {
f.CallExtern(f.builtins()->leave_global_lock);
}
}
#endif
return 0;
} else {
// L = 0
XEINSTRNOTIMPLEMENTED();
return 1;
}
f.StoreContext(offsetof(PPCContext, scratch),
f.ZeroExtend(f.LoadGPR(i.X.RT), INT64_TYPE));
return 0;
}
void RegisterEmitCategoryControl() {

View File

@@ -195,8 +195,8 @@ int InstrEmit_fmsubsx(PPCHIRBuilder& f, const InstrData& i) {
int InstrEmit_fnmaddx(PPCHIRBuilder& f, const InstrData& i) {
// frD <- -([frA x frC] + frB)
Value* v = f.Neg(
f.MulAdd(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC), f.LoadFPR(i.A.FRB)));
Value* v = f.NegatedMulAdd(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC),
f.LoadFPR(i.A.FRB));
f.StoreFPR(i.A.FRT, v);
f.UpdateFPSCR(v, i.A.Rc);
return 0;
@@ -204,8 +204,8 @@ int InstrEmit_fnmaddx(PPCHIRBuilder& f, const InstrData& i) {
int InstrEmit_fnmaddsx(PPCHIRBuilder& f, const InstrData& i) {
// frD <- -([frA x frC] + frB)
Value* v = f.Neg(
f.MulAdd(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC), f.LoadFPR(i.A.FRB)));
Value* v = f.NegatedMulAdd(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC),
f.LoadFPR(i.A.FRB));
v = f.ToSingle(v);
f.StoreFPR(i.A.FRT, v);
f.UpdateFPSCR(v, i.A.Rc);
@@ -214,8 +214,8 @@ int InstrEmit_fnmaddsx(PPCHIRBuilder& f, const InstrData& i) {
int InstrEmit_fnmsubx(PPCHIRBuilder& f, const InstrData& i) {
// frD <- -([frA x frC] - frB)
Value* v = f.Neg(
f.MulSub(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC), f.LoadFPR(i.A.FRB)));
Value* v = f.NegatedMulSub(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC),
f.LoadFPR(i.A.FRB));
f.StoreFPR(i.A.FRT, v);
f.UpdateFPSCR(v, i.A.Rc);
return 0;
@@ -223,8 +223,8 @@ int InstrEmit_fnmsubx(PPCHIRBuilder& f, const InstrData& i) {
int InstrEmit_fnmsubsx(PPCHIRBuilder& f, const InstrData& i) {
// frD <- -([frA x frC] - frB)
Value* v = f.Neg(
f.MulSub(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC), f.LoadFPR(i.A.FRB)));
Value* v = f.NegatedMulSub(f.LoadFPR(i.A.FRA), f.LoadFPR(i.A.FRC),
f.LoadFPR(i.A.FRB));
v = f.ToSingle(v);
f.StoreFPR(i.A.FRT, v);
f.UpdateFPSCR(v, i.A.Rc);

View File

@@ -834,6 +834,7 @@ int InstrEmit_stdcx(PPCHIRBuilder& f, const InstrData& i) {
// Issue memory barrier for when we go out of lock and want others to see our
// updates.
f.MemoryBarrier();
return 0;

View File

@@ -77,7 +77,8 @@ ThreadState::ThreadState(Processor* processor, uint32_t thread_id,
// Allocate with 64b alignment.
context_ = reinterpret_cast<ppc::PPCContext*>(AllocateContext()); // memory::AlignedAlloc<ppc::PPCContext>(64);
context_ = reinterpret_cast<ppc::PPCContext*>(
AllocateContext());
processor->backend()->InitializeBackendContext(context_);
assert_true(((uint64_t)context_ & 0x3F) == 0);
std::memset(context_, 0, sizeof(ppc::PPCContext));
@@ -93,6 +94,7 @@ ThreadState::ThreadState(Processor* processor, uint32_t thread_id,
// Set initial registers.
context_->r[1] = stack_base;
context_->r[13] = pcr_address;
// fixme: VSCR must be set here!
}
ThreadState::~ThreadState() {
@@ -105,7 +107,7 @@ ThreadState::~ThreadState() {
if (context_) {
FreeContext(reinterpret_cast<void*>(context_));
}
// memory::AlignedFree(context_);
// memory::AlignedFree(context_);
}
void ThreadState::Bind(ThreadState* thread_state) {