[x64] vcfux single rounding for 0x80000000+

This commit is contained in:
Triang3l
2020-12-11 21:20:13 +03:00
parent d0b849aad7
commit 5c47a3a588
4 changed files with 85 additions and 10 deletions

View File

@@ -143,6 +143,55 @@ enum Opcode {
OPCODE_TRUNCATE,
OPCODE_CONVERT,
OPCODE_ROUND,
// Note that 2147483648.0 + (src & 0x7FFFFFFF) is not a correct way of
// performing the uint -> float conversion for large numbers on backends where
// only sint -> float is available.
//
// Take 0b11000000000000000000000101000001 as an example,
// or 1.1000000000000000000000101000001 * 2^31.
// This one has 31 mantissa bits (excluding the implicit 1.), and needs to be
// rounded to 23 bits - 8 mantissa bits need to be dropped:
// 10000000000000000000001_01000001
//
// Rounding to the nearest even (the only rounding mode that exists on
// AltiVec, and the likely rounding mode in the implementations) should be
// done downwards - 01000001 of 1_01000001 is in [00000000, 01111111].
// The correct mantissa in this case is:
// 1.10000000000000000000001 * 2^31.
//
// With a two-step conversion, rounding is done twice instead, which gives an
// incorrect result.
//
// First, converting the low 31 bits to float:
// The number is 0.1000000000000000000000101000001 * 2^31.
// Normalizing it, we get 1.000000000000000000000101000001 (30 significand
// bits).
// We need to round 30 bits to 23 - 7 bits need to be dropped:
// 00000000000000000000010_1000001
//
// Rounding to the nearest even is done upwards in this case - 1000001 of
// 0_1000001 is in [1000001, 1111111].
// The result of the sint -> float conversion is:
// 1.00000000000000000000011 * 2^30.
//
// Now 2147483648.0 (1 * 2^31) needs to be added. Aligning the exponents, we
// get:
// 0.|10000000000000000000001|1 * 2^31
// + 1.|00000000000000000000000| * 2^31
// = 1.|10000000000000000000001|1 * 2^31
//
// At "infinite precision", the result has 24 significand bits, but only 23
// can be stored, thus rounding to the nearest even needs to be done. 1_1 is
// (odd + 0.5). 0.5 is ambiguous, thus tie-breaking to the nearest even -
// which is above in this case - is done. The result is:
// 1.10000000000000000000010 * 2^31.
//
// This is incorrect - larger than the correctly rounded result, which is:
// 1.10000000000000000000001 * 2^31.
//
// Test cases checked on real hardware via vcfux: 0xFFFDFF7E, 0xFFFCFF7D -
// should be 0x4F7FFDFF and 0x4F7FFCFF respectively, not 0x4F7FFE00 and
// 0x4F7FFD00.
OPCODE_VECTOR_CONVERT_I2F,
OPCODE_VECTOR_CONVERT_F2I,
OPCODE_LOAD_VECTOR_SHL,