Applies factors before the MIN/MAX operation - in FSI/ROV via runtime
blending, in FBO/RTV by pre-multiplying shader output when dstFactor is
ONE only, as we don't have access to dest to provide full support as we
do in the FSI/ROV paths.
With render target HLE, directly store linear values as R16G16B16A16_UNORM
without gamma conversion, as this format provides more than enough bits
(need at least 11 per component due to the maximum scale being 2^3 in the
piecewise linear gamma curve) to represent linear values without precision
loss.
This makes blending work correctly in linear space, improving quality of
transparency, lighting passes, and fixing issues such as transparent parts
of impact and footstep decals in 4D5307E6 being bright instead.
The new behavior is enabled by default, as it hugely improves the accuracy
of emulation of this format, that is pretty commonplace in Xbox 360 games,
with likely just a small GPU memory and bandwidth usage increase, compared
to the alternatives that were previously available on the HLE RB path.
It's currently implemented only on Direct3D 12, as most of the current GPU
emulation code is planned to be phased out and redone, and no methods other
than 8-bit with pre-conversion were implemented on Vulkan previously.
To implement on Vulkan later, same conversion as in the Direct3D 12
implementation will need to be done in ownership transfer and resolve
shaders. Currently it's somewhat inconvenient to decouple the conversion
functions in `SpirvShaderTranslator` from an instance of the translator due
to vector constant usage. Later, simpler SPIR-V generation functions may be
added (`spv::Builder` usage in general is overly verbose).
The previously default method (8-bit storage with pre-conversion in shaders
and incorrect blending) can be re-enabled by setting the
"gamma_render_target_as_unorm16" configuration option to `false`. This may
be useful if the game, for instance, switches between 8_8_8_8_GAMMA and
8_8_8_8 formats for the same data frequently, as switching will result in
EDRAM range ownership transfer data copying now. Also, the old path is
preserved for Vulkan devices not supporting R16G16B16A16_UNORM with
blending.
The other workaround that was available previously, replacing the PWL
encoding with host hardware sRGB with linear-space blending in render
target management and in texture fetching, was also inherently inaccurate
in many ways (especially when games have their own PWL encoding math, like
4541080F that displayed incorrect colors on the loading screen), and
required tracking of the encoding needed for ranges in the memory.
The sRGB workaround therefore was deleted in this commit, greatly
simplifying the code in the parts of render target, texture and memory
management and shader generation that were involved in it.
Also fixes addressing of MSAA samples 2 and 3 for 64bpp color render targets in the ROV RB implementation on Direct3D 12.
Additionally, with FSI/ROV, alpha test and alpha to coverage are done only if the render target 0 was dynamically written to (according to the Direct3D 9 rules for writing to color render targets, though not sure if they actually apply to the alpha tests on Direct3D 9, but for safety).
There is also some code cleanup for things spotted during the development of the feature.
For more optimal usage of exports and the parameter cache on the host regardless of how effective the optimizations in the host GPU driver are. Also reserve space for Vulkan/Metal/D3D11-specific HostVertexShaderTypes to use one more bit for the host vertex shader type in the shader modification bits, so that won't have to be done in the future as that would require invalidating shader storages (which are invalidated by this commit) again.
