The console appears to behave against standard IEEE754 specification
here, in particular around how NaNs are handled. NaNs appear to have no
effect on the result, and are treated the same as positive or negative
infinity, based on the sign bit.
However, when the result would be NaN (inf - inf, or (-inf) - (-inf)),
this results in a completely fogged color, or unfogged color
respectively. We handle this by returning a constant zero for the A
varaible, and positive or negative infinity for C depending on the sign
bits of the A and C registers. This ensures that no NaN value is passed
to the GPU in the first place, and that the result of the fog
calculation cannot be NaN.
This will generate one shader per copy format. For now, it is the same
shader with the colmat hard coded. So it should already improve the GPU
performance a bit, but a rewrite of the shader generator is suggested.
Half of the patch is done by linkmauve1:
VideoCommon: Reorganise the shader writes.
Also skips swapping the window system buffers in headless mode, as there
may not be a surface which can be swapped in the first place. Instead,
we call glFlush() at the end of a frame in this case.
Cel-damage uses the color from the lighting stage of the vertex pipeline
as texture coordinates, but sets numColorChans to zero.
We now calculate the colors in all cases, but override the color before
writing it from the vertex shader if numColorChans is set to a lower value.
Ideally Common.h wouldn't be a header in the Common library, and instead be renamed to something else, like PlatformCompatibility.h or something, but even then, there's still some things in the header that don't really fall under that label
This moves the version strings out to their own version header that doesn't dump a bunch of other unrelated things into scope, like what Common.h was doing.
This also places them into the Common namespace, as opposed to letting them sit in the global namespace.
SetFormat() is only ever used internally. ResetBuffer() is only
used to implement the VertexManagerBase class interface, so
there's no need to make it protected.
The casts to u32* are technically undefined behavior. The u8* cast is
left, as char/unsigned char is exempted from this rule to allow for
bvtewise inspection of objects (and this is what s8/u8 are typedefs of
on platforms we support).
Improve bookkeeping around formats. Hopefully make code less confusing.
- Rename TlutFormat -> TLUTFormat to follow conventions.
- Use enum classes to prevent using a Texture format where an EFB Copy format
is expected or vice-versa.
- Use common EFBCopyFormat names regardless of depth and YUV configurations.
Stenzek