nv2a: Ensure SurfaceBinding.size can always fit surface

It is possible to specify a pitch that is smaller than the native size of a
surface (i.e., `pitch < width * bytes_per_pixel`). On hardware this is handled
gracefully, whereas in xemu it will cause buffer overruns when dealing with
swizzling/unswizzling.

Fixes #1152

[Test](https://github.com/abaire/nxdk_pgraph_tests/blob/main/src/tests/surface_pitch_tests.cpp)
[HW Results](https://github.com/abaire/nxdk_pgraph_tests_golden_results/wiki/Results-Surface_pitch)

Note that xemu's swizzling behavior still does not match HW with this change,
it just prevents the test from crashing.
This commit is contained in:
Erik Abair 2022-07-02 09:06:09 -07:00 committed by mborgerson
parent d219d92214
commit f7b2acbb79
1 changed files with 3 additions and 4 deletions

View File

@ -5684,15 +5684,14 @@ static void pgraph_download_surface_data_to_buffer(NV2AState *d,
* FIXME: Consider swizzle in shader
*/
assert(pg->surface_scale_factor == 1 || downscale);
swizzle_buf = (uint8_t *)g_malloc(surface->height * surface->pitch);
swizzle_buf = (uint8_t *)g_malloc(surface->size);
gl_read_buf = swizzle_buf;
}
if (downscale) {
pg->scale_buf = (uint8_t *)g_realloc(
pg->scale_buf, pg->surface_scale_factor * pg->surface_scale_factor *
surface->height * surface->pitch *
surface->fmt.bytes_per_pixel);
surface->size);
gl_read_buf = pg->scale_buf;
}
@ -6000,7 +5999,7 @@ static void pgraph_populate_surface_binding_entry_sized(NV2AState *d,
entry->width = width;
entry->height = height;
entry->pitch = surface->pitch;
entry->size = height * surface->pitch;
entry->size = height * MAX(surface->pitch, width * fmt.bytes_per_pixel);
entry->upload_pending = true;
entry->download_pending = false;
entry->draw_dirty = false;