The description in the previous commit is accurate, but the problem runs deeper and was on the whole a complete failure for me to appreciate the difference between active and swapped in on memoryblocks. Bleeecch.
This was broken by 175556529e, with two related issues: When we allowed for some operations to happen even when the block is inactive, we didn't account for the fact that in swapin, the block technically is not active yet (the lock is not on the self), and similarly in swapout, the lock has already been moved out of self. The former caused all memory areas to revert to RWX at the host OS level after a swap, so no dirty detection was done. After the former was fixed, the latter caused saved costacks to still get missed.
At the same time we ran into a perfect storm with costacks on Windows; if a stack page is not yet dirty, but we hit a fault for something else, Windows will not call our VEH handler unless the TIB stack extents are satisfactory, since it needs userspace to fix up the TIB extents via VEH or SEH handler, but there's already an exception pending.
The compiler now can fully inline the co_switch, and with most registers being specified as clobbers and not saved explicitly, the compiler can choose to save only what it needs to (we don't have to defensively save everything).
Practically speaking, the co_switch calls are usually inlined, but the functions they're in don't seem to be that big and don't make direct use of r12..r15 too much anyway, so (push r12..r15, switch, pop r12..r15) is a common emit. But I see a miniscule FPS increase.
This broke any waterbox core that called in to native code in the same EnterExit() right after sealing. All nyma cores were broken, 32x was not, didn't check the rest. Regressed in 175556529e.
It worked fine in release mode, theoretically
Set up a second mirror of guest memory; easily accomplished because we were already using memfd_create / CreateFileMappingW.
This lets us simplify a lot of host code that has to access guest memory that may not be active right now, or might have been mprotect()ed to something weird. Activate is only needed now to run guest code, or when the C# side wants to peer into guest memory for memory domains and such (waterboxhost does not share the mirror address with the C# side).
adelikat