The goal of the separate stacks was to allow this, but I never quite finished the job. Now, when a SEH exception (generally a Rust panic in a guest syscall handler, or a C# Exception in a callback) tries to unwind through guest code, it works. Note that we don't actually unwind the guest stack, as there's nothing useful to be gained from that; When an emulator core throws an exception like this, it should be considered completely hosed. Throw it out and get a new one.
There were two bugs stopping this from working.
First of all, we had custom thunks that lacked sufficient unwind information for RtlUnwind to get through. For the sysv <-> msabi adapter, this was fixed by making it regular Rust code instead of hand assembled junkus. So the compiler generates valid unwind information for all of that. Then we just JIT a small stub on top in the MsHostSysVGuest code, which needs no unwind information because it won't throw an exception itself and transparently passes execution to something with valid unwind information without invalidating that information. (NB: Clr JIT stubs use the same strategy.) For the host <-> guest stack transition code, a small hand generated unwind stub was added to interop.s that is registered with `RtlAddFunctionTable`. I've seen the unwind work successfully without this second set of unwind information, but better safe than sorry.
Secondly, our misuse of SubSystemTib caught up with us. It's an old field, allegedly from OS/2, that we repurposed to hold TLS information needed for the waterbox stack transitions. Most people think nothing uses it any more, but in fact if it's set to a non-NULL value, but doesn't contain valid information, `KERNELBASE!GetModuleFileNameW` will crash when it tries to get a module name from there. The fix here was to simply tighten up our usage of SubSystemTib: We were already nulling it out when returning from guest code, but not when calling back to host code in guest code.
Fixes#2487. Unwinding of this sort has never worked well in waterbox; the reason why that issue is more recent is that the particular reproducing case of firmware didn't cause an exception in a callback in older code; the exception happened in pure managed code.
72ea5ce866