fix: reduce address space pressure around dynarec JIT caches

[apommel]

That was a funny one to debug.

On Miyoo Flip, starting a game through gpSP sometimes crashes, seemingly randomly, and the log does not give that much information, except that it happens in retro_load_game.

After several attempts at reproducing the crash, I was able to obtain a kernel fault trace, and the problem was that gpSP could not allocate its JIT translation cache. It probes for a free 10.5 MB RWX block during retro_init, and returns NULL if it finds none. It doesn't check the result, so the first code emission during retro_load_game writes through a NULL pointer.

The JIT translation cache must be within plus or minus 128 MB of the core's text (direct-branch range) and at this point this address space is mostly consumed by two things: glibc malloc arenas and 8 MB thread stacks, plus the shared libraries. In some cases there's not any appropriate address block left, and it crashes.

I believe this does not happen on tg5040, because the GLES driver is a 1.5 MB library. On my355 and tg5050 it is libmali, which maps 54 MB and 47 MB of address space respectively. As it is sightly smaller on tg5050, it may allow it to mostly avoid this issue by a fine margin, but it could actually be also exposed.

This PR implements rather a workaround than a fix: reduce address space pressure around the JIT cache. To do that, I changed the following things:

• Use a 1 MB stack size for threads instead of the default 8 MB.
• Set the amount of maximum malloc arenas to 2 (and all threads should allocate into those).

After these changes, I did not notice any further crashes. I checked for performance impacts (reducing the number of arenas could impact the speed of memory allocations in multi-threaded applications), but I did not notice anything.

[apommel]

I forgot to mention in the main post but the issue is probably related to this one in gpSP: libretro/gpsp#248.

[frysee]

Now that you mention it, I think Ive seen this or a similar issue on tg5050 with Advance Wars 2. Not sure if the ROM matters, but it happened a few times but I wasnt able to track down what causes it.

[apommel]

From what I see, the game being loaded should not have any direct impact, as the failure happens before it's loaded in memory.

But we never know, there might be indirect things that make it more likely to happen with a specific ROM.

[frysee]

I'm assuming the chances of an upstream fix are rather slim - is this something we could patch?

Otherwise I dont have any strong feeling against that, other than maybe referencing back to this issue from the comment in code (easier than blaming down the line).

[apommel]

I have given as many details as I can upstream, but the only 'easy fixes' there also seem to be correctness or workarounds rather than true fixes, which would require a bigger refactoring, and may have potential side effects. As the repository does not seem to be very active, even in the best case scenario, I don't think that any fix would be landing soon there, or that we want to have to maintain such core patches in NextUI.

So from my point of view, as our only real direct lever is working around the problem, we may as well do it the 'simple' way as I did here. Also, I don't know if there are any other cores with the same JIT design, but it could be beneficial for those too.

I can definitely add references to the specific issues in the comments for clarity.