This month we worked on some optimizations and a couple accuracy improvements.

Superblocks

Until now, felix86 would end each block on conditional jump instructions like JZ or JA. Each conditional jump would look something like this:

beqz condition, label_false
label_true:
j block_true
label_false:
j block_false

Conditions would then be constructed from the flags. However, on RISC-V there’s no flags, and these sorts of comparisons can happen directly in the branch instruction. This way, an instruction sequence such as this one:

cmp rax, rbx
jl block_less
add rcx, 1
ret

block_less:
add rdx, 1
ret

can roughly be translated like so:

blt x5, x8, stub_block_less
addi x26, x26, 1
ret

stub_block_less:
// Back to dispatcher...

Here we continue compiling past the conditional jump. We don’t compile block_less, it will get compiled if and when that path is taken. Additionally, superblocks allow for future optimizations such as lifting the x87 stack to registers across multiple blocks.

Benchmarks

After posting the 26.09 benchmark measurements, we noticed that the native zstd benchmarks (which had surprising % of native performance compared to the rest) were using multiple byte loads, because the compiler was assuming they are faster than unaligned 8-byte loads when they really weren’t. For this reason, the benchmarks from 26.09 were recompiled using -mtune=generic-ooo (which makes the native version faster), remeasured, and compared with felix86 26.11.

The following runs are on a SpacemiT K3 board, pinned to a single core, and all of them except Stockfish do not start any threads.

By calculating the geometric mean, we find that felix86 26.11 is 22% faster than felix86 26.09 in these benchmarks.

Self-validating blocks

Our SMC handling code would protect the page or pages containing the x86 code we recompiled as read-only. Upon write, a SIGSEGV happens and the writer marks the blocks as invalid.

However, there were two issues. First, if the instructions a thread is compiling get overwritten before it finishes, it would publish an outdated block. We now ensure there’s no such race. The compiling thread will keep track of atomic counters of the page’s SMC hits, and then protect the pages. After compiling the block, it will re-check the counters. If they are different, it will discard the recompiled code and try again.

The second issue was one of performance. In some games, writeable data is mixed with executable code. In previous felix86 versions, those games would cause constant invalidations on those pages. While we could track the block start and end and only invalidate blocks that are actually written to, it would still mean that every write has to go through a fault and scan of all blocks in the page, which is very expensive.

Instead, felix86 will now switch to a different SMC strategy for hot pages, called self-validating blocks. When a block crosses a threshold of invalidations, it will turn into a self-validating block. Upon entry, it will scan its own code and ensure that it matches what it was when the block was compiled. If not, it invalidates itself.

This means that writes to those hot pages will stop triggering invalidations. In the game Need for Speed: Heat, self-validating blocks brought invalidations from 3000 blocks per second to 200, which noticeably reduces stutter.

The thresholds can be configured with FELIX86_SMC_THRESHOLD and FELIX86_SMC_WINDOW_MS. By default, a block will turn self-validating if SMC happens 10 times on it, each SMC hit being a maximum of 1 second apart.

The downside of self-validating blocks, of course, is the overhead of the block checking its own bytes on entry. Our code is vectorized to compare multiple bytes at once to reduce this overhead.

FIP emulation

We fixed an issue with emulation of SecuROM games, such as Split/Second. This game reads the (very rarely used) x87 FPU instruction pointer (FIP) as some sort of DRM protection. This is now properly emulated in felix86, allowing the game to work!

Split/Second on RISC-V, running via felix86

Accurate reciprocal instructions

As a side-quest, we fixed a bug that has existed for more than 3 years in shadPS4. Since PS4 games are compiled for a specific AMD CPU, sometimes they would ship with issues that would happen on Intel CPUs that were not visible until PS4 emulators came around.

Two such bugs were known, one was the character model being deformed in Dark Souls Remastered, and one was bad lighting in the DLC areas of Bloodborne. These issues were caused by inexact VRCPPS and VRSQRTPS emulation. These instructions produce an approximate reciprocal or reciprocal square root, whose approximation algorithm is not defined and is left up to implementation. It turns out that Intel and AMD CPUs each have their own algorithms that have been the same across multiple generations of CPUs. A program that, intentionally or not, relies on exact AMD behavior would thus fail to work on an Intel CPU. In fact, it turns out that emulating these with more accuracy than either CPU (e.g. by doing a full 1/x or 1/sqrt(x) calculation) doesn’t solve the issues – they need exact AMD CPU behavior.

We fixed this bug in shadPS4, by patching the reciprocal instructions to do a correction on the Intel results. The reciprocal results only rely on the top 21 bits of the input, and when all 2^32 possible values are XORed between the two processors, there are only 17 distinct values. This allows us to build a 17-entry table of XOR values and a 2^21 index table. The index table that consists of single byte indices ends up being only a kilobyte compressed and a couple megabytes uncompressed. A similar table is then used for reciprocal square root. You can view the full PR here.

Eventually this level of accuracy will be implemented in felix86. It is less important in our case, as PC games usually target both Intel and AMD CPUs so these sorts of Intel-only or AMD-only bugs don’t fly under the radar. In ARM land there’s FEAT_RPRES which “increases the reciprocal precision”, (really it makes it be Intel’s implementation) so perfect x86 reciprocal emulation is easy on hardware with that extension. It is extremely unlikely we’ll get anything similar on RISC-V in the standard, so ARM will always have such architectural advantages over RISC-V when it comes to x86 emulation.

Thanks for reading this post.

If you like this project, please give us a star on GitHub: https://github.com/OFFTKP/felix86