AVX/AVX2 support is here!

This version of felix86 emulates all AVX and AVX2 instructions using RVV 1.0! Thanks to the FEX-Emu test suite, each instruction is extensively tested and all tests pass.

AVX is an x86 SIMD extension that adds 256-bit vectors and new instructions. Additionally, it introduces a new version for most of the SSE instructions that can use the 256-bit vectors and perform unaligned access, amongst other benefits.

Overall, 288 new instruction handlers were added, and 333 single-instruction tests pass. Binary tests can now detect AVX support and pass.

Fun parts

Here’s some fun parts of AVX emulation.

VZEROALL, VZEROUPPER

The RVV 1.0 statically allocated registers for YMM0-YMM15 were changed to v16-v31. This means we can use LMUL=8 to perform VZEROALL as two grouped VXOR instructions, and VZEROUPPER as two grouped and masked VXOR instructions.

Group writeback/restore

Since the XmmReg struct is now 256 bits, we can writeback and restore the XMM state in two LMUL=8 stores/loads, in VLEN=256 systems. This may improve performance in some applications that end up exiting to the dispatcher too often.

VGATHER

The VGATHER set of instructions in x86 (not to be confused with RISC-V’s VRGATHER) are basically indexed loads that work slightly differently from RISC-V. The indices may be optionally scaled and may need to be sign-extended before passed to the equivalent RISC-V instruction.

With the Zicclsm extension, which is mandatory in RVA23, indices that end up performing a misaligned load are supported, without needing to resort to a potentially slower instruction sequence.

Currently, there’s no support for VGATHER instructions that may fault during one of the element loads. We’ll implement this feature when there’s a program that relies on it.

Annoying parts

Below are a couple annoyances with emulating AVX on RVV.

No zeroing of agnostic bits

RVV 1.0 has two modes of dealing with tail and mask bits: Undisturbed and agnostic.

Undisturbed mode doesn’t affect the tail and mask bits. This is helpful for SSE emulation, since the legacy SSE instructions don’t modify the top 128 bits of the YMM registers.

Agnostic mode does something weird. Based on implementation, it will work the same as undisturbed, or replace the bits with ones. In version RVV 0.7.1, the bits were replaced with zeroes. This was changed in RVV 1.0 to replacing with ones, with the justification:

The value of all 1s instead of all 0s was chosen for the overwrite value to discourage software developers from depending on the value written.

This is unfortunate, because it means that there’s no instruction-free way of zeroing the tail or mask bits. Thus, performing the zeroing requires 1-3 extra instructions.

This zeroing of tail and mask bits is important for emulating AVX instructions that deal with XMMs, as they need to zero the upper 128 bits. It is also important for instructions like VMASKMOVPD, which zero the masked elements. It will also be important when/if we emulate AVX-512, as every instruction can perform similar zeroing masking.

Unfortunately, since RVV 1.0 is ratified, changing the agnostic bits to be zeroes instead of ones is not possible for software compatibility reasons. We hope there’s a mode that allows for zeroing agnostic bits in a future version, perhaps via a bit in vtype.

Cumbersome shuffles

VRGATHER is a powerful instruction in RISC-V, however it has some limitations. The iota cannot be an immediate, except in the case of VRGATHER.VI, which can only be used to broadcast a single element. The iota also cannot be picked from bits of a GPR. This means the iota needs to be loaded from a literal pool in memory in most cases.

An optimal instruction would be able to perform shuffles between two sources, using an immediate iota, similar to x86 shuffles. This would almost certainly need to be a >32-bit instruction, which RISC-V is capable of.

Until then, we need to use literal pools and two VRGATHER instructions to emulate most VSHUFPS instructions.

BMI1 support

BMI1 is a bit manipulation extension. Most of its instructions don’t match one-to-one with RISC-V equivalents, but are relatively simple to emulate.

F16C support

F16C is an extension that adds conversions from 16-bit floats to 32-bit floats. It is equivalent to the Zvfhmin extension, and is now supported when the extension is present, which is mandatory in RVA23.

Flatpaks

This month we also added initial support for Flatpaks!

The Hytale launcher, which is a Flatpak, running with felix86

Unfortunately there may be unrelated bugs with Hytale, causing it to not work. Nevertheless, Flatpaks should be able to be installed now.

In felix86 --shell, use flatpak install --user /path/to/my/game.flatpak to install and flatpak run com.MyOrg.MyGame to run your Flatpak app. System-wide installations are not tested.

Preliminary seccomp support

Flatpaks needed seccomp to run, particularly the filter mode. The BPF is now supported and recompiled to RISC-V and validates each syscall. The support is limited to the scope of supporting Flatpaks, and not all filters are supported.

AES support

The AES extension in RISC-V adds hardware acceleration for AES encryption and decryption. It is now supported in felix86!

The encryption instructions AESENC and AESENCLAST match up perfectly with the RISC-V equivalents. The decryption instruction AESDECLAST also matches up 1-to-1.

The AESDEC instruction performs InvMixColumns and AddRoundKey in the opposite order that VAESDM.VV does. My initial solution was to apply MixColumns to the key (by applying InvMixColumns via AES64IM 3 times in a row for each 64-bit word in the key) and then use VAESDM.VV.

As camel-cdr pointed out, there’s a much better solution, which is to use VAESDF.VV with a key of zeroes to apply the InvShiftRows and InvSubBytes transformations without AddRoundKey. Then, using a specific constant in VAESDM.VV can isolate the InvMixColumns transformation, and finally AddRoundKey can be done manually with a XOR. Overall, a great improvement over my initial solution.

VAESDM.VV can also be used in this way to emulate AESIMC, thus not needing the scalar AES64IM instruction.

AESKEYGENASSIST doesn’t match 1-to-1 with any instruction. It is possible to emulate some of its functionality with VAESKF1.VI, but until we encounter it in a game it is emulated in software.

Optimizations

Felix86 is >1.5 years old now, and some of the older instruction handlers were quite bad. One such example was the PHMINPOSUW instruction. While it would use VREDMINU to find the minimum element, a slow loop was used to find its index. When thinking about implementing VPHMINPOSUW and looking at my PHMINPOSUW implementation, I realized I can simply use VFIRST to find the position of the minimum element, vastly improving the instruction performance.

Another example is UNPCKHPS, which would use an ugly sequence of VRGATHER instructions, while widening adds and slides suffice.

Thanks for reading this post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

Vulkan X11 thunking support

The native Vulkan userspace driver can now be used on X11 with thunking. We also implemented the signatures for some missing extensions used by DXVK, which means DXVK on Wine can now use Vulkan thunking. This can be enabled on 64-bit games with FELIX86_ENABLED_THUNKS=vk felix86 --shell, as it is currently disabled by default.

Thunking improves performance and compatibility and reduces time spent on recompilation.

Witcher 3 with DXVK and Vulkan thunking on Milk-V Jupiter

Zink support

The Vulkan extension signatures necessary for Zink were also added. For ease of use, there’s now a zink profile that enables thunking and Zink. Test with FELIX86_PROFILE=zink felix86 --shell.

SpacemiT K3

SpacemiT was kind enough to give us early SSH access to a Linux environment with SpacemiT K3 to test felix86 and run benchmarks!

Hardware

SpacemiT K3 is RVA23 compliant, which means we get access to vector crypto (important for x86 AES & others), unaligned non-atomic accesses supported in hardware (necessary, see next month’s post as to why), Zvfhmin which we’ll eventually use for F16C, Zvbb which allows for some optimizations in our SIMD code, and Zfa which also allows for some optimizations.

Additionally, RVA23 covers the extensions necessary for felix86 operation, which are RVV 1.0 and Zba/Zbb/Zbs.

Benchmarks

The benchmarks we tried ran a lot faster on the SpacemiT K3 than on the K1!

First, the x86-64 7z benchmark run through felix86 26.02 on RISC-V hardware:

// SpacemiT K1, x86-64 7z
                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       2081   567    357   2024  |      41069   746    469   3502
23:       2040   579    359   2079  |      42298   780    469   3659
24:       2042   589    373   2196  |      41572   779    468   3648
25:       2049   606    386   2340  |      40578   779    464   3611
----------------------------------  | ------------------------------
Avr:      2053   585    369   2160  |      41379   771    468   3605
Tot:             678    418   2882

// SpacemiT K3, x86-64 7z
                       Compressing  |                  Decompressing
Dict     Speed Usage    R/U Rating  |      Speed Usage    R/U Rating
         KiB/s     %   MIPS   MIPS  |      KiB/s     %   MIPS   MIPS

22:       7392   579   1243   7192  |     123233   778   1351  10508
23:       7199   590   1244   7336  |     121907   784   1345  10545
24:       7248   609   1280   7794  |     119814   783   1343  10513
25:       7232   622   1328   8258  |     117826   787   1331  10484
----------------------------------  | ------------------------------
Avr:      7268   600   1274   7645  |     120695   783   1343  10513
Tot:             691   1308   9079

As you can see, on SpacemiT K3 the 7z benchmark run through felix86 gets a 3.54x speedup on average in compression and a 2.91x speedup in decompression compared to the same benchmark through felix86 on SpacemiT K1.

Next, the x86 stockfish benchmark (run as stockfish bench inside felix86 --shell).

// SpacemiT K1, x86-64 Stockfish 17.1, felix86 26.02
Total time (ms) : 99364
Nodes searched  : 2030154
Nodes/second    : 20431

// SpacemiT K3, x86-64 Stockfish 17.1, felix86 26.02
Total time (ms) : 24425
Nodes searched  : 2030154
Nodes/second    : 83117

With SpacemiT K3 the Stockfish benchmark is 4.07x faster.

An interesting observation, is that the native RISC-V Stockfish runs slower than the x86-64 Stockfish through felix86:

// SpacemiT K3, RISC-V Stockfish 17.1
Total time (ms) : 42895
Nodes searched  : 2030154
Nodes/second    : 47328

This isn’t too unexpected, as the x86-64 Stockfish is significantly more optimized with assembly routines, and the RISC-V version should catch up in the future, but it shows how felix86 can translate the optimized x86-64 routines of Stockfish to improve performance.

Next, a run of node -e "console.log('hello')":

// SpacemiT K1, x86-64 Node v24.13.0
2.72s user 0.70s system 110% cpu 3.100 total

// SpacemiT K3, x86-64 Node v24.13.0
real    0m1.408s
user    0m1.156s
sys     0m0.400s

Finally, a run of ffmpeg -f lavfi -i testsrc=duration=10:size=1920x1080:rate=30 -c:v libx264 -benchmark -preset medium -f null -

// SpacemiT K1, x86-64 ffmpeg N-122467-gc3d3377fe1-20260116
bench: utime=612.988s stime=6.089s rtime=99.349s

// SpacemiT K3, x86-64 ffmpeg N-122467-gc3d3377fe1-20260116
bench: utime=175.554s stime=1.746s rtime=32.661s

The benchmark completes three times faster.

A100 cores

The SpacemiT K3 has 8 X100 RVA23 cores with VLEN=256 and 8 A100 non-RVA23 cores with VLEN=1024. Due to differences in VLEN, the same process can’t be scheduled on X100 and A100 cores. Nevertheless, felix86 can work on the VLEN=1024 cores.

To make any child processes get sent to the A100 cores, you need to write the PID of the current process to /proc/set_ai_thread. In many shell programs, such as bash or zsh, this can be done with echo $$ > /proc/set_ai_thread.

Running a felix86 instance will now show us we have a VLEN of 1024:

Extensions enabled for the recompiler: g,v1024,c,b,zicond,zfa,zvbb,zvkned

Which means it should be possible to start some x86 processes on these cores through felix86 to squeeze some extra compute power out of the system if the X100 cores are fully occupied.

Final thoughts

Performance has significantly improved with the SpacemiT K3, so we’re very excited to see it in boards and test games with it.

It seems that the SpacemiT K3 has a 39-bit address space, same as the SpacemiT K1. A 48-bit address space would be nice in future SoCs, as it enables emulation of some programs that rely on the larger address space that x86 offers. One example is PS4 emulators like shadPS4.

The Zacas extension is missing and it is important for x86-64 emulation on RISC-V. It gives us a way to do 128-bit CAS, which is heavily used by Unity. When absent, we use a global lock for all CMPXCHG16B operations, which is not actually atomic with regards to other memory operations. As we get faster cores, it is likely that the lack of Zacas leads to crashes in x86-64 games that use CMPXCHG16B. It is a relatively new extension so it’s understandable for it to not be implemented yet.

Some unaligned atomic support would be nice, such as Zama16b.

Both Zacas and Zama16b are optional extensions for RVA23, so we hope to see them in a future core.

Overall, a great improvement in performance. Looking forward to seeing how games perform under felix86!

Compatible with RVA23

After a few bug fixes, felix86 now officially supports RVA23 hardware with VLEN=256. We also ran tests on QEMU with cpu=max, and after some more bug fixes there’s support for VLEN=128 with all the extensions that QEMU supports.

Miscallaneous niceties

Install script improvements

The felix86 install script (bash <(curl -s https://install.felix86.com)) now supports installing a specific commit that was made in the last 90 days.

Additionally, some oversights in rootfs installation were addressed. The rootfs is now correctly owned by root, and only the $HOME folder inside the rootfs is owned by the user.

Documentation

We now have documentation at https://felix86.com/docs, for developers and users alike. Check it out!

RSS feed

There’s now an RSS feed available at https://felix86.com/feed.xml.

Bluesky

There’s now a Bluesky account for felix86 at @felix86-emu.

Thanks for reading this post.

Big things are coming next month. Stay tuned!

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

New logo

Felix86 has a new logo!

For more info about the logo and the artist, visit this page:
https://github.com/felix86-emu/site/wiki/Logo

apt fixes

For a long time apt would be flaky on felix86. Packages would install, but often you would see a Segmentation fault happen after they finish installing, which was unpleasant and confusing.

Since an issue was opened for this, we decided to take a closer look. It turns out it was an issue with how we handle the exit syscall when it is called from inside a signal handler. This would also cause issues in the dash shell, which will longjmp out of the signal handler without ever calling sigreturn, and when it was time to exit the same bug would appear. This is now fixed and apt no longer crashes on exit!

Another bug the issue report mentions is apt-key exiting unexpectedly. The reason for this bug has to do with how we handled config files in felix86. Each child instance would try to access the config.json file to read the configuration. However, if a child process doesn’t have access to the user’s home directory then felix86 will exit during startup. This is fixed by passing configurations to child processes via environment variables.

With these fixes, apt update and apt install can now cleanly install packages on your rootfs: Installing cowsay in the rootfs thru felix86

Game fixes

Genshin Impact

Genshin Impact had a couple issues that prevented it from loading. It is not currently playable, as the 2FA messagebox is now a Chromium-based browser which seems to have some issue in felix86 that needs further investigation.

16-bit bswap

Genshin Impact would use the bswap instruction with an operand size override prefix. This is not defined in the Intel manual, but is not an illegal instruction on real hardware. This instruction is now implemented.

Memory usage reduction

Each thread would use significant amounts of memory which did not have a positive impact on performance. One such area was a 2^20 entry address cache, allowing for lookups of recently translated guest addresses using their bottom 20 bits. This was reduced to 2^16 entries, using the bottom 16 bits. Each entry would be 16 bytes, containing a guest and host address pair. Thus, in total, this reduces memory usage by 15 MiB per thread. Additionally, the maximum code cache size was changed from 64 MiB to 32 MiB. The initial code cache size was also decreased from 8 MiB to 4 MiB. This means that applications that spawn a large amount of threads will use significantly less memory, thus starting up faster and not leading to OOM problems.

This change shaves off 2 GiB of memory usage from Genshin Impact.

Deponia

We have a custom allocator for the 32-bit address space that is used for mmap/mremap/munmap/… syscalls. This is necessary even for 64-bit applications, because of the x86-only mmap flag MAP_32BIT. We would previously not correctly mark the executable itself as allocated if it was loaded in the 32-bit address space, which would cause problems because the kernel and the emulator would disagree on allocatable regions.

With this bug fixed, the game Deponia now works.

Execution profiles

felix86 can now load different “execution profiles”, specified by the FELIX86_PROFILE environment variable. Each profile is a partial or full configuration file, which will override the default configuration file. This allows for easily selecting a different array of configurations to do different things. You can have a profile that disables unsafe optimizations, or a profile that enables debugging options.

Additionally, this version introduces a configuration that will pass any specified environment variables to the guest program, separated by a semicolon. For example, FELIX86_ENVIRONMENT=WINEDEBUG=+seh;GALLIUM_HUD=fps will pass WINEDEBUG=+seh and LD_DEBUG to the guest program, without the emulator being affected by these environment variables. This feature allows execution profiles to define certain environment variables.

This feature can be used to automatically apply a bunch of different environment variables and felix86 configurations to games.

REPL color coding

The felix86 REPL can now color code each translation for easily understanding which RISC-V instruction corresponds to which x86 instruction. This can be enabled with the color command.

Each translation is a different color

Thanks for reading this post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

Quality of life changes

Trusted directories

In the past, felix86 would only allow you to run applications that exist inside the rootfs. This is because filesystem syscalls are containerized to only allow access inside the rootfs, which is where the x86 libraries and binaries are. This isn’t very user friendly, because it requires copying or moving everything you want to run inside the rootfs. With felix86 25.12, you can run x86 programs outside the rootfs. The first time you do so, felix86 will prompt you whether you want to add that directory to the trusted directories. This will allow felix86 to access this directory and its subdirectories and files without it being inside the rootfs.

Running an x86 executable from outside the rootfs for the first time

For more info and technical details, refer to the pull request.

Remove felix86-mounter

Previously, an executable with higher privileges called felix86-mounter would be responsible for mounting /dev, /proc and other directories inside the rootfs. Since we introduced fake mounts with trusted directories, these directories are now fake mounted inside the rootfs so there’s no dependency on felix86-mounter. This means that felix86 can now run without any root privileges!

--shell argument

You can now use felix86 --shell to enter the rootfs, similarly to doing felix86 /rootfs/bin/bash but with FELIX86_QUIET set and a nice prompt.

REPL environment

A lot of felix86 work is optimizing the various x86 instructions into an optimal RISC-V instruction pattern. This version of felix86 introduces a REPL environment that allows easily viewing what each x86 instruction translates to.

You can also compile instruction sequences by separating them with semicolons!

It allows for viewing how an x86 instruction compiles on 32-bit and 64-bit programs, and you can also disable flag generation. Additionally there’s the option of compiling multiple instruction sequences to view how multiple instructions interact with each other.

Opcode fusing at work, with FELIX86_FUSE_OPCODES=1

Even more optimizations

Scan ahead multiple blocks

The FELIX86_SCAN_AHEAD_MULTI option is now enabled by default. For a refresher on what it does, check the felix86 25.11 post.

It can now work with the FELIX86_UNSAFE_FLAGS option, which makes the recompiler not calculate flags for blocks ending in call or ret. This helps with blocks that have a conditional jump to a ret instruction and a block that overwrites flags. In these cases, the flags don’t need to be calculated as long as the program is ABI conforming. This flag is disabled by default for now, as it may potentially break programs.

Code cache is placed near the executable

Constructing big immediates is a pain in RISC-V and it can take a hard-to-guess number of instructions. Most compilers have multiple paths and may even make multiple attempts at generating the best instruction sequence when loading a big immediate.

One source of big immediates in our case is x86 rip-relative access. We now place the recompiled code cache near the executable, so that most rip-relative addresses can be constructed with an AUIPC+ADDI combo. This only needs to be done for 64-bit programs, as 32-bit programs have smaller addresses that can already be constructed in two instructions.

Instruction optimizations

This version comes with more instruction optimizations, such as optimizing the PMADDWD instruction, BSWAP, 32-bit effective address generation, and more.

TZCNT and LZCNT

The TZCNT behavior matches nicely with RISC-V’s CTZ instruction. When the source operand is 0, TZCNT will set the destination operand to the operand width. RISC-V does the same with CTZ and CTZW, so translation is now 1-to-1 (excluding flags). Previously we would use a branch to check if the source operand is zero, but this is unnecessary.

TZCNT can also operate on 16-bit operands. There’s no CTZH instruction, but only two instructions are needed to emulate it. One is a BSETI to set the 16th bit, followed by a CTZW. This way if the low 16 bits of the source are zero, the CTZW will return 16 since the 16th bit is set.

LZCNT was optimized in a similar fashion using CLZ and CLZW.

Requirement upgrade

The felix86 requirements used to be rv64gv, with RVV 1.0. As we get closer to RVA23 compatible chips hitting the market, this requirement is upped to rv64gvb, where b is Zba/Zbb/Zbc/Zbs. It is highly unlikely we’ll get any new high-performance cores that have RVV 1.0 but no bit-manipulation instructions, so this is a pretty easy requirement.

Other improvements

The felix86 install script has been improved. It is now hosted at install.felix86.com and its source code can be found at https://github.com/felix86-emu/install.

Rootfs

New rootfs options are now available. You can choose the No Wine rootfs, which saves ~1.6 GiB of space by not having wine installed, in case you don’t need to run Windows apps. There’s also the Tiny rootfs, which has just the bare minimum, weighing in at ~150 MiB uncompressed.

There’s plans for a future version of felix86 that allows running without a rootfs. This would require you to manually install the x86 libraries yourself and set LD_LIBRARY_PATH accordingly. It would also give the emulated executable unrestricted access to your filesystem.

Testing

Our CI was extended with a bunch of new tests. These will verify correctness and massively help with catching regressions.

Currently we use prebuilt tests from GCC, Valgrind and libuv. In the future, we want to expand the testing infrastructure with tests from Gvisor, POSIX, Node, Chromium, and more.

Benchmarks

We ran GeekBench 6 with felix86 25.11 and felix86 25.12. This month’s version is around 6% faster, with some benchmarks being up to ~20% faster. Here is the full benchmark comparison. Keep in mind that benchmarks are not really indicative of gaming performance, however it is nice to see that our performance is improving each month.

Thanks for reading this post. See you in 2026!

If you want to support this project, please consider donating: https://ko-fi.com/felix86

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

RISC-V Summit North America

The presentation “RISC-V for gaming: Emulating x86 on RISC-V” showcased the internals of the felix86 emulator, particularly the translation process. If you missed it, you can watch a recording below!

SSE 4.2

The instructions missing for SSE 4.2 were implemented. This includes horizontal instructions like PHADDW/PHADDD/... from SSSE 3. Applications that optionally use these instructions may see a performance improvement.

The x86 extension PCLMUL was also implemented. This extension adds a single instruction, PCLMULQDQ, which performs carryless multiplication. In RISC-V land we have CLMUL and CLMULH to achieve the same effect. Currently, support for this extension is disabled by default but can be enabled with the environment variable FELIX86_PCLMULQDQ.

Additionally, the CRC32 instruction from SSE 4.2 was implemented. RISC-V doesn’t have a dedicated CRC32 instruction (and even if it did, it’s unlikely it would use the same polynomial), but it’s possible to perform a CRC32 using the carryless multiplication instructions. There’s an Intel white paper describing how PCLMULQDQ can be used to calculate a CRC32 with any polynomial, and there’s also a blog post by merryhime for further reading. This allows us to implement the CRC32 instruction while taking advantage of the carryless multiplication instructions, thus greatly reducing instruction count.

More flag optimizations

It used to be the case that felix86 would omit flag calculations only in the current block. This would mean that whenever a block ends, the flags of the latest instruction that changed them need to be calculated in software. However, it is often the case that all targets of a branch don’t make use of the flags. A new optimization was made which allows checking the branch targets for flag usage. This works especially well in hot loops, since it shaves many instructions off the end of the block.

This optimization is also disabled by default and can be enabled with the environment variable FELIX86_SCAN_AHEAD_MULTI.

Thanks for reading this post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

But first, we have an important announcement to make!

RISC-V Summit North America

We’re going to be presenting at the RISC-V Summit North America on October 23, 2025. The presentation will be about how we emulate x86 on RISC-V, focusing on the low-level translation aspects.

Check it out!

x87 performance and compatibility improvements

As promised last month, we worked hard on improving x87 performance. We also fixed a few bugs along the way!

Stack optimizations

x87 registers are accessed like a stack. It consists of a “stack pointer” that points to one of eight registers. Then, whenever an instruction refers to st0, it refers to the register at that pointer, st1 is the one right after, and so on. In RISC-V, we have to model that with memory loads and stores. We tried allocating the stack on registers and moving them around on push and pop operations but it was bad for performance as push and pop are very frequent.

The new solution involves a couple optimizations. First, if we already loaded a register, we don’t load it again on further uses. If we use st1, it will get loaded from memory, then if two new values are pushed and we use st3, the register that was previously allocated for st1 will be used. The second optimization is that if we push values to the stack we don’t actually modify the top, tag word or stack until the end of the block. We keep track of what was pushed and do all the modifications at the end, saving on redundant stores.

Bug fixes

We had a few bugs with our FPU tag word handling. After fixing them and verifying the correct behavior with tests, a bunch of older titles can now run on felix86. Titles include Far Cry, DOOM 3, Assassin’s Creed and others.

Hitman: Blood Money is now playable on felix86 25.10

Future work

The x87 registers support 80-bit floats and some games rely on this extra precision to function. Felix86 uses normal 64-bit floats to emulate them, which is fine for most but not all games. We plan to emulate this extra precision mode eventually.

32-bit signals

Felix86 now has some preliminary support for 32-bit signals, allowing games that use trivial signals to work. More testing is necessary, as well as supporting legacy x86 signal frames that some games use.

Thanks for reading this post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

Optimizations

We managed to squeeze in a few optimizations. We now automatically compress RISC-V instructions whenever possible, using a new feature implemented in Biscuit. Additionally, we optimized the implementation of some x86 instructions.

New playable games

After some bug fixes in the latest version, the compatibility list has been updated with new working titles, notably Sleeping Dogs and Counter-Strike: Source.

What we are working on

32-bit signal support

Some 32-bit games require signals to function. This is usually the case with Windows games run through wine. Currently, felix86 only supports signals for 64-bit applications, but we are working on supporting 32-bit signals. This may allow several 32-bit Windows games to run.

Not all Windows games require signals. Signals are primarily used in Wine for emulating the Asynchronous Procedure Call mechanism. Some 32-bit Windows games operate fine without signal support, such as Fallout 2 or Trackmania Nations Forever.

Reducing vector pipeline stalls

After a chat with the folks at FEX-Emu, it came to our attention that instructions that change the vector configuration may be a decent overhead in current hardware.

While modern hardware may utilize register renaming to avoid a pipeline stall whenever the configuration is changed, current RISC-V hardware likely lacks this optimization.

It is the case that felix86 avoids emitting vsetivli instructions whenever the configuration wouldn’t be changed at a basic block level. However, we have something in most SSE instruction handlers that would emit an additional vsetivli whenever the instruction has a memory operand. It was falsely assumed that all SSE instructions can do unaligned memory access (this is not true, it’s something that only exists in AVX land) so when loading the memory operands we would set the vector configuration to sixteen 8-bit elements, allowing for unaligned access. Since this is not necessary, it can reduce vsetivli instructions when a memory operand is involved, potentially improving performance. We should also look into instruction handlers that use more vector configuration changes than necessary!

Thanks for reading the August post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

Browsers

We were able to run the Chromium browser under felix86!

You may be wondering: why would anyone want to do that?

The Linux version of Steam, running on RISC-V with felix86

A lot of modern apps use some kind of embedded version of Chromium. This ranges from the Windows Start Menu to Discord to Steam.

But Steam was always a huge goal for felix86. It has become the de facto launcher for games. Being able to launch games that use Steam DRM is great and allows us to test a wide variety of titles.

Steam itself is a 32-bit app. It uses a separate 64-bit tool called the steamwebhelper to render the GUI using the Chromium Embedded Framework. It also uses a tool called pressure-vessel for containerization. It does some tricky stuff with mounts and pivot_root.

If you’d like to try Steam, check out the Steam setup guide.

Installation guide

There’s also a quick installation guide for felix86 and Steam:

Optimizations

Most of the work this month was on getting Steam to work. However, we also optimized a few instructions.

One of the more interesting optimizations we made was the fusing of cmp instructions.

On RISC-V, if you want to check if, for example, register A is less than register B, you use the slt instruction.

On x86, you use the cmp instruction. The cmp instruction does a generic comparison between two registers and sets the flags accordingly. If you wanted to then set a register if A is less than B, you’d use the setl instruction after the cmp. The real operation of the setl instruction is checking if the overflow flag is not equal with the negative flag.

RISC-V doesn’t have flags, so felix86 computes them in software. This is expensive, particularly for the overflow flag, which takes numerous instructions to calculate. Luckily, cmp has no side-effects other than flags. If we know that the flags are only used for the setl and then overwritten, we could emit the cmp and setl combo as a single slt RISC-V instruction.

This is currently only done for cmp and cmovcc, but in the future we will expand it to work with setcc. The most used combo would probably be cmp and jcc, but supporting this would require analyzing more than one block at once, which we currently don’t do.

It should be noted that this optimization only works if the cmp is immediately followed by the instruction that uses the resulting flags. This is usually the case, but it’s possible for a compiler to emit instructions that don’t modify the flags between the two instructions. If deemed necessary, we could fuse cmp instructions even if the instruction that uses the flags is further down the line.

This optimization is currently disabled by default as it needs more testing. To enable it, run export FELIX86_FUSE_OPCODES=1.

Thanks for reading this post.

If you like this project, please give us a star on Github: https://github.com/OFFTKP/felix86

AAA games

Some AAA games are now playable! Titles include Linux games like Witcher 2, but also Windows games like Witcher 3 and Crysis!

Non-trivial Windows games would not work with the previous version

These just started working, so we haven’t had enough time to profile or optimize them yet. So don’t expect great performance yet!

Witcher 3 on Milk-V Jupiter

User-friendliness

The recommended way of using felix86 now is through the emulated bash.

To do so, install felix86 using the installation script (or register it with binfmt_misc using sudo -E felix86 -b), then run the x86-64 version of bash:

FELIX86_QUIET=1 felix86 /bin/bash

This should start an instance of the emulated bash that is now inside the rootfs. You can use it to run your games.

To learn more, check out the updated usage guide.

AppImage support

AppImages had a permission issue - basically they would try to run a privileged app like mount, but when this was run through the emulator, the emulator would not inherit the privileges.

Since we now install felix86 in binfmt_misc, the emulator should inherit the setuid bit of these apps, and AppImage executables should now work!

Better filesystem emulation

We try to restrict all guest file accesses inside the rootfs. In recent PRs we improved a few things.

/dev, /proc, /sys, /run and /tmp were symlinks. The reason for this is that mounting requires superuser privileges. Using namespaces doesn’t cut it, because then we wouldn’t be able to run apps like sudo or mount. Additionally, using namespaces to get mount privileges means that other instances of felix86 won’t see the same mounts as that instance.

But symlinks have their own set of problems. For example, readlink(/proc) would be equal to /proc, causing recursion in the guest. This was previously solved by detecting accesses to /proc (and the others) and redirecting them to the host /proc. All in all, it was hacky and horrible.

We now use a separate executable, called felix86-mounter. The installation script will install it and give it special privileges. Then, felix86 will call felix86-mounter with the rootfs path as the argument. felix86-mounter will then do a few things:

• Create a temporary dir called /run/felix86/mounts/mount-XXXXXX (where the X’s are random using mkdtemp)
• Create a file that has the actual rootfs path in /run/felix86/mounts/mount-XXXXXX/path.txt
• Create a directory /run/felix86/mounts/mount-XXXXXX/rootfs
• Mount the rootfs there (this is good because now the rootfs itself is a mount, just like / is)
• Mount the necessary dirs (/dev, /proc, …) inside the rootfs
• Return the /run/felix86/mounts/mount-XXXXXX/rootfs path

Further runs of felix86-mounter will detect that a mounted path already exists using the path.txt file.

This is better than doing it on the felix86 executable:

• If we did this inside the felix86 executable itself, it would need special permissions which is definitely scarier if we don’t properly drop them after mounting
• The felix86 executable frequently changes and needs to be given permissions again, felix86-mounter will not
• Having to run sudo everytime you want to run felix86 would be clunky
• Having to run sudo on anything you attach to felix86 (gdb, strace, …) would also suck

Symlink resolving

We had a bug in our previous implementation, which happened to go undetected:

The symlink syscall would resolve the old path to be inside the rootfs. For example, if you ran symlink(/oldpath, /linktarget), /linktarget would now resolve to /path/to/rootfs/oldpath.

This would work with most applications. However, as we start to worry about programs calling chroot and changing our root, this is no longer a viable solution.

Symlink shouldn’t resolve the old path to be inside the rootfs. But if we don’t do that, then we need to resolve paths ourselves using the same algorithm the kernel uses, while placing results in the rootfs. This is difficult to implement while avoiding potential bugs.

Luckily, the openat2 syscall has a flag called RESOLVE_IN_ROOT. This is perfect for our use case, as it can open a file while resolving symlinks to be inside a file descriptor – our rootfs file descriptor. We can then readlink the /proc/self/fd/<fd> to get the absolute resolved path that is inside the rootfs.

This also helps with containerization, as it makes it harder for programs to escape the rootfs. felix86 isn’t a security application and should only be used with trusted programs, but some programs would do stuff like fd = open("/"), openat(fd, "..") and depend on the file descriptors being the same, which we previously had to hack around.

Thanks for reading the July post!

It was an exciting month for felix86 progress. See you in a month!

Easier installation

On Ubuntu/Debian/Bianbu and possibly other distros, you can now install felix86 and a rootfs with a single command:

curl -s https://raw.githubusercontent.com/OFFTKP/felix86/master/src/felix86/tools/install.sh -o /tmp/felix86_install.sh && bash /tmp/felix86_install.sh && rm /tmp/felix86_install.sh

So go ahead and try it out!

Unity, UE3, and 32-bit games

There were many bug fixes during the month of May! One of the features we miss the most in RISC-V for x86-64 emulation is 128-bit compare exchange (CAS). This is necessary for proper emulation of the cmpxchg16b instruction. Unfortunately, no hardware currently supports the Zacas extension, which adds 128-bit CAS. The cmpxchg16b instruction is used extensively by Unity games, and improper emulation leads to frequent crashes and bugs.

Until we get Zacas in hardware, we have implemented a half-solution of using a global lock on every cmpxchg16b instruction. This makes it atomic with respect to other cmpxchg16b instructions, but not with other memory reads/writes. Turns out this is good enough to fix Unity games!

Cuphead, a Unity game, now playable on RISC-V with felix86

Other bug fixes have made at least one Unreal Engine 3 game playable, and maybe more that I haven’t tested.

Not many Unreal Engine 3 games for Linux out there

Additionally, after even more bug fixes, 32-bit ioctl marshalling support for radeon, and new x87 instructions, some 32-bit games are playable!

Now you’re thinking with portals

Thunking GLX

OpenGL thunk libraries now work! Games can load an overlay libGL.so/libGLX.so which will make calls into host code.

If you installed felix86 using the script, you can enable thunking using export FELIX86_ENABLED_THUNKS=glx. It is currently disabled by default, as it needs more testing.

If you compiled felix86 make sure to set FELIX86_THUNKS to /path/to/felix86/source/src/felix86/hle/guest_libs.

Technical ramble

One benefit of userspace emulators is they are allowed to employ some tricks for performance gains. For example, many libraries have a stable API across architectures. This means that if a program runs glDrawArrays for example, the host glDrawArrays could be used.

Quick brief on OpenGL inner workings

Things aren’t so simple, however. OpenGL works with a global context. The functions are loaded from an implementation-defined function that returns function pointers. On Linux, this will be either glXGetProcAddress or eglGetProcAddress. This means that thunking OpenGL essentially means thunking GLX and/or EGL.

Thunking GLX brings other problems. GLX interacts with X11 – it uses structs from X11 like the Display struct. The inner layout of this struct is architecture dependent. This not only means that you’d need to thunk libX11, but that it’s also difficult to do so as some structs differ in their layout.

FEX to the rescue

Thankfully there’s a way to get around needing to thunk libX11. You can create a host-side Display object and convert between the host and guest Display pointers when calling GLX functions that use Display. This idea is from FEX-Emu and it saves us a lot of frustration.

Thunking LuaJIT

Some games use Lua scripts. Other games like Balatro are entirely made in Lua. A popular speedy implementation of Lua is LuaJIT, which recompiles Lua code to machine code. However, LuaJIT does some things which invalidate our recompiled RISC-V code. This would previously cause significant slowdown in Balatro.

We now thunk the libluajit.so library, running the RISC-V LuaJIT in place of the x86-64 LuaJIT! This has caused significant performance improvements in Balatro and potentially other LÖVE engine games.

Balatro went from 2 FPS to 30 FPS and 100% GPU usage, so a GPU upgrade could push it even further!

LuaJIT can call C code from quite a few places, which means our host code needs to wrap the callbacks. You can see this happening in thunks_luajit.cpp. Whenever a C function is registered to be callable from Lua, we wrap it in code that will invoke the recompiler and fix up the arguments from the x86-64 ABI to the RISC-V ABI.

Enabling this needs you to compile LuaJIT for RISC-V, install it, and run export FELIX86_ENABLED_THUNKS=lua.

You can also thunk multiple libraries like we did for the Balatro image: export FELIX86_ENABLED_THUNKS=glx,lua

Thanks for reading the June post!

Contributions are welcome! Anybody interested in RISC-V or x86 can help!

If you find this project interesting, please star the repository.