posh.wiki

This morning, I tried to invoke a windows binary (git.exe) from WSL for the first time after a restart, and I got this error:

bash-5.3# git.exe push origin main
Could not find an available signal
Aborted                    git.exe push origin main

After trying the first step of debugging (turn it off and back on again - by systemctl reboot, wsl --terminate, and even restarting the Windows host), the problem didn't go away.

Googling didn't help at all. "Could not find an available signal" yields results about faulty video cables, and tagging on "WSL" or "Linux" just gets results about being unable to connect to a network.

After some investigation, I found that this applied to all Windows binaries, and no Linux ones. I managed to shake out a new error message by trying to call cmd.exe:

bash-5.3# /mnt/c/Windows/System32/cmd.exe /c echo hello
Cannot open assembly '[/mnt/c/Windows/System32/cmd].exe': File does not contain a valid CIL image.

Googling this was marginally more helpful. I ended up finding microsoft/WSL#8531, #13000 and #9146, plus a summary on Ask Ubuntu. All suggest that the root cause was an installation of mono, and an immediate term fix of sudo update-binfmts --disable cli or sudo rm /var/lib/binfmts/cli, but I had a few problems with these:

1. They claimed the issue was fixed in WSL v2.5.7; I experienced it in WSL v3.0.1.0.
2. They don't actually explain why they work.
3. I'm on Arch - I don't have update-binfmts or /var/lib/binfmts, so the solutions don't work for me.

(Also, a secret fourth thing that I found out while experimenting, neither of these are a particularly permanent fix, with update-binfmts needing to run after every restart and /var/lib/binfmts not being conceptually owned by the user and subject to change at any time).

So, here's my long-term, distro-agnostic solution, in case it helps someone else who finds themself here.

My theory

I'm quite sure that my Windows programs are valid, but I agree that they don't contain valid .NET Common Intermediate Language (CIL) code. Let's think about why Linux wants them to.

Native Linux "ELF" binaries will always begin with the same 4 "magic" bytes: 0x7f454c46 ("[DEL]ELF" in ASCII). When a program begins with magic bytes other than that, Linux uses a kernel feature called binfmt_misc to route execution to an handler which can run the program.

The kernel itself doesn't ship with any knowledge of magic bytes and their associated handlers, so it has to be told about them. If you're using systemd, that's the responsibility of the systemd-binfmt config loader.

One of the handlers usually registered in a WSL environment is called WSLInterop. It associates the magic bytes of a Windows binary (0x4d5a or "MZ") with the interpreter /init, which asks WIndows to execute the program.

My initial theory was that if some other handler was registered for these magic bytes, it might be trying and failing to execute the Windows binaries before WSL ever got a chance.

Testing the theory

To test this theory, I ran the following bash, which enumerates all active handler registrations (available as files under /proc/sys/fs/binfmt_misc/) and prints out their filenames and contents.

for f in /proc/sys/fs/binfmt_misc/*; do
    [[ $(basename "$f") == register || $(basename "$f") == status ]] && continue
    echo "=== $f ==="
    cat "$f"
done

I was looking for one or more entries using magic 4d5a and an interpreter other than /init. I found one, at /proc/sys/fs/binfmt_misc/CLR. Note that if you're following along, your handler, its interpreter, and its config file may be different from mine, so make sure you understand where they come from and don't just copy my commands exactly.

=== /proc/sys/fs/binfmt_misc/CLR ===
enabled
interpreter /usr/bin/mono
flags:
offset 0
magic 4d5a
...other handlers omitted for brevity...
=== /proc/sys/fs/binfmt_misc/WSLInterop ===
enabled
interpreter /init
flags: PF
offset 0
magic 4d5a

The interpreter associated with the CLR handler was /usr/bin/mono, which confirms the theory: A handler other than WSLInterop (namely, CLR) was claiming Windows binaries and trying to execute them using mono. I must have installed it at some point, which caused the registration's config to exist, and taken effect when I restarted my Linux installation and consequently rebuilt the handler registrations.

Fixing the problem

The immediate fix was simple:

echo -1 > /proc/sys/fs/binfmt_misc/CLR

But that doesn't last. The configuration for the handler still exists, and will be loaded by systemd-binfmt the next time we restart or otherwise rebuild the registrations. To fix it in the long term, we must attack the root cause, rather than the symptom.

Linux and systemd don't make it easy for us to tell which config file caused a particular registration to exist, so we have to find out the hard way. systemd-binfmt config files are always in one of three directories: /etc/binfmt.d, /usr/lib/binfmd.d, and /run/binfmt.d. We can grep for the path to the interpreter associated with our offending handler (in my case, /usr/bin/mono) to find the name of the config file that caused it to exist:

grep -RilE '\/usr\/bin\/mono' \
    /etc/binfmt.d \
    /usr/lib/binfmt.d \
    /run/binfmt.d 2>/dev/null

For me, the output was /usr/lib/binfmt.d/mono.conf. Specifically, we're interested in the basename, mono.conf. If we cat the file, we can confirm that it defines a handler named CLR, associated with magic bytes MZ and interpreter /usr/bin/mono.

:CLR:M::MZ::/usr/bin/mono:

To suppress this association, we need to tell systemd that we don't want this loaded. As the user, we ironically do not own /usr/lib/binfmt.d, so deleting the file there will just result in it being re-populated by a future update, and we'll start playing whack-a-mole with the handler config. /run/binfmt.d is even less useful to us, because it is volatile and disappears on reboot. We need to work in the one folder we own and can trust to remain unchanged by the system or package manager: /etc/binfmt.d. Thankfully, this directory also has the highest precedence to systemd-binfmt.

While we could create an empty file at /etc/binfmt.d/mono.conf, the convention for suppressing vendor-supplied config in systemd is actually to create a symlink to /dev/null. The command to do this is:

ln -s /dev/null /etc/binfmt.d/mono.conf

Finally, we need to rebuild our handler registrations for the change to take effect. A reboot will do this, but a less aggressive option is:

sudo systemctl restart systemd-binfmt