How not to create a Node executable for ARM

There is one thing I would like to do properly for a quite some time:

Create executable NodeJS application on an x64 machine for ARMv6

Now I have mastered the process to some extent, with the following steps:

Write an app on a host x64 machine
scp sources to the target ARMv6 machine (i.e. Raspberry pi based computer)
Make an executable with pkg by sending a command commands over ssh
Delete the sources on the target machine
OPTIONALLY: Run a native systemd or pm2 service that keeps the app running

This approach is automated to a single npm run command at this point. However it has several drawbacks, which I'd like to remove over time:

It requires the machine with the target architecture running and ssh-able (this could be replaced with QEMU and/or Docker, but I have not gotten so far yet)
De-bugging is less straightforward
It unnecessarily transfers sources, some of them coule be accidentally left there
It requires node_modules/ on the target machine - they can be several tens times larger than the actual app (which bundles the node executable in a range of 35 ~ 70 MB, depending on version an architecture)

Of course it would be easier to just build the executable and tansfer it to the target machine. C, Go and Rust can do it without much hassle.

Preparation

Multiple poeple would like to do the cross-compiling to other-architecture as well, looking at #136, #145, #363, #605, #784 among other sources. One solution is to obtain a binary for a target architecture. The repository also allows you to prepare the binaries on your machine using a Docker image, although users reported that the process takes 10 hours or more to complete.

I have built it in the past to see what happens, but currently the repository already contains a lot of prebuilt binaries, the fetched-v14.4.0-linux-armv6 should suit me well enough, by saving it in the right location (currently ~/.pkg-cache/v2.6/). Let's try:

pkg -t arm64 app.js
> [email protected]
> Warning Failed to make bytecode node14-arm64 for file /snapshot/test/app.js

What does not work

First solution suggested in the foremetioned issue threads is to use --no-bytecode. The results are unsatisfactory:

pkg -t arm64 app.js --no-bytecode
> [email protected]
> Error! --no-bytecode and no source breaks final executable
  /home/peterbabic/app.js
  Please run with "-d" and without "--no-bytecode" first, and make
  sure that debug log does not contain "was included as bytecode".

Does -d, which stands for --debug outpus useful information? Well, we get to it in a minute.

Adding an architecture

Digging deeper, Debian based distributions have an apparent solutions in adding an architecture libraries from a repository:

dpkg --add-architecture i386
apt-get
apt-get install -y libc6:i386 libstdc++6:i386

Unfortunately, there is no readily available equivalent command set on an Arch Linux. I am tempted to try it in a VM. But for now, I will expplain steps that I went through trying to make cross-compilation run.

Building ARM binaries on AMDx64

Steps to actually compile an executable ARMx86 (ARMv6/ARMv7) and ARMx64 (ARMv8) binary on my laptop, which has a 64-bit Intel notebook starts with a toolchain.

yay -S arm-linux-gnueabihf-gcc

As of writing, the package is flagged out of date and won't install. Not good. There is also a x64 toolchain available, it is supported by a compolier named aarch64-linux-gnu-gcc. I did not know which package it belonged to, so I run this command:

pacman -Fq aarch64-linux-gnu-gcc | sudo pacman -S -

Yeah, the package has the same name as a command. What was I thinking. Nevermind, it was just a few unnecessary keystrokes, but now we can build a Hello World! for ARM x64 on an x86_64 machine:

aarch64-linux-gnu-gcc hello.c -o hello

Running ARM binaries on AMDx64

Running it straight away won't work:

$ file hello
hello: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=12eca1ab69cdf6c78169cb8a9c86cf21ea8c5873, for GNU/Linux 3.7.0, not stripped

$ ./hello
zsh: exec format error: ./cross

We need qemu-aarch64 to run ARM aarch64 executable:

pacman -Fq qemu-aarch64 | sudo pacman -S -

Run our cross compiled hello executable:

qemu-aarch64 hello

The executable should greet us, insted of displaying an error.

Running a pre-compiled NodeJS ARM x64 executable

With our newly acquired knowledge, we can try to run the node executable from the beginning that will be bundled with pkg. Since our toolcahin for ARM x86 is not currently working, for a try, we download x64 one.

wget https://github.com/robertsLando/pkg-binaries/releases/download/v1.0.0/fetched-v14.4.0-linux-arm64 -P ~/.pkg-cache/v2.6

As a side note, it probably needs a little time till the ARM x64 will be widespread. Raspberry pi 4 already touches that problem, although I did not touch one yet. But beigh prepared for the future is soemtimes also worth it.

Changing into the download directory and examining the file gives us the expected ARM aarch64:

$ file fetched-v14.4.0-linux-arm64
fetched-v14.4.0-linux-arm64: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=c80da3252b3b6bc0dedfa29f77b38de5f55e771e, with debug_info, not stripped

Running this binary should not work:

$ ./fetched-v14.4.0-linux-arm64
zsh: exec format error: ./fetched-v14.4.0-linux-arm64

What was not expected is that running it with qemu-aarch64, which proved fruitful before also fails:

$ qemu-aarch64 fetched-v14.4.0-linux-arm64
/lib/ld-linux-aarch64.so.1: No such file or directory

What package provides such a file?

$ pacman -F ld-linux-aarch64.so.1
community/aarch64-linux-gnu-glibc 2.32-1 [installed]
    usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1

Package aarch64-linux-gnu-glibc was installed a few steps back alongside the cross-compiler aarch64-linux-gnu-gcc, as seen here:

$ pacman -Si aarch64-linux-gnu-glibc | rg Required
Required By      : aarch64-linux-gnu-gcc

Quick checks for sanity, file really exists:

$ file /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1
/usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1: symbolic link to ld-2.32.so

And the one that the executable wanted does not:

$ file /lib/ld-linux-aarch64.so.1
/lib/ld-linux-aarch64.so.1: cannot open `/lib/ld-linux-aarch64.so.1' (No such file or directory)

The obvious dirty solution, that would pollute your machine's /lib with libraries for different architectures, and would probably fail later on with more dependencies would be copy (worse) or symlink (better):

sudo ln -s /usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1 /lib

The precompiled binary should run now. Remove the symlink, if you tried. There is a better solution:

qemu-aarch64 -L /usr/aarch64-linux-gnu/ fetched-v14.4.0-linux-arm64

This way, qemu knows where to look for the libraries. You can make it an alias to shorten it and call it done, if you woud just like to run the binaries via command. This is however not what is our goal here, remember? We need to find a more global way to tell the emulator where are the required libraries located. One way to do that is to provide this information via an environment variable QEMU_LD_PREFIX, which is equivalent to the -L parameter:

QEMU_LD_PREFIX=/usr/aarch64-linux-gnu/ qemu-aarch64 fetched-v14.4.0-linux-arm64

If we only use qemu for one architecture at a time, we can export the variable:

export QEMU_LD_PREFIX=/usr/aarch64-linux-gnu/

With the variable exported we can now run the pre-built binary:

$ qemu-aarch64 fetched-v14.4.0-linux-arm64
internal/validators.js:121
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at validateString (internal/validators.js:121:11)
    at Object.resolve (path.js:980:7)
    at resolveMainPath (internal/modules/run_main.js:12:40)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:65:24)
    at internal/main/run_main_module.js:17:47 {
  code: 'ERR_INVALID_ARG_TYPE'
}

This sure looks familiar to the NodeJS users, doesn't it? We have in fact made it run. The reason it probably failed is because it is not bundled yet, which would actually include a code to run, which it cannot find yet.

Sadly, even if we can now run this binary on our host machine, the pkg command in a development directory would still fail:

$ npx pkg -t arm64 app.js -d

... long output omitted ...

/home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: /home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: cannot execute binary file
/home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: /home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: cannot execute binary file
> Warning Failed to make bytecode node14-arm64 for file /snapshot/app.js

We see that it corectly calls the binary that we were ale to run separately just a few moments before, but now the problem is that the pkg has now way to know that it needs to call qemu-aarch64 to execute that binary transparently. For this, we need to setup binfmt.

Transparently execute an alien binary

I have found that to run alien binaries natievely, I can do this:

yay -S binfmt-qemu-static

It also has an optional package worth noting, that I keep istall as well:

$ yay -Si binfmt-qemu-static | rg Optional
Optional Deps   : qemu-user-static

Now with QEMU_LD_PREFIX in place, we can run the pre-compolied binary like this:

~/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64

Yet, this does not allow us to run pkg.

$ npx pkg -t arm64 app.js -d

... long output omitted ...

/lib/ld-linux-aarch64.so.1: No such file or directory
/lib/ld-linux-aarch64.so.1: No such file or directory
> Warning Failed to make bytecode node14-arm64 for file /snapshot/app.js

The things start to get blurry for me around this point, because QEMU_LD_PREFIX seems to be ignored when pkg needs it (it is being run via npm/npx/pkg, either of which does not provide any infor with a ldd command).

I was able to move on by resorting to symlinking the library to /lib mentioned before, to move further:

$ npx pkg -t arm64 app.js -d

... long output omitted ...

/home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
/home/peterbabic/.pkg-cache/v2.6/fetched-v14.4.0-linux-arm64: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
> Warning Failed to make bytecode node14-arm64 for file /snapshot/v2.6/app.js

This is the dead end for me. Now matter what symlink voodoo I tried, it refused to find that file, comfortably at location /usr/aarch64-linux-gnu/lib/libdl.so.2. My machine also has a /lib/libdl.so.2 file, which is of course compild for x86_64, so symnlinking is definitely risky and there has to be other way. If you know more, please let me know.

Side note

Also, it occasionally hangs on missing libstdc++.so.6. This library is present in /usr/aarch64-linux-gnu/lib64. Searching the Internets high and low I have found a hacky solution:

export LD_LIBRARY_PATH=/usr/aarch64-linux-gnu/lib64

But this env variable should be avoided, because of reasons I did not fully comprehend yet.

Conclusion

The process to prepare a working binary of a NodeJS application on AMDx64 machine for an ARMx86 architecture would allow me for a faster build cycle. Unfortunately, No matter what I tried so far, the solution seems to elude me.

The journey documented in this article served for as a rich educational course for me, so it is not all lost. Hopefully, you find something interesting here as well.