I don’t blog much these days, apparently I just use it to announce roughly one new thing each year. But I do want to post more writing, so here’s a description of how I think about cross-compilation in nix.

This post doesn’t assume much nix knowledge. It’s not a how-to post describing the (complex) process of how to cross-compile software, it’s more of an exploration of how cross-compilation conceptually works, because it’s something I learnt recently and found interesting.

What is nix?

Nix is an operating system, a programming language, and a massive (80k+) set of software packages written in that language. Today, we’re focussing on nixpkgs, the set of packages.

What is cross-compilation?

Most software is not cross-compiled. Typically you build software on a computer, and run it on the same kind of computer. Easy peasy. You want a Linux binary? Compile it on a Linux computer or VM, my friend.

But cross compiling means building a native executable for a different kind of computer. The standard terminology here is that the place where you build is the Build platform, and the place where you will run it is the Host platform. There’s also a concept of a Target platform, but that’s a historical oddity and doesn’t matter for modern compilers.

In my case, I’ve been writing some rust software and building it on a Mac. But I want to run it on Linux too. I could do a bunch of stuff with docker, but that’s boring. And more seriously I also want to compile for the newer ARM-based Macs, without having to juggle multiple computers.

To cross-compile a trivial rust app, I’d need a rust compiler on my Mac, and I’d tell it to build myApp as a Linux binary, using --target=linux-x86_64. Rust is a clever modern compiler, it can produce binaries for many supported targets out of the box.

But this doesn’t work for non-rust shared libraries. If you have a rust library which links against openssl for example, the one you have on your system is going to be the Mac one, not the Linux one. Oh dear.

Dependency injection

Nix uses a pattern not usually found in package definitions, but very common in programming: dependency injection.

Each package is not a static thing like a YAML file, but rather a function which requires some dependencies, and then produces a result - the fully concrete specification used to build some software. Here’s a stripped-down version of the GNU hello world program to demonstrate:

{ stdenv, fetchurl }:

stdenv.mkDerivation (rec {
  pname = "hello";
  version = "2.12.1";

  src = fetchurl {
    url = "mirror://gnu/hello/hello-${version}.tar.gz";
    sha256 = "sha256-jZkUKv2SV28wsM18tCqNxoCZmLxdYH2Idh9RLibH2yA=";
  };
})

This is a function which accepts two named dependencies (stdenv and fetchurl), then returns a derivation, which is the concrete type that nix knows how to build, producing files on disk.

fetchurl is a straightforward utility - it takes a URL and a checksum, and does the job of turning that into local files. If hello had any package dependencies, they would be passed in a similar way.

stdenv is a little more mysterious, it’s just “the standard environment”. Its mkDerviation function is ubiquitous in nixpkgs, and I don’t tend to think much about it.

That changed recently, when I started digging into cross-compilation.

The Nix package universe

Zooming out, nixpkgs defines this universe of package expressions. Like I said, there’s more than 80 thousand of them. Here’s a dramatically simplified version:

And they’re lazily evaluated, which is how you can get away with a single expression containing the entire universe. If you evaluate a single package, it will only load/evaluate the stuff it depends on, not the entire universe:

The interesting thing is, every single package uses this same pattern to depend on stdenv. This is where it gets interesting.

stdenv.mkDerivation is this ubiquitous function which takes some attributes and returns a derivation - the concrete thing to build. And it’s injected into each and every package:

Different stdenvs

It turns out, there isn’t just one but many different implementations of stdenv you could inject.

The fun one for today is the one which knows how to build stuff on a Mac, and produces Linux code.

If you inject this stdenv into the expression that builds a nix universe, you get a second universe where that stdenv is being used to build every single package in the universe. This is the universe of those same 80 thousand packages, except each of them are built on mac, producing binaries that run on Linux.

The Nix package universe multiverse

So the final slightly mind blowing thing, is that the different universes are linked.

This build-for-linux stdenv automatically knows that for packages specified in the buildDependencies list, each should actually be taken from the normal build-for-Mac universe. Whereas a runtime dependency like openssl or libc needs to be taken from the current build-for-linux universe.

In practice there’s way more dependencies, but this is the general shape:

And notice how we actually have two versions of libc here. Rust needs a Mac libc at build time, while my app needs a Linux version at runtime.

How this inter-universe connection is actually implemented is pretty wild for historic reasons, but thankfully you don’t have to know too much about that to make use of it.

Is this novel?

Typical distributions like Debian have this whole multi-arch packaging system where the architecture becomes part of the package key.

That obviously works, Linux package managers are extremely competent at cross-compilation and I’m assuming it’s a perfectly reasonable system. But with nix, you can just build as many different universes as you want. By plumbing together expressions that refer to other expressions in a particular way, it produces a cross compiled binary without requiring cross-compilation support from the nix language or nix-build tool itself.

So… yeah. I learnt this recently when working on runix and I just enjoyed this strangely beautiful mix of compiler toolchain hacking and elegant functional programming concepts.

Ok so how do I actually… cross compile stuff?

Unfortunately, actually learning the ins & outs of writing and debugging nix expressions for cross-compilation can be quite tricky. My additional overlay for cross-compiling runix is not pretty, and it took a lot of trial and error to get this working. But the results are pretty amazing, because now I can build everything on a single machine, fully automated with no need to install various toolchains.

These are some resources I found helpful when learning cross-compilation in nix, hopefully they help if you want to dive deeper: