I don’t blog much these days, apparently I just use it to announce roughly one new thing each year. But I do want to post more writing, so here’s a description of how I think about cross-compilation in nix.

This post doesn’t assume much nix knowledge. It’s not a how-to post describing the (complex) process of how to cross-compile software, it’s more of an exploration of how cross-compilation conceptually works, because it’s something I learnt recently and found interesting.

I think nix is fantastic. Language-agnostic, cross-platform, reproducible, cacheable software building and distribution. It’s not an easy thing to learn, but the payoff is tremendous.

But one thing about nix is that you typically need to be all-in. When building software, this makes sense - Nix’s reproducibility only works if all your dependencies are themselves available within nix. But as a colleague casually suggested one day: shouldn’t it be possible to run nix-built software without installing nix?

I told him a few reasons why it’s harder than it sounds, mainly due to the hardcoded /nix/store path which is assumed by the entire ecosystem. But my mind dwelt on it in the background, and it turns out it turned out to be much easier than I first thought.

Runix

So, I’m announcing runix, which does just that - it’s a small, unobtrusive executable which allows running nix software from any binary cache. Features:

• small (<4mb compressed)
• fast (~10 microsecond overhead after initial download)
• no configuration required
• unobtrusive (no need for root access or a /nix directory)
• can use software from any nix-compatible binary cache (including cache.nixos.org and cachix)
• conveniently distribute software via runscripts

Runscripts are a runix invention, they’re a tiny wrapper around a list of derivations and binary caches. In addition, they support:

• self-bootstrapping (install runix itself if missing)
• multiplatform (execute a different nix derivation per platform)

runix intentionally lacks all the development and authoring features of nix itself - you can’t evaluate nix expressions or build software locally, you can only run existing software which someone else has built and pushed to a binary cache. But for distributing software, it’s a much easier way for users to access nix-built software.

chored is a utility for handling repetitive chores and files.

There are many repositories. Lots of them have similar (tedious!) things they need to do, which aren’t particular to that repository:

• building
• linting
• testing
• generation of configuration for tools:
  • CI configuration
  • build settings
  • all sorts if common or shared files
• release management
• documentation generation
• pull request automation
• updating dependencies

chored allows you to reuse solutions to these problems, and any others you can think of. It’s minimally intrusive and lightweight (just add one small wrapper script to your repo).

And even when tasks aren’t exactly the same, chored facilitates code reuse as easily as importing a URL, but as powerful as real libraries in a real programming language (because it is both of those things).

Update:

Doing things my own way is too much effort, I just use niv these days :)

My last post was more than a year ago, in which I described my long journey towards better package management with nix.

Well, it turns out that journey wasn’t over. Since then, I’ve been using the tools I created and found them unfortunately lacking.

So, I’ve built just one more tool, to replace those described in the previous post.

I’m writing this assuming you’ve read the previous post, otherwise I’d be repeating myself a lot. If you haven’t, just check out nix-wrangle and don’t worry about my previous attempts ;)

Step 7: nix-wrangle for development, dependency management and releases

After a lot of mulling over the interconnected problems described in that previous post (and discussions with the creators of similar tools), I came at it from a new direction. Nix wrangle is the result of that approach. I’ve been using it fairly successfully for a while now, and I’m ready to announce it to the world.

How does it differ from my previous state of using nix-update-source and nix-pin?

• One JSON file for all dependencies, allowing bulk operations like show and update, as well as automating dependency injection (since all dependencies are known implicitly).
• A splice command which takes a nix derivation file, injects a synthetic src attribute baking in the current source from the nix-wrangle JSON file. This allows you to keep an idiomatic nix file for local development (using a local source), and automatically derive an expression using the latest public sources for inclusion in nixpkgs proper.
• Project-local development overrides. The global heuristic of nix-pin caused issues and some confusion, nix-wrangle supports local sources for exactly the same purpose, but with an explicit, project level scope.

Please check it out if you use nix! And if you don’t use nix, check that out first :)

Update:

Doing things my own way is too much effort, I just use niv these days :)

This post is targeted at users of nix who write / maintain package derivations for software they contribute to. I’ve spent a lot of time doing (and thinking about) this, although it’s probably quite a niche audience ;)

tl;dr: you should check out nix-pin and nix-update-source if you want to have a happy life developing and updating nix expressions for projects you work on.

I believe nix is a technically excellent mechanism for software distribution and packaging.

But it’s also flexible enough that I want to use it for many different use cases, especially during development. Unfortunately, there are a few rough edges which make a good development setup difficult, especially when you’re trying to build expressions which serve multiple purposes. Each of these purpose has quite a few constraints:

Often you need to progressively build up a set of commandline arguments in bash, like so:

FLAGS=""
if [ -n "$LOGFILE" ]; then
  FLAGS="$FLAGS --log $LOGFILE"
fi
someprogram $FLAGS ...

This usually works, but is a bit rubbish:

• this will break if $LOGFILE has a space in it, because bash will split it into multiple arguments
• adding a flag is kind of tedious with the FLAGS="$FLAGS ..." boilerplate
• $FLAGS ends up with a leading space, which is entirely fine but still feels ugly

Arrays solve these issues nicely. They can store elements with spaces, and there’s a nice append syntax:

FLAGS=()
if [ -n "$LOGFILE" ]; then
  FLAGS+=(--log "$LOGFILE")
fi
someprogram "${FLAGS[@]}" ...

You need to remember the weird "${VAR[@]}" syntax, but you get used to that (writing "$@" to pass along “all of this scripts arguments” is actually shorthand for "${@[@]}", which may help you remember).

Problem: “there’s no such thing as an empty array”

The problem is that in bash, an empty array is considered to be unset. I can’t imagine any reason why this should be true, but that’s bash for you. My problem is that I always use set -u in scripts I write, so that a command will fail if I reference a variable which doesn’t exist (just like a real programming language). But in bash, this will fail:

$ set -u
$ FLAGS=()
$ echo "${FLAGS[@]}"
bash: FLAGS[@]: unbound variable

Ugh.

The solution is even more weird bash syntax:

$ echo ${FLAGS[@]+"${FLAGS[@]}"}

(thanks, Stack Overflow)

Which roughly translates to “if FLAGS[@] is set, then insert the value of FLAGS[@], otherwise expand to nothing”.

Note the placement of the quotes - quoting the first instance of ${FLAGS[@]} will lead to an empty string argument (instead of no argument) if $FLAGS is empty. And failing to quote the second instance of ${FLAGS[@]} will mean it breaks arguments on spaces, which was the whole reason we used an array in the first place.

One more trick in your bag of weird bash tricks

Depending on your outlook, this is either another useful trick to help you write more robust bash, or yet another example of how bash actively discourages decent programming practices, highlighting how you really really really shouldn’t use bash for anything nontrivial.