GFX::Monk Home

Posts tagged: "programming"

Nix remains my superpower

I’ve long considered fluency in Nix to be a superpower that pays off way more than you might imagine, if you haven’t experienced it yourself.

Sure, it helps with the obvious practical things you’d expect - my system setup is declarative, reproducible, and suffers from vanishingly few chaotic state-based issues that tend to plague less reproducible systems (like brew in particular).

There are plenty of downsides too - I run into nix-specific issues that my colleagues with more normal setups don’t suffer. Interactions with tools like bundler building native extensions can be frustrating at best.

Those are the unsurprising, surface level tradeoffs when using a good-but-novel package manager. The real superpower comes through the staggering amount of things which are not just possible, but downright straightforward due to the reliable, principled way that nix works. Here’s a good example from last week:

Nix cross-compilation: what even is it?

I don’t blog much these days, apparently I just use it to announce roughly one new thing each year. But I do want to post more writing, so here’s a description of how I think about cross-compilation in nix.

This post doesn’t assume much nix knowledge. It’s not a how-to post describing the (complex) process of how to cross-compile software, it’s more of an exploration of how cross-compilation conceptually works, because it’s something I learnt recently and found interesting.

runix: run nix software without nix

I think nix is fantastic. Language-agnostic, cross-platform, reproducible, cacheable software building and distribution. It’s not an easy thing to learn, but the payoff is tremendous.

But one thing about nix is that you typically need to be all-in. When building software, this makes sense - Nix’s reproducibility only works if all your dependencies are themselves available within nix. But as a colleague casually suggested one day: shouldn’t it be possible to run nix-built software without installing nix?

I told him a few reasons why it’s harder than it sounds, mainly due to the hardcoded /nix/store path which is assumed by the entire ecosystem. But my mind dwelt on it in the background, and it turns out it turned out to be much easier than I first thought.

Runix

So, I’m announcing runix, which does just that - it’s a small, unobtrusive executable which allows running nix software from any binary cache. Features:

  • small (<4mb compressed)
  • fast (~10 microsecond overhead after initial download)
  • no configuration required
  • unobtrusive (no need for root access or a /nix directory)
  • can use software from any nix-compatible binary cache (including cache.nixos.org and cachix)
  • conveniently distribute software via runscripts

Runscripts are a runix invention, they’re a tiny wrapper around a list of derivations and binary caches. In addition, they support:

  • self-bootstrapping (install runix itself if missing)
  • multiplatform (execute a different nix derivation per platform)

runix intentionally lacks all the development and authoring features of nix itself - you can’t evaluate nix expressions or build software locally, you can only run existing software which someone else has built and pushed to a binary cache. But for distributing software, it’s a much easier way for users to access nix-built software.

Introducing chored

chored is a utility for handling repetitive chores and files.

There are many repositories. Lots of them have similar (tedious!) things they need to do, which aren’t particular to that repository:

  • building
  • linting
  • testing
  • generation of configuration for tools:
    • CI configuration
    • build settings
    • all sorts if common or shared files
  • release management
  • documentation generation
  • pull request automation
  • updating dependencies

chored allows you to reuse solutions to these problems, and any others you can think of. It’s minimally intrusive and lightweight (just add one small wrapper script to your repo).

And even when tasks aren’t exactly the same, chored facilitates code reuse as easily as importing a URL, but as powerful as real libraries in a real programming language (because it is both of those things).

nix-wrangle: Manage nix sources and dependencies with ease

Update:

Doing things my own way is too much effort, I just use niv these days :)

My last post was more than a year ago, in which I described my long journey towards better package management with nix.

Well, it turns out that journey wasn’t over. Since then, I’ve been using the tools I created and found them unfortunately lacking.

So, I’ve built just one more tool, to replace those described in the previous post.

I’m writing this assuming you’ve read the previous post, otherwise I’d be repeating myself a lot. If you haven’t, just check out nix-wrangle and don’t worry about my previous attempts ;)

Step 7: nix-wrangle for development, dependency management and releases

After a lot of mulling over the interconnected problems described in that previous post (and discussions with the creators of similar tools), I came at it from a new direction. Nix wrangle is the result of that approach. I’ve been using it fairly successfully for a while now, and I’m ready to announce it to the world.

How does it differ from my previous state of using nix-update-source and nix-pin?

  • One JSON file for all dependencies, allowing bulk operations like show and update, as well as automating dependency injection (since all dependencies are known implicitly).
  • A splice command which takes a nix derivation file, injects a synthetic src attribute baking in the current source from the nix-wrangle JSON file. This allows you to keep an idiomatic nix file for local development (using a local source), and automatically derive an expression using the latest public sources for inclusion in nixpkgs proper.
  • Project-local development overrides. The global heuristic of nix-pin caused issues and some confusion, nix-wrangle supports local sources for exactly the same purpose, but with an explicit, project level scope.

Please check it out if you use nix! And if you don’t use nix, check that out first :)

A journey towards better nix package development

Update:

Doing things my own way is too much effort, I just use niv these days :)


This post is targeted at users of nix who write / maintain package derivations for software they contribute to. I’ve spent a lot of time doing (and thinking about) this, although it’s probably quite a niche audience ;)

tl;dr: you should check out nix-pin and nix-update-source if you want to have a happy life developing and updating nix expressions for projects you work on.

I believe nix is a technically excellent mechanism for software distribution and packaging.

But it’s also flexible enough that I want to use it for many different use cases, especially during development. Unfortunately, there are a few rough edges which make a good development setup difficult, especially when you’re trying to build expressions which serve multiple purposes. Each of these purpose has quite a few constraints: