Update:

Doing things my own way is too much effort, I just use niv these days :)

This post is targeted at users of nix who write / maintain package derivations for software they contribute to. I’ve spent a lot of time doing (and thinking about) this, although it’s probably quite a niche audience ;)

tl;dr: you should check out nix-pin and nix-update-source if you want to have a happy life developing and updating nix expressions for projects you work on.

I believe nix is a technically excellent mechanism for software distribution and packaging.

But it’s also flexible enough that I want to use it for many different use cases, especially during development. Unfortunately, there are a few rough edges which make a good development setup difficult, especially when you’re trying to build expressions which serve multiple purposes. Each of these purpose has quite a few constraints:

Purpose 1: Inclusion in nixpkgs proper

This is in some ways the most restrictive part - if I didn’t need to include my packages in nixgpks, I could make my derivations as funky and complex as I like. But there’s a lot of value to being in nixpkgs. The general constraints are that nixpkgs derivations should be idiomatic and functionality which isn’t needed for nixpkgs is frowned upon:

• simple packages should be a single function which takes its dependencies in a callPackage-compatible way
• splitting a simple derivation across multiple files is frowned upon (e.g. for the purposes of a common build.nix but different src.nix between nixpkgs and your upstream repository)
• externalizing attributes into a machine-readable / writable format (e.g. JSON) for easy automation will get your changes reverted (sadly)

Purpose 2: Allowing easy updates

Updating a package in nix is kind of tedious. For a typical github-hosted package you’ll need to update the rev attribute to the new version, but then you also need to update the sha256, and that’s not actually trivial. There are two options:

• run `nix-prefetch-url on the github tarball address, or
• break the existing sha256 attribute by modifying one or more characters (you can’t use an empty digest, that won’t get past the sanity checks), then:
  • build your package
  • copy-paste the correct sha256 from the error message into your package definition

I can never remember github’s tarball format, so both of these are pretty tedious. It may not be that much work each time, but the more packages (and versions) you commit to nixpkgs, the more this feels like a computer should be doing the tedious part for you.

Purpose 3: Standalone development (i.e. as part of the project being built)

I don’t know how common this is, but I keep nix expressions inside all of my repos. I think it’s a good idea to version your nix expressions with your code, especially when you’re working on changes to both at the same time. And since I love nix, why wouldn’t I want all of my projects to have a nix expression so I can get dependencies setup trivially and know that my build works, without accidentally depending my system environment?

Sometimes they’re just shell.nix files used for development, but a lot of the times they’re fully buildable nix derivations.

For the packages which are in nixpkgs, I want to keep a verbatim copy in my repo, but I still want the repo copy for testing changes, and so anyone with a copy of my project also has the appropriate build expression for any given revision.

Purpose 4: Inter-project development

This one may be quite uncommon, but I’ve ended up running into it a lot. The scenario is that I have some package base, and package app which depends on base. I want to either add a feature or fix something broken in app, but it turns out this requires fixes to either the source code or the nix expression for base. Now you can’t just make changes in one repo, but you need to build & integrate changes across two (or on a bad day, a whole handful of) projects.

Attempts I’ve made in the pursuit of a perfect workflow

You could probably skip to the end if you just want to know where I’m at now, but I think it often helps to explain the journey, so you know what problem I’m trying to solve, what attempts I’ve made that didn’t work out well enough, and why.

Step 1: Nothing fancy

When developing a nix expression, use only public, tagged archives.

This is fine for a long-running or third-party project, where you don’t need (nor have access) to make changes to the project itself. All you’re doing is wrapping it up in a nix expression, and updates to your nix expression come after upstream source code releases.

However, what about when the upstream project is your project. And what if you want to make sure your nix expression works as expected before you release a new version, so that you can fix anything which is broken? I really want to be able to test changes that I make end-to-end, and that includes testing that it’ll actually work in nix.

Another awkward issue is that you typically need two expressions - a default.nix which can be called with nix-build, and a myPackage/default.nix which accepts its dependencies as arguments. This typically just means default.nix just does a callPackage on myPackage/default.nix, but this is a piece of boilerplate you need in each project.

Problems:

• Every change to the source code must be versioned, tagged, pushed and released before you can update or test your nix derivation. This is extremely limiting for iterative development, as you may end up publishing a number of broken versions because you haven’t tested them.

Step 2: Scripted updates

Even when you’re doing the simplest possible thing, bumping a package version in nix is kind of tedious. For a typical github-hosted package you’ll need to update the rev attribute to the new version tag, but then you also need to update the sha256. It’s not hard, but it’s pretty annoying and clunky - the more you do it, the more convinced you become there must be a better way.

The straightforward way to automate updating source code (nix is a programming language, not a data format) is typically to extract the automatable bit into its own, simple file format. For example, if I was generating some data for a python program I wouldn’t try to modify python source code in-place, but I’d instead generate a JSON file, then use python’s JSON support to load in that file.

The same can be done with nix - we can store the arguments to fetchFromGitHub in a JSON file, and import that from our nix expression. It’s pretty easy to write a program to read & update a JSON file without needing to be able to edit nix source code. So I did, and it’s called nix-update-source. After much debate, I finally got it merged only to have it reverted a week later by edolstra (the creator of nix himself!) because he disagreed with the approach :(

So I ended up adding the option to modify nix source code inline (instead of generating JSON) using a hacky regex-based approach. For derivations that need to be accepted by nixpkgs, this is best option to avoid causing a stink.

Bonus tip: you can add an updateScript attribute to embed your update process into each package itself (here’s an example), and use nix-shell maintainers/scripts/update.nix --argstr package MY_PACKAGE to invoke it.

Problems:

• This is an improvement, but (as in Step 1) it still requires all changes go through the release process before you can test them.

Step 3: Anonymous git SHAs

Rather than fetching a specific git tag, you can use fetchgit to fetch a specific commit by its SHA. This means you can push commits to some testing branch, and use that to test out changes to your nix expression. When you’re happy, squash / rebase / merge the changes and update your commit to the published version.

Problems:

• You still need to push your commits somewhere public (e.g. github), so if you’re doing a lot of back and forth you’ll be committing / amending a commit, pushing, and then updating your src attribute before you can test the new version.
• People push back against using fetchgit in nixpkgs proper, since it’s less efficient than fetchFromGitHub (you’re cloning a full repo and depending on git, rather than fetching a single tarball). And it’s hard to justify, since each of your updates are to published versions, it’s only the work-in-progress versions which require anonymous commits.
• You’ll want to script this somehow, because forgetting to update the commit ID or digest will silently run your old code instead of the code you think you’re testing.
• You need to manage multiple branches if you don’t want testing commits going to master, and remember to update your commit ID and digest after merging / rebasing.

Step 4: Add a second derivation which uses a local tarball

This technique involves multiple nix expressions - one for the local tarball version, and one for the official published version. You can go about this in a few ways:

Option 1: use overrideDerivation to inject your local source, e.g.:

# local.nix
with import <nixpkgs> {};
lib.overrideDerivation (callPackage ./default.nix {}) (orig: {
	src = ./local.tgz;
})

Option 2: inject src as part of a second set of arguments. e.g:

# build.nix
{ stdenv, lib, curl, python }:
{ src, version }:
stdenv.mkDerivation {
	inherit src version;
	# ...
}

Option 2 is nice from an engineering perspective - there’s no “default” src, you have to explicitly provide one. When using Option 1 it’s easy to accidentally reference src in a way that overrideDerivation would be unable to intercept (e.g. by referencing it in your buildPhase directly), which ends up with very confusing issues.

I did actually get away with this. But not without sideways glances from nixpkgs maintainers about it being weird - it’s certainly not idiomatic.

Problems:

• You still need to script the creation of a tarball somehow. I came up with a handy script, but it assumes you’re using the gup build system and I have to copy it into every project.
• You need multiple nix expressions (local and published), which can get confusing.

A third option is just to use ./. as the source, skipping the whole tarball business. But I don’t like that, because:

• it’s unnecessarily slow and wastes disk space to copy the whole directory into a new store location on every build, including ignored files (not much fun with hundreds of megabytes of node_modules or built VM images)
• now your derivation needs to handle src being either a directory or a tarball. It’s kind of awkward.

Step 5: Use local tarballs and environment variables

The trouble with having a separate nix expression for your local version is that it’s not used by anyone else. Let’s say I have a project b which depends on a. I need to add a feature or fix an issue with b, but that also requires altering a in order to support that feature / fix.

If I’m building a off a local tarball in local.nix, project b won’t be able to see that since it’s using the official expression for a (which lives in nixpkgs, or is perhaps fetched from git). I could modify a temporarily to use the local.nix version instead, but that’s an awkward thing to juggle (and remember not to commit), especially when there are more than just two packages involved, and when this is a common part of your workflow. And if a is fetched from git instead of living in nixpkgs, then I’d also need to publish my changes before I can use them in b. And then what if I have a package c which wants to use my local modifications to both a and b?

So I started introducing environment variables to switch between published and development versions. e.g. each project depending on opam2nix would respect an $OPAM2NIX_DEVEL variable which caused it import that derivation instead of the published opam2nix. I could then set this variable while testing changes, and not worry about this change accidentally making its way into my source code.

Problems:

• Similar to “Step 4”, plus more complexity - even I get confused as to what version of which packages was being used where.
• Nobody wants this in nixpkgs, it’s ugly and weird.
• I don’t think it would even work for a multi-user setup.

Step 6: nix-pin for development, nix-update-source for releases

a.k.a hopefully the end of this journey?

I have built what I think is a pretty workable solution for local testing while also keeping your derivations completely idiomatic for easy acceptance in nixpkgs.

The development tool is called nix-pin, and it’s a small, generic tool for splicing local checkouts of a project into nixpkgs in an unobtrusive way. You can read the readme for the full details, but the basic workflow is:

• checkout a project to work on (which contains its own nix expression)
• add this project as a pin - you give it a name, a path, and the path to the nix expression inside the repo

You can now use nix-pin build / nix-pin shell instead of nix-build / nix-shell, to run a version of nixpkgs where the pins on your system are automatically spliced into nixpkgs. You can explicitly update a pin to include all uncommitted changes in your working directory, or pin a project to a specific git SHA. If you’re building a pinned package you’ll get the pinned version, and if you’re building any package that depends on a pinned package¹, it’ll get be the pinned version which is injected.

When it comes time to actually release commits that you’ve tested with nix-pin, you can use nix-update-source (although you don’t have to if you have your own release workflow).

It’s kind of simple when you describe it, but the big advantage is that your nix expressions don’t need to be aware of nix-pin. Your nix expressions stay idiomatic for easy inclusion in nixpkgs, and it already works with any idiomatic package definition you’ll find in nixpkgs.

nix-pin is still experimental, and there’s a chance it might break under certain awkward cases. But I think the idea is sound, and I’d love to see it get more adoption within the community, because I really think it can make development with nix much less painful.