GFX::Monk Home

Offcuts

An aside: Top and Bottom:

I’ve come to think of ZeroInstall and Nix as a kind of top and bottom state of software distribution.

It may be a stretched analogy, but as I see it:

  • a ZeroInstall implementation is a very “pure” description of a piece of software. It defines an exact implementation (an archive of files) which is completely relocatable. It also includes explicit and complete dependencies (also relocatable), and specifies exactly how the environment is to be set up such that all dependencies are available. This operation (setting environment variables) is fast, cheap, stateless (no I/O), and isolated.

Because of the above, I believe ZeroInstall implementations sit at the top of an imaginary pile of package formats - given a ZeroInstall implementation, it is almost certainly possible to automate the conversion to some more specific package format (like apt / rpm). The reverse is not true - you cannot convert an apt or rpm package into a ZeroInstall implementation trivially, because they are frequently nonportable, and often rely on stateful behaviour (e.g install scripts) or other specifics of the distribution or repository which they belong to.

  • Nix, on the other hand, sits at the very bottom of the pack. Given a nix expression for a package, you probably couldn’t easily convert it into another format, or use it along with anything but nix-specific tools, as it hardcodes all sorts of information and structure that only makes sense in the context of a nix install.

The upside of this is that, usually, it’s incredibly easy to convert some other (more flexible) package description (even just build scripts) into a nix expression, because it can support pretty much everything you throw at it. All paths can be hard-coded, dependencies are provided as easily as string interpolation, having multiple versions of package available is as natural as having multiple instances of a class in OO programming. And best of all, its a very efficient and low-maintenance system - caching is pervasive and always correct, thanks to its purity and use of cryptographic digests of all inputs that make up a given expression.

So while I believe that a ZeroInstall feed is the best thing a developer can support (because it can be used for any other purpose), for the end-user nix is incredibly versatile (because you can put anything into it, and it also has many of the same user-facing benefits as Zero Install. But as a developer, providing only a nix expression for your software would be rather silly, because nothing else could use it.


So if you’re using Docker for its provisioning aspects, it could be better. But its isolation features are still quite useful. Unless you’re running NixOS, which has builtin support for isolated containers as part of its declarative config. For example, here’s how you can run a postgres server in an isolated container:

containers.database = {
	config = { config, pkgs, ... }: {
			services.postgresql.enable = true;
			services.postgresql.package = pkgs.postgresql92;
		};
	privateNetwork = true;
	hostAddress = "192.168.100.10";
	localAddress = "192.168.100.11";
};

This is pretty concise, and specifies only what you need. And as a bonus, there’s no split between “packages in the container” and “packages on the host” - reapplying the machine config after a security update will affect both the host and container. And because this expression lives at the same level as the rest of your system config, you can use abstractions (functions, modules, shared variables) to reduce duplication between container and host, or even multiple containers.

Hopefully it’s obvious why this would be better than puppet. With puppet, anything left unspecified is implicitly left alone, as “whatever state was there previously”. But with NixOS, you specify the entire system - anything not specified will not be present on the system (even if it used to be). This sounds extremely inefficient, but nix’s purity and caching means that “rebuilding the entire OS” after changing some config is often quicker than applying that same change via puppet.

So while puppet succeeds at layering a declarative system on top of a stateful machine, its non-exhaustiveness (and other impurities) make it a poor solution.

As I described at the start of this post, puppet tries to overlay a declarative system onto a stateful model. It definitely succeeds, but I don’t think the result is actually useful.

While puppet provides a declarative system specification for a machine, it’s one of the leakiest abstractions I’ve ever encountered - applying the same puppet config to different machines can give wildly different results, because it’s totally dependent on the previous state of the system. This is because the state specified by any given puppet config is a tiny fraction of the machine’s possible state, and anything left unspecified effectively means “whatever is currently there”. So you end up having to specify the absence of things when you realise that’s important, like saying “this file / user / package should not be present”. So puppet makes you specify not only things that should be the case, but also everything that should not be the case, which is very difficult to do with any confidence.


Ending a stream in nodejs

stream.end();

Nope. Nodejs is single-threaded, and almost nothing useful happens synchronously. At this point, you have registered your interest in ending the stream, but some I/O (and possibly even further logic) needs to happen before that’s actually done. So your data isn’t actually written, and can still fail.

Ok then, lets use a callback to get notified of when the stream is actually done!

stream.end(null, null, function() {
	console.log("OK, now it's done!");
});

Well, yes and no.

This is part of the new nodejs stream API, introduced in version 0.10. But many third-party “streams” don’t implement it, and will silently ignore any arguments you pass to end(). Which means your “after end” code may never run, depending on how the stream is implemented.

Ok, so let’s use events! Events are The NodeJS Way(tm)!

stream.on('end', function() {
	console.log("OK, now it's done!");
});
stream.end();

We’re getting close. And in fact, I believe this is correct for streams that are part of the code API in nodejs<0.10. But it’s wrong if you’re on nodejs 0.10 or later, and it’s also wrong if you’re on nodejs 0.8 but using a third-party stream which supports nodejs 0.10.

In nodejs 0.10 and later, streams emit end when they’re ending, but not necessarily done with whatever I/O needs to happen after that. So you need to wait for the finish event, instead:

stream.on('finish', function() {
	console.log("OK, now it's done!");
});
stream.end();

Great!

Oh, except that (just like the callback version of end()), some APIs don’t implement this. So again, your code just sits there waiting forever, and you don’t know why it’s stuck.

But it turns out some (most? I have no idea) of those stream implement close instead[1]. So no worries, you can just wait for both close and finish.

var done = function() {
	console.log("OK, now it's done!");
} ;
stream.on('close', done);
stream.on('finish', done);
stream.end();

..and now you’re running done (a.k.a “the rest of your program”) twice in some circumstances, because a lot of streams emit both close and finish.

var alreadyDone = false;
var done = function() {
	if(alreadyDone) return;
	else alreadyDone = true;
	console.log("OK, now it's done!");
} ;
stream.on('close', done);
stream.on('finish', done);
stream.end();

Ok, so now we’re listening for close or finish, and only running the rest of our code once.

..but of course that won’t work on nodejs 0.8.

So you have to make a choice: do you wait for end? For some streams, this leaves you with incomplete data. For other streams, it’s the only correct thing to do. On the other hand you can wait for both finish and close, and make sure you don’t accidentally run the rest of your program multiple times. That’ll work, as long as you’re not dealing with a stream that only emits end, in which case your code will never continue.

Are we having fun yet?

What happens if you do it wrong?

Well, if you’re incredibly lucky, your code will stop doing anything and you’ll have a clue why.

If you’re less lucky, your code will stop doing anything and you’ll have no idea why.

..and if you’re even less lucky, everything will look like it’s working. Except that data will be lost because you’re not waiting for the right event, and your code will prematurely continue. One fun example of this is unpacking a tar file: if you wait for the wrong end event, files that should have been extracted are simply missing from your disk. Which is always a good feature of archive extraction. And since it depends on timing, it may even work properly 90% of the time. Even better, right?

Nixos

If I had my way, I would never deploy or administer a linux server that isn’t running NixOS.

I’m not exactly a prolific sysadmin - in my time, I’ve set up and administered servers numbering in the low tens. And yet every single time, it’s awful.

Firstly, you get out of the notion of doing anything manually, ever. Anytime you do something manually you create a unique snowflake, and then 3 weeks (or 3 years!) down the track you tear your hair out trying to recreate whatever seemingly-unimportant thing it is you did last time that must have made it work.

So you learn about automated deployment. There are no shortage of tools, and they’re mostly pretty similar. I’ve personally used these, and learned about many more in my quest not to have a sucky deployment experience:

  • capistrano
  • fabric
  • puppet
  • ansible

Even after trying all of these tools, I still have a sucky deployment experience.

The problem

Most of those are imperative, in that they boil down to a list of steps - “install X”, “upload file A -> B”, etc. This is the obvious approach to automating deployment, kind of like a shell script is the obvious approach to automating a process. It takes what you currently do, and turns it into one or more concrete files that you can modify and replay later.

And obviously, the entire problem of server deployment is deeply stateful - your server is quite literally a state machine, and each deployment attempts to modify its current state into (hopefully) the expected target state.

Of course, in such a system it’s very difficult to predict how the current state will interact with your deployment scripts. Performing the same deployment to two servers that started in different states can have drastically different results. Usually one of them failing, because you didn’t realise your deployment depended on some state that was (perhaps accidentally) enacted by a previous deployment (perhaps a failed one) somewhere in the entire history of your server.

Puppet is a little different, in that you don’t specify what you want to happen, but rather the desired state. Instead of writing down the steps required to install the package foo, you simply state that you want foo to be installed, and puppet knows what to do to get the current system (whatever its state) into the state you asked for.

Which would be great, if it weren’t a pretty big lie.

The thing is, it’s a fool’s errand to try and specify your system state in puppet. Puppet is built on traditional linux (and even windows) systems, with their stateful package managers and their stateful file systems and their stateful user management and their stateful condiguration directories, and… Well, you get the idea. There are plenty of places for state to hide, and puppet barely scratches the surface.

If you deploy a puppet configuration that specifies “package foo must be installed”, but then you remove that line from your config at time T, what happens? Well, now any servers deployed before T will have foo installed, but new servers (after T) will not. You did nothing wrong, it’s just that puppet’s declarative approach is only a thin veneer over an inherently stateful system.

To correctly use puppet, you would have to specify not only what you do want to be true about a system, but also all of the possible things that you do not want to be true about a system. This includes any package that may have ever been installed, any file that may have ever been created, any users or groups that may have ever been created, etc. And if you miss any of that, well, don’t worry. You’ll find out when it breaks something.

Enter Nix

So servers are deeply stateful. And deployment is typically imperative. This is clearly a bad mix for something that you want to be as reproducable and reliable as possible.

Puppet tries to fix the “imperative” part of deployment, but can’t really do anything about the statefulness of its hosts. Can we do better?

Well, yeah.

Nix is a novel package manager where the entire set of packages exists as a single expression in a pure, lazy programming language. Which is pretty whacky to start with. Internally, package names don’t matter - you reference actual expressions. Whatever your toplevel expression (a list of installed packages) references will be built only when referenced, and so on for their dependencies. And because it’s pure, it doesn’t rebuild anything that hasn’t changed, but always rebuild something that has changed. This can cause unnecessary rebuilds when the change wasn’t “important”, but that’s a small price to pay for always having a consistent result.

The language is declarative and fairly simple, but it’s not dumb (like XML or JSON) - it has modules, functions, data types, and all the rest.

This is useful for various reasons on its own. But where nix really shines is when you extend its reach from being a package system to (very nearly) an entire OS.

Because the nix language is both pure and lazy, it evaluates only what you need.

Nix is a package manager that does away with the weak linking of packages having a globally unique name, but instead defines all packages inside a pure, functional programming language. In nix, you don’t refer to package named “python”, but rather to the expression pkgs.python. This may seem like a spurious difference, but it’s an important one. Now that packages are expressions, you can pass them around, and substitute different ones. You can depend on pkgs.python, pkgs.python27, or just as easily, a version of python that is built with your own custom patches. This needs no globally unique name, it’s just like an object instance in a process - people with a reference to it can use it, and it has no global presence so it won’t interfere with anything else in the system.

Nix has a novel approach to package management, and NixOS extends that model to almost the entire system.

I am fairly opinionated when it comes to packaging and software distribution. Years ago I discovered Zero Install, which I believe is a tremendously elegant system for software distribution. But most of the new packaging systems I encounter are a re-hash of the same old patterns - usually a home rolled system that works for the very narrow scope of a single language.

So when I say that I’m rather impressed with nix (and nixos), that’s pretty high praise - I’ve though about software distribution a lot, and am not easily impressed.

thoughts on code grepping

We developers look through code a lot.

  • Ack is good, but its options for ignoring files are pretty weak. I often end up manually specifying a number of folders to avoid getting useless results, which is very slow, and means I sometimes miss useful results because I didn’t think to include them.

  • the_silver_searcher (ag) is faster than ack. It’s written in C (so manual compilation required), but I don’t mind that too much. It claims to support .gigitnore files and .agignore files, but it is wrong. Or at least, this feature has been totally broken in both versions I’ve tried so far. I may try to fix this, but programming in C usually annoys me, so I might also give up.

  • Selecta handles “just the fuzzy finding UI”, which feels quite unixy. It means you can pipe any file-source into it (git ls-files, find, a list of manually-maintaned quick-access paths. Or even stuff that has nothing to do with files (like ps output).

    It doesn’t integrate with gvim though (it only interacts via /dev/tty, which works in terminal vim, but I use gvim).

    It can’t really do caching. Ideally, I’d like my file-finding tool to start up quick with a cache from last time, and then revise the set of files incrementally as it updates this cache with the current on-disk state.

    It also waits for all of stdin to be complete before it lets you start typing, which is a bust. This may be easily fixable, but it’s ruby, so that might mean fun with threads. Given the entire program is ~600 lines, it may be easier to port it to something with cooperative threading. Ocaml is starting to look like a pretty nice hammer lately (particularly for quick start-up utilities), if you can stomach the need for compilation.

  • A long time ago I wrote file-finder, which works pretty well as a persistent process that sits in a terminal. It uses inotify to watch for changes, so it should always be up to date.

    But you need to keep it running in a dedicated terminal window for it to be quick (on startup, it just scans the disk like a chump).

    You have to massage its command-line options carefully to specify ignore rules, and they’re pretty rudimentary.

    By default, it opens files with gvim --remote-tab, which is what I usually want. But you may not want that, and even then it opens files in the first vim instance it finds, which may not be the one you wanted. So often I have to resort to ctrl+c, and then go into vim and do :tabedit <ctrl+v>. Which really takes the “fast” out of “fast file opening”.

So what I want is:

1) Most likely, a canonical “source of files” program or library. This needs to be as hands-off as possible - it should support .gitignore and friends by default, and make it unlikely that I even need to configure it specially. It should allow its own .ignore file as overrides, because sometimes I want to include files in searches but not git (and vice versa).

2) A grep program that gets it files from (or at least can use the same ignore logic as) (1). Returning matches in files I don’t care about is way worse than being slightly slower, because my brain works much slower than a computer to filter unwanted results.

  • using a good grep porgram like ack or ag for (1) would get me this for free, but both of those have poor ignore functionality (ack’s functionality is rudimentary, while ag should work in theory but is completely busted in both versions I tried, and doesn’t have a commandline option to just dump the file list)

3) A fuzzy finder that can integrate with my editor, and get its files from (1). It may have a plain CLI mode as well, but to be honest I don’t see myself using that much.

Escaping an array of command-line arguments in C#

Let’s say you have an array of strings:

args = [ "arg1", "an argument with whitespace", 'even some "quotes"']

..and you want to pass them to a command, exactly as is. You don’t want it split on spaces, you don’t want quotes to disappear. You just want to pass exactly these strings to the command you’re running. In python, you would do something like:

subprocess.check_call(["echo"] + args)

In low-level C, it’s more effort, but it’s not really harder - you just use the execv* family of system calls, which takes an array of strings. At least on a UNIX-like OS.

But what if you’re using C# on Windows? Then it’s going to cost you a veritable screenful of code if you want to not screw it up. And you’ll probably screw it up. The internet has plenty of examples that happen to work well enough for simple data. But then they break when you add spaces, or double quotes, or backslashes, or multiple backslashes followed by a double quote. You don’t want that code. You want this code.

I’m honestly floored that nobody has published this code anywhere before (that I could find). So with the firm belief that it’s insane for anybody to have to implement this ridiculous escaping scheme for themselves, here it is:

Doing stuff when files change

There’s a common pattern in development tools to help with rapid feedback: you run a long-lived process that does a certain task. At the same time, it watches the filesystem and restarts or re-runs that task whenever a file that you’re monitoring changes.

This is an extremely useful tool for rapid feedback (which is why we’ve integrated nodemon into our Conductance app server), but is not very flexible - most tools are integrated into a web framework or other environment, and can’t easily be used outside of it. There are a few generic tools to do this kind of thing - I personally use watchdog a lot, but it’s sucky in various ways:

  • Configuring it to watch the right file types is hard
  • Configuring it to ignore “junk” files is hard, leading to infinite feedback loops if you get it wrong
  • It sees about 6 events from a single “save file” action, and then insists on running my build script 6 times in a row
  • It takes a bash string, rather than a list of arguments - so you have to deal with double-escaping all your special characters

And yet for all of those issues, I haven’t found a better tool that does what I need.

My build workflow

Lately, I’ve been making heavy use of my relatively-new build system, gup. It’s a bit like make, but way better in my humble-and-totally-biased-opinion. But going to a terminal window and typing up, enter (or the wrist-saving alternative crtl-p, ctrl-m) to rebuild things is tedious. But there’s no way I’m going to implement yet another watch-the-filesystem-and-then-re-run-something gup-specific tool, at least not until the lazy alternatives have been exhausted.

Obviously, my workflow isn’t just up, enter. It’s (frequently):

  • save file in vim
  • go to terminal
  • press up, enter
  • go to browser
  • refresh

And you know what? Monitoring every file is kind of dumb for this workflow. I don’t have gremlins running around changing files in my working tree at random (I hope), I almost always want to reload in response to me changing a file (with vim, of course). So why not just cooperate?

The simple fix

So I’ve written a stupid-dumb vim plugin, and a stupid-dumb python script. The vim plugin touches a file in $XDG_USER_DIR whenever vim saves a file. And then the script monitors just this file, and does whatever you told it to do each time the file is modified. The script automatically communicates with vim to enable / disable the plugin as needed, so it has no overhead when you’re not using it.

It’s called vim-watch, and I just gave you the link.

Addendum: restarting servers

While writing this post, I was a little disappointed that it still doesn’t quite replace tools that automatically restart a server when something changes, because it expects to run a build-style command that exits when it’s done - but servers run forever. Some unix daemons (like apache) restart themselves when you send them a HUP signal, but that’s not so common in app servers. So now huppy exists, too.

It’s a tiny utility that’ll run whatever long-lived process you tell it to, and when it receives a HUP signal it’ll kill that process (with SIGINT) if it’s still running, then restart it. It seems surprising that this didn’t exist before (maybe my google-fu is failing me) but on the other hand it’s less than 60 lines of code - hardly an expensive wheel to reinvent.

You can use it like:

$ # step 1: start your server
$ huppy run-my-server

$ # step 2: use vim-watch to reload the server on changes
$ vim-watch killall -HUP huppy

$ # Or, if you need to rebuild stuff before restarting the server,
$ vim-watch bash -c 'gup && killall -HUP huppy'