GFX::Monk Home

Heads up: new handle

Just a quick heads up, in case anyone comes here looking for verification: I’ve changed my twitter & github handle from @gfxmonk to @timbertson.

I don’t have particular plans to rename this site, since that’s a lot more complex and it would break All The Links. But “gfxmonk” was a name I picked when I was big into computer graphics (and not a monk, though it sounded cool at the time). A decade or so later I’m more of an enthusiastic observer when it comes to graphics, and still not all that monk-like. So I figured I’d adopt a handle that was based on my name, since I probably won’t decide to change that any time soon.

Figuring out what transducers are good for (by trying to use them for a bunch of problems in JavaScript)

I’ve been aware of transducers for a little while, but haven’t actually used them, or even really felt like I fully grokked what they were good for. They come from the clojure community, but are making their way into plenty of other languages and libraries too. I’ve seen claims that they are a game-changing, breathtaking new concept, which didn’t really square with what they looked like.

So I thought I’d learn more about them by just attempting some plausible but detailed examples with them in JavaScript. If you’ve heard about transducers but aren’t really sure what they’re good for, perhaps this’ll help clarify. And if you’ve never heard of transducers, feel free to take a detour via the clojure documentation.

Force a specific shell for sshd

I use fish-shell as my default shell on my own computer, because it’s a pretty nice shell.

Occasionally, though, this causes issues. Software tests in particular have a habit of sloppily running shell command. Typically, this can be fixed by just being more explicit, using execvp(['bash', '-c', '<command>']) (or just use execv* directly instead of going through a shell).

But one case I couldn’t figure out is SSH. When you’re testing an SSH client, the most reliable way to do that is to run some shell scripts over a local SSH session, and check that it does what you expect. SSH has no way of passing a nice array of arguments, all you get is a string which will be interpreted by the user’s SHELL.

If you want to use bash for an SSH command, many online resources will tell to you run “ssh bash ...". But that won't work for tests, which want to run _real_ commands against a POSIX shell (including edge cases around argument parsing). Other suggestions include changing your default shell, but I don't want to forsake my preferred shell just to appease some automated tests!

What you can do however, is funnel the whole shell string through to your desired shell using this slightly underhand sshd configuration:

ForceCommand sh -c 'eval "$SSH_ORIGINAL_COMMAND"'

This still requires a default shell that’s normal enough to not do any interpolation within single quotes, but that’s a much simpler requirement (and true of fish-shell).

For the Conductance test suite, we run an unprivileged SSHD daemon with its own checked-in config during testing. So applying this globally is fine. But if you are doing this on actual SSH server, you might want to use a Match directive to make sure this rule only applies to trusted user (e.g I haven’t tested how this works on a locked-down account with its shell set to /bin/nologin, it could conceivably create a security hole).

OS Technologies To Watch

It’s the new year, and it seems to be a vibrant time for novel Operating System technologies. This is not intended to be an objective list of “the best things”, it’s just some up-and-coming technologies that I’m particularly excited about right now:

NixOS and Stateless Deployment

If I had my way, I would never deploy or administer a linux server that isn’t running NixOS.

I’m not exactly a prolific sysadmin - in my time, I’ve set up and administered servers numbering in the low tens. And yet every single time, it’s awful.

Firstly, you get out of the notion of doing anything manually, ever. Anytime you do something manually you create a unique snowflake, and then 3 weeks (or 3 years!) down the track you tear your hair out trying to recreate whatever seemingly-unimportant thing it is you did last time that must have made it work.

So you learn about automated deployment. There are no shortage of tools, and they’re mostly pretty similar. I’ve personally used these, and learned about many more in my quest not to have an awful deployment experience:

All of these work more or less as advertised, but all of them still leave me with a pretty crappy deployment experience.

The problem

Most of those are imperative, in that they boil down to a list of steps - “install X”, “upload file A -> B”, etc. This is the obvious approach to automating deployment, kind of like a shell script is the obvious approach to automating a process. It takes what you currently do, and turns it into one or more concrete files that you can modify and replay later.

And obviously, the entire problem of server deployment is deeply stateful - your server is quite literally a state machine, and each deployment attempts to modify its current state into (hopefully) the expected target state.

Unfortunately, in such a system it can be difficult to predict how the current state will interact with your deployment scripts. Performing the same deployment to two servers that started in different states can have drastically different results. Usually one of them failing.

Puppet is a little different, in that you don’t specify what you want to happen, but rather the desired state. Instead of writing down the steps required to install the package foo, you simply state that you want foo to be installed, and puppet knows what to do to get the current system (whatever its state) into the state you asked for.

Which would be great, if it weren’t a pretty big lie.

The thing is, it’s a fool’s errand to try and specify your system state in puppet. Puppet is built on traditional linux (and even windows) systems, with their stateful package managers and their stateful file systems and their stateful user management and their stateful configuration directories, and… well, you get the idea. There are plenty of places for state to hide, and puppet barely scratches the surface.

If you deploy a puppet configuration that specifies “package foo must be installed”, but then you remove that line from your config at time t, what happens? Well, now any servers deployed before t will have foo installed, but new servers (after t) will not. You did nothing wrong, it’s just that puppet’s declarative approach is only a thin veneer over an inherently stateful system.

To correctly use puppet, you would have to specify not only what you do want to be true about a system, but also all of the possible things that you do not want to be true about a system. This includes any package that may have ever been installed, any file that may have ever been created, any users or groups that may have ever been created, etc. And if you miss any of that, well, don’t worry. You’ll find out when it breaks something.

So servers are deeply stateful. And deployment is typically imperative. This is clearly a bad mix for something that you want to be as reproducible and reliable as possible.

Puppet tries to fix the “imperative” part of deployment, but can’t really do anything about the statefulness of its hosts. Can we do better?

Well, yeah.

Escaping an array of command-line arguments in C#

Let’s say you have an array of strings:

args = [ "arg1", "an argument with whitespace", 'even some "quotes"']

..and you want to pass them to a command, exactly as is. You don’t want it split on spaces, you don’t want quotes to disappear. You just want to pass exactly these strings to the command you’re running. In python, you would do something like:

subprocess.check_call(["echo"] + args)

In low-level C, it’s more effort, but it’s not really harder - you just use the execv* family of system calls, which takes an array of strings. At least on a UNIX-like OS.

But what if you’re using C# on Windows? Then it’s going to cost you a veritable screenful of code if you want to not screw it up. And you’ll probably screw it up. The internet has plenty of examples that happen to work well enough for simple data. But then they break when you add spaces, or double quotes, or backslashes, or multiple backslashes followed by a double quote. You don’t want that code. You want this code.

I’m honestly floored that nobody has published this code anywhere before (that I could find). So with the firm belief that it’s insane for anybody to have to implement this ridiculous escaping scheme for themselves, here it is: