GFX::Monk Home

Posts tagged: linux

nix-wrangle: Manage nix sources and dependencies with ease

My last post was more than a year ago, in which I described my long journey towards better package management with nix.

Well, it turns out that journey wasn’t over. Since then, I’ve been using the tools I created and found them unfortunately lacking.

So, I’ve built just one more tool, to replace those described in the previous post.

I’m writing this assuming you’ve read the previous post, otherwise I’d be repeating myself a lot. If you haven’t, just check out nix-wrangle and don’t worry about my previous attempts ;)

Step 7: nix-wrangle for development, dependency management and releases

After a lot of mulling over the interconnected problems described in that previous post (and discussions with the creators of similar tools), I came at it from a new direction. Nix wrangle is the result of that approach. I’ve been using it fairly successfully for a while now, and I’m ready to announce it to the world.

How does it differ from my previous state of using nix-update-source and nix-pin?

  • One JSON file for all dependencies, allowing bulk operations like show and update, as well as automating dependency injection (since all dependencies are known implicitly).
  • A splice command which takes a nix derivation file, injects a synthetic src attribute baking in the current source from the nix-wrangle JSON file. This allows you to keep an idiomatic nix file for local development (using a local source), and automatically derive an expression using the latest public sources for inclusion in nixpkgs proper.
  • Project-local development overrides. The global heuristic of nix-pin caused issues and some confusion, nix-wrangle supports local sources for exactly the same purpose, but with an explicit, project level scope.

Please check it out if you use nix! And if you don’t use nix, check that out first :)

A journey towards better nix package development


I’ve continued my journey, and described my next steps (and implementation) in a new post.

This post is targeted at users of nix who write / maintain package derivations for software they contribute to. I’ve spent a lot of time doing (and thinking about) this, although it’s probably quite a niche audience ;)

tl;dr: you should check out nix-pin and nix-update-source if you want to have a happy life developing and updating nix expressions for projects you work on.

I believe nix is a technically excellent mechanism for software distribution and packaging.

But it’s also flexible enough that I want to use it for many different use cases, especially during development. Unfortunately, there are a few rough edges which make a good development setup difficult, especially when you’re trying to build expressions which serve multiple purposes. Each of these purpose has quite a few constraints:

Running gnome-shell nested in a Xephyr window

TL;DR: install nix and Xephyr, then try this script.

I’ve worked on a GNOME Shell tiling window extension (shellshape) for 5 years now, since before the first release of gnome-shell. The shell itself is impressively extensible, and it’s pretty amazing that I can distribute a tiling window extension which as just a bunch of javascript. But the development process itself has always been awful:

  • you have to restart your window manager all the time, which typically loses the sizing and workspace affinity of every window, leaving you with a tangled mess of windows
  • if your extension doesn’t work then you have a broken shell
  • it is painfully easy to cause a segfault (from JavaScript code :( )
  • you’d better be editing your code in a tmux session so you can fix it from a VTE
  • sometimes when restarting the shell, all your DBus-based integrations get messed up so you can’t change volume, use multimedia keys or shutdown
  • testing against a new gnome-shell version basically means either upgrading your OS or trying to do a fresh install in a VM, which is a whole new layer of annoyance.

Maybe I’m spoiled from working on projects which are easily run in isolation - I bet kernel developers scoff at the above minor inconveniences. But it makes development annoying enough that I dread it, which means I’ll only fix bugs when they get more annoying than development itself.

All of which is to say that this is freakin’ awesome. As of a couple days ago I’ve been able to run the latest version of GNOME Shell (which isn’t packaged for my distro) in a regular window, completely disconnected from my real session, running the development version of shellshape.

Big thanks go to whichever mysterious developers were responsible for fixing whatever gnome-shell / graphics / Xephyr issues have always prevented gnome-shell from running nested (it does now!), and to the nixpkgs folks maintaining the latest GNOME releases so that I can run new versions of GNOME without affecting the rest of my system.

Unfortunately I can’t guarantee it’ll work for you, since this stuff is heavily dependant on your graphics card and drivers, plus it only seems to work with my system version of Xephyr, not the nixpkgs one. But if this interests you, you should definitely give it a go. You’ll need nix and Xephyr. If you don’t want to use nix, you can probably extract what you need from the script to run your system version of gnome-shell in a Xephyr window.

Force a specific shell for sshd

I use fish-shell as my default shell on my own computer, because it’s a pretty nice shell.

Occasionally, though, this causes issues. Software tests in particular have a habit of sloppily running shell command. Typically, this can be fixed by just being more explicit, using execvp(['bash', '-c', '<command>']) (or just use execv* directly instead of going through a shell).

But one case I couldn’t figure out is SSH. When you’re testing an SSH client, the most reliable way to do that is to run some shell scripts over a local SSH session, and check that it does what you expect. SSH has no way of passing a nice array of arguments, all you get is a string which will be interpreted by the user’s SHELL.

If you want to use bash for an SSH command, many online resources will tell to you run “ssh bash ...". But that won't work for tests, which want to run _real_ commands against a POSIX shell (including edge cases around argument parsing). Other suggestions include changing your default shell, but I don't want to forsake my preferred shell just to appease some automated tests!

What you can do however, is funnel the whole shell string through to your desired shell using this slightly underhand sshd configuration:

ForceCommand sh -c 'eval "$SSH_ORIGINAL_COMMAND"'

This still requires a default shell that’s normal enough to not do any interpolation within single quotes, but that’s a much simpler requirement (and true of fish-shell).

For the Conductance test suite, we run an unprivileged SSHD daemon with its own checked-in config during testing. So applying this globally is fine. But if you are doing this on actual SSH server, you might want to use a Match directive to make sure this rule only applies to trusted user (e.g I haven’t tested how this works on a locked-down account with its shell set to /bin/nologin, it could conceivably create a security hole).

NixOS and Stateless Deployment

If I had my way, I would never deploy or administer a linux server that isn’t running NixOS.

I’m not exactly a prolific sysadmin - in my time, I’ve set up and administered servers numbering in the low tens. And yet every single time, it’s awful.

Firstly, you get out of the notion of doing anything manually, ever. Anytime you do something manually you create a unique snowflake, and then 3 weeks (or 3 years!) down the track you tear your hair out trying to recreate whatever seemingly-unimportant thing it is you did last time that must have made it work.

So you learn about automated deployment. There are no shortage of tools, and they’re mostly pretty similar. I’ve personally used these, and learned about many more in my quest not to have an awful deployment experience:

All of these work more or less as advertised, but all of them still leave me with a pretty crappy deployment experience.

The problem

Most of those are imperative, in that they boil down to a list of steps - “install X”, “upload file A -> B”, etc. This is the obvious approach to automating deployment, kind of like a shell script is the obvious approach to automating a process. It takes what you currently do, and turns it into one or more concrete files that you can modify and replay later.

And obviously, the entire problem of server deployment is deeply stateful - your server is quite literally a state machine, and each deployment attempts to modify its current state into (hopefully) the expected target state.

Unfortunately, in such a system it can be difficult to predict how the current state will interact with your deployment scripts. Performing the same deployment to two servers that started in different states can have drastically different results. Usually one of them failing.

Puppet is a little different, in that you don’t specify what you want to happen, but rather the desired state. Instead of writing down the steps required to install the package foo, you simply state that you want foo to be installed, and puppet knows what to do to get the current system (whatever its state) into the state you asked for.

Which would be great, if it weren’t a pretty big lie.

The thing is, it’s a fool’s errand to try and specify your system state in puppet. Puppet is built on traditional linux (and even windows) systems, with their stateful package managers and their stateful file systems and their stateful user management and their stateful configuration directories, and… well, you get the idea. There are plenty of places for state to hide, and puppet barely scratches the surface.

If you deploy a puppet configuration that specifies “package foo must be installed”, but then you remove that line from your config at time t, what happens? Well, now any servers deployed before t will have foo installed, but new servers (after t) will not. You did nothing wrong, it’s just that puppet’s declarative approach is only a thin veneer over an inherently stateful system.

To correctly use puppet, you would have to specify not only what you do want to be true about a system, but also all of the possible things that you do not want to be true about a system. This includes any package that may have ever been installed, any file that may have ever been created, any users or groups that may have ever been created, etc. And if you miss any of that, well, don’t worry. You’ll find out when it breaks something.

So servers are deeply stateful. And deployment is typically imperative. This is clearly a bad mix for something that you want to be as reproducible and reliable as possible.

Puppet tries to fix the “imperative” part of deployment, but can’t really do anything about the statefulness of its hosts. Can we do better?

Well, yeah.

Doing stuff when files change

There’s a common pattern in development tools to help with rapid feedback: you run a long-lived process that does a certain task. At the same time, it watches the filesystem and restarts or re-runs that task whenever a file that you’re monitoring changes.

This is an extremely useful tool for rapid feedback (which is why we’ve integrated nodemon into our Conductance app server), but is not very flexible - most tools are integrated into a web framework or other environment, and can’t easily be used outside of it. There are a few generic tools to do this kind of thing - I personally use watchdog a lot, but it’s sucky in various ways:

  • Configuring it to watch the right file types is hard
  • Configuring it to ignore “junk” files is hard, leading to infinite feedback loops if you get it wrong
  • It sees about 6 events from a single “save file” action, and then insists on running my build script 6 times in a row
  • It takes a bash string, rather than a list of arguments - so you have to deal with double-escaping all your special characters

And yet for all of those issues, I haven’t found a better tool that does what I need.

My build workflow

Lately, I’ve been making heavy use of my relatively-new build system, gup. It’s a bit like make, but way better in my humble-and-totally-biased-opinion. But going to a terminal window and typing up, enter (or the wrist-saving alternative crtl-p, ctrl-m) to rebuild things is tedious. But there’s no way I’m going to implement yet another watch-the-filesystem-and-then-re-run-something gup-specific tool, at least not until the lazy alternatives have been exhausted.

Obviously, my workflow isn’t just up, enter. It’s (frequently):

  • save file in vim
  • go to terminal
  • press up, enter
  • go to browser
  • refresh

And you know what? Monitoring every file is kind of dumb for this workflow. I don’t have gremlins running around changing files in my working tree at random (I hope), I almost always want to reload in response to me changing a file (with vim, of course). So why not just cooperate?

The simple fix

So I’ve written a stupid-dumb vim plugin, and a stupid-dumb python script. The vim plugin touches a file in $XDG_USER_DIR whenever vim saves a file. And then the script monitors just this file, and does whatever you told it to do each time the file is modified. The script automatically communicates with vim to enable / disable the plugin as needed, so it has no overhead when you’re not using it.

It’s called vim-watch, and I just gave you the link.

Addendum: restarting servers

While writing this post, I was a little disappointed that it still doesn’t quite replace tools that automatically restart a server when something changes, because it expects to run a build-style command that exits when it’s done - but servers run forever. Some unix daemons (like apache) restart themselves when you send them a HUP signal, but that’s not so common in app servers. So now huppy exists, too.

It’s a tiny utility that’ll run whatever long-lived process you tell it to, and when it receives a HUP signal it’ll kill that process (with SIGINT) if it’s still running, then restart it. It seems surprising that this didn’t exist before (maybe my google-fu is failing me) but on the other hand it’s less than 60 lines of code - hardly an expensive wheel to reinvent.

You can use it like:

$ # step 1: start your server
$ huppy run-my-server

$ # step 2: use vim-watch to reload the server on changes
$ vim-watch killall -HUP huppy

$ # Or, if you need to rebuild stuff before restarting the server,
$ vim-watch bash -c 'gup && killall -HUP huppy'

Passing arrays as arguments in bash

I tend to avoid bash wherever possible for scripting, because it has dangerously bad defaults and will happily munge your data unless you take great care to wrap it up properly (particularly whitespace). But sometimes you have no choice, so you might as well know how to do it safely.

Here’s how to capture argv as a bash array, and pass it on to another command without breaking if some argument contains a space:

echo "${args[@]}"

You can also just pass “$@” directly, but the above syntax works for any array.

Don’t forget any of those quotes, or bash will silently ruin everything (until you have data with spaces, at which point it might loudy ruin everything).

Here’s how to convert a line-delimited string (e.g a list of files in the current directory) into an array and pass that on:

mapfile -t arr <<<"$(ls -1)"
echo "${arr[@]}"

Note that a sensible-looking:

ls -1 | mapfile -t args

will not work, as a builtin on the receiving end of a pipe gets run in a subshell.

If you don’t have mapfile (added in bash v4), you’ll have to resort to:

oldIFS="$IFS"; IFS=$'\n' read -d '' -r -a arr <<< "$(ls -1)"; IFS="$oldIFS"; unset oldIFS
echo "${arr[@]}";

I look forward to the day when I don’t have to know that.

pkcon, the cross-distro package manager CLI

I seem to keep switching between Ubuntu and Fedora every 6 months, for various reasons. I think switching is good for you in that it allows you to evaluate the benefits of each distro, and also encourages you “travel light” as it were, discouraging one-off hacks that you’ll have to figure out each and every time you change distro (or potentially upgrade). That’s not to say I don’t do a heck-tonne of customisation, but I do it in a structured, repeatable and (when possible) cross-distro manner.

To be honest, the main thing that irks me is having to remember to mentally switch between apt-get/dpkg and yum/rpm. I try to declare package dependencies for tools I commonly use or libraries I depend on in a cross-platform way using zero-install feeds (this really helps when switching, since I don’t have to remember the differing names of many packages), but inevitably I still need to use the command-line package managers fairly often.

Today I ran across fpm (“Effing Package Management”) again, which is a tool that a (likely frustrated) user has built to abstract away the uninteresting details of building packages for such-and-such package manager. It turns out that system packages are boring, and do pretty much the same thing in 99% percent of cases. The other 1% is probably a bad idea, or obscure enough of a feature that you’ll do it wrong1. I took a closer look to see if it also dealt with the uninteresting inconsistencies of using a package manager to search for an install packages, but no such luck.

Enter pkcon

Fortunately, a pretty decent tool is already available, and already installed in many modern distros. It’s called pkcon, and it’s part of the PackageKit project to provide a consistent API for the common operations across the plethora of linux package managers. PackageKit has a whole heap of supported backends, most likely more than you’ve even heard of. It covers the basic operations that are generally present in all package managers, it does not attempt to expose advanced features. Fedora’s default package GUI is built on it, and I believe ubuntu’s software center either is already or soon will be built on it it too. It’s also the mechanism by which zero-install can install required system packages as needed on most distros, which earns it a special place in my heart ;)

Using it

Anyway, what can you do with pkcon? Close to everything I’ve ever needed to. Technically speaking it’s largely a testing tool for exercising the PackageKit API, but (by no coincidence) the PackageKit API happens to be a straightforward set of the main things a user actually cares about. For the most part, the commands are exactly what you’d expect if you’ve used apt or yum, and in other cases they make more sense. Here’s a list of the stuff I use, with apt and yum equivalents for context:

install (apt-get install, yum install)

$ pkcon install [packagename]

remove (apt-get remove, yum remove)

$ pkcon remove [packagename]

check for updates (apt-get update, yum check-update)

$ pkcon refresh

upgrade all packages (apt-get upgrade, yum update)

$ pkcon update

search (apt-cache search, yum search)

$ pkcon search [name|details|group|file] [query]

file is pretty useful when you want to find “who owns that file on my hard drive”, but unfortunately you can’t use wildcards to find files you’re missing but don’t know the full location of (at least it’s not working for me). Still, good for satisfying poorly-documented build dependencies that are expecting certain files to exist.

what-provides (rpm -qf, dpkg -S)

$ pkcon what-provides /usr/bin/python

Seems to be the same as pkcon search file above. Much easier to remember than the rpm/dpkg flags.

get-details (apt-cache show, yum info)

$ pkcon get-details [packagename]

I mainly use this for getting the version of a package installed, or to check that a given package is what I want to install.

get-files (dpkg -L, rpm -ql)

$ pkcon get-files python

This one even works on packages you haven’t installed, unlike both dpkg and rpm.

User Beware

Unfortunately, pkcon is not without its rough edges. While the package management actions are well-tested, the command line itself is a bit flaky. For example, both --help-all and refresh --force give parse errors on Fedora 17, despite being recommended in the help text. More importantly the --filter=installed flag does nothing on my system. Without that filter, you will need to disambiguate (by picking from a numbered list) whenever you perform an action on a package that is installed but also has an available update, or which is available in both 32 and 64 bit versions.

It also requires DBus as a dependency (since PackageKit is implemented as a DBus API), which may make it a poor option for administering servers. But you probably shouldn’t be doing most of this manually on a server anyway.

  1. Even relatively common features like debian’s postrm scripts are frequently used incorrectly to painful effect. I’ve seen multiple packages that forever have to workaround dumb bugs in the control scripts of earlier version of the package, because the author didn’t fully understand how they worked (full disclosure: I wrote one such package ;)). The reason you need to work around this forever is that you never know if the previous version a user had installed was one of the buggy versions (since it’s the outgoing version whose postrm script gets run). 

My new bash script prelude

For a while my preferred “bash script prelude” (the stuff you write in every bash script, before the actual content) has been a modest:

set -eux

Those settings break down to:

  • -e: fail when any subcommand fails (the number of scripts that proceed to do completely incorrect things after an earlier command failed is criminal). You can always silence errors by using || or an if statement, but (much like exceptions in a high-level language) silence should not be the default.
  • -u: fail when an unknown variable is referenced - mostly guards against silly typos, but why wouldn’t you? It’s sometimes not what you want with variables that may be optionally set by the user, so sometimes I’ll move the set -u down in the script after I’ve dealt with setting those to their defaults if they are not present.
  • -x: trace all commands, after expansion. E.g with a line like: x=123; echo $x, Your output will be:

    • x=123
    • echo 123 123

This gets very noisy, but frequently I find it’s better to have more output than less - when a script breaks, often you can tell why just by looking at the last command trace (especially when combined with the -e option). And it’s easier to ignore excess output than it is to guess what went wrong.

The new hotness

Today, I discovered / created a much longer (and uglier) prelude, but I think it’s going to be worth it. Here ‘tis:

set -eu
set -o pipefail
export PS4='+ ${FUNCNAME[0]:+${FUNCNAME[0]}():}line ${LINENO}: '
syslogname="$(basename "$0")[$$]"
exec 3<> >(logger -t "$syslogname")
echo "Tracing to syslog as $syslogname"
unset syslogname
debug() { echo "$@" >&3; }
set -x

A mouthful, indeed. Lets go though it line-by-line:

Systemd Socket Activation in Python

For those unaware: systemd is a replacement for the traditional unix init process. It is the first process to be brought up by the kernel, and is responsible for all the user-space tasks in booting a system. Conveniently, it also has a --user mode switch that allows you to use it yourself as a session-level init service. As someone who hates being root and loves repeatable configuration, I’m tremendously pleased to be able to offload as much of my session management as I can to something that’s built for the task (and doesn’t rely on hard-coded, root-owned paths).

I’ve heard a lot of complaining from sysadmins who seem to prefer a tangled mess of shell scripts in /etc/init.d/, but I’m a big fan of systemd for its user-friendly features - like restarting stuff when it dies, listing running services, showing me logging output, and being able to reliably kill daemons. There are other ways to accomplish all of these, but systemd is easy to use, and works very well in my experience.

Socket activation

Recently I was wondering how to implement systemd socket activation in python. The idea is much like inetd, in which you tell systemd which port to listen on, and it opens up the port for you. When a request comes in, it immediately spawns your server and hands over the open socket to seamlessly allow your program to service the request. That way, services can be started only as necessary, making for a quicker boot and less resource usage for infrequently accessed services.

The question is, of course, how am I supposed to grab a socket from systemd? It turns out it’s not hard, but there are some tricks. For my app, I am using the BaseHTTPServer module. Specifically, the threaded variant that you can construct using multiple inheritance:

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):

I couldn’t find much existing information on how to do this in python, so I thought I’d write up what I found. One useful resource was this tutorial for ruby, since python and ruby are pretty similar.

Nuts and bolts

Obviously, systemd needs to tell you that it has a socket open for you. The way it does this is by placing your process ID into $LISTEN_PID. So to tell if you should try and use an existing socket, you can check:

if os.environ.get('LISTEN_PID', None) == str(os.getpid()):
	# inherit the socket
	# start the server normally

Given that you normally pass a host and port into the HTTPServer class’ constructor, how can you make it bind to a socket systemd gives you? It turns out to be fairly simple:

class SocketInheritingHTTPServer(ThreadedHTTPServer):
	"""A HttpServer subclass that takes over an inherited socket from systemd"""
	def __init__(self, address_info, handler, fd, bind_and_activate=True):
		ThreadedHTTPServer.__init__(self, address_info, handler, bind_and_activate=False)
		self.socket = socket.fromfd(fd, self.address_family, self.socket_type)
		if bind_and_activate:
			# NOTE: systemd provides ready-bound sockets, so we only need to activate:

You construct this just like you would the normal ThreadedHTTPServer, but with an extra fd keyword argument. It passes bind_and_activate=False to prevent the parent class from binding the socket, overrides the instance’s self.socket, and then activates the server.

The final piece of the puzzle is the somewhat-arbitrary knowledge that systemd passes you in sockets beginning at file descriptor #3. So you can just pass in fd=3 to the SocketInheritingHTTPServer. If you have a server that has multiple ports configured in your .socket file, you can check $LISTEN_FDS

And that’s it! I’ve only just learnt this myself, so I may be missing some detail. But it seems to work just fine. If you want to see the full details, you can have a look at the commit I just made to edit-server, which includes a simple script to simulate what systemd does when it hands you an open socket, for quick testing. You can also have a look at the service files to see how the .socket and .service systemd units are set up.

Shellshape arrives on

shellshape logo

It’s been a long time coming, but shellshape (my tiling window manager extension for gnome-shell) is finally available via Get it while it’s hot!

You’ll need the bleeding edge (3.4.1) version of gnome-shell, as it’s the first version that allows normal extensions to register new keybindings. If you’re stuck on 3.4 you can still use the 0launch method described on the shellshape homepage.

(view link)

Why Piep

piep (pronounced “pipe”) is a new command line tool for processing text streams with a slightly modified python syntax, inspired by the main characters of a typical unix shell (grep, sed, cut, tr, etc). To pluck a random example. here’s how you might rename all files (but not directories) in the current folder to have a “.bak” extension (because you have a very strange and manual backup scheme, apparently):

$ ls -1 | piep 'not os.path.isdir(p) | sh("mv", p, p + ".bak")'

In this simple example we can see filtering, piping (note that the pipes between expressions are internal to piep’s single argument, and thus not interpreted by the shell), and shelling out to perform useful work.

Here’s another, to print out the size of files in the current directory that are greater than 1024 bytes:

$ ls -l | piep 'pp[1:] | p.splitre(" +", 7) | size=int(p[4]) | size > 1024 | p[7], "is", p[4], "bytes"'

Or, if hacking through the output of ls -l isn’t your thing (it’s most likely a terrible idea), you can do things the pythonic way:

$ ls -1 | piep --import 'stat' 'size=os.stat(p).st_size | size > 1024 | p, "is", size, "bytes"'

For a proper introduction, you should read the online documentation. But I wanted to address one specific point here, about the origins of piep.

Recently I came across pyp, The Pied Piper. It seemed like a great idea, but after I played with it for a little while I uncovered some unfortunate shortcomings, some of which are deal breakers. My list included:

  • stream-based operation: there’s a beta implementation with “turbo” (line-wise) mode, but it seems very limited. I believe it should be the norm, and wanted to see if I could do things in a way that was just as convenient, but with all the benefits of lazy stream-based processing.
  • Command execution: commands are made up by string concatenation, requiring manual effort to escape metacharacters including the humble space 1. Also, errors are silently ignored.
  • Purity of data: things are routinely strip()ed and empty strings are frequently dropped from computations. Breaking up a line into a list of data would (sometimes?) see each list merged back into the input stream, rather than maintained as a list.
  • stream confusion: second stream, file inputs, etc. Not really sure why there are so many special cases
  • a not-very-extensible extension mechanism, which is fairly manual and appears to preclude sharing or combining extensions
  • lots of unnecessary machinery that complicates the code: macros, history, –rerun, three file input types, etc. Some of this may be useful once you use the tool a lot, but it hindered my ability to add the features I wanted to pyp.

I initially tried my hand at modifying pyp to fix some of the things I didn’t like about it, but the last point there really got in my way. History is baked in, and doesn’t really work in the same manner for stream-based operations. The entire pp class had to be rewritten, which is actually what I started doing when I decided to turn it into a separate tool (since it then became difficult to integrate this new pp class with the rest of the system. Anyway, I hope this isn’t taken as an offence by the developers of pyp - I really like the ideas, so much so that I was compelled to write my ideal version of them.

  1. I humbly submit that concatenating strings is the worst possible way to generate shell commands, leading to countless dumb bugs that only rear their heads in certain situations (and often in cascading failures). Observe piep’s method on a filename containing spaces:

    $ ls -1 | piep 'sh("wc", "-c", p)'
    82685610 Getting the Most Out of Python Imports.mp4

    Compared to that of pyp (and countless other tools):

    $ ls -1 | pyp 'shell("wc -c " + p)'
    wc: Getting: No such file or directory
    wc: the: No such file or directory
    wc: Most: No such file or directory
    wc: Out: No such file or directory
    wc: of: No such file or directory
    wc: Python: No such file or directory
    wc: Imports.mp4: No such file or directory
    [[0]0 total]
    $ echo $?

    It is unacceptable for a language with simple and convenient sequence types to instead rely on complex string escaping rules to prevent data from being misinterpreted. To be honest, this on its own may be reason enough to use piep over alternatives. 

Cool projects I learnt about at lca2012

I went to this week, and learnt about some pretty awesome tech (as well as hearing some entertaining, inspiring, touching and terrifying talks from the likes of Paul Fenwick, Bruce Perens, Karen Sandler and Jacob Appelbaum).

So here’s a quick dump of cool projects I learnt about, with links where I could find them.

browser id:

Openid is not so great, due to:

  • usability / confusion (requiring a url as an id)
  • reliability (if your provider goes down, you can’t log in)
  • lock in, (hard for the user to migrate providers)
  • privacy (your provider knows every url you log into, every time)

Browserid serves one simple purpose: to prove you own an email. It’s distributed. The browser, (not a third-party server) is the login intermediary.

As an example: The login process to gmail generates a shortlived (in the order of hours or maybe days, I guess) cryptographically signed statement that you own that email, which your browser stores. Other sites just need to grab gmail’s public key and then they can themselves verify that the assertion you sent them proves you own (or can log in to)

Demo site:

Not awesome yet: There’s no browser or email provider support. But it’s developed by mozilla, has scaffolding in js to work already in all browsers. Certs are stored on for now, until browsers implement native support (so they are just as bad as a server intermediary, but only as a stopgap).

mediastreams processing api:

Very cool demos, which are online (although they require a dev build of firefox).

css calc()

Computation in css. Very cool. There are implementations in IE and Firefox so far, and work on webkit is in progress. This is not just a convenience like SASS and friends - it allows for previously impossible mixing of units like 100% - 10px.

News on arbitrary metadata in git:

Apparently many people feel this is important for VCS interop (git-svn, fastimport, etc), and there are proposals for adding support (but it’s difficult to figure out how to do right in a way that won’t prohibit other useful things in the future).


Uses orchestra by ubuntu for deployments.

gerrit for code review. Use Jenkins plugins, pushes state notifications to launchpad. openstack wrote a patch submission tool: git-review. Rebases & pushes patchset to gerrit.

testing ctdb

Autocluster is a tool for testing clustered samba (and other things, presumably). Virtual kvm based clusters. Uses guestfish to manage volumes inside the kvm guests.

dbench for automated performance tests. Can describe any kind of IO workload.

Assessing the potential impact of package updates

Tools for spelunking package relationships (on debian at least):

  • Recursive depends, reverse depends, apt-cache, germinate.

misc projects / pointers

sozi is an inkscape plugin for chopping up vectors into slide presentations.

fossil scm is a VCS that tracks docs, wikis, bugs, etc in with the source code. Cool idea, apparently some bad implementation decisions though.

safe-rm: replaces rm and has a blacklist of folders you probably don’t really want to remove, such as /usr/. No more bumblebeeing (can’t find the link, but there was once a bug in the bumblebee uninstall script that removed all of /usr as root).

handbag makes android accessory dev easy(er)

libvirt-sandbox: new library and command-line tools for app-level sandboxing with LCX and/or KVM. Should be coming in Fedora 17.


freedombox: 100% free tiny personal server project. Uses all open-source & federated social software like diaspora, buddycloud

instamorph: malleable plastic that’s solid at room temperature, for making ad-hoc hooks, connectors, docks, etc. The aussie equivalent is apparently called “Polymorph”.

Shellshape: A Tiling Window Manager for Gnome Shell

shellshape Today I released the first version of shellshape, a tiling window manager plugin for gnome-shell. It’s definitely pre-alpha software, and currently requires a custom fork of the mutter window manager. I’ve had some trouble getting it running due to awful packaging things (I now know far too much about dynamic linking path resolution on linux), but it should work, at least on Fedora 15. Please give it a go if you use gnome-shell - and if not, there’s more information and a demo video at the above link.

It’s got a long way to go - there are certainly bugs, and some features aren’t done right yet. But it works, and that’s pretty exciting to me after working towards it on and off for about four months.

From Ubuntu to Fedora

I recently moved from Ubuntu to Fedora. It seemed like the right time for it. I just got a new computer, so I’d have to do a fresh install of something anyway. And on the day that my computer arrived, the Fedora 15 beta was released. Perhaps it was meant to be?

Tip: Transient Notifications in notify-send

In gnome-shell (and in contrast to Ubuntu), notifications are persistent by default - they don’t go away until you dismiss them.

I have been using notify-send for some time to pop up quick messages that do not need to stick around, so I’m happy to have found how you can make individual notifications transient. If you want to make a transient notify-send message, you can do the following:

notify-send --hint=int:transient:1 [standard-arguments ...]

I’ve created a notify-send script in my ~/bin directory that sets this by default, although that depends on you having added ~/bin before /usr/bin in your $PATH

New GVim Icon

The official gvim icon (left) is showing its age. The palette seems to be from the days of 256color, there is no anti-aliasing to speak of in many variants, and it’s just generally bland. I have found a few more modern variants, most notably the tango version, but none of them looked terribly great to me.

So I set out to create a more modern rendering of the classic logo. I exchanged the round serifs for a round diamond, added some psuedo-3d shading, gradients and shadows. All in all I’m quite pleased with how it’s turned out: 128px / svg

If you’d like to use this for gvim, you should do something like this1:

mkdir -p ~/.icons
cd ~/.icons
tar zxf gvim-icon-gfxmonk.tgz
rm gvim-icon-gfxmonk.tgz

Then go to your system’s theme selector and pick the “gfxmonk” icon theme.

If all the other (non-gvim) icons have now reverted to the system default and you didn’t want that, you will need to edit ~/.icons/gfxmonk/index.theme and set the inherits value to the name of your preferred icon set2.

Note: the svg icon may need the norasi font installed to display properly.

  1. These instructions have been tested to work on fedora, but I fear other distros may have wildly different mechanisms for overriding icons. 

  2. Forgive me if there is a better way for packaging/installing icons, this is my first attempt. 

Natty UI Fumbles

While my main computer is out of action, I’ve been trying beta 2 of ubuntu 11.04 (Natty Narwhal). The biggest change in this version is the introduction of unity, but there are also a number of other UI changes. Here are some of the bad bits:

  • The new scrollbars break a few interactions that I am rather fond of:
    • Sometimes the grabber is inside the window, sometimes it’s outside.
    • If you scroll up or down by pressing the buttons a few times, and then pause to read, and then try to click the arrow again, it may have disappeared - leading to awfully surprising results.
    • You can’t use the neat trick of middle-clicking anywhere in the scrollbar pane to immediately bring the scrubber there (instead you have to find its current location, then drag it to where you want it to be).
    • If the orange scrubber is taller than the pair of arrows and you want to hit one of the arrows, it remains a moving target until you actually leave the window’s boundary.
    • After scrolling with the mouse wheel, the orange scrubber turns practically invisible, making it impossible to tell how much scrolling remains.
  • The new terminal theme makes it really hard to tell which tab is active.
  • Whenever I press alt (for example, when starting alt-tab or another keyboard shortcut) I get really distracted by the global menu flashing in and out of my peripheral vision.
  • Alt-tab raises windows but does not focus them (bug).
  • Focus-follows-mouse (my preferred setting) is incompatible with the new global menu. You can remove the global menu, but that didn’t work for me. For now, I’m having to get used to pressing F10. There’s a proposal to fix the global menu behaviour when using focus-follows-mouse, but nobody seems interested in implementing it.
  • There’s no way to make “fullscreen” mean “everything but the dock”, as it does by default for other non-auto-hide docks. I do like the combination of titlebar & top-bar though.

Given my progress on a tiling window manager for gnome-shell (more on that when it’s working, I hope), I may well end up switching to gnome-shell soon. I’ll probably try out fedora 15 in the process, I wonder how much of ubuntu I’ll miss.

My Ideal Window Manager

I’ve been using xmonad (with a slightly modified bluetile setup) for about a year now, and it’s been pretty great. But I still feel locked in to its grid sometimes, and miss the direct manipulation that a “normal” window manager (like metacity) provides - specifically allowing quick movement and resizing by using alt + mouse dragging. Bluetile has the option of floating windows, but actually moving or resizing them is so cumbersome that it’s not really worth it. I also sometimes wish that my windows could overlap, so that (while still tiled) a window can extend beyond the bounds of its tile if I want it to.

I also am a sucker for shiny things, and xmonad is far from a shiny thing (in terms of graphics). I tried out gnome-shell yesterday, and while buggy, it is exceedingly shiny. And considering that gnome-shell will not allow alternate window managers (that was a surprise to me), I have put some thought into what my ideal window manager would look like.

I’m keen to try and implement this somewhere. It’s unlikely to be xmonad, as I want builtin compositing support (and haskell is a great language, but I can barely figure out how to configure xmonad, let alone extend it). So I’m wondering if the following can be done as a plugin to either gnome-shell or mutter. Hopefully gnome-shell, as I can stomach javascript hacking a lot easier than compiling C extensions.

Also, if people know of an existing project (with compositing!) that has these sorts of features, I’d be interested to know - I don’t want to have to reinvent the wheel, but it seems like most tiling window managers are too rigid and keyboard-based for me, while most “grid” extensions to floating window managers are too manual.

So, here’s the plan:

Why Zero-Install Will Succeed

Also known by the longer title: I Sure Hope Zero-Install Succeeds, Or Else We Might All Give Up On Package Managers Entirely.

If you’ve tried to run any of my software lately, you may have noticed that it’s all distributed and packaged via Zero-Install. I’ve posted about how awesome it is, but that was just an initial impression - I’m now familiar enough with the system to expand on those impressions.

After reading that the distros have killed python yesterday, I felt compelled to write in a little more detail how zero-install solves this and many other problems right now, across platforms and languages, and for much less effort than the current packaging practices.

Bluetile: A friendly tiling window manager

I’ve recently started using bluetile instead of compiz, and I think I’m going to stick with it. Bluetile is a tiling window manager, but it’s not like all those other tiling window managers*:

  • it’s simple
  • it’s easy to learn
  • it’s built with GNOME support out of the box
  • it still supports direct manipulation style using the mouse if you just want to change a window’s size, or swap window locations. I think this is crucial for keeping it accessible to new people like myself.

* well okay, it is like those other tiling window managers. Specifically, it’s built on xmonad. But it presents a simple veneer over the vast complication that is xmonad (I should know, I tried to configure it just last week ;)).

Bluetile is, sadly, notoriously difficult to install. But there are good guides around, I used this one for ubuntu karmic.

The good news is, bluetile has recently been merged back into the xmonad mainline. And xmonad is apt-gettable (at least on ubuntu). So in theory, the next release of xmonad will make it pretty easy to run in bluetile mode, without having to compile-your-own-window-manager (which certainly does not make for a gentle learning curve). If anyone is considering playing around with alternate window managers, it’s definitely worth your while to give bluetile a go.

Etoile first impression

I’ve long been interested in étoilé. It’s a linux distribution built on top of GNUStep, aiming to provide a more modern (read: prettier and generally a bit more Apple-Like) operating system than the GNUStep base. Obviously as a mac user who has switched to linux for its openness, this is a pretty tempting idea.

Anyways, I recently downloaded the VirtualBox image to take it for a test run. Aside from there not really being much software to use yet (making it hard to take for a spin), I noticed this in the system menu. It’s probably somewhat more important to get your character encoding right if you’re going to be all fancy and have accents in your name ;)

etoile system menu

Zenity: for the user-friendly scripter

I just discovered zenity, which is a great tool for simple gtk+ interactions with a script.

It has the following types of interactions:

  • calendar (date picker)
  • file / directory selection
  • error / info message
  • list picker
  • question (yes/no)
  • progress dialog
  • systray notification
  • and more… ( see the man page )

The usage is brilliantly simple, and it’s very unixy despite being a GUI tool. I just wrote a pygtk+ app to do some photo importing stuff, but doing it with zenity would have been far simpler.

Well worth keeping in mind for your next user-friendly shell scripts (particularly for the more advanced progress, calendar and list picker dialogs).

Is there such thing as a Snapping Window Manager? which I propose a potentially-new window management feature, and hope that somebody has already done it so that I won’t have to…

A thought or two about application launch window focus models

Note that here I’m talking about windows getting focus when they launch, rather than focus-follows-mouse or window click-through (I’ll leave that one to Gruber).

Recently, I’ve learnt something about how window focus works on OSX, and subsequently been fairly disappointed by how it works in X (and probably Windows, but correct me if I’m wrong).

OSX-style horizontal mouse scrolling for linux

OSX has this great feature where if you hold down SHIFT at the same time as using your mouse scroll wheel, it’ll scroll horizontally instead of vertically. If you don’t have a laptop, this is an immensely useful trick. Sadly, I couldn’t find any way to get this to happen on linux.

But now, thanks to some direction from stackoverflow, I finally figured out how to do it myself. The world of X11 input hackery is somewhat twisted and full of projects either abandoned or in disrepair, but I finally stumbled across the right set of tools.

If you’d like to get this (rather excellent) feature in linux, you will need the following:

  • Install the packages xbindkeys and xautomation: sudo apt-get install xbindkeys xautomation

  • Save the following file as ~/.xbindkeysrc.scm :

      ; bind shift + vertical scroll to horizontal scroll events
      (xbindkey '(shift "b:4") "xte 'mouseclick 6'")
      (xbindkey '(shift "b:5") "xte 'mouseclick 7'")
  • Use your favourite mechanism to ensure that the xbindkeys command is run at the beginning of your xsession (I added it to ubuntu’s “startup items” preference, but you can surely use init.d if you’re comfortable with that).

Remap shift+space to underscore

So I had this great idea yesterday (for coders, at least): remap [shift+space] to [underscore]. Turns out I am far from the first to think of this, which only enforces its awesomeness as an idea.

Mac: Put this in ~/Library/KeyBindings/DefaultKeyBinding.dict:

	/* turn shift + space into underscore */
	"$ " = ("insertText:", "_");

Linux: for a PC keyboard, try:

xmodmap -e 'keycode 65 = space underscore'

or on a mac keyboard:

xmodmap -e 'keycode 57 = space underscore'

Or if neither of those work, run:

xmodmap -pk | grep space | awk '{print $1}'

and use that number instead of 65 or 57 above.

You can put this keybinding in ~/.xmodmaprc or somesuch if you like it.

A better snap-open (for gedit)

snap-open is handy. Unfortunately, it’s terribly crippled if you have a lot of files, since:

  • it uses the linux find command, instead of an index-based file list
  • it doesn’t run in a separate thread, so it locks up your entire GUI while it goes off to trawl your file heirarchy every time you change a letter in the find box

So I fixed it a bit. It is now threaded, and uses the linux locate command. This is still not ideal, it would be much better to use beagle. But I couldn’t figure out how to do that real quick, so locate it is (for now).

Update: As MadsBuus Points out in the comments, SnapOpen has improved a lot since then. Also, I’ve taken a similar plugin (gedit-openfiles) and turned it into a console-based file finder that allows you to open files with your preferred editor. Since I no longer use gedit, I think having a file finder independent of your editor is a pretty neat feature.