GFX::Monk Home

Posts tagged: application

Doing stuff when files change

There’s a common pattern in development tools to help with rapid feedback: you run a long-lived process that does a certain task. At the same time, it watches the filesystem and restarts or re-runs that task whenever a file that you’re monitoring changes.

This is an extremely useful tool for rapid feedback (which is why we’ve integrated nodemon into our Conductance app server), but is not very flexible - most tools are integrated into a web framework or other environment, and can’t easily be used outside of it. There are a few generic tools to do this kind of thing - I personally use watchdog a lot, but it’s sucky in various ways:

  • Configuring it to watch the right file types is hard
  • Configuring it to ignore “junk” files is hard, leading to infinite feedback loops if you get it wrong
  • It sees about 6 events from a single “save file” action, and then insists on running my build script 6 times in a row
  • It takes a bash string, rather than a list of arguments - so you have to deal with double-escaping all your special characters

And yet for all of those issues, I haven’t found a better tool that does what I need.

My build workflow

Lately, I’ve been making heavy use of my relatively-new build system, gup. It’s a bit like make, but way better in my humble-and-totally-biased-opinion. But going to a terminal window and typing up, enter (or the wrist-saving alternative crtl-p, ctrl-m) to rebuild things is tedious. But there’s no way I’m going to implement yet another watch-the-filesystem-and-then-re-run-something gup-specific tool, at least not until the lazy alternatives have been exhausted.

Obviously, my workflow isn’t just up, enter. It’s (frequently):

  • save file in vim
  • go to terminal
  • press up, enter
  • go to browser
  • refresh

And you know what? Monitoring every file is kind of dumb for this workflow. I don’t have gremlins running around changing files in my working tree at random (I hope), I almost always want to reload in response to me changing a file (with vim, of course). So why not just cooperate?

The simple fix

So I’ve written a stupid-dumb vim plugin, and a stupid-dumb python script. The vim plugin touches a file in $XDG_USER_DIR whenever vim saves a file. And then the script monitors just this file, and does whatever you told it to do each time the file is modified. The script automatically communicates with vim to enable / disable the plugin as needed, so it has no overhead when you’re not using it.

It’s called vim-watch, and I just gave you the link.

Addendum: restarting servers

While writing this post, I was a little disappointed that it still doesn’t quite replace tools that automatically restart a server when something changes, because it expects to run a build-style command that exits when it’s done - but servers run forever. Some unix daemons (like apache) restart themselves when you send them a HUP signal, but that’s not so common in app servers. So now huppy exists, too.

It’s a tiny utility that’ll run whatever long-lived process you tell it to, and when it receives a HUP signal it’ll kill that process (with SIGINT) if it’s still running, then restart it. It seems surprising that this didn’t exist before (maybe my google-fu is failing me) but on the other hand it’s less than 60 lines of code - hardly an expensive wheel to reinvent.

You can use it like:

$ # step 1: start your server
$ huppy run-my-server

$ # step 2: use vim-watch to reload the server on changes
$ vim-watch killall -HUP huppy

$ # Or, if you need to rebuild stuff before restarting the server,
$ vim-watch bash -c 'gup && killall -HUP huppy'

Oni Conductance

This past week, we (Oni Labs) announced Conductance, the next-generation web app server built on the StratifiedJS language (which we also built, and which has seen a number of steadily improving public releases over the past couple of years).

For a long time, I’ve been convinced that plan JavaScript is simply inappropriate for building large scale, reliable applications. That doesn’t mean it’s impossible, but the effort required to correctly write a heavily-asynchronous application in javascript involves a frankly exhausting amount of careful error checking and orchestration, and there are whole classes of confusing bugs you can get into with callback-based code which should not be possible in a well-structured language.

So I was extremely happy to join the Oni Labs team to help work on StratifiedJS, because it’s a much more extensive (and impressive) attempt to solve the same problems with asynchronous JavaScript that I was already trying to solve.

Conductance is a logical progression of this work: now that we have StratifiedJS, we’ve used its features to build a new kind of app server: one which maintains all the performance benefits of asynchronous JavaScript (it’s built on nodejs, after all), but which makes full use of the structured concurrency provided by StratifiedJS for both server and client-side code. And not just for nice, modular code with straightforward error handling - but actually new functionality, which would be impossible or extremely ungainly to achieve with normal JavaScript.

If you’re interested in building web apps (whether you already do, or would like to start), please do check out conductance.io for more details, and plenty of resources to help you get started building Conductance applications.

Announcing the gup build tool

gup is a software build tool. It is designed to be general purpose, and does not care:

  • what kind of project you are building
  • what language you are building
  • what language you write your build scripts in

It has (almost) no syntax, instead it defines a simple protocol for where build scripts are located. Instead of declaring dependencies up-front, build scripts declare dependencies as they use them. This allows your dependencies to be enumerated at runtime, while building, rather than existing in some separate, statically-declared list which has to be manually updated if you wish your build to Not Be Wrong.

It’s similar to djb’s redo, which has been implemented by Avery Pennarun. In fact, I even took a bunch of code from redo. If you’ve used it before, gup will seem pretty familiar.

Please check out the project on github for more details and documentation. It’s still young, and it may change. But I’ve been using it for both work and person projects for a few months now, and it’s already proven much more solid than redo in my own usage.

Why didn’t I just help make redo better?

I tried, but I believe redo's design is impossible (or at least very difficult) to implement in a way that does not Do The Wrong Thing silently (and somewhat often). That is absolutely not a property I want from my build system.

The core problem springs from the fact that redo relies on local file state to determine whether a file is actually a target. The only difference between a build target and a source file is that a target is one which didn’t exist when you first tried to build it - i.e if something looks like a target but it already exists, then it is actually a source, and will never be built.

There is quite a bit of state locked up in the above definition, and it turns out that it’s perilously difficult to manage that state correctly. The end result in many cases is that redo thinks a built file is actually a source file, and it silently ignores all requests to build it1. Remedying this situation is manual - it cannot easily be scripted, and the actions required depend entirely on the state of the local workspace.

gup fixes this problem by requiring you to be more explicit about your targets. In gup, something is a target if (and only if) you’ve told gup how to build it. It also means that the set of targets is defined by the part of your project that’s tracked by source control, rather than the state of your local filesystem.

  1. When updating from Fedora 19 -> 20 recently, this happened to every single file redo had ever built. This may not be redo’s fault, but it shows how fragile the mechanism is.

Shellshape arrives on extensions.gnome.org

shellshape logo

It’s been a long time coming, but shellshape (my tiling window manager extension for gnome-shell) is finally available via extensions.gnome.org. Get it while it’s hot!

You’ll need the bleeding edge (3.4.1) version of gnome-shell, as it’s the first version that allows normal extensions to register new keybindings. If you’re stuck on 3.4 you can still use the 0launch method described on the shellshape homepage.

(view link)

Why Piep

piep (pronounced “pipe”) is a new command line tool for processing text streams with a slightly modified python syntax, inspired by the main characters of a typical unix shell (grep, sed, cut, tr, etc). To pluck a random example. here’s how you might rename all files (but not directories) in the current folder to have a “.bak” extension (because you have a very strange and manual backup scheme, apparently):

$ ls -1 | piep 'not os.path.isdir(p) | sh("mv", p, p + ".bak")'

In this simple example we can see filtering, piping (note that the pipes between expressions are internal to piep’s single argument, and thus not interpreted by the shell), and shelling out to perform useful work.

Here’s another, to print out the size of files in the current directory that are greater than 1024 bytes:

$ ls -l | piep 'pp[1:] | p.splitre(" +", 7) | size=int(p[4]) | size > 1024 | p[7], "is", p[4], "bytes"'

Or, if hacking through the output of ls -l isn’t your thing (it’s most likely a terrible idea), you can do things the pythonic way:

$ ls -1 | piep --import 'stat' 'size=os.stat(p).st_size | size > 1024 | p, "is", size, "bytes"'

For a proper introduction, you should read the online documentation. But I wanted to address one specific point here, about the origins of piep.


Recently I came across pyp, The Pied Piper. It seemed like a great idea, but after I played with it for a little while I uncovered some unfortunate shortcomings, some of which are deal breakers. My list included:

  • stream-based operation: there’s a beta implementation with “turbo” (line-wise) mode, but it seems very limited. I believe it should be the norm, and wanted to see if I could do things in a way that was just as convenient, but with all the benefits of lazy stream-based processing.
  • Command execution: commands are made up by string concatenation, requiring manual effort to escape metacharacters including the humble space 1. Also, errors are silently ignored.
  • Purity of data: things are routinely strip()ed and empty strings are frequently dropped from computations. Breaking up a line into a list of data would (sometimes?) see each list merged back into the input stream, rather than maintained as a list.
  • stream confusion: second stream, file inputs, etc. Not really sure why there are so many special cases
  • a not-very-extensible extension mechanism, which is fairly manual and appears to preclude sharing or combining extensions
  • lots of unnecessary machinery that complicates the code: macros, history, –rerun, three file input types, etc. Some of this may be useful once you use the tool a lot, but it hindered my ability to add the features I wanted to pyp.

I initially tried my hand at modifying pyp to fix some of the things I didn’t like about it, but the last point there really got in my way. History is baked in, and doesn’t really work in the same manner for stream-based operations. The entire pp class had to be rewritten, which is actually what I started doing when I decided to turn it into a separate tool (since it then became difficult to integrate this new pp class with the rest of the system. Anyway, I hope this isn’t taken as an offence by the developers of pyp - I really like the ideas, so much so that I was compelled to write my ideal version of them.

  1. I humbly submit that concatenating strings is the worst possible way to generate shell commands, leading to countless dumb bugs that only rear their heads in certain situations (and often in cascading failures). Observe piep’s method on a filename containing spaces:

    $ ls -1 | piep 'sh("wc", "-c", p)'
    82685610 Getting the Most Out of Python Imports.mp4
    

    Compared to that of pyp (and countless other tools):

    $ ls -1 | pyp 'shell("wc -c " + p)'
    wc: Getting: No such file or directory
    wc: the: No such file or directory
    wc: Most: No such file or directory
    wc: Out: No such file or directory
    wc: of: No such file or directory
    wc: Python: No such file or directory
    wc: Imports.mp4: No such file or directory
    [[0]0 total]
    $ echo $?
    0
    

    It is unacceptable for a language with simple and convenient sequence types to instead rely on complex string escaping rules to prevent data from being misinterpreted. To be honest, this on its own may be reason enough to use piep over alternatives.

Stereoscoper and the Depth of Awesomeness

A few days ago I got a shiny new toy: a 3d camera from thinkgeek (I’d link to it, but it seems to have disappeared from their catalogue). I’m a massive fan of 3d photos / video, so it’s pretty cool to have a device that allows me to take stereoscopic photo pairs simultaneously (you can do it manually with a static scene, but those get boring).

Sadly (although not surprisingly), the quality is not great. The limited resolution is not really an issue given how you’re likely to view them, but the pictures come out awkwardly stretched to half the expected horizontal resolution. They are also pre-combined in a single JPEG, they are the way around for cross-eyed viewing, and the colour balance is frequently off between the two sensors (which can be really jarring).

Seeing a lot of manual photo fixing in my future, I set out to automate it. And thus stereoscoper was born, as a way to bulk-convert stereo images to other formats. Aside from the obvious geometry changes (the horizontal resolution and image placement), I also learnt all about histogram matching in order to make the colour balance consistent across stereo pairs. And in order to make animated gifs that match up nicely, there’s even an interactive mode where you can fine-tune the alignment of the image pairs.

In the wiggly-animated spirit of 3ERD (note: some images there are NSFW), here’s some fun we had in the park with my new toy:

(click to toggle each animation. It’s off by default to save your brain from having a fit ;)

(stereo)

(stereo)

(stereo)

(stereo)

Update: Click the (stereo) link under each image for a cross-eyed viewing version.

Shellshape: A Tiling Window Manager for Gnome Shell

shellshape Today I released the first version of shellshape, a tiling window manager plugin for gnome-shell. It’s definitely pre-alpha software, and currently requires a custom fork of the mutter window manager. I’ve had some trouble getting it running due to awful packaging things (I now know far too much about dynamic linking path resolution on linux), but it should work, at least on Fedora 15. Please give it a go if you use gnome-shell - and if not, there’s more information and a demo video at the above link.

It’s got a long way to go - there are certainly bugs, and some features aren’t done right yet. But it works, and that’s pretty exciting to me after working towards it on and off for about four months.

daglink: organise your system configuration

I just released daglink, a tiny program for managing your system configuration. The readme has most of the information, but as a quick overview daglink allows you to maintain symbolic links from well-known locations (e.g /etc/apt/sources.list.d/) into locations of your choosing (mine live in ~/dev/app-customisations/).

But more than just creating symlinks, it allows you to create the appropriate links based on tags of your choosing, for example distro, release and keyboard type (yes, I do actually have multiple of each of those that I frequently use).

I’ve seen others use git or puppet for this, and until now I just had a hand-rolled script and some well-named files. So I figured I might as well write something proper. Did I mention it has built-in support for zero install URLs instead of paths, if you’re into that sort of thing? I’m not sure how useful that is (or should be), but it could turn out useful for some very niche cases.

mocktest 0.5

Over the holidays, I’ve finally had the time to rewrite my mocking library for python, mocktest.

The original version had what turned out to be a confusing distinction between mock anchdors, mock wrappers and raw mocks. You should no longer need to know about that distinction when using mocktest 0.5, as it takes a more traditional approach using global methods like when() and expect() to differentiate between setting up the mock object and actually using it.

Please check out the brand-new documentation if you’re looking into a mocking library for python - it works with the standard unittest infrastructure, so it’ll work just fine with your favourite test runner (nosetests, surely….)

Speaking of documentation, this is my first time using sphinx. I am very impressed, and really quite keen to properly document a lot of my other code, when I get the time.

Pagefeed Android App

After getting an android phone about a month ago, I finally got around to writing a workable pagefeed sync app. This app can handle share intents from any app (I use NewsRob mostly), and save that link to pagefeed on next sync.

It also downloads the links (not the content, this is a lightweight app) so you can delete links from the app and have them deleted from the webapp as well.

You can download the apk here (I haven’t signed up for the market yet). The source is also up on github.

Android as a platform

Overall, I found android frustrating to learn when dealing with some of the newer and less-documented parts (like SyncAdapters and how they interact with ContentProviders, for example), but overall I’m finding it to be a very well-designed system that is easy to write for.

For comparison (because I’ve written an iPhone app or two), this app could not exist on the iPhone because interaction with other apps (via intents) is crucial, as is background operation (which has only just arrived in OS4). It would also be a royal pain in the ass to obtain, store and validate user credentials - all of which android makes general and reusable (I am happily reusing the google-provided authenticators).

Oh yeah, and it’s written in scala! I seem to have an aversion to platform-specified languages, having used python for my biggest iPhone app. Of course, that meant that I had to void my warranty and only run on jailbroken iPhones. Android doesn’t care how you write your app, and that’s the way it should be.

Save My Text

savemytext is a tiny app engine webapp I wrote last night.

When you write text in it, it will save your text. When you come back, your text will still be there. You can have multiple text boxes. Since it’s an appengine app, you’ll need a google account to use it.

A trivial app, to be sure. But all too often I find myself writing gmail drafts just in order to get a persistent text box between computers. I figured, why not do it properly?

(I was going to just call it “textarea”, but someone already stole that name (and tens of similar ones, without actually having an app to put up. How rude))

Introducing: PageFeed

pagefeed logo PageFeed is a simple web service to help you organise the pages you’d like to read, but just don’t have the time (or desire) to read through right now.

If you’ve ever used Instapaper, or the newer Read It Later, Pagefeed will feel pretty familiar. The idea is that when you come across an interesting page that you don’t have time to read right now, you just save it to PageFeed via a handy bookmarklet. You can close the page, and PageFeed will remember all the pages you’ve saved for reading later.

Instapaper and Read It Later both have their own iPhone apps for reading stuff offline. Which is great, but it seems a little unnecessary - and it makes for yet another app you have to remember to open and sync every day. And you know what? There’s already a super robust way to subscribe to a stream of HTML items - it’s called RSS. PageFeed puts the HTML contents of your saved pages into your own private RSS feed. You can then use your favourite online / offline feed reader - you can take the content wherever you like and use it on whatever platform you fancy. You can also use the PageFeed home page for managing your list of saved pages away from your RSS reader.

PageFeed isn’t perfect; it can’t put HTML contents in a feed if it can’t parse the HTML page properly. And if there’s one thing the internet is good at, it’s malformed HTML. But it does pretty well, and if it can’t save the page contents then it will still keep the URL around for you to visit later.

PageFeed runs on AppEngine and uses google accounts, so chances are you already have an account.

The newest version (0.8) of GRiS (the RSS reader I wrote for the iPhone) has support for PageFeed - when you click a link in an article’s contents, you can opt for it to be saved to PageFeed instead of opened in MobileSafari. If you’d like to offer similar functionality in your RSS (or other) application, get in touch!

Autonose: continuous test runner for python's nosetests

Today I’ve put up the first “releaseable” version of autonose. Basically, it analyses your code’s imports, and determines exactly which tests rely on which code. So whenever you change a file, it’ll automatically run the tests that depend on the changed file (be it directly or transitively). Give it a go, and please let me know your feedback.

All you need do is:

$ easy_install autonose
$ cd [project_with_some_tests_in_it]
$ autonose

See the github project page for further information (and a screenshot).

A better snap-open (for gedit)

snap-open is handy. Unfortunately, it’s terribly crippled if you have a lot of files, since:

  • it uses the linux find command, instead of an index-based file list
  • it doesn’t run in a separate thread, so it locks up your entire GUI while it goes off to trawl your file heirarchy every time you change a letter in the find box

So I fixed it a bit. It is now threaded, and uses the linux locate command. This is still not ideal, it would be much better to use beagle. But I couldn’t figure out how to do that real quick, so locate it is (for now).

Update: As MadsBuus Points out in the comments, SnapOpen has improved a lot since then. Also, I’ve taken a similar plugin (gedit-openfiles) and turned it into a console-based file finder that allows you to open files with your preferred editor. Since I no longer use gedit, I think having a file finder independent of your editor is a pretty neat feature.

google-reader-iphone-sync

A small, cobbled-together tool which allows you to read your Google Reader feed items offline on your iPhone or iPod Touch.

[i made this!]

(view link)

HTML to PDF converter

Given the integration of PDF into Mac OS X, I was surprised to find that there didn’t seem to be any tool to convert HTML files into PDFs. So, like any frustrated coder I wrote my own little script to do it: html2pdf.py

Requires python and OSX 10.5 (Leopard). It uses a WebKit view for the rendering, so in theory it should work with any URL that safari can handle - but I haven’t exactly tested it thoroughly…

Trashy.app

In any OSX “document window”, there is a little icon representing the current document. If you click and hold this icon, it becomes a draggable alias for the file. You can then use that alias much as you would the file itself (as if you had dragged it from the finder) - but one thing you can’t do is delete the file by dropping it in the trash.

Trashy is a simple program to fix that. Put it in your dock, and it will send anything you drag onto it into the trash. Click here to download Trashy.

Here’s the entire source code applescript, if you’re curious (or just naturally suspicious of running random programs you found on internet, as you ought to be):

on open fileList
	tell application "Finder"
		repeat with f in fileList
			set n to 0
			repeat while class of f is alias file
				if n is less than 5 then
					set n to n + 1
					set f to original item of f
				end if
			end repeat
			move f to the trash
		end repeat
	end tell
end open

on run
	tell application "Finder" to open the trash
end run

MetaMonkey.app

[I made this]

A Lightweight Metadata manager for OSX, allowing you to easily tag and rate any type of file (which is then indexed by spotlight). This is basically my iPhoto replacement now, after spending a week trying to fix its broken database, orphan files, duplicates and all that rubbish. This is much simpler, and it can be used to tag any type of file I want.

This is the result of 2 days work, and just 400 lines of code. That’s pretty decent in my book, the combination of python and cocoa is a pretty powerful beast. Once you get past all the runtime errors and pyobjc bridge troubles, that is…

(view link)