Today we (Oni Labs) released StratifiedJS 0.14. This is the first release since I started working here full-time, and it’s a big one: loads of useful new syntax, as well as a thoroughly kitted-out standard library.

StratifiedJS is a Javascript-like language that compiles to Javascript, but which supports advanced syntax and semantics, like:

• blocking-style code for asynchronous operations (no callbacks!)
• try/catch error handling works even for asynchronous code
• a structured way of managing concurrent code (waitfor/or, waitfor/and, waitforAll, waitforWirst, etc).
• ruby-style blocks
• lambda expressions (arrow functions)
• quasi-quote expressions

Check it out at onilabs.com/stratifiedjs.

NodeJS’ require() method is special. npm is special. Some of that is good - its efforts to dissuade people from installing anything globally are commendable, for a start. But some of it is bad. It’s probably better to be aware of the bad parts than to learn them when they bite you.

Let’s run through a quick example of what happens when I install a package. For example, installing the bower package will:

• install bower’s code under node_modules/bower
• under node_modules/bower, install each of bower’s direct dependencies.

Of course, this is recursive - for each of bower’s direct dependencies, it also installs all of its dependencies. But it does so individually, so you end up with paths like (this is a real example):

node_modules/
  bower/
    node_modules/
      update-notifier/
        node_modules/
          configstore/
            node_modules/
              yamljs/
                node_modules/
                  argparse/
                    node_modules/
                      underscore

Unlike pretty much every package manager I’ve encountered, npm makes no attempt to get just one copy of a given library. After installing bower, NPM has unpacked the graceful-fs package into 4 different locations under bower. I’ve also installed the karma test runner recently, which indirectly carries with it another 10 copies of graceful-fs. My filesystem must be exceedingly graceful by now.

I’ve just released version 1.0 of 0env. Its purpose is similar to the “interactive” mode of operation of tools like rvm, virtualenv, etc. That is, “entering” some environment with an interactive shell. But instead of being part of some language-specific development tool, it works with any ZeroInstall feeds (which are language-agnostic and cross-platform). The readme on the linked page pretty much explains it all, but I’ll summarize the important features here:

• You can try out one (or more) ZeroIntall feeds interactively
• All work happens in a subshell, with a modified shell prompt to clarify what context you’re in
• There is nothing to roll back, modify or undo - it’s completely stateless
• It works for published feeds (URLs) as well as unpublished or development local feeds
• It works cross-platform

I really feel this is an important tool for helping people adopt and use ZeroInstall feeds. ZeroInstall is a great way to publish software, but until now it has been awkward to try one or more feeds out interactively, partly because there is nothing to install.

(view link)

I seem to keep switching between Ubuntu and Fedora every 6 months, for various reasons. I think switching is good for you in that it allows you to evaluate the benefits of each distro, and also encourages you “travel light” as it were, discouraging one-off hacks that you’ll have to figure out each and every time you change distro (or potentially upgrade). That’s not to say I don’t do a heck-tonne of customisation, but I do it in a structured, repeatable and (when possible) cross-distro manner.

To be honest, the main thing that irks me is having to remember to mentally switch between apt-get/dpkg and yum/rpm. I try to declare package dependencies for tools I commonly use or libraries I depend on in a cross-platform way using zero-install feeds (this really helps when switching, since I don’t have to remember the differing names of many packages), but inevitably I still need to use the command-line package managers fairly often.

Today I ran across fpm (“Effing Package Management”) again, which is a tool that a (likely frustrated) user has built to abstract away the uninteresting details of building packages for such-and-such package manager. It turns out that system packages are boring, and do pretty much the same thing in 99% percent of cases. The other 1% is probably a bad idea, or obscure enough of a feature that you’ll do it wrong¹. I took a closer look to see if it also dealt with the uninteresting inconsistencies of using a package manager to search for an install packages, but no such luck.

Enter pkcon

Fortunately, a pretty decent tool is already available, and already installed in many modern distros. It’s called pkcon, and it’s part of the PackageKit project to provide a consistent API for the common operations across the plethora of linux package managers. PackageKit has a whole heap of supported backends, most likely more than you’ve even heard of. It covers the basic operations that are generally present in all package managers, it does not attempt to expose advanced features. Fedora’s default package GUI is built on it, and I believe ubuntu’s software center either is already or soon will be built on it it too. It’s also the mechanism by which zero-install can install required system packages as needed on most distros, which earns it a special place in my heart ;)

Using it

Anyway, what can you do with pkcon? Close to everything I’ve ever needed to. Technically speaking it’s largely a testing tool for exercising the PackageKit API, but (by no coincidence) the PackageKit API happens to be a straightforward set of the main things a user actually cares about. For the most part, the commands are exactly what you’d expect if you’ve used apt or yum, and in other cases they make more sense. Here’s a list of the stuff I use, with apt and yum equivalents for context:

install (apt-get install, yum install)

$ pkcon install [packagename]

remove (apt-get remove, yum remove)

$ pkcon remove [packagename]

check for updates (apt-get update, yum check-update)

$ pkcon refresh

upgrade all packages (apt-get upgrade, yum update)

$ pkcon update

search (apt-cache search, yum search)

$ pkcon search [name|details|group|file] [query]

file is pretty useful when you want to find “who owns that file on my hard drive”, but unfortunately you can’t use wildcards to find files you’re missing but don’t know the full location of (at least it’s not working for me). Still, good for satisfying poorly-documented build dependencies that are expecting certain files to exist.

what-provides (rpm -qf, dpkg -S)

$ pkcon what-provides /usr/bin/python

Seems to be the same as pkcon search file above. Much easier to remember than the rpm/dpkg flags.

get-details (apt-cache show, yum info)

$ pkcon get-details [packagename]

I mainly use this for getting the version of a package installed, or to check that a given package is what I want to install.

get-files (dpkg -L, rpm -ql)

$ pkcon get-files python

This one even works on packages you haven’t installed, unlike both dpkg and rpm.

User Beware

Unfortunately, pkcon is not without its rough edges. While the package management actions are well-tested, the command line itself is a bit flaky. For example, both --help-all and refresh --force give parse errors on Fedora 17, despite being recommended in the help text. More importantly the --filter=installed flag does nothing on my system. Without that filter, you will need to disambiguate (by picking from a numbered list) whenever you perform an action on a package that is installed but also has an available update, or which is available in both 32 and 64 bit versions.

It also requires DBus as a dependency (since PackageKit is implemented as a DBus API), which may make it a poor option for administering servers. But you probably shouldn’t be doing most of this manually on a server anyway.

Here’s a few 3D photos from a recent trip beachwards, run through slowmoVideo for smooth transitions:

(stereo version)

(stereo version)

(stereo version)

For a while my preferred “bash script prelude” (the stuff you write in every bash script, before the actual content) has been a modest:

#!/bin/bash
set -eux

Those settings break down to:

• -e: fail when any subcommand fails (the number of scripts that proceed to do completely incorrect things after an earlier command failed is criminal). You can always silence errors by using || or an if statement, but (much like exceptions in a high-level language) silence should not be the default.
• -u: fail when an unknown variable is referenced - mostly guards against silly typos, but why wouldn’t you? It’s sometimes not what you want with variables that may be optionally set by the user, so sometimes I’ll move the set -u down in the script after I’ve dealt with setting those to their defaults if they are not present.

• -x: trace all commands, after expansion. E.g with a line like: x=123; echo $x, Your output will be:

  • x=123
  • echo 123 123

This gets very noisy, but frequently I find it’s better to have more output than less - when a script breaks, often you can tell why just by looking at the last command trace (especially when combined with the -e option). And it’s easier to ignore excess output than it is to guess what went wrong.

The new hotness

Today, I discovered / created a much longer (and uglier) prelude, but I think it’s going to be worth it. Here ‘tis:

#!/bin/bash
set -eu
set -o pipefail
export PS4='+ ${FUNCNAME[0]:+${FUNCNAME[0]}():}line ${LINENO}: '
syslogname="$(basename "$0")[$$]"
exec 3<> >(logger -t "$syslogname")
BASH_XTRACEFD=3
echo "Tracing to syslog as $syslogname"
unset syslogname
debug() { echo "$@" >&3; }
set -x

A mouthful, indeed. Lets go though it line-by-line: