gup is a software build tool. It is designed to be general purpose, and does not care:

• what kind of project you are building
• what language you are building
• what language you write your build scripts in

It has (almost) no syntax, instead it defines a simple protocol for where build scripts are located. Instead of declaring dependencies up-front, build scripts declare dependencies as they use them. This allows your dependencies to be enumerated at runtime, while building, rather than existing in some separate, statically-declared list which has to be manually updated if you wish your build to Not Be Wrong.

It’s similar to djb’s redo, which has been implemented by Avery Pennarun. In fact, I even took a bunch of code from redo. If you’ve used it before, gup will seem pretty familiar.

Please check out the project on github for more details and documentation. It’s still young, and it may change. But I’ve been using it for both work and person projects for a few months now, and it’s already proven much more solid than redo in my own usage.

Why didn’t I just help make redo better?

I tried, but I believe redo's design is impossible (or at least very difficult) to implement in a way that does not Do The Wrong Thing silently (and somewhat often). That is absolutely not a property I want from my build system.

The core problem springs from the fact that redo relies on local file state to determine whether a file is actually a target. The only difference between a build target and a source file is that a target is one which didn’t exist when you first tried to build it - i.e if something looks like a target but it already exists, then it is actually a source, and will never be built.

There is quite a bit of state locked up in the above definition, and it turns out that it’s perilously difficult to manage that state correctly. The end result in many cases is that redo thinks a built file is actually a source file, and it silently ignores all requests to build it¹. Remedying this situation is manual - it cannot easily be scripted, and the actions required depend entirely on the state of the local workspace.

gup fixes this problem by requiring you to be more explicit about your targets. In gup, something is a target if (and only if) you’ve told gup how to build it. It also means that the set of targets is defined by the part of your project that’s tracked by source control, rather than the state of your local filesystem.

I tend to avoid bash wherever possible for scripting, because it has dangerously bad defaults and will happily munge your data unless you take great care to wrap it up properly (particularly whitespace). But sometimes you have no choice, so you might as well know how to do it safely.

Here’s how to capture argv as a bash array, and pass it on to another command without breaking if some argument contains a space:

args=("$@")
echo "${args[@]}"

You can also just pass “$@” directly, but the above syntax works for any array.

Don’t forget any of those quotes, or bash will silently ruin everything (until you have data with spaces, at which point it might loudy ruin everything).

Here’s how to convert a line-delimited string (e.g a list of files in the current directory) into an array and pass that on:

mapfile -t arr <<<"$(ls -1)"
echo "${arr[@]}"

Note that a sensible-looking:

ls -1 | mapfile -t args

will not work, as a builtin on the receiving end of a pipe gets run in a subshell.

If you don’t have mapfile (added in bash v4), you’ll have to resort to:

oldIFS="$IFS"; IFS=$'\n' read -d '' -r -a arr <<< "$(ls -1)"; IFS="$oldIFS"; unset oldIFS
echo "${arr[@]}";

I look forward to the day when I don’t have to know that.

Today we (Oni Labs) released StratifiedJS 0.14. This is the first release since I started working here full-time, and it’s a big one: loads of useful new syntax, as well as a thoroughly kitted-out standard library.

StratifiedJS is a Javascript-like language that compiles to Javascript, but which supports advanced syntax and semantics, like:

• blocking-style code for asynchronous operations (no callbacks!)
• try/catch error handling works even for asynchronous code
• a structured way of managing concurrent code (waitfor/or, waitfor/and, waitforAll, waitforWirst, etc).
• ruby-style blocks
• lambda expressions (arrow functions)
• quasi-quote expressions

Check it out at onilabs.com/stratifiedjs.

NodeJS’ require() method is special. npm is special. Some of that is good - its efforts to dissuade people from installing anything globally are commendable, for a start. But some of it is bad. It’s probably better to be aware of the bad parts than to learn them when they bite you.

Let’s run through a quick example of what happens when I install a package. For example, installing the bower package will:

• install bower’s code under node_modules/bower
• under node_modules/bower, install each of bower’s direct dependencies.

Of course, this is recursive - for each of bower’s direct dependencies, it also installs all of its dependencies. But it does so individually, so you end up with paths like (this is a real example):

node_modules/
  bower/
    node_modules/
      update-notifier/
        node_modules/
          configstore/
            node_modules/
              yamljs/
                node_modules/
                  argparse/
                    node_modules/
                      underscore

Unlike pretty much every package manager I’ve encountered, npm makes no attempt to get just one copy of a given library. After installing bower, NPM has unpacked the graceful-fs package into 4 different locations under bower. I’ve also installed the karma test runner recently, which indirectly carries with it another 10 copies of graceful-fs. My filesystem must be exceedingly graceful by now.

I’ve just released version 1.0 of 0env. Its purpose is similar to the “interactive” mode of operation of tools like rvm, virtualenv, etc. That is, “entering” some environment with an interactive shell. But instead of being part of some language-specific development tool, it works with any ZeroInstall feeds (which are language-agnostic and cross-platform). The readme on the linked page pretty much explains it all, but I’ll summarize the important features here:

• You can try out one (or more) ZeroIntall feeds interactively
• All work happens in a subshell, with a modified shell prompt to clarify what context you’re in
• There is nothing to roll back, modify or undo - it’s completely stateless
• It works for published feeds (URLs) as well as unpublished or development local feeds
• It works cross-platform

I really feel this is an important tool for helping people adopt and use ZeroInstall feeds. ZeroInstall is a great way to publish software, but until now it has been awkward to try one or more feeds out interactively, partly because there is nothing to install.

(view link)

For a while my preferred “bash script prelude” (the stuff you write in every bash script, before the actual content) has been a modest:

#!/bin/bash
set -eux

Those settings break down to:

• -e: fail when any subcommand fails (the number of scripts that proceed to do completely incorrect things after an earlier command failed is criminal). You can always silence errors by using || or an if statement, but (much like exceptions in a high-level language) silence should not be the default.
• -u: fail when an unknown variable is referenced - mostly guards against silly typos, but why wouldn’t you? It’s sometimes not what you want with variables that may be optionally set by the user, so sometimes I’ll move the set -u down in the script after I’ve dealt with setting those to their defaults if they are not present.

• -x: trace all commands, after expansion. E.g with a line like: x=123; echo $x, Your output will be:

  • x=123
  • echo 123 123

This gets very noisy, but frequently I find it’s better to have more output than less - when a script breaks, often you can tell why just by looking at the last command trace (especially when combined with the -e option). And it’s easier to ignore excess output than it is to guess what went wrong.

The new hotness

Today, I discovered / created a much longer (and uglier) prelude, but I think it’s going to be worth it. Here ‘tis:

#!/bin/bash
set -eu
set -o pipefail
export PS4='+ ${FUNCNAME[0]:+${FUNCNAME[0]}():}line ${LINENO}: '
syslogname="$(basename "$0")[$$]"
exec 3<> >(logger -t "$syslogname")
BASH_XTRACEFD=3
echo "Tracing to syslog as $syslogname"
unset syslogname
debug() { echo "$@" >&3; }
set -x

A mouthful, indeed. Lets go though it line-by-line: