defer is something I’ve been working towards in various fashions for many months now. It’s a way of writing asynchronous javascript in a more straightforward and synchronous-looking way. My current approach has been to modify the CoffeeScript compiler to introduce a defer keyword. There is currently some debate on the issue (parts one and two have even more history) as to whether the functionality is necessary, useful or appropriate. Here, I hope to show the reasons behind the idea, why it does what it does, and how it helps programmers write more concise and readable code.

Update:

There has been further development on this feature in the issue thread, the gist of which is that I will not be continuing work on this feature for CoffeeScript. I still think it’s an excellent feature that should be implemented, and I hope that someone will do so in the future. But right now, it doesn’t make sense for me to keep trying. Please read the above issue thread for my more detailed reasoning.

On a related note, oni apollo has recently seen its first release. It covers most of the same concepts as I’ve talked about here, but it’s built on javascript’s syntax (rather than CoffeeScript’s). If you’re looking for a defer-like mechanism right now, I would suggest investigating oni.

Table of contents:

Part 1: What is asynchronous programming, and why is it so damn awkward?

Asynchronous programming, at least in the context of javascript, comes about as a necessary part of making use of browser functions that are themselves asynchronous. The most common example of this is in AJAX programming, where you ask the browser to perform a request, and you give it a callback to run when the request is complete. But in the time that the request takes to load, the javascript on the page continues to run, and the user interface remains responsive to the user. This is good - if you try to make a synchronous AJAX call, the entire browser will be unresponsive to the user until the request is complete. Here’s an example of an AJAX request in jQuery, using a callback parameter:

jquery.get("/page", null, function(data) {
	alert("I've got the data! it is " + data);
});

And it’s not just AJAX that is asynchronous. More and more complex APIs are being provided by browsers and libraries, and in order to remain responsive to the user, they make use of asynchronous callbacks. In fact, most functions that takes a callback parameter do so in order to remain asynchronous. A very important API introduced in HTML5 is offline storage - the ability to allow a webapp to store data on a client’s computer. This is an important ability for a webapp to have, if it is to remain useful when the network drops out or is otherwise unavailable. Lawnchair is a simple-to-use client-side persistence library that can make use of multiple backends. It has a very simple API, and most of its methods are asynchronous. For example, to get a document with id “123”, you would do the following:

lawnchair.get("123", function(object) {
	alert("object with id 123 is " + object);
});

Look familiar? Most asynchronous methods look very similar, taking a callback argument as the last parameter. This convention is something we’ll use to our advantage later.

What’s wrong with this picture?

For the case of AJAX programming, it seems reasonable to write a specific callback parameter for a request - you probably don’t make that many different AJAX requests, and each one is likely to be a transaction on its own - you are unlikely to perform multiple AJAX requests in response to a single event, nor are you likely to perform a different AJAX request based on the outcome of a previous request (at least not often).

That situation becomes quite different for the case of offline storage. In this case, it’s quite reasonable to perform many actions on the local database (like fetching a set of objects, each accessed by their id). And similarly, it’s quite common to make a series of requests - first get a list of IDs, then fetch the associated document or object for each ID.

What’s the difference between these two cases? In the AJAX case, the asynchronous task is the entire point of a piece of code. You set up data, you post it, and when it comes back you do something with it. It’s reasonable to put this “do something with it” in a callback, because it’s usually self-contained. But in the case of an asynchronous data store, the data store call is not actually the point of a given piece of code - you just use it to get some information, and then you continue with whatever processing you need to do once you have that information. Importantly, this often includes more asynchronous calls to fetch appropriate data, with additional processing on that data. Now you don’t simply have two parts to your code - “before AJAX” and “after AJAX” as before - you have many partitions in your code between calls to the data store that you have to break up separately, purely because those calls happen to return asynchronously. Code that is broken up in many pieces for this reason is often confusing and can get difficult to manage, but the real problem comes about when these partitions occur inside control-flow statements.

The effect of callbacks on control-flow

Control flow is a big part of how applications are structured. Conditional statements (if/else) and looping constructs (while/for) are the building blocks of programming. The problem is, callbacks make proper control flow difficult, because asynchronous events happen outside of the regular program flow. For example, consider a synchronous database get() method:

item = itemDatabase.get("123");
if(item.author == current_user) {
	return render_own_post(item);
} else {
	author_info = authorDatabase.get(item.author);
	return render_post(item, author);
}

But what if the database only had asynchronous methods? The return value of get becomes useless, because at the time the function returns, the item will not necessarily have been retrieved from the database. It’s only when the callback in invoked that you can use its content to affect control flow. That means the control flow must be moved to inside the callback, like so:

itemDatabase.get("123", function(item) {
	if(item.author == current_user) {
		return render_own_post(item);
	} else {
		authorDatabase.get(item.author, function(author_info) {
			render render_post(item, author);
		});
	}
});

But what has happened to the returns? The functions that are returning useful values are the anonymous callbacks we provided to get() - the database is the one invoking those when it has a result for us, and it certainly won’t know what to do with our return value - we intended them for the caller of our code. The only way to solve this is for this piece of code to itself accept a callback paramater, and to indicate a “return” by calling this function. Like so:

function render_item(callback) {
	itemDatabase.get("123", function(item) {
		if(item.author == current_user) {
			callback(render_own_post(item));
		} else {
			authorDatabase.get(item.author, function(author_info) {
				callback(render_post(item, author));
			});
		}
	});
}

It then becomes absolutely critical that the author of this function remembers to call callback, instead of just returning. And consider the caller of this function. Just like the database get method, our render_item method is also now asynchronous. The caller of our function must now provide a callback to us - and hence our caller also needs to accept a callback parameter, and remember to call it with the result instead of simply returning it. And the same goes for the caller of that function - it’s turtles all the way down, if turtles were callbacks.

Don’t break the chain

Now that every caller in any stack that ultimately makes a call to an asynchronous API must use a callback instead of returning its result, what happens if just one of these functions doesn’t call its callback?

Nothing.

At some point, nothing happens. The program starts to run, and for example the item may be retrieved from the database and the appropriate callback executed. But if any function in the callback chain fails to call its own callback under any circumstance, it would be as if the program just suddenly stopped running - as if a function never returned, but without the ability to interrupt the code and see what function failed to return - because it did return, just not into the callback you were expecting. In a heavily asynchronous program, this is going to happen to you during development. Tracking down where is incredibly difficult.

Aside: in other programming languages, this could be dealt with differently by blocking, waiting for the call to complete - turning an asynchronous call into a synchronous one. For better or worse (probably better, given that the entire browser locks up for synchronous javascript calls), there’s no way to block execution and wait for an event in javascript.

Also note that since one of the backends that lawnchair uses is asynchronous, all of the methods must be written in asynchronous style - there’s no way to abstract away the need for callbacks in javascript. In that way, asynchronous methods are somewhat viral - if anyone uses them, everyone must use them.

A brief interlude: CoffeeScript

CoffeeScript is a small language that aims to gloss over some of javascript’s blemishes, and generally give a cleaner syntax and useful features from languages like ruby and python into javascript. For the most part it reads like a cross between python and ruby, and compiles fairly straightforwardly into javascript. From here on in, I’ll be using CoffeeScript for example code instead of javascript. If you don’t understand CoffeeScript, it may be worth taking a detour via coffeescript.org.

The reason I’m using CoffeeScript for the examples is because the current implementation of the feature I’m talking about is implemented in my branch of the CoffeeScript language. That means I can give real examples that actually work in my version of the CoffeeScript language.

Asynchronous control flow is hard

Writing a loop that will perform multiple asynchronous operations is tough. For example, consider retrieving a set of documents by their ID. In synchronous code with CoffeeScript’s list comprehensions, it is pleasantly concise:

documents: (database.get(id) for id in document_ids)

or if you prefer a more functional approach:

documents: map(database.get, document_ids)

but what about using the asynchronous get method? Since it doesn’t return anything useful, we can’t simply collect the return values. I guess we’ll need to use a proper loop:

documents: []
for id in document_ids
	database.get(id, (item) -> documents.push(item))

That’s not so bad, but wait - how do you know when you’ve got all of the documents? You’ll have to use a callback, but only after you’ve received all of the documents:

get_all_documents: (callback) ->
	documents: []
	for id in document_ids
		database.get id, (item) ->
			documents.push(item)
			if documents.length == document_ids.length
				callback(documents)

Only now you don’t necessarily know what order the documents are in - since get is asynchronous, it could in theory return objects in any order. If you care about object order more than massively parallel database retrieval (and let’s face it, it’s probably not a massively parallel database store), you’ll want to do something like the following:

get_all_documents: (callback) ->
	documents: []

	get_next_document() ->
		if document_ids.length == 0
			return callback(documents)

		document_id: document_ids.shift()
		database.get(document_id, (document) ->
			documents.push(document)
			get_next_document()

	get_next_document()

Wow, our one-line list comprehension has ballooned into a 10-line recursive function. While fans of recursion may prefer this, there’s no doubt that it is far less concise - and all because we had to use callbacks!

Aside: we’ve also just modified the document_ids object, because we didn’t make a copy of it beforehand. This may or may not be a problem in your program, but it certainly opens up possibilities for confusing bugs

Now, you could certainly write a library or helper function to handle this case for you - that way, you only need to write it once, and you can re-use it. But it would still require the use of callbacks, and this is not alone amongst cases where callbacks make simple synchronous code bloated and difficult to follow. Consider the following use of if/else:

do_the_thing_with_the_document: ->
	if some_condition
		document: database1.get id
		do_something(document)
	else
		document: database2.get id
		do_something_else(document)
	document.read_count += 1
	return document

With callbacks, it transforms into this:

do_the_thing_with_the_document: (callback) ->
	rest: (document) ->
		document.read_count += 1
		callback(document)

	if some_condition
		database1.get id, (document) ->
			do_something(document)
			rest(document)
	else
		database2.get id, (document) ->
			do_something_else(document)
			rest(document)

Not only is it more noisy (remember, CoffeeScript is terse – it’s even worse in javascript), but the code is now in a different order to the way it’s executed - not a great recipe for readable code. That’s because we had to pull out rest as a function, because there’s no way for the execution of the two asynchronous calls to rejoin as execution does after an if/else branch. We effectively can’t make use of control flow after we’ve made an asynchronous call.

Some will argue that the restrictions of using callbacks simply means you have to factor your code more carefully to avoid such awkward pieces of code. That certainly may be true in some examples, but not all. If, for and friends exist in every procedural language for a good reason - they’re useful constructs. Callback-based code cannot fully make use of these constructs, so we have to jump through awkward hoops and refactoring just so that the right pieces of code appear in atomic function blocks. It’s not a good way to structure code.

Whew. So, hopefully I’ve explained the reasons why I think asynchronous programming is problematic, difficult to get right, and causes a lot of unnecessary noise in code. It’s time for the sell:

Part 2: So what is defer, and why should I use it?

Continuations

A Continuation is an object that represents “the rest of the currently executing program”. If you took a given program, started running it and then paused at some point, there are two parts of the program. There is all of the things that have happened since the program started running. And then there are all the things that, if the program were to be resumed, will happen up until the program completes. A continuation is simply the name for an object that represents that second part of the program - its entire future.

Scheme is one of a small number of languages that have support for continuations provided by the runtime, via call-with-current-continuation. This makes a function call, but passes it one additional paramater - the continuation. If we had support for continuations in javascript runtimes, you could very easily make asynchronous functions appear synchronous. For example:

data: jquery.get.call_with_current_continuation('/data/', null)
alert("the data is: " + data)

Even though this code looks synchronous, what really happens is that at the point of the call to get, the program’s future execution would be wrapped up in a second argument to get. It wouldn’t continue to run immediately, rather it would only resume once the jquery library called this continuation object with the now-retrieved data from the server.

Continuation-passing-style

If javascript had continuations, I would be a happy chappy. Sadly, it does not. What this reduces us to is called continuation passing style - our program runs in much the same way as it would if we had true continuations, except that we’re writing the continuation objects ourselves, manually. As I’ve shown above, this is both error-prone and awkward. Surely this is a task that a smart compiler should be able to help us with.

Let the computer write your callbacks

So here we are. We aren’t going to get continuation support from browsers any time soon. And without continuation support, you can’t do this sort of thing as a library - asynchronous flow still gets in the way of your program’s flow, so you end up having to represent program flow as data instead of code. Lisp fans may find this appealing, but it’s not very much like javascript, nor have I seen any cases of it being easy to read and debug when things go awry.

What other tools do we have at our disposal? Compilers, of course! In theory, a compiler could take our synchronous-looking code, and spit out code that will automatically provide a continuation object to the asynchronous functions that we call. Lets see what that would like:

  • input CoffeeScript

      get_an_item: (id, callback) ->
      item: defer database.get(id)
      item.read_count += 1
      callback(item)

defer is the word I’ve chosen to represent that a function call is not to happen synchronously, but that a continuation should be added to the argument list, and that the rest of this function will execute later, when the callback is executed. In this way, the execution of the rest of the function is deferred until the asynchronous result is available. What will it generate?

  • output CoffeeScript (it will actually generate javascript, but having the input and output represented in the same language will give a better understanding of what work the compiler is doing)

      get_an_item: (id, callback) ->
      item: database.get(id, (item) ->
          item.read_count += 1
          callback(item)
      )

So it’s not a huge change when comparing the source of those two examples. But defer represents a much greater change than just reducing the number of anonymous functions and indentation in your code. Because the compiler has access to the structure of your code, it can (and does) transform your code into a form that will actually follow standard procedural rules when used inside and around control flow statements like if branches and for loops. Consider the example from earlier, when we had to retrieve a list of documents from an asynchronous database, given a list of ids. Here is how it ended up in javascript:

get_all_documents: (callback) ->
	documents: []

	get_next_document() ->
		if document_ids.length == 0
			return callback(documents)

		document_id: document_ids.shift()
		database.get(document_id, (document) ->
			documents.push(document)
			get_next_document()

	get_next_document()

..and here is how it can be written in CoffeeScript, with defer:

get_all_documents: (callback) ->
	documents: (defer database.get(id) for id in document_ids)
	callback(documents)

That looks familiar, doesn’t it? defer means that the same useful constructs for synchronous control-flow can be used with asynchronous code, which helps tremendously to keep your code readable.

It’s worth mentioning here that defer does not (yet) solve the problem of requiring your function to both take and make use of a callback parameter, as true continuations would. I plan to write a follow-up post with thoughts on how that could be addressed, but since javascript is a dynamic language, there may be no way to avoid this that isn’t likely to cause confusion when synchronous functions meet asynchronous ones.

The benefit this gives you on a small scale might not appear significant. But when it comes to writing an actual application in javascript, the difference is huge. I started this journey by attempting to write an offline-capable RSS reader almost entirely in javascript, and I quickly learned that writing a real application (as opposed to a script) is seriously error-prone and difficult when you have to write asynchronous javascript yourself.

And why wouldn’t you want the compiler to write your error-prone callbacks for you? It keeps your code cleaner, and will never get confused or forget a critical piece of boilerplate.

Try it out!

My deferred branch of CoffeeScript is fully functional, if a little slow in keeping up with changes to the main language. You can check it out and compile some defer-enabled CoffeeScript like so:

git clone https://gfxmonk@github.com/gfxmonk/coffee-script.git
cd coffee-script
bin/coffee -c <your-script>.coffee

You can also just look at some (contrived, but thorough) examples in the test suite. All of the cases in the test_defer.coffee test file work as expected, and give you a good example of the specific features of defer.

Currently, the code it generates is not optimised. If you look at the code, it might seem to be doing some unnecessary things, but these are in order to keep the simplicity of the compiler to a minimum - I care more about functioning code than optimised code. If you want optimised code, you may want to try applying the google closure compiler to the generated javascript.

Make it official!

I’d very much like to have defer merged in to the official CoffeeScript branch, so that everyone can use it. My branch is functional, but there is a lot of work involved in keeping it up to date with the main CoffeeScript source, and I certainly can’t promise to do that forever.

If you think the feature should be included in CoffeeScript, I’d appreciate if you could mention this (and your reasons) on the issue page. Without enough support for the feature it may never get included, and only my version of CoffeeScript will have it. Everyone could still use it if they wanted to, but I don’t want to support an entire fork of a language for just one (admittedly important) feature.

Part 3: Addendum

defer is not perfect

Since defer is a relatively simple translation at the compiler level, and since javascript is so dynamic, there are some things that can’t reasonably be done. As I mentioned above, having continuations work across function boundaries (with return) is something that has some options, but all current ideas have their downsides.

The other big issue is exceptions. Nothing special is done when exceptions are thrown, which makes them hard to track down (without a stack trace). This is no worse than what already happens with exceptions in asynchronous code, but sadly it is also no better.

Additionally, there is currently no support for providing a defer callback as anything other than the last argument to a function call. There’s no reason it can’t be done if required, but it’s not a high priority for me, as it’s very easy to write a one-off wrapper for the few functions that don’t follow this convention.

Similar projects

I am not the first person to try and improve the state of asynchronous programming in javascript. In fact, I did so only after trying out a number of other projects, and found each to be lacking in some way. If you’re interested, please check out the following projects to see how others have tried to solve similar problems:

narrativejs: A javascript-to-javascript compiler. It deals with the more general case of pausing javascript execution, as opposed to just dealing with callbacks. Produces unsightly and hard-to-debug code, and I encountered multiple critical bugs in the implementation. The original project is also abandoned.

strands: A continuation of narrative-js, after the original author abandoned it. Has many of the same issues.

async.js: An attempt to create a javascript library to deal with asynchronous code. It is a good idea, but using it in practice is prone to errors and there are a lot of things you must remember, or else your code will just fail to work. I don’t think the problem can be solved in a library, and these issues are a good example of why.