GFX::Monk Home

Posts tagged: javascript

Figuring out what transducers are good for (by trying to use them for a bunch of problems in JavaScript)

I’ve been aware of transducers for a little while, but haven’t actually used them, or even really felt like I fully grokked what they were good for. They come from the clojure community, but are making their way into plenty of other languages and libraries too. I’ve seen claims that they are a game-changing, breathtaking new concept, which didn’t really square with what they looked like.

So I thought I’d learn more about them by just attempting some plausible but detailed examples with them in JavaScript. If you’ve heard about transducers but aren’t really sure what they’re good for, perhaps this’ll help clarify. And if you’ve never heard of transducers, feel free to take a detour via the clojure documentation.

Oni Conductance

This past week, we (Oni Labs) announced Conductance, the next-generation web app server built on the StratifiedJS language (which we also built, and which has seen a number of steadily improving public releases over the past couple of years).

For a long time, I’ve been convinced that plan JavaScript is simply inappropriate for building large scale, reliable applications. That doesn’t mean it’s impossible, but the effort required to correctly write a heavily-asynchronous application in javascript involves a frankly exhausting amount of careful error checking and orchestration, and there are whole classes of confusing bugs you can get into with callback-based code which should not be possible in a well-structured language.

So I was extremely happy to join the Oni Labs team to help work on StratifiedJS, because it’s a much more extensive (and impressive) attempt to solve the same problems with asynchronous JavaScript that I was already trying to solve.

Conductance is a logical progression of this work: now that we have StratifiedJS, we’ve used its features to build a new kind of app server: one which maintains all the performance benefits of asynchronous JavaScript (it’s built on nodejs, after all), but which makes full use of the structured concurrency provided by StratifiedJS for both server and client-side code. And not just for nice, modular code with straightforward error handling - but actually new functionality, which would be impossible or extremely ungainly to achieve with normal JavaScript.

If you’re interested in building web apps (whether you already do, or would like to start), please do check out conductance.io for more details, and plenty of resources to help you get started building Conductance applications.

StratifiedJS 0.14 released

Today we (Oni Labs) released StratifiedJS 0.14. This is the first release since I started working here full-time, and it’s a big one: loads of useful new syntax, as well as a thoroughly kitted-out standard library.

StratifiedJS is a Javascript-like language that compiles to Javascript, but which supports advanced syntax and semantics, like:

  • blocking-style code for asynchronous operations (no callbacks!)
  • try/catch error handling works even for asynchronous code
  • a structured way of managing concurrent code (waitfor/or, waitfor/and, waitforAll, waitforWirst, etc).
  • ruby-style blocks
  • lambda expressions (arrow functions)
  • quasi-quote expressions

Check it out at onilabs.com/stratifiedjs.

Module resolution with npm / nodejs

NodeJS’ require() method is special. npm is special. Some of that is good - its efforts to dissuade people from installing anything globally are commendable, for a start. But some of it is bad. It’s probably better to be aware of the bad parts than to learn them when they bite you.

Let’s run through a quick example of what happens when I install a package. For example, installing the bower package will:

  • install bower’s code under node_modules/bower
  • under node_modules/bower, install each of bower’s direct dependencies.

Of course, this is recursive - for each of bower’s direct dependencies, it also installs all of its dependencies. But it does so individually, so you end up with paths like (this is a real example):

node_modules/
  bower/
    node_modules/
      update-notifier/
        node_modules/
          configstore/
            node_modules/
              yamljs/
                node_modules/
                  argparse/
                    node_modules/
                      underscore

Unlike pretty much every package manager I’ve encountered, npm makes no attempt to get just one copy of a given library. After installing bower, NPM has unpacked the graceful-fs package into 4 different locations under bower. I’ve also installed the karma test runner recently, which indirectly carries with it another 10 copies of graceful-fs. My filesystem must be exceedingly graceful by now.

Looking for a good javascript mocking library

Lately I’ve been looking for a good mocking library for node.js. It’s not easy.

Here are some (I would have said obvious) features that seem to be missing in most of the libraries I’ve seen:

  1. create an anonymous mock object on which to add expected methods (no need to provide a template object)
  2. create a (named) mock function (i.e a directly callable mock)
  3. argument matchers (at least eq(), given how terrible javascript equality is)
  4. stub a single method of a real object for the duration of the test
  5. verify all expectations and revert all replaced methods (see #4) with a single call (to be called from a common tearDown())

I don’t want it to be tied to my test runner, I’m quite happy with mocha.

I prefer the rspec style of setting expectations on mocks before the code is run and having them verified at the end of the test, but it’s not a requirement.

I plan need it to run in node.js, but would like it to work in the browser (even if I have to use some sort of commonJS-shim).

Here are the libraries I tried to use or looked at, and reasons they will not suffice:

Honestly, I looked at most of the unit testing modules listed on the node.js wiki that sounded like they did mocking.

jasmine’s mocking support seems somewhat reasonable (I’ve used it before), but unfortunately it seems to be tied to the jasmine test runner, which is not acceptable for async tests.

I’m happy to be shown wrong about my conclusions here, or to be pointed to any mocking library that succeeds in most (or at least more) of my requirements. If all else fails I may consider porting my mocktest python library to javascript as best as the language will allow, but it’s probably a lot of effort - surely someone has written a good javascript mocking library somewhere amongst all this? What do other folks use?

Obligate.js

I’ve been doing some browser-side javascript lately, and getting frustrated at the mess that is browser-side modules.. So here’s a tiny library that, given a tree containing javascript files, will give you a single javascript file containing all of the modules, as well as a commonJS require method to use in browser-side code. No more accidental globals, your variables are local and you just add your module’s public interface to properties on the module-local exports object.

Of course, it also includes a tool to grab all the javascript code of your own and that of your recursive dependencies, specified in a zero install feed.

(view link)

Ruby's split() function makes me feel special (in a bad way)

Quick hand count: who knows what String.split() does?

Most developers probably do. Python? easy. Javascript? probably. But if you’re a ruby developer, chances are close to nil. I’m not trying to imply anything about the intelligence or skill of ruby developers, it’s just that the odds are stacked against you.


So, what does String.split() do?

In the simple case, it takes a separator string. It returns an array of substrings, split on the given string. Like so:

py> "one|two|three".split("|")
["one", "two", "three"]

Simple enough. As an extension, some languages allow you to pass in a num_splits option. In python, it splits only this many times, like so:

py> "one|two|three".split("|", 1)
["one", "two|three"]

Ruby is similar, although you have to add one to the second argument (it talks about number of returned components, rather than number of splits performed).

Javascript is a bit odd, in that it will ignore the rest of the string if you limit it:

js> "one|two|three".split("|", 2)
["one", "two"]

I don’t like the javascript way, but these are all valid interpretations of split. So far. And that’s pretty much all you have to know for python and javascript. But ruby? Pull up a seat.

Node.js child processes

me: Hey node, how can I tell if my child_process.spawn() call failed before it even runs the desired command? Is there a return value? An exception? Maybe there’s an error event I can listen for?

node: Even better! Here’s an example from the documentation to do exactly that:

var spawn = require('child_process').spawn,
    child = spawn('bad_command');

child.stderr.setEncoding('utf8');
child.stderr.on('data', function (data) {
  if (/^execvp\(\)/.test(data)) {
    console.log('Failed to start child process.');
  }
});

me: Thanks, that’s… That’s just stunning.

(For extra points, this doesn’t even work if the child process inherits stderr, as my actual code does. Is this just a silly example from the documentation, or is there really nothing better?)

Tame.JS: Async Flow Control

If you are interested in my defer work with async control flow in CofeeScript, you’ll probably be interested in Tame.JS. The guys from OkCupid have a history with this sort of thing, apparently they have been using a similar mechanism they built for C++ for years.

Tame allows for more explicit control over parallelism than defer, and is a pretty simple mechanism. Contrasted to Stratified JS it seems to be simpler and more interoperable with existing javascript codebases, but also has fewer features - Tame.js is at a similar level to defer, while Stratified JS offers additional features like promise values (strata), parallel composition, alternative composition and more.

Regardless of which you prefer, it’s good to see people tackling the problem despite the common wisdom seeming to be that there is no problem (or worse, that it can be adequately addressed with libraries alone).

As usual, there’s some good commentary going on at hacker news. There are even a bunch of people wondering when such a useful mechanism will arrive in CoffeeScript ;)

(view link)

Dealing with non-local control flow in deferred CoffeeScript

…a continuation (hah!) of defer: Taming asynchronous javascript with coffeescript.

In my last post, I outlined the ideas behind defer and its current state. One of the things I mentioned is that how to deal with return statements in asynchronous code is not yet decided. I’d like to explore a few of those ideas here. If you haven’t read the previous post, I suggest that you do.

defer: Taming asynchronous javascript with CoffeeScript

defer is something I’ve been working towards in various fashions for many months now. It’s a way of writing asynchronous javascript in a more straightforward and synchronous-looking way. My current approach has been to modify the CoffeeScript compiler to introduce a defer keyword. There is currently some debate on the issue (parts one and two have even more history) as to whether the functionality is necessary, useful or appropriate. Here, I hope to show the reasons behind the idea, why it does what it does, and how it helps programmers write more concise and readable code.

Battling javascript contortion with lisp (of all things)

I’m pleased to report that my crazy notions of replacing javascript metaprogramming with lisp metaprogramming appear to be headed steadily in the direction of success. It’d take a while to explain exactly what I’m up to (and the reasoning behind it), but essentially I’m trying to solve the same problem that async.js, narrative javascript and strands all try to solve: asynchronous callbacks are ugly, error-prone and downright confusing.

All of the above mentioned tools are unsatisfactory in various ways. Both narrative javascript and strands are complex enough that I uncovered serious bugs in both reasonably quickly. async.js is much more stable (and I actually have a working application with it), but still requires very careful programming and only works on mozilla browsers.

So I set forth with parenscript, which is essentially javascript dressed up as lisp. It doesn’t make for any prettier javascript, but it does make for one hell of a metaprogramming opportunity using lisp macros.

The aim is to convert straightforward, procedural-style code into the contortion required to appease asynchronous callbacks. I’m certainly not done yet, but I do have some compelling proof to show that it’s a plausible thing to do with lisp. Here’s a contrived example for, say, getting all items from whatever feed the supplied item-id belongs to. This involves getting the item to find the feed it belongs to, then returning all items contained within that feed. The “store” objects in this scenario refer to lawnchair stores, which do asynchronous local datastore lookups:

    (asyncfun get-sibling-items (item-id)
      (defer item (item-store.get id))
      (console.log (+ "item belongs to feed: " item.feed-id))
      (defer feed (feed-store.get item.feed-id))
      (ret feed.all-items))

And here is the generated javascript:

    function getSiblingItems(itemId, cb) {
      itemStore.get (id, function (item) {
         console.log('item belongs to feed: ' + item.feedId);
         feedStore.get(item.feedId, function (feed) {
            cb(feed.allItems);
          });
       });
    };

This is just a simple example, but the lisp code still shows a remarkable reduction of both noise and contorted control-flow, and the generated code ought to work in all browsers. Worth pursuing, certainly.

Javascript Object Promotion

So it’s not something you should have to worry about (often), but I just spent an inordinate amount of time figuring out exactly why a particularly innocuous-looking piece of code was failing. It’s also yet another surprising part of the javascript language, and if you often deal with javascript then it’s in your best interests to keep track of such oddities (there are a lot of them!).

To illustrate, consider the following array intersection function:

    var list1 = ["a", "b", "c"];
    var list2 = ["a", "b", "d"];

    function intersect(a,b) {
      // find the intersection of two arrays
      var intersection = [];
      jQuery.each(a, function() {
        if(b.indexOf(this) != -1) {
          intersection.push(this);
        }
      }
      return intersection;
    }

You might expect calling intersect(list1, list2) to return ["a", "b"], since those are the common elements in the two sets. However, what you really get is [].

The problem is that I was a bit lazy, and used jQuery.each() to iterate over my collection (I hate the for x in y construct, and for loops are just so C). The each method calls your provided function on each member of the provided array, setting this in the context of each call to the current array element.

But it turns out that when you call a method on a primitive string object, the javascript (well, ECMAScript really) language specifies that this call takes place not on the string primitive type, but on the String object type (it’s kinda like a boxed value in Java). That is, fun.call("some string") actually ends up as if you had written fun.call(new String("some string")).

The String and string objects will compare equal when using the “==” operator - but since they are of different types, they will not compare as equal using the “===” operator. Evidently that is the type of equality that indexOf() uses, therefore none of the String objects will ever appear in the array of string primitives.

Normally, you will neither notice nor care. However, when extending the String class or calling a function with a primitive type as the subject, you should keep this conversion in mind. Note that the conversion does not apply to function arguments, only to the subject of a function application (i.e this).


I’m continually amazed by the high quality answers found on stackoverflow.com - I posted this question asking why the object promotion occurs (and if that is indeed what’s going on), and was rewarded with an explanation and a link to the precise part in the ECMAScript spec where the behaviour is detailed.

Making sense of async.js

async.js is a fascinating library for taming asynchronous javascript. It’s highly experimental, and as far as I know, it only works in firefox. But the idea is an important (and useful) one, so I think it’s definitely worth knowing about.

There are a few approaches to dealing with asynchronous javascript:

  1. Compile [some language] to javascript This is the approach taken by GWT, pyjamas, and many others. It’s usually extremely heavyweight, so it mainly makes sense for big apps.

  2. Compile [almost-javascript] to javascript Most notably this includes Narrative Javascript and its successor, Strands. This is a reasonable approach, but the lack of maturity / tool support makes debugging extremely hard. Also, I ran into a number of bugs in both these libraries. The thought of finding and fixing more of those bugs is not at all fun.

  3. Write a javascript library This is doable, but typically looks hideous, convoluted, and is usually quite burdensome to try and use.

Async.js is the most plausible attempt at #3 that I’ve seen. It uses a reasonably clever (but not unknown) trick to to turn Javascript 1.7’s generator functionality into an event-based coroutine system. Specifically, this allows for a program to “wait” for a callback, while not actually blocking the javascript interpreter (as a synchronous AJAX call would). Go read the async.js page to learn more, because the following isn’t going to make much sense if you don’t know roughly how to use async.js.

Javascript's fragile "this" statement

Javascript’s this statement must be the most fragile and confusing statement I’ve come across. Many languages have the concept of “this”, but none mess with it to hard as javascript does. As I have recently discovered, there are two fundamental issues with javascript’s this:

  1. this does not get captured when storing an object member into a function object
  2. you can never actually be certain what this will be

For object-oriented programming, these two facts are entirely terrifying. Let me illustrate each one:

1. this does not get captured when storing an object member into a function object

function Obj() {
	this.method = function() {
		return "method() called - I am " + this;
	};

	this.toString = function() {
		return "[Obj instance]";
	}
}

Now, consider the following scenarios:

var obj = new Obj();
obj.method();
// returns "method() called - I am [Obj instance]"

var obj_method = obj.method;
obj_method();
// returns "method() called - I am [object DOMWindow]"

What happened to this? I can find no explanation anywhere as to why it has been lost (and the window object used in its place), but it is consistent. This may not seem like a big deal, but even aside from the awkwardness, it’s completely non-obvious - and therefore a great candidate for sneaky bugs.

2. You can never actually be certain what this will be

Some see this as a feature, and it is in some cases. But the fact remains that the caller of any function can set this to be any object they choose is cause for great suspicion on the part of any callback code.

What’s worse, as a library writer there are cases where it’s impossible to not mess with the value of this. Other languages have the concept of a unsplat operator. Google it if you don’t know this term, but basically it will turn a list of objects into an argument list. That is, func(1,2,3) is the same as func(*[1,2,3]) (where * is the un-splat operator). This is very important for higher-order / functional programming, where you might write a proxy function that wraps a normal function call with some useful behaviour.

Anyways, javascript does have an unsplat operator. Kind of… The following code will work:

function call_other() {
	var _arguments = Array.prototype.slice.call(arguments);
	var func = _arguments[0];
	var func_args = _arguments.slice(1);
	// do whatever proxy stuff you need to do here
	func.appy(null, func_args);
}

Except for that first parameter to the apply function. Whatever you pass in there is what this will be set to in the context of the called function. With no discernible means of extracting what this would normally be for the given function, it becomes impossible not to clobber the otherwise extremely-useful this statement.

navim update

So I’ve kept working on navim, my jQuery plugin for easily adding vim-style keyboard navigation to web pages. New features include:

  • shift+enter to open links in a new window (and the ability to tell if shift is pressed from a custom action callback)
  • fixed a bug that interfered with pressing return to submit a form
  • using focus(), blur() and tab-key navigation to better effect

I’ve also now implemented navim in my “read later” webapp, pagefeed. It was trivial enough to add “d” as an additional keyboard shortcut to delete the currently active item, which serves as a good example for anyone wanting to add their own custom action keys. The code is simply:

$(window).keypress(function(e) {
	if(e.which == 100) { // 'd'
		$(".navim_active").children("form.del").eq(0).submit();
		return false;
	}
});

The rise of vi keybindings on the web

Vi? On the internet? No, not that one. I just mean vi keybindings, not the rest of vi. And I’ve just written a jQuery plugin to help make this happen.

On the UNIX terminal, many commands use common keyboard shortcuts to navigate screenfuls of text. j/k for down/up, h/l for left/right, and so on. As far as I know, vi was the first program to use these shortcuts. vi is famous for its modal interface, where different keys mean different things depending on what mode you’re in. In text insertion mode, “hjkl” means exactly those letters. But in normal (or visual) mode, they are the keys you use for navigation. Using standard letters instead of the arrow keys could have come about for a number of reasons. Firstly, there’s the issue that different terminals send different key-codes for special keys like the arrow keys. That’s generally been solved these days, but the other reason still remains: By using the whole alphabet as control keys, you get a startling amount of “command bandwidth” - that is, commands in vi generally require much less finger-contortion than pretty much every other text editor.

vi keybindings are useful on the internet for a third (but related) reason - keybinding collisions. We already have meanings for what the arrow keys do (scroll), and trying to use control-key combinations for webpage-specific functionality is fraught with user frustration, confusion and technical issues.

I’ve noticed it already with a bunch of google products, especially since I started using gmail’s online version most of the time now (instead of Mail.app). Gmail and Google reader are the two big ones that I use, but I’m sure there are more. Both use vi-style keyboard navigation, and both are a delight to use with the keyboard. I’d be surprised if bespin lasts long before a vi-mode is added.

I guess many people think only data-heavy, webapps are worth learning keyboard shortcuts for. But the real benefit comes when everything (or at least most things you care about) use the same convenient shortcuts. I was absolutely delighted when I noticed that every issue of The Big Picture allows you to use j/k to jump to successive images - incredibly useful in this case, because page up/down rarely manages to line up to image boundaries. It’s an unobtrusive addition, and it won’t hurt regular users. But for those who do use it, it quickly becomes indispensable.

So what I’d love is for all websites with many conceptual “items” on a page to implement the j/k keybinding as a minimum (with horizontal and inter-page navigation coming later, hopefully). To this end, I have written a jQuery plugin that should make the process fairly trivial, and allow for “active item” decoration via CSS. Click that link for a demo you can try out yourself.

If there’s enough interest, maybe someone could turn it into a community-based firefox extension (a-la AutoPager) that allows users to define the navigation items for websites that haven’t supplied their own (google search would be a handy one).

It’s worth mentioning Vimperator, which brings vi keybindings to firefox’s UI. To clarify, I’m not talking about browser functionality. I’m happy to use the existing controls for a browser, and leave the alphabet keys for use by the web-page. That way there’s no overlap, and webpages can provide contextual navigation controls that are much more powerful than the basic “scroll down 30 pixels” that a browser provides.