This post summarises some recent experiments and learnings around concurrency & Koka. There’s no immediate application yet, just a bunch of thoughts which might be interesting if you’re into concurrency, parallelism, or Koka. If you’ve never heard of Koka before that’s OK, you don’t really need any prior knowledge (but I wrote about it here).

Koka and concurrency:

There are a few active avenues of interest when it comes to Koka and concurrency.

First, there’s the async effect, currently implemented in the community stdlib. That effect allows a function to suspend execution and await the result of a callback, as well as the ability to execute multiple async operations concurrently. It’s basically an implementation of the async / await semantics in JS and many other languages, but as a library (rather than built into the compiler). Currently this only supports koka code compiled into the JS backend. Under the hood, an async operation is represented as a continuation function - when the operation is complete, that function is invoked and (from the perspective of the Koka code you write), the async code resumes from where it left off.

There’s also work going on to create libuv bindings for koka. There’s a fully-featured attempt in this community repo, as well as a more minimal version in koka itself with just the core scheduling primitives. This work allows execution of async koka code in compiled binaries (the C target).

Both of these are currently limited to single-thread concurrency1, i.e. concurrency-without-parallelism. This is nothing to sneeze at, it’s done well enough for NodeJS for more than a decade, and it’s a step up from many scripting languages which only support synchronous IO.

Koka and parallelism:

But at the same time, Koka has some great fundamentals when it comes to true parallelism (with multi-threading). The way to share mutable state is via a ref, which is already thread-safe. And Koka’s reference counting algorithm was designed to perform well for both single and multi-threaded environments.

All this is to say: single-threaded concurrency is cool and all, but there’s no technical reason Koka couldn’t support true parallel concurrency like Go, Rust, OCaml and Guile Scheme.

What’s Guile Scheme? Don’t ask me, I’ve never used it. but I think of it often when it comes to concurrency. Years ago, I read Andy Wingo’s excellent series on implementing Concurrent ML primitives in Guile Scheme. It’s stayed in the back of my mind as an interesting point in the concurrency design space, and seems to keep coming up as a lesser-known approach which ought to be more widely known and adopted. See also Concurrent ML has a branding problem, where the tl;dr is “Concurrent ML’s primitives are great but the terms used to describe it are confusing so people ignore it”.

It’s been a while since I’ve been excited about a new programming language.

I’ve learnt and used a number of languages, either because of necessity or curiosity, but there’s only been a few languages in the past 20+ years where I’ve thought “this could be my new favourite language”.

For a long time Python was my preferred language for its simplicity, and then StratifiedJS for its powerful approach to asynchrony (and JS interop). Since then most of my interest has been in statically typed languages, particularly OCaml and Rust.

I’ve used plenty of other languages for my day job, and I consider myself lucky to have a day job where I’m mostly writing Scala - a really good mix of “powerful language” and “something I can get paid to write”. But I’m a big believer that there should be a language which is great for nearly everything I want to do, and none of those are it.

Koka?

So when I came across Koka recently and realised how strongly it aligns with the kinds of things I like in a language, I got pretty excited! Of course, it’s far too early to know if it will stay that way as the language and ecosystem (hopefully!) grow and become production-ready, but for now I’m very optimistic.

I’ve long considered fluency in Nix to be a superpower that pays off way more than you might imagine, if you haven’t experienced it yourself.

Sure, it helps with the obvious practical things you’d expect - my system setup is declarative, reproducible, and suffers from vanishingly few chaotic state-based issues that tend to plague less reproducible systems (like brew in particular).

There are plenty of downsides too - I run into nix-specific issues that my colleagues with more normal setups don’t suffer. Interactions with tools like bundler building native extensions can be frustrating at best.

Those are the unsurprising, surface level tradeoffs when using a good-but-novel package manager. The real superpower comes through the staggering amount of things which are not just possible, but downright straightforward due to the reliable, principled way that nix works. Here’s a good example from last week:

I don’t blog much these days, apparently I just use it to announce roughly one new thing each year. But I do want to post more writing, so here’s a description of how I think about cross-compilation in nix.

This post doesn’t assume much nix knowledge. It’s not a how-to post describing the (complex) process of how to cross-compile software, it’s more of an exploration of how cross-compilation conceptually works, because it’s something I learnt recently and found interesting.

I think nix is fantastic. Language-agnostic, cross-platform, reproducible, cacheable software building and distribution. It’s not an easy thing to learn, but the payoff is tremendous.

But one thing about nix is that you typically need to be all-in. When building software, this makes sense - Nix’s reproducibility only works if all your dependencies are themselves available within nix. But as a colleague casually suggested one day: shouldn’t it be possible to run nix-built software without installing nix?

I told him a few reasons why it’s harder than it sounds, mainly due to the hardcoded /nix/store path which is assumed by the entire ecosystem. But my mind dwelt on it in the background, and it turns out it turned out to be much easier than I first thought.

Runix

So, I’m announcing runix, which does just that - it’s a small, unobtrusive executable which allows running nix software from any binary cache. Features:

• small (<4mb compressed)
• fast (~10 microsecond overhead after initial download)
• no configuration required
• unobtrusive (no need for root access or a /nix directory)
• can use software from any nix-compatible binary cache (including cache.nixos.org and cachix)
• conveniently distribute software via runscripts

Runscripts are a runix invention, they’re a tiny wrapper around a list of derivations and binary caches. In addition, they support:

• self-bootstrapping (install runix itself if missing)
• multiplatform (execute a different nix derivation per platform)

runix intentionally lacks all the development and authoring features of nix itself - you can’t evaluate nix expressions or build software locally, you can only run existing software which someone else has built and pushed to a binary cache. But for distributing software, it’s a much easier way for users to access nix-built software.

chored is a utility for handling repetitive chores and files.

There are many repositories. Lots of them have similar (tedious!) things they need to do, which aren’t particular to that repository:

• building
• linting
• testing
• generation of configuration for tools:
  • CI configuration
  • build settings
  • all sorts if common or shared files
• release management
• documentation generation
• pull request automation
• updating dependencies

chored allows you to reuse solutions to these problems, and any others you can think of. It’s minimally intrusive and lightweight (just add one small wrapper script to your repo).

And even when tasks aren’t exactly the same, chored facilitates code reuse as easily as importing a URL, but as powerful as real libraries in a real programming language (because it is both of those things).