Implementing Async/Await using JavaScript Generators

I use async/await pretty often at work, but how the syntax is applied under the hood has always been a bit of a mystery to me.

Like many JavaScript developers, I know that await allows us to write asynchronous operations in a synchronous-like manner. I also know that async functions always return a JavaScript promise. I kind of know that async/await is built off JavaScript generators, but it’s not clear exactly how the two concepts might be connected.

This last point, in particular, is the bit that gets to me. What if we didn’t have async/await? How would we derive it from other JavaScript features? Could we build our own asyncify function?

Iterators and Generators

Let’s start with a quick refresher on iterators and generators.

Generators are simply special iterators. An iterator is an object that implements the iterator API. This API expects any iterator object to contain a .next() method which, when called, returns an object in the form of { done: [some boolean], value: [some value] }.

Iterators generally deal with collections of data. For example:

// Given an array
const arr = [1, 2, 3];

// We can access its default iterator
const iter = arr[Symbol.iterator]();

console.log(iter.next()); // { value: 1, done: false }
console.log(iter.next()); // { value: 2, done: false }
console.log(iter.next()); // { value: 3, done: false }
console.log(iter.next()); // { value: undefined, done: true }

Generators are special in the sense that they allow us to go beyond working with static collections of data. With generators, we can repeatedly start and pause a function’s execution, which enables us to produce dynamic collections of data.

// Defining a generator function
function* addGen(x, y) {
  const first = yield x;
  const second = yield y;
  return first + second;
}

// Instantiating a generator
const gen = addGen(1, 2);

// Running a generator
console.log(gen.next()); // { value: 1, done: false }
console.log(gen.next(5)); // { value: 2, done: false }
console.log(gen.next(5)); // { value: 10, done: true }

Running to Completion (Synchronously)

The ability to pause a function and continue executing it later is an incredible language feature. In fact, it’s crucial in the context of writing synchronous-looking asynchronous code –– but we’ll come to that later.

For now, let’s forget about asynchronicity and just figure out out how we can run synchronous generators to completion.

Recall that every .next() call accepts arguments that can be freely defined by the developer.

// Defining a generator function
function* addGen(x, y) {
  // `first`: Arg passed into gen.next() *after* yielding 1
  const first = yield x;
  // `second`: Arg passed into gen.next() *after* yielding 2  
  const second = yield y; 
  return first + second;
}

This is why our final call to gen.next in the previous example returned 10 instead of 3, despite the fact that addGen was called with 1 and 2:

// Instantiating a generator
const gen = addGen(1, 2);

// Running a generator
console.log(gen.next()); // { value: 1, done: false }
console.log(gen.next(5)); // { value: 2, done: false }
console.log(gen.next(5)); // { value: 10, done: true }

To make our generators run to completion without the need of any kind of developer-defined input, we’ll make it requirement that gen.next() is always called with the previously-yielded value:

// Instantiating a generator
const gen = addGen(1, 2);

// Running a generator
const yielded1 = gen.next(); 
const yielded2 = gen.next(yielded1);
console.log(gen.next(yielded2)); // { value: 3, done: true }

If you think about it, this is practically equivalent to “converting” addGen into something like:

function add(x, y) {
  const first = x;
  const second = y;
  return first + second;
}

It’s almost as if we’ve removed the generator syntax and stripped addGen of the yield keyword.

Let’s define a conversion function, getGenRunner, which returns a new function that executes the provided generator function as if it wasn’t a generator.

function getGenRunner(generatorFunc) {
  return function genRunner(...args) {
    const gen = generatorFunc(...args);

    function processNextState(nextState) {
      const { done, value } = nextState;

      // Terminal case: Generator is done
      if (done) {
        return value;
      }
      return processNextState(gen.next(value));
    }

    const returnValue = processNextState(gen.next());
    return returnValue;
  }
}

const add = getGenRunner(addGen);
add(1, 2); // 3

In the above code, all genRunner does is that it repeatedly invokes gen.next with the previously-yielded value until the generator has been marked as done.

At this point, you might be thinking, “Why go through all this trouble of turning generators into non-generators?” If building an asyncify function is our goal, it seems like we’ve taken a step back rather than a step forward.

To see why, I’ll need you to bear with me for just a little while longer! Even though we’ve sort of hidden the fact that add is built on top of a generator, we’ll soon be using these hidden generator features to handle asynchronicity.

Running to Completion (Asynchronously)

Consider the following code:

// Defining a generator function
function* delayedAddGen(x, y) {
  const first = yield new Promise((resolve) => {
    setTimeout(() => resolve(x), 1000);
  });
  const second = yield y;
  return first + second;
}

// Instantiating a generator
const gen = delayedAddGen(1, 2);

// Running a generator
const nextState = gen.next();
console.log(nextState) // { value: <Promise>, done: false }

Running the generator for the first time yields an object wrapping a Promise. We can’t simply invoke gen.next with this Promise. Instead, we’ll need to wait for the Promise to resolve and then continue the generator’s execution with the resolved value.

To forward the resolved value to the next generator call, we could subscribe gen.next as a promise fulfillment callback on the next line:

// Running a generator
const nextState = gen.next();
console.log(nextState) // { value: <Promise>, done: false }
nextState.value.then(gen.next);

This is nice, but how do we integrate this with getGenRunner? It seems that our requirements have expanded, as we now need to handle synchronously-evaluated values as well as asynchronously-evaluated ones (i.e. Promises).

Since we’re no longer only dealing with synchronous behaviours, it seems fitting to rename getGenRunner to asyncify. This is what we’ve been working towards the whole time –– asyncify will help us convert generator functions into async/await-like functions.

function asyncify(generatorFunc) {
  return function asyncifiedFunc(...args) {
    const gen = generatorFunc(...args);

    function processNextState(nextState) {
      const { done, value } = nextState;

      if (done) {
        return value;
      }

      // Case 1: Yielded a Promise
      if (
        typeof value === "object" &&
        typeof value.then === "function"
      ) {
        return value.then(resolvedValue => {
          return processNextState({   
            done,
            value: resolvedValue
          })
        })
      }

      // Case 2: Yielded a non-Promise
      return processNextState(gen.next(value));
    }

    const returnValue = processNextState(gen.next());
    return Promise.resolve(returnValue);
  }
}

const delayedAdd = asyncify(delayedAddGen);
delayedAdd(1, 2).then(x => console.log(x)); // 3

Note that I’ve called Promise.resolve on the returnValue. Since async functions always return a Promise, we want to replicate that behaviour here as well.

In other words, even generators that purely contain synchronous operations will still be asyncify-ed into Promise-returning functions:

const add = asyncify(addGen);
add(1, 2).then(res => console.log(res)); // 3

Considerations: What is this?

Our implementation of asyncify is not terrible, but there are a number of things we ought to improve on.

First of all, let’s think about how context objects will be bound. What if our input generator functions contained references to this?

function* contextualAddGen() {
  const first = yield this.x;
  const second = yield this.y;
  return first + second;
}

The key here is to use .call to bind the context of the underlying generator in asyncify:

function asyncify(generatorFunc) {
  return function asyncifiedFunc(...args) {
    // Invoke `.call`
    const gen = generatorFunc.call(this, ...args);

    function processNextState(nextState) {
      // ...
    }

    return processNextState(gen.next())
  }
}

const myObj = {
  x: 1,
  y: 2,
  add: asyncify(contextualAddGen)
}

myObj.add().then(res => console.log(res)); // 3

To learn more about binding generators, I recommend checking out this superb article by Ben Nadel.

Considerations: Error Handling

The other critical consideration relates to error handling. Given that we’re handling synchronous and asynchronous code paths, we want to make sure that both types of errors eventually cause the Promise returned by the asyncified function to be rejected.

function asyncify(generatorFunc) {
  return function asyncifiedFunc(...args) {
    const gen = generatorFunc.call(this, ...args);

    function processNextState(nextState) {
      const { done, value } = nextState;

      if (done) {
        return value;
      }

      if (
        typeof value === "object" &&
        typeof value.then === "function"
      ) {
        return value.then(resolvedValue => {
          return processNextState({   
            done,
            value: resolvedValue
          })
        }).catch(err => {
          // Invoke gen.throw
          return processNextState(gen.throw(err))
        });
      }

      return processNextState(gen.next(value));
    }
  
    try {
      const returnValue = processNextState(gen.next());
      return Promise.resolve(returnValue);
    } catch (err) {
      return Promise.reject(err);
    }
  }
}

There are two cases to handle.

First, if one of our yielded Promises gets rejected, we make sure to propagate that error to the generator in the .catch() callback. We’ll use gen.throw(), which allows us to fire errors at generator entry points (i.e. places where the generator resumes its execution).

Second, if any errors remain uncaught in the generator itself, then the asyncified function’s returned Promise must be in a rejected state –– so we return Promise.reject(err) instead of returning a resolved value.

Ideally, the generator functions we pass to asyncify should already contain their own try/catch blocks and handle any errors in advance for us. That said, we cannot assume that this will always be the case. Thus, it is necessary for us to guard against such errors within asyncify itself.

Concluding Thoughts

With that, we’ve implemented a pretty decent version of async/await using our very own asyncify function. To be clear, I don’t believe that the underlying behaviours surrounding async/await map entirely to the implementation I’ve outlined above –– certain aspects, such as error messages annd their associated stack traces, are likely to differ.

Regardless, the goal of this article was to hammer home the strong relationship between async/await and generator functions, and I hope it’s achieved that. In particular, I hope this article has helped you better appreciate the language features and coding patterns that have enabled us to reason about asynchronous flows in JavaScript a lot more intuitively.