I am Hack Sparrow
Captain of the Internets.

Node.js Async Programming

One of the most important highlights of Node.js is that it is asynchronous. At the same time its async nature can also become its Achilles' heel - if one doesn't know how to manage the nested async callback functions.

Take a look at this piece of code with three levels deep async callbacks.

var fs = require('fs');
var http = require('http');
var path = './headers.txt';

// i. check if headers.txt exists
fs.stat(path, function(err, stats) {
    if (stats == undefined) {
        // ii. fetch the HTTP headers
        var options = {
            host: 'www.wikipedia.org',
            port: 80
        };        
        http.get(options, function(res) {
            var headers = JSON.stringify(res.headers);
            // iii. write the headers to headers.txt
            fs.writeFile(path, headers, function(err) {
                console.log('Great Success!');
            });
        });
    }
    else { console.log('headers already collected'); }
});

With every level it gets worse and worse. Imagine it was 10 levels deep! You must have figured the async callback problem by now. Let's see how we can fix it.

This problem becomes a no-problem, with the use of an aptly named Node.js module called async. Let's find out how to use it.

First install async:

$ npm install async

Now here is the modified code with async:

var fs = require('fs');
var async = require('async');
var http = require('http');
var path = './headers.txt';

async.waterfall(
    [
        // i. check if headers.txt exists
        function(callback) {
            fs.stat(path, function(err, stats) {
                if (stats == undefined) { callback(null); }
                else { console.log('headers already collected'); }
            });
        },
        
        // ii. fetch the HTTP headers
        function(callback) {
            var options = {
                host: 'www.wikipedia.org',
                port: 80
            };        
            http.get(options, function(res) {
                var headers = JSON.stringify(res.headers);
                callback(null, headers);
            });
        },
        
        // iii. write the headers to headers.txt
        function(headers, callback) {
            fs.writeFile(path, headers, function(err) {
                console.log('Great Success!');
                callback(null, ':D');
            });
        }
    ],
    
    // the bonus final callback function
    function(err, status) {
        console.log(status);
    }
);

Isn't that much neater? With the use of async, callbacks don't grow horizontally, they grow vertically, very much like synchronous calls.

While the async module offers many other useful methods, only two of them are enough for most async callbacks in Node.js - series() and waterfall().

Both these methods accept an array of functions to be executed one after another, and an optional callback function which will be executed after all the functions have executed without any errors.

The difference between series() and waterfall() is that the functions in series() are executed one after another and their results stored in a response array in their corresponding indexes, and the response array is passed to the optional final callback function. In waterfall(), the result of each function is passed on to the next function, and the result of the last function is passed to the optional final callback function.

In the above example, we used waterfall() because we needed to pass the result of step ii to the function in step iii.

Another simple example of using waterfall():

async.waterfall(
    [
        function(callback) {
            callback(null, 'Node.js', 'JavaScript');
        },
        function(arg1, arg2, callback) {
            var caption = arg1 +' and '+ arg2;
            callback(null, caption);
        },
        function(caption, callback) {
            caption += ' Rock!';
            callback(null, caption);
        }
    ],
    function (err, caption) {
        console.log(caption);
        // Node.js and JavaScript Rock!
    }
);

The first parameter, null, in callback(null, caption) is for any error you might want to throw. If the error parameter is a non-null value, other functions in the queue are not executed, the callback function is called with the error value immediately.

When the error value is null, the following parameters after null are passed on to the next callback function. Don't expect to find the error parameter in the next function even though it is the first parameter in callback(). It never makes it to the next function. The last parameter of callback() is understood as the next callback function.

Here is an example of using series():

async.series(
    [
        function(callback) {
            callback(null, 'Node.js');
        },
        function(callback) {
            callback(null, 'JavaScript');
        },
    ],
    function(err, response) {
        // response is ['Node.js', 'JavaScript']
    }
);

The second parameter of series() is the optional callback function to which has access to the array of results from all the function series. You may or may not use the callback function, depending on your requirements.

An interesting thing about series() is that you can pass an object instead of an array, with the functions attached to the object keys. The results of functions replace their corresponding functions in the object. Here is an example:

async.series({
        one: function(callback) {
                callback(null, 'Node.js');
            },
        two: function(callback) {
                callback(null, 'JavaScript');
            },
    },
    function(err, response) {
        // response == {one: 'Node.js', two: 'JavaScript'}
    }
);

So there you have it, two efficient methods for handling asynchronous callbacks in Node.js. For other methods of async, visit it's GitHub page. If you need any clarifications about series() and waterfall(), ping me in the comments.

5 Responses to “Node.js Async Programming”

  1. Lyman says:

    cool ;)
    good post about the nodejs cb toooo much levels problem!

  2. mroff.com says:

    nice article. thanks!

  3. Mau says:

    Thanks for sharing. Understanding how async works was driving me nuts. Your tuto put things clear.
    Just one question: why if I make a second call to async only the first will execute?

    For instance:

    async.waterfall(
    [
    function(callback) {
    callback(null, 'Node.js', 'JavaScript');
    },
    function(arg1, arg2, callback) {
    var caption = arg1 +' and '+ arg2;
    callback(null, caption);
    },
    function(caption, callback) {
    caption += ' Rock!';
    callback(null, caption);
    }
    ],
    function (err, caption) {
    console.log(caption);
    // Node.js and JavaScript Rock!
    }
    );

    async.series(
    [
    function(callback) {
    callback(null, 'Node.js');
    },
    function(callback) {
    callback(null, 'JavaScript');
    },
    ],
    function(err, response) {
    // response is ['Node.js', 'JavaScript']
    console.log(response);
    }
    );

  4. Mau says:

    Please ignore my last comment. My bad, it does work pretty good actually. Thanks again.

  5. lelouch says:

    Thanks :D that was very simple .. Saved my life

Make a Comment