Abstract

We perform an empirical study to characterize JavaScript callback usage across a representative corpus of 138 JavaScript programs, with over 5 million lines of JavaScript code.

We find that on average, every 10th function definition takes a callback argument, and that over 43% of all callback-accepting function callsites are anonymous. Furthermore, the majority of callbacks are nested, more than half of all callbacks are asynchronous, and asynchronous callbacks, on average, appear more frequently in client-side code (72%) than server-side (55%).

We also study three well-known solutions designed to help with the complexities associated with callbacks, including the error-first callback convention, Async.js library, and Promises. Our results inform the design of future JavaScript analysis and code comprehension tools.

Downloads

A preprint of our ESEM 2015 paper is now available here:

Download Preprint


Full dataset can be downloaded from here:

Download Data


29 Examples of nesting with the highest depth of 8 we found among the test subjects can be downloaded here:

Download Examples


Source code used to analyze the subject systems can be accessed here:

Source Code

Subject Systems

The 86 NPM modules we study are the most depended-on modules in the NPM repository. The other subject systems were selected from GitHub Showcases, where popular and trending open source repositories are organized around different topics.

DataViz

# Project Client Server Revision Git URL
1 Chart.js 1cfa42bc24e152cfef79523c310fae7d4c458635 https://github.com/nnnick/Chart.js
2 Leaflet 7cafaa63469ee4e2bebd0a8e390c8d798df51d4a https://github.com/Leaflet/Leaflet
3 arbor c5f7ec33c9f4ca28218779ca65e4bdcdec481884 https://github.com/samizdatco/arbor
4 cesium c71f81c935783aa3de065a5fe043ee073b7a6b3c https://github.com/AnalyticalGraphicsInc/cesium
5 chartist-js 51553dc5dfe7016fd451d674edc7f2f8cbf493a1 https://github.com/gionkunz/chartist-js
6 d3 f82dd6fb414a15bca4f9c39c7c9442295ddea416 https://github.com/mbostock/d3
7 dc.js 29dfbedcb0ef8cdd96daf254b3ce32b51f015bbf https://github.com/dc-js/dc.js
8 echarts 0b7cf670ae60ada1de04aef08958ed53af13df9b https://github.com/ecomfe/echarts
9 envisionjs 747cccd6f72f3b13539685c61904f8394cf6a0ac https://github.com/HumbleSoftware/envisionjs
10 kartograph.js 4b8525e2e9ab4cfd97c680a21158416360e3d6ef https://github.com/kartograph/kartograph.js
11 modestmaps-js fedc94504ef95140a8ccc3c19907445987d13c6d https://github.com/stamen/modestmaps-js
12 peity d99db2ca721e6f2bcb1bbdfb9c848d2cef35a809 https://github.com/benpickles/peity
13 raw 3555470a81d49d7a31742e48751f86b84c15e6ef https://github.com/densitydesign/raw
14 recline 5b8b02b296b1274c5d0f246bc3ac9dddf59c8c1a https://github.com/okfn/recline
15 sigma.js 389c2f4230d944568b9efd977ca91728774dbe39 https://github.com/jacomyal/sigma.js
16 vega 1215de8e996b742907b5961d1c831a276cfd6903 https://github.com/trifacta/vega

Game Engines

# Project Client Server Revision Git URL
17 Babylon.js 4ea3332f48299a84b4bca8dc703125bf0eba558a https://github.com/BabylonJS/Babylon.js
18 Crafty 912a0097bd855b7f0de0788df48933127c04e6db https://github.com/craftyjs/Crafty
19 PhysicsJS 2e062a0c069fb7b4056c3aa742931ac2d940bac0 https://github.com/wellcaffeinated/PhysicsJS
20 cocos2d-html5 fa68c45229f8929fab5b73a7679896804880cb58 https://github.com/cocos2d/cocos2d-html5
21 cutjs 2e8fd61a496307a883918c3f39e9286f76cab347 https://github.com/piqnt/cutjs
22 engine 7e576c84423da52153fc202559dfac639e418e21 https://github.com/playcanvas/engine
23 kiwi.js 1ea2272fb0c5e313a2b9f4c9f0386437d1bf283e https://github.com/gamelab/kiwi.js
24 lycheeJS 58f719815c2e0b0bb63d78e5bcb23443ee1da78d https://github.com/LazerUnicorns/lycheeJS
25 melonJS b0c445f3520873a4a9f89fcf26c1f1d797e67c7f https://github.com/melonjs/melonJS
26 phaser 62b7fce9c8fc358733df8635fdb7256b92fd6b44 https://github.com/photonstorm/phaser
27 pixi.js ae42e515b9046c67c8c5623f62712c65127bdddc https://github.com/GoodBoyDigital/pixi.js

MVC Frameworks

# Project Client Server Revision Git URL
28 angular.js fbbf6ac161a2087a64d50878b02ed7f618b182cc https://github.com/angular/angular.js
29 backbone 70636d6d854b32cb73bb8c16ba827b590b33bc88 https://github.com/jashkenas/backbone
30 brick 4cf4100085e736805f5f57b6f09da0e724cd4099 https://github.com/mozbrick/brick
31 ember.js 977efcf517ba5039c708915042f3839d4546fc57 https://github.com/emberjs/ember.js
32 knockout 624ddacd5a879ef5bf555d89e648f982a6bd6a4e https://github.com/knockout/knockout
33 polymer 2d000d10fec1b2415fa4edabfed2b8641264deee https://github.com/Polymer/polymer
34 spine b7fd062637d8fbf4abc28b8dad693dfb13c85f52 https://github.com/spine/spine
35 todomvc dd13679ae101c3a735e23e96bca9622a96a91ad3 https://github.com/tastejs/todomvc

Games

# Project Client Server Revision Git URL
36 2048 538546423f9321a57d8081254cfbaf512e33f82a https://github.com/gabrielecirulli/2048
37 BrowserQuest af32d247cac3495ca430d0effbb88dd5f3250b2c https://github.com/mozilla/BrowserQuest
38 adarkroom 76098f2b59194055bb66e389269634eb0aa172a0 https://github.com/Continuities/adarkroom
39 clumsy-bird f84ecc76d1d9a85a7c27357cc4419dde41e1e986 https://github.com/ellisonleao/clumsy-bird
40 hextris b9dfef0575bc9f324910293e5fa706885af6df36 https://github.com/Hextris/hextris
41 untrusted ccd0718e209e9ed9a2189ae4a736d29a6bcb1b13 https://github.com/AlexNisnevich/untrusted

Web Applications

# Project Client Server Revision Git URL
42 KiwiIRC de1607bf1e9ba4429a10f895a6ec45c54daa3161 https://github.com/prawnsalad/KiwiIRC
43 NodeBB e49dfc72660c0cd453ba1990899f7e393aba22cf https://github.com/NodeBB/NodeBB
44 browsenpm.org eee2c7d70884288f4b0898c46a28b64903bd1df9 https://github.com/nodejitsu/browsenpm.org
45 david-www e324c121e9d6fccb6c46283037d986a05e3f1442 https://github.com/alanshaw/david-www
46 iloveopensource 4ee090e47cab9188a5c7658d478761e29a1eb058 https://github.com/codio/iloveopensource
47 keystone 8e6653d236281ecac477008c6109353159e36ae8 https://github.com/keystonejs/keystone
48 live4chan 96ab6d14a163e7af5be6f86fdf0bf2a3e4a2b01f https://github.com/emgram769/live4chan
49 mash.li d3b81b89f562fbc8db9b4bbdfcfc9c2bc31b42a2 https://github.com/chipersoft/mash.li
50 mediacenterjs 4ec3181f2443ac4be8b164e6bf20e1bd3e405885 https://github.com/jansmolders86/mediacenterjs
51 newww 426e9e3cdd0759c3234f254125aa8261f381cdb8 https://github.com/npm/newww
52 npm-www 87e2ea494ac7c9a0b4c2456db52c84b5f02ce023 https://github.com/npm/npm-www
53 open-marriage fa71d73c8802073e144be0cf045f1105ba1834d5 https://github.com/ericf/open-marriage
54 pencilblue 9d7892f37bf6f540e7a74352f506e2df32867878 https://github.com/pencilblue/pencilblue
55 shields 74f0466c69a8fcc5e5a1f7b8d86e4899d11fc7e1 https://github.com/badges/shields
56 slate ebe63e37a980cc0bfb3d99b4ac02cad74f4d532d https://github.com/slate/slate
57 strider 61a449960d2ca702633e25cd13b7c88569e606b4 https://github.com/Strider-CD/strider

NPM Modules

# Project Client Server Revision Git URL
58 Inquirer.js b273ca70fd8435c7931ecfaae4b481d2edcea335 https://github.com/SBoudrias/Inquirer.js
59 UglifyJS2 f36a1eaa8b5203ab7e4616108c33a0b68668a8db https://github.com/mishoo/UglifyJS2/
60 async 232121251f601c9be7209f6382ee32180df676f1 https://github.com/caolan/async/
61 aws-sdk-js 8e2e66c9c53146ba69e358c73bc2369644bf2fe5 https://github.com/aws/aws-sdk-js
62 backbone 84db2ba31ce92bba95df90eb3bead5cd4cbcc579 https://github.com/jashkenas/backbone
63 bluebird 7b5dfb24e787b2ad42e99fc7559ef595bdacae63 https://github.com/petkaantonov/bluebird
64 body-parser df1d88aaa33c36ccced4a0d0788a0ab92e56875a https://github.com/expressjs/body-parser
65 chalk 0f22588c59f95bece6c206e98566ea097d046c33 git://github.com/sindresorhus/chalk
66 cheerio 9ee6695e4cab6660a80199591044b7e9addf0d5f https://github.com/MatthewMueller/cheerio
67 clean-css 1018f6a332abc5da77adb0799bf22206698a2aa8 https://github.com/GoalSmashers/clean-css
68 cli-color 838bd81de5c993f207706c22c3a447c20e17ac11 https://github.com/medikoo/cli-color
69 coffeescript 485aa8efcf9c02566dc2e38572627eb7947eaf46 git://github.com/jashkenas/coffeescript.git
70 colors.js e9e6557cc0fa26dba1a20b0d45e92de982f4047c https://github.com/Marak/colors.js
71 commander.js 79ea1205ece8f8012aa17ce72754ec6b07d38268 https://github.com/tj/commander.js.git
72 concat-stream 9c14e0eb65c966a307afc4f73e0f905dcdc0a675 http://github.com/maxogden/concat-stream/
73 connect 0509cb8cfa4cd0922076e1c8926b13970d8a6ef5 https://github.com/senchalabs/connect
74 debug 953162b4fa8849268d147f4bac91c737baee55bb git://github.com/visionmedia/debug.git
75 ejs c1b18858de424bb80ab921759ef26061ee6bff5c https://github.com/visionmedia/ejs
76 escodegen fae5989a67bba85a603bdc8042efb6d8f6381d25 https://github.com/Constellation/escodegen/
77 express ac56cf46063e461fbaf53c2c869a1a657e8adbe1 https://github.com/strongloop/express
78 generator 8e2ef622524403e89c538e19299913498265c246 https://github.com/yeoman/generator
79 grunt 09f8fc498b570ef1ed8f8e61a5ef63efef82fc76 git://github.com/gruntjs/grunt.git
80 gulp-util 5ffb34b9e09d7aca85038abf615892b70d2c214e https://github.com/wearefractal/gulp-util/
81 gulp 433c8686ac7abc97e3f70ffa4852712cc2a3c373 https://github.com/gulpjs/gulp
82 hiredis-node 974eec184dc8595b92deee91272185b57e4b24e3 https://github.com/gulpjs/gulp
83 iconv-lite 7374716daa2f2308fff05a9c9a29daf3a346a88f https://github.com/ashtuchkin/iconv-lite
84 inherits 3af5a10c6b51f9e99d9f90394645d7ea630d5eaa https://github.com/isaacs/inherits/
85 jade 87554b38fa6fced3ecfadc0eb0b55757fcbb3dac https://github.com/visionmedia/jade
86 jquery f7e60dc83d81cbf892de9dab39642dd67c49bd23 https://github.com/jquery/jquery.git
87 js-yaml a6cd5657603496c7128d3b9fa621ce9274283194 https://github.com/nodeca/js-yaml
88 jsdom 77990df1527b39e88a877a313c9eb418e377d6cf git://github.com/tmpvar/jsdom.git
89 less.js 59b763a10cb3829389ffa7e0937db9bd54c9b13a https://github.com/less/less.js.git
90 lodash 0d87b7d7207faedcb03ee776bace6a42cbb65d36 https://github.com/lodash/lodash.git
91 log4js-node ec5f4485f85fcac84cee4b1581b36fcac4eeebdc https://github.com/nomiddlename/log4js-node
92 marked 3e02a69921b9b4009d0b17aa1fe0ae2546f96de2 https://github.com/chjj/marked
93 minimatch c2261b9220d9dcc5802d76054a9421fcd5f3fe30 https://github.com/isaacs/minimatch/
94 minimist e2563e462be40c344c6c65b7cde85091fe261976 https://github.com/substack/minimist
95 mocha 7057f7b3324e180fb9a49cbe559bf57ff99d7710 git://github.com/mochajs/mocha.git
96 moment b4ffdd99f842ef5b4ce9d8f531ea6fd0a72114ff https://github.com/moment/moment.git
97 mustache.js 32db14a490ae19455d4367c35f412bd0ecdba17c https://github.com/janl/mustache.js
98 nconf 7b01b46313a10404dad84dcf70853699b0d56aeb https://github.com/flatiron/nconf/
99 ncp e9e6fae157e94e6c55b8edae70779bf12b0d2476 https://github.com/AvianFlu/ncp.git
100 node-browserify 691efa39fd9fb8a2746dba7ffaacef8d82b47085 http://github.com/substack/node-browserify.git
101 node-extend 6ec360b195e965a3b2916e35c1483b14ceb32312 https://github.com/justmoon/node-extend
102 node-fs-extra dcd3cde2b45cb5d0cda137d6768e44d41def3d0e https://github.com/jprichardson/node-fs-extra
103 node-glob 99568bf4819ba80a80314db8a01599d3fae5a02b https://github.com/isaacs/node-glob/
104 node-graceful-fs d3fd03247ccc4fa8a3eee399307fd266c70efb06 https://github.com/isaacs/node-graceful-fs
105 node-http-proxy aba505d159a7dd46e46dd3ceb1b9565308b4d6af https://github.com/nodejitsu/node-http-proxy
106 node-mime f310936307516fed2971b1461b776fa8bbf939d4 https://github.com/broofa/node-mime/
107 node-mkdirp b98bedf92798ed73c87413eaf413d01dbe094a09 https://github.com/substack/node-mkdirp
108 node-mysql 9d7b4868d37a8ede0a14fdd573c1053a6feb1dd8 https://github.com/felixge/node-mysql
109 node-oauth 9e27f4c1287af26ae23ae5e49897e43a8b1de0a2 https://github.com/ciaranj/node-oauth/
110 node-open 2d20d7c8946b2da243a59d9c727953bf06d77611 https://github.com/pwnall/node-open.git
111 node-optimist 680451c01535cf5e1d0a62c22b7d3de6ac8e4011 http://github.com/substack/node-optimist.git
112 node-semver 1a8c3006b7c8c844b6a0c08201b030dfeb50dbde https://github.com/isaacs/node-semver
113 node-uuid 2ea67f5252b5e629c3817df2d302e6037379433b https://github.com/shtylman/node-uuid
114 node-xml2js 8c081138cbc9dae125c0f281be93ef5995eaf851 https://github.com/Leonidas-from-XIV/node-xml2js
115 node_redis de6692774fa563f6fbdd8b6d68d627c5f919bc39 https://github.com/mranney/node_redis
116 nomnom 1f486c53e1a2900570ec9ec523fb4f8efc73e833 https://github.com/harthur/nomnom/
117 nopt 4296f7aba7847c198fea2da594f9e1bec02817ec https://github.com/isaacs/nopt/
118 npm 3f8e2ff8342d327d6f1375437ecf4bd945dc360f https://github.com/npm/npm
119 promise 7710d6d277aeeee2ac5d712a50a386621fc718bb https://github.com/then/promise
120 prompt e1d3df66acfe9de33a573bef1c0a1b18d18cc698 https://github.com/flatiron/prompt/
121 q d6bc41d5556e1526a95b30ed793af13d95f12b6c git://github.com/kriskowal/q.git
122 qs 9250c4cda5102fcf72441445816e6d311fc6813d https://github.com/hapijs/qs.git
123 r.js e764f5c476fa665fd9ac60d82b20ad4b0e65f3fe https://github.com/jrburke/r.js/
124 react e3e0bf5de2aa0b6cb09ec8860a17f9697ceb2a46 https://github.com/facebook/react
125 readable-stream b1036fbc18c4162a7000b90c03ad6071b9c89345 git://github.com/isaacs/readable-stream
126 request da3453532abd5f6f9c742e9d07ff46141c0d4e49 https://github.com/request/request
127 rimraf 6c9968c26e5cd90d3e14803c87aebbc94a6730e2 https://github.com/isaacs/rimraf/
128 shelljs 3611190124f893d689c039a5de69675b0898399d http://github.com/arturadib/shelljs
129 should.js 1f07d245a78c58eb09af37e051b0bf985f61a7f9 https://github.com/shouldjs/should.js.git
130 socket.io-client 6debb043596e84edd851ff6187e6d69f22114be2 https://github.com/Automattic/socket.io-client
131 socket.io 973e6cc9820fcc8e6d5345fa109923739e9e3328 https://github.com/Automattic/socket.io/
132 stylus 7bf5cae588c2d27154356909e3f58170de7e8a42 git://github.com/LearnBoost/stylus
133 superagent 96948c416d3939321425d211f808272059f438b6 https://github.com/visionmedia/superagent
134 through 19ed9b7e84efe7c3e3c8be80f29390b1620e13c0 http://github.com/dominictarr/through
135 through2 0fa42c010aa58b562fd3eabfb53c0bc5ddaa60f7 https://github.com/rvagg/through2.git
136 underscore.string ee72fe2e73ae5399664ef611e9f5607856180477 https://github.com/epeli/underscore.string.git
137 validator.js 422d05423f8c427ec373258229763863a9d24b4c http://github.com/chriso/validator.js
138 winston 4a2f035f6f4b1025c0f73998fcceccfff045439f https://github.com/flatiron/winston.git
139 wrench-js 13f486d867d28fb72947c06f57ce4e5b8e656e86 http://github.com/ryanmcgrath/wrench-js/
140 ws 22ea90f319593d84938c6a498c13ebc244642cb9 https://github.com/einaros/ws
141 xtend 94a95d76154103290533b2c55ffa0fe4be16bfef https://github.com/Raynos/xtend
142 yargs 220048e3243cc900cdb26e779e89b643e986c4b3 https://github.com/chevex/yargs
143 yosay 0c69e56321dc32a13868bd5058cfc1258696be7f git://github.com/yeoman/yosay

Asynchronous APIs

A callback passed into a function can be deferred and invoked at a later time. This deferral happens through known system APIs, which deal with the task queue in common browsers and Node.js environments. Our analysis detects a variety of APIs, including DOM events, network calls, timers, and I/O. Following table lists examples of these asynchronous APIs.

Category Method Client Server
DOM events addEventListener()
onclick
Network calls XMLHTTPRequest.open()
Timers (macro-Task) setImmediate()
setTimeout()
setInterval()
Timers (micro-task) process.nextTick()
I/O fs.rename(oldPath, newPath, callback)
fs.ftruncate(fd, len, callback)
fs.truncate(path, len, callback)
fs.chown(path, uid, gid, callback)
fs.fchown(fd, uid, gid, callback)
fs.lchown(path, uid, gid, callback)
fs.chmod(path, mode, callback)
fs.fchmod(fd, mode, callback)
fs.lchmod(path, mode, callback)
fs.stat(path, callback)
fs.lstat(path, callback)
fs.fstat(fd, callback)
fs.link(srcpath, dstpath, callback)
fs.symlink(srcpath, dstpath[, type], callback)
fs.readlink(path, callback)
fs.realpath(path[, cache], callback)
fs.unlink(path, callback)
fs.rmdir(path, callback)
fs.mkdir(path[, mode], callback)
fs.readdir(path, callback)
fs.close(fd, callback)
fs.open(path, flags[, mode], callback)
fs.utimes(path, atime, mtime, callback)
fs.futimes(fd, atime, mtime, callback)
fs.write(fd, buffer, offset, length[, position], callback)
fs.write(fd, data[, position[, encoding]], callback)
fs.read(fd, buffer, offset, length, position, callback)
fs.readFile(filename[, options], callback)
fs.writeFile(filename, data[, options], callback)
fs.appendFile(filename, data[, options], callback)
fs.watchFile(filename[, options], listener)
fs.unwatchFile(filename[, listener])
fs.watch(filename[, options][, listener])
fs.exists(path, callback)
fs.access(path[, mode], callback)
net.createServer([options][, connectionListener])
net.connect(options[, connectionListener])
net.createConnection(options[, connectionListener])
net.connect(port[, host][, connectListener])
net.createConnection(port[, host][, connectListener])
net.connect(path[, connectListener])
net.createConnection(path[, connectListener])
net.server.listen(port[, host][, backlog][, callback])
net.server.listen(path[, callback])
net.server.listen(handle[, callback])
net.server.listen(options[, callback])
net.server.close([callback])
net.server.getConnections(callback)
child_process.exec(command[, options], callback)
child_process.execFile(file[, args][, options][, callback])
child_process.fork(modulePath[, args][, options])
cluster.disconnect([callback])
crypto.pbkdf2(password, salt, iterations, keylen[, digest], callback)
crypto.randomBytes(size[, callback])
crypto.pseudoRandomBytes(size[, callback])
dns.lookup(hostname[, options], callback)
dns.lookupService(address, port, callback)
dns.resolve(hostname[, rrtype], callback)
dns.resolve4(hostname, callback)
dns.resolve6(hostname, callback)
dns.resolveMx(hostname, callback)
dns.resolveTxt(hostname, callback)
dns.resolveSrv(hostname, callback)
dns.resolveSoa(hostname, callback)
dns.resolveNs(hostname, callback)
dns.resolveCname(hostname, callback)
dns.reverse(ip, callback)
domain.bind(callback)
domain.intercept(callback)
http.createServer([requestListener])
http.Server.listen(port[, hostname][, backlog][, callback])
http.Server.listen(path[, callback])
http.Server.listen(handle[, callback])
http.Server.close([callback])
http.Server.setTimeout(msecs, callback)
http.ServerResponse.setTimeout(msecs, callback)
http.ServerResponse.write(chunk[, encoding][, callback])
http.ServerResponse.end([data][, encoding][, callback])
http.request(options[, callback])
http.get(options[, callback])
http.ClientRequest.write(chunk[, encoding][, callback])
http.ClientRequest.end([data][, encoding][, callback])
http.ClientRequest.setTimeout(timeout[, callback])
http.IncomingMessage.setTimeout(msecs, callback)
https.Server.setTimeout(msecs, callback)
https.createServer(options[, requestListener])
https.Server.listen(port[, host][, backlog][, callback])
https.Server.listen(path[, callback])
https.Server.listen(handle[, callback])
https.Server.close([callback])
https.request(options, callback)
https.get(options, callback)
net.createServer([options][, connectionListener])
net.connect(options[, connectionListener])
net.createConnection(options[, connectionListener])
net.connect(port[, host][, connectListener])
net.createConnection(port[, host][, connectListener])
net.connect(path[, connectListener])
net.createConnection(path[, connectListener])
net.Server.listen(port[, host][, backlog][, callback])
net.Server.listen(path[, callback])
net.Server.listen(handle[, callback])
net.Server.listen(options[, callback])
net.Server.close([callback])
net.Server.getConnections(callback)
net.Socket.connect(port[, host][, connectListener])
net.Socket.connect(path[, connectListener])
net.Socket.write(data[, encoding][, callback])
net.Socket.setTimeout(timeout[, callback])
readline.Interface.question(query, callback)
tls.createServer(options[, secureConnectionListener])
tls.connect(options[, callback])
tls.connect(port[, host][, options][, callback])
tls.Server.listen(port[, host][, callback])
tls.TLSSocket.renegotiate(options, callback)
dgram.createSocket(type[, callback])
dgram.createSocket(options[, callback])
dgram.Socket.send(buf, offset, length, port, address[, callback])
dgram.Socket.bind(port[, address][, callback])
dgram.Socket.bind(options[, callback])