CODEWITHSUNDEEP
javascriptnode.jsmultithreadingconcurrencyweb-worker
Andreas Gassmann
Andreas Gassmann

Reputation: 6544

Make HTTP request inside Web Worker

I am trying to use web-workers or threads in my node application for the first time. I am using the webworker-threads npm module.

Basically I would like each worker to make requests to a server, measure the response time and send it back to the main thread.

I tried it many different ways, but I just can't seem to get it working. The basic examples from the docs work. But when I try to require a module ("request" in my case), the workers just seem to stop working, without any error messages. I saw in the docs that require doesn't work inside a worker, so I tried "importScripts()", which doesn't work either. When using threadpools I tried to use .all.eval() but it didn't work either.

Since this is the first time working with web-workers / threads in node, I might misunderstand how to use those things in general. Here is one example I tried:

server.js

var Worker = require('webworker-threads').Worker;
var worker = new Worker('worker.js');

worker.js

console.log("before import");
importScripts('./node_modules/request/request.js');
console.log("after import");

This basic example only prints before import and then stops.

Upvotes: 5

Views: 13871

Answers (1)

barczag
barczag

Reputation: 751

Web workers are native javascript only so you can't achieve what you want with them. Worker threads don't support node.js api or npm packages(like http or request.js). For concurrency you don't need any multithread magic just use async.js or promises. If you want to play with threads then child_processes is the way to go. You could also use an API to manage child_processes like https://github.com/rvagg/node-worker-farm

Considering your example you could write something like this:

main.js

var workerFarm = require('worker-farm')
, workers    = workerFarm(require.resolve('./child'))
, ret        = 0;

var urls = ['https://www.google.com', 'http://stackoverflow.com/', 'https://github.com/']; 

urls.forEach(function (url) {
    workers(url, function (err, res, body, responseTime) {
        console.log('Url ' + url + 'finished in ' + responseTime + 'ms');    
        //Ugly code here use async/promise instead
        if (++ret == urls.length)
            workerFarm.end(workers);
    });
});

child.js

var request = require('request');

module.exports = function(url, cb) {
    var start = new Date();
    request(url, function(err, res, body) {
        var responseTime = new Date() - start;
        cb(err, res, body, responseTime);
    });
};

Upvotes: 6

Related Questions