CODEWITHSUNDEEP
javascriptnode.jsasynchronous
user2255700
user2255700

Reputation: 113

Need assistance with an issue due to async nature of Node

I am having trouble figuring out how to accomplish what I am trying given the asynchronous nature of Node. On a conceptual level I want to do this:

  1. Use the Twitter API to run a search
  2. Iterate over the resulting tweets to run a sentiment check using alchemylab's api - each tweet with its associated sentiment info is pushed onto an array
  3. Render the page which can then loop through the array displaying the tweets and sentiment info

Here is the code I have:

    client.get('search/tweets', {q: 'node.js'}, function(error, tweets, response)
    {
        for (var i=0; i<tweets.statuses.length; i++){
           var ttext = tweets.statuses[i].text;
           var tsrc = tweets.statuses[i].source;
           var lang = '';
           var sentiment = '';
           var score = '';

           var tweet = {
                         text: tweets.statuses[i].text,
                         source: tweets.statuses[i].source
                       };
          
           alchemyapi.sentiment("text", ttext, {}, function(response) {
 
              if (response.status == 'OK' ){
                tweet.lang = response.language;
                tweet.sentiment = response['docSentiment'].type;
                tweet.score = response['docSentiment'].score;
                console.log("pushing: " + util.inspect(tweet) );
                tweet_data.push( tweet );
              }

           });

        }

        

        res.render('twitter', {
            myText: myText,
        });
    });

What's happening is the page is being rendered before any of the alchemyapi calls have completed. I am fairly new to thinking in async terms and haven't wrapped my head fully around it yet.

Any help would be greatly appreciated!

Thanks, Andy

Edit -

I am trying to implement the deferred approach using Promises and not having much luck.. I have parseTweet defined now as a function as you suggested, if the response is 'OK' it calls resolve(tweet) - not really sure if this is doing anything. My for() loop now looks like this:

            promises.push(parseTweet(tweets.statuses[i]));

            Promise.all(promises).then(function(){
                console.log('all promises complete');
            }, function(){
                console.log('error occurred with promises');
            });

I have comments in the parseTweet function and I see all of the tweets being parsed but not inside the Promise.all block - and the program never completes, keeps running until I kill it...

Upvotes: 0

Views: 99

Answers (2)

Jose Mato
Jose Mato

Reputation: 2799

Ok, function "alchemyapi.sentiment" is async and you are calling a lot of times, exactly "tweets.statuses.length", and, at the end of your code you are calling function "res.send" so you call this function before "for loop" end, so, this is my approach to the solution (pseudocode):

1) Create a function that receive a tweet, create the tweet data (your logic + alchemyapi.sentiment). This function return a javascript promise (in my example I will use jquery defer to write less code, but you can use native promise):

function parseTweet(tweet) {
        $promise = $.defer();
        var tweetDataObject = {}; // this is not important, put your required data

        // call alchemyapi async
        alchemyapi.sentiment("text", ttext, {}, function(response) {
            if (response.status == 'OK' ) {
                // parse data and "return" valid object
                $promise.resolve(tweetDataObject);
            } else { // data is not valid, return null
                $promise.reject(null); // null, empty object, what you need
            }
        }); 

        return $promise;
    }

2) On your main method, do the for loop, call function parseTweet (that return something like promise) and wait for ALL promise:

var promises = [];
for (var i=0; i<tweets.statuses.length; i++) {
    promises.push(parseTweet(tweets.statuses[i]));

    $.when(promises).then(function() {
        // here, on "arguments" you have array of objects that parseTweet "return" when called resolve or reject

        console.log(arguments);
    });
}

I wrote the example using jquery defer, but it's easy with native promises (specially at this case that we have a lot number of promises to wait on $.when function).

Please, read carefully this article: http://www.html5rocks.com/en/tutorials/es6/promises/

UPDATED: Example

I did an example on github. If you have doubts let me know, but I think that it's very clear https://github.com/josemato/stackoverflow/tree/master/es-promise-all-no-order

Upvotes: 1

duffymo
duffymo

Reputation: 308753

It'd be easy with vert.x: when the request event comes in, defer it to a worker module and let it do the long running processing. Register a handler that processes the "all finished" event for display.

Maybe that process works for node.js as well: Make the long running alchemy calls a separate thread that registers an "all finished" event when it's done. Ask the UI to register itself with the callback to render when it receives that event.

Upvotes: 0

Related Questions