CODEWITHSUNDEEP
javascriptnode.jsexpressredis
KM_
KM_

Reputation: 41

Loop through Redis tables, and get same keys in Nodejs and Expressjs

I'm saving different data on different databases on Redis, with the same key. For example at any time this script runs, I have the keys test_1 and test_2 on databases 0...100 in Redis. What I'm trying to do, is for each database, get the keys that have the same name, and save the data to a file. Here's a snippet of what I've tried. 

const redis = require('redis');
const client = redis.createClient();
const fs = require('fs');

for (let i = 1; i <= 100; i++) {
    client.select(i, function(err,res){
        client.mget(['test_1', 'test_2'], function(err,res){
            let results = JSON.parse(res);

            fs.appendFileSync('testfile.json', results);
        });
    });
}

I've also tried wrapping this in an async function and also wrapping await client.mget in an async function aswell, but nothing seems to work.

Upvotes: 0

Views: 511

Answers (1)

Elliot Blackburn
Elliot Blackburn

Reputation: 4174

You can't make async calls like this in a for loop. What you're doing at the moment is telling node to run 100 select statements, it's firing those off and they will all return at different times, then they'll each start their own mget and eventually you might get them attempt to append to the file. This isn't what you want at all, even if it were to work you don't get any reasonable order to your file either.

There are a number of ways you could do this, I'd say the simplest is probably with something like a promise library since you want to fire off a bunch of async requests, aggregate the results, and then write a file.

In this case I'd do something such as the below. Short disclaimer, I've not run the code, but I believe this should roughly work with a couple of tweaks.

const bluebird = require('bluebird');
const redis = require('redis');
const client = redis.createClient();

// Use bluebird to convert client.select and client.mget into
// promise returning functions rather than callback funcs.
// you could do this with the built in node utils, but we need
// some of bluebirds helper functions later on.
const rSelect = bluebird.promisify(client.select);
const rMget = bluebird.promisify(client.mget);

// Get 0..100 array, there are tons of ways to do this...
const dbIds = Array.from(Array(100).keys());

const operations = dbIds.map(id => {
    return rSelect(i)
        .then(res => {
            return rMget(['test_1', 'test_2']);
        });
});

// You now have an array of promises, one for each database id (0..100).
// We can now execute these promises using a bluebird helper.
// Here I'm using map, if you need them done in a specific order you could
// use mapSeries. You could also use reduce if you'd prefer writing it in
// a reducer style, but that doesn't seem to be what you want.
return bluebird.map(operations)
    .then(allResponses => {
        // "allResponses" now contains the response of every operation.
        // lets write those out to a file.
        var result = JSON.parse(allResponses);
        fs.appendFileSync('testfile.json', result);
    })

In the above example I use the Bluebirdjs library, you're free to use any you like but this should get the job done. It's a bit easier to work with bluebird for iterative async processes than callbacks in my experience as it provides some nice helper functions.

Upvotes: 1

Related Questions