CODEWITHSUNDEEP
javascriptnode.jsfirebasepromisefirebase-storage
pete
pete

Reputation: 2046

Node.JS: Why do nested promise thens seem to run out of order?

I need each download and getFiles call to run separately because firebase seems unable to handle too many connections. 

const bucket = storage.bucket(bucketName); // firebase storage bucket
var queue = Promise.resolve();

websiteNames.reduce(function(promise, websiteName) {
    return promise
    .then(() => {
        mkdirp(websiteName)
    })
    .then(() => {
        console.log("now getting files: " + dirPath);
        return bucket.getFiles({prefix: dirPath})
    })
    .then(data => {
        console.log("got files");
        var files = data[0];
        files.forEach(function (file) {
            var storePath = tmpDir + "/" + file.name;
            queue = queue
            .then(() => {
                console.log("now downloading: " + file.name);
                return file.download({destination: storePath});
            })
            .then(() => {
                console.log(`downloaded to ${storePath}.`);
                return readFile(storePath, 'utf8');
            })
            .then(buffer => {
                console.log("readed file: " + storePath + " of buf size " + buffer.length);
            });
        });
    });
}, queue);

I expected that the big then block will be adding download actions to a queue which already has all the getFiles actions. So I expect it to make all the getFiles calls, before starting to download the files. But console logs like this: 

now getting files: foo/a
got files
now downloading: foo/a/0.csv
now getting files: foo/b
got files
now getting files: foo/c
got files
downloaded to /tmp/foo/a/0.csv.

Why does foo/a/0.csv start downloading before all the bucket getFiles calls are made? Shouldn't the file download thens have been added to the end of the queue after it already had the getFiles thens? Moreover foo/a has multiple files, so why did only 1 file download sneak in between the getFiles calls?

Upvotes: 0

Views: 81

Answers (1)

Bernd Strehl
Bernd Strehl

Reputation: 2920

The nested structure is the problem. There is no mechanism to prevent the inner loop to promises to the queue, when all the outer ones aren't in the queue yet. As soon as you create the first Promise it will start to "work" and the handlers will be called, so if you aren't done with adding all the Promises, and the first Promise reaches the point where it adds Promises to the queue again, you get what your observed.

Your nested structure is what makes it impossible. Take a look at Promise.all. First build a distinct Array with all the file download Promises. Ask the Array when all the promises have resolved with Promise.all, then attach the then handler to your Promise.all with the inner loop.

Promise.all will have an array of all files, that you can use to perform more steps, by building another distinct queue.

Upvotes: 1

Related Questions