CODEWITHSUNDEEP
javascriptnode.jsfs
Strontium_99
Strontium_99

Reputation: 1803

Downloading image using Node JS

I'm looping through a number of image uri's to download them locally.

This is the code I'm using:

const optionsImg = {
url: basePath,
   headers: {
       'User-Agent': 'request'
   }
};

let download = function(url, filename, callback) {
   optionsImg.url = url
   request.head(optionsImg, (err, res, body) => {
      console.log('content-type:', res.headers['content-type']);
      console.log('content-length:', res.headers['content-length']);

      request(optionsImg).pipe(fs.createWriteStream(filename)).on('close', callback);
   });
};

But it's only ever saving the last image in the list. What I suspect is happening is it's trying to download async, and the fs.createWriteStream keeps being interrupted, till the list has finished. Therefore only downloading the last image successfully.

The console.logs do show different content-length.

How best to get round this issue? Thanks in advance.

Edit: This is the loop code:

for (let x = 0; x < pics.length; x++) {
    download(pics[x], localPath + x + '.jpeg', function() {
        console.log('done');
    });
}

Upvotes: 0

Views: 582

Answers (1)

RickN
RickN

Reputation: 13500

By the time request.head has finished—which, because it is a network request it might as well be "an eternity" as far as a computer is concerned—optionsImg.url will always have the same value because you keep re-using the same object.

Consider this:

const obj = { a: undefined };
for(let i = 0; i <= 3; i++) { 
  obj.a = i; 
  setTimeout(console.log, 100, obj);   
}
// will log {a:3} always

You'd fix the above problem by cloning the object:

const obj = { a: undefined };
for(let i = 0; i <= 3; i++) { 
  const obj2 = Object.assign({}, obj);
  obj2.a = i; 
  // or const obj2 = { ...obj, a: i };
  setTimeout(console.log, 100, obj2);   
}
// will log {a:0}, {a:1}, {a:2}, {a:3}

And some goes for your code:

let download = function(url, filename, callback) {
   const options = {
       ...optionsImg,
       url: url,
   }
   /* alternatively you can do:
   const options = Object.assign({}, optionsImg);
   options.url = url; */
   request.head(options, (err, res, body) => {
      console.log('content-type:', res.headers['content-type']);
      console.log('content-length:', res.headers['content-length']);

      request(options).pipe(fs.createWriteStream(filename)).on('close', callback);
   });
};

Object.assign and {...optionsImg} do a shallow copy, so headers would still be re-used.

Now every time download() is called it and the request() calls it triggers will have 'its own' copy of the options object.

Edit: you don't need this for the stated problem in the OP, but should you also want to copy the headers object—perhaps you want to manipulate its value for every request—then you could do:

const options = {
  ...optionsImg,
  headers: { 
    ...optionsImg.headers,
    // as an example:
    'X-this-is-request-number': `${x} out of ${pics.length}`,
  },
  url: url,
};

You could also search for a recursive object cloning utility; there are many.

Upvotes: 2

Related Questions