CODEWITHSUNDEEP
javascriptnode.js
Christopher Carswell
Christopher Carswell

Reputation: 131

Node.js downloaded images corrupted

I'm trying to find a way to download large images (~2MB and upwards) using node.js. I've searched high and low, trying all kinds of solutions and even though the images are downloaded, some are consistently corrupted.

For instance, downloading this sample image using the following code works, but the image is only half downloaded.

var http = require('http')   
var fs = require('fs')   

var options = {
    host:'images.clipartpanda.com',
    port:80,
    path:'http://images.clipartpanda.com/writing-clip-art-xTgn7KXTA.jpeg'
};
    var downloadImage = function (options, fileName) {
        http.get(options, function (res) {
            var imageData = '';
            res.setEncoding('binary');
            res.on('data', function (chunk) {
                imageData += chunk;
            });
            res.on('end', function () {
                fs.writeFile(fileName, imageData, 'binary', function(err){
                    if(err) throw err;
                    console.log('File: ' + fileName + " written!");
                })
            });
        });
    };

downloadImage(options,'writing-clip-art-xTgn7KXTA.jpeg');

I've also tried using npm image-downloader but I'm getting the same results

var image_downloader = require('image-downloader');     
var options = {
    url: 'http://images.clipartpanda.com/writing-clip-art-xTgn7KXTA.jpeg',
    dest: '/home',
    done: function (err, filename, image) {
        if (err) {
            console.log(err);

        } else {
            console.log('File saved to', filename);
        }
    },
};


image_downloader(options);

I'm using an Ubuntu v16.04 Virtual Machine with Node.js v4.2.6.

-UPDATE-

Tried using the following code as per answers & comments below (now using createWriteStream), but file is still being corrupted on download. 

var http = require('http')   
var fs = require('fs')   

var options = {
    host:'images.clipartpanda.com',
    port:80,
    path:'http://images.clipartpanda.com/writing-clip-art-xTgn7KXTA.jpeg'
};
var fileName = 'writing-clip-art-xTgn7KXTA.jpeg';

http.get(options, function (res) {
  var stream = fs.createWriteStream(fileName);
  res.pipe(stream);
});

Every now and then I also receive an error like this: 

Command: node   "/home/test.js" 
events.js:141
      throw er; // Unhandled 'error' event
      ^
Error: read ECONNRESET
    at exports._errnoException (util.js:870:11)
    at TCP.onread (net.js:544:26)
Program exited with code 1

-- UPDATE 2 --

After searching for countless hours to resolve this issue, I think I've figured it out. I had to add {forever: true} to the request in the image-download.js file on line 21. 

request({url: options.url, encoding: null, forever:true}, function (err, response, body) {

It seems that the socket connection kept dying and keepAlive:true needed to be used.

Upvotes: 0

Views: 2064

Answers (1)

rsp
rsp

Reputation: 111336

You are getting binary data but you are appending it to a string.

What you should do is work on Buffers. Here is how you append Buffers:

var newBuffer = Buffer.concat([buffer1, buffer2]);

So instead of appending the data to a string, add every buffer to an array and use Buffer.concat to concatenate all of the buffers at the end. See: Buffer.concat().

Or even better - you can work with streams (thanks to Steven Schobert for pointing it out in the comments). You can do something like this plus some error checking and logging:

http.get(options, function (res) {
  var stream = fs.createWriteStream(fileName);
  res.pipe(stream);
});

Alternatively use a higher-level API like request.

Of course you cannot rule out the possibility that the server may give you incomplete or invalid responses to your requests because you exceed some rate limits or other terms of use.

Upvotes: 1

Related Questions