CODEWITHSUNDEEP
node.jsaxiosfs
onkami
onkami

Reputation: 9441

axios file download and storing leads in a corrupt file (encoding problem)?

I have a typical JPG file that I download using axios and store on a disk like so:

// Just create axios instance that does not care about HTTPS certificate validity
// since I need to download from an untrusted internal service. 
// Included for completeness. Do not borrow this bit unless you know what you are doing. 
const axiosInstance = axios.create({
    httpsAgent: new https.Agent({
        rejectUnauthorized: false
    })
});
// Download file
fileData = await axiosInstance.get(imageUrl, {
    responseType: 'arraybuffer',

});
console.log(`Dowloaded OK, size ${fileData.data.length} bytes`));
// Dump file to disk
await fs.writeFile(`${filename}`, fileData.data, (err) => {}
});

I observe that while console.log reports correct number of bytes and fileData.data looks OK in debugger, the file recorded has twice as many bytes and naturally is corrupted. Seems that non-English letter became 2 bytes instead of one. I suspect it has something to do with the encoding.

How I could possibly correct this so the file is recorded correctly?

Upvotes: 2

Views: 3204

Answers (1)

Anastazy
Anastazy

Reputation: 4776

I guess the problem is how you call writeFile, I suggest you try something along

fs.writeFile(`${filename}`, { encoding: 'binary' }, fileData.data, (err) => {})
// or
fs.writeFile(`${filename}`, Buffer.from(fileData.data), (err) => {})

However, the correct approach is to use streams as the data is incoming, and write directly to disk.

This way the program will use much less memory, and will be time-efficient. Program will not wait for all the data to be downloaded and kept in RAM, and then attempt writing to disk, but will write to disk "on the fly".

Working example:

const axios = require('axios')
const path = require('path')
const fs = require('fs')

const target_path = path.resolve(__dirname, 'some_random_filename')
const input = fs.createWriteStream(target_path, 'binary')

axios
  .get('https://via.placeholder.com/150', {
    responseType: 'stream'
  })
  .then(res => res.data.pipe(input))
  .catch(err => console.error(err))

References:

Upvotes: 6

Related Questions