CODEWITHSUNDEEP
node.jsaxios
ThomasDEV
ThomasDEV

Reputation: 95

Downloading many images with node.js + axios stops downloading suddenly after a while with no errors

its that time again when I'm clueless & come humbly to ask for help!

I am trying to download 4500 images at once, average 1mb size, all the images get created & download starts, after about 2gb downloaded (so half) some images are complete, some partial, some empty, task manager confirms the download stops suddenly.

network graph

What could possibly be the issue? No matter how much I wait, nothing happens, at least if I got an error I would try something else...

Please advice if possible, thank you!

//get all json files from a folder
const fs = require("fs");
const path = require("path");
const axios = require("axios");

let urlsArray = [];
const collection = "rebels";
const folder = collection + "_json";

const getFiles = (folder) => {
  const directoryPath = path.join(__dirname, folder);
  return fs.readdirSync(directoryPath);
};

const files = getFiles(folder);

//inside the folder there are json files with metadata
//for each json file parse it and get the image url

files.forEach((file) => {
  const filePath = path.join(__dirname, folder, file);
  const fileContent = fs.readFileSync(filePath, "utf8");
  const parsedJson = JSON.parse(fileContent);
  const imageFromMetadata = parsedJson.image;
  const url = imageFromMetadata.replace("ipfs://", "https://ipfs.io/ipfs/");
  let nr = file.replace(".json", "");
  urlsArray.push({ url, nr });
});

//foreach url create a promise to download with axios

const downloadImage = (url, nr) => {
  const writer = fs.createWriteStream(
    process.cwd() + `/${collection}_images2/${nr}.png`
  );

  return axios({
    url,
    method: "GET",
    responseType: "stream",
  }).then((response) => {
    return new Promise((resolve, reject) => {
      response.data.pipe(writer);
      writer.on("finish", resolve);
      writer.on("error", reject);
    });
  });
};

const promiseAll = async () => {
  const promises = urlsArray.map((data) => {
    console.log(`trying to download image nr ${data.nr} from ${data.url}`);
    return downloadImage(data.url, data.nr);
  });

  await Promise.allSettled(promises);
};

promiseAll();

//download all

Upvotes: 0

Views: 535

Answers (1)

jfriend00
jfriend00

Reputation: 707436

Since Promise.allSettled() never rejects, nothing in your code will report on any rejected promises that it sees. So, I'd suggest you iterate its results and see if you have any rejected promises there.

You can do that like this:

const results = await Promise.allSettled(promises);
console.log(`results.length = ${results.length}`);
for (const r of results) {
    if (r.status === "rejected") {
         console.log(r.reason);
    }
}
console.log("all done");

This will verify that you got through the end of the Promise.allSettled(promises) and will verify that you got non-zero results and will log any rejected promises you got.

Upvotes: 1

Related Questions