CODEWITHSUNDEEP
node.jsazureexpressazure-storageasync.js
VladN
VladN

Reputation: 739

Deleting a list of azure blobs blocks express/nodejs

Given a list of blobs I want to delete them one by one (since there's no batch/bulk delete of blobs in azure storage sdk) without blocking node's event loop or the express server. The deletion is triggered by (and runs on) a request. But although the entire flow is made as async as possible, the express server becomes unresponsive once the deletion starts, sometimes some endpoints returning 500.

const async       = require('async');
const azure       = require('azure-storage');
const Promise     = require('bluebird');

const blobService = azure.createBlobService(CONFIG.CONNECTION_STRING);

// Endpoint
app.delete(`${BASE_URL}/files/:date`, deleteFiles);

const deleteFiles = async (req, res) => {

  const date = req.params.date;

  if (!date) {

    return res.status(400).send();

  }

  await deleteBlobs(date).catch((err) => {

    return res.status(500).send(err);

  });

  res.status(200).send();

};

const deleteBlobs = (date) => {

  return new Promise(async (resolve, reject) => {

    // findFiles calls blobService.listBlobsSegmentedWithPrefix and performs some async.map on the returned entrieds
    const blobNames = await findFiles(date)
      .catch(err => log.error('findFiles failed in deleteBlobs.', err));

    async.each(
      blobNames,
      (blobName, callback) => {

        blobService.deleteBlob(CONTAINER, blobName, (deleteError) => {

          if (deleteError) {

            return callback(deleteError);

          }

          callback();

        });

      },
      err => {

        if (err)  {

          return reject(err);

        }

        resolve();

      });

  });

};

So what am I missing? Why the server becomes unresponsive till the deletion finishes?

Mentions (maybe it helps): The express server/app is hosted in Azure on a Basic (B1) service plan.

Upvotes: 0

Views: 1352

Answers (1)

Xiaoning Liu - MSFT
Xiaoning Liu - MSFT

Reputation: 329

I think there are 1 possible reason which could lead your express unresponsive.

Too many delete async operations, which creates lots of callback I/O events at a short time. Node.js may serve these event at first, and stuck the Node.js to deal with other incoming HTTP requests.

Can you use async.eachLimit instead of async.each to set a lower concurrency limitation?

http://caolan.github.io/async/docs.html#eachLimit

Upvotes: 4

Related Questions