Node.js Multer: Intermittent File Upload Failures - Files Not Uploading to Server Randomly

Question

I am facing an intermittent issue with file uploads using Multer in my Node.js application. The issue is that files are not always being uploaded to the server, and the req.files array sometimes contains only partial information about the files.

Description of the Issue: In my application, users upload files through a browser, which are then saved in the state in React.js. The file types can be .pdf, .jpg, .step, .stp, .iges, .stl, etc. The following code is used to handle file selection and save the files in the state:

Note: I am using React DropZone for file uploading

const onSelectFile = async (filesData, ext) => {
  try {
    setIsQuoteChanged(true);
    const filesCollection = [...files];

    for (let file of filesData) {
      const singleValue = {
        selectedFile: file,
        partFileName: file?.path || file?.name,
        ...commandFields
      };

      valueCollection.push(singleValue);

      filesCollection.push({
        ...file,
        isNotFromMongoDB: true
      });
    }

    setFiles(filesCollection);
  } catch (err) {
    //Error handling
  }
};

When it’s time to submit the form, the following code appends the files to formData and makes the API call:

const saveApiCall = (user, updateType) => {
    return new Promise(async (resolve, reject) => {
      try {
        setLoading(true);
 
        let formData = new FormData();
        files.map((val, i) => {
          formData.append('selectedFile', val?.selectedFile || val)
        })

        for (const key in twoDFiles) {
          const value = twoDFiles[key];
          formData.append('twoDFiles', value?.file);
        }

        formData.append('twoDFilesData', JSON.stringify(twoDFiles))
        //other formdata appends.....

        if (_id) {
          const updateResponse = await postApi(`comman/update-quote/${_id}`, formData, token);
          return resolve(updateResponse.data);
        } else {
          const response = await postApi('comman/save-quote', formData, token);
          return resolve(response.data);
        }
      } catch (err) {
        //Error handling
      }
    })
  }


//code for postApi
export const postApi = async (endPoint, payload,token) => {
  return new Promise(async (resolve, reject) => {
    try {
      //code for headers

      let response = await axios.post(`${config.backEnd}/${endPoint}`, payload, { headers: header });
      return resolve(response.data);
    } catch (error) {
      return reject(error);
    }
  });
}

Backend Code: Here's the backend route handling the file upload using Multer:

const multer = require('multer');
const multerIns = multer();
const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'public/uploads/'),
  filename: (req, file, cb) => cb(null, Date.now() + file.originalname),
});
const upload = multer({ storage: storage });


router.post('/save-quote', multerIns.any(), async (req, res) => {
  try {
    // code for userdata
    const response = await CommonController.createQuote(req.body, req.files, userData.userType)
    return res.status(200).send({ status: true, message: 'save data', data: response })
  } catch (err) {
    //error handling
  }
})

Issue: Out of 100%, the code works fine in 95% of situations. However, at certain random moments, the files data is not available in req.files. Instead of receiving the complete file information, I get:

{
  path: <File name>
}

and the files are not available on the server. This issue is intermittent and cannot be replicated at will but is occurring more frequently in production, causing significant business impact.

Environment: Node.js version: v18.14.0 Multer version: ^1.4.5-lts.1 Operating System: Ubuntu 20.04.5 LTS

Steps to Reproduce: Set up a basic Node.js server with the above code. Use an API client (e.g., Postman) to upload files repeatedly. Observe that some files do not appear in the uploads/ directory.

Any insights or suggestions would be greatly appreciated. Thank you!

What I've Tried: Ensured the upload directory exists and has the correct permissions. Checked server logs for any errors (none found). Increased the logging level to debug but still no useful information.

Questions: What could be causing these random failures in file uploads? Are there any known issues with Multer or its configuration that might lead to this behavior? What additional debugging steps can I take to identify the root cause?

Answer 1

Multer with MemoryStore:

Sorry, there is no such missing I could reproduce while running the code below. The code uses MemoryStore as opposed to file store in the other answer. The upload finds so consistent.

server2.js

const express = require('express');
const multer = require('multer');
const { writeFile } = require('fs/promises');

const app = express();
const router = express.Router();
const folder = __dirname + '/public/uploads/';

app.use(express.static('public'));
app.use('/', router);

const storage = multer.memoryStorage();
const upload = multer({ storage: storage });

router.post('/save-quote', upload.any(), async (req, res) => {

  req.files.forEach((file) => {
    writeFile(folder + file.originalname, file.buffer).catch((err) =>
      res.send
        .status(500)
        .send({ status: false, message: 'error saving data', data: null })
    );
  });
  res
    .status(200)
    .send({ status: true, message: 'data saved successfully', data: 'some data' });
});

app.listen(3000, () => console.log('L@3000'));

Test results:

Expectation : 100 files uploaded

Resulted: 
Run 1 : 100 files Uploaded 
Run 2 : 100 files Uploaded
Run 3 : 100 files Uploaded

index.html This file is the same mentioned in the other answer.

<!DOCTYPE html>
<html>
  <head>
    Passing formData to an API gateway
  </head>
  <body>
    <h1>Passing formData to an API gateway</h1>
  </body>
  <script>
    const obj = { greetings: 'Hello' };
    const blob = new Blob([JSON.stringify(obj, null, 2)], {
      type: 'application/json',
    });
    const formData = new FormData();
    for (i = 1; i <= 100; i++) {
      formData.append(i, blob, i);
    }
    // uploading to the Gateway API
    fetch('/save-quote', {
      method: 'POST',
      body: formData,
    }).then((response) => {
      if (!response.ok) {
        throw new Error('Error on uploading file');
      }
    });
  </script>
</html>

Answer 2

Multer with FileStore:

Yes, there is something failing in the following line of code. The upload is consistent when this line is omitted. It is inconsistent when the same line is enabled, as the below code demonstrates.

  filename: (req, file, cb) => cb(null, Date.now() + file.originalname),

Code showing inconsistent upload. This inconsistency is there in every run of this code.

server.js

const express = require('express');
const multer = require('multer');

const multerIns = multer();
const app = express();
const router = express.Router();

// new code to server index.html
app.use(express.static('public'));
app.use('/', router);

const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'public/uploads/'),
  filename: (req, file, cb) => cb(null, Date.now() + file.originalname),
});
const upload = multer({ storage: storage });

router.post('/save-quote', upload.any(), async (req, res) => {
  res
    .status(200)
    .send({ status: true, message: 'save data', data: 'some data' });
});

app.listen(3000, () => console.log('L@3000'));

index.html

<!DOCTYPE html>
<html>
  <head>
    Passing formData to an API gateway
  </head>
  <body>
    <h1>Passing formData to an API gateway</h1>
  </body>
  <script>
    const obj = { greetings: 'Hello' };
    const blob = new Blob([JSON.stringify(obj, null, 2)], {
      type: 'application/json',
    });
    const formData = new FormData();
    for (i = 1; i <= 100; i++) {
      formData.append(i, blob);
    }
    // uploading to the Gateway API
    fetch('/save-quote', {
      method: 'POST',
      body: formData,
    }).then((response) => {
      if (!response.ok) {
        throw new Error('Error on uploading file');
      }
    });
  </script>
</html>

Test results:

Expectation : 100 files uploaded

Resulted: 
Run 1 : 5 files Uploaded 
Run 2 : 3 files Uploaded
Run 3 : 3 files Uploaded

Test results: when the following line is omitted.

// filename: (req, file, cb) => cb(null, Date.now() + file.originalname),


Expectation : 100 files uploaded

Resulted: 
Run 1 : 100 files Uploaded 
Run 2 : 100 files Uploaded
Run 3 : 100 files Uploaded

Solution

It depends upon your exact requirement. As an easy solution I may propose as below. Please use the third argument to append. This would be assigned as the blob content file name. The code below does the same.

Syntax : append(name, value, filename)

for (i = 1; i <= 100; i++) {
  formData.append(i, blob, i);
}

Now on the server, the below code would do the job. Use the original file name as it is passed to append in the frontend.

const storage = multer.diskStorage({
  destination: (req, file, cb) => cb(null, 'public/uploads/'),
  filename: (req, file, cb) => cb(null, file.originalname),
});

Test results

It gives a consistent upload.