CODEWITHSUNDEEP
node.jsexpressfsmulter
Miroslav Saracevic
Miroslav Saracevic

Reputation: 1466

Consume content of the file without saving the file with express.js?

I am writing an application where I have an CVS file upload. I have an CVS parser so from my frontend I can upload the CSV to backend, process it and save it to database. After that I am deleting the file.

I am using multer to accept the file and it get's saved to hard drive, then I can read the file and consume the content of the file and delete the file. All fine and all good there.

I am trying to figure out if there is a way to skip actually saving the file altogether. Is there an easy way of just submitting the file from 'multipart/form-data' form and reading the content directly with express without having to save it on file system?

Just for reference, this is what I have now and it is working as expected

On frontend: 

static fileUpload(file) {
  const url = '/api/statement-upload';
  const formData = new FormData();
  formData.append('file', file);
  const config = {
    headers: {
      'content-type': 'multipart/form-data'
    }
  };
  return post(url, formData, config);
}

On my express server I'm using multer and have defined route like this:

import multer from 'multer';

export default function (router) {
  const upload = multer({dest: 'uploads/'});

  router.use('/api/statement-upload', upload.single('file'), (req, res) => {
    cvsParser(req.file.path)
      .then(() => res.json({error: false}))
      .catch(e => res.json({error: e}));
  });

  return router;
};

And in my cvsParser then I have this:

export default function (filePath) {
  let content = fs.readFileSync(filePath, 'binary');  
  fs.unlink(filePath, () => {});

  // do nasty stuff to content and save it gently in the database
}

So in short again, is there a way to do this without having to resort to saving the CSV to file system before I can consume the content of the file?

Is this even possible considering the file encodings, etc, maybe there is no way around using fs?

Upvotes: 9

Views: 15656

Answers (4)

CrackerKSR
CrackerKSR

Reputation: 1926

using busboy middleware with express.js

app.post('/image-metadata', (req, res, next) => {
        busboy.on('file', async(fieldname, file, filename) => {
          
          try {
            file.on('data', async(data) => {

              // Handle incoming file data chunks
              // const metadata = await sharp(data).metadata()
            });
         ...

Upvotes: 0

Surojit Paul
Surojit Paul

Reputation: 1282

    var csvtojson = require("csvtojson");
    app.post('/', async function (req, res) {
      let sampleFile = req.files.yourUploadedCsvFileName.data.toString();
      // convert it to JSON
      const json_data=await csvtojson().fromString(sampleFile);
      
    })

Upvotes: 0

Shashank Vivek
Shashank Vivek

Reputation: 17514

This is what can be done when you don't want to store the csv in a file system and read the content. In my case, I had to pass the content of csv file as a string to another server (without saving in local system of server).

const multer = require('multer');
const upload = multer({ storage: multer.memoryStorage() })

app.post(
        '/users/bulkUpload',
        upload.single('csvFile'),
        this.usersRoutes.uploadUserData
 );

and in uploadUserData():

uploadUserData = async(
    req: Request,
    res: Response,
    next: any
): Promise<any> => {
    try {
        const options = {
                formData : {
                    'upload': String(req.file.buffer)
                },
                headers: {
                    authorization: req.token,
                    'Content-type': 'multipart/form-data'
                },
                json: true
        };
        const response = await this.http.post.exec(
                '/somePostUrl/',
                options
        );
        return res.status(200).json(response.body);
    }catch (error) {
        return next(error);
    }
}

Here, console.log(String(req.file.buffer)) would show you the content of csv file as a string.

I hope it helps you.

Upvotes: 10

Federico B.
Federico B.

Reputation: 189

With multer library you can store the file uploaded in memory ad use this as a buffer. On Readme file of the project you can find the documentation: github.com/expressjs/multer#memorystorage

Upvotes: 2

Related Questions