NodeJS - How to parse multipart form data without frameworks?

Question

I'm trying to do a basic thing: to send a form using FormData API and parse it in NodeJS.

After searching SO for an hour only to find answers using ExpressJS and other frameworks I think it deserves its own question:

I have this HTML:

<form action="http://foobar/message" method="POST">
  <label for="message">Message to send:</label>
  <input type="text" id="message" name="message">
  <button>Send message</button>
</form>

JS:

var xhr = new XMLHttpRequest();
xhr.open('POST', 'http://foobar/message');
xhr.send(new FormData(form));

In NodeJS I'm doing:

var qs = require('querystring');

var requestBody = '';
request.on('data', function (chunk) {
  requestBody += chunk;
});
request.on('end', function () {
  var data = qs.parse(requestBody);
  console.log(data.message);
});

But in data.message I get the Webkit Boundary thing (from the multipart form data format) instead of the expected message. Is there another built-in lib to parse multipart post data instead of querystring? If not then how to do it manually (high-level, without reading the source code of Express)?

Answer 1

I came here looking for a function that would replace the req.formData method available when using fetch(req) when it's not possible to use fetch. As in the original question here, I am only interested in returned key:value strings. Unlike here though, I want the result to be in the same format as .formData converted with Object.fromEntries(). The accepted answer here plus the getReqData(req) function at https://www.section.io/engineering-education/a-raw-nodejs-rest-api-without-frameworks-such-as-express/ helped me make this (typescript) function that works for me.

function getFormData(request:any) {
  return new Promise<{[key:string]:string}>((resolve, reject) => {
    try {
      const contentTypeHeader = request.headers["content-type"];
      const boundary = "--" + contentTypeHeader.split("; ")[1].replace("boundary=","");
      const body = [] as any;
      request.on('data', (chunk:any) => { body.push(chunk) });
      request.on('end', () => {
        const formDataSubmitted: {[key:string]:string} = {};
        const bodyParts = Buffer.concat(body).toString().split(boundary);
        bodyParts.forEach((val:string) => {
          // After name=.. there are 2 \r\n before the value - that's the only split I want
          // So, the regex below splits at the first occurance of \r\n\r\n, and that's it
          // This way, newlines inside texarea inputs are preserved
          const formDataEntry = returnKeyValObj(
            val.replace("Content-Disposition: form-data; ","").split(/\r\n\r\n(.*)/s)
          );
          if (formDataEntry) Object.assign(formDataSubmitted, formDataEntry);
        })
        if (Object.keys(formDataSubmitted).length) resolve(formDataSubmitted);
      });
    } catch (error) {
      reject(error);
    }
  });
}

function returnKeyValObj(arr:Array<string>){
  if (!Array.isArray(arr) || arr.length < 2) return false;
  let propKey = '';
  const formDataEntries: {[key:string]:string} = {};
  const [pKey, ...pValArray] = arr;
  // pValArray[0] ends with \r\n (2 characters total)
  const propVal = pValArray[0].slice(0,-2)
  // pKey looks like '\r\nname=\"key\"', where \r and \n and \" count as one character each
  // So, need to remove 8 from start of pKey and 1 from end of pKey
  if (pKey && pKey.includes('name=\"')) propKey = pKey.slice(8).slice(0,-1);
  if (propKey) formDataEntries[propKey] = propVal;
  if (Object.keys(formDataEntries).length) return formDataEntries;
  return false;
}

So, instead of

        const data = await req.formData();
        const dataEntries = Object.fromEntries(data.entries());

I can use

      const dataEntries = await getFormData(req);

And, all the rest of my code stays the same.

Answer 2

Hello there i see that you are trying to transfer form data from the frontend to the backend using an http request. Let's do that in an easier and more efficient way: You shall use packetgun-frontend and packetgun-backend npm packages like so (I made those. Don't read their docs it's bad):

In the frontend project folder run the following command

npm install packetgun-frontend

Then write the following in your code (replace ip with the ip of your server as well as port with the port you're listening with)

Html

<!DOCTYPE html>
<html>
    <head>
        <title>Cool form chat website</title>
    </head>
    <body>
        <div id="form">
            <h1 id="title">Enter a message:</h1>
            <input type="text" id="message" placeholder="Hello i am a cool dev">
            <button id="send">Send</button>
        </div>
    </body>
</html>
<!--The path may be a bit different sometimes-->
<script src="node_modules/packetgun-frontend/packetgun-frontend.js"></script>
<script src="./app.js"></script>

Javascript [app.js]

//Get textbox
var message = document.getElementById("message");
//Get button
var send = document.getElementById("send");
var open = function(){
    packetgun.recommended("127.0.0.1:1234", function (client) {
        send.onclick = function () {
            //On button click send form data...
            client.send({
                "exit_code": 0,
                "data": JSON.stringify({
                    "message": message.value
                })
            })
            //Empty textbox
            message.value = "";
        }
        message.onkeydown = function (event) {
            //If pressed enter key while in the textbox
            if (event.keyCode === 13) {
                //Click the button
                send.click();
            }
        }
        client.on("serverDisconnect", function(){
            //On client close restart the process
            open();
        })
    })
}
open()

In your node.js project folder run the following command

npm install packetgun-backend

Then write the following in your code (replace port with the port you want to use):

//Require packetgun
var packetgun = require("packetgun-backend");

//Init packetgun
packetgun.init()
//Listen with recommended method
packetgun.listen.recommended(port, function(client){
    //When receiving data from client
    client.on("data", function(data){
        //Ignore exit_code and client_exit_code just get client data
        data = data.data;
        //Check if data requires preprocessing
        if (typeof data === "string"){
            try{
                //Preprocess data
                data = JSON.parse(data)
            }
            catch (error){
                //Client data is corrupted
                console.error("Client data is corrupted :(");
                client.close();
            }
        }
        //Close client to mimic an http request (disconnect client)
        client.close();
        //data is the JSON object you sent from the frontend side containing the form data, use that data as you whish
        //For exemple let's stringify and log the data
        console.log(JSON.stringify(data, null, 4));
    })
})

I tested this code, and it works!

Answer 3

You Buffer.from POST body to string then .split it by the boundary value provided in the Content-Type: request header. This gives your body parts in an array.

Now process them to determine which is a file and which is a key:val pair. The below code illustrates this.

NodeJS / Server Side Example

const SERVER = http.createServer(async function(request, response) {
  let statusCode = 200;
  if(request.url === '/app') {
    let contentTypeHeader = request.headers["content-type"];
    let boundary = "--" + contentTypeHeader.split("; ")[1].replace("boundary=","");
    if (request.method == 'POST') {
      let body = [];
      request.on('data', chunk => {
        body.push(chunk)
      });
      request.on('end', async () => {
        body = Buffer.concat(body).toString();
        let bodyParts = body.split(boundary);
        let result = [];
        bodyParts.forEach(function(val,index){
          val = val.replace("Content-Disposition: form-data; ","").split(/[\r\n]+/);
          if(isFile(val)){
            result.push(returnFileEntry(val))
          }
          if(isProperty(val)){
            result.push(returnPropertyEntry(val))
          }
        })
        console.log(result)
      });  
      response.end();  
    }
    response.end();

Then the processing functions

function returnPropertyEntry(arr){
  if (!Array.isArray(arr)) {return false};
  let propertyName = '';
  let propertyVal = undefined;
  arr.forEach(function(val,index){
    if(val.includes("name=")){
      propertyName = arr[index].split("name=")[1];
      propertyVal = arr[index + 1]
    }
  })
  return [propertyName,propertyVal];
}

function returnFileEntry(arr){
  if (!Array.isArray(arr)) {return false};
  let fileName = '';
  let file = undefined;
  arr.forEach(function(val,index){
    if(val.includes("filename=")){
      fileName = arr[index].split("filename=")[1];
    }
    if(val.toLowerCase().includes("content-type")){
      file = arr[index + 1];
    }
  })
  return [fileName,file];
}
function isFile(part){
  if(!Array.isArray(part)){return false};
  let filenameFound = false;
  let contentTypeFound = false;
  part.forEach(function(val,index){
    if (val.includes("filename=")){
      filenameFound = true;
    }
    if (val.toLowerCase().includes("content-type")){
      contentTypeFound = true;
    }
  });
  part.forEach(function(val,index){
    if (!val.length){
      part.splice(index,1)
    }
  });
  if(filenameFound && contentTypeFound){
    return part;
  } else {
    return false;
  }
}
function isProperty(part){
  if(!Array.isArray(part)){return false};
  let propertyNameFound = false;
  let filenameFound = false;
  part.forEach(function(val,index){
    if (val.includes("name=")){
      propertyNameFound = true;
    }
  });
  part.forEach(function(val,index){
    if (val.includes("filename=")){
      filenameFound = true;
    }
  });
  part.forEach(function(val,index){
    if (!val.length){
      part.splice(index,1)
    }
  });
  if(propertyNameFound && !filenameFound){
    return part;
  } else {
    return false;
  }
}

Answer 4

I encounter the same problem; because the webserver I am using is written in C++ with a Javascript API (which is not the same as Node.js, although standard compliant). So I have to make my own wheel.

Then I come across this npm module parse-multipart-data. It works, and you can read the source code, it's just one file, and the author explain very clearly what it is, and how to do it.

P.S. as you go higher - you need to go lower. Experience programmer will know what I mean :)