CODEWITHSUNDEEP
javascriptnode.jsjsoncsvparsing
Sergiy B.
Sergiy B.

Reputation: 11

Chalenging nodejs CSV to JSON parsing problem

I'm currently faced with a specific problem. I have to parse CSV file to the JSON, and usually csv-parser works well, however, this time I have received a CSV file with terrible separators: terrible separators

The lines look like

val||"val"||"val"||"val"

The question is how do you think it is possible to go through all CSV lines and change all the "||" to simple comas?

Here is my code for parsing:

const fs = require('fs')
const path = require('path')
const csv = require('csv-parser')

// Read CSV and Write to the JSON

const results = [];

fs.createReadStream('./CSV/users-test.csv')
  .pipe(csv({separator: '||'}))
  .on('data', (data) => results.push(data))
  .on('end', () => {      
      console.log(results);
      fs.writeFileSync('./OutputJSON/data.json', JSON.stringify(results),'utf-8', (err) => {
          if (err) console.log('That is bad')
      })
  });

And here is what I get in the JSON: Broken JSON

It works when all user's data are separated by ',' but I have no idea how to find and replace all that "||" in the CSV file before I parse it.

Upvotes: 1

Views: 358

Answers (1)

O. Jones
O. Jones

Reputation: 108651

csv-parser only allows one-character delimiters. You can preprocess your incoming file into lines with event-stream and then remove the nasty field separators. Something like this ... not debugged.

const fs = require('fs')
const path = require('path')
const csv = require('csv-parser')
const es = require('event-stream')

const results = [];

fs.createReadStream('./CSV/users-test.csv')
  .pipe(es.split())             // line by line
  .pipe(es.replace('||','|')    // turn || into |
  .pipe(csv({separator: '|'}))  // handle CSV with | separator
  .on('data', (data) => results.push(data))
  .on('end', () => {      
      console.log(results);
      fs.writeFileSync('./OutputJSON/data.json', JSON.stringify(results),'utf-8', (err) => {
          if (err) console.log('That is bad')
      })
  });

Upvotes: 1

Related Questions