CODEWITHSUNDEEP
node.jscsv
hweiu321
hweiu321

Reputation: 123

How to remove duplicate rows of csv file using nodejs

I am trying to look for any simple way or any node packages which provides this functionality.

My csv data is as following:

"date","time1","height1","time2","height2","time3","height3","time4","height4"
"30 Ogos 2021","05:08","175","11:30","93","17:00","148","23:04","88"
"31 Ogos 2021","06:00","160","12:39","104","18:40","132",,
"30 Ogos 2021","05:08","175","11:30","93","17:00","148","23:04","88"
"31 Ogos 2021","06:00","160","12:39","104","18:40","132",,

Different row will have dynamic number of columns.

Upvotes: 0

Views: 1656

Answers (1)

Ashley
Ashley

Reputation: 579

This solution is inefficient, I believe it's O(n^2), but it will work, if we assume that your CSV is represented in javascript as a 2D array of strings (i.e. string[][] in typescript notation).

const table = loadCsv(...); // this line is pseudocode

/**
 * Check if two arrays are the same
 */
function rowCompare(rowA, rowB) {
  if (rowA.length !== rowB.length) return false;
  for (let i = 0; i < rowA.length; i++) {
    if (rowA[i] !== rowB[i]) return false;
  }
  return true;
}

const dedupedTable = table.filter((row, ind, arr) => {
  return ind === arr.findIndex(r => rowCompare(row, r))
});

The dedupedTable variable should now contain your table, with any duplicate rows removed.

Upvotes: 2

Related Questions