Having trouble with reading csv values for a specific column using csv-parser

Question

I am writing an api where I am sending back some data after a get request is being made. I am reading the data from a CSV file and then just sending back the data of a specific column name. The columns I have available are

[
  'YEAR',
  'TICKETS SOLD',
  'TOTAL BOX OFFICE',
  'TOTAL INFLATION ADJUSTED BOX OFFICE',
  'AVERAGE TICKET PRICE',
  ''
]

Here are some sample data from the Kaggle dataset I am using.

YEAR,TICKETS SOLD,TOTAL BOX OFFICE,TOTAL INFLATION ADJUSTED BOX OFFICE,AVERAGE TICKET PRICE,
2021,"42,37,74,881","$3,881,777,912","$3,881,777,912",$9.16,
2020,"22,36,38,958","$2,048,534,616","$2,048,534,616",$9.16,
2019,"1,22,85,41,629","$11,253,443,955","$11,253,444,050",$9.16,

I can read the data fine and can even display the data properly on the client side. Problem occurs when I try to access the YEAR field. For that specific field, I get undefined for all rows. That does not happen with the other fields. When I try to access the other fields, they return the value for each row correctly. I am not sure why this is happening.

This is my app.js (The main file)

const express = require('express');
const app = express();

// Middleware

// Routes
app.use('/', require('./routes/test'));
app.use('/movies', require('./routes/annualTicketSales'));

// Start the server
const port = process.env.PORT || 3000;
app.listen(port);
console.log(`Server Listening at ${port}`);

And this is my route file

const express = require('express');
const route = express.Router();
const csv = require('csv-parser');
const fs = require('fs');

let data = [];
fs.createReadStream('archive/AnnualTicketSales.csv')
    .pipe(csv())
    .on('headers', (headers) => {
        console.log(headers)
    })
    .on('data', async(d) => {
        console.log(d['YEAR']);                    // Returns Undefined for all the values
        console.log(d['AVERAGE TICKET PRICE']);    // Returns the correct value
        data.push(d);
    })
    .on('end', async() => {
        console.log("Successfully Read Data");
    });

route.get('/ticketSalesAnnual', async(req, res) => {
    return res.send(data);
})

module.exports = route;

As you can see, I am using fs and csv-parser to read the data. When I try to do console.log(d['YEAR']);, I get back undefined for each row. But when I use another header value, it returns the correct values for that column. I am not sure why this is happening, but can someone help me with this weird behavior?

Following is what I get for console.log(headers)

[
  'YEAR',
  'TICKETS SOLD',
  'TOTAL BOX OFFICE',
  'TOTAL INFLATION ADJUSTED BOX OFFICE',
  'AVERAGE TICKET PRICE',
  ''
]

The output in the console when I just do console.log(d) is as following

{
  'YEAR': '2021',
  'TICKETS SOLD': '42,37,74,881',
  'TOTAL BOX OFFICE': '$3,881,777,912',
  'TOTAL INFLATION ADJUSTED BOX OFFICE': '$3,881,777,912',
  'AVERAGE TICKET PRICE': '$9.16',
  '': ''
}
{
  'YEAR': '2020',
  'TICKETS SOLD': '22,36,38,958',
  'TOTAL BOX OFFICE': '$2,048,534,616',
  'TOTAL INFLATION ADJUSTED BOX OFFICE': '$2,048,534,616',
  'AVERAGE TICKET PRICE': '$9.16',
  '': ''
}
{
  'YEAR': '2019',
  'TICKETS SOLD': '1,22,85,41,629',
  'TOTAL BOX OFFICE': '$11,253,443,955',
  'TOTAL INFLATION ADJUSTED BOX OFFICE': '$11,253,444,050',
  'AVERAGE TICKET PRICE': '$9.16',
  '': ''
}

Following is the output when I use console.log(d['YEAR])

undefined
undefined
undefined

Following is what I get for console.log(d['AVERAGE TICKET PRICE']);

$9.16
$9.16
$9.16

Answer 1

Mostly it is due to BOM encoding. You can check if BOM encoding is there in file using below command.

file <file-path>

One of the best way to remove BOM encoding using below vi command

vi -c ":set nobomb" -c ":wq" <file-path>

There are other option one can try. Link

Answer 2

I figured it out. The file was encoded with UTF-8 BOM. I had to remove those. It is working now.