CODEWITHSUNDEEP
pythonpandas
Diogo Neiss
Diogo Neiss

Reputation: 117

Pandas read_csv not ignoring commas inside quotted string

I have a exported csv dataset which allows html text from users and I need to turn it into a DataFrame.

The columns with possible extra commas are quotted with ", but the parser is using the commas inside them as separators.

This is the code I'm using, and I've already tried solutions from a github issue and another post here.

pd.read_csv(filePath,sep=',', quotechar='"', error_bad_lines=False)

results in

Result

Here is the csv file itself, with the columns and first entry. csvFile

I don't know what the issue is, quotechar was supposed to work, maybe the extra " inside the quotted string?

Upvotes: 0

Views: 621

Answers (1)

DataSci Live
DataSci Live

Reputation: 28

Here's the issue you're running into:

You set quote (") as your quotechar. Unfortunately, you also have quotes in your text:

<a href ="....">

And so... after that anchor tag, the next few commas are NOT considered inside quotes. Your best bet is probably to remake the original csv file with something else as quotechar (that doesn't appear at all in your text).

Upvotes: 1

Related Questions