CODEWITHSUNDEEP
pythonpython-3.xcsv
Adarsh Ravi
Adarsh Ravi

Reputation: 953

Python: Getting Memory Error for large CSV files

I have the following code snippet and I get the memory error on the last line rows = list(reader)

for file in fileList:
fileName, fileExtension = os.path.splitext(file)
if fileExtension == ".csv":
    with open(path + '\\' + file, "rU") as f:
        reader = csv.reader(f, delimiter=',', dialect="excel")
        rows = list(reader)

Is there any other approach I can use?

Upvotes: 0

Views: 633

Answers (1)

blhsing
blhsing

Reputation: 106435

Since you've now stated in the comments that you simply want to fix the formatting of the rows, you definitely don't need all the rows at once. You should iterate through the csv reader one row at a time, fix the formatting of the row, write the row to another csv file, and then move on to the next row:

with open(path + '\\' + file, "rU") as f, open(path + '\\' + file + '.fixed', "w") as o:
    reader = csv.reader(f, delimiter=',', dialect="excel")
    writer = csv.writer(o, dialect='excel')
    for row in reader:
        # fix the formatting of the row here
        writer.writerow(row)

Upvotes: 2

Related Questions