CODEWITHSUNDEEP
pythonpandas
David Vilkomerson
David Vilkomerson

Reputation: 11

How big a file will read_csv handle? Works for 10 lines of excel, but not 100000 -- says file doesn't exist

Can't read in a large Excel file using read_csv - python error that file doesn't exist.

Smaller versions of same excel file open easily.

import pandas as pd
data = pd.read_csv("E:\rawdata_50K.csv")
print(data[0:5])

Top 20 lines of excel file load perfectly; the large version does not.

Upvotes: 1

Views: 65

Answers (2)

David Vilkomerson
David Vilkomerson

Reputation: 11

Thanks so much! The 2nd solution in Answer 5 of the 6GB answer worked well and fast....

Trying suggested methods

import pandas as pd Fileread = pd.read_csv("E:\dataraw.csv", chunksize=500) dfList = [] for df in Fileread: dfList.append(df)

df = pd.concat(dfList,sort=False)

print(df[99950:100000])

and perhaps someone can explain why the same CSV file worked when named as dataraw but did NOT work if renamed rawdata...(????)

Upvotes: 0

Trenton McKinney
Trenton McKinney

Reputation: 62403

Note the r in front of the path if using Windows \

data = pd.read_csv(r"E:\rawdata_50K.csv")

or

Note the direction of the / in the path, doesn't require r

data = pd.read_csv("E:/rawdata_50K.csv")

File paths with pathlib:

pathlib

from pathlib import Path

drive_path = Path('E:/')
file_path = drive_path / 'rawdata_50K.csv'
data = pd.read_csv(file_path)

Upvotes: 2

Related Questions