CODEWITHSUNDEEP
pythoncsvubuntupandasram
Shoobi
Shoobi

Reputation: 98

memory error while loading csv file?

i have a Ubuntu laptop with 8 GB ram .and also have a 2 GB CSV file but when i use pandas method read_csv to load my data the ram is completely filled while there was 7 GB ram free . how does 2 GB file fill 7 GB ram ?

Upvotes: 2

Views: 5225

Answers (2)

MaxU - stand with Ukraine
MaxU - stand with Ukraine

Reputation: 210812

try to make use of chunksize parameter:

df = pd.concat((chunk for chunk in pd.read_csv('/path/to/file.csv', chunksize=10**4)),
               ignore_index=True)

Upvotes: 0

harshil9968
harshil9968

Reputation: 3244

The reason you get this low_memory warning might be because guessing dtypes for each column is very memory demanding. Pandas tries to determine what dtype to set by analyzing the data in each column.

In case using 32-bit system : Memory errors happens a lot with python when using the 32bit version in Windows. This is because 32bit processes only gets 2GB of memory to play with by default.

Try this : 

tp = pd.read_csv('file_name.csv', header=None, chunksize=1000)
df = pd.concat(tp, ignore_index=True)

Upvotes: 3

Related Questions