CODEWITHSUNDEEP
pythonpickle
Mohamed Emad
Mohamed Emad

Reputation: 150

how to append to a pickle file in batches

So I had this large Dataset of files and I created a program to put them in a pickle file but I Only Have 2GBs of RAM so I can`t Have the entire file in an array so how can I append the array on multiple batches "stuff data inside the array, append to the pickle file, clear the array, repeat " how can I do that,

thanks

Upvotes: 1

Views: 1639

Answers (1)

Felix
Felix

Reputation: 6359

Actually I don't think that it's possible to append data to a pickle file and if if it was, you would run into memory issues when trying to read the pickle file.

Pickle files are not designed for large data storage, so it might be worth switching to another file format.

You could go with text-based formats like csv, json, ... or binary formats like hdf5 which is specifically optimized for large amounts of numerical data.

Upvotes: 2

Related Questions