CODEWITHSUNDEEP
python-3.xoracle-databasejupyter-notebookbigdata
Taimur Islam
Taimur Islam

Reputation: 990

Big Data Load in Pandas Data Frame

As I am new in Big Data Platform, I would like like to do some feature engineering work with my data. The Database size is about 30-50 Gb. Is is possible to load the full data (30-50Gb) in a data frame like pandas data frame?

The Database used here is Oracle. I tried to load it but I am getting out of memory error. Furthermore I like to work in Python.

Upvotes: 0

Views: 378

Answers (1)

codebr
codebr

Reputation: 11

pandas is not good if you have GBS of data it would be better to use distributed architecture to improve speed and efficiency. There is a library called DASK that can load large data and use distributed architecture.

Upvotes: 1

Related Questions