CODEWITHSUNDEEP
pythontensorflowkeras
tryingtolearn
tryingtolearn

Reputation: 2636

Generating data for training in keras

My training set is really quite large. The entire thing takes up about 120GB of RAM and so I can't even generate the numpy.zeros() array to store the data.

From what I've seen, using a generator works well when the entire dataset is already loaded into an array but then is incrementally fed into the network and then deleted afterwards.

Is it alright for the generator to create the arrays, insert the data, load the data into the network, delete the data? Or will that whole process take too long and I should be doing something else?

Thanks

Upvotes: 1

Views: 683

Answers (1)

regina_fallangi
regina_fallangi

Reputation: 2198

You do not need to load the whole data at once, you can load just as much as your batch needs. Check out this answer.

Upvotes: 2

Related Questions