CODEWITHSUNDEEP
tensorflowkerastensorflow2.x
SheppLogan
SheppLogan

Reputation: 352

Keras alternative to ImageDataGenerator for loading arbitrary numpy tensor

The Keras's ImageDataGenerator looks great for simply progressively loading images and passing an iterator to the model.fit function. However, it seems to be only usable for images and for classification tasks.

I want to do regression, i.e., my labels are also arrays of the same shape as the training set ones. In practice, they are multidimensional (>1 channels) arrays like images but they are not images.

Any suggestions on what class to use to simply spit batches of data to a keras model.fit() for training a deep neural net?

The problem, of course, is that my datasets are much too large to fit in memory, which is why I need to use these generators/iterators.

Upvotes: 1

Views: 1943

Answers (2)

Timbus Calin
Timbus Calin

Reputation: 15003

The best solution for your case is to use tf.data.Dataset().

While it may take a relatively short time to accustom to it, it is the recommended way to load your data and use model.fit().

You can consult the documentation here: https://www.tensorflow.org/api_docs/python/tf/data/Dataset

Is is new, fast, beautifully designed and easily extensible.

For instance, for your problem you may want to use tf.data.Dataset.from_tensor_slices(); I will leave you discover its features :D.

Upvotes: 1

Deepanshu Yadav
Deepanshu Yadav

Reputation: 11

A quick solution would be to use Colab whose GPU instance has got 24 GB RAM to work with . You could also reduce your memory when you load the numpy array like the way I did here

Upvotes: 0

Related Questions