CODEWITHSUNDEEP
pythontensorflowkeras
mmrbulbul
mmrbulbul

Reputation: 400

Why does Keras ImageDataGenerator throwing Memory Error?

I am building a simple "Cat vs Dog Classifier" in Keras. While fitting the ImageDataGenerator I am getting MemoryError. My Code looks like this: 

from keras.preprocessing.image import ImageDataGenerator

image_gen = ImageDataGenerator(shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True) 

image_gen.fit(X)

X has a shape of (25000,150,150,3)

What am I doing wrong or how to fix this?

I have already checked this and this.

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-10-2fd88662a693> in <module>
----> 1 image_gen.fit(X)

/opt/conda/lib/python3.6/site-packages/keras_preprocessing/image/image_data_generator.py in fit(self, x, augment, rounds, seed)
    943             np.random.seed(seed)
    944 
--> 945         x = np.copy(x)
    946         if augment:
    947             ax = np.zeros(

/opt/conda/lib/python3.6/site-packages/numpy/lib/function_base.py in copy(a, order)
    790 
    791     """
--> 792     return array(a, order=order, copy=True)
    793 
    794 # Basic operations

MemoryError:

Upvotes: 1

Views: 1348

Answers (1)

lbragile
lbragile

Reputation: 8122

You are using data augmentation in the generator, which essentially triples the number of images you have. Most likely your computer cannot handle 75k images in memory (due to low RAM especially GPU RAM). Your choices are either to reduce the image sizes, reduce the augmentation, or have the data-generator read your images from folders without storing them in memory (by batch).

As shown here it will look like this:

train_datagen = ImageDataGenerator(shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

# Change to match your problem
model.fit_generator(
        train_generator,
        steps_per_epoch=2000,
        epochs=50,
        validation_data=validation_generator,
        validation_steps=800)

Upvotes: 1

Related Questions