Keras.fit_generator takes more time for epoch

Question

I am doing image classification by using Keras , I have 8k images(input) in training sample and 2k images(input) in test sample , defined epoch as 25 . I noticed that epoch is very slow (approx takes an hour for first iteration) .

can any one suggest how can I overcome this , and what is the reason it takes hell lot of time?

code below..

PART-1
initialise neural network
from keras.models import Sequential

#package to perfom first layer , which is convolution , using 2d as it is for image , for video it will be 3d
from keras.layers import Convolution2D

#to perform max pooling on convolved layer
from keras.layers import MaxPool2D

#to convert the pool feature map into large feature vector, will be input for ANN
from keras.layers import Flatten 

#to add layeres on ANN
from keras.layers import Dense

#STEP -1
#Initializing CNN
classifier = Sequential()

#add convolution layer
classifier.add(Convolution2D(filters=32,kernel_size=(3,3),strides=(1, 1),input_shape= (64,64,3),activation='relu'))

#filters - Number of feature detecters that we are going to apply in image

#kernel_size - dimension of feature detector

#strides moving thru one unit at a time

#input shape - shape of the input image on which we are going to apply filter thru convolution opeation,
#we will have to covert the image into that shape in image preprocessing before feeding it into convolution
#channell 3 for rgb and 1 for bw , and  dimension of pixels

#activation - function we use to avoid non linearity in image

#STEP -2 

#add pooling
#this step will significantly reduce the size of feature map , and makes it easier for computation

classifier.add(MaxPool2D(pool_size=(2,2)))

#pool_size - factor by which to downscale


#STEP -3
#flattern the feature map

classifier.add(Flatten())

#STEP -4 
#hidden layer
classifier.add(Dense(units=128,activation='relu',kernel_initializer='uniform'))

#output layer
classifier.add(Dense(units=1,activation='sigmoid'))


#Compiling the CNN using stochastic gradient descend

classifier.compile(optimizer='adam',loss = 'binary_crossentropy',
                  metrics=['accuracy'])

#loss function should be categorical_crossentrophy if output is more than 2 class

#PART2 - Fitting CNN to image

#copied from keras documentation 

from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory(
        '/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/training_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

test_set = test_datagen.flow_from_directory(
    '/Users/arunramji/Downloads/Sourcefiles/CNN_Imageclassification/Convolutional_Neural_Networks/dataset/test_set',
        target_size=(64, 64),
        batch_size=32,
        class_mode='binary')

classifier.fit_generator(
        training_set,
        steps_per_epoch=8000,   #number of input (image)
        epochs=25,
        validation_data=test_set,
        validation_steps=2000)          # number of training sample

 classifier.fit(
                 training_set,
        steps_per_epoch=8000,   #number of input (image)
        epochs=25,
        validation_data=test_set,
        validation_steps=2000)

Answer 1

You are setting steps_per_epoch to the wrong value (this is why it takes longer than necessary): it is not set to the number of data points. steps_per_epoch should be set to the size of the dataset divided by the batch size, which should be 8000/32 = 250 for your training set, and 63 for your validation set.

Answer 2

Update:

As Matias in his answer pointed out, your steps_per_epoch parameter setting in your fit method led for the huge slowing down per epoch. From the fit_generator documentation:

steps_per_epoch:
Integer. Total number of steps (batches of samples) to yield from generator before declaring one epoch finished and starting the next epoch. It should typically be equal to ceil(num_samples / batch_size) Optional for Sequence: if unspecified, will use the len(generator) as a number of steps.

validation_steps:
Only relevant if validation_data is a generator. Total number of steps (batches of samples) to yield from validation_data generator before stopping at the end of every epoch. It should typically be equal to the number of samples of your validation dataset divided by the batch size. Optional for Sequence: if unspecified, will use the len(validation_data) as a number of steps.

Actually Keras has an inconsistency at handling the two parameters, as fit method raises an Valuerror if you uses a simple dataset instead of datagenerator and set the parameters like batch_size=batch_size, steps_per_epoch=num_samples:

ValueError: Number of samples 60000 is less than samples required for specified batch_size 200 and steps 60000

But when data comes from datagenerator it doesn't handle the same problem letting you to have an issue like the current one.

I made a little example code to check these up.

The fit method with steps_per_epoch=num_samples:

Number of samples: 60000
Number of samples per batch: 200
Train for 60000 steps, validate for 50 steps
Epoch 1/5
263/60000 [..............................] - ETA: 4:07:09 - loss: 0.2882 - accuracy: 0.9116

with ETA (estimated time): 4:07:09,

as this is for 60000 steps, each of 200 samples per batch.

The same fit with steps_per_epoch=num_samples // batch_size:

Number of samples: 60000
Number of samples per batch: 200
Train for 300 steps, validate for 50 steps
Epoch 1/5
28/300 [=>............................] - ETA: 1:15 - loss: 1.0946 - accuracy: 0.6446

with ETA: 1:15

Solution:

steps_per_epoch=(training_set.shape[0] // batch_size)
validation_steps=(validation_set.shape[0] // batch_size)

Further possible issues regarding performance:

As @SajanGohil wrote in his comment train_datagen.flow_from_director make some tasks like file operations, preprocessings before actual traning process which sometimes takes more time as the traning itself.

So to avoid these extratime, you can do the preprocessing task before the whole traning process separately only once. Then you can use these preprocessed data at traning time.

Anyway CNNs with vast images are rather time and resource consuming tasks, which assumes GPU usage for this reason.