CODEWITHSUNDEEP
pythontensorflowkerasartificial-intelligence
user7466500
user7466500

Reputation:

Epoch's steps taking too long on GPU

I'm trying to train a model which in my opinion is taking too long compared to other datasets given that it's taking about 1h and 20min to complete one epoch. I think that the problem is because the dataset is not being stored on ram, but I'm not sure of this.

The code is the following:

def load_data():

    train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
    train_generator = train_datagen.flow_from_directory(path1, target_size=(200, 200), batch_size=32, class_mode="binary")

    test_datagen = ImageDataGenerator(rescale=1./255)
    test_generator = test_datagen.flow_from_directory(path2, target_size=(200, 200),batch_size=32, class_mode="binary")

    return train_generator, test_generator

Model:

Fit:

model.fit_generator(x, steps_per_epoch=500, epochs=50, validation_data=y, validation_steps=len(y)/32, callbacks=[tensorboard])

If you could help me I'd appreciate it. Thank you very much!

EDIT: I've done what Matias Valdenegro suggested, even though it's true that the time it takes to complete an epoch is lower, what I realized is that it takes my GPU 10s to complete a step. This is what I really wanted to improve. Sorry for the confusion.

Upvotes: 2

Views: 4449

Answers (1)

Dr. Snoopy
Dr. Snoopy

Reputation: 56357

Your steps_per_epoch are completely wrong, you should set this parameter to number of training images divided by the batch size.

What this parameter does is it tells keras how many batches to pull from the generator in order to declare an epoch. If you set it too high (as you did), then keras is pulling multiple epochs of data before showing you one epoch, which explains the performance issue that you see.

You cannot set this parameter arbitrarily.

Upvotes: 1

Related Questions