CODEWITHSUNDEEP
pythontensorflowkerasconv-neural-network
Obiii
Obiii

Reputation: 824

Accuracy reduced when shuffle set to True in Keras fit_generator

The data I am working with is very imbalanced.

I am training an image classifier using VGG16. I freezed all the layers in VGG16 accept the last two fully connected layers.

BATCH_SIZE = 128

EPOCHS = 80

When I set shuffle = False, the precision and recall for each class is very high (between .80-.90) but when I set shuffle = True, the precision and recall, for each class, drops to 0.10-0.20. I am not sure what is going on. Can some please help?

Below is the code:

img_size = 224
trainGen = trainAug.flow_from_directory(
    trainPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)
valGen = valAug.flow_from_directory(
    valPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)

testGen = valAug.flow_from_directory(
    testPath,
    class_mode="categorical",
    target_size=(img_size, img_size),
    color_mode="rgb",
    shuffle=False,
    batch_size=BATCH_SIZE)

baseModel = VGG16(weights="imagenet", include_top=False,input_tensor=Input(shape=(img_size, img_size, 3)))
headModel = baseModel.output
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(512, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(PFR_NUM_CLASS, activation="softmax")(headModel)
# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)
# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
    layer.trainable = False

The class weights are calculated as:

from sklearn.utils import class_weight
import numpy as np

class_weights = class_weight.compute_class_weight(
               'balanced',
                np.unique(trainGen.classes), 
                trainGen.classes)

These are the class weights:

array([0.18511007, 2.06740331, 1.00321716, 3.53018868, 2.48637874,
       2.27477204, 1.57557895, 6.68214286, 1.04233983, 4.02365591])

and code for training is:

# compile our model (this needs to be done after our setting our layers to being non-trainable
print("[INFO] compiling model...")
opt = SGD(lr=1e-5, momentum=0.8)
model.compile(loss="categorical_crossentropy", optimizer=opt, metrics=["accuracy"])
# train the head of the network for a few epochs (all other layers
# are frozen) -- this will allow the new FC layers to start to become
#initialized with actual "learned" values versus pure random
print("[INFO] training head...")
H = model.fit_generator(
    trainGen,
    steps_per_epoch=totalTrain // BATCH_SIZE,
    validation_data=valGen,
    validation_steps=totalVal // BATCH_SIZE,
    epochs=EPOCHS,
    class_weight=class_weights,
    verbose=1,
    callbacks=callbacks_list)
# reset the testing generator and evaluate the network after
# fine-tuning just the network head

Upvotes: 3

Views: 2803

Answers (1)

Timbus Calin
Timbus Calin

Reputation: 15003

In your case, the problem with setting the shuffle=True is that if you shuffle on your validation set, the results will be chaotic. It happens that the prediction is correct but compared to wrong indices can lead to misleading results, just like it happened in your case.

Always shuffle=True on the training set and shuffle=False on the validation set and test set.

Upvotes: 5

Related Questions