Multiclass Classification model not training properly. Why is the training loss constant?

Question

I am trying to train a model using keras, for multiclass classification. There are 5 classes from which to predict. This is an image classification problem, as mentioned before there are five classes of images, bedroom, bathroom, living room, dining room, and kitchen. The problem is the model doesn't seem to learn, it's always stuck at 20% accuracy and the loss never changes from epoch 1. I'm using the convolutional base from the Xception model with my classifier on top. The train, test, and validation datasets are set up using the tf.data API.

Can someone please point out what I am doing wrong?

This is the dataset generation

train_dir = "House_Dataset/Train"
valid_dir = "House_Dataset/Valid"
test_dir = "House_Dataset/Test"

train_ds = trainAug.flow_from_directory(
        train_dir,
        target_size=(224,224),
        shuffle= False,
        class_mode= "sparse"
)
valid_ds = image_dataset_from_directory(
        valid_dir,
        image_size=(224,224),
        shuffle=False,
        
)

test_ds = image_dataset_from_directory(
        test_dir,
        image_size=(224,224),
        shuffle=False,
        
)

This is the importing of the exception convolution base.

conv_base = keras.applications.Xception(include_top=False, weights="imagenet", input_shape=(224,224,3))
conv_base.trainable = False

This is the model building function.

def pre_trained():
    
    inputs = keras.Input(shape=(224,224,3))
    #x = data_augmentation(inputs)
    x = keras.applications.xception.preprocess_input(inputs)
    x = conv_base(x)
    x = layers.GlobalAveragePooling2D()(x)
    x = layers.BatchNormalization()(x)
    x = layers.Dropout(0.5)(x)
    outputs = layers.Dense(5, activation = "softmax")(x)
    
    
    
    
    model = keras.Model(inputs, outputs)
    
    model.compile(optimizer="rmsprop", loss="sparse_categorical_crossentropy", metrics = ["accuracy"])
    
    return model

Training function call

history = pre_trained_model.fit(train_ds, epochs=25)

This is the picture of the epochs.

Answer 1

try my cnn network and see if you get 87% accuracy. cnn extract features in each layer as a filter. the filter then feeds to the category softmax function.

model=Sequential()

model.add(Conv2D(32, (3,3),activation='relu',input_shape=(IMG_SIZE,IMG_SIZE,3)))
#model.add(Dropout(0.25))
model.add(MaxPooling2D(2))

#model.add(BatchNormalization())
model.add(Conv2D(64, (3,3), activation="relu"))
model.add(MaxPooling2D(2,2))
model.add(Conv2D(128, (3,3), activation="relu"))
model.add(MaxPooling2D(2,2))
model.add(Conv2D(128, (3,3), activation="relu"))
model.add(MaxPooling2D(2,2))
#model.add(BatchNormalization())
model.add(Flatten())
model.add(Dropout(0.2))
model.add(Dense(512,activation='relu'))
model.add(Dense(5, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics = ['accuracy'])
model.summary()

Answer 2

While the exact cause for this remains unclear to me, I have found where the problem occurred, and the solution for it.

I added some parameters in the dataset generator function.

train_dir = "House_Dataset/Train"
valid_dir = "House_Dataset/Valid"
test_dir = "House_Dataset/Test"

train_ds = image_dataset_from_directory(
        train_dir,
        image_size=(224,224),
        shuffle= True,
        seed=1,
        labels="inferred",
        label_mode = "categorical"
)
valid_ds = image_dataset_from_directory(
        valid_dir,
        image_size=(224,224),
        shuffle=True,
        seed=1,
        labels="inferred",
        label_mode = "categorical"
)

test_ds = image_dataset_from_directory(
        test_dir,
        image_size=(224,224),
        shuffle=True,
        seed=1,
        labels="inferred",
        label_mode = "categorical"
        
)

I added the option to shuffle with some seed, and changed the label mode to categorical, which will produce a one-hot encoding of labels. Likewise I also changed the loss from sparse_categorical_crossentropy to categorical_crossentropy. These changes have allowed the model train, and there have been significant improvements in both training and validation loss as well as accuracy.