Deep learning model fails to predict one specific class among multiple classes

Question

I am attempting to classify a total of 8 classes using the VGG16 network. The classification is conducted using the Stratified 5 Fold Cross Validation method. In this process, there are instances where all samples from the "AFL" class are predicted as the "AFIB" class in each fold. Additionally, there are cases where, except for one sample, all are predicted as "AFIB," and in another scenario, around 60% accuracy is achieved in predicting "AFL."

Figure 1. Fold_1 Confusion matrix

Figure 2. Fold_2 Confusion matrix

Figure 3. Fold_4 Confusion matrix

In summary,

Is there a possibility that, even if a deep learning model makes incorrect predictions, there exists a highly biased scenario where all AFL samples are predicted as the same different class (as shown in Figure 1)? Could this be a result of errors in the code?

*Domain Explanation: AFIB and AFL have similar features, making them challenging to distinguish from each other.

def create_additional_model(input_shape=(150, 530, 1), learning_rate=0.00005):
    additional_model = Sequential()
    additional_model.add(Conv2D(input_shape=input_shape, filters=64, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=64, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding="same"))
    additional_model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding="same"))
    additional_model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding="same"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding="same"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same", activation="relu"))
    additional_model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding="same"))
    additional_model.add(Flatten())
    additional_model.add(Dense(units=512, activation="sigmoid"))
    additional_model.add(Dense(units=256, activation="sigmoid"))
    additional_model.add(layers.Dropout(0.5))
    additional_model.add(Dense(units=8, activation="softmax"))

    optimizer = RMSprop(lr=learning_rate, rho=0.9, epsilon=None, decay=0.0)
    additional_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])

    return additional_model


additional_model = create_additional_model(input_shape=(150, 530, 1), learning_rate=0.00005)

checkpoint = ModelCheckpoint(f'lightBSMgeneration_Resized_(15_x_max_length)x10_Hold_one_out.h5', 
                             monitor='val_accuracy', 
                             save_best_only=True,
                             mode='max', 
                             verbose=1)
    
history=additional_model.fit(X_train, y_train, epochs=40, batch_size=16,validation_data=(X_test, y_test),callbacks=[checkpoint])

Answer 1

As you stated, if AFIB and AFL have very similar feature distributions it is very likely that a very deep network topology is likely to not distinguish perfectly between them, and you may be overfitting to the majority class of those two classes. (But definitely the error is not in your code)

Actually, a shallower architecture may be more accurate for your problem. I suggest you to build a two-step classifier. Join AFIB and AFL in one only class. If the first classifier provides you a prediction of a sample being part of that new class, then you pass the sample to a second classifier only trained on AFIB and AFL samples, but in this case, separated in two different classes.