Implementing Cross Validation for CNN model

Question

I have built my CNN model to classify images for 8 classes. The Training and testing steps have been done through randomly splitting 80% for training images and 20% for testing images, where Acuuracy and F-measure results have been calculated.

I have noticed that my training accuracy results are slightly less, compared to my testing results (where I believe that the training accuracy should be higher). After conducting a lot of search, I found two reasons:

1- The use of droput(0.5), which affects the training accuracy results

2-The testing dataset is simple to be classified.

I am planning to evaluate my CNN classifier by conducting 10-k cross validation. However, since I am still new in this area, most of the answers I found were for .csv files, where I have the image files.

How can I code to get the cross validation?

Can I get a confusion matrix for cross validation?

My code:

from keras.models import Sequential
from keras.layers import Conv2D,Activation,MaxPooling2D,Dense,Flatten,Dropout
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from IPython.display import display
import matplotlib.pyplot as plt
from PIL import Image
from sklearn.metrics import classification_report, confusion_matrix
import keras
from keras.layers import BatchNormalization
from keras.optimizers import Adam

classifier = Sequential()
classifier.add(Conv2D(32,(3,3),input_shape=(200,200,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Flatten())
classifier.add(Dense(128))
classifier.add(Activation('relu'))
classifier.add(Dropout(0.5))
classifier.add(Dense(8))
classifier.add(Activation('softmax'))
classifier.summary()
classifier.compile(optimizer =keras.optimizers.Adam(lr=0.001),
                   loss ='categorical_crossentropy',
                   metrics =['accuracy'])
train_datagen = ImageDataGenerator(rescale =1./255,
                                   shear_range =0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip =True)
test_datagen = ImageDataGenerator(rescale = 1./255)

batchsize=10
training_set = train_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/Journal_Paper/Train/',
                                                target_size=(200,200),
                                                batch_size= batchsize,
                                                class_mode='categorical')

test_set = test_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/Journal_Paper/Test/',
                                           target_size = (200,200),
                                           batch_size = batchsize,
                       shuffle=False,
                                           class_mode ='categorical')
history=classifier.fit_generator(training_set,
                        steps_per_epoch = 3067 // batchsize,
                        epochs = 50,
                        validation_data =test_set,
                        validation_steps = 769 // batchsize)


Y_pred = classifier.predict_generator(test_set, steps= 769 // batchsize + 1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
print(confusion_matrix(test_set.classes, y_pred))
print('Classification Report')
target_names = test_set.classes
class_labels = list(test_set.class_indices.keys()) 
target_names = ['coinhive','emotent','fareit', 'flystudio', 'gafgyt','gamarue', 'mirai','razy'] 
report = classification_report(test_set.classes, y_pred, target_names=class_labels)
print(report) 

# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

Answer 1

According to Wikipedia

In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data.

For example, like you want to do 10-fold cross validation, you will

1. Partition the data into ten random, equal parts

2. Train ten separate models with nine subsets of the data,

3. Validate on a different subset of data for each model
4. Find the confusion matrix on each model separately

The code for training and testing will remain the same, just the data input can be taken via either custom generators, or by using SKLearn's KFold.