My model has high accuracy and val_accuracy but giving wrong result on test data

Question

I have created some images using opencv and i am running a deep neural network classifier on it. It gives around 97% accuracy and 95% val_accuracy but when i test it, it gives wrong predictions.

Here is my code to create images.

import cv2
import numpy as np
import random
import os
size = 64

def circle(i,d):
    img = np.zeros(shape=(size,size,3))
    point = (random.randint(1,size),random.randint(1,size))
    img = cv2.circle(img,point,random.randint(1,size),(255,255,0),thickness=2,lineType=8)

    if not os.path.exists(d+"/circle"):
        os.makedirs(d+"/circle")
    cv2.imwrite(d+"/circle/"+str(i)+"circle.png",img)
    #print("created circle"+str(i))


def rectangle(i,d):
    img = np.zeros(shape=(size,size,3))
    point = (random.randint(1,size),random.randint(1,size))
    w = random.randint(1,size);
    h = random.randint(1,size);
    point2 = (point[0] + w,point[1]+h)
    img = cv2.rectangle(img,point,point2,(255, 255, 0), 2)
    if not os.path.exists(d+"/react"):
        os.makedirs(d+"/react")
    cv2.imwrite(d+"/react/"+str(i)+"react.png",img)
    #print("created reactangle"+str(i))

def traingle(i,d):
    img = np.zeros(shape=(size,size,3))
    point1 = (random.randint(1,size),random.randint(1,size))
    point2 = (random.randint(1,size),random.randint(1,size))
    point3 = (random.randint(1,size),random.randint(1,size))

    img = cv2.line(img,point1,point2,(255, 255, 0), 2)
    img = cv2.line(img,point2,point3,(255, 255, 0), 2)
    img = cv2.line(img,point3,point1,(255, 255, 0), 2)
    if not os.path.exists(d+"/tra"):
        os.makedirs(d+"/tra")
    cv2.imwrite(d+"/tra/"+str(i)+"tra.png",img)
    #print("created trangle"+str(i))


if not os.path.exists("data_train"):
    os.makedirs('data_train')
for i in range(1,2000):
    circle(i,"data_train")
    rectangle(i,"data_train")
    traingle(i,"data_train")
print("Created test data")   
if not os.path.exists("data_test"):
    os.makedirs('data_test')
for i in range(1,500):
    circle(i,"data_test")
    rectangle(i,"data_test")
    traingle(i,"data_test")

And here is my code for classification.

# importing libraries 
from keras.preprocessing.image import ImageDataGenerator 
from keras.models import Sequential 
from keras.layers import MaxPooling2D,Dropout, Convolution2D
from keras.layers import Flatten, Dense 
from keras import backend as K 


img_width, img_height = 64, 64

train_data_dir = 'data_train'
validation_data_dir = 'data_test'
nb_train_samples = 5997
nb_validation_samples = 1497
epochs = 3
batch_size = 15

if K.image_data_format() == 'channels_first': 
    input_shape = (3, img_width, img_height) 
else: 
    input_shape = (img_width, img_height, 3) 
model = Sequential() 

model.add(Convolution2D(32, 3, 3, input_shape = input_shape,activation="relu")) 
model.add(MaxPooling2D(pool_size =(2, 2))) 

model.add(Convolution2D(32, 3, 3,activation="relu")) 
model.add(MaxPooling2D(pool_size =(2, 2))) 

model.add(Flatten()) 
model.add(Dropout(0.2)) 
model.add(Dense(output_dim=180,activation="relu")) 
model.add(Dropout(0.2)) 
model.add(Dense(3,activation="softmax")) 

model.compile(loss ='categorical_crossentropy', 
                    optimizer ='adam', 
                metrics =['categorical_accuracy']) 

train_datagen = ImageDataGenerator( 
                rescale = 1. / 255, 
                shear_range = 0.2, 
                zoom_range = 0.2, 
            horizontal_flip = False) 

test_datagen = ImageDataGenerator(rescale = 1. / 255) 

train_generator = train_datagen.flow_from_directory(train_data_dir, 
                            target_size =(img_width, img_height), 
                    batch_size = batch_size, class_mode ='categorical') 

validation_generator = test_datagen.flow_from_directory( 
                                    validation_data_dir, 
                target_size =(img_width, img_height), 
        batch_size = batch_size, class_mode ='categorical') 

model.fit_generator(train_generator, 
    steps_per_epoch = nb_train_samples, 
    epochs = epochs, validation_data = validation_generator, 
    validation_steps = nb_validation_samples) 

I have tried 1. Change the number of hidden layer 2. Add dropout layer before final layer and after first layer. 2. Add conv layer.

Any please suggest me what i am doing wrong.

Thanks in advance.

Answer 1

Often this problem is caused by wrong normalization configuration during inference.

In case of overfitting is unlikely to have high validation accuracy unless the validation set choice is really bad and has excessive affinity to training set.

Answer 2

Another probable reason for this issue is overfitting since you are getting high training and validation accuracy The commonly used methodologies are:

Cross- Validation: A standard way to find out-of-sample prediction error is to use 5-fold cross validation. Early Stopping: Its rules provide us the guidance as to how many iterations can be run before learner begins to over-fit. Pruning: Pruning is extensively used while building related models. It simply removes the nodes which add little predictive power for the problem in hand. Regularization: It introduces a cost term for bringing in more features with the objective function. Hence it tries to push the coefficients for many variables to zero and hence reduce cost term.

Answer 3

Most likely reason of this issue is your test set and training set are not from the same sample. This is so common in classification problems. Before training, you should compare the class distributions and the feature distributions of training and test sets. If they are not close to each other, the rules learned from the training set doesn't generalize to test set.

For example a training set class distributions are 70% of class 1, 20% class 2 and 10% class 3. Since the cross-validation comes from the training set, the model has a high training and cross-validation accuracy. However, the model may not perform well if the test set class distributions are like 10% class 1, 20% class 2 and 70% class 3.