Deep learning and neural network

Question

# set the matplotlib backend so figures can be saved in the background
import matplotlib
matplotlib.use("Agg")

# import the necessary packages
from keras.layers.core import Dropout, Activation
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.optimizers import SGD
import matplotlib.pyplot as plt

from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint
from keras.layers import Dense, Conv2D, Flatten
from keras.layers.convolutional import MaxPooling2D

(trainX, testX, trainY, testY) = train_test_split(data,labels, test_size=0.25, random_state=42)

lb = LabelBinarizer()
trainY = lb.fit_transform(trainY)
testY = lb.transform(testY)

#create model
model = Sequential()
#add model layers
model.add(Conv2D(32, kernel_size=3, activation="relu", input_shape=(32,32,3)))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Conv2D(32, kernel_size=3, activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Conv2D(64, kernel_size=3, activation="relu"))
model.add(MaxPooling2D(pool_size=(2,2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation("relu"))
model.add(Dropout(0.5))
model.add(Dense(3, activation="softmax"))

# initialize our initial learning rate and # of epochs to train for
INIT_LR = 0.001
EPOCHS = 500

opt = SGD(lr=INIT_LR, clipvalue=0.5)
model.compile(loss="categorical_crossentropy", optimizer=opt,metrics=["accuracy"])

es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model_500Epoch.h5', monitor='val_accuracy', mode='max', verbose=1, save_best_only=True)

H = model.fit(trainX, trainY, validation_data=(testX, testY),epochs=EPOCHS, batch_size=32,callbacks=[es, mc])

Using following script for prediction.

from keras.models import load_model
import pickle
import cv2
import os
import matplotlib.pyplot as plt
from keras import backend as k


new_model = load_model('model_name.h5')
lb = pickle.loads(open("Label_Binarizer", "rb").read())

dirName = "Other_than_class"
listOfFile = os.listdir(dirName)



# Iterate over all the entries
for entry in listOfFile:
    # Create full path
    fullPath = os.path.join(dirName, entry)
    # If entry is a directory then get the list of files in this 
    directory
    image = cv2.imread(fullPath)
    output = image.copy()
    image = cv2.resize(image, (32, 32))

    # scale the pixel values to [0, 1]
    image = image.astype("float") / 255.0

    # check to see if we should flatten the image and add a batch
    # dimension
    image = image.flatten()
    image = image.reshape((1, image.shape[0]))


    # preds = new_model.predict(image)
    preds = new_model.predict(image.reshape(1, 32, 32, 3))
    print(preds[0])

    k.clear_session()

    # find the class label index with the largest corresponding 
    probability
    i = preds.argmax(axis=1)[0]
    label = lb.classes_[i]


    plt.grid(False)
    plt.imshow(output)
    plt.xlabel("Actual: " + str(entry))
    plt.title("Prediction: " + str(preds[0][i] * 100)+"  "+str(label))
    plt.show()

I have developed model using above architecture for 3-classes cat,dog and flower. it giving good result when i am predicting any unseen image of these classes. but when I am predicting it for house.jpg or laptop.jpg or images other than these 3-classes then also it predicting among these 3-classes which is so disgusting. what's i am doing wrong?

The accuracy of predicting of house.jpg or laptop.jpg is also above 85%. what to do so that it must not predict images out of the classes.

Answer 1

You are doing everything well. If you don't give your network the possibility to choose another class for your laptop.jpg, the job your model will try to do is understanding whether the picture of that laptop has more in common with a cat, a dog or a flower. Let's say it has with a flower, your network may predict the class flower for any laptop picture that you give to it!

Bye :)

Answer 2

Your problem is that your network has only three options (either "cat", "dog" or "flower").

The best way to go here is adding a fourth option ("Unknown").

Therefore you would need to add some random pictures to your training data with the label "Unknown". Those pictures of course must not contain cats, dogs or flowers.

That way your network does not only learn to predict the given objects it also learns to tell if there is none of the known objects in the picture.

A little more generally spoken: You should train your network as close as possible to its practical application.

In your case: why do you have a house.jpg but do not use it for training?

Answer 3

But when I am predicting it for house.jpg or laptop.jpg or images other than these 3-classes then also it predicting among these 3-classes.

This is the normal behaviour because neural network in the last layer

model.add(Dense(3, activation="softmax"))

it returns probabilities for every class from your problem.

So, if you're using a laptop.jpg image maybe it returns three small probabilities and the biggest one it gives the output for you.

Since you're not using laptop images in your training set, then neural network has no ideea about it.

One approach could be setting a threshold probability, let's say 50% and if no one from those 3 probabilities exceeds this threshold, then print Unknown.

With other words, if you are using a softmax distribution for your classification, then you could determine what your baseline max probability is for correctly classified samples, and then infer if a new sample doesn't belong to any of your known classes if its max probability is below some kind of threshold.

This idea comes from a research paper which explains this situation: A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks