Can't get accuracy in own digit image dataset for mnist

Question

I am new to machine learning and I tried the mnist dataset and I got an accuracy of around 97% but then I tried working on my image dataset and I got an accuracy of 0%. Please help me out.

This is the 97% accuracy model code:

from keras.models import Sequential                  
from keras.layers import Dense, Dropout, Conv2D, Flatten  
from keras.callbacks import ModelCheckpoint               


x_train = tf.keras.utils.normalize(x_train, axis =1)
x_test = tf.keras.utils.normalize(x_test, axis = 1)

model = Sequential()
model.add(Flatten())
model.add(Dense(128, activation = 'relu')) 
model.add(Dense(128, activation = 'relu'))
model.add(Dense(10, activation = 'softmax')) 


model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics = ['accuracy'])

checkpointer = ModelCheckpoint(filepath = 'mnist.model.weights.best.hdf5',verbose = 1,save_best_only = True, monitor = 'loss')

model.fit(x_train, y_train, epochs = 3, callbacks = [checkpointer], 
          batch_size = 32,verbose = 2,shuffle = True)

Now I tried with my 10 images and none of them were predicted correctly. Below is the code:

from skimage import io
from skimage import color
import numpy as np
import tensorflow as tf
import keras

img_0 = color.rgb2gray(io.imread("0.jpg"))
img_2 = color.rgb2gray(io.imread("2.jpg"))
img_3 = color.rgb2gray(io.imread("3.jpg"))
img_4 = color.rgb2gray(io.imread("4.jpg"))
img_5 = color.rgb2gray(io.imread("5.jpg"))
img_6 = color.rgb2gray(io.imread("6.jpg"))
img_7 = color.rgb2gray(io.imread("7.jpg"))
img_8 = color.rgb2gray(io.imread("8.jpg"))
img_9 = color.rgb2gray(io.imread("9.jpg"))
array = [img_0, img_2, img_3, img_4, img_5, img_6, img_7, img_8, img_9]

#normalized the data between 0-1
array = tf.keras.utils.normalize(array, axis = 1)

#used the loop to increase the dimensions of the input layer as 1,28,28 which will be converted into 1*784
for i in array:
    i = np.expand_dims(i,axis = 0)
    print(i.shape)


new_model = tf.keras.models.load_model('mnist_save.model')
new_model.load_weights('mnist.model.weights.best.hdf5')
predictions = new_model.predict(array)

Can you please help me out with my problem.

Answer 1

If I were you, I will check the following three things.

1. Visualize both training and testing data side by side

This is the simplest way to see whether the low performance is reasonable. Basically, if the testing data is very different looking than the training data, there is no way for your pretrained model to achieve high performance in this new testing domain. Even this is not the case, visualization should be helpful to decide what simple domain adaptation can be applied to achieve better performance.

2. Double check with your L2normalization

I took a look at the source code of keras.utils.normalize

@tf_export('keras.utils.normalize')
def normalize(x, axis=-1, order=2):
  """Normalizes a Numpy array.
  Arguments:
      x: Numpy array to normalize.
      axis: axis along which to normalize.
      order: Normalization order (e.g. 2 for L2 norm).
  Returns:
      A normalized copy of the array.
  """
  l2 = np.atleast_1d(np.linalg.norm(x, order, axis))
  l2[l2 == 0] = 1
  return x / np.expand_dims(l2, axis)

Since you are using the tensorflow backend, normalize along the 1st axis means what? Normalize each row? This is strange. The right way to do normalization is to (1) vectorize your input image, i.e. each image becomes a vector; and (2) normalize the resulting vector (at axis=1).

Actually, it is somewhat inappropriate especially when you want to apply a pretrained model in a different domain. This is because L2normalization is more sensitive to nonzero values. In MNIST samples, almost binarized, i.e. either 0s or 1s. However, in a grayscale image, you may meet values in [0,255], which is a completely different distribution.

You may try the simple (0,1) normalization, i.e.

x_normalized = (x-min(x))/(max(x)-min(x))

but this requires you to retrain a new model.

3. Apply domain adaptation techniques

This means you want to do the following things before feeding a testing image to your model (even before normalization).

• binarize your testing image, i.e. convert to 0/1 images
• negate your testing image, i.e. make 0s to 1s and 1s to 0s
• centralize your testing image, i.e. shift your image such that its mass center is the image center.

Of course, what techniques to apply is dependent on the domain differences that you observe in visualization results.