keras model with high loss

Question

I have created a Keras model of an Nvidia self-driving car.https://arxiv.org/pdf/1604.07316.pdf But there was a problem the loss is very high and the accuracy is 0.

It's an image dataset of 45000 images that I was using.

I crop the size of the images.

def LoadTrain(batch_size):
global train_batch_pointer
x_out=[]
y_out=[]
for i in range(0, batch_size):
    Image.open(train_xs[(train_batch_pointer + i) % num_train_images])
    # reading width, height
    width, height = im.size
    left, top, right, bottom = 150,height/2, width, height
    im_new = im.crop((left, top, right, bottom))
    img_arr = np.array(im_new)
    x_out.append(img_arr)
    y_out.append([(train_batch_pointer + i) % num_train_images])
return x_out, y_out

X_train, y_train = LoadTrain(len(train_xs))
X_test, y_test = LoadTrain(len(val_xs))


X_train_new = asarray(X_train, dtype=np.float32)
X_test_new = asarray(X_test, dtype=np.float32)
y_train_new = asarray(y_train, dtype=np.float32)
y_test_new = asarray(y_test, dtype=np.float32)

Model

def keras_model(input_data):
model = Sequential()
model.add(Lambda(lambda x: x/255., input_shape=input_data))
model.add(BatchNormalization())
model.add(Conv2D(24, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(36, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(48, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(64,(3,3), activation='relu', strides=(1, 1)))
model.add(Conv2D(64,(3,3), activation='relu', strides=(1, 1)))

model.add(Flatten())
model.add(Dropout(0.3))
model.add(Dense(1164, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1, activation='relu'))
return model

Model call

model = keras_model(X_train_new[0].shape)
keras.utils.plot_model(model, 'generator.png', show_shapes=True)


Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lambda (Lambda)              (None, 128, 305, 3)       0         
_________________________________________________________________
batch_normalization (BatchNo (None, 128, 305, 3)       12        
_________________________________________________________________
conv2d (Conv2D)              (None, 62, 151, 24)       1824      
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 29, 74, 36)        21636     
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 13, 35, 48)        43248     
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 11, 33, 64)        27712     
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 9, 31, 64)         36928     
_________________________________________________________________
flatten (Flatten)            (None, 17856)             0         
_________________________________________________________________
dropout (Dropout)            (None, 17856)             0         
_________________________________________________________________
dense (Dense)                (None, 1164)              20785548  
_________________________________________________________________
dense_1 (Dense)              (None, 100)               116500    
_________________________________________________________________
dense_2 (Dense)              (None, 50)                5050      
_________________________________________________________________
dense_3 (Dense)              (None, 10)                510       
_________________________________________________________________
dense_4 (Dense)              (None, 1)                 11        
=================================================================
Total params: 21,038,979
Trainable params: 21,038,973
Non-trainable params: 6
_________________________________________________________________
None

enter image description here

Compile

adam = Adam(learning_rate=0.0001)
model.compile(optimizer=adam, loss='mean_squared_error', metrics=['Accuracy'])

fit

model.fit(X_train_new, y_train_new, epochs=10, batch_size=64, verbose=1, validation_data=. 
(X_test_new, y_test_new), callbacks = callback) 

Output

Epoch 1/30

568/568 [==============================] 3762s 7s/step - loss: 195265259.8383 - accuracy: 0.0000e+00 - val_loss: 148408736.0000 - val_accuracy: 0.0000e+00

Epoch 2/30

568/568 [==============================] - 2313s 4s/step - loss: 111066534.5659 - accuracy: 0.0000e+00 - val_loss: 197166144.0000 - val_accuracy: 0.0000e+00

I don't know why loss is so high.

Solution There was a problem with the PIL that why loss is very high.I use cv2 to resize the image and the loss is very low.

Answer 1

There are few issues with your code. I'll try to elaborate on all of these below.

TLDR; Remove the last relu, add maxpooling2d and reduce neurons of dense layer with 1164 as it costs you 99.16% of all your training weights in the model is a major bottleneck.

NOTE: As commented by you, the input shape I am using is (160,305,3) -

1. Output Activation is incorrect

Since you are working with a single output regression problem with arbitrary output, you don't want to use a relu but instead, you would want to not have any activation in the final layer. A relu will just return continuous arbitrary values greater than 0 instead of an arbitrary output range.

Check this table for a good reference on what to do for different problem types when it comes to output activations.

2. Metric is not defined correctly

If you were doing a regression problem then you would not use accuracy as a measure since that is for calculating the accuracy of classification. Instead, you would use mse itself or mean absolute error instead for such problems.

---

SIDE NOTE: If you were working on a classification problem, you would have used accuracy and not Accuracy.

THE REASON WHY TENSORFLOW2 DOESNT THROW AN ERROR AND INSTEAD JUST OUTPUTS A 0.00 ACCURACY IS A TENSORFLOW 2 BUG WHICH I HAVE PREVIOUSLY IDENTIFIED, TRACED AND SOLVED ON STACKOVERFLOW.

Check this SO post: Neural network accuracy is always 0.

I traced the error in detail in the tensorflow2 source code and have raised a bug report and made a pull request to fix it as well, which I am awaiting a response on. If interested, do check my explanation on why this occurs.

---

The updated code with these issues fixed should be somewhat like the following -

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Lambda, BatchNormalization, Conv2D, Flatten, Dropout
from tensorflow.keras.optimizers import Adam

input_dim = (160,305,3)

model = Sequential()
model.add(Lambda(lambda x: x/255., input_shape=input_dim))
model.add(BatchNormalization())
model.add(Conv2D(24, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(36, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(48, (5,5), activation='relu', strides=(2, 2)))
model.add(Conv2D(64,(3,3), activation='relu', strides=(1, 1)))
model.add(Conv2D(64,(3,3), activation='relu', strides=(1, 1)))

model.add(Flatten())
model.add(Dropout(0.3))
model.add(Dense(1164, activation='relu'))
model.add(Dense(100, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1))

adam = Adam(learning_rate=0.0001)
model.compile(optimizer=adam, loss='mse')

model.summary()

Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lambda_6 (Lambda)            (None, 160, 305, 3)       0         
_________________________________________________________________
batch_normalization_6 (Batch (None, 160, 305, 3)       12        
_________________________________________________________________
conv2d_30 (Conv2D)           (None, 78, 151, 24)       1824      
_________________________________________________________________
conv2d_31 (Conv2D)           (None, 37, 74, 36)        21636     
_________________________________________________________________
conv2d_32 (Conv2D)           (None, 17, 35, 48)        43248     
_________________________________________________________________
conv2d_33 (Conv2D)           (None, 15, 33, 64)        27712     
_________________________________________________________________
conv2d_34 (Conv2D)           (None, 13, 31, 64)        36928     
_________________________________________________________________
flatten_6 (Flatten)          (None, 25792)             0         
_________________________________________________________________
dropout_6 (Dropout)          (None, 25792)             0         
_________________________________________________________________
dense_30 (Dense)             (None, 1164)              30023052  
_________________________________________________________________
dense_31 (Dense)             (None, 100)               116500    
_________________________________________________________________
dense_32 (Dense)             (None, 50)                5050      
_________________________________________________________________
dense_33 (Dense)             (None, 10)                510       
_________________________________________________________________
dense_34 (Dense)             (None, 1)                 11        
=================================================================
Total params: 30,276,483
Trainable params: 30,276,477
Non-trainable params: 6
_________________________________________________________________

3. Other additions

Few more changes I would recommend you to try out -

• You have a large bottleneck right after the flatten layer. That one layer accounts for 99.16% (30,023,052 / 30,276,477) of all the training parameters in your model. You may want to reduce that. Your best bet is to add the maxpooling2d layers that you have missed out on after each conv2d layer. You could also reduce the dense neurons for that layer as well (1164 is a very large number of neurons, especially since you suddenly go down to 100).
• With so many Dense layers, you could either reduce the number of neurons or reduce the number of layers initially to see how the model is performing. I would recommend increasing the dropout to 0.5 and placing it after or between the dense layers as well.