CODEWITHSUNDEEP
pythonkerasneural-networkdeep-learningfeed-forward
iprof0214
iprof0214

Reputation: 701

Understanding Feed Forward Neural Network Output

I have build a feed forward neural network with 3 hidden layers for regression problem. The metrics I'm using for validation is MAPE. Following are the model parameters

#Define the model
NN_model = Sequential()

# The Input Layer :
NN_model.add(Dense(128, kernel_initializer='normal',input_dim = X_train.shape[1], activation='relu'))

# The Hidden Layers :
NN_model.add(Dense(256, kernel_initializer='normal',activation='relu'))
NN_model.add(Dense(256, kernel_initializer='normal',activation='relu'))
NN_model.add(Dense(256, kernel_initializer='normal',activation='relu'))

# The Output Layer :
NN_model.add(Dense(1, kernel_initializer='normal',activation='linear'))

# Compile the network :
NN_model.compile(loss='mean_absolute_percentage_error', optimizer='adam', metrics=['mean_absolute_percentage_error'])
##NN_model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mean_absolute_error'])
NN_model.summary()

The sample output looks like the following

Train on 18000 samples, validate on 4500 samples
Epoch 1/500
18000/18000 [==============================] - 3s 148us/step - loss: 672.5252 - mean_absolute_percentage_error: 672.5252 - val_loss: 29.3799 - val_mean_absolute_percentage_error: 29.3799

Epoch 00001: val_loss improved from inf to 29.37992, saving model to Weights-001--29.37992.hdf5
Epoch 2/500
18000/18000 [==============================] - 2s 133us/step - loss: 739.9019 - mean_absolute_percentage_error: 739.9019 - val_loss: 220.4918 - val_mean_absolute_percentage_error: 220.4918

Epoch 00002: val_loss did not improve from 29.37992
Epoch 3/500
18000/18000 [==============================] - 2s 129us/step - loss: 840.8005 - mean_absolute_percentage_error: 840.8005 - val_loss: 18.5716 - val_mean_absolute_percentage_error: 18.5716

My question is, in each epoch I see mean absolute percentage error and validation mean absolute percentage error. The later seems to be lower which is what I expect, Why is mean absolute percentage error so different and much higher than validation mean absolute percentage error?

Also, why is validation mean absolute percentage error fluctuating so much?

Appreciate any input.

Upvotes: 0

Views: 185

Answers (2)

jkm
jkm

Reputation: 704

The MAPE may be fairly low-ish because there isn't much spread in terms of input values.

Lets pretend we're talking about classical linear regression to make things easier to visualize. Unfortunately, you happen to be running your linear regression on a nice, classical sigmoid - zero to one, with an inflection point at 0.5, from -inf to inf.

Your best possible effort will be a line defined as 0x+0.5. You simply physically cannot do any better with the model you are using. And yet, your MAPE will be at 50%, not something crazy like 10000%.

That model is utter garbage, of course, but its wrongness is constrained to a fairly small range of values.

Now, with a DNN, it's trickier, because you have all those extra dimensions to worry about, but it might be a decent lead for debugging your model.

As far as fluctuations go, hard to tell ATM - you've just shown the first three epochs, it may still be trying to find its footing on the data. I'd see if the issue persists during further training before making any diagnoses.

Upvotes: 0

pafi
pafi

Reputation: 679

It looks like your model is not able to learn anything at all. Did you check your input-data?

Another problem could be that the gradients are to huge, which leads to a "no convergence behaviour". Therefore your loss is just "randomly" fluctuating.

So check your training data, try to scale it and try to use a different loss function! Hope that helps.

Upvotes: 0

Related Questions