neuralnet prediction returns the same values for all predictions

Question

I'm trying to build a neural net with the neuralnet package and I'm having some trouble with it. I've been successful with the nnet package but no luck with the neuralnet one. I have read the whole documentation package and can't find the solution, or maybe I'm not able to spot it.

The training command I'm using is

nn<-neuralnet(V15 ~ V1 + V2 + V3 + V4 + V5 + V6 + V7 + V8 + V9 + V10 + V11 + V12 + V13 + V14,data=test.matrix,lifesign="full",lifesign.step=100,hidden=8) 

and for prediction

result<- compute(nn,data.matrix)$net.result

The training takes a whole lot longer than the nnet training. I have tried using the same algorithm as nnet (backpropagation instead of resilent backpropagation) and nothing, changed the activation function too (and the linear.output=F) and pretty much everything else, and the result didn't improved. Predicted values are all the same. I don't understand why the nnet works for me, while the neuralnet one doesn't.

I could really use some help, my (lack of) understanding of both things (neural nets and R) it's probably the cause, but can't find why.

My dataset is from UCI. I want to use the neural network for a binary classification. A sample of the data would be:

25,Private,226802,11th,7,Never-married,Machine-op-inspct,Own-child,Black,Male,0,0,40,United-States,<=50K.
38,Private,89814,HS-grad,9,Married-civ-spouse,Farming-fishing,Husband,White,Male,0,0,50,United-States,<=50K.
28,Local-gov,336951,Assoc-acdm,12,Married-civ-spouse,Protective-serv,Husband,White,Male,0,0,40,United-States,>50K.
44,Private,160323,Some-college,10,Married-civ-spouse,Machine-op-inspct,Husband,Black,Male,7688,0,40,United-States,>50K.
18,?,103497,Some-college,10,Never-married,NA,Own-child,White,Female,0,0,30,United-States,<=50K.
34,Private,198693,10th,6,Never-married,Other-service,Not-in-family,White,Male,0,0,30,United-States,<=50K.
29,?,227026,HS-grad,9,Never-married,?,Unmarried,Black,Male,0,0,40,United-States,<=50K.
63,Self-emp-not-inc,104626,Prof-school,15,Married-civ-spouse,Prof-specialty,Husband,White,Male,3103,0,32,United-States,>50K.
24,Private,369667,Some-college,10,Never-married,Other-service,Unmarried,White,Female,0,0,40,United-States,<=50K.
55,Private,104996,7th-8th,4,Married-civ-spouse,Craft-repair,Husband,White,Male,0,0,10,United-States,<=50K.
65,Private,184454,HS-grad,9,Married-civ-spouse,Machine-op-inspct,Husband,White,Male,6418,0,40,United-States,>50K.
36,Federal-gov,212465,Bachelors,13,Married-civ-spouse,Adm-clerical,Husband,White,Male,0,0,40,United-States,<=50K.
26,Private,82091,HS-grad,9,Never-married,Adm-clerical,Not-in-family,White,Female,0,0,39,United-States,<=50K.

Converted into a matrix, with the factors as numerical values:

V1  V2  V3  V4  V5  V6  V7  V8  V9  V10 V11 V12 V13 V14 V15
39  7   77516   10  13  5   1   2   5   2   2174    0   40  39  0
50  6   83311   10  13  3   4   1   5   2   0   0   13  39  0
38  4   215646  12  9   1   6   2   5   2   0   0   40  39  0
53  4   234721  2   7   3   6   1   3   2   0   0   40  39  0
28  4   338409  10  13  3   10  6   3   1   0   0   40  5   0
37  4   284582  13  14  3   4   6   5   1   0   0   40  39  0
49  4   160187  7   5   4   8   2   3   1   0   0   16  23  0
52  6   209642  12  9   3   4   1   5   2   0   0   45  39  1
31  4   45781   13  14  5   10  2   5   1   14084   0   50  39  1
42  4   159449  10  13  3   4   1   5   2   5178    0   40  39  1
37  4   280464  16  10  3   4   1   3   2   0   0   80  39  1
30  7   141297  10  13  3   10  1   2   2   0   0   40  19  1
23  4   122272  10  13  5   1   4   5   1   0   0   30  39  0

Summary of the predicted values:

      V1           
 Min.   :0.2446871  
 1st Qu.:0.2446871  
 Median :0.2446871  
 Mean   :0.2451587  
 3rd Qu.:0.2446871  
 Max.   :1.0000000  

Value of the Wilcoxon-Mann-Whitney test (area under the curve) shows that the prediction performance is virtualy the same as a random.

performance(predneural,"auc")@y.values
[1] 0.5013319126

Answer 1

If anyone has the same problem, I solved it by using the parameter rep when defining the neural network. It seems that the training of the network is done only once if you don't set this parameter and that leads to the network returning a vector of identical values (or values which are very similar, e.g. 0.99872 and 0.97891).

I believe that the problem could also be in the default value of err.fct parameter, which I set to ce for binary classification.

This is the code which led to normal results:

library(neuralnet)

model <- neuralnet(formula = allow ~ .,
                   data = podaci_train,
                   linear.output = FALSE, 
                   err.fct = "ce",
                   hidden = 4,
                   rep = 3)

predictions <- compute(model, subset(podaci_test, select = -c(allow)))
predictions <- predictions$net.result
max(predictions)
min(predictions)

This is the output (maximum probability - 94.57%, minimum probability - 0.01%):

[1] 0.9456731
[1] 0.0009583263

The usage of rep leads to some weird behavior in RStudio when plotting, because there are multiple models in different training iterations. Therefore, if you don't want to make your environment crash from too much plotting, use an additional parameter:

plot(model, rep = 'best')

Answer 2

I'm adding this here for anyone who might have the same problem I had.

if any of the above didn't work and you're using TensorFlow with a custom training loop. make sure to set training=True as in:

predictions = model(inputs, training=True)

Answer 3

Similar to the answer from @sashkello, I faced a similar issue earlier when my data was not properly normalized. Once I normalized the data everything ran correctly.

Recently, I faced this issue again and after debugging, I found that there can be another reason for neural networks giving the same output. If you have a neural network that has a weight decay term such as that in the RSNNS package, make sure that your decay term is not so large that all weights go to essentially 0.

I was using the caret package for in R. Initially, I was using a decay hyperparameter = 0.01. When I looked at the diagnostics, I saw that the RMSE was being calculated for each fold (of cross validation), but the Rsquared was always NA. In this case all predictions were coming out to the same value.

Once I reduced the decay to a much lower value (1E-5 and lower), I got the expected results.

I hope this helps.

Answer 4

The first reason to consider when you get weird results with neural networks is normalization. Your data must be normalized, otherwise, yes, the training will result in skewed NN which will produce the same outcome all the time, it is a common symptom.

Looking at your data set, there are values >>1 which means they are all treated by NN essentially the same. The reason for it is that the traditionally used response functions are (almost) constant outside some range around 0.

Always normalize your data before feeding it into a neural network.