Wills
Wills

Reputation: 89

MATLAB Neural Network Predictions Converging to 1

I am trying to train a neural network but I am hitting a snag. I have a simplified version of the problem I am facing:

Setup:

NN = trainnet(X_data, Y_data, NN, 'crossentropy', options);

    featureInputLayer(3, "Name", "InputLayer")
    fullyConnectedLayer(32, "Name", "HiddenLayer1","WeightsInitializer", "he") 
    reluLayer("Name", "ReLU")
    dropoutLayer(0.2, "Name", "Dropout") 
    fullyConnectedLayer(1, "Name", "OutputLayer","WeightsInitializer", "glorot")
    sigmoidLayer("Name", "SigmoidOutput") 
];
NN = dlnetwork(NN);
options = trainingOptions("adam", ...
    LearnRateSchedule = "piecewise", ...
    LearnRateDropFactor = 0.2, ...
    LearnRateDropPeriod = 5, ...
    MaxEpochs = 1, ... 
    MiniBatchSize = 128, ...
    ExecutionEnvironment = "cpu", ...
    Plots = "none");

Problem:

When I run the single line NN = trainnet(X_data, Y_data, NN, 'crossentropy', options);, the model runs 194 iterations with the final trainingloss being 0.044545, however the tested accuracy of the model is only ~45% on similar test data, but more concerning is predictions generated with similar data are very skewed towards one. In fact, the lowest prediction is 0.6876 and the mean is 0.9030. This is a massive jump and one that does not make sense to me. I would expect the model's predictions to stay approximately normal about 0.5 (not considering for the moment any learning) or maybe a bit less to match the slighly lower amount of labels of 1. Why is this happening and how can I fix it?

Note: What I am really doing is running a loop that runs the line of code I talked about above. The problem I have descibed is just the first iteration of the loop; as it continues, the skew becomes more extreme until all predictions are just 1.00000. The data is different each time through the loop, however it is very similar and there are trends in it that should be learnable (found from other types of regression modeling).

Upvotes: 0

Views: 24

Answers (0)

Related Questions