CODEWITHSUNDEEP
javadeep-learningneural-networkneuroph
Pieter Svenson
Pieter Svenson

Reputation: 301

Neuroph API Custom Network Not Configured Correctly (Outputs 0)

I'm trying to build a custom neural network the Neuroph API. In the constructor, I've added all my layers and connected them but, even before I train it, it constantly outputs 0 no matter what the inputs are.

The class is below. Am I missing any steps to prepare the network? Am I correctly calculating the output? If it helps, the learning process doesn't do anything either (doesn't change output or error) so I think it's just that I setup the network wrong.

FYI, I would like to avoid out-of-the-box options if I can because this is part of a research project and I would like to have full autonomy over the characteristics of the network.

Also, Neuroph seems to be a fairly lightweight solution and I don't mind it being slow, but if anyone has any other simple java NN solutions, I would happily take suggestions.

imports ...

public class ScoringNetwork extends NeuralNetwork<LMS> implements Serializable {

  private static final long serialVersionUID = 2L;
  private static final transient int seed = 123456;
  private final transient Lock lock = new ReentrantLock();  // For concurrency
  private final transient Random random = new Random(seed);

  public ScoringNetwork() {
    // create input layer
    NeuronProperties inputNeuronProperties = new NeuronProperties(InputNeuron.class, Linear.class);
    Layer inputLayer = LayerFactory.createLayer(8, inputNeuronProperties);
    inputLayer.addNeuron(new BiasNeuron());
    this.addLayer(inputLayer);

    NeuronProperties hiddenNeuronProperties = new NeuronProperties(TransferFunctionType.RECTIFIED, true);
    Layer hiddenLayer = LayerFactory.createLayer(50, hiddenNeuronProperties);
    hiddenLayer.addNeuron(new BiasNeuron());
    this.addLayer(hiddenLayer);
    ConnectionFactory.fullConnect(inputLayer, hiddenLayer);

    // Create output layer
    NeuronProperties outputNeuronProperties = new NeuronProperties(TransferFunctionType.RECTIFIED, false);
    Layer outputLayer = LayerFactory.createLayer(1, outputNeuronProperties);
    this.addLayer(outputLayer);
    ConnectionFactory.fullConnect(hiddenLayer, outputLayer);

    NeuralNetworkFactory.setDefaultIO(this);

    this.setLearningRule(new LMS());
    this.getLearningRule().setLearningRate(0.1);
    this.getLearningRule().setMaxError(0.1);
    this.getLearningRule().setErrorFunction(new MeanSquaredError());
  }

  // My method to set the inputs, calculate the (single) output, then return the output
  public double calculateOutputs(/* Custom Input Parameters */) {
    lock.lock();
    this.setInput(/* Custom Input Parameters into 8 network input parameters of type double */);
    this.calculate();
    double output = this.getOutput()[0];
    System.out.println("output: " + output);
    lock.unlock();
    return output;
  }

}

Upvotes: 0

Views: 86

Answers (1)

Pieter Svenson
Pieter Svenson

Reputation: 301

Fixed by using a Leaky ReLu function to solve the Dying ReLu problem.

More info

I found my problem: I was using a ReLu function everywhere but gave negative weights for half of the random values, resulting in deactivated nodes everywhere. I believe that if the nodes weren't already deactivated, they would get there within one iteration.

I tried fixing my problem by bringing all my weights to positive values but I found that my error value still was stuck at basically a constant value (after the first iteration, which did show a change in error). So that was a bust!

Finally, I fixed the actual problem by switching to the Leaky ReLu function and returned to a distribution of random weights including negatives. It seems that my network fell prey to the Dying ReLu Problem. Now my network runs like a charm!

Upvotes: 0

Related Questions