CODEWITHSUNDEEP
javamathneural-networkminimize
Goldenprime
Goldenprime

Reputation: 364

Neural Network Minimize Function

I created a simple neural network; in order to actually train it, I would need to know in which direction the weights and biases need to be tweaked. I've read some articles on the topic, but I'm not exactly great at math and the only thing I understood was that the cost functions (which I managed to get working) need to be minimized. It would be great if someone could at least tell me in theory how this works. If required, I could also post more of the code. The minimize function should in the end replace evolve():

import java.util.Random;

public class Neuron {
Neuron[] input;
float[] weight;
float bias;
Float value = null;
public Neuron(Neuron[] input) {
    this.input = input;
    weight = new float[input.length];
    setRandom();
}
public void setValue(float val) {
    this.value = val;
}
public float getValue() {
    if(this.value == null) {
        return calculate();
    }
    else {
        return this.value;
    }
}
private float calculate() {
    float res = 0;
    for(int i = 0; i < input.length; i++) {
        res += input[i].getValue() * weight[i];
    }
    res -= bias;
    return sigmoid(res);
}
private void setRandom() {
    Random rand = new Random();
    float max = 0;
    for(int i = 0; i < weight.length; i++) {
        weight[i] = rand.nextFloat();
        max += weight[i];
    }
    this.bias = max * 0.8f - rand.nextFloat();
}
public void evolve() {
    Random rand = new Random();
    for(int i = 0; i < weight.length; i++) {
        weight[i] += rand.nextFloat() - 0.5f;
    }
    this.bias += rand.nextFloat() - 0.5f;
}

public static float sigmoid(float x) {
    return (float)(1/( 1 + Math.pow(Math.E,(-1*(double)x))));
  }
}

Upvotes: 0

Views: 164

Answers (1)

Kirtiman Sinha
Kirtiman Sinha

Reputation: 1011

Cost function is basically a function of the difference between the real datapoints and your predictions (i.e. it's your penalty). Say for argument's sake, your neural network is f(x) = 2x + 1. Now, say your observed real datapoint is x = 1, y = 4. Therefore your prediction (f(1)) is 3.

If your cost function is the absolute difference between actual observed value and prediction i.e. |f(x) - y| the value of your cost function is 1 (for x = 1) and you would need to minimize this cost function. However, if your cost function is 100 - |f(x) - y| you would want to maximize it. In this cost function your maximum reward is 100.

So your weights and bias need to move in the direction that would get you closer to minimizing your penalty and maximizing your reward. The closer your prediction is to the observed dataset value, the higher the reward and smaller the penalty.

Notes:

  1. This is a gross oversimplification of the math involved but it should help you get started. Also read about overfitting in machine learning.

  2. For understanding machine learning theory Cross Validated would be better forum.

Upvotes: 1

Related Questions