CODEWITHSUNDEEP
pythonnumpymathneural-networksigmoid
Practical1
Practical1

Reputation: 678

Why does my neural network give the correct outputs after backpropagation, but doesn't on new inputs?

I've been trying my hand in neural networks, and below is a simple one which uses a sigmoid function to output 1 (or a number REALLY close) if the number is odd, and 0 if the number is even. After training the neural network the output is correct. When new values are introduced the output is zero. Why is the output zero? 

import numpy as np

inputs = np.array([9, 45, 62, 87, 88, 49]) 
outputs = np.array([1, 1, 0, 1, 0, 1]) # 1 for odd 0 for even
weights = np.random.random(()) # weights are initialized as random.
lr = 0.1 # learning rate

mw  = np.dot(inputs, weights)

def sigmoid(x, deriv=False):
  if deriv == True:
    return sigmoid(x) * (1 - sigmoid(x)) # derivative of sigmoid 
  else:
    return 1 / (1 + np.exp(-x))

z = sigmoid(mw)
print("Results before training: {}".format(z)) 
# Results before backpropagation with random value as weight


for x in range(20000): # training loop
  error = (z - outputs)
  adjustments = sigmoid(z, deriv=True) * error * inputs
  weights = weights - lr * adjustments 
# readjusting the weights to minimize error

# After the training loop with the readjusted weights

new_mw = (weights * inputs)  
new_z = sigmoid(new_mw) # sigmoid of new weights * input
print("New results after training:{}".format(new_z)) # -> [1, 1, 0, 1, 0, 1]

def think(x, weights):
    print("New situation: {}".format(x))
    xw = np.dot(x, weights)
    print("New results after thinking: {}".format(sigmoid(xw)))


x = np.array([2, 4, 6, 7, 17, 53]) #array of new test data


think(x, weights) # -> 0.0

Upvotes: 0

Views: 133

Answers (1)

Peteris
Peteris

Reputation: 3325

Impossible solution

There is no possible weight (single! at least in your code) that would result in your function being able to output 1 (or a number REALLY close) if the number is odd, and 0 if the number is even.

It's plausible that during training it learns a relationship that it should output close to 1 for large numbers and 0 for smaller numbers, as that's probably the best it can do with a single multiplicative weight. Looking at your training data, it may be that the boundary is somewhere around 50 or 60, which would result in all your test samples being 0, because they're small-ish - but you could (and perhaps should!) draw a graph for all the values of your learned function from, say, 1 to 100 to illustrate this.

It's worth spending some time thinking why you believe(d) that some weight in f(x) = sigmoid(x * [trainable weight]) would result in a function that distinguishes even numbers from odd numbers. If this is not obvious, plotting the learned function might be informative.

Upvotes: 2

Related Questions