How to teach keras neural network to solve sqrt

Question

I'm learning machine learning with python and keras. I created a neural network to predict square root from even integers from range {1, 4, 9, 16, 25, 36, ..., 100}. I've written code to do that, but the results are far from true (whatever number I will provide to the network, it predicts that it's 1.0).

I've tried changing the number of layers, number of neurons in every layer, activation functions, but nothing helped.

Here is the code I've written so far:

from numpy import loadtxt
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers

# laod dataset
# dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
dataset = loadtxt('sqrt.csv', delimiter=',')

# split into input (X) and output (y) variables
X = dataset[:,0:1] * 1.0
y = dataset[:,1] * 1.0

# define the keras model
model = Sequential()
model.add(Dense(6, input_dim=1, activation='relu'))
model.add(Dense(1, activation='linear'))

# compile the keras model
opt = optimizers.adam(lr=0.01)
model.compile(loss='mean_squared_error', optimizer=opt, metrics=['accuracy'])

# fit the keras model on the dataset (CPU)
model.fit(X, y, epochs=150, batch_size=10, verbose=0)

# evaluate the keras model
_, accuracy = model.evaluate(X, y, verbose=0)
print('Accuracy: %.2f' % (accuracy*100))

# make class predictions with the model
predicitions = model.predict_classes(X)

# summarize the first 10 cases
for i in range(10):
    print('%s => %.2f (expected %.2f)' % (X[i].tolist(), predicitions[i], y[i]))

And here is the dataset:

1,1
4,2
9,3
16,4
25,5
36,6
49,7
64,8
81,9
100,10

When I run this network I get the following results:

[1.0] => 0.00 (expected 1.00)
[4.0] => 0.00 (expected 2.00)
[9.0] => 1.00 (expected 3.00)
[16.0] => 1.00 (expected 4.00)
[25.0] => 1.00 (expected 5.00)
[36.0] => 1.00 (expected 6.00)
[49.0] => 1.00 (expected 7.00)
[64.0] => 1.00 (expected 8.00)
[81.0] => 1.00 (expected 9.00)
[100.0] => 1.00 (expected 10.00)

What am I doing wrong?

Answer 1

This is a regression problem. So you should use model.predict() instead of model.predict_classes().

Dataset is not large enough too. However, you can get some sensible predictions with the following code.

from numpy import loadtxt
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers

# laod dataset
# dataset = loadtxt('pima-indians-diabetes.csv', delimiter=',')
dataset = loadtxt('sqrt.csv', delimiter=',')

# split into input (X) and output (y) variables
X = dataset[:,0:1] * 1.0
y = dataset[:,1] * 1.0

# define the keras model
model = Sequential()
model.add(Dense(6, input_dim=1, activation='relu'))
model.add(Dense(10, activation='relu'))
model.add(Dense(1))

# compile the keras model
opt = optimizers.adam(lr=0.001)
model.compile(loss='mean_squared_error', optimizer=opt)

# fit the keras model on the dataset (CPU)
model.fit(X, y, epochs=1500, batch_size=10, verbose=0)

# evaluate the keras model
_, accuracy = model.evaluate(X, y, verbose=0)
print('Accuracy: %.2f' % (accuracy*100))

# make class predictions with the model
predicitions = model.predict(X)

# summarize the first 10 cases
for i in range(10):
    print('%s => %.2f (expected %.2f)' % (X[i].tolist(), predicitions[i], y[i]))

Output:

[1.0] => 1.00 (expected 1.00)
[4.0] => 2.00 (expected 2.00)
[9.0] => 3.32 (expected 3.00)
[16.0] => 3.89 (expected 4.00)
[25.0] => 4.61 (expected 5.00)
[36.0] => 5.49 (expected 6.00)
[49.0] => 6.52 (expected 7.00)
[64.0] => 7.72 (expected 8.00)
[81.0] => 9.07 (expected 9.00)
[100.0] => 10.58 (expected 10.00)

EDIT:

As @desertnaut pointed out in the comments, the metric accuracy has no meaning in a regression task. A custom R_squared value (AKA coefficient of determination) is therefore generally used as the metric. R_squared value represents the goodness of a fit in a regression model. Following is the code to calculate R_squared.

def r_squared(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square(y_true - y_pred)) 
    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) 
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

Now, you could compile the model with;

model.compile(loss='mean_squared_error', optimizer=opt, metrics=[r_squared])