Simple regression with Keras seems not working properly

Question

I am trying, just for practising with Keras, to train a network to learn a very easy function. The input of the network is 2Dimensional . The output is one dimensional. The function can indeed represented with an image, and the same is for the approximate function. At the moment I'm not looking for any good generalization, I just want that the network is at least good in representing the training set. Here I place my code:

import matplotlib.pyplot as plt
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
import random as rnd
import math

m = [
[1,1,1,1,0,0,0,0,1,1],
[1,1,0,0,0,0,0,0,1,1],
[1,0,0,0,1,1,0,1,0,0],
[1,0,0,1,0,0,0,0,0,0],
[0,0,0,0,1,1,0,0,0,0],
[0,0,0,0,1,1,0,0,0,0],
[0,0,0,0,0,0,1,0,0,1],
[0,0,1,0,1,1,0,0,0,1],
[1,1,0,0,0,0,0,0,1,1],
[1,1,0,0,0,0,1,1,1,1]]                      #A representation of the function that I would like to approximize

matrix = np.matrix(m)

evaluation = np.zeros((100,100))

x_train = np.zeros((10000,2))
y_train = np.zeros((10000,1))

for x in range(0,100):
    for y in range(0,100):
         x_train[x+100*y,0] = x/100.                #I normilize the input of the function, between [0,1)
         x_train[x+100*y,1] = y/100.
         y_train[x+100*y,0] = matrix[int(x/10),int(y/10)] +0.0


#Here I show graphically what I would like to have
plt.matshow(matrix, interpolation='nearest', cmap=plt.cm.ocean, extent=(0,1,0,1))

#Here I built the model
model = Sequential()
model.add(Dense(20, input_dim=2, init='uniform'))
model.add(Activation('tanh'))
model.add(Dense(1, init='uniform'))
model.add(Activation('sigmoid'))

#Here I train it
sgd = SGD(lr=0.5)
model.compile(loss='mean_squared_error', optimizer=sgd)

model.fit(x_train, y_train,
          nb_epoch=100,
          batch_size=100,
          show_accuracy=True)

#Here (I'm not sure), I'm using the network over the given example
x = model.predict(x_train,batch_size=1)

#Here I show the approximated function
print x
print x_train
for i in range(0, 10000):
    evaluation[int(x_train[i,0]*100),int(x_train[i,1]*100)] = x[i]

plt.matshow(evaluation, interpolation='nearest', cmap=plt.cm.ocean, extent=(0,1,0,1))
plt.colorbar()
plt.show()

As you can see, the two function are completely different, and I can't understand why. I think that maybe model.predict doesn't work as I axpect.

Answer 1

I have changed just your network structure and added a training dataset. The loss decreases down to 0.01.

# -*- coding: utf-8 -*-
"""
Created on Thu Mar 16 15:26:52 2017

@author: Administrator
"""

import matplotlib.pyplot as plt
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
import random as rnd
import math
from keras.optimizers import Adam,SGD
m = [
[1,1,1,1,0,0,0,0,1,1],
[1,1,0,0,0,0,0,0,1,1],
[1,0,0,0,1,1,0,1,0,0],
[1,0,0,1,0,0,0,0,0,0],
[0,0,0,0,1,1,0,0,0,0],
[0,0,0,0,1,1,0,0,0,0],
[0,0,0,0,0,0,1,0,0,1],
[0,0,1,0,1,1,0,0,0,1],
[1,1,0,0,0,0,0,0,1,1],
[1,1,0,0,0,0,1,1,1,1]]                      #A representation of the function that I would like to approximize

matrix = np.matrix(m)

evaluation = np.zeros((1000,1000))

x_train = np.zeros((1000000,2))
y_train = np.zeros((1000000,1))

for x in range(0,1000):
    for y in range(0,1000):
         x_train[x+1000*y,0] = x/1000.                #I normilize the input of the function, between [0,1)
         x_train[x+1000*y,1] = y/1000.
         y_train[x+1000*y,0] = matrix[int(x/100),int(y/100)] +0.0


#Here I show graphically what I would like to have
plt.matshow(matrix, interpolation='nearest', cmap=plt.cm.ocean, extent=(0,1,0,1))

#Here I built the model
model = Sequential()
model.add(Dense(50, input_dim=2, init='uniform'))## init是关键字，’uniform’表示用均匀分布去初始化权重
model.add(Activation('tanh'))
model.add(Dense(20, init='uniform'))
model.add(Activation('tanh'))
model.add(Dense(1, init='uniform'))
model.add(Activation('sigmoid'))

#Here I train it
#sgd = SGD(lr=0.01)
adam = Adam(lr = 0.01)
model.compile(loss='mean_squared_error', optimizer=adam)

model.fit(x_train, y_train,
          nb_epoch=100,
          batch_size=100,
          show_accuracy=True)

#Here (I'm not sure), I'm using the network over the given example
x = model.predict(x_train,batch_size=1)

#Here I show the approximated function
print (x)
print (x_train)
for i in range(0, 1000000):
    evaluation[int(x_train[i,0]*1000),int(x_train[i,1]*1000)] = x[i]

plt.matshow(evaluation, interpolation='nearest', cmap=plt.cm.ocean, extent=(0,1,0,1))
plt.colorbar()
plt.show()

Answer 2

Your understanding is correct; it's just a question of hyperparameter tuning.

I just tried your code, and it looks like you're not giving your training enough time:

Look at the loss, under 100 epochs, it's stuck at around 0.23. But try using the 'adam' otimizer instead of SGD, and increase the number of epochs up to 10,000: the loss now decreases down to 0.09 and your picture looks much better.

If it's still not precise enough for you, you may also want to try increasing the number of parameters: just add a few layers; this will make overfitting much easier ! :-)