CODEWITHSUNDEEP
pythontensorflowneural-networkfeed-forward
sagi trabilse
sagi trabilse

Reputation: 11

How to train my neural network properly?

My neural network solving a nonlinear problem, but the testing loss is very high. When I use a neural network without hidden layer the testing loss is lower than with hidden layer but also high. Someone know why? and how I can improve the loss?

#data

    train_X = data_in[0:9001, :]
    train_Y = data_out[0:9001, :]
    test_X = data_in[9000:10001, :]
    test_Y = data_out[9000:10001, :
    n = train_X.shape[1] 
    m = train_X.shape[0]
    d = train_Y.shape[1]  
    l = test_X.shape[0]

#parameters

    trainX = tf.placeholder(tf.float32, [batch_size, n])
    trainY = tf.placeholder(tf.float32, [batch_size, d])
    testX = tf.placeholder(tf.float32, [l, n])
    testY = tf.placeholder(tf.float32, [l, d])
    def multilayer(trainX, h1, h2, hout, b1, b2, bout):
        layer_1 = tf.matmul(trainX, h1) + b1
        layer_1 = tf.nn.sigmoid(layer_1)
        layer_2 = tf.matmul(layer_1, h2) + b2
        layer_2 = tf.nn.sigmoid(layer_2)
        out_layer = tf.matmul(layer_2, hout) + bout
        return out_layer
    h1 = tf.Variable(tf.zeros([n, n_hidden_1]))
    h2 = tf.Variable(tf.zeros([n_hidden_1, n_hidden_2]))
    hout = tf.Variable(tf.zeros([n_hidden_2, d]))
    b1 = tf.Variable(tf.zeros([n_hidden_1]))
    b2 = tf.Variable(tf.zeros([n_hidden_2]))
    bout = tf.Variable(tf.zeros([d]))
    pred = multilayer(trainX, h1, h2, hout, b1, b2, bout)
    predtest = multilayer(testX, h1, h2, hout, b1, b2, bout)
    loss = tf.reduce_sum(tf.pow(pred - trainY, 2)) / (batch_size)
    losstest = tf.reduce_sum(tf.pow(predtest - testY, 2)) / (l)
    optimizer = 
    tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)

# Initializing the variables

    init = tf.global_variables_initializer()
    a = np.linspace(0, m - batch_size, m / batch_size, dtype=np.int32)
    with tf.Session() as sess:
        sess.run(init)
        for i in (a):
            x = train_X[i:i + batch_size, :]
            y = train_Y[i:i + batch_size, :]
            for epoch in range(training_epochs):
                sess.run(optimizer, feed_dict={trainX: np.asarray(x), trainY: 
                          np.asarray(y)})
                c = sess.run(loss, feed_dict={trainX: np.asarray(x), trainY: 
                            np.asarray(y)})
                print("Batch:", '%04d' % (i / batch_size + 1), "Epoch:", '%04d'%
                      (epoch + 1), "loss=", "{:.9f}".format(c))
# Testing
    print("Testing... (Mean square loss Comparison)")
    testing_loss = sess.run(losstest, feed_dict={testX: np.asarray(test_X), 
    testY: np.asarray(test_Y)})
    pred_y_vals = sess.run(predtest, feed_dict={testX: test_X})
    print("Testing loss=", testing_loss)

Upvotes: -1

Views: 79

Answers (1)

Sunreef
Sunreef

Reputation: 4552

From what I can see in your training loop, you're iterating on the epochs before iterating on the batches. Which means that your network will be trained several times (training_epochs times) on the same batch and then go on to the next batch. It will never come back on a batch it's seen before.

Intuitively, I would say that your network is heavily overfitting on the last batch it's seen during training. Which explains the high loss during testing.

Invert the two loops in your training and you should be fine.

Upvotes: 0

Related Questions