Why is tensorflow having a worse accuracy than keras in direct comparison?

Question

I made a direct comparison between TensorFlow vs Keras with the same parameters and the same dataset (MNIST).

The strange thing is that Keras achieves 96% performance in 10 epochs, while TensorFlow achieves about 70% performance in 10 epochs. I have run this code many times in the same instance and this inconsistency always occurs.

Even setting 50 epochs for TensorFlow, the final performance reaches 90%.

Code:

import keras
from keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

# One hot encoding
from keras.utils import np_utils
y_train = np_utils.to_categorical(y_train) 
y_test = np_utils.to_categorical(y_test) 

# Changing the shape of input images and normalizing
x_train = x_train.reshape((60000, 784))
x_test = x_test.reshape((10000, 784))
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation

# Creating the neural network
model = Sequential()
model.add(Dense(30, input_dim=784, kernel_initializer='normal', activation='relu'))
model.add(Dense(30, kernel_initializer='normal', activation='relu'))
model.add(Dense(10, kernel_initializer='normal', activation='softmax'))

# Optimizer
optimizer = keras.optimizers.Adam()

# Loss function
model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc'])

# Training
model.fit(x_train, y_train, epochs=10, batch_size=200, validation_data=(x_test, y_test), verbose=1)

# Checking the final accuracy
accuracy_final = model.evaluate(x_test, y_test, verbose=0)
print('Model Accuracy: ', accuracy_final)

TensorFlow code: (x_train, x_test, y_train, y_test are the same as the input for the Keras code above)

import tensorflow as tf
# Epochs parameters
epochs = 10
batch_size = 200

# Neural network parameters
n_input = 784 
n_hidden_1 = 30 
n_hidden_2 = 30 
n_classes = 10 

# Placeholders x, y
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_classes])

# Creating the first layer
w1 = tf.Variable(tf.random_normal([n_input, n_hidden_1]))
b1 = tf.Variable(tf.random_normal([n_hidden_1]))
layer_1 = tf.nn.relu(tf.add(tf.matmul(x,w1),b1)) 

# Creating the second layer 
w2 = tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2]))
b2 = tf.Variable(tf.random_normal([n_hidden_2]))
layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1,w2),b2)) 

# Creating the output layer 
w_out = tf.Variable(tf.random_normal([n_hidden_2, n_classes]))
bias_out = tf.Variable(tf.random_normal([n_classes]))
output = tf.matmul(layer_2, w_out) + bias_out

# Loss function
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = output, labels = y))
# Optimizer
optimizer = tf.train.AdamOptimizer().minimize(cost)

# Making predictions
predictions = tf.equal(tf.argmax(output, 1), tf.argmax(y, 1))

# Accuracy
accuracy = tf.reduce_mean(tf.cast(predictions, tf.float32))

# Variables that will be used in the training cycle
train_size = x_train.shape[0]
total_batches = train_size / batch_size

# Initializing the variables
init = tf.global_variables_initializer()

# Opening the session
with tf.Session() as sess:
    sess.run(init)

    # Training cycle
    for epoch in range(epochs):

        # Loop through all batch iterations
        for i in range(0, train_size, batch_size): 
            batch_x = x_train[i:i + batch_size]
            batch_y = y_train[i:i + batch_size]

            # Fit training
            sess.run(optimizer, feed_dict={x: batch_x, y: batch_y})

        # Running accuracy (with test data) on each epoch    
        acc_val = sess.run(accuracy, feed_dict={x: x_test, y: y_test})
        # Showing results after each epoch
        print ("Epoch: ", "{}".format((epoch + 1)))
        print ("Accuracy_val = ", "{:.3f}".format(acc_val))

    print ("Training Completed!")

    # Checking the final accuracy
    checking = tf.equal(tf.argmax(output, 1), tf.argmax(y, 1))
    accuracy_final = tf.reduce_mean(tf.cast(checking, tf.float32))  
    print ("Model Accuracy:", accuracy_final.eval({x: x_test, y: y_test}))

I'm running everything in the same instance. Can anyone explain this inconsistency?

Answer 1

I think it's the initialization that's the culprit. For example, one real difference is that you initialize bias in TF with random_normal which isn't the best practice, and in fact Keras defaults to initializing the bias to zero, which is the best practice. You don't override this, since you only set kernel_initializer, but not bias_initializer in your Keras code.

Furthermore, things are worse for the weight initializers. You are using RandomNormal for Keras, defined like so:

keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=None)

But in TF you use tf.random.normal:

tf.random.normal(shape, mean=0.0, stddev=1.0, dtype=tf.dtypes.float32,    seed=None, name=None)

I can tell you that using standard deviation of 0.05 is reasonable for initialization, but using 1.0 is not.

I suspect that if you changed these parameters, things would look better. But if they don't, I'd suggest dumping the TensorFlow graph for both models and just checking by hand to see the differences. The graphs are small enough in this case to double-check.

To some extent this highlights the difference in philosophy between Keras and TF. Keras tries hard to set good defaults for NN training that correspond to what is known to work. But TensorFlow is completely agnostic - you have to know those practices and explicitly code them in. The standard deviation thing is a stellar example: of course it should be 1 by default in a mathematical function, but 0.05 is a good value if you know it will be used to initialize an NN layer.

Answer originally provided by Dmitriy Genzel on Quora.