Massively worse performance in Tensorflow compared to Scikit-Learn for Logistic Regression

Question

I’m trying to implement a logistic regression classifier on a numerical dataset. I was having trouble getting good accuracy and loss with a model I built in Tensorflow, so to check it wasn’t something in the data, I tried using scikit-learn’s own LogisticRegression, and got massively better results. The difference is so big that I suspect I’m doing something really basic wrong on the .tf side...

Data preprocessing:

dt = pd.read_csv('data.csv', header=0)
npArray = np.array(dt)
xvals = npArray[:,1:].astype(float)
yvals = npArray[:,0]
x_proc = preprocessing.scale(xvals)

XTrain, XTest, yTrain, yTest = train_test_split(x_proc, yvals, random_state=1)

If I now do Logistic Regression with sklearn:

log_reg = LogisticRegression(class_weight='balanced')
log_reg.fit(XTrain, yTrain)
yPred = log_reg.predict(XTest)
print (metrics.classification_report(yTest, yPred))
print ("Overall Accuracy:", round(metrics.accuracy_score(yTest, yPred),2))

...I get the following confusion matrix:

        precision    recall  f1-score   support
      1       1.00      0.98      0.99        52
      2       0.96      1.00      0.98        52
      3       0.98      0.96      0.97        51
      4       0.98      0.97      0.97        58
      5       1.00      0.95      0.97        37
      6       0.93      1.00      0.96        65
      7       1.00      0.95      0.97        41
      8       0.94      0.98      0.96        50
      9       1.00      0.98      0.99        45
     10       1.00      0.98      0.99        49
 avg/total    0.98      0.98      0.98       500

 Overall Accuracy: 0.98

Great stuff, right? Here's the Tensorflow code instead from the same point right after the split:

yTrain.resize(len(yTrain),10) #the labels are scores between 1 and 10
yTest.resize(len(yTest),10)

tf.reset_default_graph()

X = tf.placeholder(tf.float32, [None, 8], name="input") 
Y = tf.placeholder(tf.float32, [None, 10])

W = tf.Variable(tf.zeros([8, 10])) 
b = tf.Variable(tf.zeros([10])) 

out = (tf.matmul(X, W) + b)
pred = tf.nn.softmax(out, name="output")

learning_rate = 0.001
training_epochs = 100
batch_size = 200
display_step = 1

L2_LOSS = 0.01

l2 = L2_LOSS * \
    sum(tf.nn.l2_loss(tf_var) for tf_var in tf.trainable_variables())

# Minimize error using cross entropy
cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits = out, labels = Y)) + l2
# Gradient Descent
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

train_count = len(XTrain)

#defining optimizer and accuracy
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(Y, 1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

#----Training the model------------------------------------------
saver = tf.train.Saver()

history = dict(train_loss=[], 
                 train_acc=[], 
                 test_loss=[], 
                 test_acc=[])

sess=tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for i in range(1, training_epochs + 1):
    for start, end in zip(range(0, train_count, batch_size),
                      range(batch_size, train_count + 1,batch_size)):
        sess.run(optimizer, feed_dict={X: XTrain[start:end],
                                   Y: yTrain[start:end]})

    _, acc_train, loss_train = sess.run([pred, accuracy, cost], feed_dict={
                                        X: XTrain, Y: yTrain})

    _, acc_test, loss_test = sess.run([pred, accuracy, cost], feed_dict={
                                        X: XTest, Y: yTest})

    history['train_loss'].append(loss_train)
    history['train_acc'].append(acc_train)
    history['test_loss'].append(loss_test)
    history['test_acc'].append(acc_test)

   if i != 1 and i % 10 != 0:
       continue

   print(f'epoch: {i} test accuracy: {acc_test} loss: {loss_test}')

predictions, acc_final, loss_final = sess.run([pred, accuracy, cost], feed_dict={X: XTest, Y: yTest})

print()
print(f'final results: accuracy: {acc_final} loss: {loss_final}')

Now I get the following:

epoch: 1 test accuracy: 0.41200000047683716 loss: 0.6921926140785217
epoch: 10 test accuracy: 0.5 loss: 0.6909801363945007
epoch: 20 test accuracy: 0.5180000066757202 loss: 0.6918861269950867
epoch: 30 test accuracy: 0.515999972820282 loss: 0.6927152872085571
epoch: 40 test accuracy: 0.5099999904632568 loss: 0.6933282613754272
epoch: 50 test accuracy: 0.5040000081062317 loss: 0.6937957406044006
epoch: 60 test accuracy: 0.5019999742507935 loss: 0.6941683292388916
epoch: 70 test accuracy: 0.5019999742507935 loss: 0.6944747567176819
epoch: 80 test accuracy: 0.4959999918937683 loss: 0.6947320103645325
epoch: 90 test accuracy: 0.46799999475479126 loss: 0.6949512958526611
epoch: 100 test accuracy: 0.4560000002384186 loss: 0.6951409578323364

final results: accuracy: 0.4560000002384186 loss: 0.6951409578323364

Thoughts? I have experimented with initialising the weights (with the 2nd answer here: How to do Xavier initialization on TensorFlow), changing the learning rate, epochs, batch size, L2 loss etc., all to no real effect. Any help would be really appreciated...

Answer 1

I think I got to the root of the problem - yTrain.resize and yTest.resize were stupid logic- and maths-wise and once I replaced them with one-hot encoded arrays (with help from Convert array of indices to 1-hot encoded numpy array) it all started to work far better. Got the same accuracy in the end as in sk-learn (I think)!