CODEWITHSUNDEEP
pythonmachine-learningkerasoutput
Rami
Rami

Reputation: 43

Accuracy calculation in keras not matching

Currently doing a multi-label classifier. I receive suspiciously high accuracies (0.94) calling the evaluate method in keras(the evaluate method returns accuracy and loss). I then tried to program the method manually, as described here: How does Keras evaluate the accuracy?

However, I receive way lower accuracy doing it on my own. F1_mico :0.7157894736842105 Accuracy:0.3530864197530864

I do not understand why. I thought it has to do something with imbalanced classes. The f1_mico value is high, while the f1_macro is around 0.45, so it seems to make sense(I have a hierarchy, so some classes naturally appear more often than others). However, how is keras getting such a high accuracy, it does not make sense to me. The code:

model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['accuracy'])
#print("Traning Model...")
model.fit(X_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, callbacks=[checkpoint], validation_data=(X_test, y_test))  # starts training
print("Testing Model")
output = model.predict(X_final, batch_size = batch_size)
for pred_i in output:
    pred_i[pred_i >=0.5] = 1
    pred_i[pred_i < 0.5] = 0
print "F1: " + str(f1_score(y_final, output, average='micro'))
print "Accuracy: " + str(accuracy_score(y_final, output))

mscores = model.evaluate(X_final, y_final, batch_size = batch_size)
print mscores

Output:

Creating Model...
Testing Model
F1: 0.7157894736842105
Accuracy: 0.3530864197530864
405/405 [==============================] - 8s 20ms/step
['0.15227678694106914', '0.9422222640779283']

Upvotes: 1

Views: 359

Answers (1)

Crigges
Crigges

Reputation: 1243

The Acccuracy calculated by keras compares each value of the one-hot encoded vector and looks if it matches. If data is rather sparse this causes the system to have a high accuracy since most values are going to be assigned to zero. The scilearn accuracy only returns one for the data if the complete prediction matches.

Upvotes: 1

Related Questions