CODEWITHSUNDEEP
pythonscikit-learnkeras
Antoine Caté
Antoine Caté

Reputation: 107

Difference between accuracy_score in scikit-learn and accuracy in Keras

I have implemented and trained a multiclass Convolutional Neural Network on Keras. The resulting testing accuracy is 0.9522. However, when I calculate the accuracy using accuracy_score from scikit-learn, I get 0.6224. Here is what I did:

X_train = X[:60000, :, :, :]
X_test = X[60000:, :, :, :]
y_train = y[:60000, :]
y_test = y[60000:, :]
print ('Size of the arrays:')
print ('X_train: ' + str(X_train.shape))
print ('X_test: ' + str(X_test.shape))
print ('y_train: ' + str(y_train.shape))
print ('y_test: ' + str(y_test.shape))

Results:

Size of the arrays:
X_train: (60000, 64, 64, 3)
X_test: (40000, 64, 64, 3)
y_train: (60000, 14)
y_test: (40000, 14)

Fitting the Keras model (I do not add the entire model here to keep the code simple):

model = Sequential()
model.add(Conv2D(10, (5,5), padding='same', input_shape=(64, 64, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(14))
model.add(Activation('softmax'))
model.compile(optimizer='rmsprop', loss='mean_squared_error', metrics['accuracy'])
model.fit(X_train, y_train, batch_size=100, epochs=5, verbose=1, validation_data=(X_test, y_test))

Accuracy with Scikit-Learn:

y_pred = model.predict(X_test, batch_size=100)
y_pred1D = y_pred.argmax(1)
y_pred = model.predict(X_test, batch_size=100)
y_test1D = y_test.argmax(1)
print ('Accuracy on validation data: ' + str(accuracy_score(y_test1D, y_pred1D)))

Score:

Accuracy on validation data: 0.6224

Accuracy with Keras:

score_Keras = model.evaluate(X_test, y_test, batch_size=200)
print('Accuracy on validation data with Keras: ' + str(score_Keras[1]))

Results:

Accuracy on validation data with Keras: 0.95219109267

My question is: why are the two accuracies different, and which one should I use to evaluate the performance of my multiclass classifier?

Thanks in advance!

Upvotes: 5

Views: 3977

Answers (1)

Coding thermodynamist
Coding thermodynamist

Reputation: 1403

There is a typo in your code, why are you defining y_pred twice ?

y_pred = model.predict(X_test, batch_size=100)
y_pred1D = y_pred.argmax(1)
y_pred = model.predict(X_test, batch_size=100)
y_test1D = y_test.argmax(1)
print ('Accuracy on validation data: ' + str(accuracy_score(y_test1D, y_pred1D)))

Should be :

y_pred = model.predict(X_test, batch_size=100)
y_pred1D = y_pred.argmax(1)
y_test1D = y_test.argmax(1)
print ('Accuracy on validation data: ' + str(accuracy_score(y_test1D, y_pred1D)))

Despite that, you should provide the values and shapes of y_pred1D and y_test1D, the error lies when you do y_pred1D = y_pred.argmax(1) and y_test1D = y_test.argmax(1) in order to use the scikit learn metric. My guess is, that it is not what you think it is, otherwise the two metrics would be the same.

Upvotes: 2

Related Questions