CODEWITHSUNDEEP
pythontensorflowkerasscikit-learnauc
Arjunsai
Arjunsai

Reputation: 176

how to use sklearn AUC in tensorflow.keras model metrics?

I'm trying to use sklearn AUC in tf.keras as model metrics , for that I used custom made function from this link AUC

Below is my model :

def auc(y_true, y_pred):
    return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)

model = Model(inputs= [text_x,state_x,grade_x,cat_x,subcat_x,teach_x,num_x],outputs = [output_layer])
model.compile(optimizer = 'Adam', loss= 'binary_crossentropy', metrics=[auc])

history = model.fit(x = input_data , y= y_train,batch_size = 180, epochs = 15, callbacks = [es, mc], validation_data = (val_data, y_val))

Train on 69918 samples, validate on 17480 samples
Epoch 1/15
69918/69918 [==============================] - 278s 4ms/sample - loss: 0.3086 - auc: 0.8516 - val_loss: 0.4711 - val_auc: 0.6896
Epoch 2/15
69918/69918 [==============================] - 275s 4ms/sample - loss: 0.1417 - auc: 0.9738 - val_loss: 0.6638 - val_auc: 0.6692
Epoch 3/15
69918/69918 [==============================] - 275s 4ms/sample - loss: 0.0506 - auc: 0.9964 - val_loss: 0.9611 - val_auc: 0.6824
Epoch 4/15
69918/69918 [==============================] - 276s 4ms/sample - loss: 0.0329 - auc: 0.9983 - val_loss: 0.9462 - val_auc: 0.6719

I'm getting this error when evaluate the model, ValueError:

test_input_data = [text_test_1,state_test,grade_test,cat_test,subcat_test,teach_test,num_test]
score = model.evaluate(test_input_data, y_test,verbose = 1)
print('test_loss: ',score[0])
print('test_acc: ',score[1])

<ipython-input-103-336c032c70f4> in <module>()
      1 test_input_data = [text_test_1,state_test,grade_test,cat_test,subcat_test,teach_test,num_test]
----> 2 score = model.evaluate(test_input_data, y_test,verbose = 1)
      3 print('test_loss: ',score[0])
      4 print('test_acc: ',score[1])

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in __call__(self, *args, **kwargs)
   1456         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1457                                                self._handle, args,
-> 1458                                                run_metadata_ptr)
   1459         if run_metadata:
   1460           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 209, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 355, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/base.py", line 76, in _average_binary_score
    return binary_metric(y_true, y_score, sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 323, in _binary_roc_auc_score
    raise ValueError("Only one class present in y_true. ROC AUC score "

ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.


     [[{{node metrics_15/auc/PyFunc}}]]
  (1) Invalid argument: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 209, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 355, in roc_auc_score
    sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/base.py", line 76, in _average_binary_score
    return binary_metric(y_true, y_score, sample_weight=sample_weight)

  File "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/ranking.py", line 323, in _binary_roc_auc_score
    raise ValueError("Only one class present in y_true. ROC AUC score "

ValueError: Only one class present in y_true. ROC AUC score is not defined in that case.


     [[{{node metrics_15/auc/PyFunc}}]]
     [[metrics_15/auc/PyFunc/_1683]]
0 successful operations.
0 derived errors ignored.

I tried tf.keras.metrics.AUC, then its working fine but when using sklearn AUC I having this error. how to set sklearn's AUC in tf.keras.model metric function. any help would be appreciated..Thank you.

Upvotes: 4

Views: 3917

Answers (1)

Vijay Maurya
Vijay Maurya

Reputation: 349

I had a same problem but found this code on Github : pranaya-mathur account you can follow same

from sklearn.metrics import roc_auc_score
def auc_score(y_true, y_pred):
    if len(np.unique(y_true[:,1])) == 1:
        return 0.5
    else:
        return roc_auc_score(y_true, y_pred)
def auc(y_true, y_pred):
    return tf.py_func(auc1, (y_true, y_pred), tf.double)
#in model.compile you can use auc function name
model.compile(optimizer=optimizer,loss='categorical_crossentropy',metrics=[auc])

Upvotes: 2

Related Questions