CODEWITHSUNDEEP
kerasscikit-learnroc
sensationti
sensationti

Reputation: 337

sklearn ROC curver

I have 10 classes, and my y_test has shape (1000, 10) and it looks like this:

array([[0, 0, 1, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ...,
       [0, 0, 0, ..., 0, 0, 1],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]], dtype=int64)

If I use the following where i is the class number

fpr[i], tpr[i], _ = roc_curve(y_test[:, i], y_pred[:, i])

should y_pred be

y_pred = model.predict(x_test)

OR

y_pred = np.argmax(model.predict(x_test), axis=1)
lb = LabelBinarizer()
lb.fit(y_test)
y_pred = lb.transform(y_pred)

The first option gives me something like this:

[[6.87280996e-11 6.28617670e-07 9.96915460e-01 ... 3.08361766e-03
  3.47333212e-14 2.83545876e-09]
 [7.04240659e-30 1.51786850e-07 8.49807921e-28 ... 6.62584656e-33
  6.97696034e-19 1.01019222e-20]
 [2.97537670e-14 2.67199534e-24 2.85646610e-19 ... 2.19898160e-15
  7.03626012e-22 7.56072279e-18]
 ...
 [1.63774752e-15 1.32784101e-06 1.23182635e-05 ... 3.60217566e-14
  6.01247484e-05 2.61179358e-01]
 [2.09420733e-35 6.94865276e-10 1.14242395e-22 ... 5.08080394e-22
  1.20934697e-19 1.77760468e-17]
 [1.68334747e-13 8.53335252e-04 4.40571597e-07 ... 1.70050384e-06
  1.48684137e-06 2.93400045e-03]]

with shape (1000,10).

where the latter option gives

[[0 0 1 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]

with shape (1000,10)

Which way is the correct approach? in other words, what would this y_pred be when passing to sklearn.metrics.roc_curve().

Forget to mention, using the first option gives me extremely high (almost 1) AUC values for all classes, whereas the second option seems to generate reasonable AUC values.

The ROC curves using the two options are below, which one looks more correct?

first option second option

Upvotes: 0

Views: 63

Answers (1)

Daniel Möller
Daniel Möller

Reputation: 86630

There is nothing wrong with the first option, and that's what the documentation asks for:

y_scorendarray of shape (n_samples,)

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).

Also, the first graph looks like a ROC curve, while the second is weird.

And finally, ROC curves intend to study "different classification thresholds". That means you need predictions "as probabilities" (confidences), not as 0's and 1's.

When you take an argmax, you throw away the probabilities/confidences, making it impossible to study thresholds.

Upvotes: 1

Related Questions