Scikit-learn recall and precision

Question

I would like to understand how scikit-learn's recall and precision work. Here is the documentation.

So, precision formula is: tp / (tp + fp) and recall formula: tp / (tp + fn). tp = true positives, fp = false positivies and fn = false negatives.

Let's figure this out with an example: 99 patients are healthy, 1 patient is sick.

Example 1: The classifier decides that everyone of the 100 patient are healthy. In that case:

precision: 99 / (99 + 1) = 0,9

recall: 99 / (99 + 0) = 1

Example 2: Every of the 100 patients is classified sick:

precision: 0 / (0 + 0) = n.d.

recall: 0 / (0 + 99) = 0

I found another explanation of precision and recall in this book:

Precision is the fraction of detections reported by the model that were correct, while recall is the fraction of true events that were detected. A detector that says no one has the disease would achieve perfect precision, but zero recall.

This is the case from example 1, right? No one has the disease means everyone is healthy. A perfect precision means 1 but in my calculations it's 0.9. Where is the mistake? Also why 0 recall?

A detector that says everyone has the disease would achieve perfect recall, but precision equal to the percentage of people who have the disease (0.0001 percent in our example of a disease that only one people in a million have).

This is example 2: everyone is sick. How can precision have another value than undefined? I calculated a recall of 0, how can it be 1 in this example?

Answer 1

The first and the foremost decision that you need to make here is what will be your positive class.

In the use-case that is specified, it makes more sense to detect sick patients, so logically the positive class should be sick, and hence the negative class becomes healthy (yeah, it's a little counter-intuitive).

Now, considering this background, the confusion matrix(C-M) will look something like below:

And, hence if all are identified as healthy, the C-M will look like:

Now, The precision and recall can be calculated based on the formulas that are posted in the question.

precision = TP/(TP+FP) = 0/(0+0) = n.a. #This is encountered when classifier predicts all scenarios as negative label.Check Here

recall = TP/(TP+FN) = 0/(0+1) = 0

In a similar way, if all the patients are identified as sick. The confusion matrix would look something like this:

And the precision-recall will look like:

precision = TP/(TP+FP) = 1/(1+99) = 1/100

recall = TP/(TP+FN) = 1/(1+0) = 1

This is how scikit-learn also takes recall. Based on what you consider as the positive class for your use-case.

Adding python code for the same:

(Scenario 1):

from sklearn.metrics import confusion_matrix, recall_score, precision_score
qq_gt = [0]*100
qq_gt[0] = 1
qq_predict = [1]*100
print('Recall: ', recall_score(qq_gt, qq_predict))
print('Precision: ', precision_score(qq_gt, qq_predict))

Scenario 2:

qq_gt = [0]*100
qq_gt[0] = 1
qq_predict = [0]*100
print('Recall: ', recall_score(qq_gt, qq_predict))
print('Precision: ', precision_score(qq_gt, qq_predict))