CODEWITHSUNDEEP
pythontensorflowmetrics
pangasio
pangasio

Reputation: 13

Meaning of values returned by tensorflow accuracy metric

I'm a bit confused about the values returned by the functions of the module tf.metrics (for example tf.metrics.accuracy).

A simple piece of code in which I calculate accuracy using tf.metrics.accuracy and using tp, tn, fp and fn.

import tensorflow as tf

# true and predicted tensors
y_p = tf.placeholder(dtype=tf.int64)
y_t = tf.placeholder(dtype=tf.int64)

# Count true positives, true negatives, false positives and false negatives.
tp = tf.count_nonzero(y_p * y_t)
tn = tf.count_nonzero((y_p - 1) * (y_t - 1))
fp = tf.count_nonzero(y_p * (y_t - 1))
fn = tf.count_nonzero((y_p - 1) * y_t)

acc = tf.metrics.accuracy(y_p, y_t)

# Calculate accuracy, precision, recall and F1 score.
accuracy = (tp + tn) / (tp + fp + fn + tn)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())

    for i in range(4):
        if i == 0:
            yop = [0,0,0,0,0,0,0,0,0,0]
        elif i == 1:
            yop = [0,0,0,0,0,0,0,0,1,1]
        elif i == 2:
            yop = [1,1,1,0,0,0,0,0,0,1]
        else:
            yop = [0,1,1,1,1,1,1,0,0,0]
        tf_a = sess.run(acc, feed_dict={y_p: [0,0,0,0,0,0,0,0,0,0], y_t: yop})
        my_a = sess.run(accuracy, feed_dict={y_p: [0,0,0,0,0,0,0,0,0,0], y_t: yop})
        print("TF accuracy: {0}".format(tf_a))
        print("My accuracy: {0}".format(my_a))

which outputs

TF accuracy: (0.0, 1.0)
My accuracy: 1.0
TF accuracy: (1.0, 0.9)
My accuracy: 0.8
TF accuracy: (0.9, 0.8)
My accuracy: 0.6
TF accuracy: (0.8, 0.7)
My accuracy: 0.4

I understand that the second returned value of tf.metrics.accuracy (the update_op) is the averaged accuracy over the number of times the function was called. However, I can't make sense of the first value, which should represent the accuracy. Why does it differ from the value of the accuracy I calculate myself? Isn't there a way to obtain the non-cumulative value for the accuracy?

Thanks in advance.

Upvotes: 1

Views: 1850

Answers (1)

Vikash Singh
Vikash Singh

Reputation: 14001

import tensorflow as tf
from sklearn.metrics import accuracy_score

# true and predicted tensors
y_p = tf.placeholder(dtype=tf.int64)
y_t = tf.placeholder(dtype=tf.int64)

# Count true positives, true negatives, false positives and false negatives.
tp = tf.count_nonzero(y_p * y_t)
tn = tf.count_nonzero((y_p - 1) * (y_t - 1))
fp = tf.count_nonzero(y_p * (y_t - 1))
fn = tf.count_nonzero((y_p - 1) * y_t)

acc = tf.metrics.accuracy(predictions=y_p, labels=y_t)

# Calculate accuracy, precision, recall and F1 score.
accuracy = (tp + tn) / (tp + fp + fn + tn)

with tf.Session() as sess:
    for i in range(4):
        sess.run(tf.global_variables_initializer())
        sess.run(tf.local_variables_initializer())


        if i == 0:
            yop = [0,0,0,0,0,0,0,0,0,0]
        elif i == 1:
            yop = [0,0,0,0,0,0,0,0,1,1]
        elif i == 2:
            yop = [1,1,1,0,0,0,0,0,0,1]
        else:
            yop = [0,1,1,1,1,1,1,0,0,0]
        print('accuracy_score', accuracy_score([0,0,0,0,0,0,0,0,0,0], yop))
        tf_a = sess.run(acc, feed_dict={y_p: [0,0,0,0,0,0,0,0,0,0], y_t: yop})
        my_a = sess.run(accuracy, feed_dict={y_p: [0,0,0,0,0,0,0,0,0,0], y_t: yop})
        print("TF accuracy: {0}".format(tf_a))
        print("My accuracy: {0}".format(my_a))
        print()

output:

accuracy_score 1.0
TF accuracy: (0.0, 1.0)
My accuracy: 1.0

accuracy_score 0.8
TF accuracy: (0.0, 0.8)
My accuracy: 0.8

accuracy_score 0.6
TF accuracy: (0.0, 0.6)
My accuracy: 0.6

accuracy_score 0.4
TF accuracy: (0.0, 0.4)
My accuracy: 0.4

Simply moving the tf.local_variables_initializer() inside the loop makes sure that the values in accuracy metric tensor get's re-initialised.

Why does it work?

As per the documentation

The accuracy function creates two local variables, total and count that are used to compute the frequency with which predictions matches labels.

If we don't reinitialise the local variables then value from previous iteration remains in it, leading to wrong results as you were experiencing.

The other approach is to use:

tf.contrib.metrics.accuracy instead of tf.metrics.accuracy. But this gives some residual value at the end like 0.800000011920929 instead of 0.8. It's also deprecated as pointed out by OP in comments.

Source:

https://www.tensorflow.org/api_docs/python/tf/metrics/accuracy

https://github.com/tensorflow/tensorflow/issues/3971

How to properly use tf.metrics.accuracy?

Upvotes: 3

Related Questions