How Can I Extract Predictions from A Softmax Layer on Tensorflow

Question

I'm trying to extract predictions, use predictions in calculating accuracy/precision/recall/F1 and prediction probability. I know I have 10 output classes therefore I can't calculate precision per see but I will be doing all these in other models moreover I'd like to be able to extract prediction probabilities. My model is as follows. I've checked GitHub and StackOverflow however I have yet to find a way to extract those properties. Most of the answers come close but never answer what I needed. I've used some low epoch numbers there in order to check out model fast and keep the output screen less crowded.

import tensorflow as tf
from tensorflow.contrib.layers import fully_connected

from sklearn.datasets import fetch_mldata
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split

mnist = fetch_mldata('MNIST original', data_home="data/mnist/")
lb = LabelBinarizer().fit(mnist.target)
X_train, X_test, y_train, y_test = train_test_split(mnist.data, lb.transform(mnist.target), train_size=0.9, test_size=0.1)

X = tf.placeholder(tf.float32, shape=(None, 784))
y = tf.placeholder(tf.int64, shape=(None, 10))

lOne = fully_connected(inputs=X, num_outputs=100, activation_fn=tf.nn.elu)
logits = fully_connected(inputs=lOne, num_outputs=10, activation_fn=tf.nn.softmax)

pred = logits
acc = tf.metrics.accuracy(labels=y, predictions=pred)

loss = tf.losses.softmax_cross_entropy(logits=logits, onehot_labels=y)
trainOP = tf.train.AdamOptimizer(0.001).minimize(loss)

import numpy as np
bSize = 100
batches = int(np.floor(X_train.shape[0]/bSize)+1)
def batcher(dSet, bNum):
    return(dSet[bSize*(bNum-1):bSize*(bNum)])

epochs = 2
init = tf.global_variables_initializer()
with tf.Session() as sess:
    sess.run(init)
    for epoch in range(0, epochs):
        for batch in range(1, batches):
            X_batch = batcher(X_train, batch)
            y_batch = batcher(y_train, batch)
            sess.run(trainOP, feed_dict={X: X_batch, y: y_batch})
        lossVal = sess.run([loss], feed_dict={X: X_test, y: y_test})
        print(lossVal)


    sess.close()

Answer 1

I'm writing in case anyone may stumble upon this particular case. I've built a network following basic MNIST examples, I've used tf.nn.softmax in the final layer and expected to get results from said layer. It looks like I need to use softmax function again to get the results from a layer such as yPred = tf.nn.softmax(logits) with logits being the name of the output layer. I'm adding fixed code below.

I can add a line to save the model, load it later on and made predictions on saved model. Since this is just an example for me building the model, I've omitted the saving part.

import tensorflow as tf
from tensorflow.contrib.layers import fully_connected

from sklearn.datasets import fetch_mldata
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split

mnist = fetch_mldata('MNIST original', data_home="data/mnist/")
lb = LabelBinarizer().fit(mnist.target)
X_train, X_test, y_train, y_test = train_test_split(mnist.data, lb.transform(mnist.target), train_size=0.9, test_size=0.1, stratify = mnist.target, random_state=42)

X = tf.placeholder(tf.float32, shape=(None, 784))
y = tf.placeholder(tf.int64, shape=(None, 10))


lOne = fully_connected(inputs=X, num_outputs=100, activation_fn=tf.nn.elu)
lTwo = fully_connected(inputs=lOne, num_outputs=100, activation_fn=tf.nn.elu)
logits = fully_connected(inputs=lTwo, num_outputs=10, activation_fn=tf.nn.softmax)

pred = tf.nn.softmax(logits)
acc_bool = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
acc_Num = tf.cast(acc_bool, tf.float32)
acc_Mean = tf.reduce_mean(acc_Num)

loss = tf.losses.softmax_cross_entropy(logits=logits, onehot_labels=y)
trainOP = tf.train.AdamOptimizer(0.001).minimize(loss)


import numpy as np
bSize = 1024
batches = int(np.floor(X_train.shape[0]/bSize)+1)
def batcher(dSet, bNum):
    return(dSet[bSize*(bNum-1):bSize*(bNum)])



epochs = 250
init = tf.global_variables_initializer()
trainA = []
testA = []

with tf.Session() as sess:
sess.run(init)
for epoch in range(0, epochs):
    for batch in range(1, batches):
        X_batch = batcher(X_train, batch)
        y_batch = batcher(y_train, batch)
        sess.run(trainOP, feed_dict={X: X_batch, y: y_batch})
    if epoch % 25 == 1:
        trainLoss, trainAcc = sess.run([loss, acc_Mean], feed_dict={X: X_train, y: y_train})
        testLoss, testAcc = sess.run([loss, acc_Mean], feed_dict={X: X_test, y: y_test})
        yPred = sess.run(pred, feed_dict={X: X_test[0].reshape(1,-1), y: y_test[0].reshape(1,-1)})
        print(yPred)

sess.close()

Answer 2

The code shared in the question covers training, but not "using" (infering) with the resulting model.

Two issues:

• The trained model is not serialized, so future runs will run on an untrained model, and predict whatever their initialization tells them to. Hence a question comment suggesting to save the trained model, and restore it when predicting.
• The logits are the output of a SoftMax function. A common way to get a class from logits is to select the highest value in the tensor (here a vector).

With TensorFlow, the last point can be done with tf.argmax ("Returns the index with the largest value across axes of a tensor."):

tf.argmax(input=logits, axis=1)

All in all, the question's code covers only partially the MNIST tutorial from the TensorFlow team. Perhaps more pointers there if you get stuck with this code.