CODEWITHSUNDEEP
python-3.xnumpymachine-learningtensorflowconv-neural-network
buydadip
buydadip

Reputation: 9407

Evaluating prediction extremely slow

I am training a CNN using the following placeholders...

# mnist data image of shape [batch_size, 28, 28, 1].
x = tf.placeholder(tf.float32, [None, 28, 28, 1])

# 0-9 digits recognition => 10 classes.
y = tf.placeholder(tf.float32, [None, 10])

I train it within a session, and as I must reshape the data every time before I feed the data to the placeholder, here is roughly what I am doing...

with tf.Session() as sess:

    for epoch in range(25):

        total_batch = int(features.train.num_examples/500)

        avg_cost = 0

        for i in range(total_batch):

            batch_xs, batch_ys = features.train.next_batch(10)

            _, c = sess.run([train_op, loss], feed_dict={x:batch_xs.reshape([-1, 28, 28, 1]), y:batch_ys})

   ...
   ...

As you can see I am reshaping the data before I feed it to the placeholder. The CNN seems to work fine, however an issue arises when I try to calculate the accuracy of the prediction. I am predicting it using a softmax tensor and my logits layer...

# Logits Layer.
# Create a dense layer with 10 neurons => 10 classes
# Output has a shape of [batch_size, 10]
logits = tf.layers.dense(inputs=dropout, units=10)

# Softmax layer for deriving probabilities.
pred = tf.nn.softmax(logits, name="softmax_tensor")

...so after all the training take place, this is how I calculate the predictions...

from tensorflow.examples.tutorials.mnist import input_data
...
...
features = input_data.read_data_sets("/tmp/data/", one_hot=True)
...
...

# list of booleans to determine the correct predictions
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))

print(correct_prediction.eval({x:features.test.images.reshape([10000, 28, 28, 1]), y:features.test.labels}))

Unfortunately, I must convert every features data into the correct format since the placeholder x only accepts the format [batch_size, 28, 28, 1]. There are 10,000 images and labels. When I do this, my computer slows down tremendously and a lot of computation is used, and I need to restart my computer every time. Clearly converting them into this format is not very computationally friendly.

What's a better way to do this that doesn't cause my computer to crash? I've used the following code for other Neural Networks...

print(correct_prediction.eval({x:features.test.images, y:features.test.labels}))

...but I've never had this issue which leads me to believe the cause is stemming from the reshape function.

Upvotes: 1

Views: 1099

Answers (1)

Maxim
Maxim

Reputation: 53758

I might need to take a look at the whole script, but from the provided snippets, I think the issue is that you're pushing the whole test set for evaluation at once.

Note that not only it allocates 10000*28*28*4 (30Mb) for the input tensor and its pooled transformations, the convolutional layer performs approx 10000*25*25*1*filter_size convolutions across the whole batch (the first layer alone). It is very computationally expensive, and that's what most probably takes the whole time, not reshape.

You should perform test evaluation by batch. This is simplified code, need to be more careful to compute it correctly with arbitrary batch_size:

total_accuracy = 0.0
for batch_xs, batch_ys in iterate_batches(features.test, batch_size):
  accuracy = correct_prediction.eval({x: batch_xs.reshape([-1, 28, 28, 1]), y: batch_ys})
  total_accuracy += np.mean(accuracy)
total_batches = features.test.num_examples / batch_size
total_accuracy /= total_batches

But it should work for batch_size=10 or batch_size=100.

Upvotes: 2

Related Questions