CODEWITHSUNDEEP
tensorflowmachine-learningneural-networkdeep-learningconv-neural-network
ivan
ivan

Reputation: 111

Tensorflow Dataset/Iterator for evaluating with train and test data in CNN

The setting: I want to predict a value by training a CNN on input batches in a regression setting. I also want to evaluate and calculate the loss after every epoch, so I need to switch between datasets at runtime.

Input: [num_examples, height, width, channels] -> [num_examples, y]

I want to use the new Dataset API because I want to avoid feeding batches myself during the training.

I also don't want to store my dataset in the computation graph because the Dataset is bigger than 2GB, but small enough to be stored in memory.

This is my current setup:

def initialize_datasets(x, y,...):
    dataset_train = tf.data.Dataset.from_tensor_slices((x, y))
    dataset_train = dataset_train.apply(tf.contrib.data.shuffle_and_repeat(buffer_size=examples_train, count=epochs))
    dataset_train = dataset_train.batch(batch_size)

    dataset_test = tf.data.Dataset.from_tensor_slices((x, y))
    dataset_test = dataset_test.apply(tf.contrib.data.shuffle_and_repeat(buffer_size=examples_test, count=-1))
    dataset_test = dataset_test.batch(batch_size)

    # Iterator 
    iterator_train = dataset_train.make_initializable_iterator()
    iterator_test = dataset_test.make_initializable_iterator()

    return iterator_train, iterator_test


def get_input_batch_data(testing, iterator_train, iterator_test):
    features, labels = tf.cond(testing, lambda: iterator_test.get_next(), lambda: iterator_train.get_next())
    return features, labels

Then in my model() function:

#1
iterator_train, iterator_test = initialize_datasets(x, y, ...)
#2
features, labels = get_input_batch_data(testing, iterator_train, 
iterator_test)

# forward pass, loss, etc
...

with tf.Session as sess:
   #initialize with train data, trainX[num_examples, height, width, channels]
    sess.run(iterator_train.initializer, feed_dict={x: trainX, y: trainY, 
    batch_size: batchsize})

   #initialize with test data
    sess.run(iterator_test.initializer, feed_dict={x: testX, y: testY, 
    batch_size: NUM_EXAMPLES_TEST})

for i in range(EPOCHS)
    for j in range(NUM_BATCHES)
        _, batch_loss = sess.run([train_step, loss], feed_dict={testing: 
              False,  i: iters_total, pkeep: p_keep})   

    # after 1 epoch, calculate loss and whole test data set
    epoch_test_loss = sess.run(loss, feed_dict={testing: True, i: 
                    iters_total, pkeep: 1})

This is the output:

Iter: 44, Epoch: 0 (8.46s), Train-Loss: 103011.18, Test-Loss: 100162.34
Iter: 89, Epoch: 1 (4.17s), Train-Loss: 93699.51, Test-Loss: 92130.21
Iter: 134, Epoch: 2 (4.13s), Train-Loss: 90217.82, Test-Loss: 88978.74
Iter: 179, Epoch: 3 (4.14s), Train-Loss: 88503.13, Test-Loss: 87515.81
Iter: 224, Epoch: 4 (4.18s), Train-Loss: 87336.62, Test-Loss: 86486.40
Iter: 269, Epoch: 5 (4.10s), Train-Loss: 86388.38, Test-Loss: 85637.64
Iter: 314, Epoch: 6 (4.14s), Train-Loss: 85534.52, Test-Loss: 84858.43
Iter: 359, Epoch: 7 (4.29s), Train-Loss: 84693.19, Test-Loss: 84074.78
Iter: 404, Epoch: 8 (4.20s), Train-Loss: 83973.64, Test-Loss: 83314.47
Iter: 449, Epoch: 9 (4.40s), Train-Loss: 83149.73, Test-Loss: 82541.73

Problems:

I also uploaded the whole model here: https://github.com/toemm/TF-CNN-regression/blob/master/BA-CNN_so.ipynb

Upvotes: 1

Views: 1842

Answers (1)

Patwie
Patwie

Reputation: 4460

The obvious answer is: You do not want to do this within the same graph as the evaluation graph is different from the training graph.

  • Dropout has fixed multiplier (no sampling)
  • BatchNorm uses accumulated statistics and does not update the EMA

So the solution is really to build two different things like

import numpy as np
import tensorflow as tf


X_train = tf.constant(np.ones((100, 2)), 'float32')
X_val = tf.constant(np.zeros((10, 2)), 'float32')

iter_train = tf.data.Dataset.from_tensor_slices(
    X_train).make_initializable_iterator()
iter_val = tf.data.Dataset.from_tensor_slices(
    X_val).make_initializable_iterator()


def graph(x, is_train=True):
  return x


output_train = graph(iter_train.get_next(), is_train=True)
output_val = graph(iter_val.get_next(), is_train=False)

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  sess.run(iter_train.initializer)
  sess.run(iter_val.initializer)

  for train_iter in range(100):
    print(sess.run(output_train))

  for train_iter in range(10):
    print(sess.run(output_val))

Upvotes: 4

Related Questions