CODEWITHSUNDEEP
pythontensorflow
Vince Gatto
Vince Gatto

Reputation: 415

Feeding tensors for training vs validation data

In the tensorflow examples, feed_dict is used to send either training or validation input into the same model graph. Unfortunately, you cannot feed tensors:

    Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.

I've been using an input pipeline and TFRecordReader, so my data never really enters python. Having to call run to get the data into python just to feed it back to tensorflow seems silly and is definitely slow.

Does anyone have a good solution for this?

Currently, I just create two identical copies of the model subgraph that use the same parameters. This works, but forces me to organize my code in an odd way.

EDIT

For example, I'm currently doing something like:

model_params = BuildModelParams()
train_model = BuildModel(model_params, train_input)
test_model = BuildModel(model_params, test_input)

so that the test model uses the parameters learned by training. The nice thing about feed_dict is that I only need to define the model once and I do not have to separate the model's parameters from its structure.

Upvotes: 7

Views: 1593

Answers (2)

mdaoust
mdaoust

Reputation: 6367

Warning:

This solution can cause significant problems when input queues are involved. See: https://groups.google.com/a/tensorflow.org/forum/#!msg/discuss/mLrt5qc9_uU/sGNbC7GpAwAJ

Thanks to @fwalch for pointing this out in the comments

There's no way to do exactly what you're asking, see the answer to my question here.

But the newly public "cond" from version 0.7 can fill your use case:

# Here are the two data streams.
train_data = tf.Variable(999)
test_data = tf.Variable(1000)

# This selects which stream to use.
select_test = tf.placeholder(dtype=bool,shape=[],name='select_test')
data = tf.cond(
    select_test,
    lambda:test_data,
    lambda:train_data
)

# Here is the model.
model = data-500;

init = tf.initialize_all_variables()
with tf.Session():
    init.run()

    # You just have to feed `select_test` when you evaluate the model.
    print(model.eval({select_test:False})) #499
    print(model.eval({select_test:True})) #500

You can use the same sort of trick for switching Batch Normalization to use a moving average during test.

Upvotes: 8

erik-e
erik-e

Reputation: 3891

TL;DR

You can use your input pipeline with the train, test and validation step instead of using a placeholder. When you already have a Tensor then I don't see a reason not to use it.

Relevant Documentation

Further information can be found on TF's site.

  1. Feeding Data from Python
  2. Placeholders

Example

In this example, we'll send the model and expected output to train and test functions. The only difference is that for some we use a placeholder and others we're using Tensors.

import tensorflow as tf


def train(model, T):
    """Example train operation which returns sum of parameters."""
    return model + T


def test(model, T):
    """Example test operation which returns parameters multiplied."""
    return model * T


# Placeholders which will be required in the feed_dict once we execute a TF_Run.
x_placeholder_model = tf.placeholder(tf.int32)
t_placeholder = tf.placeholder(tf.int32)

# Tensors, using constants for brevity but these could be from an input pipeline
#  or some other set of operations.
x_tensor_model = tf.constant([1, 2, 3])
t_tensor = tf.constant([1, 2, 3])

using_placeholder_train = train(x_placeholder_model, t_placeholder)
using_tensor_train = train(x_tensor_model, t_tensor)

using_placeholder_test = test(x_placeholder_model, t_placeholder)
using_tensor_test = test(x_tensor_model, t_tensor)

with tf.Session() as sess:
    print(sess.run(
        using_placeholder_train, feed_dict={
            x_placeholder_model: [1, 2, 3],
            t_placeholder: [1, 2, 3]}))

    print(sess.run(
        using_placeholder_test, feed_dict={
            x_placeholder_model: [1, 2, 3],
            t_placeholder: [1, 2, 3]}))

    print(sess.run(using_tensor_train))
    print(sess.run(using_tensor_test))

Executing this code will result in the following output:

[2 4 6]
[1 4 9]
[2 4 6]
[1 4 9]

Illustrating that calls to:

print(sess.run(
    using_placeholder_train, feed_dict={
        x_placeholder_model: [1, 2, 3],
        t_placeholder: [1, 2, 3]}))

Return the same output as:

print(sess.run(using_tensor_train))

Without any additions to the feed_dict because we already have the tensors available on the graph.

Upvotes: 1

Related Questions