CODEWITHSUNDEEP
tensorflow
user2489252
user2489252

Reputation:

Are there any downsides of creating TensorFlow placeholders for variable sized vs. fixed sized inputs?

I am wondering is there are any substantial downsides (e.g., regarding computational efficiency, memory...) in creating TensorFlow placeholders for variable sized inputs (vs. fixed sized inputs)?

Say, I am doing mini-batch learning and initialize the graph with a placeholder where I assume a fixed batch_size upfront:

tf.placeholder(..., shape=[batch_size, ...])

Alternatively, I can initialize the placeholder variable so that it accepts variable sized inputs:

tf.placeholder(..., shape=[None, ...])

I am not that familiar with the low level tensorflow implementations under the hood, but wouldn't the latter have to check dimensions, allocate memory, and create new arrays at each iteration to account for a case where my minibatch size changes during training? So, depending on the implementation, wouldn't that be computationally wasteful if I am working with a fixed batch dimension?

Upvotes: 9

Views: 887

Answers (1)

mrry
mrry

Reputation: 126154

There is a definite tension between providing fully defined shapes (which can yield better performance) and allowing dimensions to vary (which makes the dataflow graph easier to reuse). As you suspect, there are some downsides to using variable-shaped tf.placeholder() ops to represent input in a TensorFlow model:

  • TensorFlow is often able to simplify the dataflow graph when shapes are fully known. For example, a call to tf.shape(x) returns a tf.Tensor containing the true dynamic shape of a tensor x. If that shape is fully defined at graph construction time, TensorFlow will replace the shape computation with a tf.constant(), and this will be used in constant folding optimizations to reduce the amount of work done at runtime.

  • As an extreme case, the XLA compiler requires that all input tensor shapes be fully defined before code is generated, so that it can generate much more efficient kernel code where array bounds (etc.) are compile-time constants. XLA recompiles the kernel code for each combination of input shapes, so using fixed-size tensors will avoid the recompilation overhead.

  • TensorFlow's memory allocator currently does allocate new arrays for every intermediate and output tensor at each call to tf.Session.run(). However, the underlying memory allocators (the BFC allocator for GPU memory, and tcmalloc or jemalloc for CPU memory) tend to perform better if they have a static distribution of allocation requests (since the requests can be satisfied from buffers that were recently freed).

Upvotes: 7

Related Questions