CODEWITHSUNDEEP
tensorflowrecurrent-neural-network
Kemp
Kemp

Reputation: 91

Multi-dimension dynamic rnn with tensorflow

In tensorflow's dynamic_rnn function, I was surprised by the output shape and I was hoping someone could help improve my understanding of the RNN cells.

For example, if the input is defined as: 

x = tf.placeholder(tf.float32, [110, seq_size, input_dim])

where seq_size = 5 and input_dim = 2 (ie. two time series) and 110 is the batch size; and the cell is defined as,

cell = rnn_cell.BasicLSTMCell(hidden_dim)

where hidden_dim = 6

When I create a dynamic_rnn 

outputs, states = rnn.dynamic_rnn(cell, x, dtype=tf.float32)

and check the size of output it is [110 5 6]. These dimensions are batch size by seq_size by hidden_dim.

Questions:

1: These dimensions imply there are 6 hidden nodes per time step in the time series for a total of 30 hidden nodes (5 x 6) rather than 6 hidden nodes total. Is this the correct interpretation?

2: Since my input dimension is 5 x 2 (5 steps in each sequence and 2 sequences) how is tensorflow connecting the inputs to the hidden nodes at each time step? Is the tensorflow assuming a fully connected graph with 2 inputs, 12 weights, and 6 biases before each hidden cell? Or something else?

Upvotes: 0

Views: 1291

Answers (1)

Yao Zhang
Yao Zhang

Reputation: 5781

You can think of a sequence as a sentence and an input as a word. The sequence length is the number of words in the sentence, which is also the number of hidden nodes in LSTM; each input/word is corresponding to one hidden node, which maps the input to one output. This is why the number of output is seq_size (5).

A word is a vector, that is positioned in a multi-dimensional space, whose number of dimensions is input_dim. In LSTM, a word is mapped from this input space to a higher dimensional space whose number of dimensions is hidden_dim. This is why the size of each output is hidden_dim (6).

I believe epoch is an irrelevant concept for this discussion. Please see The meaning of batch_size in ptb_word_lm (LSTM model of tensorflow)

Upvotes: 1

Related Questions