placeholders definition for nd-array input in tensorflow

Question

I'm trying to build LSTM RNN based on this guide: http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/ My input is ndarray with the size of 89102*39 (89102 rows, 39 features). There are 3 labels for the data - 0,1,2 It seems like I'm having a problem with the placeholders definition but I'm not sure what it is.

My code is:

    data = tf.placeholder(tf.float32, [None, 1000, 39])
    target = tf.placeholder(tf.float32, [None, 3])
    cell = tf.nn.rnn_cell.LSTMCell(self.num_hidden)

    val, state = tf.nn.dynamic_rnn(cell, data, dtype=tf.float32)
    val = tf.transpose(val, [1, 0, 2])
    last = tf.gather(val, int(val.get_shape()[0]) - 1)


    weight = tf.Variable(tf.truncated_normal([self.num_hidden, int(target.get_shape()[1])]))
    bias = tf.Variable(tf.constant(0.1, shape=[target.get_shape()[1]]))

    prediction = tf.nn.softmax(tf.matmul(last, weight) + bias)

    cross_entropy = -tf.reduce_sum(target * tf.log(tf.clip_by_value(prediction, 1e-10, 1.0)))

    optimizer = tf.train.AdamOptimizer()
    minimize = optimizer.minimize(cross_entropy)

    mistakes = tf.not_equal(tf.argmax(target, 1), tf.argmax(prediction, 1))
    error = tf.reduce_mean(tf.cast(mistakes, tf.float32))


    init_op = tf.initialize_all_variables()
    sess = tf.Session()
    sess.run(init_op)
    batch_size = 1000
    no_of_batches = int(len(train_input) / batch_size)
    epoch = 5000
    for i in range(epoch):
        ptr = 0
        for j in range(no_of_batches):
            inp, out = train_input[ptr:ptr + batch_size], train_output[ptr:ptr + batch_size]
            ptr += batch_size
            sess.run(minimize, {data: inp, target: out})
        print( "Epoch - ", str(i))

And I'm getting to following error:

File , line 133, in execute_graph
sess.run(minimize, {data: inp, target: out})

  File "/usr/local/lib/python3.5/dist-
packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)

  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 975, in _run
    % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))

ValueError: Cannot feed value of shape (1000, 39) for Tensor 'Placeholder:0', which has shape '(1000, 89102, 39)'

Any idea what might be causing the problem?

Answer 1

As indicated here, The dynamic_rnn function takes the batch inputs of shape

[batch_size, truncated_backprop_length, input_size]

In the link that you provided, the shape of the placeholder was

data = tf.placeholder(tf.float32, [None, 20,1]) 

This means that they chose truncated_backprop_length=20 and input_size=1.
Their data was the following 3D array:

[
 array([[0],[0],[1],[0],[0],[1],[0],[1],[1],[0],[0],[0],[1],[1],[1],[1],[1],[1],[0],[0]]), 
 array([[1],[1],[0],[0],[0],[0],[1],[1],[1],[1],[1],[0],[0],[1],[0],[0],[0],[1],[0],[1]]), 
 .....
]

Based on your code, it seems that train_input is a 2D array and not a 3D array. Hence, you need to transform it into a 3D array. In order to do that, you need to decide which parameters you want to use for truncated_backprop_length and input_size. Afterwards, you need to define data appropriately.

For example, if you want truncated_backprop_length and input_size to be 39 and 1 respectively, you can do

import numpy as np
train_input=np.reshape(train_input,(len(train_input),39,1))
data = tf.placeholder(tf.float32, [None, 39,1]) 

I changed your code according to the above discussion and run it on some random data that I produced. It runs without throwing an error. See the code below:

import tensorflow as tf
import numpy as np
num_hidden=5
train_input=np.random.rand(89102,39)
train_input=np.reshape(train_input,(len(train_input),39,1))
train_output=np.random.rand(89102,3)

data = tf.placeholder(tf.float32, [None, 39, 1])
target = tf.placeholder(tf.float32, [None, 3])
cell = tf.nn.rnn_cell.LSTMCell(num_hidden)

val, state = tf.nn.dynamic_rnn(cell, data, dtype=tf.float32)
val = tf.transpose(val, [1, 0, 2])
last = tf.gather(val, int(val.get_shape()[0]) - 1)


weight = tf.Variable(tf.truncated_normal([num_hidden, int(target.get_shape()[1])]))
bias = tf.Variable(tf.constant(0.1, shape=[target.get_shape()[1]]))

prediction = tf.nn.softmax(tf.matmul(last, weight) + bias)

cross_entropy = -tf.reduce_sum(target * tf.log(tf.clip_by_value(prediction, 1e-10, 1.0)))

optimizer = tf.train.AdamOptimizer()
minimize = optimizer.minimize(cross_entropy)

mistakes = tf.not_equal(tf.argmax(target, 1), tf.argmax(prediction, 1))
error = tf.reduce_mean(tf.cast(mistakes, tf.float32))


init_op = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init_op)
batch_size = 1000
no_of_batches = int(len(train_input) / batch_size)
epoch = 5000
for i in range(epoch):
    ptr = 0
    for j in range(no_of_batches):
        inp, out = train_input[ptr:ptr + batch_size], train_output[ptr:ptr + batch_size]
        ptr += batch_size
        sess.run(minimize, {data: inp, target: out})
    print( "Epoch - ", str(i))