CODEWITHSUNDEEP
neural-networktensorflowdeep-learningrecurrent-neural-networklstm
anony
anony

Reputation: 13

(tensorflow) Is it possible to combine GridLSTM and MLP/CNN?

I'm trying to train on images using GridLSTM and MLP/CNN.

So basically what I would like to try is to input an image into a GridLSTM and then take its output and feed into MLP/CNN as the next layer.

e.g.:

2-D image -> GridLSTM -> MLP or CNN -> GridLSTM -> MLP or CNN -> Output

I tried running the example code of GridLSTM and the tutorial on Cifar10 (for CNN).

But the input and output for both examples are in a different type (list for GridLSTM and tensor for CNN).

And I was wondering whether it is possible to combine GridLSTM and MLP/CNN at the first place.

Upvotes: 1

Views: 1512

Answers (2)

vu.pham
vu.pham

Reputation: 136

As in "Scene Labeling with LSTM Recurrent Neural Networks", I believe the feedforward layer takes as input the output of LSTM layers at every step. Assuming that the LSTM layers give you a list of N outputs, you will then need to iterate through this list, and apply the feedforward layer to every element of the list.

This is why in Figure 1 of the paper, you see blocks of 3xnxn in the LSTM layer mapped to a block of size 1x1 in the Feedforward layer. You will need to iterate through the whole image in order to compute the whole output for the Feedforward layer. This also explains why they used LSTM and feedforward layers of only a few dozens of units.

Now, how to implement that in Tensorflow is up to you. You can use tf.scan() to iterate through the list of tensor given by LSTM, and apply the feedforward layer on each of the element; or you can concat the tensors in the list with tf.concat() and apply a convolutional operation with suitable sizes. Depending on your network, one way can be faster than the other.

Hope it helps.

Upvotes: 1

Pete Warden
Pete Warden

Reputation: 2878

I reached out to a colleague involved in the GridLSTM implementation, and here is her reply:

You can't. So the LSTM will remove the correlation in time-freq so I don't think you can pass it to a convolutional layer. That being said, you can have multiple grid LSTM layers. See this as an example.

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/06/TFLSTM-1.pdf

Upvotes: 0

Related Questions