How do you pass video features from a CNN to an LSTM?

Question

After you pass a video frame through a convnet and get an output feature map, how do you pass that data into an LSTM? Also, how do you pass multiple frames to the LSTM thru the CNN?
In other works I want to process video frames with an CNN to get the spatial features. Then I want pass these features to an LSTM to do temporal processing on the spatial features. How do I connect the LSTM to the video features? For example if the input video is 56x56 and then when passed through all of the CNN layers, say it comes out as 20: 5x5's. How are these connected to the LSTM on a frame by frame basis? ANd shoudl they go through a fully connected layer first? Thanks, Jon

Answer 1

The architecture of cnn+lstm model will look like the below diagram Basically you have to create time distributed wrapper for CNN layer and then pass the output of CNN to the LSTM layer

cnn_input= Input(shape=(3,200,100,1))   #Frames,height,width,channel of imafe
conv1 = TimeDistributed(Conv2D(32, kernel_size=(50,5),    activation='relu'))(cnn_input)
conv2 = TimeDistributed(Conv2D(32, kernel_size=(20,5), activation='relu'))(conv1)
pool1=TimeDistributed(MaxPooling2D(pool_size=(4,4)))(conv2)
flat=TimeDistributed(Flatten())(pool1)
cnn_op= TimeDistributed(Dense(100))(flat)

After this you can pass your CNN output to LSTM

lstm = LSTM(128, return_sequences=True, activation='tanh')(merged)
op =TimeDistributed(Dense(100))(lstm)
fun_model = Model(inputs=[cnn_input], outputs=op)

please remember the input to this time distributed CNN must be (# of frames,row_size,column_size,channels)

And Finally you can apply softmax at the last layer to get some predictions

Answer 2

Basically, you can flatten each frame features and feed them into one LSTM cell. With CNN, it's the same. You can feed each output of CNN into one LSTM cell.

For FC, it's up to you.

See a network structure from http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-180.pdf.