CODEWITHSUNDEEP
pythonmachine-learningkeras
Кирилл Малышев
Кирилл Малышев

Reputation: 163

How to apply two sequences of images to the input of the model?

I have a set of sequences of left and right images. I need to feed two such sequences to the model input. I figured out how to input one sequence:

seq = Sequential()
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   input_shape=(None, 150, 150, 1),
                   padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   padding='same', return_sequences=True))
seq.add(BatchNormalization())
seq.add(Conv3D(filters=1, kernel_size=(3, 3, 3),
               activation='sigmoid',
               padding='same', data_format='channels_last'))

What needs to be changed to introduce two sequences, i.e. an array with shape (2, 150, 150, 1)?

Upvotes: 0

Views: 161

Answers (1)

GeorgPoe
GeorgPoe

Reputation: 352

I think you could just go with setting the second image to the second channel like this (input_shape=(None, 150, 150, 2)):

seq.add(ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   input_shape=(None, 150, 150, 2),
                   padding='same', return_sequences=True))

However, if you want to calculate 2 different streams: In Keras, there are 2 ways of setting up the network: Via the "Sequential" and "Functional" APIs. While Sequential covers the vast majority of the use-cases that you will ever have, Functional is more flexible and allows, well, non-sequential computation structures, i.e. where one layer's output is not always the only input to the next layer. In your case, you need 2 Inputs and some layers covering each of the two images, that are joined at some later stage. So, what you need is a Network generated by the Functional API.

https://keras.io/getting-started/functional-api-guide/

You will need 2 "streams" that feed into your network. Note the slightly different syntax for adding layers.

import keras
from keras.models import Model
from keras.layers import Dense, ConvLSTM2D, Input, Concatenate, BatchNormalization

input1 = Input(shape=[150, 150, 1])
nn1 = ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   input_shape=(None, 150, 150, 1),
                   padding='same', return_sequences=True)(input1)
nn1 = BatchNormalization()(nn1)
... # and so on

# compute 2nd stream
input2 = Input(shape=[150, 150, 1])
nn2 = ConvLSTM2D(filters=40, kernel_size=(3, 3),
                   input_shape=(None, 150, 150, 1),
                   padding='same', return_sequences=True)(input2)
nn2 = BatchNormalization()(nn2)
... # and so on

# finally, join the nn1 and nn2 streams into nn
nn = Concatenate()([nn1, nn2])
# and work with this new nn layer
nn = Dense(64, activation='relu')(nn)

# finally, compile the model...
model = Model([input1, input2], nn)

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

# and fit it, note how it now receives two inputs
model.fit([pix_train_1, 
           pix_train_2], 
           y, ...
         )

Upvotes: 1

Related Questions