Multiple image input for Keras Application

Question

I'm trying to create a CNN based on the Keras Application DenseNet121 which can take multiple images as input.

I've been able to figure out of how to join multiple ImageDataGenerators to provide the data for the network according to this thread and how to take multiple inputs and feed them to the a single CNN using Keras functional API according to this thread.

Even if I feel like I've done the same thing as theabove threads I still encounter the following error when trying to connect the inputs to the network.

ValueError: Invalid input_shape argument [(None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3), (None, 32, 32, 3)]: model has 1 tensor inputs.

This is the code used for setting up the model with the 11 input images:

in1 = Input(shape=(32,32,3))
in2 = Input(shape=(32,32,3))
in3 = Input(shape=(32,32,3))
in4 = Input(shape=(32,32,3))
in5 = Input(shape=(32,32,3))
in6 = Input(shape=(32,32,3))
in7 = Input(shape=(32,32,3))
in8 = Input(shape=(32,32,3))
in9 = Input(shape=(32,32,3))
in10 = Input(shape=(32,32,3))
in11 = Input(shape=(32,32,3))
inputs = [in1,in2,in3,in4,in5,in6,in7,in8,in9,in10,in11]

densenet_121_model = DenseNet121(include_top=False, weights='imagenet', input_shape=input_shape, pooling='avg')
model_base = densenet_121_model(inputs)
output = Dense(units=n_output_units, activation=activation_fn)(model_base)
model = Model(inputs=inputs, outputs=output)

I'm aware that this technically violates the input_shape parameter for densenet121, but that also seems to be the case in thushv89's accepted answer to the second thread referenced above.

I cite the code they recommended below for convenience.

from tensorflow.python.keras import layers, models, applications

# Multiple inputs
in1 = layers.Input(shape=(128,128,3))
in2 = layers.Input(shape=(128,128,3))
in3 = layers.Input(shape=(128,128,3))

# CNN output
cnn = applications.xception.Xception(include_top=False)


out1 = cnn(in1)
out2 = cnn(in2)
out3 = cnn(in3)

# Flattening the output for the dense layer
fout1 = layers.Flatten()(out1)
fout2 = layers.Flatten()(out2)
fout3 = layers.Flatten()(out3)

# Getting the dense output
dense = layers.Dense(100, activation='softmax')

dout1 = dense(fout1)
dout2 = dense(fout2)
dout3 = dense(fout3)

# Concatenating the final output
out = layers.Concatenate(axis=-1)([dout1, dout2, dout3])

# Creating the model
model = models.Model(inputs=[in1,in2,in3], outputs=out)
model.summary()```

Any help would be greatly appreciated!

Answer 1

Here I have built DenseNet121 with 3 inputs with the help of inputs you have mentioned in the question. Can you check if this is what you were looking for.

%tensorflow_version 1.x
import tensorflow
import keras
from keras import Input, Model
from keras.applications.densenet import DenseNet121
from keras.layers import Dense, Flatten, Concatenate
from keras.activations import relu

# Multiple inputs
in1 = Input(shape=(128,128,3))
in2 = Input(shape=(128,128,3))
in3 = Input(shape=(128,128,3))

# CNN output
cnn = DenseNet121(include_top=False)
#cnn.summary()

out1 = cnn(in1)
out2 = cnn(in2)
out3 = cnn(in3)

# Flattening the output for the dense layer
fout1 = Flatten()(out1)
fout2 = Flatten()(out2)
fout3 = Flatten()(out3)

# Getting the dense output
dense = Dense(1, activation='softmax')

dout1 = dense(fout1)
dout2 = dense(fout2)
dout3 = dense(fout3)

# Concatenating the final output
out = Concatenate(axis=-1)([dout1, dout2, dout3])

# Creating the model
model = Model(inputs=[in1,in2,in3], outputs=out)
model.summary()

Output - Summary of the model look like this.

TensorFlow is already loaded. Please restart the runtime to change versions.
Model: "model_2"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_306 (InputLayer)          (None, 128, 128, 3)  0                                            
__________________________________________________________________________________________________
input_307 (InputLayer)          (None, 128, 128, 3)  0                                            
__________________________________________________________________________________________________
input_308 (InputLayer)          (None, 128, 128, 3)  0                                            
__________________________________________________________________________________________________
densenet121 (Model)             multiple             7037504     input_306[0][0]                  
                                                                 input_307[0][0]                  
                                                                 input_308[0][0]                  
__________________________________________________________________________________________________
flatten_1 (Flatten)             (None, 16384)        0           densenet121[1][0]                
__________________________________________________________________________________________________
flatten_2 (Flatten)             (None, 16384)        0           densenet121[2][0]                
__________________________________________________________________________________________________
flatten_3 (Flatten)             (None, 16384)        0           densenet121[3][0]                
__________________________________________________________________________________________________
dense_12 (Dense)                (None, 1)            16385       flatten_1[0][0]                  
                                                                 flatten_2[0][0]                  
                                                                 flatten_3[0][0]                  
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 3)            0           dense_12[0][0]                   
                                                                 dense_12[1][0]                   
                                                                 dense_12[2][0]                   
==================================================================================================
Total params: 7,053,889
Trainable params: 6,970,241
Non-trainable params: 83,648
__________________________________________________________________________________________________