Dimension errors in neural network in Keras

Question

I am trying to implement a neural network where I merge/concatenate a fully connected neural network with a convolution neural network. But when I fit the model, I get the following error:

ValueError: All input arrays (x) should have the same number of samples. Got array shapes: [(1, 100, 60, 4500), (100, 4500)]

I have two different inputs:

1. image(dimensions: 1,100,60,4500) where 1 is the channel, 100: # of sample, 60*4500 (dimension of my image). This goes to my convolution neural network

2. positions(dimensions: 100,4500): where 100 refers to samples.

Dimension for my output is 100,2.

The code for my neural network is:

###Convolution neural network
b1 = Sequential()
b1.add(Conv2D(128*2, kernel_size=3,activation='relu',data_format='channels_first',
              input_shape=(100,60,4500)))
b1.add(Conv2D(128*2, kernel_size=3, activation='relu'))
b1.add(Dropout(0.2))

b1.add(Conv2D(128*2, kernel_size=4, activation='relu'))
b1.add(Dropout(0.2))


b1.add(Flatten())
b1.summary()

###Fully connected feed forward neural network
b2 = Sequential()
b2.add(Dense(64, input_shape = (4500,), activation='relu'))
b2.add(Dropout(0.1))
b2.summary()

model = Sequential()

###Concatenating the two networks
concat = concatenate([b1.output, b2.output], axis=-1)

x = Dense(256, activation='relu', kernel_initializer='normal')(concat)
x = Dropout(0.25)(x)
output = Dense(2, activation='softmax')(x)

model = Model([b1.input, b2.input], [output])

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adam(),
              metrics=['accuracy'])

history = model.fit([image, positions], Ytest, batch_size=10,
              epochs=1,
              verbose=1)

Also, the reason why my 'image' array is 4 dimensional is because in the beginning it was just (100,60,4500) but then I ran into the following error:

ValueError: Error when checking input: expected conv2d_10_input to have 4 dimensions, but got array with shape (100, 60, 4500)

And upon googling I found out that it expects # of channels as an input too. And after I added the # of channel, this error went away but then I ran into the other error that I mentioned in the beginning.

So can someone tell me how to solve for the error (the one I specified in the beginning)? Help would be appreciated.

Answer 1

It is not a good practice to mix Sequential and Functional API.

You can implement the model like this

i1 = Input(shape=(1, 60, 4500))

c1 = Conv2D(128*2, kernel_size=3,activation='relu',data_format='channels_first')(i1)
c1 = Conv2D(128*2, kernel_size=3, activation='relu')(c1)
c1 = Dropout(0.2)(c1)
c1 = Conv2D(128*2, kernel_size=4, activation='relu')(c1)
c1 = Dropout(0.2)(c1)
c1 = Flatten()(c1)

i2 = Input(shape=(4500, ))
c2 = Dense(64, input_shape = (4500,), activation='relu')(i2)
c2 = Dropout(0.2)(c2)

c = concatenate([c1, c2])

x = Dense(256, activation='relu', kernel_initializer='normal')(c)
x = Dropout(0.25)(x)
output = Dense(2, activation='softmax')(x)

model = Model([i1, i2], [output])

model.summary()

Note the shape of i1 is shape=(1, 60, 4500). You have set data_format='channels_first' in Conv2D layer hence you need 1 in the beginning.

Compiled the model like this

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

Placeholder data

import numpy as np

X_img = np.zeros((100, 1, 60, 4500))
X_pos = np.ones((100, 4500))
Y = np.zeros((100, 2))

Training

history = model.fit([X_img, X_pos], Y, batch_size=1,
              epochs=1,
              verbose=1)

Answer 2

You number of samples (batch size) should always be the first dimension. So, your data should have shape (100, 1, 60, 4500) for image and (100, 4500) for positions. The argument channels_first for the Conv2D layer means that the channels is the first non-batch dimension.

You also need to change the input shape to (1, 60, 4500) in the first Conv2D layer.