CODEWITHSUNDEEP
mergekeras
user2179331
user2179331

Reputation: 97

Merging two tensors by convolution in Keras

I'm trying to convolve two 1D tensors in Keras. I get two inputs from other models:

  1. x - of length 100
  2. ker - of length 5

I would like to get the 1D convolution of x using the kernel ker. I wrote a Lambda layer to do it:

import tensorflow as tf
def convolve1d(x):
    y = tf.nn.conv1d(value=x[0], filters=x[1], padding='VALID', stride=1)
    return y

x = Input(shape=(100,))
ker = Input(shape=(5,))
y = Lambda(convolve1d)([x,ker])
model = Model([x,ker], [y])

I get the following error:

ValueError: Shape must be rank 4 but is rank 3 for 'lambda_67/conv1d/Conv2D' (op: 'Conv2D') with input shapes: [?,1,100], [1,?,5].

Can anyone help me understand how to fix it?

Upvotes: 1

Views: 1345

Answers (3)

Zhu Tiny
Zhu Tiny

Reputation: 1

I guess I have understood what you mean. Given the wrong example code below:

input_signal = Input(shape=(L), name='input_signal')
input_h = Input(shape=(N), name='input_h')
faded= Lambda(lambda x: tf.nn.conv1d(input, x))(input_h)

You want to convolute each signal vector with different fading coefficients vector. The 'conv' operation in TensorFlow, etc. tf.nn.conv1d, only support a fixed value kernel. Therefore, the code above can not run as you want.

I have no idea, too. The code you given can run normally, however, it is too complex and not efficient. In my idea, another feasible but also inefficient way is to multiply with the Toeplitz matrix whose row vector is the shifted fading coefficients vector. When the signal vector is too long, the matrix will be extremely large.

Upvotes: 0

Mitiku
Mitiku

Reputation: 5412

From given code it is difficult to point out what you mean when you say

is it possible

But if what you mean is to merge two layers and feed merged layer to convulation, yes it is possible. 

x = Input(shape=(100,))
ker = Input(shape=(5,))
merged = keras.layers.concatenate([x,ker], axis=-1)
y = K.conv1d(merged, 'same')
model = Model([x,ker], y)

EDIT:

@user2179331 thanks for clarifying your intention. Now you are using Lambda Class incorrectly, that is why the error message is showing. But what you are trying to do can be achieved using keras.backend layers. Though be noted that when using lower level layers you will lose some higher level abstraction. E.g when using keras.backend.conv1d you need to have input shape of (BATCH_SIZE,width, channels) and kernel with shape of (kernel_size,input_channels,output_channels). So in your case let as assume the x has channels of 1(input channels ==1) and y also have the same number of channels(output channels == 1).

So your code now can be refactored as follows

from keras import backend as K
def convolve1d(x,kernel):
    y = K.conv1d(x,kernel, padding='valid', strides=1,data_format="channels_last")
    return y
input_channels = 1
output_channels = 1
kernel_width = 5
input_width = 100
ker = K.variable(K.random_uniform([kernel_width,input_channels,output_channels]),K.floatx())
x = Input(shape=(input_width,input_channels)
y = convolve1d(x,ker)

Upvotes: 1

mpariente
mpariente

Reputation: 660

It was much harder than I expected because Keras and Tensorflow don't expect any batch dimension in the convolution kernel so I had to write the loop over the batch dimension myself, which requires to specify batch_shape instead of just shape in the Input layer. Here it is :

import numpy as np
import tensorflow as tf
import keras
from keras import backend as K
from keras import Input, Model
from keras.layers import Lambda

def convolve1d(x):
    input, kernel = x
    output_list = []
    if K.image_data_format() == 'channels_last':
        kernel = K.expand_dims(kernel, axis=-2)
    else:
        kernel = K.expand_dims(kernel, axis=0)
    for i in range(batch_size): # Loop over batch dimension
        output_temp = tf.nn.conv1d(value=input[i:i+1, :, :],
                                   filters=kernel[i, :, :],
                                   padding='VALID',
                                   stride=1)
        output_list.append(output_temp)
        print(K.int_shape(output_temp))
    return K.concatenate(output_list, axis=0)

batch_input_shape = (1, 100, 1)
batch_kernel_shape = (1, 5, 1)

x = Input(batch_shape=batch_input_shape)
ker = Input(batch_shape=batch_kernel_shape)
y = Lambda(convolve1d)([x,ker])
model = Model([x, ker], [y])

a = np.ones(batch_input_shape)
b = np.ones(batch_kernel_shape)
c = model.predict([a, b])

In the current state :

  • It doesn't work for inputs (x) with multiple channels.
  • If you provide several filters, you get as many outputs, each being the convolution of the input with the corresponding kernel.

Upvotes: 1

Related Questions