CODEWITHSUNDEEP
pythontensorflowmachine-learningkerasdeep-learning
user14349917
user14349917

Reputation:

Is RepeatVector layer the same as Concatenate layer?

Here was my code:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector

Model = Sequential([
              LSTM(2, input_shape=(3,2)),
              RepeatVector(5)])

inputs = [[[1,2], [3,4], [5,6]]]
print(Model.predict(inputs))

which produces the output:

[[[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]
[0.03203796 0.00486411]]]

Now, it seems that adding Repeat Vector just repeated the same output n times. If RepeatVector just repeats the output n times, does this mean that RepeatVector(n) is the same as Concatenating the output to itself n times?

Upvotes: 1

Views: 1101

Answers (2)

Nicolas Gervais
Nicolas Gervais

Reputation: 36684

According to the docs of RepeatVector and concatenate, one of the differences is the output shape.

For RepeatVector, the output shape is: (num_samples, n, features)

For concatenate, the output shape is: (num_samples, features*n) (if axis=-1)

Let's visualize that. Here's some dummy input:

import tensorflow as tf

x = tf.reshape(tf.range(1, 10), (3, 3))

<tf.Tensor: shape=(3, 3), dtype=int32, numpy=
array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])>

Let's repeat the vector:

tf.keras.layers.RepeatVector(n=3)(x)

<tf.Tensor: shape=(3, 3, 3), dtype=int32, numpy=
array([[[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3]],
       [[4, 5, 6],
        [4, 5, 6],
        [4, 5, 6]],
       [[7, 8, 9],
        [7, 8, 9],
        [7, 8, 9]]])>

Let's concatenate:

tf.keras.layers.concatenate([x, x, x])

<tf.Tensor: shape=(3, 9), dtype=int32, numpy=
array([[1, 2, 3, 1, 2, 3, 1, 2, 3],
       [4, 5, 6, 4, 5, 6, 4, 5, 6],
       [7, 8, 9, 7, 8, 9, 7, 8, 9]])>

The other difference is of course, the conceptual difference between repeating and concatenating.

Repeating:

[0, 0, 0, 1, 1, 1, 2, 2, 2]

Concatenating:

[0, 1, 2, 0, 1, 2, 0, 1, 2]

Upvotes: 2

Lescurel
Lescurel

Reputation: 11651

Those two layers, Concatenate and RepeaVector are syntactic sugar over the tensorflow ops tf.concat and tf.repeat respectively.

Those 2 ops are there to mimic the numpy operations numpy.concatenate and numpy.repeat.

Those 2 ops are different:

  • repeat will repeat each element of the Tensor n times
  • concat will concat the tensor with itself
import numpy as np
import tensorflow as tf
a = np.arange(3)
>>> a
array([0, 1, 2])
>>> tf.repeat(a,3)
<tf.Tensor: shape=(9,), dtype=int64, numpy=array([0, 0, 0, 1, 1, 1, 2, 2, 2])>
>>> tf.concat((a,)*3,axis=0)
<tf.Tensor: shape=(9,), dtype=int64, numpy=array([0, 1, 2, 0, 1, 2, 0, 1, 2])>

tf.repeat was introduced in a feature request on github

Upvotes: 1

Related Questions