CODEWITHSUNDEEP
tensorflowkeras
liyuan
liyuan

Reputation: 553

Keras: why must an embedding layer be used only as the first layer?

In the keras documentation it states that the embedding layer "can only be used as the first layer in a model." This makes no sense to me, I might want to do a reshape/flatten on an input before passing it to the embedding layer, but this is not allowed. Why must the embedding layer be used only as the first layer?

Upvotes: 3

Views: 1275

Answers (1)

Amir
Amir

Reputation: 16587

"can only be used as the first layer in a model." This makes no sense to me

Generally, an embedding layer maps discrete values to continuous values. In the subsequence layers, we have continuous vector representation which means there is no need to convert the vectors again.

I might want to do a reshape/flatten on input before passing it to the embedding layer

Of course, you can reshape or flatten an input but in most cases is meaningless. For example, assume we have sentences with a length of 30 and want to flatten them before passing them to embedding:

input_layer = Input(shape=(30))
flatten = Flatten()(input_layer)
embedd = Embedding(1000, 100)(flatten)

In the above example, flattened layer has no effect at all. Before and after flattening our vector size is [batch, 30].

Let's look at another example, assume our input is a 2D vector with the shape of [batch, 30, 2]. After flatting the input, the vectors have the size of [batch, 60]. We can feed them into the Embedding layer but in most of the scenarios, it has no meaning. In fact, we destroy the logical relationship between features.

input_layer = Input(shape=(30, 2))
flatten = Flatten()(input_layer)
embedd = Embedding(1000, 100)(flatten)

Upvotes: 3

Related Questions