CODEWITHSUNDEEP
tensorflowword-embedding
Daniel T.
Daniel T.

Reputation: 489

In tf.keras.layers.Embedding, why it is important to know the size of dictionary?

Same as the title, in tf.keras.layers.Embedding, why it is important to know the size of dictionary as input dimension?

Upvotes: 2

Views: 5446

Answers (2)

sebrockm
sebrockm

Reputation: 6002

Because internally, the embedding layer is nothing but a matrix of size vocab_size x embedding_size. It is a simple lookup table: row n of that matrix stores the vector for word n.

So, if you have e.g. 1000 distinct words, your embedding layer needs to know this number in order to store 1000 vectors (as a matrix).

Don't confuse the internal storage of a layer with its input or output shape. The input shape is (batch_size, sequence_length) where each entry is an integer in the range [0, vocab_size[. For each of these integers the layer will return the corresponding row (which is a vector of size embedding_size) of the internal matrix, so that the output shape becomes (batch_size, sequence_length, embedding_size).

Upvotes: 7

gorjan
gorjan

Reputation: 5555

In such setting, the dimensions/shapes of the tensors are the following:

  • The input tensor has size [batch_size, max_time_steps] such that each element of that tensor can have a value in the range 0 to vocab_size-1.
  • Then, each of the values from the input tensor pass through an embedding layer, that has a shape [vocab_size, embedding_size]. The output of the embedding layer is of shape [batch_size, max_time_steps, embedding_size].
  • Then, in a typical seq2seq scenario, this 3D tensor is the input of a recurrent neural network.
  • ...

Here's how this is implemented in Tensorflow so you can get a better idea:

inputs = tf.placeholder(shape=(batch_size, max_time_steps), ...)
embeddings = tf.Variable(shape=(vocab_size, embedding_size], ...)
inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)

Now, the output of the embedding lookup table has the [batch_size, max_time_steps, embedding_size] shape.

Upvotes: 3

Related Questions