CODEWITHSUNDEEP
attention-model
Yoo Inhyeok
Yoo Inhyeok

Reputation: 131

Implementing Attention

I'm implementing the Attention in PyTorch. I got questions during implementing the attention mechanism.

  1. What is the initial state of the decoder $s_0$? Some post represents it as zero vector and some implements it as the final hidden state of the encoder. So what is real $s_0$? The original paper doesn't mention it.

  2. Do I alternate the maxout layer to dropout layer? The original paper uses maxout layer of Goodfellow.

  3. Is there any differences between encoder's dropout probability and decoder's? Some implementation sets different probabilities of dropouts for encoder and decoder.

  4. When calculating $a_{ij}$ in the alignment model (concat), there are two trainable weights $W$ and $U$ . I think the better way to implement it is using two linear layers. If I use a linear layer, should I remove bias term in the linear layers?

  5. The dimension of the output of the encoder(=$H$) doesn't fit the decoder's hidden state. $H$ is concatenated, so it has to be 2000 (for the original paper). However, the decoder's hidden dimension is also 1000. Do I need to add a linear layer after the encoder to fit the encoder's dimension and the decoder's dimension?

Upvotes: 0

Views: 170

Answers (1)

Jindřich
Jindřich

Reputation: 11220

In general, many answers are: it is different in different implementations. The original implementation from the paper is at https://github.com/lisa-groundhog/GroundHog/tree/master/experiments/nmt. For later implementations that reached better translation quality, you can check:

Now to your points:

  1. In the original paper, it was a zero vector. Later implementations use a projection of either of the encoder final state or the average of the encoder states. The argument for using average is that it propagates the gradients more directly into the encoder states. However, this decision does not seem to influence the translation quality much.

  2. Maxout layer is a variant of a non-linear layer. It is sort of two ReLU layers in one: you do two independent linear projections and take the maximum of them. You can happily replace Maxout with ReLU (modern implementations do so), but you still should use dropout.

  3. I don't know about any meaningful use case in MT when I would set the dropout rates differently. Note, however, that seq2seq models are used in many wild scenarios when it might make sense.

  4. Most implementations do use bias when computing attention energies. If you use two linear layers, you will have the bias split into two variables. Biases are usually zero-initialized, they will thus get the same gradients and the same updates. However, you can always disable the bias in a linear layer.

  5. Yes, if you want to initialize s0 with the decoder states. In the attention mechanism, matrix U takes care of it.

Upvotes: 2

Related Questions