CODEWITHSUNDEEP
pythonkerasdeep-learninglstm
Stupid420
Stupid420

Reputation: 1419

How can I add attention layer binary classification RNN

I want to add attention after my LSTM layer. Following is the code. 

visible = Input(shape=(251,))    
embed=Embedding(vocab_size, 50)(visible)
bilstm = Bidirectional(LSTM(units=25, return_sequences=True))(embed)    
att==??    
predictions=Dense(1, activation='sigmoid')(att)

Is there any layer in keras for attention like lstm or gru etc.

Upvotes: 2

Views: 420

Answers (1)

Giacomo
Giacomo

Reputation: 116

You can use this.

bilstm = Bidirectional(LSTM(units=25, return_sequences=True))(embed) 
attention=TimeDistributed(Dense(1, activation = 'tanh'))(bilstm)
attention=L.Softmax(axis=1)(attention)
context=L.Multiply()([attention, bilstm])
cout=L.Lambda(lambda x: K.sum(x,axis=1))(context)

Which allows you to retrieve each layer. By default it will return weights*sequence as output from the bidirectional LSTM. You can avoid the Lambda layer if you want to retain the sequences.

Found the code from a keras github issue. The solution i wrote is from one of keras devlopers (i suppose?). So i cannot give credits unfortunately.

Upvotes: 1

Related Questions