CODEWITHSUNDEEP
pythonkerasnlplstmrecurrent-neural-network
danci_bald
danci_bald

Reputation: 1

How to use custom embeddings with keras LSTM?

I want to use pre-trained word embeddings with an LSTM.

That is I already have a model of form:

embedding_for_word = model[word]

I have data of the following form:

1. "word1 word2 word3" label 0
2. "word4 word5 word6 word7" label 1
3. "word8 word9" label 1
...
..
.

I know that for a standard LSTM (if timesteps are fixed) we can have:

model = Sequential()
model.add(LSTM(N, input_shape=(n_timesteps, 1), return_sequences=True))
model.add(TimeDistributed(Dense(1, activation='sigmoid')))
model.compile(loss='binary_crossentropy', optimizer='adam')

How do I give sequential input of the form:

batch_1[embedding_word1,
embedding_word2,embedding_word3 .. some_end_of_sequence_character] --> label 0
batch_2[embedding_word4,
embedding_word5,embedding_word,embedding_word7,some_end_of_sequence_character] --> label 1
...
..
.

How do I engineer the data and create the model (For the model, I'm only asking what the input layer would look like) for the example above?

Assume:

size_of_embeddings = K batch_size = B

Upvotes: 0

Views: 651

Answers (1)

Daniel Möller
Daniel Möller

Reputation: 86600

You should have something to pad the missing words up to the length of the sequence. Probably a "sentence end" embedding would be interesting.

So, you will need:

  • one array with shape: (total_sentences, words, embedding_size) - This is X, inputs
  • one array with shape: (total_sentences, total_labels) - This is Y, outputs

You can then use fit(X, Y), maybe with a train test split.

The first layer of the model can be the LSTM(input_shape=(words, embedding_size), ...).

Upvotes: 1

Related Questions