CODEWITHSUNDEEP
pythontensorflowdeep-learninglstmlanguage-model
user3480922
user3480922

Reputation: 574

How to run custom seq2seq learning (using pre-calculated word embeddings) encoder-decoder in Tensorflow?

I need to run a encoder-decoder model in Tensorflow. I see that using the available APIs basic_rnn_seq2seq(encoder_input_data, decoder_input_data, lstm_cell) etc, a encoder-decoder system can be created.

  1. How can we enter the embeddings such as word2vec in such model? I am aware that we can do embedding lookup but as per the API encoder_input_data is a list of 2D Tensor of size batch_size x input_size. How can each word be represented using its respective word embedding in this setup? Even embedding_rnn_seq2seq internally extracts the embeddings. How to give pre-calculated word embeddings as input?
  2. How can we get the cost/perplexity through the API?
  3. In case of test instances, we may not know the corresponding decoder inputs. How to handle such case?

Upvotes: 0

Views: 830

Answers (1)

Phillip Bock
Phillip Bock

Reputation: 1889

First question: Probably not the best way, but what I did was, after building the model, before training starts:

for v in tf.trainable_variables():
  if v.name == 'embedding_rnn_seq2seq/RNN/EmbeddingWrapper/embedding:0':
    assign_op = v.assign(my_word2vec_matrix)
    session.run(assign_op)  # or `assign_op.op.run()`

my_word2vec_matrix is a matrix of shape vocabularysize x embedding size and filled in my precomputed embedding-vectors. Use this (or something similar) if you believe your embeddings are really good. Otherwise the seq2seq-Model, over time, will come up with its own trained embedding.

Second question: In seq2seq.py there is a call to model_with_buckets() which you can find in python/ops/seq2seq.py. From there the loss is returned.

Third question: In the test case each decoder input is the decoder output from the timestep before (i.e. the first decoder input is a special GO-symbol, the second decoder input is the decoder output of the first timestep, the third decoder input is the decoder output of the second timestep, and so on)

Upvotes: 3

Related Questions