CODEWITHSUNDEEP
pytorchkeras-layerbert-language-modelhuggingface-transformers
gezgine
gezgine

Reputation: 47

BERT Embeddings in Pytorch Embedding Layer

I'm working with word embeddings. I obtained word embeddings using 'BERT'.

I have a data like this

1992 regular unleaded 172 6 MANUAL all wheel drive 4 Luxury Midsize Sedan 21 16 3105 200

and as a label:

df['Make'] = df['Make'].replace(['Chrysler'],1)

I try to give embeddings as a LSTM inputs.

Using below code for BERT:

 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')                                                     
 bert_model = BertModel.from_pretrained('bert-base-uncased')

For tokinizing this code:

def tokenize_text(df, max_seq):
    return [ 
        tokenizer.encode(text, add_special_tokens=True)[:max_seq] for text in df
    ]


def pad_text(tokenized_text, max_seq):
    return np.array([el + [0] * (max_seq - len(el)) for el in tokenized_text])


def tokenize_and_pad_text(df, max_seq):
    tokenized_text = tokenize_text(df, max_seq)
    padded_text = pad_text(tokenized_text, max_seq)
    return padded_text

and :

train_indices = tokenize_and_pad_text(df_train, max_seq)

for BERT embedding matrix:

def get_bert_embed_matrix():
    bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    bert_embeddings = list(bert.children())[0]
    bert_word_embeddings = list(bert_embeddings.children())[0]
    mat = bert_word_embeddings.weight.data.numpy()
    return mat

embedding_matrix = get_bert_embed_matrix()

and LSTM Model:

embedding_layer =Embedding(embedding_matrix.shape[0], embedding_matrix.shape[1], weights=[embedding_matrix], input_length=max_seq_len, trainable=True)
model = Sequential()
model.add(embedding_layer)
model.add(LSTM(128, dropout=0.3, recurrent_dropout=0.3))
model.add(Dense(1, activation='softmax'))

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-5),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)

model.summary()

and for model fit this code:

model.fit(train_indices, y_train, epochs=20, verbose=1)

I give a output like this:

Epoch 1/20
1/1 [==============================] - 3s 3s/step - loss: 0.0000e+00 - accuracy: 0.3704
Epoch 20/20
1/1 [==============================] - 0s 484ms/step - loss: 0.0000e+00 - accuracy: 0.3704

I don't know what I'm missing.

Firstly, what can we do about it?
Secondly, how can we implement Pytorch Model?

Thanks a lot.

Upvotes: 3

Views: 3945

Answers (1)

Xiao Huang
Xiao Huang

Reputation: 71

Here is a related question. The op tried 'embedding_layer.set_weights()' method and it worked.

Keras initialize large embeddings layer with pretrained embeddings

Upvotes: 0

Related Questions