CODEWITHSUNDEEP
python-3.xpytorchhuggingface-transformershuggingface-tokenizers
sheming
sheming

Reputation: 161

Hugging face: tokenizer for masked lm question

I am using transformer version 3.0.0 for my project and have some questions.

I want to use a bert model with masked lm pretraining for protein sequences. To get a character level tokenizer I derived from the BertTokenizer

from transformers import BertTokenizer
class DerivedBertTok(BertTokenizer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
    def tokenize(self, text):
        if isinstance(text, np.ndarray):
            assert len(text) == 1
            text = text[0]
        return [x if x in self.vocab else self.unk_token for x in text]

my vocab looks like this

[PAD]
[CLS]
[SEP]
[UNK]
[MASK]
A
R
N
D
B
C
E
Q
Z
G
H
I
L
K
M
F
P
S
T
W
Y
V

The usage seems quite similar to what i have seen in the docs:

d_tokenizer = DerivedBertTok(
    vocab_file=vocab_path,
    do_lower_case=False,
    do_basic_tokenize=False,
    tokenize_chinese_chars=False
)
d_tokenizer.encode_plus(np.array(["AXEF"])[0], 
                      max_length=20,
                      pad_to_max_length=True,
                      add_special_tokens=True,
                      truncation=True,
                      return_tensors='pt')

From this I was building a pytorch Dataset with a custom collate function. all the collate function does is taking all input tensors and stacking them

from transformers import BatchEncoding
    def collate_fn(self, batch):
        # this function will not work for higher dimension inputs
        elem = batch[0]
        elem_type = type(elem)
        if isinstance(elem, BatchEncoding):
            new_shapes = {key: (len(batch), value.shape[1]) for key, value in elem.items()}
            outs = {key: value.new_empty(new_shapes[key]) for key, value in elem.items()}
            if torch.utils.data.get_worker_info() is not None:
                [v.share_memory_() for v in outs.values()]
            return {key: torch.stack(tuple((d[key].view(-1) for d in batch)), 0, out=outs[key]) for key in elem.keys()}
        else:
            raise ValueError(f"type: {elem_type} not understood")

Question 1: So I was wondering if the BatchEncoding or another class is already capable of doing this (and doing it possibly better?). Or using a different Dataset/ DataLoader class altogether.

Question 2: Additionally, I want to mask some of the Inputs as required for the masked LM, however I did not manage find any implementation in the transformer library. Are there any recommendations for doing this?

Upvotes: 2

Views: 3465

Answers (1)

sheming
sheming

Reputation: 161

After some more digging I found a DataCollator, which implements replacing token randomly with the mask token at: https://github.com/huggingface/transformers/blob/615be03f9d961c0c9722fe10e7830e011066772e/src/transformers/data/data_collator.py#L69. So I changed my DataSource to return raw text instead of the BatchEncoding in the __getitem__ method and then do the encoding and masking in the collate function.

Upvotes: 4

Related Questions