CODEWITHSUNDEEP
nlpbert-language-modelhuggingface-transformers
mannt-jp
mannt-jp

Reputation: 63

huggingface's ReformerForMaskedLM configuration issue

I'm trying to pass the all of the huggingface's ...ForMaskedLM to the FitBert model for fill-in-the-blank task and see which pretrained yields the best result on the data I've prepared. But in the Reformer module I have this error says that I need to do 'config.is_decoder=False' but I don't really get what this means (This is my first time using huggingface). I tried to pass a ReformerConfig(is_decoder=False) to the model but still get the same error. How can I fix this?

My code:

pretrained_weights = ['google/reformer-crime-and-punishment', 
                      'google/reformer-enwik8']
configurations = ReformerConfig(is_decoder=False)
for weight in pretrained_weights:
    print(weight)
    model = ReformerForMaskedLM(configurations).from_pretrained(weight)
    tokenizer = ReformerTokenizer.from_pretrained(weight)
    fb = FitBert(model=model, tokenizer=tokenizer)
    predicts = []
    for _, row in df.iterrows():
        predicts.append(fb.rank(row['question'], options=[row['1'], row['2'], row['3'], row['4']])[0])
    print(weight,':', np.sum(df.anwser==predicts) / df.shape[0])

Error:

AssertionError                            Traceback (most recent call last)
<ipython-input-5-a6016e0015ba> in <module>()
      4 for weight in pretrained_weights:
      5     print(weight)
----> 6     model = ReformerForMaskedLM(configurations).from_pretrained(weight)
      7     tokenizer = ReformerTokenizer.from_pretrained(weight)
      8     fb = FitBert(model=model, tokenizer=tokenizer)
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1032 
   1033         # Instantiate model.
-> 1034         model = cls(config, *model_args, **model_kwargs)
   1035 
   1036         if state_dict is None and not from_tf:

/usr/local/lib/python3.7/dist-packages/transformers/models/reformer/modeling_reformer.py in __init__(self, config)
   2304         assert (
   2305             not config.is_decoder
-> 2306         ), "If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention."
   2307         self.reformer = ReformerModel(config)
   2308         self.lm_head = ReformerOnlyLMHead(config)

AssertionError: If you want to use `ReformerForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.

Upvotes: 2

Views: 238

Answers (1)

cronoik
cronoik

Reputation: 19365

You can override certain model configurations by loading the model config separately and providing it as parameter for the from_pretrained() method. This will assure that you are using the proper model configuration with the changes you have made:

from transformers import ReformerConfig, ReformerForMaskedLM

config = ReformerConfig.from_pretrained('google/reformer-crime-and-punishment')
print(config.is_decoder)
config.is_decoder=False
print(config.is_decoder)
model = ReformerForMaskedLM.from_pretrained('google/reformer-crime-and-punishment', config=config)

Output:

True
False

Upvotes: 2

Related Questions