Is it possible to convert BertTokenizer to BartTokenizer?

Question

I understand BART and BERT are two architectures available in HuggingFace Transformers. I wondered if I could convert BertTokenizer to a BartTokenizer.

The reason for converting is that a training script requires BartTokenizer format, while the tokenizer I am using is a BertTokenizer.

I saved my tokenizer and the model using save_pretrained to a local folder. The files are as follows:

• added_tokens.json
• config.json
• generation_config.json
• pytorch_model.bin
• special_tokens_map.json
• tokenizer_config.json
• tokenizer.json
• vocab.txt

For BartTokenizer, it looks for the following files:

• vocab.json
• merges.txt
• tokenizer.json
• config.json

How can I "convert" the BertTokenizer to BartTokenizer files? Thanks.

---

Using the following versions:

• transformers 4.28.1
• tokenizers 0.13.3
• Python 3.10.10

and on another machine (using the latest version available):

• transformers 4.45.2
• tokenizers 0.20.1
• Python 3.12.4

Answer 1

I am not sure that there's a simple way for obtaining merges (merges.txt) from your berttokenizer, but the easiest you can do - add tokens from your bertokenizer:

from transformers import BartTokenizerFast

bert_tokenizer # your tokenizer
bart_tokenizer = BartTokenizerFast.from_pretrained("facebook/bart-base")
bert_tokens = list(bert_tokenizer.vocab.keys())
bart_tokenizer.add_tokens(bert_tokens)