CODEWITHSUNDEEP
huggingface-transformers
AmirHJ
AmirHJ

Reputation: 837

How to initialize BertForSequenceClassification for different input rather than [CLS] token?

BertForSequenceClassification uses [CLS] token's representation to feed a linear classifier. I want to leverage another token (say [X] in the input sequence) rather than [CLS]. What's the most straightforward way to implement that in Transformers?

Upvotes: 1

Views: 461

Answers (1)

Eric Lozano
Eric Lozano

Reputation: 441

You can define the special tokens when creating the tokenizer.

This is an example of how to modify an special token of a pretrained tokenizer:

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", cls_token="[X]")

Please check the BertTokenizer documentation to see which other special tokens you can modify (unk_token, sep_token, pad_token, cls_token, mask_token, etc.).

Upvotes: 1

Related Questions