How is get predict accuracy score in Bert Classification

Question

I am using Bert Classifier for my Chatbot project. I perform the necessary tokenizer operations for the incoming text message. Then I insert it into the model and make a prediction. How can I get the accuracy of this estimate?

for text in test_texts:
    encoded_dict = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=max_len,
        pad_to_max_length=True,
        return_attention_mask=True,
        return_tensors='pt',
    )

    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])

input_ids = torch.cat(input_ids, dim=0)
attention_masks = torch.cat(attention_masks, dim=0)
print("input_ids ",input_ids)
print("attention_masks ",attention_masks)

batch_size = 32

prediction_data = TensorDataset(input_ids, attention_masks)
prediction_sampler = SequentialSampler(prediction_data)
prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=batch_size)

print("prediction_data ",prediction_data)
print("prediction_sampler ",prediction_sampler)
print("prediction_dataloader ",prediction_dataloader)
model.eval()
predictions, true_labels = [], []

for batch in prediction_dataloader:
    batch = tuple(t.to(device) for t in batch)
    b_input_ids, b_input_mask = batch
    print("b input ids",b_input_ids)
    with torch.no_grad():
        outputs = model(b_input_ids, token_type_ids=None,
                        attention_mask=b_input_mask.to(device))

    logits = outputs[0]
    logits = logits.detach().cpu().numpy()
    label_ids = b_input_mask.to('cpu').numpy()

    predictions.append(logits)
    true_labels.append(label_ids)
    print("logits ",logits)

    print("label_ids ",label_ids)
    print("true_labels ",true_labels)

print('Prediction completed')

prediction_set = []

for i in range(len(true_labels)):
    pred_labels_i = np.argmax(predictions[i], axis=1).flatten()
    prediction_set.append(pred_labels_i)

prediction= [item for sublist in prediction_set for item in sublist]


print("prediction:", prediction[0])

I am looking for a percentage value. will respond or pass depending on the result of this percentage value.

Answer 1

Accuracy can be directly computed using some libraries.

For example, you can use sklearn:

from sklearn.metrics import accuracy_score
print("Accuracy:", accuracy_score(true_labels, predictions)) # Value between 0 and 1

print("Accuracy Percentage {} %:".format(100*accuracy_score(true_labels, predictions))) # Value between 0 and 100
``