CODEWITHSUNDEEP
pythondeep-learningpytorchfast-ai
brenodacosta
brenodacosta

Reputation: 640

How to change the threshold of a prediction of multi-label classification using FASTAI library

I have a multi-label dataset that I'm using to train my model using fast-ai library for Python, using as metrics an accuracy function such as:

def accuracy_multi1(inp, targ, thresh=0.5, sigmoid=True):
    "Compute accuracy when 'inp' and 'targ' are the same size"
    if sigmoid: inp=inp.sigmoid()
    return ((inp>thresh) == targ.bool()).float().mean()

And my learner is like:

learn = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi1,thresh=0.1))
learn.fine_tune(2,base_lr=3e-2,freeze_epochs=2)

After training my model, I want to predict an image considering the threshold I used a argument, but the method learn.predict('img.jpg') only considers the default thres=0.5. In the following example, my predidction should return True for 'red, 'shirt' and 'shoes', as their probabilities are above 0.1 (but shoes is below 0.5, so it is not considered as True):

def printclasses(prediction,classes):
    print('Prediction:',prediction[0])
    for i in range(len(classes)):
        print(classes[i],':',bool(prediction[1][i]),'|',float(prediction[2][i]))

printclasses(learn.predict('rose.jpg'),dls.vocab)

Output:

Prediction: ['red', 'shirt']
black : False | 0.007274294272065163
blue : False | 0.0019288889598101377
brown : False | 0.005750810727477074
dress : False | 0.0028723080176860094
green : False | 0.005523672327399254
hoodie : False | 0.1325301229953766
pants : False | 0.009496113285422325
pink : False | 0.0037188702262938023
red : True | 0.9839697480201721
shirt : True | 0.5762518644332886
shoes : False | 0.2752271890640259
shorts : False | 0.0020902694668620825
silver : False | 0.0009014935349114239
skirt : False | 0.0030087409541010857
suit : False | 0.0006510693347081542
white : False | 0.001247694599442184
yellow : False | 0.0015280473744496703

Is there a way to impose the threshold when I'm doing a prediction of a image I'm referencing? Something that would look like:

learn.predict('img.jpg',thresh=0.1)

Upvotes: 4

Views: 866

Answers (1)

Thomas J.
Thomas J.

Reputation: 60

I've encountered the same problem. I remain interested in a better solution, but since accuracy_mult only seems to provide user-friendly evaluation of the model during the training process (and is not involved in the prediction), I created a work-around for my data.

The basic idea is to take the tensor with the actual predictions (which is the third entry in the 3-tuple returned by the predict() function), apply the threshold & get the corresponding labels from the vocab.

def predict_labels(x, model, thresh=0.5):
  '''
  function to predict multi-labels in text (x)

  arguments:
  ----------
  x: the text to predict
  model: the trained learner
  thresh: thresh to indicate which labels should be included, fastai default is 0.5

  return:
  -------
  (str) predictions separated by blankspace
  '''

  # getting categories according to threshold
  preds = model.predict(x)[2] > thresh
  labels = model.dls.multi_categorize.vocab[preds]

  return ' '.join(labels)

Upvotes: 1

Related Questions