CODEWITHSUNDEEP
c#botframeworkazure-language-understandingazure-bot-serviceazure-qna-maker
parthraj panchal
parthraj panchal

Reputation: 121

How to set confidence score threshold to 50% in azure bot (webchat)?

I'm working with QnAmker bot with azure bot.I haven't coded a single line of it, I'm just using it online. Now i want to set confidence score threshold to 50% . I'm editing bot code with online editor and deploying it with KUDU console. Where can i edit or code confidence score threshold? My bot platform is C#.

Upvotes: 0

Views: 1469

Answers (2)

billoverton
billoverton

Reputation: 2885

I'm using nodejs instead of C# but the implementation should be similar. I tried the method mentioned to set the threshold in the call to generateanswer, but this will succeed only in giving you the default answer if the threshold wasn't reached, something like "No answers were found for that query." In most cases you're not going to want that response for your users, and I find checking the confidence score after you receive the answer to be more useful.

I am doing a check in the actual dialog and setting the outputActivity from there. Again, this is nodejs but you should be able to do the same thing in C#. Here is how I implemented it.

// Apply confidence filter
if (qnaResult[0].score > MINIMUM_SCORE) {
    outputActivity = MessageFactory.text(qnaAnswer);
}
else {
    // If low confidence, send to social talk
    var socialResult = await SocialTalkHelper.queryQnAService(query, oldState);
    if (socialResult[0].score > MINIMUM_SCORE) {
        outputActivity = MessageFactory.text(socialResult[0].answer);
    }
    else {
        // If low confidence for social talk, use default answer
        outputActivity = MessageFactory.text(defaultAnswer);
    }
}

I've actually cascaded this to use a second QnA Maker call to an Enterprise Social Talk KB if my minimum score isn't met. I've set a custom defaultAnswer to use if that call doesn't meet my threshold either. Note that MINUMUM_SCORE is out of 100, not 1.0 (e.g. 50% confidence is 50, not 0.5).

Not sure which sample you are using. My answer is based on the "experimental" version but in any case you should be getting the same object back from QnA Maker and it's just a matter of how you handle it.

Upvotes: 1

JJ_Wailes
JJ_Wailes

Reputation: 2227

To change the threshold to a different value, set the threshold score as a property of the GenerateAnswer API JSON body. This means you set it for each call to GenerateAnswer.

If you're using the bots on Azure, you're going to have to edit the BotServices.cs file to account for the QnAMaker metadata, which is returned in the call to GenerateAnswer. In particular, you're going to have to add the following:

var qnaOptions = new QnAMakerOptions();
var metadata = new Microsoft.Bot.Builder.AI.QnA.Metadata();

metadata.Name = Constants.MetadataName.Intent;
metadata.Value = topIntent;

qnaOptions.StrictFilters = new Microsoft.Bot.Builder.AI.QnA.Metadata[] { metadata };
qnaOptions.Top = Constants.DefaultTop;
qnaOptions.ScoreThreshold = 0.5F;

The important bits are var metadata... and qnaOptions.ScoreThreshold = 0.5F; The only sample that shows this in action is the following sample on the Botbuilder-Samples repo: Support Bot in QnA Maker.

HOWEVER, two things to keep in mind if you decide to practice with this sample: 1) it's in the experimental branch of the samples repo, which means there is no guarantee to perfect functionality. 2) That sample show multiple different QnAMaker functionalities. I suggest you clone the code and look through it before just putting it in your code on Azure.

Upvotes: 4

Related Questions