Dialogflow Detect Intent Matching Confidence And Entity Value Selection

Question

I am looking to understand how detect intent confidence impacts entity value selection in Dialogflow. For example, using two user generated phrases:

• Phrase 1: "For snack yesterday I had an apple and peanut butter". This phrase has an intent detection confidence of '1' and 'snack' and 'yesterday' are tagged correctly to their respective entities, and the foods, 'apple' and 'peanut butter' are correctly matched within their entity [food], with values of 'apple' and 'peanut butter' respectively.

• Phrase 2: "Are the snack yesterday I had an apple and peanut butter". This phrase was mumbled by the user or garbled by Siri (we use an iOS voice app). Here the intent detection confidence is '0.852' and while 'snack' and 'yesterday' are tagged to their entities correctly, the foods are not treated as above. Specifically, while both are tagged to the correct entity [food] and 'apple' was correctly tagged to 'apple', the 'peanut' of 'peanut butter' was tagged as one food [value = 'peanut'] and the 'butter' of 'peanut butter' was tagged as another food [value = 'butter'].

As context we have ~500 intents, the intent matched above has ~400 training phrases (clearly not including 'Are the...') and ~200 entities, the largest of which has 29,998 values.

So it appears the intent detection confidence impacts the entity parameter value matching. Can anyone shed any light on this? From our viewpoint, it is not a useful 'feature'. Quite the opposite.

Answer 1

Over the last few weeks we have interacted at some length with GCP Support who've interacted with the DF engineering team.

The short answer is they indicate the entity value extraction is 'as designed', and specifically, when using a composite entity (a nested entity structure), values that match input terms (here, 'peanut', 'butter' and 'peanut butter') are extracted and matched at random. So a user's utterance of 'peanut butter' may be matched to 'peanut' and 'butter' or 'peanut butter' at random.

This behavior cannot be controlled by adding additional training phrases to the agent.

From our point of view, the behavior is not as desired but we understand the implications for our design. Hopefully this 'answer' helps others navigate this and similar issues.

Answer 2

When searching for a matching intent, Dialogflow scores potential matches with an intent detection confidence, also known as the confidence score.

These values range from 0.0 (completely uncertain) to 1.0 (completely certain). Without taking the other factors described in this document into account, once intents are scored, there are three possible outcomes:

• If the highest scoring intent has a confidence score greater than or equal to the ML Classification Threshold setting, it is returned as a match.

• If no intent meets the threshold, a fallback intent is matched.

• If no intents meet the threshold and no fallback intent is defined, no intent is matched.

Update: As per the GCP DialogFlow development team: Scores are referred to intent recognition confidence Parameter extraction is not taken into account when scores are computed" In other words, there is no relationship between intent classification confidence and entity extraction.

The described behavior could potentially be a bug within DialogFlow or something specific to your GCP project and further inspection for your GCP project is required with GCP Support to investigate why this is happening. You can create a GCP Support Case.