CODEWITHSUNDEEP
machine-learningnlpclassificationmaxent
nobillygreen
nobillygreen

Reputation: 1588

"Consensus" Among Maximum Entropy Classifications

Imagine we have three classes: A, B, and C, and we classify a document 'd' using a standard MaxEnt classifier, and come up with the following probabilities:

P(d, A) = 0.50
P(d, B) = 0.25
P(d, C) = 0.25

I feel like that is very different, in a way, from this set of probabilities:

P(d, A) = 0.50
P(d, B) = 0.49
P(d, C) = 0.01

Is there a way to score the difference between these two?

Upvotes: 2

Views: 203

Answers (2)

Ben Allison
Ben Allison

Reputation: 7394

What you're looking for is cross-entropy: specifically, you want to calculate the cost of approximating the true distribution with the one output by your classifier. Probabilistic multi-class classifiers will optimise this directly in many cases. Take a look at this.

Upvotes: 1

lejlot
lejlot

Reputation: 66835

The problem you are facing is often called the "consensus" among classifiers. As multilabel MaxEnt can be seen as N independent classifiers, you can think about it as a group of models "voting" for different classes.

Now, there are many measures of calculating such "consensus", including:

  • "naive" calculation of the margin - difference between the "winning" class probability and the second one - bigger the margin - more confident the classification
  • entropy - smaller the entropy of the resulting probability distribution, the more confident the decision
  • some further methods involving KL divergence etc.

In general you should think about methods od detecting "uniformity" of the resulting distribution (impling less confident decison) or "spikeness" (indicating more confident classification).

Upvotes: 3

Related Questions