supervised classification of multiple categories with a natural ordering

Question

I try to train a model to classify samples into three categories: weak, medium, and strong. As far as I know, the best way to do this is to use weighted kappa score to measure the model performance. It seems weighted kappa is not implemented in scikit-learn. How do people do with scikit-learn for this types of classification? It is a fairly routine task.

Answer 1

You can use skll to get the kappa score to measure the performance.

from skll.metrics import kappa
kappa_val = kappa(y_true, y_pred)

There are also many other use cases of skll. For details, please visit the link here