Null Classes in Machine Learning

Question

How do we handle null classes at test time in a machine learning system. If I train my model on lets say 10 classes, and then I observe a class that does not belong to any of the 10 classes, is there a way to detect this occurrence? It is needed for activity recognition in a sliding window approach where each time step yields one of the 10 classes, however, actually, there are time-steps where nothing happens and so the algorithm should not classify.

Answer 1

This would be whats called outlier or novelty detection. Some basic info here. You would want to use a outlier detection algorithm first (where all 10 classes are your inliers) to filer out new classes that you haven't seen before. Then if it passes the outlier detector, you feed it into your classifier. There will be some false positives/negatives on the outlier stage - which will have an impact on what fraction of the data you classify correctly.

however, actually, there are time-steps where nothing happens and so the algorithm should not classify.

Perhaps then what you really should consider is a 11'th class of "no activity". If its real data that occurs regularly, you should treat it as such.