How to merger NaiveBayesClassifier object in NLTK

Question

I am working on a project using the NLTK toolkit. With the hardware I have, I am able to run the classifier object on a small data set. So, I divided the data into smaller chunks and running the classifier object in them while storing all these individual object in a pickle file.

Now for testing I need to have the whole object as one to get better result. So my question is how can I combine these objects into one.

objs = []

while True:
    try:
        f = open(picklename,"rb")
        objs.extend(pickle.load(f))
        f.close()
    except EOFError:
        break

Doing this does not work. And it gives the error TypeError: 'NaiveBayesClassifier' object is not iterable.

NaiveBayesClassifier code :

 classifier = nltk.NaiveBayesClassifier.train(training_set)

Answer 1

I am not sure about the exact format of your data, but you can not simply merge different classifiers. The Naive Bayes classifier stores a probability distribution based on the data it was trained on, and you can not merge probability distributions without access to the original data.

If you look at the source code here: http://www.nltk.org/_modules/nltk/classify/naivebayes.html an instance of the classifier stores:

self._label_probdist = label_probdist
self._feature_probdist = feature_probdist

these are calculated in the train method using relative frequency counts. (e.g P(L_1) = (# of L1 in training set) / (# labels in training set). To combine the two, you would want to get (# of L1 in Train 1 + Train 2)/(# of labels in T1 + T2).

However, the naive bayes procedure isn't too hard to implement from scratch, especially if you follow the 'train' source code in the link above. Here is an outline, using the NaiveBayes source code

1. Store 'FreqDist' objects for each subset of the data for the labels and features.

   label_freqdist = FreqDist()
   feature_freqdist = defaultdict(FreqDist)
   feature_values = defaultdict(set)
   fnames = set()
   
   # Count up how many times each feature value occurred, given
   # the label and featurename.
   for featureset, label in labeled_featuresets:
       label_freqdist[label] += 1
       for fname, fval in featureset.items():
           # Increment freq(fval|label, fname)
           feature_freqdist[label, fname][fval] += 1
           # Record that fname can take the value fval.
           feature_values[fname].add(fval)
           # Keep a list of all feature names.
           fnames.add(fname)
   
   # If a feature didn't have a value given for an instance, then
   # we assume that it gets the implicit value 'None.'  This loop
   # counts up the number of 'missing' feature values for each
   # (label,fname) pair, and increments the count of the fval
   # 'None' by that amount.
   for label in label_freqdist:
       num_samples = label_freqdist[label]
       for fname in fnames:
           count = feature_freqdist[label, fname].N()
           # Only add a None key when necessary, i.e. if there are
           # any samples with feature 'fname' missing.
           if num_samples - count > 0:
               feature_freqdist[label, fname][None] += num_samples - count
               feature_values[fname].add(None)
   # Use pickle to store label_freqdist, feature_freqdist,feature_values
   

2. Combine those using their built-in 'add' method. This will allow you to get the relative frequency across all the data.

   all_label_freqdist = FreqDist()
   all_feature_freqdist = defaultdict(FreqDist)
   all_feature_values = defaultdict(set)
   
   for file in train_labels:
       f = open(file,"rb")
       all_label_freqdist += pickle.load(f)
       f.close()
   
   # Combine the default dicts for features similarly
   

3. Use the 'estimator' to create a probability distribution.

   estimator = ELEProbDist()
   
   label_probdist = estimator(all_label_freqdist)
   
   # Create the P(fval|label, fname) distribution
   feature_probdist = {}
   for ((label, fname), freqdist) in all_feature_freqdist.items():
       probdist = estimator(freqdist, bins=len(all_feature_values[fname]))
       feature_probdist[label, fname] = probdist
   
   classifier = NaiveBayesClassifier(label_probdist, feature_probdist)
   

The classifier will not combine the counts across all the data and produce what you need.