CODEWITHSUNDEEP
javamachine-learningclassificationgaussian
LucasSeveryn
LucasSeveryn

Reputation: 6262

Multivariate gaussian classifier implementation. Trouble understanding, going from Naive Gaussian

Thank you for checking this question out. I am trying to understand how to use the multivariate gaussian classifier.

To introduce you better to my problem, I will show how currently I classify data.

I have a library of these objects:

public class AccFeat {
    int id;

    Double[] mean = new Double[3];
    Double[] sd = new Double[3];
    Double[] avPeakDistance = new Double[3];
    int[][] histogram = new int[3][10];
    int[][] fftHistogram = new int[3][10];
    int[] crossingCount = new int[3];
    double resultantAcc;
    int type;

And then I get an object with type field indicating that it is not identified.

Procedure:

  1. Load a library of 30 training samples per class.

  2. Calculate mean and variance of each feature for each sample class, store these values in arrays, one array of 73 mean/variance pairs per class. (Because there are total of 73 features, including 6 histograms 10 bins each (60 numbers))

  3. Create an array of 73 values which correspond to features of AccFeat object to be indentified.

  4. Calculate probability using what I understand is Naive Bayesian Classifier.

we check for i from 0 to 8, because there are 9 sample classes.

    for (int i = 0; i < 9; i++) {
            result = 1;
            for (int j = 0; j < SAMPLEFEATURES.get(i).size(); j++) {
                result = result * p(QUERYFEATURES.get(j), SAMPLEFEATURES.get(i).get(j)); 
//this is the p function, first argument is value of feature, 
//second argument is mean-variance pair for this feature in this particular i class.

            }
            results[i] = result;
        }
    }

p(x) function is simply this:

  1. https://i.sstatic.net/1iws5.jpg

And then I have 9 probability values, for each class, my classifier will say that it is class that corresponds to the highest probability value.

Now I want to create a Multivariate Gaussian Classifier.

This is the formula that is used to calculate probability in this case:

  1. https://i.sstatic.net/JNiwT.jpg

  2. So I create a variance-covariance matrix for each of 9 classes. Here I'm not sure if I do it right, I take all 73 features, which, again, includes 6 histograms of 10 bins each, so 60 of these features are histogram of acceleration frequencies and acceleration values

This I find a bit dodgy, should I just put all these values together into one matrix? Calculating covariance between frequencies of accelerations in range 10-20 on x-axis and peak distance of y accelerations seems a bit... odd.

But I do it, and create a 73x73 matrix for each class using this formula for each cell:

Cov(feature a, feature b) = sum ( ( featureA[i] - mean_featureA ) * ( featureB[i] - mean_featureB ) ) / n-1

  1. The next thing I need is a mean vector, so I create a 73-element vector of means of each feature for each group of smaples associated with each class, total of 9 vectors.

  2. From what I understand, the x in the formula in case of my program is a 73-element vector of feature values of unidentified AccFeat object.

  3. So I implement this formula, thinking: I have to run it using covariance matrices and means for each class, and the one with highest outcome is most likely candidate for indentification

Problems:

What is wrong with the way I'm using the classifier? Unfortunaltely I have nobody to help me with this, and I base all of my weak understanding on online lecture slides....

Can anyone help me in how to use it?

Upvotes: 0

Views: 1426

Answers (1)

Ben Allison
Ben Allison

Reputation: 7394

I haven't thoroughly checked whether what you're doing is valid, but I can help you with the error you're getting. You don't have enough data points to estimate the covariance matrix, so the maximum likelihood estimate is singular.

One option is to move to a variant of Linear Discriminant Analysis, where you use a pooled covariance matrix across all the classes. This may get you a non-singular covariance matrix, if you have enough data pooled across all classes. The other option is Naive Bayes, which is what you have, where you have a diagonal covariance matrix.

Otherwise, is there any reason you're looking to use the MVN classifier? You could just use something else, for example an SVM with kernels designed to compare histograms.

Upvotes: 0

Related Questions