Why does this clean data provide strange SVM classification results?

Question

My problem and question are bolded below.

I have used support vector machines from Accord.NET successfully by following examples on their documentation pages like this one. However, when using a KernelSupportVectorMachine with a OneclassSupportVectorLearning to train it, the training process results in large error values and incorrect classifications.

The following miminal example shows what I mean. It generates a dense cluster of training points, then trains an SVM to classify points as inliers or outliers to the cluster. The training cluster is just a 0.6 by 0.6 square centered at the origin, and the training points are spaced at intervals of 0.1:

static void Main(string[] args)
{
    // Model and training parameters
    double kernelSigma = 0.1;
    double teacherNu = 0.5;
    double teacherTolerance = 0.01;


    // Generate input point cloud, a 0.6 x 0.6 square centered at 0,0.
    double[][] trainingInputs = new double[49][];
    int inputIdx = 0;
    for (double x = -0.3; x <= 0.31; x += 0.1) {
        for (double y = -0.3; y <= 0.31; y += 0.1) {
            trainingInputs[inputIdx] = new double[] { x, y };
            inputIdx++;
        }
    }


    // Generate inlier and outlier test points.
    double[][] outliers =
    {
        new double[] { 1E6, 1E6 },  // Very far outlier
        new double[] { 0, 1E6 },    // Very far outlier
        new double[] { 100, -100 }, // Far outlier
        new double[] { 0, -100 },   // Far outlier
        new double[] { -10, -10 },  // Still far outlier
        new double[] { 0, -10 },    // Still far outlier
    };
    double[][] inliers =
    {
        new double[] { 0, 0 },      // Middle of cluster
        new double[] { .15, .15 },  // Halfway to corner of cluster
        new double[] { -0.1, 0 },   // Comfortably inside cluster
        new double[] { 0.25, 0 }    // Near inside edge of cluster
    };


    // Construct the kernel, model, and trainer, then train.
    Console.WriteLine($"Training model with parameters:");
    Console.WriteLine($"  kernelSigma = {kernelSigma.ToString("#.##")}");
    Console.WriteLine($"  teacherNu={teacherNu.ToString("#.##")}");
    Console.WriteLine($"  teacherTolerance={teacherTolerance}");
    Console.WriteLine();

    var kernel = new Gaussian(kernelSigma);
    var svm = new KernelSupportVectorMachine(kernel, inputs: 1);
    var teacher = new OneclassSupportVectorLearning(svm, trainingInputs)
    {
        Nu = teacherNu,
        Tolerance = teacherTolerance
    };
    double error = teacher.Run();

    Console.WriteLine($"Training complete - error is {error.ToString("#.##")}");
    Console.WriteLine();


    // Test trained classifier.
    Console.WriteLine("Testing outliers:");
    foreach (double[] outlier in outliers) {
        WriteResultDetail(svm, outlier);
    }
    Console.WriteLine();
    Console.WriteLine("Testing inliers:");
    foreach (double[] inlier in inliers) {
        WriteResultDetail(svm, inlier);
    }
}

private static void WriteResultDetail(KernelSupportVectorMachine svm, double[] coordinate)
{
    string prettyCoord = $"{{ {string.Join(", ", coordinate)} }}".PadRight(20);
    Console.Write($"Classifying: {prettyCoord} Result: ");

    // Classify coordinate, print results.
    double result = svm.Compute(coordinate);
    if (Math.Sign(result) == 1) {
        Console.Write("Inlier");
    }
    else {
        Console.Write("Outlier");
    }
    Console.Write($" ({result.ToString("#.##")})\n");
}

Here is the output for a reasonable parameter set:

Training model with parameters:
  kernelSigma = .1
  teacherNu=.5
  teacherTolerance=0.01

Training complete - error is 222.4

Testing outliers:
Classifying: { 1000000, 1000000 } Result: Inlier (2.28)
Classifying: { 0, 1000000 }       Result: Inlier (2.28)
Classifying: { 100, -100 }        Result: Inlier (2.28)
Classifying: { 0, -100 }          Result: Inlier (2.28)
Classifying: { -10, -10 }         Result: Inlier (2.28)
Classifying: { 0, -10 }           Result: Inlier (2.28)

Testing inliers:
Classifying: { 0, 0 }             Result: Inlier (4.58)
Classifying: { 0.15, 0.15 }       Result: Inlier (4.51)
Classifying: { -0.1, 0 }          Result: Inlier (4.55)
Classifying: { 0.25, 0 }          Result: Inlier (4.64)

The number in parentheses is the score given by the SVM for that coordinate. With SVMs from Accord.NET (and in general), a negative score is one class and a positive score is another. Here, everything has a positive score. Inliers are classified correctly, but outliers (even very distant ones) are also classified as inliers.

Note that any other time I've trained models with Accord.NET, the training error has been quite close to zero, but here it's over 200.

Here's another parameter set's output:

Training model with parameters:
  kernelSigma = .3
  teacherNu=.8
  teacherTolerance=0.01

Training complete - error is 1945.67

Testing outliers:
Classifying: { 1000000, 1000000 } Result: Inlier (20.96)
Classifying: { 0, 1000000 }       Result: Inlier (20.96)
Classifying: { 100, -100 }        Result: Inlier (20.96)
Classifying: { 0, -100 }          Result: Inlier (20.96)
Classifying: { -10, -10 }         Result: Inlier (20.96)
Classifying: { 0, -10 }           Result: Inlier (20.96)

Testing inliers:
Classifying: { 0, 0 }             Result: Inlier (44.52)
Classifying: { 0.15, 0.15 }       Result: Inlier (41.62)
Classifying: { -0.1, 0 }          Result: Inlier (43.85)
Classifying: { 0.25, 0 }          Result: Inlier (40.53)

Again, very high training error, all positive scores.

The models are definitely getting something out of training - the scores are different between inliers and outliers. But why doesn't this simple scenario give results that differ by positive and negative sign as they should?

---

PS. Here is a similar program that tests many combinations of training and model parameters, and here is its output. Again, everything results in positive classification scores, high error values, and incorrectly classified outliers.

Answer 1

The issue raised in the question has been addressed in version 3.7.0 of Accord.NET. A unit test with an example similar to yours has also been added in commit be81aab.