CODEWITHSUNDEEP
c#performancevisual-studiounit-testing
Chris Riether
Chris Riether

Reputation: 33

c# unit test akward runtime

I am developing a C# tool in Visual Studio 2013 on Win8.

I just started to pay depts for unit tests (shame on me ;) ) on a pretty easy part:

have a class for geometric vectors. Wanted to check if init works. Code below.

Found that the runtime of the test methods is pretty strange: when ran as below the timing of the tests is 

TestCreateNonZeroVertex  <1ms
TestCreateZeroVertex      4ms
TestModifyVertex         <1ms

executed tests multiple times in sequence (to eleminate some buffering stuff of the system) stabilized on this values.

First thought: Garabage Collector -> forced GC in the TearDown (with waiting for finishing). Result: timespan increased (as to expect) but proportionally

TestCreateNonZeroVertex  2ms
TestCreateZeroVertex     8ms (with peaks to 13ms)
TestModifyVertex         2ms

For the little of code and the fact that TestModifyVertex is defacto TestCreateZeroVertex + X, I don't understand this discrepancy. Is it because TestCreateZeroVertex is the first method in the TestSuite? If so, it sounds like a bug in the test environment for me. (Why is the tear down part of the meassuring anyway?)

If someone can explain, why the timing drifts so strong, and maybe how to eleminate this shift (for compairability), I'd be very thankful.

Here the begin of the class:

public class Vertex
{
    public double X { get; set; }
    public double Y { get; set; }
    public double Z { get; set; }

    public Vertex()
    {
        X = 0;
        Y = 0;
        Z = 0;
    }

    public Vertex(double x, double y, double z)
    {
        X = x;
        Y = y;
        Z = z;
    }
...

pretty simple.

I did some test methods:

    [TestMethod]
    public void TestCreateZeroVertex()
    {
        Vertex v = new Vertex();
        Assert.AreEqual<double>(0.0, v.X, "Expected X value to be zero, is: " + v.X);
        Assert.AreEqual<double>(0.0, v.Y, "Expected Y value to be zero, is: " + v.Y);
        Assert.AreEqual<double>(0.0, v.Z, "Expected Z value to be zero, is: " + v.Z);
    }

    [TestMethod]
    public void TestCreateNonZeroVertex()
    {
        double x = 1.1, y = 2.2, z = 3.3;
        Vertex v = new Vertex(x, y, z);
        Assert.AreEqual<double>(x, v.X, "Expected X value to be " + x + ", is: " + v.X);
        Assert.AreEqual<double>(y, v.Y, "Expected Y value to be " + y + ", is: " + v.Y);
        Assert.AreEqual<double>(z, v.Z, "Expected Z value to be " + z + ", is: " + v.Z);
    }

    [TestMethod]
    public void TestModifyVertex()
    {
        double x = 1.1, y = 2.2, z = 3.3;
        Vertex v = new Vertex();
        Assert.AreEqual<double>(0.0, v.X, "Expected X value to be zero, is: " + v.X);
        Assert.AreEqual<double>(0.0, v.Y, "Expected Y value to be zero, is: " + v.Y);
        Assert.AreEqual<double>(0.0, v.Z, "Expected Z value to be zero, is: " + v.Z);

        v.X = x;
        v.Y = y;
        v.Z = z;
        Assert.AreEqual<double>(x, v.X, "Expected X value to be " + x + ", is: " + v.X);
        Assert.AreEqual<double>(y, v.Y, "Expected Y value to be " + y + ", is: " + v.Y);
        Assert.AreEqual<double>(z, v.Z, "Expected Z value to be " + z + ", is: " + v.Z);
    }

Upvotes: 3

Views: 157

Answers (1)

ChriPf
ChriPf

Reputation: 2780

When running your tests separately throw VS 2012 Test Explorer they show more or less equal durations - some ms each. Executing all of them however shows similar results as shown in the question, although in my case TestModifyVertex is the one running longer.

Without having done deeper research I would suspect some class loading issue here - might be your code or (which is probably the case here, as your example is quite compact) some MSTest (or whatever testrunner you are using) code. Subsequent tests might gain some benefit from already loaded classes.

So to have comparable results you could try to run each of the tests separately, but I don't think this is a good idea. We have tried detecting performance decreases in a similar way but gave up due to too much false positives. MSTest just seems not to be the appropriate tools for this.

To get more information about what is happening, depending on your Visual Studio version you code try to profile the test.

Upvotes: 1

Related Questions