CODEWITHSUNDEEP
c#.netperformanceunit-testingmicrosoft-fakes
avivr
avivr

Reputation: 1523

Unit tests become slow when using Microsoft fakes in OTHER tests

We have a unit tests project which is still using the "old-style" private accessors in many tests. Since they're a maintenance nightmare, we're trying to get rid of them, and move to new new Microsoft Fakes framework, using Shims where needed.

Recently we wrote some new unit tests which use Shims, and noticed that for some reason this caused a few OTHER, old, tests, which were not modified, to run considerably slower. By slower I mean run times of about ~10sec instead of ~900millisec for the affected tests. Running the affected tests on their own didn't seems to have this effect though - it only occurs when running them after tests with Shims.

Initially we thought this might be simply due to initialization problems, causing tests to influence one another. However, after some experimentation, we found that the slowdown occurs even without actually adding any new test code. Simply adding the following snippet before one of the slowed-down tests caused the same effect of the test running slower:

using (ShimsContext.Create()) {}

Debugging seemed to show that the the code being tested was indeed running much slower (not the unit test code itself) , but we couldn't identify which part of it. We're not able to identify why these tests are affected while others are not. At this point we tried profiling these tests (using the new "profile test" option in VisualStudio). However, it turns out that profiling tests with Shims is not possible for some reason. The following exception was thrown:

Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationException: UnitTestIsolation instrumentation failed to initialize. Please restart Visual Studio and rerun this test

As a last resort we also tried moving all tests using Shims to a separate test project in the same solution. This did seem to help, and all test run times returned to normal. We used test playlists to run each project's tests before the other's, and in both cases run times were OK. It's not really a solution though, and feels more like circumventing the actual issue. So, we're not sure how to proceed. Any thoughts and ideas would be helpful.

Thanks.

Upvotes: 3

Views: 1111

Answers (2)

Jaded
Jaded

Reputation: 1831

"Other" tests should be explicitly executed in shimless context (ShimsContext.ExecuteWithoutShims), because it looks like even disposed ShimsContext in other tests may have detours to logic that doesn't use shims.

Upvotes: 0

JakJackal
JakJackal

Reputation: 121

The Microsoft documentation, Better Unit Testing with Microsoft Fakes (RTM).pdf, states that you will see a performance decrease when using Shims.

This article also goes over the performance impact of Shims: http://technet.microsoft.com/en-us/windows/jj863250.aspx

Upvotes: 3

Related Questions