How can I unit test Roslyn diagnostics?

Question

How can I unit test my own custom analyzers and Code Fix providers?

_{I'm sitting in front of my computer with my hands on the keyboard but I don't know what to type.}

Answer 1

There are the Microsoft.CodeAnalysis.Testing (GitHub) packages from Microsoft, where currently pre-release versions are available on MyGet. These are also included in the Visual Studio 16.6 template Analyzer with Code Fix (.NET Standard).

In order to consume these packages, you have to add their package source to the NuGet Config file: Go to /configuration/packageSources and add:

<add key="roslyn-analyzers" value="https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json" />

If you have no NuGet Config file yet, you can create a new one with dotnet new nugetconfig.

When adding/updating these packages with a NuGet Package Manager, you may have to include prereleases.

These packages support testing DiagnosticAnalyzer, CodeFixProvider, and CodeRefactoringProvider, written in either C# or Visual Basic, via MSTest V2, NUnit, or xUnit.net, by adding the particular PackageReference:

Microsoft.CodeAnalysis.[CSharp|VisualBasic].[Analyzer|CodeFix|CodeRefactoring].Testing.[MSTest|NUnit|XUnit]

Testing CompletionProvider is currently not supported.

Warning: the Visual Studio 16.6 template contains errors and will not compile. Refer to the documentation for the correct namespaces and method names.

Answer 2

Testing analyzers with the default infrastructure provided by the template project is quite complicated. There is a nice project RoslynNUnitLight create by Dustin Campbell which makes testing of Roslyn analyzers, code fixes and refactorings super easy. Unfortunately, it's no longer maintained. I created a fork and made a few adjustments, like:

• Removed dependencies from the unit test framework. Now you can create a test with your favorite framework: xunit, nunit, mstest, etc.
• Added ability to locate diagnostic position by the line number
• Added infrastructure for testing CompletionProviders
• Improved error messages
• Presents code diff using diff tool in debug mode

This fork is called RoslynTestKit and it's available on GitHub https://github.com/cezarypiatek/RoslynTestKit

Nuget package: https://www.nuget.org/packages/SmartAnalyzers.RoslynTestKit/

You can find sample tests built with RoslynTestKit in this project https://github.com/smartanalyzers/MultithreadingAnalyzer/tree/master/src/MultithreadingAnalyzer.Test

Answer 3

A good place to start is by creating a new solution using the "Diagnostics and Code Fix" template. This will create a unit test project that comes with a few classes which allow you to very easily test your diagnostics.

However this also shows its weakness: the classes are hardcoded in your codebase and are not a dependency which you can easily update when needed. In a still constantly changing codebase like Roslyn, this means you will fall behind quickly: the test classes are aimed at Beta-1 while Roslyn is already at RC2 at the time of writing.

There are two solutions I propose:

1. Read through the rest of this post where I give a broad layout of what is being done in those classes and what their key aspects are. Afterwards you can create your own implementation according to your needs.

2. Remove all those classes and instead use the RoslynTester NuGet package which I created based on these helpers. This will allow you to immediately get started with the RC2 version of Roslyn and keep it more easily updated. For more information, take a look at my blog or the Github page.

---

The idea

The idea behind the helpers is simple: given one or more strings that represent class files and one or more objects that represent expected diagnostic results, create an in-memory project with the given classes and execute the analyzers.

In the case of a CodeFix provider, you can also specify how the code should look after it's transformed.

The execution

How is it called?

This is an example test that shows a warning when you have an asynchronous method whose name doesn't end with "Async" and provides a CodeFix to change the name.

[TestMethod]
public void AsyncMethodWithoutAsyncSuffixAnalyzer_WithAsyncKeywordAndNoSuffix_InvokesWarning()
{
    var original = @"
using System;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
   class MyClass
   {   
       async Task Method()
       {

       }
   }
}";

    var result = @"
using System;
using System.Text;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
   class MyClass
   {   
       async Task MethodAsync()
       {

       }
   }
}";

    var expectedDiagnostic = new DiagnosticResult
    {
        Id = AsyncMethodWithoutAsyncSuffixAnalyzer.DiagnosticId,
        Message = string.Format(AsyncMethodWithoutAsyncSuffixAnalyzer.Message, "Method"),
        Severity = EmptyArgumentExceptionAnalyzer.Severity,
        Locations =
        new[]
        {
            new DiagnosticResultLocation("Test0.cs", 10, 13)
        }
    };

    VerifyCSharpDiagnostic(original, expectedDiagnostic);
    VerifyCSharpFix(original, result);
}

As you can see the setup is very straightforward: you determine how the faulty code looks, you specify how it should look and you indicate the properties of the warning that should be displayed.

Creating the project

The first step is to create the in-memory project. This consists of a few steps:

• Create the workspace (new AdhocWorkspace())
• Add a new project to it (.CurrentSolution.AddProject())
• Add references to relevant assemblies (.AddMetadataReferences())
• Add documents to the solution(solution.AddDocument())

Gathering the diagnostics

Here we will use the documents we just created. These two lines are most important:

var compilation = project.GetCompilationAsync().Result;
var diagnostics = compilation.WithAnalyzers(ImmutableArray.Create(analyzer))
                             .GetAnalyzerDiagnosticsAsync()
                             .Result;

Verifying the diagnostics

At this point you have everything you need: you have the actual results and you have the expected results. All that is left is verifying that the two collections match.

Applying a Code Fix

This roughly follows the same pattern as the diagnostics but adds a little bit to it.

• Create a document
• Get the analyzer
• Create a CodeFixContext

var actions = new List<CodeAction>();
var context = new CodeFixContext(document, analyzerDiagnostics[0], 
              (a, d) => actions.Add(a), CancellationToken.None);
codeFixProvider.RegisterCodeFixesAsync(context).Wait();

• Apply the Code Fix

var operations = codeAction.GetOperationsAsync(CancellationToken.None).Result;
var solution = operations.OfType<ApplyChangesOperation>().Single().ChangedSolution;

• Optionally: verify no new diagnostics have been triggered due to your refactor
• Verify the expected source and the resulting source are the same

---

If everything is still a little blurry, definitely take a look at the exact source code. If you want clearer examples of how to create your own diagnostics, take a look at my VSDiagnostics repository or my blogpost on writing your own.