"Analyzer with Code Fix" project template is broken

Question

Short question: How to setup a roslyn code analyzer project with a working unit-test project in Visual Studio 2019 v16.6.2?

A few months (and a few Visual Studio updates) ago I experimented with setting up a code analyzer project using the "Analyzer with Code Fix (.NET Standard)" project template. It worked well and just as documented in all the available how-tos.

Today (VS2019 v16.6.2) I wanted to start a real analyzer project, but unfortunatly the updated template seems to be broken or to be released with a lot of unfinished work in progress. (One minor issue is the fact that suddenly the package manager seems unable to restore packages, because it does not like the vsix project using the same assembly name as the analyzer project.)

The template contains a unit-test project. In the earlier version I experimented with, this test project contained a lot of code that acted like a test infrastructure to make it easy for developers to test the analyzer against real code.

All this code now seems to be integrated in a lot of language-specific nuget packages. But these packages

• are no longer hosted in https://dotnet.myget.org/F/roslyn/api/v3/index.json, but in https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json instead
• they are all pre-release versions and
• they do not contain the expected classes/code.

These are the package references apparently required (ommitted test framework packages):

• Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.MSTest v1.0.1-beta1.20384.1
• Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.MSTest v1.0.1-beta1.20384.1
• Microsoft.CodeAnalysis.CSharp.CodeRefactoring.Testing.MSTest v1.0.1-beta1.20384.1

And this is the sample unit-test code:

using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;

using Verify = Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.MSTest.CodeFixVerifier<
    Analyzer1.Analyzer1Analyzer,
    Analyzer1.Analyzer1CodeFixProvider>;

namespace Analyzer1.Test
{
    [TestClass]
    public class UnitTest
    {
        //No diagnostics expected to show up
        [TestMethod]
        public async Task TestMethod1()
        {
            var test = @"";

            await Verify.VerifyCSharpDiagnosticAsync(test);
        }

        //...

I managed to get the packages installed by adding the correct source url, and I updated all referenced packages to the latest (pre-release) version. But whatever I tried, this template code could not be compiled, because the namespace

Microsoft.CodeAnalysis.CSharp.CodeFix

(used in the alias declaration of Verify) could not be found. Googling for that namespace only brought me back to the myget.org sites for the Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.* packages.

---

So, what do I have to do to setup a working unit-test (ideally mstest) project for my code-analyzer project? I would also be fine using the "old" version that included all the helper code directly in the project, not using the apparently unfinished nuget packages.

---

UPDATE: After implementing AndrewSilvers's answer this second test from the template fails:

    [TestMethod]
    public async Task TestMethod2()
    {
        var test = @"
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;

namespace ConsoleApplication1
{
    class TypeName
    {   
    }
}";

        var fixtest = @"
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Diagnostics;

namespace ConsoleApplication1
{
    class TYPENAME
    {   
    }
}";

        var expected = Verify.Diagnostic("Analyzer1").WithLocation(11, 15).WithArguments("TypeName");
        await Verify.VerifyCodeFixAsync(test, expected, fixtest);
    }
}

stating that one diagnostic was found, but none expected (which is obviously wrong, the test clearly expects a diagnostic).

So the question remains: How to fix this test project?

Answer 1

The roslyn (mstest) packages have moved to a different package repository:

Your nuget.config should look like:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- Only specify feed for Arcade SDK (see https://github.com/Microsoft/msbuild/issues/2982) -->
  <packageSources>
    <clear />
    <add key="dotnet5" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet5/nuget/v3/index.json" />
    <add key="dotnet-eng" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json" />
    <add key="dotnet-tools" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
  <disabledPackageSources>
    <clear />
  </disabledPackageSources>
</configuration>

See for example https://github.com/dotnet/roslyn-analyzers/blob/master/NuGet.config

Answer 2

Quick update for Microsoft Visual Studio Community 2019 16.8.3,

The template now works after the following modifications:

• Initialize a new solution with the 'Analyzer with Code Fix'
• Add a NuGet.Config file in the root of your solution, I.e. run the following command: dotnet new nugetconfig
• Modify the NugetConfig file to add the roslyn-analyzers MyGet feed, like so:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <!--To inherit the global NuGet package sources remove the <clear/> line below -->
    <clear />
    <add key="nuget" value="https://api.nuget.org/v3/index.json" />
    <add key="roslyn-analyzers" value="https://dotnet.myget.org/F/roslyn-analyzers/api/v3/index.json" />
  </packageSources>
  
  <activePackageSource>
    <!-- this tells that all of them are active -->
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

• Close visual studio, Delete the .vs folder in the solution directory
• Run a nuget restore and build it should work now.
• Optionally run Get-Project -all | Update-Package in the package manager console, so all packages are updated to the latest version.

Answer 3

I found a couple of repositories with working unit tests:

• https://github.com/dotnet/samples/tree/master/csharp/roslyn-sdk/Tutorials/MakeConst
• https://github.com/dotnet/roslyn/tree/master/src/Analyzers/CSharp

All of them seem to be using "manual" approach with helper code included in the project. While they don't shed any light on what's going on with project templates and helper code bundled into "beta" nugets, at least they provide a working starting point.

UPDATE: Microsoft has updated its documentation; "Build your first analyzer and code fix" tutorial now has a note in Prerequisites section explaining that there's a bug in the template that should be fixed in Visual Studio v16.7. The note also provides steps for fixing generated projects until then.

UPDATE 2: Followed the steps in the note. In fact, for test sources they are the same as provided by AndrewSilver in his answer. Got the same result, the second test fails. Oh well, back to square one.

UPDATE 3: Ok, I think I got it. Short version: add the following line to your analyzer's Initialize method:

context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None);

Long version

Spent some time debugging the library. The main culprit appeared to be this line that would declare your expected diagnostic message a subject to exclusion and discard it from the list of expected results, thus failing the test due to mismatch between actual (1) and expected (now 0) diagnostics. Was about to file an issue but decided to check if anything similar already exists. It does, indeed, and it points to another lengthy thread with this solution.

By the way, project's README file provides a few more usage examples.

Answer 4

I've run into the same issue recently. Thank you for mentioning changed nuget URL. After changing it in VS settings, I've been able to compile project by removing .CodeFix part in here

using Verify = Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.MSTest.CodeFixVerifier<
    Analyzer1.Analyzer1Analyzer,
    Analyzer1.Analyzer1CodeFixProvider>;

and after that by replacing VerifyCSharpDiagnosticAsync with VerifyAnalyzerAsync in the first unit test and VerifyCSharpFixAsync with VerifyCodeFixAsync in the second one.

Unfortunately, one of the unit tests is not working yet. I would appreciate any help with this issue.