Do you put unit tests in same project or another project?

Question

Do you put unit tests in the same project for convenience or do you put them in a separate assembly?

If you put them in a separate assembly like we do, we end up with a number of extra projects in the solution. It's great for unit testing while coding but how do you release the application without all of these extra assemblies?

Answer 1

The past decade has taught us that keeping unit tests near source files provides an unbelievably elevated developer experience and better-structured projects—in other languages. Unfortunately, the shortcomings of the .NET ecosystem make it a tough tradeoff. I would not put test files near source files at the cost of needing to ship the tests together with production code in assemblies.

Answer 2

Another disadvantage of shipping unit test code to production is that we are shipping the knowledge of how to actually use that code. This can be bad because the attacker will have an easier time understanding the business logic after successful decompilation of the unit tests. This can be really harmful. For example, your code could contain special logic to deal with the particularities of taxes between states and countries. This knowledge would then be easily stolen by the attacker, who will have the possibility to sell this to other players in the same area...just by copying the code in production. In my experience, almost no code in .NET is correctly obfuscated in production. There are many tools that can easily decompile the code in a very intelligible form.

Answer 3

After spending some time in TypeScript projects, where tests are often placed in a file alongside the code they are testing, I grew to prefer this approach over keeping them separate:

• It is quicker to navigate to the test file.
• It is easier to remember to rename the tests when you rename the class being tested.
• It is easier to remember to move the tests when you move the class being tested.
• It is immediately obvious if a class is missing tests.
• You don't need to manage two duplicate file structures, one for tests and one for code.

So when I started a new .NET Core project recently I wanted to see if it was possible to mimic this structure in a C# project without shipping the tests or test assemblies with the final release.

Putting the following lines in the project file appears to be working well so far:

  <ItemGroup Condition="'$(Configuration)' == 'Release'">
    <Compile Remove="**\*.Tests.cs" />
  </ItemGroup>
  <ItemGroup Condition="'$(Configuration)' != 'Release'">
    <PackageReference Include="nunit" Version="3.11.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.12.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />
  </ItemGroup>

The above ensures that in the Release configuration all the files named *.Tests.cs are excluded from compilation, and also that the required unit testing package references are removed.

If you still want to be able to unit test the classes in their release configuration you can just create a new configuration derived from Release called something like ReleaseContainingTests.

---

Update: After using this technique for a while I've also found it's helpful to customize your icons in VS Code to make the tests (and other things) stand out a bit more in the explorer pane:

To do this, use the Material Icon Theme extension and add something like the following to your VS Code preferences JSON:

"material-icon-theme.files.associations": {
  "*.Tests.cs": "test-jsx",
  "*.Mocks.cs": "merlin",
  "*.Interface.cs": "Raml",
}

---

Update 2:

If you get the error Program has more than one entry point defined then you can follow the instructions here, which is basically to add <GenerateProgramFile>false</GenerateProgramFile> inside a <PropertyGroup> element in your project's .csproj file, and this will stop the testing framework from adding its own entry point.

Answer 4

I have recently been deploying in docker. I don't see the value in having a separate project when the Dockerfile can easily copy the /src directory and leave the /test directory. But I am new to dotnet and might be missing something.

Answer 5

As others have answered - put tests in separate projects

One thing that hasn't been mentioned is the fact that you can't actually run nunit3-console.exe against anything but .dll files.

If you're planning on running your tests via TeamCity, this will pose a problem.

Let's say you have a console application project. When you compile, it returns an executable .exe to the bin folder.

From the nunit3-console.exe you wouldn't be able to run any tests defined in that console application.

In other words, a console application returns an exe file and a class library returns dll files.

I just got bit by this today and it sucks :(

Answer 6

I know this is a very old question, but I would like to add my experience there I recently change unit testing habit from separate projects to same one.

Why?

First I am very tend to keep main project folder structure same with test project. So, if I have a file under Providers > DataProvider > SqlDataProvider.cs then I am creating same structure in my unit test projects like Providers > DataProvider > SqlDataProvider.Tests.cs

But after project is getting bigger and bigger, once you move files around from one folder to another, or from one project to another, then it is getting very cumbersome work to sync those up with unit test projects.

Second, it is not always very easy to navigate from class to be tested to unit test class. This is even harder for JavaScript and Python.

Recently, I started to practice that, every single file I created (for example SqlDataProvider.cs) I am creating another file with Test suffix, like SqlDataProvider.Tests.cs

At the beginning it seems it will bloat up files and library references, but at long term, you will eliminate moving file syndrome at first glance, and also you will make sure, every single file those are candidates of being tested will have a pair file with .Tests suffix. It gives you easy of jumping into test file (because it is side by side) instead of looking through separate project.

You can even write business rules to scan through project and identify class which does not have .Tests file, and report them to the owner. Also you can tell your test runner easily to target .Tests classes.

Especially for Js and Python you will not need to import your references from different path, you can simply use same path of target file being tested.

I am using this practice for a while, and I think it is very reasonable trade-off between project size vs maintainability and learning curve for new comers to the project.

Answer 7

If NUnit framework is used, there is an additional reason to put the tests in the same project. Consider the following example of the production code mixed with unit tests:

public static class Ext
{
     [TestCase(1.1, Result = 1)]
     [TestCase(0.9, Result = 1)]
     public static int ToRoundedInt(this double d)
     {
         return (int) Math.Round(d);
     }
}

The unit tests here serve as documentation and specification to the code being tested. I do not know how to achieve this effect of self-documenting, with the tests located in a separate project. The user of the function would have to search for the tests to see those test cases, which is unlikely.

Update: I know that such usage of TestCase attribute was not that the developers of NUnit intented, but why not?

Answer 8

I would say keep them separate.

On top of the other reasons mentioned, having code and tests together skews test coverage numbers. When you report on unit test coverage - reported coverage is higher because the tests are covered when you run unit tests. When you report on integration test coverage, the reported coverage is lower because integration tests would not run unit tests.

Answer 9

Put Unit tests in the same project as the code to achieve better encapsulation.

You can easily test internal methods, which means you wont make methods public that should have been internal.

Also it's really nice to have the unit tests close to the code you're writing. When you write a method you can easily find the corresponding unit tests because it's in the same project. When you build a assembly that includes unitTests, any errors in the unitTest will give you an compilereerror, so you must keep your unittest up-to-date, just to build. Having unittest in a seperate project, might cause some developers to forget building the unittest-project, and missing the broken tests for a while.

And you can remove the unit tests from the production code, by using compilation tags (IF #Debug).

Automatic Integration Tests (made i NUnit) should be in a seperate project since they don't belong to any single project.

Answer 10

I am really inspired by the unit testing framework of the Flood NN library by Robert Lopez. It uses a different project for every single unit tested class, and has one solution holding all these projects, as well as a main project that compiles and runs all the tests.

The neat thing is also the layout of the project. The source files are in a folder, but then the folder for the VS project is below. This allows you to make different subfolders for different compilers. All the VS projects are shipped with the code, so it is very easy for anyone to run any or all of the unit tests.

Answer 11

I do not understand the frequent objection to deploying tests with production code. I led a team at a small microcap (grew from 14 to 130 people). We had a half-dozen or so Java apps and we found it EXTREMELY valueable to deploy tests into the field to execute them on a specific machine that was exhibiting unusual behavior. Random problems occur in the field and being able to throw a few thousand unit tests at the mystery with zero cost was invaluable and often diagnosed problems in minutes...including installation problems, flaky RAM problems, machine-specific problems, flaky network problems, etc, etc. I think it is incredibly valuable to put tests into the field. Also, random problems pop up at random times and it is nice to have the unit tests sitting there already waiting to be executed at a moments notice. Hard-drive space is cheap. Just like we try to keep data and functions together (OO design), I think there is something fundamentally valuable in keeping code and tests together (function + tests that validate the functions).

I would like to put my tests in the same project in C#/.NET/Visual Studio 2008, but I still haven't investigated this enought to achieve it.

One big benefit of keeping Foo.cs in the same project as FooTest.cs is that developers are constantly reminded when a class is missing a sibling test! This encourages better test-driven coding practices...holes are more apparent.

Answer 12

In my opinion, unit tests should be placed in a separate assembly from production code. Here are just a few cons of placing unit tests in the same assembly or assemblies as production code are:

1. Unit tests get shipped with production code. The only thing shipped with product code is production code.
2. Assemblies will be unnecessarily bloated by unit tests.
3. Unit tests can affect build processes like automated or continuous build.

I don't really know of any pros. Having an extra project (or 10) isn't a con.

Edit: More Info On Build and Shipping

I would further recommend that any automated build process place production and unit tests into different locations. Ideally, the unit test build process only runs if the production code builds, and copies the product files into the unit tests directory. Doing it this way results in the actual bits being separated for shipping, etc. Additionally, it is fairly trivial to run automated unit testing at this point on all tests in a particular directory.

To summarize, here is the general idea for a daily build and testing and shipping of bits and other files:

1. Production build runs, placing production files into a specific "production" directory.
   1. Build production projects only.
   2. Copy compiled bits and other files into a "production" directory.
   3. Copy bits and other files into a release candidate directory, aka a Christmas release directory would be "Release20081225".
2. If production build succeeds, unit test build runs.
   1. Copy production code to "tests" directory.
   2. Build unit tests to "tests" directory.
   3. Run unit tests.
3. Send build notifications and unit tests results to developers.
4. When a release candidate (like Release20081225) is accepted, ship these bits.

Answer 13

I put them in separate projects. The name of the assembly mirrors that of the namespaces, as a general rule for us. So if there is a project called Company.Product.Feature.sln, it has an output (assembly name) of Company.Product.Feature.dll. The test project is Company.Product.Feature.Tests.sln, yielding Company.Product.Feature.Tests.dll.

You are best keeping them in a single solution and controlling the output via the Configuration Manager. We have a named configuration for each of the main branches (Development, Integration, Production) in lieu of using the default Debug and Release. Once you have your configurations setup, you can then include or exclude them by clicking on the "Build" checkbox in the Configuration Manager. (To get the Configuration Manager, right-click the solution and go to Configuration Manager.) Note, that I find the CM in Visual Studio to be buggy at times. A couple of times, I have had to go into the project and/or solution files to clean up the targets that it created.

Additionally, if you are using Team Build (and I am sure that other .NET build tools are the same) you can then associate the build with a named configuration. This means that if you don't build your unit tests for your "Production" build, for example, the build project can be aware of this setting as well and not build them since they were marked as such.

Also, we used to do XCopy drops off of the build machine. The script would just omit copying anything named *.Tests.Dll from being deployed. It was simple, but worked.

Answer 14

I fluctuate between same project and different projects.

If you're releasing a library releasing the test code with the production code is a problem, otherwise I find it usually isn't (although there's a strong psychological barrier before you try).

When putting tests in the same project I find it easier to switch between tests and the code they test, and easier to refactor/move them around.

Answer 15

Separate project, but in the same solution. (I've worked on products with separate solutions for test and production code - it's horrible. You're always switching between the two.)

The reasons for separate projects are as stated by others. Note that if you're using data-driven tests, you might end up with quite a significant amount of bloat if you include the tests in the production assembly.

If you need access to the internal members of the production code, use InternalsVisibleTo.

Answer 16

My unit tests always go in a separate project. In fact, for every project I have in my solution, there is a separate test project that goes along with it. Testing code is not application code and should not be intermingled with it. One advantage to keeping them in separate projects -- at least using TestDriven.Net -- is that I can right-click on a test project and run all the tests in that project, testing an entire library of application code with one click.