CODEWITHSUNDEEP
c#unit-testingdebuggingtesting
Pones
Pones

Reputation: 348

Unit test passes when in debug but fails when run

A search method returns any matching Articles and the most recent Non-matching articles up to a specified number.

Prior to being returned, the IsMatch property of the matching articles is set to true as follows:

articles = matchingArticles.Select(c => { c.IsMatch = true; return c; }).ToList();

In a test of this method, 

    [Test]
    public void SearchForArticle1Returns1MatchingArticleFirstInTheList()
    {
        using (var session = _sessionFactory.OpenSession())
        {
            var maxResults = 10;
            var searchPhrase = "Article1";
            IArticleRepository articleRepository = new ArticleRepository(session);
            var articles = articleRepository.GetSearchResultSet(searchPhrase, maxResults);
            Assert.AreEqual(10, articles.Count);
            Assert.AreEqual(1, articles.Where(a => a.Title.Contains(searchPhrase)).Count());
            var article = articles[0];
            Assert.IsTrue(article.Title.Contains(searchPhrase));
            Assert.IsTrue(article.IsMatch);
        }
    }

All assertions pass when the test is run in debug, however the final assertion fails when the test is run in release:

Expected: True But was: False

In the app itself the response is correct.

Any ideas as to why this is happening?

Edit:

I figured out what the problem is. It's essentially a race condition. When I am setting up the tests, I am dropping the db table, recreating it and populating it with the test data. Since the search relies on Full Text search, I am creating a text index on the relevant columns and setting it to auto populate. When this is run in debug, there appears to be sufficient time to populate the text index and the search query returns matches. When I run the test I don't think the index has been populated in time, no matches are returned and the test fails. It's similar to issues with datetimes. If I put a delay between creating the catalog and running the test the test passes.

Upvotes: 15

Views: 28150

Answers (3)

Eternal21
Eternal21

Reputation: 4664

Obviously the problem will be different for other users, but I just hit it, and figured my solution may help some. Basically when you are running in debug mode, you are running a single test only. When you are running in run mode, you are running multiple tests in addition to the one you are having a problem with.

In my situation the problem was those other tests writing to a global list that I was not explicitly clearing in my test setup. I fixed the issue by clearing the list at the beginning of the test.

My advice to see if this is the type of problem you are facing would be to disable all other tests and only 'run' the test you have an issue with. If it works when ran by itself, but not with others, you'll know you have some dependency between tests.

Another tip is to use Console.WriteLine("test") statements in the test. That's actually how I found my list had items with it leftover from another test.

Upvotes: 6

Basheer AL-MOMANI
Basheer AL-MOMANI

Reputation: 15327

try to print out the actual result that you are comparing them with expected on debug and normal run

in my case, I created entities (JBA) in the test method

in debug mode, the generated ids were 1, 2 and 3

but in the normal running mode, they ware different

that caused my hard-coded values to make the test fail, so I changed them to get id from entity instead of the hard-coded way

hope this helps

Upvotes: 0

wal
wal

Reputation: 17719

Pones, you have since clarified that the unit test fails when not debugging.

At this stage it could be anything however you should continue to run the unit test not debugging and insert the following statement somewhere you know (or think you know) is true

 if(condition)
            Debugger.Launch();

This will do the obvious and allow you to zone in on whats going wrong. 1 Place i suggest is on the IsMatch property (for starters)

Another common place you can run into issues like this is using DateTime's. If your unit test is running 'too fast' then it may break an assumption you had.

Upvotes: 12

Related Questions