CODEWITHSUNDEEP
c#visual-studio-2010selenium-idenunit-2.5
Graeme Secondwave
Graeme Secondwave

Reputation: 693

Why do my tests fail when run together, but pass individually?

When I write a test in Visual Studio, I check that it works by saving, building and then running the test it in Nunit (right click on the test then run).

The test works yay... so I Move on...

Now I have written another test and it works as I have saved and tested it like above. But, they dont work when they are run together.

Here are my two tests that work when run as individuals but fail when run together:

using System;
using NUnit.Framework;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium;

namespace Fixtures.Users.Page1
{
    [TestFixture]
    public class AdminNavigateToPage1 : SeleniumTestBase
    {
        [Test]
        public void AdminNavigateToPage1()
        {
            NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
            NavigateTo<Page1>();
            var headerelement = Driver.FindElement(By.ClassName("header"));

            Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
            Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
        }

        [Test]
        public void AdminNavigateToPage1ViaMenu()
        {
            NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
            Driver.FindElement(By.Id("menuitem1")).Click();
            Driver.FindElement(By.Id("submenuitem4")).Click();
            var headerelement = Driver.FindElement(By.ClassName("header"));

            Assert.That(headerelement.Text, Is.EqualTo("Page Title"));
            Assert.That(Driver.Url, Is.EqualTo("http://localhost/Page Title"));
        }
    }
}

When the second test fails because they have been run together

Nunit presents this:

Sse.Bec.Web.Tests.Fixtures.ManageSitesAndUsers.ChangeOfPremises.AdminNavigateToChangeOfPremises.AdminNavigateToPageChangeOfPremisesViaMenu: OpenQA.Selenium.NoSuchElementException : The element could not be found

And this line is highlighted:

var headerelement = Driver.FindElement(By.ClassName("header"));

Does anyone know why my code fails when run together, but passes when run alone?

Any answer would be greatly appreciated!

Upvotes: 69

Views: 94940

Answers (11)

Điệp Trần Thị
Điệp Trần Thị

Reputation: 21

I fixed it by changing the list MockData to a function. So, each time we use the list data source, it returns a new instance.

My old code:

public static class MockData {
    private static List<PeopleEntity> AllPeople =
    [...]
}

My new code:

public static class MockData
{
    private static List<PeopleEntity> AllPeople =>
    [...]
}

Upvotes: 0

user487779
user487779

Reputation: 630

One way in which tests fail when run together, but pass individually is with DateTime comparisons. A thread sleep helps here.

Upvotes: 0

Juha Lipponen
Juha Lipponen

Reputation: 198

I had a similar problem with C++. It turned out I was using uninitialized variable values, which caused undefined behavior, varying upon contents of unreserved memory.

I suspected some kind of memory problem and decided to run the unit test with valgrind.

Valgrind gave errors of the form:

==1795== Conditional jump or move depends on uninitialised value(s)

And indeed, my code was using uninitialized values, which of course causes undeterministic behavior, depending on what garbage happens to be in that memory location when running the code.

The type of bug I had was like this:

    class MyClass
    {
        public:
        MyClass();
        void InitStuff(const bool flagIn)
        
        private:
        bool flag;
    };
    
    void MyClass::InitStuff(const bool flagIn)
    {
        if (flag) // Bug! I'm using member variable flag, not flagIn
        {
            DoStuff();
        }
        flag = flagIn;
    }

So, I suggest checking your code for usage of uninitialized values.

Upvotes: 0

Pogrindis
Pogrindis

Reputation: 8091

I realize this is an extremely old question but I just ran into it today and none of the answers addressed my particular case.

Using Selenium with NUnit for front end automation tests.

For my case I was using in my startup [OneTimeSetUp] and [OneTimeTearDown] trying to be more efficient.

This however has the problem of using shared resources, in my case the driver itself and the helper I use to validate/get elements.

Maybe a strange edge case - but took me a few hours to figure it out.

Upvotes: 0

chandmk
chandmk

Reputation: 3481

Two things you can try

  1. put the break point between the following two lines. And see which page are you in when the second line is hit

  2. Introduce a slight delay between these two lines via Thread.Sleep

    Driver.FindElement(By.Id("submenuitem4")).Click();
var headerelement = Driver.FindElement(By.ClassName("header"));

Upvotes: 4

user3613932
user3613932

Reputation: 1397

Such a situation normally occurs when the unit tests are using shared resources/data in some way.

  1. It can also happen if your system under test has static fields/properties which are being leveraged to compute the output on which you are asserting.
  2. It can happen if the system under test is being shared (static) dependencies.

Upvotes: 21

Marc Sputnik
Marc Sputnik

Reputation: 106

I think you need to ensure, that you can log on for the second test, this might fail, because you are logged on already?

-> putting the logon in a set up method or (because it seems you are using the same user for both tests) even up to the fixture setup -> the logoff (if needed) might be put in the tear down method

     [SetUp]
     public void LaunchTest()
     {
        NavigateTo<LogonPage>().LogonAsCustomerAdministrator();
     }

     [TearDown]
     public void StopTest()
     {
        // logoff
     }
     [Test]
     public void Test1()
     {...}
     [Test]
     public void Test2()
     {...}

If there are delays in the DOM instead of a thread.sleep I recommend to use webdriver.wait in combination with conditions. The sleep might work in 80% and in others not. The wait polls until a timeout is reached which is more reliable and also readable. Here an example how I usually approach this:

    var webDriverWait = new WebDriverWait(webDriver, ..);
    webDriverWait.Until(d => d.FindElement(By.CssSelector(".."))
        .Displayed))

Upvotes: 0

Loul G.
Loul G.

Reputation: 1095

If none of the answers above worked for you, i solved this issue by adding Thread.Sleep(1) before the assertion in the failing test...

Looks like tests synchronization is missed somewhere... Please note that my tests were not order dependant, that i haven't any static member nor external dependency.

Upvotes: 3

Jodi
Jodi

Reputation: 900

Without knowing how Selenium works, my bet is on Driver which seems to be a static class so the 2 tests are sharing state. One example of shared state is Driver.Url. Because the tests are run in parallel, there is a race condition to set the state of this object.

That said, I do not have a solution for you :)

Upvotes: 2

casals
casals

Reputation: 353

Are you sure that after running one of the tests the method

NavigateTo<LogonPage>().LogonAsCustomerAdministrator();

is taking you back to where you should be? It'd seem that the failure is due to improper navigation handler (supposing that the header element is present and found in both tests).

Upvotes: 0

default
default

Reputation: 11635

look into the TestFixtureSetup, Setup, TestFixtureTearDown and TearDown.
These attributes allow you to setup the testenvironment once, instead of once per test.

Upvotes: 2

Related Questions