UI automation testing with Selenium and NUnit Order of Tests Executions

Question

It's almost my beginning of Automation Testing so I would like to apologize if I cannot make you clear!

So, after a two weeks of reading blogs about automation, I have decided to use NUNIT with Selenium Webdriver for UI Automation.

My application is of enterprise level that has remaint under developement for 3 years. It is a project and portfolio management system.

I have hundreds of pages and about 50% of which perform CRUD operations.

I decided the following strucutre for my application:

I am using test data as JSON in following format and retrive this data to my view models:

[
  {
    "Name": "Finance",
    "Description": "division with complete test data",
    "Color": "#ff0000",
    "ExpectedStatus": {
      "WillBeAdded": true,
      "WillBeDeleted": true,
      "WillBeUpdated": true
    },
    "PerformableActions": {
      "ShouldAdd": true,
      "ShouldDelete": false,
      "ShouldUpdate": true
    }
  },
  {
    "Name": "IT",
    "Description": "IT projects",
    "Color": "pink",
    "ExpectedStatus": {
      "WillBeAdded": true,
      "WillBeDeleted": true,
      "WillBeUpdated": true
    },
    "PerformableActions": {
      "ShouldAdd": true,
      "ShouldDelete": false,
      "ShouldUpdate": true
    }
  },
  {
    "Name": "Business",
    "Description": "division with name and color name",
    "Color": "yellow",
    "ExpectedStatus": {
      "WillBeAdded": true,
      "WillBeDeleted": true,
      "WillBeUpdated": true
    },
    "PerformableActions": {
      "ShouldAdd": true,
      "ShouldDelete": false,
      "ShouldUpdate": true
    }
  },
  {
    "Name": "",
    "Description": "division without name and color name, should add white color",
    "Color": "",
    "ExpectedStatus": {
      "WillBeAdded": true,
      "WillBeDeleted": true,
      "WillBeUpdated": true
    },
    "PerformableActions": {
      "ShouldAdd": true,
      "ShouldDelete": true,
      "ShouldUpdate": true
    }
  },
  {
    "Name": "",
    "Description": "without name and color name and will not be added",
    "Color": "black",
    "ExpectedStatus": {
      "WillBeAdded": false,
      "WillBeDeleted": false,
      "WillBeUpdated": false
    },
    "PerformableActions": {
      "ShouldAdd": true,
      "ShouldDelete": false,
      "ShouldUpdate": false
    }
  }
]

I am here using two things:

1. PerformableActions i.e what actions can I perform by using this test case data. E.g shouldAdd means should this TestCase be performed for adding record, shouldDelete means should this testCase be run for deleting the record, similarly shouldUpdate runs.

2. ExpectedStatus i.e what will be the expected result of the test case, WillBeAdded means would this data be added to grid. Similarly WillBeDeleted and WillBeUpdated work.

Questions:

1) I am using same test data for all the CUD operations, the reason for this is that I have to maintain the CUD flow in my app. Since only need to test UI so I am just assuming that Create, retrieve and delete will work in order.

Is this the right way of doing testing in cases of CRUD operations?

2) Think that I have to run all the tests in my project, it may be hundreds or thousands, How would I maintain order, since I can see the only possible way is to use Order attribute in NUNit. How can I use that or is there any alternate to test group of independent modules and dependent modules seperately?

Answer 1

1. There is nothing wrong with your approach. You can create a function for each case and make C, U and D in the order you wish. Maybe you need add waits between them to let the backend do it's enterprisy stuff.
2. OrderAttribute with NUnit works good enough, even if there are better ways. You can also make lists of test cases and run them in the order successively. Why not? Don't limit yourself to classes and methods.

3. I know, you tagged your question with C#, but for this amount of work maybe it is worth to look at some other Libraries than your have now. If your not afraid of very little learning, you can enjoy F#:

   • with Expecto you have full control in which order and which test you run
   • with Canopy you have a nice DSL for the selenium. Here is the starter kit with little example. And you can even run it in the docker with chrome in headless mode
   • and you can even get intellisense for your Json test definitions with Json Type Provider

UPDATE: for the ordering you can define the own attributes, or even use existing from NUnit, and call the methods per reflection in correct order. Or you define own DSL for tests and traverse the lists and extract the info you need.

    public struct CRUD
{
    public bool C;
    public bool R;
    public bool U;
    public bool D;
    public CRUD(bool c, bool r, bool u, bool d)
    {
        this.C = c;
        this.R = r;
        this.U = u;
        this.D = d;
    }
}
public enum ActionKind
{
    Create,
    Read,
    Update,
    Delete
}
public struct TestResult
{
    public ActionKind Action;
    public bool IsPerformed;
}
public class TestCase
{
    public string Name { get; set; }
    public string Description { get; set; }
    public string Color { get; set; }
    public CRUD Action { get; set; }
    public List<TestResult> Actual { get; set; }
    public List<TestResult> Expected { get; set; } // or List<TestResult>

}
public class Tests
{
    public void TestAListOfCases()
    {
        var cases = new List<TestCase>
        {
            new TestCase { Name = "Finance", Description = "division with complete test data", Color = "#ff0000",
                Expected = new List<TestResult> {new TestResult {Action = ActionKind.Create, IsPerformed = true },
                                                 new TestResult {Action = ActionKind.Update, IsPerformed = false},
                                                 new TestResult {Action = ActionKind.Delete, IsPerformed = true }
                }},
            new TestCase { Name = "IT", Description = "Description"}
        };

        cases
            .Where(x => x.Name == "IT")
            .Where(x => x.Color != "pink")
            .Select(RunTestCase)
            .Where(x => !x.Item2)
            .Select(x => OutputFailedTestCase(x.Item1)); // boah c# is verbose
    }
    public Tuple<TestCase, bool> RunTestCase(TestCase testCase)
    {
        foreach(var exp in testCase.Expected)
        {
            switch (exp.Action) {
                case ActionKind.Delete:
                    // do delete if needed and create actual result
                    var actual = exp.IsPerformed
                        ? new TestResult { Action = exp.Action, IsPerformed = true }
                        : new TestResult { Action = exp.Action, IsPerformed = false };
                    break;

            }
        }
        var isFailed =
            Enumerable.Zip(
                testCase.Actual, 
                testCase.Expected, 
                (expected, actual) => expected.Action == actual.Action && expected.IsPerformed == actual.IsPerformed)
           .All(x=>x);
        // your selenium stuff
        return Tuple.Create(testCase, isFailed);
    }
    public bool OutputFailedTestCase(TestCase testCase)
    {
        // write to console or do something else
        Console.Write($"{testCase.Name} is failed to perform following actions: {calculateFailedActions}");
        return true;
    }
}