Test Automation framework - Tools important?

Question

I have been working on Test Automation from last few months and have been using the tool named "Testcomplete". But I have noticed that the tools do not matter a lot in the field of automation. Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application. Apart from this, you will always have to build a automation framework which will serve your needs writing code.

So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct? In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools). Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?

Answer 1

So, I figured I would give you an idea of my background in test automation and software development engineering in test, so that you can get an idea why I'm answering the way that I am. I've been an SDET for over 15 years now, and I've written, modified and built out test automation frameworks for lots of different companies in different industries (for example: RAID storage, website ads, HR software, large corporate websites, smaller websites). I've written frameworks for testing the front end (examples: web, android, iphone, java, php, python), the back end (API, SOAP, even some proprietary storage backends), database (examples: SQL, NoSql, MongoDB...), security testing, load testing, stress testing, and even a touch of big data (examples: hive, cassandra, hadoop, elasticsearch).

So, now that that's out of the way, let's get to your questions:

But I have noticed that the tools do not matter a lot in the field of automation.

I would tell you that this very much depends on the industry that you're involved with. For any company that has a website they need automated testing on, by and large, you'll see a few tools pop up over and over again (Selenium, cypress, playwright). However, there are others that absolutely require custom tooling, or have other unique scenarios that require a more specialized tool.

Only thing you expect from an automation tool is the ability of the tool to spit out the recognition strings for the different controls used in the test application.

I'm not really sure what you mean by this, but I'm going to take a guess and say that you're saying, "The only thing expected from an automation tool is for the tool to have the ability to find or detect the identified test object, and to allow some interactions to be done with it."

To be a bit more specific, if you were talking about WebUI automated testing, I believe you're saying the equivalent here of "The only thing that is expected from a Web UI testing tool is for the tool to be able to perform various browser level operations (load, close, resize, etc), and locate identified elements on the page (links, images, divs, html elements, pixels, whatever), and allow you a standardized way of interacting with the element in a standardized way that mimics (clicks, hover, roll the mouse cursor over it) a real user."

Assuming that I've interpreted that correctly, I would actually argue and say that no, that's not the only thing that I expect from a web testing tool. For companies focused in web testing, by and large if they have a test automation framework established, it is more often than not written in Selenium. Selenium is a huge name in the web ui testing scene, but I generally find it generally does not take a lot of convincing for people to be willing to refactor all of their test automation code, as Selenium will generally end up in a state where the tests take a long time to run, and throw up so many false positives for test failures that they become nearly unbearable to use, maintain or even triage.

So, most of the time, I'm able to convince the companies I've worked for to move to a different framework, as Selenium really is more of a library than it is a fully fleshed out framework.

The important differentiator here is that a library does NOT have some advanced features that you really do need in the long run if you're going to have success with test automation, long-term. Some of the features that you really do NEED were not included in the Selenium library by default (example: retry functionality to determine the source of a failure, specific error messaging when an error condition is encountered (example, the good old Element_not_found exception, but that error message by itself does not give me enough information to be able to diagnose and triage the issue, and the test s are generally so flakey and unreliable that whenever the automated test encounters a failure condition, if by some fortune the developers are aware of, and even rarer: invested in making sure the automated tests are passing, it's a real struggle to look at the logs that selenium makes and determine the root cause of the exception without having to examine a video or a screenshot taken at the time of failure.

There are multiple examples that I could use, but I'll limit myself to some of the most common issues and complaints that I've encountered. They are (in no particular order): advanced test outcome reporting that the entire company can take a quick glance at and understand if automated tests passed or failed, very specific and descriptive stack traces for failures, and test failure retry logic. Any tool available for whatever I'm testing, I want to be sure that a tool I am recommending has all of these key features, as test automation can very quickly become a fool's errand, as the level of investigation to follow through and drive towards either a passing test, or dig into the defect and assist a developer to locate and fix the issue detected. in the amount of time debugging and staring at screenshots robust, specific and descriptive reporting,traces, . use a couple of prominent examples, which are: but one of the biggest things to me that never made a lot of source code for selenium, and of the things that a framework offers that a library does not. (or, what a library doesn't have) cypress for example comes with a ton of features that you would have had to implement to use with selenium, such as: graphical reporting to share across all roles of the company, historical test data reporting, the ability to intercept, modify, add, remove or even just delay a a backend request or response before the actual server can supply that information, or the ability to have retry logic be executed when an error is encountered, etc).

networking interruptions (used to test failure conditions, or simulate edge case conditions for load testing), and logging that is so clear that you don't need to have a screenshot or a video to understand why a test reported a failure. care to look at the logs, or even understand it, honestly. will generally speaking push back upon hearing this. except ha and that immediately come to my mind are: advanced and customizable error codes, end up convincing the companies I've worked with to move to a tool that does not use Selenium. One of the biggest reasons for this is that for the longest time, you could not interrupt any calls that were being made to the backend API. Additionally, generally the error messaging with Selenium leaves a lot to be desired, and the error messages are really not all that clear on first inspection regarding why a test has failed. Generally speaking, with Selenium's failures, you have to end up taking a video of the test or taking a screenshot wherever you are running the test to be able to really determine if a defect that has been reported is even a defect at all. And believe me, when you have a repository that contains hundreds of tests, and all of a sudden most of them start reporting failures, it can literally take days to go through the logs and the screenshots, even if all you're doing is determining that they all failed for the same reason (for example, your login page was not working during the time that the tests were executed) can be an incredibly painful process.

And I haven't even started to yet speak to development engagement. So, hold on to your hat :)

Apart from this, you will always have to build a automation framework which will serve your needs writing code.

I guess this would depend on what your definition of "building a framework". To some, building a framework means that you're creating a lot of custom code, say for example a log parser that would detect when a stack trace is happening in the system logs, while to others the same terminology means you're identifying and adding locators in a page object, so it all depends. But, at least in my mind, "building a framework" means the more technical side of the equation, meaning creating custom code and log parsers and all of that. But to answer your question, I'd say that to have a truly effective and good test automation framework, there will always be some parts of "building the framework" you'll have to do. Of course, that assumes that you're going to do things that I always champion within test automation, such as log parsing when failures happen, but I suppose you could get away with not doing those things, too.

So my question is, Is my thinking that automation tools do not matter a lot in the field of automation correct?

I wouldn't agree with that statement, to be honest. I'd say that tools do matter, but you've got to be willing to be flexible and use the right tools for the task at hand. For example, if I'm testing an API, I wouldn't want to use Selenium to do that, except in a few very special cases.

In the sense, you can use any tools to get your automation running. Or Do the tools really matter? (Please ignore the costs factor of the tools).

Yes, mostly. I'd say to have a truely end to end automated testing environment, you'll have to have several tools, depending on the use case. For example, although I might use cypress to test a web-based product, and also use cypress to test the API endpoints, I'd probably want to implement a Jmeter or similar tool to do load and performance testing of that same product. Or, I might want to implement a different tool for visual regression testing, like Applitools eyes.

Also if I need to learn a new automation tool, then what do I concentrate on? Or how do I go about learing the tool? In short, what exactly does "learning a tool" mean?

Another set of really good questions. I'd say that you need to focus on the "gotchas" that exist within the framework. By this, I mean the things that are special about the framework, or are somewhat uncommon knowledge. For example, in Selenium, knowing the difference between explicit waits and implicit waits sets me head and shoulders above most other candidates when applying for a job. Or, knowing the ins and outs of iframe management within Selenium would be another good example. Anyhow.... as far as what to focus on when learning a new tool, I'd say the first thing to do is just to USE the tool, and not focus on learning the specializations within the tool at first. So, if I were to pick up a new tool, I'm going to go ahead and create a POC to address some task that I currently have assigned to me, and make sure that I can use the documentation available to implement a simple use case. After I've completed the POC and know the basics of using the framework, then I'll try and do a more complicated case. Usually tho just by using the framework itself, with enough time and exposure, you'll pick up the specialized cases of the framework.

Anyhow, overall I guess what I'd say to close things out is that in my experience, the most successful test automation frameworks that I've seen in my career have been the ones that have been implemented in the same language as whatever you're trying to test is written in, and generally lives in the same codebase as the code under test. The reasons for this are myriad, tho most of them stem from the idea of "white box" or "grey box" testing. So, I wouldn't neccesarily tell you to go and study up on this framework or that framework, rather I'd tell you that your time should be spent on learning a myriad of different programming languages (example: Python, javascript, java), so that you can implement a test automation framework that is suited to the code under test.

Hopefully I've answered your questions! Good luck!

Answer 2

Variety of Test Automation Frameworks and Test Automation Tools are already available in the market. Thus, I would not recommend the built your own Test Automation Framework at all.

As far as selection of automation tools is a concern, I would say it does matter on the following basis:

1. Support: How much level of support you have when you are choosing an automation tool for your project.
2. Community: How big community is using that tool and how responsive that community is about sharing knowledge?
3. Pricing: (Proprietary or Open Source) Last but not the least is the pricing of the automation tool that you are planning to introduce in your project.
4. QA teams' expertise also matters sometimes. For example, in case your QA team does not have a developer or semi-developer skills vs Not-Technical QA Team, etc.

Regarding the Automation framework, there are many automation frameworks also available in the market already, therefore no need to reinvent the wheel. and selection of automation framework mostly depends on your selection of scripting language.

For example, if you choose python as your scripting language, then you have option to choose UnitTest, PyTest etc. as an automation framework.

In case of Java, you have option of JUnit and TestNG as an automation framework.

and so on, the base on your selection of scripting language.

Finally, when it comes to structuring your automation framework, it solely depends on many things as the following:

1. Your nature of the project
2. Single product vs multiple products
3. and many more... Check an example of multiple product automation project directory structure. https://github.com/pancht/python-selenium-framework

I hope, in some way, I would have helped you out in giving an answer to your question.

Thanks,

Panchdev Singh Chauhan

Answer 3

An other piece of advice: sometimes you'll need wrapper classes around certain frameworks. We were using WatiN, which was really good at its time, but it lacked Chrome (it had a small percentage that time) support. The thing that killed WatiN for us was the lack of coping with new FireFox releases: FireFox 8 was out, and we had to run on our tests on FireFox 3.6... Selenium was the solution, but it had a totally different logic and we already had more than a 1000 tests.

So we had to create a wrapper class around Selenium to "fake" it was WatiN. We had some issues, but we had to rewrite only some special cases... And not all tests.

The point is, sometimes, support for frameworks just cease to exist. But with an own framework focusing on what the test actually does instead of how it works would save you in this situation.

Answer 4

Some additional thoughts on why the tool is important:

• Community - What's the user community like? Are there a lot of user-generated resources out there to help?
• Support - (if vendor) What's customer support like? Do they fix problems quickly? Is it easy to find solutions to common problems?
• Extensibility - Often in test automation, you'll need to roll your own or code work-arounds, if the tool does not support a particular type of object in your application. How easy is it to extend the product? What programming language does the tool use? What kind of support do you get from the IDE?

Answer 5

My 3 best reasons for choosing which tool to use:

• it works. This is important, not all tools work in all scenarios i.e. flash, silver light, adobe air, legacy apps with no automation support, etc.
• whole team skills. This includes not only testers, but also developers. Test automation shouldn't be an isolated effort, developers should also collaborate on it. This is far easier when dev and test are using the same language/platform.
• price. Doesn't have to be free (but it could), but of course its an important factor.

Personally we use the same test runner as the one for the unit tests. That along with extra third party automation pieces that do the plumbing for you.