How do you manage automating web based (Selenium specifically) localizations in a time efficient manner?

Question

In my job, we deal with a growing number of countries we support. We're currently using selenium to automate these test cases, but we are on a scrum team with two week sprints and a monthly release cycle.

Every two weeks, we need to run our automation suite. We have a support team dedicated to this task. However, every country adds hours to running a testing suite, and we have many testing suites. We are almost at critical load. We barely have enough time to run the suite once per sprint, and we are going to surpass this in the next few months.

Is there a way to compress the time spent per language? Are there some general assumptions you can make from testing one language, that you can skip the same page for another?

Specifically about Selenium, are there ways to speed up this process? We do have to use a lot of xpath selectors - would working with the development team to get a better HTML layout in order to use id/css selectors improve the timing significantly?

Answer 1

As you can see from the other post and comments you may need to do some experimenting on your own page to determine which approach will find your elements the fastest. One thing that I did find was that searching only descendant elements definitely speeds things up on larger pages, so we will often find a container element such as a div and then find descendants of it like this:

IWebElement myDiv = myWebDriver.FindElement(By.ID("divId")); IWebElement descendant = myDiv.FindElement(By.ID("descendantId"));

With regards to whether you can make some assumptions about different language and markets, yes you can. There are different language types. It is important to test each one of those types, but not necessarily every language within those types. You will want to test a right to left language if you support any, you will also want to test languages that contain different character sets.

With that said, I think there is an important question that is not being asked, and that is... What are you getting out of running your automated tests on multiple languages? Does changing the language ever cause issues with functionality? Certain tests for date formats, etc are useful and can be automated, but if the only difference is the language, then unless you can test that the strings are translated correctly and are correct in context (which you can't unless you speak all of those languages fluently and want to maintain a massive list of strings in various languages) then it doesn't actually buy you much. I would suggest running a small subset of your automated tests in different languages and reduce the number of those different languages down to a handful based on their character sets and/or formatting differences. The biggest concerns that you have with different languages are layout issues where long strings can cause elements to grow larger than expected and cause problems. Most of these layout issues won't be caught by your automation anyways unless you have an extremely mature (and probably not cost effective) automation model that checks for page layout. You are better off getting a Pseudo-loc build that automatically enlarges strings and ensures that you are actually translating all of your text and doing one manual test pass against that pseudo loc build.

A related post and answer about the efficiency of verifying text in multiple languages can be found here: https://sqa.stackexchange.com/questions/1949/do-you-verify-presence-of-text-in-your-automated-tests/1958#1958

Answer 2

I ran some checks on the current browser on a page with 500 elements and here are some of the benchmarks that I got. It might be worth your time to setup a benchmark test program so that periodically (new browser releases) you can recheck the benchmarks. I checked mine using Firefox 7, IE9 and Webdriver 2.11.0. Each test looked for the same element 5 times and then determined the average response.

SUMMARY:FirefoxDriver [5 tries] =========================
Find By ID Avg: 169 ms
Find By Name Avg: 17 ms
Find By Relative XPath Avg: 7 ms
Find By Absolute XPath Avg: 6 ms
Find By CssSelector Avg: 6 ms
Find By ClassName Avg: 5 ms
=========================
SUMMARY:InternetExplorerDriver [5 tries] =========================
Find By ID Avg: 309 ms
Find By Name Avg: 90 ms
Find By Relative XPath Avg: 82 ms
Find By Absolute XPath Avg: 82 ms
Find By CssSelector Avg: 53 ms
Find By ClassName Avg: 75 ms
=========================

Running Selenium Grid to run multiple instances is also a great way to improve performance. My guess is that with a virtual server you should be able to instantiate multiple test easily and that don't conflict. Also if you can use the HTMLUnitDriver as it is much more efficient than any other browser (I'm not sure if your requirements state that you need to use a real browser, multiple browsers or just validation).

Answer 3

If you have multiple sites to test with the same code, and the tests don't interfere with each other, Selenium Grid should do the job for you. It will take some extra effort - and perhaps some extra hardware - but once properly configured, the number of languages you can test is only limited by the number of concurrent tests the hardware can handle.

Regarding the selectors, moving to CSS can yield great performance improvements. As a rule of thumb: when possible, use CSS; XPath should only be used as a fallback scenario. Things that CSS doesn't support are e.g. navigation to parent nodes, or matching on the contents of an element, so don't hesitate to use XPath when beneficial. You should also check out the related posts on that blog: they offer some additional background and migration tips.