Cucumber test with Selenium and Ruby in Jenkins server fails but passes in local machine

Question

I have some cucumber tests made using the gem 'cucumber-rails'.

This tests use another gem called 'page-objects' (from cheezy) and in my tests I use Selenium and Chrome-driver.

To make the tests run in the Jenkins server (that has no X) I use the gem 'headless' and I've configured the server to use Xfvb (my local machine runs Elementary OS Freya 0.3.2 and the server runs Ubuntu 15.10).

When I run these tests in my local machine (also using headless), they all pass flawlessly every time. The same happens when I run the tests in Jenkins' server from the Terminal.

But when I run them on Jenkins CI, some fail (the error is "didn't find an element" - when the element is there).

Both tests fail because it cannot find an element (the element exists and when it fails I print the screen just to check what's happening. The page print show that it is on the right page and displays the element).

The strange thing is that they both fail at the second step (the first one is going to the login page and authenticate, which it does correctly). Also, there's another test that runs first that always passes (the tests have pretty much the same structure).

Both tests run over the same server (the webpage they access is the same and the data is the same).

I have looked at this over and over and I can't find any explanation. I know the problem is not very detailed, so if there's anything missing that might be important, just tell me and I'll update the post.

If you happen to know any reason why the tests might fail on a 'Xless' server, I would greatly appreciate it!

Thanks a lot for the help!

---

Edit: The failures are both "element not found" and the element is present.

I've also made the Jenkins user part of the sudoers. Any ideas?

Thanks a lot

Answer 1

Ok, so I found out the problem and the solution.

The problem was that the elements were still not loaded when performing their lookup. The solution was to use PageObject.wait_until to wait for them.

I realized that this was the problem after noticing the errors were sometimes these ones: Stale Element Reference

But there's a question that remains, still: why does this only happen in Jenkins CI? (whoever finds the solution will earn the bounty :) )

Answer 2

Things you can try:

Reduce the test to the very basics: Instead of running it all, just that concrete page and try to find that element. Check if that works. It probably won't...but you can pinpoint better the problem.

Add some waits in the code (sleep <number of seconds>), to see if your code is trying to find something "too fast". This happens time to time.

Instead of using headless, try to use something else and run it in a browser (with some gems like capybara you can switch between headless, firefox or chrome).

Can you find some sort of JS logs, to see if there are any errors?

Do Jenkins and your environment have exactly the same gem versions?

Another thing you can do to debug it is use something like byebug and set a breakpoint just before it fails. You can then use the interactive console to inspect the page, and run different commands to try to find the element. You would have to do this while in jenkins, but as a last resort...

Answer 3

The only way to solve these issues is to debug the error received. To debug, we can add screenshots at different places in the scenario to identify the cause of the failure. If this is failing as page not completely loaded, then to solve this issue, is to increase the default wait time to open the particular page or to locate an element. Once the page loads or element is located, set the default time to its original value.

Example: 1. You can add screen shot in you scenario:

Scenario: Click on Sign In Link
Given Joe opens "www.yahoo.com" page
When Joe takes screenshot with file name "ScreenShot1"
When Joe clicks "Sign In" link

The second line "When Joe takes screenshot with file name "ScreenShot1" can be used to take the screen shot

And(/^(\S*) takes screenshot with file name "([^"]*)"$/) do |user, file_name|
  takes_screenshot_with_file_name(file_name, "ScreenShot")
end

def takes_screenshot_with_file_name(file_name, message)
  page.save_screenshot("../../target/#{file_name}.png", :full => true)
end

2. Default wait time can be increased in below mentioned way:

The second line "And Joe waits for "Sign In" link" can be used to increase the default wait time

Scenario: Click on Sign In Link
Given Joe opens "www.yahoo.com" page
And Joe waits for "Sign In" link
When Joe clicks "Sign In" link

And(/^Joe waits for "([^"]*)" link$/) do |linkName|
  Capybara.default_wait_time = 120
  page.should have_content(linkName)
  Capybara.default_wait_time = DEFAULT_WAIT_TIME
end