How to set up working automated browser tests on Travis CI?

Question

I am trying to set up automated tests for my web app on Travis CI. I am talking about automated browser tests - running unit tests is simple, but I need a process/system test that will start a headless browser and perform scripted tests of various use cases.

My application is in PHP so I decided to go with PHPUnit, Selenium and headless Firefox.

After a lot of research and trial and error I ended with following .travis.yml file:

language: php
php:
  - '7.1'
services:
  - mysql
addons:
  firefox: latest
env:
  - MOZ_HEADLESS=1
    DISPLAY=:99.0
    SELENIUM_FIREFOX_DRIVER=/home/travis/build/lotcz/zSample/geckodriver
before_install:
  - sudo apt-get update > /dev/null
  - wget https://selenium/download/url -O selenium-server.jar
  - wget https://github.com/mozilla/geckodriver/releases/download/v0.21.0/geckodriver-v0.21.0-linux32.tar.gz
  - tar -xzf geckodriver-v0.21.0-linux32.tar.gz
install:
  - sudo apt-get install apache2
  - sudo service apache2 start
  - mysql -e 'CREATE DATABASE IF NOT EXISTS zsample;'      
before_script:      
  - nohup java -jar -Dwebdriver.gecko.driver=$SELENIUM_FIREFOX_DRIVER selenium-server.jar &
  - composer install
script:
  - phpunit --fail-on-risky --fail-on-warning --stop-on-skipped --stop-on-incomplete --verbose --debug --colors
after_failure:     
  - cat nohup.out

I edited out some pieces specific to my application. Just believe me that I set my application correctly before running the test.

Now a very simple test may look something like this:

class VisitorLoginTest extends PHPUnit_Extensions_Selenium2TestCase {

  public function setUp() {
    $this->setHost('localhost');
    $this->setPort(4444);
    $this->setBrowserUrl('http://localhost');
    $this->setBrowser('firefox');
  }

  public function tearDown() {
    $this->stop();
  }

  public function testFrontPage() {
    $this->url('/');
    $content = $this->byClass('main-title')->text();
    $this->assertEquals('Hello', $content);
  }

}

When my test is run I get this:

The Selenium Server is not active on host localhost at port 4444.   
OK, but incomplete, skipped, or risky tests!
Tests: 1, Assertions: 0, Skipped: 1.
The command "make test" exited with 0.

Now my problems are these:

• this file seems to be really long to me and too complicated given that I want to do something rather standard (I assume that automated browser tests are common these days). I would expect Travis CI to provide an easier way to perform automated browser tests. Here I have to download, install and start Apache, Selenium driver, Selenium server, use composer to get PHPUnit Selenium plugin etc...
• Selenium server doesn't seem to be running and I can't find out why.
• In the end, because PHPUnit ingeniously return 0 even when it couldn't even run the tests, Travis reports this test as successful. All those flags like --fail-on-risky or --stop-on-skipped still don't force PHPUnit to report a test failure which is in my opinion the only logical result as test clearly failed.

I know this is too broad and contains multiple questions. I am afraid that I took wrong direction somewhere and I am probably trying to do something simple in a complicated way.

Can somebody provide working example of .travis.yml file for automated browser tests? My application is in PHP, but I can write tests in Node.js, Python, Java or anything else as long as tests will really work and failure will be reported if anything goes wrong.

Answer 1

To Execute the above tests, Selenium server should be up and running on localhost with port 4444. Since the server is not up and running on the machine where execution happens try removing below mentioned lines and try.

$this->setHost('localhost');
$this->setPort(4444);