Separate Selenium tests from build process

Question

I have a question that might be very trivial for all of you, I just want to make sure I'm grasping the concept. I have done a lot of research and reading (SO post) and I'm still a bit confused, so I''m turning to you the experts!

In the articles below, the authors mention a common practice which is to separate the the selenium tests from the unit tests at build time. They say that the selenium tests should not be run on every build and they should be run on the CI server and not as part of the maven build, so they configure maven to skip the selenium tests and then they are ran in the integration phase.

I kind of get the reason why, but I'm not really that familiar with the build lifecycle and why you would have to set things up the way author.

Would somebody be kind enough to provide a simple explanation (or provide reading material) as to why you would want to do what is mentioned on those articles? I'm starting to work with selenium heavily and would like to understand these concepts better. Sorry again if this is trivial, and I'm not soliciting debate, arguments, or polling.

Thanks so much!

http://www.weblogism.com/item/334/integration-tests-with-cucumber-jvm-selenium-and-maven http://www.lagerweij.com/2011/08/24/setting-up-selenium-with-maven http://blog.tfnico.com/2007/09/continous-web-testing-with-selenium_16.html http://developershood.blogspot.com/2009/10/using-selenium-and-maven-for-junit.html

Answer 1

As I wrote one of the articles you refer to, I guess I should add to this.

There is another reason to separate out the selenium tests. Selenium tests are slow. A developer doesn't need a build that takes multiple minutes to run. He wants quick feedback from his unit tests. Selenium tests might be run at a lower frequency, say before a commit / push of the code to version control.

For that same reason, you will probably have different build jobs set-up on your build server, with non-UI tests running at every checking, and UI tests perhaps running after that, or perhaps running with a lower frequency (every hour? every night?) depending on how long they take to run, and what kind of infrastructure you can afford.

Answer 2

Reasons of separating and running only unit tests are:

1. Need for running unit test before selenium tests - to simplify reasoning of failures.
2. These kinds of tests require different settings in maven plugin configuration - parallel test running, ordering, system properties, JVM options, etc.
3. Selenium tests are end-to-end and requires lot of configuration and deployment work, resetting application state (clearing database, starting and stopping of app server) - these works cannot be done easy on all development and staging environments

Maven have set of phases for separating steps of integration tests, also there are profiles which allow to gather selenium test configuration in single place.

Here is example of our end-to-end tests with selenium (unpack and run scripts to setup DB from scratch, start app server, run tests, stop app server, verify and report test results):

<profile>
    <id>selenium</id>
    <dependencies>
        <dependency>
            <groupId>com.thenewmotion</groupId>
            <artifactId>msp-solveconnector</artifactId>
            <version>${project.version}</version>
            <classifier>sql-install</classifier>
            <type>zip</type>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-dependency-plugin</artifactId>
                    <executions>
                        <execution>
                            <id>unpack-dependencies</id>
                            <phase>package</phase>
                            <goals>
                                <goal>unpack-dependencies</goal>
                            </goals>
                            <configuration>
                                <includeGroupIds>com.thenewmotion</includeGroupIds>
                                <includeArtifactIds>msp-solveconnector</includeArtifactIds>
                                <includeClassifiers>sql-install</includeClassifiers>
                                <includeTypes>zip</includeTypes>
                                <includes>**/*.*</includes>
                                <outputDirectory>${project.build.directory}</outputDirectory>
                                <overWriteReleases>false</overWriteReleases>
                                <overWriteSnapshots>true</overWriteSnapshots>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>sql-maven-plugin</artifactId>
                    <dependencies>
                        <dependency>
                            <groupId>mysql</groupId>
                            <artifactId>mysql-connector-java</artifactId>
                            <version>5.1.16</version>
                        </dependency>
                    </dependencies>
                    <executions>
                        <execution>
                            <id>create-db</id>
                            <phase>pre-integration-test</phase>
                            <goals>
                                <goal>execute</goal>
                            </goals>
                            <configuration>
                                <driver>com.mysql.jdbc.Driver</driver>
                                <url>jdbc:mysql://localhost</url>
                                <username>${msp.db.user}</username>
                                <password>${msp.db.password}</password>
                                <orderFile>ascending</orderFile>
                                <fileset>
                                    <basedir>${project.build.directory}/db</basedir>
                                    <includes>
                                        <include>1_drop_database.sql</include>
                                        <include>2_create_database.sql</include>
                                        <include>3_create_tables.sql</include>
                                        <include>4_create_views.sql</include>
                                        <include>5_create_foreign_keys.sql</include>
                                        <include>6_data.sql</include>
                                    </includes>
                                </fileset>
                            </configuration>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.mortbay.jetty</groupId>
                    <artifactId>jetty-maven-plugin</artifactId>
                    <configuration>
                        <stopKey>foo</stopKey>
                        <stopPort>8088</stopPort>
                        <connectors>
                            <connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector">
                                <port>${msp.port}</port>
                                <maxIdleTime>60000</maxIdleTime>
                            </connector>
                        </connectors>
                    </configuration>
                    <executions>
                        <execution>
                            <id>start-jetty</id>
                            <phase>pre-integration-test</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <daemon>true</daemon>
                            </configuration>
                        </execution>
                        <execution>
                            <id>stop-jetty</id>
                            <phase>post-integration-test</phase>
                            <goals>
                                <goal>stop</goal>
                            </goals>
                        </execution>
                    </executions>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-failsafe-plugin</artifactId>
                    <version>2.12</version>
                    <configuration>
                        <systemPropertyVariables>
                            <msp.user>${msp.user}</msp.user>
                            <msp.password>${msp.password}</msp.password>
                            <msp.baseUrl>${msp.baseUrl}</msp.baseUrl>
                            <webdriver.type>${webdriver.type}</webdriver.type>
                            <webdriver.chrome.driver>${webdriver.chrome.driver}</webdriver.chrome.driver>
                        </systemPropertyVariables>
                        <includes>
                            <include>**/*FT.class</include>
                        </includes>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>sql-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</profile>