JSF Application Load Average escalates

Question

We developed a site using JSF 2 running on jBoss EAP 6.0, we are having problems in our production environment that basically after a period of time (with current levels of users about 24 hrs) the application makes the server load average go up to 100 and the server kind of stops responding (it actually doesn't stop responding it just takes too long to response so in practicall terms it's the same)

Now if we just stop and start jBoss again the application works fine (so it's not a problem of load but more something that happens over time, the more load the faster it happens)

In order to test this we created a simple application (just a war) that basically has a login page, and s very simple page that renders some text with a button and a simple popup that also just renders some text.

We are using jMeter (running in another machine) to stress this simple application hitting it with 100 concurrent threads (or as concurrent as jMeter allows anyway). The jBoss EAP running on a Mac with i5 and 8gb of ram (using 4gb for the server) lasts around 20 minutes before escalating load average to 50 and continus climbing, If instead of 100 I use 200 threads

The other thing I noticed is that If I start testing with a fresh jBoss instance it takes X amount of time for the load average to escalate, now, If I start a test, get the load average up to say 50, then stop that test and start a new test without restarting jBoss the Load Average spikes in seconds to a 50 again.

We can stress our services using the Restful services without problems (and no load average spikes) so we are guessing that it's not a general jBoss issue but more of a JSF issue.

I'm uploading some images of Visual VM as a reference (where you can see that GC is done periodically and looks fine and the curb of both CPU and load average escalation) and the contents of the pom.xml for library versions.

So my questions are basically has anyone encountered this problem? is this normal behaviour? can anyone think of a reason for this happening? (we have also created a ticket with RedHat support but we are hopping that someone in the community had already faced this issue or could help us figure out a way to find out wht's happening)

I can provide the code of the sample application and the jMeter test script if anyone is interested

TIA!

Load average with 100 threads:

CPU with 100 threads:

Memory with 100 threads:

Configuration:

Load Average starting test again without stopping jBoss:

Finally pom.xml dependencies:

 <properties>
        <project.version>1.0</project.version>
    <codi.version>1.0.5</codi.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.build.timestamp.format>yyyyMMdd'T'HHmmss</maven.build.timestamp.format>
        <jboss.home>${env.JBOSS_HOME}</jboss.home>
        <jboss.domain>default</jboss.domain>
        <arquillian.version>1.0.0.CR7</arquillian.version>
        <slf4j.version>1.6.1</slf4j.version>
        <weld.version>1.1.8.Final</weld.version>
        <seam.version>3.1.0.Final</seam.version>
        <rest.version>2.3.3.Final</rest.version>
        <version.arquillian_core>1.0.0.CR7</version.arquillian_core>
        <version.jbossas_7>7.1.0.Final</version.jbossas_7>
        <jboss-javaee6-spec.version>1.0.0.Final</jboss-javaee6-spec.version>
        <org.richfaces.bom.version>4.0.0.Final</org.richfaces.bom.version>
    </properties>

    <modelVersion>4.0.0</modelVersion>
    <build>
        <finalName>foo</finalName>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.1</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <dependencies>
        <!-- PRoject Dependencies -->
        <dependency>
            <groupId>ar.com.easytech</groupId>
            <artifactId>easyFaces-core</artifactId>
            <version>1.1</version>
        </dependency>

        <!-- Web Dependencies -->
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <scope>compile</scope>
            <version>1.6.2</version>
        </dependency>
        <!-- Seam -->
        <dependency>
            <groupId>org.jboss.seam.security</groupId>
            <artifactId>seam-security</artifactId>
            <scope>compile</scope>
            <version>${seam.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.myfaces.extensions.cdi.bundles</groupId>
            <artifactId>myfaces-extcdi-bundle-jsf20</artifactId>
            <version>${codi.version}</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-web-api</artifactId>
            <version>6.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>com.ocpsoft</groupId>
            <artifactId>prettyfaces-jsf2</artifactId>
            <scope>compile</scope>
            <version>3.3.2</version>
        </dependency>
        <!-- PDF -->
        <dependency>
            <groupId>org.apache.xmlgraphics</groupId>
            <artifactId>fop</artifactId>
            <version>1.0</version>
            <scope>compile</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.xmlgraphics</groupId>
            <artifactId>xmlgraphics-commons</artifactId>
            <version>1.4</version>
        </dependency>
        <dependency>
            <groupId>avalon-framework</groupId>
            <artifactId>avalon-framework</artifactId>
            <version>4.1.5</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons.io</groupId>
            <artifactId>commons.io</artifactId>
        <version>2.4</version>
    </dependency>
    <dependency>
        <groupId>batik</groupId>
        <artifactId>batik-bridge</artifactId>
        <version>1.6</version>
    </dependency>
    <dependency>
        <groupId>batik</groupId>
        <artifactId>batik-css</artifactId>
        <version>1.6</version>
    </dependency>
    <dependency>
        <groupId>batik</groupId>
        <artifactId>batik-util</artifactId>
        <version>1.6</version>
    </dependency>
    <dependency>
        <groupId>apache-xerces</groupId>
        <artifactId>xercesImpl</artifactId>
        <version>2.9.1</version>
    </dependency>
    <dependency>
        <groupId>com.sun.xml.ws</groupId>
        <artifactId>jaxws-rt</artifactId>
        <version>2.2.5</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

EDIT

As maple_shaft suggested I tested this in on glassfish getting an entirely different result, I configured a base glassfish server just changing Xmx Xms and NewSize and ran the test with 200 threads and the load average never spiked over 6 (actually it stayed around 5 in what i'm guessing is the confort zone for the load)

I'm adding the result images and will update RH support case accordingly

Load Average

CPU

Memory

Answer 1

Can you change the mojara implementation to myfaces, since there was some performance issue for applications that use a version of Mojarra up to 2.1.21. For more info read http://blog.oio.de/2013/05/06/jsf-performance-tuning/