spring mvc + wildfly + postgresql performance tunning

Question

I have spent plenty of hours trying to figure out what is causing my application to work really slow, maybe someone will help me to check what is the problem.

versions:

spring mvc: 4.2.5.RELEASE

hibernate: 4.3.11.Final

spring-data-jpa: 1.8.2.RELEASE

wildfly: 10

PostgreSQL Server: 9.4

OS: Debian GNU/Linux 7.9 (wheezy)

Server configuration: VPS, 1 proc, 4GB RAM

About my app:

Simple web application with static JSP landing page, information page and login which redirect to private area. Public pages are static, first query to database is being done during login and then trying to get content for logged user.

What is the problem:

When I navigate through pages which are configured like this:

<http auto-config="true"> 
        <intercept-url pattern="/" access="permitAll"/>
        <intercept-url pattern="favicon.ico" access="permitAll"/>
        <intercept-url pattern="/login/**" access="permitAll"/>
        <intercept-url pattern="/info" access="permitAll"/>
...
</http>

there is no problem, everything is being loaded on he fly, there are no pauses. It works smooth. When I Post on /login (which redirect me to /main) and then when I navigate on each site configured like this:

<intercept-url pattern="/main/**" access="hasRole('ROLE_USER')"/>
<intercept-url pattern="/statistic/**" access="hasRole('ROLE_USER')"/>
<intercept-url pattern="/admin/**" access="hasRole('ROLE_ADMIN')" />

there are some extreme delays sometimes even up to 1 minute.

Hibernate configuration

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
        <property name="driverClassName" value="${jdbc.driverClassName}"/>
        <property name="url" value="${jdbc.url}"/>
        <property name="username" value="${jdbc.username}"/>
        <property name="password" value="${jdbc.password}"/>
    </bean>

    <!-- Configure the entity manager factory bean -->
    <bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
        <property name="dataSource" ref="dataSource"/>
        <property name="packagesToScan" value="pl.portal.model"/>
        <property name="jpaVendorAdapter">
            <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
        </property>
        <property name="jpaProperties">
            <props>
                <!-- <prop key="hibernate.dialect">pl.portal.sql.ProjectPsqlDialect</prop> -->
                <prop key="hibernate.show_sql">false</prop>
                <prop key="hibernate.enable_lazy_load_no_trans">true</prop>
                <prop key="hibernate.jdbc.use_streams_for_binary">false</prop>
                <prop key="hibernate.connection.useUnicode">true</prop>
                <prop key="hibernate.connection.characterEncoding">UTF-8</prop>
                <prop key="hibernate.connection.characterSet">UTF-8</prop>

            </props>
        </property>
    </bean>

Can anyone give me a hint how to determine what is causing that delays? Checking on webinspect in my browser -> /admin, size 7,89KB, size : 54secons (where each js or css not exced 100ms)

EDIT2: removed invalid fragment of thread dump, pasted full dump of thread which start after invoking action /admin on app. Thread took 61s

Thread dump

thread on visualvm console:

Answer 1

Server configuration: VPS, 1 proc, 4GB RAM

Do you have access to the server? Try run top command also and see CPU usage. You have only one proc, maybe something huge is running.

Typical way to start find performance problems:

Condition 1 You could connect with profiler Do it. jvisualvm, jconsole, NetBeans Profiler, perf, etc.

1. Connect to your process with profiler;
2. Run task, which you think is slow and wait;
3. Save profiler results to analyse;

You should be attentive to total CPU time.

Condition 2 You can not connect with profiler

Yes, here you can use jstack to get thread dumps. jps to get java process ID.

1. Prepare cmd/bash;
2. Run jps and find your process pid, remember;
3. Run task, which you think is slow;
4. Run jstack multiple times and save output (jstack pid > output.1, jstack pid > output.2, etc.);
5. Analyse all results and see, which methods are most frequent.

Don't make pre-optimizations, meausure performance before. :)

UPDATE for answer after new information.

In Thread dump you can see a lot of "sun.nio.ch.EPollArrayWrapper.epollWait", try to look some links:

• Netty
• Github 100 cpu load

Maybe it is your case. There was a bug with epoll. But something similar occurs in nowadays too.

Answer 2

In my experience you may have db lock problem. There is an sql locking your table for a while. Your code is waiting then the lock is released and your code continues. You may monitor db locks to find which sql locks your table while your code is hanging.

Answer 3

There can be many reasons for such delays, some of the most common ones are stated below.

1. Threads are waiting for a lock.
2. Database performance issues.
3. CPU intensive task.
4. Network delays

Network delays between the web server and client can be ruled out in your case as you mentioned that the server response time for static pages and page resources such as CSS is in milliseconds.

For the rest of the things, a remote management system like VisualVM is the best tool to find out what is causing delays and which process is taking a lot of CPU time. You can also go for commercial profilers like JProfiler, but VisualVm can also do the job perfectly.

To connect to ViusalVM you need to have access to your server and add JMX options in JAVA_OPTS or wherever your options go and restart the web application.

Threads waiting for lock - There may be a piece of code where have you have a synchronized method that locks on a static field. If that method lets assume takes about 5 seconds for processing a DB query. And there are 10 requests queued up. Then the last request will be processed in 50 seconds or more.

How to detect this bottleneck situation?

You can go through the code and find out all the places where there are acquiring locks and add some debug logs. But what if the problem is not within your code and probably arising from some third party library?

I once faced a situation where the Logger (log4j library) would block all threads(for around 10 seconds) when the log file was rolled over. There was no way that this situation could have been detected with code review.

Profilers to the rescue. VisualVM is probably the best profiler to detect delays caused by thread locking. Here is what you can do.

1. Monitor the thread.
2. When you see many threads in blocked state.
3. Take thread dump.
4. Analyze the thread dump to see which function is causing this issue and solve the problem accordingly.

CPU Intensive task - If a process is taking a lot of CPU time, it can also cause delays. Again you can use VisualVM's sampler and find out which thread is taking the most amount of CPU time.

Hope this will help find you the reason for the delay!

Answer 4

With respect to how to analyse the performance you can focus on several aspects:

1. DB
2. Network
3. Application layer
4. Memory

Monitoring the DB is plane simple use a RDBMS monitoring tool like Query Monitor for DB2 or something else depending on the vendor.

Analysing the BNetwork, there are many different Tools but what you need to do is to figure out what is Your latency and what is Your network speed.

I believe we can presume that in Your case the Network is OK or at least it is not very likly that there is problem.

The Third thing to analyse is Your Application layer, this is where you need a proper profiling tool. You can use visualVM which is free and comes With the jdk or alternativly you can use yourkit which is paid. But you can get a trial Version.

The memory you can also analyse With a profiling tool.

Once you perform CPU sampling you will be able to figure out which areas of Your Application are causing the delays. Yourkit is also capable of monitoring the executed SQLs from the Application layer side. This way you will get in one measurement the SQL speed + the latency to execute it + the time to transfer the resultset.

Common performance tunning opitions are:

Hibernate. Use batching for Collections loading, do not join relations instead favour FetchStrategy.SELECT.

Make sure you don't perform any cartesian Products and such.