CODEWITHSUNDEEP
javaperformanceoracle-databasetomcatjmeter
Spring
Spring

Reputation: 11865

Server response time get extremely slower during load test

I was analyzing my web apps performance using Jmeter(Oracle,Tomcat,Spring).

Number of threads :20
Ramp up: 2
Loop: 500

I only simulate a simple login to the website with Jmeter.

When I start in these settings above. First 3 minutes it starts very fast(av response time 500ms) but it starts to slow down quickly and by min 15 the response time goes down to 5 seconds. And it takes minutes to complete a request.

I am not sure if this is normal or if the problem is Tomcat or actually my c3po pool settings are not right.

This is my settings for c3po:

    <property name="testConnectionOnCheckout" value="false" />  <!-- do not test connection isValid on checkout as it has bad performance result -->
    <property name="testConnectionOnCheckin" value="true" />    <!-- test connection isValid on checkin -->
    <property name="idleConnectionTestPeriod" value="60" />     <!-- test every 60 seconds -->

    <property name="checkoutTimeout" value="5000" />            <!-- The number of milliseconds a client calling getConnection() will wait for a Connection to be checked-in or acquired when the pool is exhausted. -->
    <property name="acquireRetryAttempts" value="30" />         <!-- Defines how many times c3p0 will try to acquire a new Connection from the database before giving up. -->
    <property name="acquireRetryDelay" value="1000" />          <!-- Milliseconds, time c3p0 will wait between acquire attempts. -->
    <property name="breakAfterAcquireFailure" value="true" />   <!-- If true, a pooled DataSource will declare itself broken and be permanently closed if a Connection cannot be obtained from the database after making acquireRetryAttempts to acquire one. -->

    <property name="maxIdleTime" value="180" />                 <!-- 3 minutes, then die -->
    <property name="maxConnectionAge" value="300" />            <!-- 5 minutes, then die -->
    <property name="minPoolSize" value="1" />
    <property name="initialPoolSize" value="2" />
    <property name="maxPoolSize" value="25" />
    <property name="acquireIncrement" value="1" />              <!-- ramp up the pool with 1 connection if all others are in use -->

    <property name="unreturnedConnectionTimeout" value="360" /> <!-- in seconds (6 minutes), actually this should be necessary if connections are not closed properly and thus are leaked -->
    <property name="numHelperThreads" value="10" />
    <property name="maxStatementsPerConnection" value="5" />

I also installed VisualVm to monitor Tomcat but not sure how to analyze it, Should I check Memory pools or Heap or Perm gen..?

Any ideas why it might be getting slow or how can I check it ?

Ps Old gen Memory Pool size graph peaks when response time starts to get slower might be any relation? My Cm connection settinsgd are 

 -Xmx1024m -XX:MaxPermSize=512M

Upvotes: 0

Views: 2335

Answers (2)

Quinnlv
Quinnlv

Reputation: 302

You may be using up your pool of DB connections since it's only set to 25 max. Once that max is hit, your server is storing all of those connection requests for several minutes before they release them back to JMeter (while JMeter never stops making requests)

You can try increasing the connection pool size, or lower the time before they are released back to the client.

You should also be investigating why it's taking so long to get a response for a login request.

Edit: Adding Example below

I don't think the issue here is inside Tomcat (I'm assuming Tomcat is just handling the request and acting as the broker between the client and a database), it may be in how your testing itself.

If JMeter is making the same login request repeatedly it can potentially backup a database because it may lock that record while the login is processing. You should not be using the same user when load testing because of this possibility (it isn't a realistic situation anyways). A better test is to create many users, and cycle through them during the load test.

What it sounds like is your creating many threads to perform a login, it's causing queuing on the database, which is causing connection queuing on tomcat, which makes the problem worse over time.

Upvotes: 2

Michael-O
Michael-O

Reputation: 18415

I'd follow @Quinnlv's statement. Implement a mock login and see how this performs. If that scales, check the pool.

Upvotes: 0

Related Questions