Reputation: 395
One of our customers has a newish problem: the application grinds to halt. Thread dump shows that all threads are hanging on network IO in JDBC calls.
We/I have never seen these 'network IO' hangs. Typically a slow machine w/ DB problems has either a) one or two long-running queries or b) some type of lock/deadlock. In either of these cases the threads 'hang' on different methods. I have never seen all 30+ threads hanging on network IO.
Below I have included an excerpt from the thread dump. All HTTP threads are hanging on the same java.net.SocketInputStream.read
call.
I talked to their dba and sysadmin. According to them 'nothing has changed' in the environment recently which would cause this problem.
db environment
MSSQL 2005 64-bit Service Pack 2 Driver: sqljdbc.jar : 1.0 809 102
Note: they are running an older jdbc driver. AFAIK they tried upgrading from 1.0 to the 1.2 driver but had some other problem.
other environment issues
They're running both the app server and the db server in VMWare VM's. I don't know how this setup affects network performance.
Apparently this is the only application with this problem. I don't know anything else about their network architecture.
Questions * any insights on this problem? * if it is network, any next steps for analyzing?
Appendix A: Excerpt from Thread dump
All HTTP connections are hanging on the same method:
"TP-Processor31" daemon prio=5 tid=0x04085b78 nid=0x970 runnable [0x0764d000..0x0764fd6c] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.microsoft.sqlserver.jdbc.DBComms.receive(Unknown Source) at com.microsoft.sqlserver.jdbc.IOBuffer.sendCommand(Unknown Source) - locked (a com.microsoft.sqlserver.jdbc.DBComms) at com.microsoft.sqlserver.jdbc.SQLServerStatement.sendExecute(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteQuery(Unknown Source)
Upvotes: 5
Views: 3527
Reputation: 354
It is the changes to JSSE in version 1.6 u29. The change is to partially patch CVE-2011-3389 and CVE-2011-3560.
If are not deploying applets and using java web-start. You might be able to just use the 1.6 u27 jsse.jar. You're still going to have the vulnerability, but it will allow the sqljdbc.jar and sqljdbc4.jar to work.
The other options to migrate to Java 7, the sqljdbc4.jar does work in that environment.
The patch is not complete fix. So expect more changes in future patches.
Upvotes: 2
Reputation: 81
We've had similar issues, and traced them back to a buggy JDK update (1.6.29).
We downloaded 1.6.27 (http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u27-oth-JPR), re-set the JAVA_HOME environment and we were back on track.
Upvotes: 8