CODEWITHSUNDEEP
javamysqlglassfishconnectionpool
mukund
mukund

Reputation: 2980

what is java.io.EOFException, Message: Can not read response from server. Expected to read 4 bytes, read 0 bytes

This question has been asked a couple of times in SO and many times in other sites. But I didn't get any satisfiable answer.

My problem:
I have a java web application which uses simple JDBC to connect to mysql database through Glassfish application server.

I have used connection pooling in glassfish server with the following configurations:
Initial Pool Size: 25
Maximum Pool Size: 100
Pool Resize Quantity: 2
Idle Timeout: 300 seconds
Max Wait Time: 60,000 milliseconds

The application has been deployed for last 3 months and it was running flawlessly too.
But from last 2 days the following error is coming at the time of login.

Partial StackTrace 

com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: No operations allowed after connection closed.Connection was implicitly closed due to underlying exception/error:  

** BEGIN NESTED EXCEPTION **  

com.mysql.jdbc.CommunicationsException  
MESSAGE: Communications link failure due to underlying exception:  

** BEGIN NESTED EXCEPTION **  

java.io.EOFException  
MESSAGE: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.  

STACKTRACE:  

java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.  
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1997)  
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2411)  
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2916)  
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1631)  
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1723)  
at com.mysql.jdbc.Connection.execSQL(Connection.java:3256)  
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1313)  
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1448)  
............  
............  
my application traces....

What caused this error suddenly ? I have lost a lot of time for this.

EDIT : The problem even persists after restarting the server. As per DBA two of the important mysql server configurations are:
wait_timeout : 1800 seconds
connect_timeout : 10 seconds
NOTE : Other applications deployed in the same server connecting to the same database and using different pools are running smoothly.

EDIT-2 : After reading a lot of things and expecting some positive outcome I made these changes to my connection pool.

Max Wait Time : 0 (previously it was 60 seconds)
Connection Validation : Required
Validation Method : table
Table Name : Demo
Validate Atmost Once : 40 seconds
Creation Retry Attempts : 1
Retry Intervals : 5 seconds
Max Connection Usage : 5

And this worked as the application is running for 3 days consistently. But I got a very strange and interesting result of out this. While monitoring the connection pool, I found these figures:

NumConnAcquired : 44919 Count
NumConnReleased : 44919 Count
NumConnCreated : 9748 Count
NumConnDestroyed : 9793 Count
NumConnFailedValidation : 70 Count
NumConnFree : 161 Count
NumConnUsed : -136 Count

How can the NumConnFree become 161 as I have Maximum Pool Size = 100 ?
How can the NumConnUsed become -136, a negative number ?
How can the NumConnDestroyed > NumConnCreated ?

Upvotes: 55

Views: 109051

Answers (12)

Nicholas
Nicholas

Reputation: 1

I found a possible solution. This is related to some sort of table corruption which can be fixed as follows:

SELECT CONCAT('REPAIR TABLE ', table_name, ';') 
FROM information_schema.tables 
WHERE table_schema = 'database_name';

This will generate bunch of sql statements like : REPAIRE TABLE abc; REPAIR TABLE xyz;

run these statements and the error will go away

Upvotes: 0

electrobabe
electrobabe

Reputation: 1647

In our case, we got the error "Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedl lost" (in IntelliJ) because of a Docker error.

We fixed it with simply running:

docker-compose --profile microservices --compatibility -f docker-compose.services.yml up

Upvotes: 0

Christopher Schultz
Christopher Schultz

Reputation: 20882

The connection has failed, possibly due to a firewall idle-timeout, etc. If you don't have your JDBC driver configured to reconnect on failure, then this error will not go away unless you open a new connection.

If you are using a database connection pool (you are using one, right?), then you probably want to enable it's connection-checking features like issuing a query to check to see if the connection is working before handing it back to the application. In Apache commons-dbcp, this is called the validationQuery and is often set to something simple like SELECT 1.

Since you are using MySQL, you ought to use a Connector/J-specific "ping" query that is lighter-weight than actually issuing a true SQL query and set your validation query to /* ping */ SELECT 1 (the ping part needs to be exact).

Upvotes: 15

Hero
Hero

Reputation: 1

it may be the server connection is full try to reduce the server connection or modify the max connection param

Upvotes: 0

myuce
myuce

Reputation: 1391

In my case, it was the java mysql driver/connector version. I.e. (Even if the server was version 8) somehow when I used mysql driver/connector version 5.1.44, then it started working.

Upvotes: 1

zhuojl
zhuojl

Reputation: 11

i meet the same problem, it caused by lock for too much time.

we have a long transcation A, if execute it, the other trascation will be locked util transcation A finished, so the others always be killed by our mysql tool(pt-kill)

Upvotes: 0

Thomas
Thomas

Reputation: 71

Got this error in PHPStorm while trying to connect and/or get some data from mysql server. Checkbox "single connection mode" fixed things for me. I'll just leave it here. Perhaps this will help some martyr in the future.

https://www.jetbrains.com/help/phpstorm/configuring-database-connections.html

Upvotes: 2

macieks
macieks

Reputation: 51

I had this problem but I it wasn't possible for me to make changes in MySQL database configuration. Therefore I ensured that in my sql connector class the connection is always closed before it's initiated again. Something like:

public static Connection getConnection() {

    if (DatabaseConnnector.conn == null) {
        initConn();
    } else {
        try {
            DatabaseConnnector.conn.close();          
        } catch (SQLException e) {
            e.printStackTrace();
        }
          initConn();
    }
    return DatabaseConnnector.conn;
}

And this solved the problem.

Upvotes: 1

Stephen C
Stephen C

Reputation: 719661

It most likely means that the database has restarted or the network connection to the database has been broken (e.g. a NAT connection has timed out) ... and your webapp is trying to use a stale database connection.

If the problem persists after restarting the web container, it could be something more serious.

You asked the following:

How can the NumConnFree become 161 as I have Maximum Pool Size = 100 ?
How can the NumConnUsed become -136, a negative number ?
How can the NumConnDestroyed > NumConnCreated ?

On the face of it, these don't make sense. However, they could simply be the result of some usage counters being updated in a non-thread-safe way. This is not necessarily related to your original problem.

Upvotes: 9

Mani
Mani

Reputation: 21

It might be Firewall related problem.

Upvotes: 2

gerrytan
gerrytan

Reputation: 41143

While I don't have definitive solutions, it seems something is interfering communication between the app server and db. Following are few things you can try to isolate the problems:

  • Try to determine wether this is mysql problem or java code problem. Try connect to mysql using the command line tool from the same host as the app server and issue similar SQL to perform login. Test using a simple java code that does a select, deploy it to the same infrastructure, see what happen etc. Also check the mysql server log, see if you can find anything useful

  • There are two way an idle connection get closed: by the connection pool code that runs inside app server, or by mysql itself. Make sure you check the config on both sides

  • Check if any network infrastructure config has changed recently. Was there any new firewall rule in place interfering with app server <--> mysql connectivity? Was there any settings that prohibits open TCP connection idling longer than X?

  • Try a different connection pooling library just to eliminate the possibility it's the connection pooling

Good luck

Upvotes: 8

Puneet
Puneet

Reputation: 611

This is EndOfFileException in java, happens when your cursor start point is on the end point or when database connection closed because of some unexpected exception.

Upvotes: 9

Related Questions