CODEWITHSUNDEEP
filesocketscassandra
chnet
chnet

Reputation: 2033

cassandra too many open files

I used cassandra 0.6.5 on two-node (A and B) cluster.
Hector is used in Client side.

One node A always has too many open files exception after running some time.
I run netstat on the node.
It shows a lot of CLOSE_WAIT tcp connections.

It is the culprit of the exception.
However, what causes so many CLOSE_WAIT connections,
Is it Client side Hector problem?
Why the other node B does not have this problem?

Upvotes: 1

Views: 7870

Answers (1)

psanford
psanford

Reputation: 5670

Instead of using netstat, try lsof -n | grep java. How many file descriptors are listed there (you can get a count with lsof -n | grep java | wc -l)?

The datastax docs suggest you might be hitting a default file descriptor limit of 1024. You can change that via ulimit or in /etc/security/limits.conf. Datastax suggests the following changes:

echo "* soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "* hard nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root soft nofile 32768" | sudo tee -a /etc/security/limits.conf
echo "root hard nofile 32768" | sudo tee -a /etc/security/limits.conf

The debian package sets the following values:

# Provided by the cassandra package
cassandra  -  memlock  unlimited
cassandra  -  nofile   100000

I would also strongly recommend that you upgrade to a more recent version of Cassandra.

Upvotes: 5

Related Questions