CODEWITHSUNDEEP
hbase
Jack
Jack

Reputation: 5880

Why all my Hbase Regionservers are down

All of my HbaseRegion Servers were crashed(stopped), because of the following Exception:

java.net.SocketException: Too many open files
    at sun.nio.ch.Net.socket0(Native Method)
    at sun.nio.ch.Net.socket(Net.java:441)
    at sun.nio.ch.Net.socket(Net.java:434)
    at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:105)
    at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
    at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
    at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:62)
    at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1620)

But I did increase the OS user File Descriptor to 65535, and give Hbase the number is 55535, how come Hbase still crashed on this config?

I was trying to insert 20Million big rows to Hbase, each row may has 1MB-10MB, the first 10Million records ran well, but the second 10Million records caused Hbase crashed.

Upvotes: 0

Views: 190

Answers (1)

Michal
Michal

Reputation: 1321

Per HBase documentation HBase Configuration you should be able to check if the ulimit you believe to set was really set:

Also, a common mistake is that administrators will up the file descriptors for a particular user but for whatever reason, HBase will be running as some one else. HBase prints in its logs as the first line the ulimit its seeing. Ensure its correct.

Upvotes: 2

Related Questions