Accumulo:There are no tablet servers

Question

./bin/accumulo shell -u root
Password: ******

2015-02-14 15:18:28,503 [impl.ServerClient] WARN : There are no tablet servers: check that zookeeper and accumulo are running.

2015-02-14 13:58:52,878 [tserver.NativeMap] ERROR: Tried and failed to load native map library from /home/hduser/hadoop/lib/native::/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.lang.UnsatisfiedLinkError: no accumulo in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886)
at java.lang.Runtime.loadLibrary0(Runtime.java:849)
at java.lang.System.loadLibrary(System.java:1088)
at org.apache.accumulo.tserver.NativeMap.<clinit>(NativeMap.java:80)
at org.apache.accumulo.tserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:155)
at org.apache.accumulo.tserver.TabletServer.config(TabletServer.java:3560)
at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:3671)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)
2015-02-14 13:58:52,915 [tserver.TabletServer] ERROR: Uncaught exception in TabletServer.main, exiting
java.lang.IllegalArgumentException: Maximum tablet server map memory 83,886,080 and block cache sizes 28,311,552 is too large for this JVM configuration 48,693,248
at org.apache.accumulo.tserver.TabletServerResourceManager.<init>(TabletServerResourceManager.java:166)
at org.apache.accumulo.tserver.TabletServer.config(TabletServer.java:3560)
at org.apache.accumulo.tserver.TabletServer.main(TabletServer.java:3671)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.accumulo.start.Main$1.run(Main.java:141)
at java.lang.Thread.run(Thread.java:745)

The above error is shown in the tserver_localhost.log. can anyone help me with this issue. I have hadoop running on single-node mode, zookeeper running, and i followed the instructions in the Readme file of accumulo. I dont know how to start a tablet server.There was no explanation regarding this in the readme,could anyone help me with this.

Answer 1

I have found the solution to this. I have removed all the config files from the config folder in accumulo and used the bootstrap_config.sh file in bin folder,..which created the config files based on the input i have given and after that i initialized accumulo again and i was able to open the shell and the error was gone.

Thanks for the help.

Answer 2

This is the confluence of two problems.

First, your Accumulo can't find the native libraries it would use for off-heaping the in-memory-map for live edits. Knowing your version of Accumulo, how you deployed accumulo, and seeing your accumulo-env.sh would be needed to diagnose why it may have failed. (asking on the user mailing list would be best) Take a look at the README for your version under the Building section for "native map support".

For example, the passage for version 1.6.1 gives the following advice for building them yourself without a full source tree:

Alternatively, you can manually unpack the accumulo-native tarball in the $ACCUMULO_HOME/lib directory. Change to the accumulo-native directory in the current directory and issue make. Then, copy the resulting 'libaccumulo' library into the $ACCUMULO_HOME/lib/native/map.

$ mkdir -p $ACCUMULO_HOME/lib/native/map $ cp libaccumulo.* $ACCUMULO_HOME/lib/native/map

Normally, not having the native libraries available is a soft failure; Accumulo will happily issue a WARN and then rely on a pure-java implementation.

Your second problem is caused by incorrect memory configuration. Accumulo relies on a single configuration parameter to tune memory use for both the native in-memory-map and the java one. The memory for the native implementation is allocated outside of the JVM heap and can be substantial (in the 1-16GB range depending on target workload). When running with the Java implementation, that same configuration value takes away space carved from the max heap size.

Based on your log output, you have configured a total max heap for tabletservers of ~46MB. You have allocated 27MB of this for the block cache and 80MB for the in-memory-map. The error you see is because those two values would result in an OOM.

You can increase the total Java Heap in accumulo-env.sh:

# Probably looks like this
test -z "$ACCUMULO_TSERVER_OPTS" && export ACCUMULO_TSERVER_OPTS="${POLICY} -Xmx48m -Xms48m "
#                                 change this part to give it more memory --^^^^^^

And/or you can tune how much space should be used for the native maps, block cache, and index cache in accumulo-site.xml

  <!-- Amount of space to hold incoming random writes -->
  <property>
    <name>tserver.memory.maps.max</name>
    <value>80M</value>
  </property>

  <!-- Amount of space for holding blocks of data read out of HDFS -->
  <property>
    <name>tserver.cache.data.size</name>
    <value>7M</value>
  </property>

  <!-- Amount of space for holding indexes read out of HDFS -->
  <property>
    <name>tserver.cache.index.size</name>
    <value>20M</value>
  </property>

How you should balance these three will depend on how much memory you have and what your workload looks like. Keep in mind that more than just those two things need to go into your total Java heap (like atleast one copy of the current cell being written / read on each RPC).