Hadoop standalone - hdfs commands are slow

Question

I'm doing development/research in an Ubuntu 14.04 VM with Hadoop 2.6.2 and I'm getting constantly held back because any commands I issue to hdfs always take about 15 seconds to run. I've tried digging around, but I am unable to locate the source of the problem or even if this is expected behavior.

I followed the directions on Apache's website and successfully got it up and running just fine in /opt/hadoop-2.6.2/

The following is a simple test command that I'm using to evaluate if I have solved the problem.

/opt/hadoop-2.6.2/bin/hdfs dfs -ls /

I have inspected the logs and found no errors or strange warnings. A recommendation that I found online was to set the logger to output the console.

HADOOP_ROOT_LOGGER=DEBUG,console /opt/hadoop-2.6.2/bin/hdfs dfs -ls /

Doing this yields something of interest. You can watch it hang between the following.

16/01/15 11:59:02 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
16/01/15 11:59:17 DEBUG util.KerberosName: Kerberos krb5 configuration not found, setting default realm to empty

Thoughts: When I first saw this I assumed that it was hanging on authentication, but not only do I not have Kerberos installed, the default configuration for core-site.xml shows the authentication mode set to "simple". This makes wonder why it would be looking for anything Kerberos related to begin with. I attempted to specifically disable it in the xml and the lag/slowness didn't go away. I kinda get the feeling that the delay is because its timing out waiting for something. Does anyone else have any ideas?

Answer 1

I just went ahead and install Kerberos anyways just to see if it would work. Large delays have disappeared now that /etc/krb5.conf is present. I wonder if I could have just created the file with nothing in it. Hrmmm...

sudo apt-get install krb5-kdc krb5-admin-server