CODEWITHSUNDEEP
hadooposx-maverickshortonworks-data-platformspring-xd
Luc
Luc

Reputation: 223

Spring XD on Hortonworks Sandbox on OSX

I am trying to store Spring XD streams to the Hortonworks sandbox version 2.0 using xd-singlenode and xd-shell. No xd directory is created and no stream is stored in Hortonworks hadoop hdfs.

Environment: Apple OSX 10.9.3, Hortonworks Sandbox running in Oracle Virtualbox (Red Hat 64bit), using bridge mode networking. I assigned in my WiFi router a fixed IP address (192.168.178.30) to the Virtualbox MAC address. When I browse with OSX Safari to 192.168.178.30:8000 I can use the Hortonworks menu's such as File Browser, Pig, Beeswax (Hive), etc. A "check for misconfiguration" in the Hortonworks menu results into:

Configuration files located in /etc/hue/conf.empty 
All OK. Configuration check passed.

I have used Homebrew to install Spring XD. I changed in OSX the /usr/local/Cellar/springxd/1.0.0.M6/libexec/xd/config/servers.yml file to include

# Hadoop properties
spring:
  hadoop:
   fsUri: hdfs://192.168.178.30:8020

and

#Zookeeper properties
# client connect string: host1:port1,host2:port2,...,hostN:portN
zk:
  client:
     connect: 192.168.178.30:2181

Within Virtualbox I changed the file /etc/hadoop/conf.empty/hadoop-env.sh to include:

export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= -Djava.security.krb5.kdc="

export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf=/dev/null"

I start Spring XD with OSX with the following commands:

./xd-singlenode --hadoopDistro hadoop22

and in a second OSX terminal: 

./xd-shell --hadoopDistro hadoop22

In xd-shell I enter:

 hadoop config fs --namenode hdfs://192.168.178.30:8020

A "hadoop fs ls /" command in the xd-shell results into:

Hadoop configuration changed, re-initializing shell...
2014-06-24 00:55:56.632 java[7804:5d03] Unable to load realm info from SCDynamicStore
00:55:56,672  WARN Spring Shell util.NativeCodeLoader:62 - Unable to load native-hadoop     library for your platform... using builtin-java classes where applicable
Found 6 items
drwxrwxrwt   - yarn   hadoop          0 2013-10-21 00:19 /app-logs
drwxr-xr-x   - hdfs   hdfs            0 2013-10-21 00:08 /apps
drwxr-xr-x   - mapred hdfs            0 2013-10-21 00:10 /mapred
drwxr-xr-x   - hdfs   hdfs            0 2013-10-21 00:10 /mr-history
drwxrwxrwx   - hdfs   hdfs            0 2013-10-28 16:34 /tmp
drwxr-xr-x   - hdfs   hdfs            0 2013-10-28 16:34 /user

When I create a Spring XD stream with the command

stream create --name twvoetbal --definition "twittersearch --consumerKey='<mykey>' --consumerSecret='<mysecret>' --query='voetbal' | file" --deploy

then in OSX a /tmp/xd/output/twvoetbal.out file is created. Spring XD seams to work including my Twitter developer secret keys.

When I create a Spring XD stream with the command

stream create --name twvoetbal --definition "twittersearch --consumerKey='<mykey>' --consumerSecret='<mysecret>' --query='voetbal' | hdfs" --deploy

then no xd directory and no file(s) are created in hadoop hdfs.

Questions:

  1. How do I solve the "Unable to load realm info from SCDynamicStore" error in xd-shell?
  2. How do I solve the "WARN Spring Shell util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable" error in xd-shell?
  3. What else could I have done wrong?

Upvotes: 1

Views: 582

Answers (1)

Thomas Risberg
Thomas Risberg

Reputation: 976

The WARN messages are just noise - I always just ignore them. Did you create the xd directory on the sandbox? You need to give the user running the xd-singlenode the rights to create the needed directories.

You can ssh to the sandbox as root (password is hadoop) and run the following:

sudo -u hdfs hdfs dfs -mkdir /xd
sudo -u hdfs hdfs dfs -chmod 777 /xd

We have a brief writeup for using Hadoop VMs with XD: https://github.com/spring-projects/spring-xd/wiki/Using-Hadoop-VMs-with-Spring-XD#hortonworks-sandbox

Upvotes: 2

Related Questions