Loading data to Hadoop 2.2 with Spring-XD

Question

I'm trying to create stream of data from multiple sources and load it into hdfs.

For proof of concept i'm using single node hadoop instalation and Spring-XD on a same machine. Hadoop was deployed with Ambari and reports to be functional and healthy (HDP 2.0). Spring-XD is installed in /var/spring-xd with environment variable $XD_HOME set to /var/spring-xd/xd; both server and client of Spring run with --hadoopDistro hdp20 flag. However i can only get those messages in log file so far:

18:00:06,346  INFO DeploymentsPathChildrenCache-0 Configuration.deprecation:840 - fs.default.name is deprecated. Instead, use fs.defaultFS
18:00:06,692  WARN DeploymentsPathChildrenCache-0 util.NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.ClasspathLoggingApplicationListener:55 - Application failed to start with classpath: [file:/var/spring-xd/xd/modules/processor/scripts/, file:/var/spring-xd/xd/config/, file:/var/spring-xd/xd/, file:/var/spring-xd/xd/lib/activation-1.1.jar, file:/var/spring-xd/xd/lib/amqp-client-3.2.4.jar, file:/var/spring-xd/xd/lib/aopalliance-1.0.jar, file:/var/spring-xd/xd/lib/args4j-2.0.16.jar, file:/var/spring-xd/xd/lib/asm-3.1.jar, file:/var/spring-xd/xd/lib/aspectjrt-1.6.6.jar, file:/var/spring-xd/xd/lib/aspectjweaver-1.6.6.jar, file:/var/spring-xd/xd/lib/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/cglib-2.2.1-v20090111.jar, file:/var/spring-xd/xd/lib/cloudfoundry-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/com.ibm.jbatch-tck-spi-1.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-1.7.0.jar, file:/var/spring-xd/xd/lib/commons-beanutils-core-1.8.0.jar, file:/var/spring-xd/xd/lib/commons-cli-1.2.jar, file:/var/spring-xd/xd/lib/commons-codec-1.7.jar, file:/var/spring-xd/xd/lib/commons-collections-3.2.1.jar, file:/var/spring-xd/xd/lib/commons-compress-1.4.1.jar, file:/var/spring-xd/xd/lib/commons-configuration-1.6.jar, file:/var/spring-xd/xd/lib/commons-daemon-1.0.13.jar, file:/var/spring-xd/xd/lib/commons-dbcp-1.2.2.jar, file:/var/spring-xd/xd/lib/commons-digester-1.8.jar, file:/var/spring-xd/xd/lib/commons-el-1.0.jar, file:/var/spring-xd/xd/lib/commons-fileupload-1.2.1.jar, file:/var/spring-xd/xd/lib/commons-httpclient-3.1.jar, file:/var/spring-xd/xd/lib/commons-io-2.4.jar, file:/var/spring-xd/xd/lib/commons-jexl-2.1.1.jar, file:/var/spring-xd/xd/lib/commons-lang-2.6.jar, file:/var/spring-xd/xd/lib/commons-math-2.1.jar, file:/var/spring-xd/xd/lib/commons-net-3.3.jar, file:/var/spring-xd/xd/lib/commons-pool-1.3.jar, file:/var/spring-xd/xd/lib/core-0.9.2.jar, file:/var/spring-xd/xd/lib/curator-client-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-framework-2.4.1.jar, file:/var/spring-xd/xd/lib/curator-recipes-2.4.1.jar, file:/var/spring-xd/xd/lib/ehcache-core-2.3.0.jar, file:/var/spring-xd/xd/lib/freemarker-2.3.15.jar, file:/var/spring-xd/xd/lib/groovy-all-2.2.2.jar, file:/var/spring-xd/xd/lib/guava-14.0.1.jar, file:/var/spring-xd/xd/lib/guice-3.0.jar, file:/var/spring-xd/xd/lib/guice-servlet-3.0.jar, file:/var/spring-xd/xd/lib/hibernate-validator-4.3.1.Final.jar, file:/var/spring-xd/xd/lib/hsqldb-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-annotations-2.3.0.jar, file:/var/spring-xd/xd/lib/jackson-core-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-core-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/jackson-databind-2.3.2.jar, file:/var/spring-xd/xd/lib/jackson-mapper-asl-1.9.13.jar, file:/var/spring-xd/xd/lib/javax.batch-api-1.0.jar, file:/var/spring-xd/xd/lib/javax.inject-1.jar, file:/var/spring-xd/xd/lib/javax.mail-1.5.0.jar, file:/var/spring-xd/xd/lib/jboss-logging-3.1.0.CR2.jar, file:/var/spring-xd/xd/lib/jcl-over-slf4j-1.7.6.jar, file:/var/spring-xd/xd/lib/jersey-guice-1.9.jar, file:/var/spring-xd/xd/lib/jersey-test-framework-grizzly2-1.9.jar, file:/var/spring-xd/xd/lib/jettison-1.2.jar, file:/var/spring-xd/xd/lib/joda-time-2.1.jar, file:/var/spring-xd/xd/lib/jolokia-core-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jmx-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-jvm-1.1.5.jar, file:/var/spring-xd/xd/lib/jolokia-spring-1.1.5.jar, file:/var/spring-xd/xd/lib/json-path-0.8.1.jar, file:/var/spring-xd/xd/lib/json-simple-1.1.jar, file:/var/spring-xd/xd/lib/json-smart-1.1.1.jar, file:/var/spring-xd/xd/lib/kite-data-core-0.12.1.jar, file:/var/spring-xd/xd/lib/kryo-2.22.jar, file:/var/spring-xd/xd/lib/lettuce-2.3.3.jar, file:/var/spring-xd/xd/lib/log4j-1.2.17.jar, file:/var/spring-xd/xd/lib/mysql-connector-java-5.1.23.jar, file:/var/spring-xd/xd/lib/netty-3.7.0.Final.jar, file:/var/spring-xd/xd/lib/objenesis-1.3.jar, file:/var/spring-xd/xd/lib/opencsv-2.3.jar, file:/var/spring-xd/xd/lib/paranamer-2.3.jar, file:/var/spring-xd/xd/lib/parquet-avro-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-column-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-common-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-encoding-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-format-1.0.0.jar, file:/var/spring-xd/xd/lib/parquet-generator-1.2.5.jar, file:/var/spring-xd/xd/lib/parquet-hadoop-1.2.5.jar, file:/var/spring-xd/xd/lib/postgresql-9.2-1002-jdbc4.jar, file:/var/spring-xd/xd/lib/slf4j-api-1.7.6.jar, file:/var/spring-xd/xd/lib/slf4j-log4j12-1.7.6.jar, file:/var/spring-xd/xd/lib/snakeyaml-1.12.jar, file:/var/spring-xd/xd/lib/snappy-java-1.0.5.jar, file:/var/spring-xd/xd/lib/spring-amqp-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-aop-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-manager-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-admin-resources-1.3.0.M1.jar, file:/var/spring-xd/xd/lib/spring-batch-core-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-infrastructure-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-batch-integration-3.0.0.M3.jar, file:/var/spring-xd/xd/lib/spring-beans-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-actuator-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-boot-autoconfigure-1.0.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-context-support-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-core-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-commons-1.6.2.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-data-redis-1.2.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-expression-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-hateoas-0.9.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-integration-amqp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-core-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-event-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-file-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-ftp-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-groovy-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-http-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-jmx-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-redis-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-integration-scripting-4.0.0.M4.jar, file:/var/spring-xd/xd/lib/spring-jdbc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-messaging-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-plugin-core-0.8.0.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-rabbit-1.3.1.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-retry-1.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-service-connector-0.9.2.jar, file:/var/spring-xd/xd/lib/spring-tx-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-web-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-webmvc-4.0.3.RELEASE.jar, file:/var/spring-xd/xd/lib/spring-xd-analytics-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-batch-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-dirt-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-hadoop-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-module-spi-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-rest-domain-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/spring-xd-tuple-1.0.0.M6.jar, file:/var/spring-xd/xd/lib/tomcat-el-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-core-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-embed-logging-juli-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-jdbc-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-jsp-api-7.0.35.jar, file:/var/spring-xd/xd/lib/tomcat-juli-7.0.42.jar, file:/var/spring-xd/xd/lib/tomcat-servlet-api-7.0.35.jar, file:/var/spring-xd/xd/lib/uuid-3.2.jar, file:/var/spring-xd/xd/lib/validation-api-1.0.0.GA.jar, file:/var/spring-xd/xd/lib/xmlenc-0.52.jar, file:/var/spring-xd/xd/lib/xmlpull-1.1.3.1.jar, file:/var/spring-xd/xd/lib/xpp3_min-1.1.4c.jar, file:/var/spring-xd/xd/lib/xstream-1.4.4.jar, file:/var/spring-xd/xd/lib/xz-1.0.jar, file:/var/spring-xd/xd/lib/zookeeper-3.4.6.jar, file:/var/spring-xd/xd/lib/hdp20/avro-1.7.5.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-annotations-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-auth-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-distcp-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-hdfs-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-core-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-jobclient-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-mapreduce-client-shuffle-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-streaming-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-api-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-client-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-common-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/hadoop-yarn-server-nodemanager-2.2.0.jar, file:/var/spring-xd/xd/lib/hdp20/jetty-util-6.1.26.jar, file:/var/spring-xd/xd/lib/hdp20/protobuf-java-2.5.0.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-batch-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-core-2.0.0.RC2-hdp20.jar, file:/var/spring-xd/xd/lib/hdp20/spring-data-hadoop-store-2.0.0.RC2-hdp20.jar]
18:00:07,944  INFO DeploymentsPathChildrenCache-0 logging.AutoConfigurationReportLoggingInitializer:101 - 

Error starting ApplicationContext. To display the auto-configuration report enabled debug logging (start with --debug)


18:00:07,946 ERROR DeploymentsPathChildrenCache-0 cache.PathChildrenCache:550 - 
java.lang.RuntimeException: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storeWriter': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: failed to initialize; nested exception is java.net.ConnectException: Call From CentOS/10.0.0.4 to localhost:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.springframework.xd.dirt.server.ContainerRegistrar.deployStreamModule(ContainerRegistrar.java:448)
    at org.springframework.xd.dirt.server.ContainerRegistrar.onChildAdded(ContainerRegistrar.java:347)

I'm still able to successfuly execute such commands as

hadoop fs ls /

from xd-shell and get results succesfuly.

I'm aware of Spring-XD API, but would like to stay within XD-Shell for now. Any help would be greatly appreciated.

Answer 1

1. Test with Spring XD Single Node - it's a little easier and no zookeeper
2. Make sure you have permissions setup in HDFS for access, I've seen that a problem before.
3. If Hadoop is running in a virtual machine on your desktop and spring xd is on your main machine, sometimes there's IP/DNS issues that will block it.

Answer 2

check firewall rules and run some standard hadoop commands.

XD is running Zookeeper which could be blocked

Answer 3

It's not able to connect to localhost:8020, i.e. the name node service. Most likely your hadoop installation is not started correctly.