CODEWITHSUNDEEP
apache-storm
Edward Khoh
Edward Khoh

Reputation: 11

Storm-kafka Hortonworks Tutorials for real time data streaming

Welcome any idea after read the problem statement.

Background: Publish message using Apache Kafka: Kafka broker is running. Kafka producers are the application that create the messages and publish them to the Kafka broker for further consumption. Therefore, in order for the Kafka consumer to consume data, Kafka topic need to create before Kafka producer and consumer starting publish message and consume message

Kafka tested successful as Kafka consumer able to consume data from Kafka topic and display result.

Before startup Storm topology, stop the Kafka consumer so that Storm Spout able to working on source of data streams from kafka topics.

Real time processing of the data using Apache Storm: With Storm topology created, Storm Spout working on the source of data streams, which mean Spout will read data from kafka topics. At another end, Spout passes streams of data to Storm Bolt, which processes and create the data into HDFS (file format) and HBase (db format) for storage purpose.

  1. Zookeeper znode missing the last child znode. From log file, 2015-05-20 04:22:43 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/truckevent/partitions

Zookeeper is the coordination service for distribution application. From the zookeeper client, we always can see the /brokers/topics/truckevent, but the last znode always missing when running storm. I managed to solve this issue once if we create the znode manually. However, same method is no longer work for subsequent testing.

  1. Storm (TrucjHBaseBolt is the java class) failed to access connection to HBase tables. From log file, 2015-05-20 04:22:51 c.h.t.t.TruckHBaseBolt [ERROR] Error retrievinging connection and access to HBase Tables

I had manually create the Hbase table as for data format at HBase. However, retrieving the connection to HBase still failed.

  1. Storm (HdfsBolt java class) reported the permission denied when storm user write the data into hdfs. From log file, 2015-05-20 04:22:43 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: Error preparing HdfsBolt: Permission denied: user=storm, access=WRITE, inode="/":hdfs:hdfs:drwxr-xr-x

Any one can help on this?

Upvotes: 1

Views: 532

Answers (1)

Pranjal Sahu
Pranjal Sahu

Reputation: 1469

Suggestion for problem 1:

Stop storm topology. Delete the znodes related to topics manually in the zookeeper where storm is running and restart storm topology. This will create new znodes.

Suggestion for problem 2:

First check using java code if you are able to connect to Hbase. Then test that same logic in Storm topology.

Answer for problem 3:

As per your logs user=storm but the directory in which you are writing is owned by hdfs. So change the user permission of that directory and make storm as the user using chown command.

Upvotes: 1

Related Questions