CODEWITHSUNDEEP
hadooptwitter
Pooja Karnik
Pooja Karnik

Reputation: 25

Unable to view the log files in localhost browsing HDFS

I have successfully started the flume-agent but can't view the log files in HDFS.

The path I have set in twitter.conf is: 

TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets/

Please help me to get rid of this error and view the data in my HDFS.

Upvotes: 1

Views: 150

Answers (1)

Ramesh Maharjan
Ramesh Maharjan

Reputation: 41977

If you have set your hadoop home in .bashrc as 

export HADOOP_HOME=<Path to your hadoop home>

Then you don't need localhost:9000 in the following

TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets/

So the correct line should be 

TwitterAgent.sinks.HDFS.hdfs.path = hdfs:///user/flume/tweets/

It should work given that your twitter.conf is as below

    # Naming the components on the current agent. 
    TwitterAgent.sources = Twitter 
    TwitterAgent.channels = MemChannel 
    TwitterAgent.sinks = HDFS

    # Describing/Configuring the source 
    TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
    TwitterAgent.sources.Twitter.consumerKey = Your OAuth consumer key
    TwitterAgent.sources.Twitter.consumerSecret = Your OAuth consumer secret 
    TwitterAgent.sources.Twitter.accessToken = Your OAuth consumer key access token 
    TwitterAgent.sources.Twitter.accessTokenSecret = Your OAuth consumer key access token secret 
    TwitterAgent.sources.Twitter.keywords = tutorials point,java, bigdata, mapreduce, mahout, hbase, nosql

    # Describing/Configuring the sink 

    TwitterAgent.sinks.HDFS.type = hdfs 
    TwitterAgent.sinks.HDFS.hdfs.path = hdfs:///user/flume/tweets/
    TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
    TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text 
    TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
    TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 
    TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 

    # Describing/Configuring the channel TwitterAgent.channels.MemChannel.type = memory 
    TwitterAgent.channels.MemChannel.capacity = 10000 
    TwitterAgent.channels.MemChannel.transactionCapacity = 100

    # Binding the source and sink to the channel 
    TwitterAgent.sources.Twitter.channels = MemChannel
    TwitterAgent.sinks.HDFS.channel = MemChannel 

    TwitterAgent.channels.MemChannel.type=memory

And your command should be similar to the following in Flume home

bin/flume-ng agent --conf ./conf/ -f conf/twitter.conf Dflume.root.logger=DEBUG,console -n itterAgent

You can check Tutorials Point for better understanding

Note : You can always debug the errors by looking for exact errors in flume.log in logs directory of flume

Upvotes: 1

Related Questions