Getting error while load data from Twitter to hdfs

Question

I am getting error while load data from twitter to hdfs

I am using ambari sandbox hortonworks hadoop-2.7

This is my flume.conf file

flume.conf:

    TwitterAgent.sources = Twitter
    TwitterAgent.channels = MemChannel
    TwitterAgent.sinks = HDFS

    TwitterAgent.sources.Twitter.type =    
    com.cloudera.flume.source.TwitterSource
    TwitterAgent.sources.Twitter.channels = MemChannel
    TwitterAgent.sources.Twitter.consumerKey =oblBU8btK3OpuSoFce8fJTOz9
    TwitterAgent.sources.Twitter.consumerSecret     
    =ofsGWmx1T4GHvi8qDcAySUAC3mVdvSS8VcfD9CPTejxzQ52izk
    TwitterAgent.sources.Twitter.accessToken =3479003538-     
    2OP1N7wKqSkAohXscehBdhbMfJhoXqSPkng7cPY
    TwitterAgent.sources.Twitter.accessTokenSecret       
    =0vrKLzdUplRnPjcTWiSNKhu9Ohe18FcoOXYMmD7OUazTt
    TwitterAgent.sources.Twitter.keywords = hadoop, big data, analytics
    TwitterAgent.sinks.HDFS.channel = MemChannel
    TwitterAgent.sinks.HDFS.type = hdfs
    TwitterAgent.sinks.HDFS.hdfs.path =/flume/tweets
    TwitterAgent.sinks.HDFS.hdfs.fileType =DataStream
    TwitterAgent.sinks.HDFS.hdfs.filePrefix =twitter
    TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
    TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
    TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
    TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
    TwitterAgent.sinks.HDFS.hdfs.rollInterval = 10

    TwitterAgent.channels.MemChannel.type = memory
    TwitterAgent.channels.MemChannel.capacity = 10000
    TwitterAgent.channels.MemChannel.transactionCapacity = 100

15/09/11 07:21:03 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:03 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:03 INFO twitter4j.TwitterStreamImpl: Waiting for 1000 milliseconds 15/09/11 07:21:04 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:04 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:04 INFO twitter4j.TwitterStreamImpl: Waiting for 2000 milliseconds 15/09/11 07:21:06 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:06 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:06 INFO twitter4j.TwitterStreamImpl: Waiting for 4000 milliseconds 15/09/11 07:21:10 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:10 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:10 INFO twitter4j.TwitterStreamImpl: Waiting for 8000 milliseconds 15/09/11 07:21:18 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:18 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:18 INFO twitter4j.TwitterStreamImpl: Waiting for 16000 milliseconds 15/09/11 07:21:34 INFO twitter4j.TwitterStreamImpl: Establishing connection. 15/09/11 07:21:34 INFO twitter4j.TwitterStreamImpl: stream.twitter.com 15/09/11 07:21:34 INFO twitter4j.TwitterStreamImpl: Waiting for 16000 milliseconds

^C15/09/11 07:21:45 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 10 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS stopped 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.start.time == 1441956061906 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.stop.time == 1441956105092 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.complete == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.empty == 7 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.batch.underflow == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.closed.count == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.creation.count == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.connection.failed.count == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.event.drain.attempt == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: HDFS. sink.event.drain.sucess == 0 15/09/11 07:21:45 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider stopping 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: MemChannel stopped 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.start.time == 1441956061903 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.stop.time == 1441956105094 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.capacity == 10000 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.current.size == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.put.attempt == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.put.success == 0 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.take.attempt == 7 15/09/11 07:21:45 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: MemChannel. channel.event.take.success == 0 [root@sandbox bin]#

Answer 1

Looks like you have not given complete hdfs path:

TwitterAgent.sinks.HDFS.hdfs.path =hdfs://localhost:8020/flume/tweets

here localhost is hostname and 8020 hdfs port.