CODEWITHSUNDEEP
hadooptwitter4jflume
Harshita Taparia
Harshita Taparia

Reputation: 21

hadoop flume extracting twitter data

I Have used this command while extracting twitter data using flume

[cloudera@localhost bin]$ ./flume-ng agent --conf ./conf/ -f ../conf/flume.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent

2015-07-14 05:42:00 [INFO ] Configuration provider starting
2015-07-14 05:42:00 [INFO ] Reloading configuration file:../conf/flume.conf
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Added sinks: HDFS Agent: TwitterAgent
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Processing:HDFS
2015-07-14 05:42:00 [INFO ] Post-validation flume configuration contains configuration for agents: [TwitterAgent]
2015-07-14 05:42:00 [INFO ] Creating channels
2015-07-14 05:42:00 [INFO ] Creating instance of channel MemChannel type memory
2015-07-14 05:42:00 [INFO ] Created channel MemChannel
2015-07-14 05:42:00 [INFO ] Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource

It is processing up to this then here is the error:

2015-07-14 05:42:01 [ERROR] Unhand-led error java.lang.NoSuchMethodError: twitter4j.TwitterStream.addListener(Ltwitter4j/StatusListener;)V at org.apache.flume.source.twitter.TwitterSource.configure(TwitterSource.java:119) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPool Executor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2015-07-14 05:44:13 [INFO ] Stopping life cycle supervisor 10 2015-07-14 05:44:13 [INFO ] Configuration provider stopping

Upvotes: 0

Views: 907

Answers (1)

Erik Schmiegelow
Erik Schmiegelow

Reputation: 2759

java.lang.NoSuchMethodError <- this indicates that you are using the wrong version of twitter4j.

The current 1.6.0 version of flume's twitter-source is built against 3.0.3:

[INFO] +- org.twitter4j:twitter4j-core:jar:3.0.3:compile
[INFO] +- org.twitter4j:twitter4j-media-support:jar:3.0.3:compile
[INFO] \- org.twitter4j:twitter4j-stream:jar:3.0.3:compile

just replace your twitter4j libs with those, then it will work.

Upvotes: 1

Related Questions