Twitter - Hadoop Data Streaming

Question

How do we get the twitter(Tweets) into HDFS for offline analysis. we have a requirement to analyze tweets.

Answer 1

Fluentd log collector just released its WebHDFS plugin, which allows the users to instantly stream data into HDFS.

• Fluentd + Hadoop: Instant Big Data Collection

Also by using fluent-plugin-twitter, you can collect Twitter streams by calling its APIs. Of course you can create your custom collector, which posts streams to Fluentd. Here's a Ruby example to post logs against Fluentd.

• Fluentd: Data Import from Ruby Applications

Answer 2

I would look for solution in well developed area of streaming logs into hadoop, since the task looks somewhat similar.
There are two existing systems doing so:
Flume: https://github.com/cloudera/flume/wiki
And
Scribe: https://github.com/facebook/scribe

So your task will be only to pull data from twitter, what I asume is not part of this question and feed one of these systems with this logs.

Answer 3

This can be a solution to your problem.

• Tools to capture Twitter tweets

  • Create PDF, DOC, XML and other docs from Twitter tweets
  • Tweets to CSV files

• Capture it in any format. (csv,txt,doc,pdf.....etc)

• Put it into HDFS.