CODEWITHSUNDEEP
pythontwittertweepy
Marcus
Marcus

Reputation: 9

Twitter API - Obtain user tweets and parse into a table/database

This is a small project I'd like to get started on in the near future. It's still in the planning stage so this post is more about being steered in the right direction

Essentially, I'd like to obtain tweets from a user and parse the tweets into a table/database, with the aim to be able to run this program in real-time.

My initial plan to tackle this was to use Beautiful Soup, a Python specific library, however, I believe the Twitter API is the better approach (advice on this subject would be appreciated)

There are still 3 unknowns:

  1. Where do I store the tweets once obtained?
  2. How to parse the tweets?
  3. Where to store the parsed data?

To answer (3), I suppose it depends on what I want to do with the data. I still haven't decided how I'll use the parsed data but I know that I'd like it put into categories so my thinking is probably a database/table/excel??

A few questions still to answer and I'd like you guys to steer me in the right direction. My programming language knowledge is limited to just C for now, but as this project means a great deal to me, I'm willing to put the effort in and learn the necessary languages/APIs.

What languages/APIs will I need to gain an understanding of to accomplish this project? From where I stand, it seems to be Twitter API and Python.

EDIT: So I have a basic script going which obtains a user tweets. It works better than expected. However, I'd like to take it another step. I'd like to only obtain the users' tweets if it contains a hashtag inside of the tweet. All other tweets should be ignored. How best to do this?

Here is a snippet of the basic code I have going:

import tweepy
import twitter_credentials

auth = tweepy.OAuthHandler(twitter_credentials.CONSUMER_KEY, twitter_credentials.CONSUMER_SECRET)
auth.set_access_token(twitter_credentials.ACCESS_TOKEN, twitter_credentials.ACCESS_TOKEN_SECRET)
api = tweepy.API(auth)

stuff = api.user_timeline(screen_name = 'XXXXXXXXXX', count = 10, include_rts = False)

for status in stuff:
    print(status.text)

Upvotes: 0

Views: 1335

Answers (1)

JeffProd
JeffProd

Reputation: 3148

Scraping Twitter (or any other social network) with for example Beautiful soup, as you said, is not a good idea for 2 reasons :

  • if the source pages changes (name attributes, div ids...), you have to keep your code up to date
  • your script can be banned because scraping is not "allowed".

To answer your questions :

1) you can store the tweets wherever you want : csv, mysql, sqlite, redis, neo4j...

2) With official API, you get JSON. Here is a Tweet Object : https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/tweet-object.html . With tweepy, for example status.text will give you the text of the tweet.

3) Same as #1. If you don't know actually what you will do with the data, store the full JSONs. You will be able later to parse them.

I suggest tweepy/python (http://www.tweepy.org/) or twit/nodejs (https://www.npmjs.com/package/twit). And read official docs : https://developer.twitter.com/en/docs/api-reference-index

Upvotes: 0

Related Questions