Parsing Twitter JSON object in Python

Question

I am trying to download tweets from twitter.

I have used python and Tweepy for this. Though I am new to both Python and Twitter API.

My Python script is as follow: #!usr/bin/python

#import modules
import sys
import tweepy
import json

#global variables
consumer_key = ''
consumer_secret = ''
token_key = ''
token_secret = ''

#Main function
def main():
    print sys.argv[0],'starts'
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
    auth.set_access_token(token_key, token_secret)
    print 'Connected to Twitter'
    api = tweepy.API(auth)
    if not api.test():
        print 'Twitter API test failed'

    print 'Experiment with cursor'
    print 'Get search method returns json objects'

   json_search = api.search(q="football")
   #json.loads(json_search())
   print  json_search


#Standard boilerplate to call main function if this file runs

if __name__ == '__main__':
    main()

I am getting result as follows:

[<tweepy.models.SearchResult object at 0x9a0934c>, <tweepy.models.SearchResult object at 0x9a0986c>, <tweepy.models.SearchResult object at 0x9a096ec>, <tweepy.models.SearchResult object at 0xb76d8ccc>, <tweepy.models.SearchResult object at 0x9a09ccc>, <tweepy.models.SearchResult object at 0x9a0974c>, <tweepy.models.SearchResult object at 0x9a0940c>, <tweepy.models.SearchResult object at 0x99fdfcc>, <tweepy.models.SearchResult object at 0x99fdfec>, <tweepy.models.SearchResult object at 0x9a08cec>, <tweepy.models.SearchResult object at 0x9a08f4c>, <tweepy.models.SearchResult object at 0x9a08eec>, <tweepy.models.SearchResult object at 0x9a08a4c>, <tweepy.models.SearchResult object at 0x9a08c0c>, <tweepy.models.SearchResult object at 0x9a08dcc>]

Now I am confused how to extract tweets from this information? I tried to use json.loads method on this data. But it gives me error as JSON expects string or buffer. Example code would be highly appreciated. Thanks in advance.

Answer 1

Instead of using global variables, I would reorganize the code in a python class:

import tweepy

class TweetPrinter():
    """
        Simple class to print tweets
    """
    def __init__(self, consumer_key, consumer_secret, access_token, 
                 access_token_secret):
        self.consumer_key = consumer_key
        self.consumer_secret = consumer_secret
        self.access_token = access_token
        self.access_token_secret = access_token_secret
        self.auth = tweepy.OAuthHandler(self.consumer_key, 
                                        self.consumer_secret)
        self.auth.set_access_token(access_token, access_token_secret)

    def tweet_print(self):
        api = tweepy.API(self.auth)
        football_tweets = api.search(q="football")
        for tweet in football_tweets:
            print(tweet.text)


def main():
    tweet_printer = TweetPrinter(my_consumer_key, my_consumer_secret, 
                              my_access_token, my_access_token_secret)

    tweet_printer.tweet_print()

if __name__ == '__main__':
    main()

Answer 2

You can use the JSON parser to achieve this, here is my code on App Engine that handles a JSONP response ready to be used in with a JQuery client:

import webapp2
import tweepy
import json
from tweepy.parsers import JSONParser

class APISearchHandler(webapp2.RequestHandler):
    def get(self):

        CONSUMER_KEY = 'xxxx'
        CONSUMER_SECRET = 'xxxx'
        ACCESS_TOKEN_KEY = 'xxxx'
        ACCESS_TOKEN_SECRET = 'xxxx'

        auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
        auth.set_access_token(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
        api = tweepy.API(auth, parser=JSONParser())

        # Query String Parameters
        qs = self.request.get('q')
        max_id = self.request.get('max_id')

        # JSONP Callback
        callback = self.request.get('callback')

        max_tweets = 100
        search_results = api.search(q=qs, count=max_tweets, max_id=max_id)
        json_str = json.dumps( search_results )

        if callback:
            response = "%s(%s)" % (callback, json_str)
        else:
            response = json_str

        self.response.write( response )

So the key point is

api = tweepy.API(auth, parser=JSONParser())

Answer 3

Tweepy gives you richer objects; it parsed the JSON for you.

The SearchResult objects have the same attributes as the JSON structures that Twitter sent; just look up the Tweet documentation to see what is available:

for result in api.search(q="football"):
    print result.text

Demo:

>>> import tweepy
>>> tweepy.__version__
'3.3.0'
>>> consumer_key = '<consumer_key>'
>>> consumer_secret = '<consumer_secret>'
>>> access_token = '<access_token>'
>>> access_token_secret = '<access_token_secret>'
>>> auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
>>> auth.set_access_token(access_token, access_token_secret)
>>> api = tweepy.API(auth)
>>> for result in api.search(q="football"):
...     print result.text
... 
Great moments from the Women's FA Cup http://t.co/Y4C0LFJed9
RT @freebets: 6 YEARS AGO TODAY: 

Football lost one of its great managers. 

RIP Sir Bobby Robson. http://t.co/NCo90ZIUPY
RT @Oddschanger: COMPETITION CLOSES TODAY!

Win a Premier League or Football League shirt of YOUR choice! 

RETWEET &amp; FOLLOW to enter. http…
Berita Transfer: Transfer rumours and paper review – Friday, July 31 http://t.co/qRrDIEP2zh [TS] #nobar #gosip
@ajperry18 im sorry I don't know this football shit😂😅
@risu_football おれモロ誕生日で北辰なんすよ笑
NFF Unveils Oliseh As Super Eagles Coach - SUNDAY Oliseh has been unveiled by the Nigeria Football... http://t.co/IOYajD9bi2 #Sports
RT @BilelGhazi: RT @lequipe : Gourcuff, au tour de Guingamp http://t.co/Dkio8v9LZq
@EDS_Amy HP SAUCE ?
RT @fsntweet: マンＣの塩対応に怒りの炎！ベトナム人ファン、チケットを燃やして猛抗議 - http://t.co/yg5iuABy3K 

なめるなよ、プレミアリーグ！マンチェスターＣのプレシーズンツアーの行き先でベトナム人男性が、衝撃的な行
RT @peterMwendo: Le football cest un sport collectif ou on doit se faire des passe http://t.co/61hy138yo8
RT @TSBible: 6 years ago today, football lost a true gentleman. Rest in Peace Sir Bobby Robson. http://t.co/6eHTI6UxaC
6 years ago today the greatest football manger of all time passed away SIR Bobby Robson a true Ipswich and footballing legend
The Guardian: PSG close to sealing £40m deal for Manchester United’s Ángel Di María. http://t.co/gAQEucRLZa
Sir Bobby Robson, the #football #legend passed away 6 years ago. 

#Barcelona #newcastle #Porto http://t.co/4UXpnvrHhS

Answer 4

take my code for tweepy:

def twitterfeed():
   auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
   auth.set_access_token(access_key, access_secret)
   api = tweepy.API(auth)
   statuses = tweepy.Cursor(api.home_timeline).items(20)
   data = [s.text.encode('utf8') for s in statuses]
   print data