CODEWITHSUNDEEP
pythontwittertwython
jxn
jxn

Reputation: 8055

how to get more than 100 tweets from the twitter search or streaming api using Twython

How can i use Twython to retrieve all tweets that mention for example "#Accounting" for say in New York?

I used the Twitter search API but i was only allowed to get 100 tweets.

I then tried to use the Twitter streaming filter API but wasn't able to narrow by geolocation and it seemed to take forever. I read that we should put all that tweets into a database and then do the aggregations there but i was wondering if there was any other way to do something quick.

Here is my code:

from twython import Twython

TWITTER_APP_KEY = 'XXXX'
TWITTER_APP_KEY_SECRET = 'XXXX'
TWITTER_ACCESS_TOKEN = 'XXXX'
TWITTER_ACCESS_TOKEN_SECRET = 'XXXX'

t = Twython(app_key=TWITTER_APP_KEY, 
            app_secret=TWITTER_APP_KEY_SECRET, 
            oauth_token=TWITTER_ACCESS_TOKEN, 
            oauth_token_secret=TWITTER_ACCESS_TOKEN_SECRET)

search = t.search(q='#Accounting', 
                    geocode='-74,40,-73,41',
                    since_id='1',
                    max_id='504082008759488512'
                  )

tweets = search['statuses']
count=0
for tweet in tweets:
    count+=1
    print tweet['id_str'], '\n', tweet['text'], '\n\n\n'

print count

And here is my code for the Streaming API:

from twython import TwythonStreamer
APP_KEY = 'XXXX'
APP_SECRET = 'XXXX' 
OAUTH_TOKEN = 'XXXX'
OAUTH_TOKEN_SECRET = 'XXXX'

class MyStreamer(TwythonStreamer):
  tweets=[] 
  def on_success(self, data):
    if 'text' in data:
        tweet= data['text'].encode('utf-8')
        if 'Accounting' in tweet:
          tweets.append(tweet)
          print tweet

  def on_error(self, status_code, data):
    print status_code
    self.disconnect()

stream = MyStreamer(APP_KEY, APP_SECRET,
                    OAUTH_TOKEN, OAUTH_TOKEN_SECRET)

stream.statuses.filter(location=['-74,40,-73,41'])

Thanks

Upvotes: 3

Views: 6648

Answers (2)

Yupsiree
Yupsiree

Reputation: 163

You can only query 100 at a time, but you have the ability to state at which ID that query should start at (or be greater or lower than).

So what you do is run your first 100, then look for the lowest ID and run another query, this time setting that ID (less 1) as your maximum ID. This will return the next 100, which you append to the previous results.

You can then run this loop as many times as you need subject to Twitter rate limits.

If you want to get the most recent tweets, you should use t.setMaxId() or t.setSinceId() set to lower or higher than your current lowest/highest ID respectively.

Upvotes: 9

Mike Helmick
Mike Helmick

Reputation: 313

Per https://dev.twitter.com/docs/api/1.1/get/search/tweets

Count optional

The number of tweets to return per page, up to a maximum of 100. Defaults to 15. This was formerly the "rpp" parameter in the old Search API.

Example Values: 100

It looks like you can only retrieve a maximum of 100 :(

Upvotes: -2

Related Questions