CODEWITHSUNDEEP
pythonerror-handlingtweepyrate-limiting
OneManRiot
OneManRiot

Reputation: 971

Working around error 104 and Twitter rate limiting

I'm working on a research project that involves analyzing large amounts of data from Twitter. The project is being built in Python using Tweepy. As you might imagine I have to work very closely within the confines of the Twitter rate limiter. As such, my authentication code looks like this.

auth1 = tweepy.OAuthHandler("...", "...")
auth1.set_access_token("...", "...")
api1 = tweepy.API(auth1, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

Which does a wonderful job of stopping and waiting before I trip my limit on requests for a small scaled down run. However, when I try and run the program on my full data set I eventually get this error while the program is sleeping:

tweepy.error.TweepError: Failed to send request: ('Connection aborted.', error(104, 'Connection reset by peer'))

My research tells me that this is happening because Twitter is disconnecting and I need to catch the error. How would I catch this error, reconnect and have my program pick up where it left off? Any advice would be welcome.

Upvotes: 2

Views: 2161

Answers (2)

gios
gios

Reputation: 1

I've also tried at the same way to wrap Tweepy calls inside a while True loop, but I got also issues with reconnections (in some cases this solution does not permit equally to solve the problem). Otherwise, I've thought to switch Auth (connected to Tweepy API instance, here "twapi") in case of error, and it seems to work properly: 

...
while True:
    try:
        users_stream = twapi.lookup_users(screen_names=[scrname_list_here])
    except tweepy.error.TweepError, ex:
        time.sleep(120)
        global twapi
        global switch_auth
        if switch_auth == False:
            twapi = tweepy.API(auths[auth_id+1])  
            switch_auth = True
        elif switch_auth == True:
            twapi = tweepy.API(auths[auth_id])    
            switch_auth = False
        continue
    break
 ...

By using a bool variable switch_auth is possible (in case arises the Tweepy error related to failed reconnection) to "switch" the auth input of Tweepy API module (it can be assumed stored in auths list) to solve the problem. The same technique can be used to 'switch' Auth when research's rate limit is reached. I hope it will be useful, just try!

Upvotes: 0

avinash pandey
avinash pandey

Reputation: 1381

The twitter disconnection error are socket exception,which is a special case of IOError exceptions.In order to catch that you need to do something like 

auth = tweepy.OAuthHandler(… # set up your oauth here
try:
    stream = tweepy.Stream(auth=auth, listener=SomeListener()) # start the stream
except IOError, ex:
    print 'I just caught the exception: %s' % ex

If it works wrap in a while True loop with an increasing backoff so to provide some pause between re-connection.Reference link

Upvotes: 1

Related Questions