CODEWITHSUNDEEP
pythontwittertweepy
Marko Polo
Marko Polo

Reputation: 159

Getting back to back error using "wait_on_rate_limit" parameter

In order to avoid rate limit error I used the parameter: 

wait_on_rate_limit

in function

api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True)

At first my program worked fine. When I surpass the rate limit I got the message:
"Rate limit reached. Sleeping for: 909". My program was sleeping for this amount of time and then my program continued to collect data. At some point I got some back to back errors.

...
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

...
urllib3.exceptions.ProtocolError: ('Connection aborted.', 
ConnectionResetError(10054, 'An existing connection was forcibly closed by 
the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

...
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

During handling of the above exception, another exception occurred:

...
tweepy.error.TweepError: Failed to send request: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))

My code:

for user in tweepy.Cursor(api.friends, id="twitter").items():
    friendsOfUser=user.screen_name
    ## Do something with friendsOfUser

Is there anything that I can do?

Upvotes: 1

Views: 1930

Answers (3)

Karan Mittal
Karan Mittal

Reputation: 181

To avoid this you can add a timeout after every request. I am using a script that allows only 15 requests every 15 minutes, so I am making a single request every minute with maximizing the data.

for page in tweepy.Cursor(api.followers, screen_name=user_name, wait_on_rate_limit=True, count=200).pages():
try:
    followers.extend(page)
    print("-->", len(followers))
    if len(followers) % 100 == 0:
        save_followers_to_csv(user_name, followers)
    time.sleep(60)
except tweepy.TweepError as e:
    print("Going to sleep:", e)
    time.sleep(60)

Upvotes: 0

domi
domi

Reputation: 566

This worked for me:

    backoff_counter = 1
    while True:
        try:
            for user in tweepy.Cursor(api.friends, id="twitter").items():
                # do something with user
            break
        except tweepy.TweepError as e:
            print(e.reason)
            sleep(60*backoff_counter)
            backoff_counter += 1
            continue

Basically, when you get the error you sleep for a while, and then try again. I used an incremental backoff to make sure that the sleeping time was enough for reestablishing the connection.

Upvotes: 2

pythad
pythad

Reputation: 4267

You can't do anything with the fact that host closes the connection. If you are waiting for the rate limit, I bet you're a bit aggressive in API usage :) Try catching TweepError and explicitly waiting for some time and them try again.

You can try something like this:

import time

...
try:
    for user in tweepy.Cursor(api.friends, id="twitter").items():
        friendsOfUser=user.screen_name
        ...
except tweepy.TweepError:
    time.sleep(120) # sleep for 2 minutes. You may try different time

Upvotes: 0

Related Questions