Getting Back To Back Error Using "wait_on_rate_limit" Parameter

In order to avoid rate limit error I used the parameter: wait_on_rate_limit in function api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True) At first my

Solution 1:

This worked for me:

    backoff_counter = 1
    while True:
        try:
            for user in tweepy.Cursor(api.friends, id="twitter").items():
                # do something with user
            break
        except tweepy.TweepError as e:
            print(e.reason)
            sleep(60*backoff_counter)
            backoff_counter += 1
            continue

Basically, when you get the error you sleep for a while, and then try again. I used an incremental backoff to make sure that the sleeping time was enough for reestablishing the connection.

Solution 2:

You can't do anything with the fact that host closes the connection. If you are waiting for the rate limit, I bet you're a bit aggressive in API usage :) Try catching TweepError and explicitly waiting for some time and them try again.

You can try something like this:

import time

...
try:
    for user in tweepy.Cursor(api.friends, id="twitter").items():
        friendsOfUser=user.screen_name
        ...
except tweepy.TweepError:
    time.sleep(120) # sleep for 2 minutes. You may try different time

Solution 3:

To avoid this you can add a timeout after every request. I am using a script that allows only 15 requests every 15 minutes, so I am making a single request every minute with maximizing the data.

for page in tweepy.Cursor(api.followers, screen_name=user_name, wait_on_rate_limit=True, count=200).pages():
try:
    followers.extend(page)
    print("-->", len(followers))
    if len(followers) % 100 == 0:
        save_followers_to_csv(user_name, followers)
    time.sleep(60)
except tweepy.TweepError as e:
    print("Going to sleep:", e)
    time.sleep(60)