CODEWITHSUNDEEP
pythontweepy
tcer
tcer

Reputation: 13

How to get tweets from user ids?

I have a list of over 9000 user IDs and I have to collect at most 500 tweets from each of the users. My code ran for about 5 days and only collected tweets from 541 user IDs. How could I get tweets from all the accounts? What am I doing wrong with my code?

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)  
auth.set_access_token(access_token, access_token_secret) 

ids = df_all["id_str"].tolist()
api = tweepy.API(auth, wait_on_rate_limit=True) 


for id_ in ids:
    df = pd.DataFrame()
    
    outtweets = []
    
    try:
    
        for tweet in tweepy.Cursor(api.user_timeline,id=id_).items(500):

            outtweets.append({'id':id_,
                  'tw_id_str': tweet.id_str, 
                  'tw_created_at':tweet.created_at, 
                  'tw_favorite_count':tweet.favorite_count, 
                  'tw_retweet_count':tweet.retweet_count, 
                  'tw_text':tweet.text.encode("utf-8").decode("utf-8")})
        df = pd.DataFrame(outtweets)
        df.to_csv("tweets_of_ids.csv", mode='a')

    except tweepy.TweepError as e:
        continue

Thanks a lot for the help!

Upvotes: 0

Views: 542

Answers (2)

tcer
tcer

Reputation: 13

I have actually found and adapted some lines of code that work much faster than the one above:

auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)

outtweets=[] 

for id_ in ids:
    
    c = 0 
    
    try: 
        alltweets = [] 

        new_tweets = api.user_timeline(user_id = str(id_),count=200)

        #save most recent tweets
        alltweets.extend(new_tweets)

        #save the id of the oldest tweet less one
        oldest = alltweets[-1].id - 1


        while len(new_tweets) > 0 and len(alltweets) <= 500:
            print(f"getting tweets before {oldest}")

            c = 500 - len(alltweets) 
            if c == 0: 
                break
            else:
                new_tweets = api.user_timeline(user_id = str(id_),count=c,max_id=oldest)
               
                alltweets.extend(new_tweets)

                oldest = alltweets[-1].id - 1

                print(f"...{len(alltweets)} tweets downloaded so far")

        for tweet in alltweets:
            outtweets.append({'id':id_, 'id_str':tweet.id_str, 'tw_created_at':tweet.created_at, 'tw_text':tweet.text})
    
    except tweepy.TweepError:
        print(id_)

    pass

Upvotes: 1

Beppe C
Beppe C

Reputation: 13883

The code is fine but Tweepy is throttling the requests to avoid going over the rate limit (using wait_on_rate_limit).

api = tweepy.API(auth, wait_on_rate_limit=True)

This approach prevents the application to throw an error when the limit exceeds.

There are Premium APIs which offer higher rates.

Upvotes: 0

Related Questions