CODEWITHSUNDEEP
pythonapitwittertweepy
jpf5046
jpf5046

Reputation: 797

Tweepy: Ignore previous tweets to improve optimization

Problem: Trying to pull tweets via tweepy using Cursor. I want to make sure I don't pull tweets I previously pulled.

Here is working code:

import tweepy
import pandas as pd
import numpy as np

ACCESS_TOKEN = ""
ACCESS_TOKEN_SECRET = ""
CONSUMER_KEY = ""
CONSUMER_SECRET = ""

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

# Creation of the actual interface, using authentication
api = tweepy.API(auth, wait_on_rate_limit=True)

csvFile = open(r'filename', 'a')

#Use csv writer
headers = ['UserName', 'Tweet', 'TweetId', 'tweet_date', 'source', 'fav_count', 'retweet_count', 'coordinates', 'geo']

# definitions for writing to CSV
csvWriter = csv.writer(csvFile, lineterminator='\n')
# write the headers once
csvWriter.writerow(headers)


handles = ['pycon', 'gvanrossum']
previousTweets = 
 ['222288832031240000',
 '222287080586362000',
 '222277240178741000',
 '221414283844653000',
 '221188011906445000',
 '205274818877210000']


for handle in handles:   
    for status in tweepy.Cursor(api.user_timeline, screen_name= handle, tweet_mode="extended").items():
        if status.id not in previousTweets:
            csvWriter.writerow([status.user.name.encode('utf-8'), status.full_text.encode('utf-8'), status.id, status.created_at, status.source, 
                    status.favorite_count, status.retweet_count, status.coordinates, status.geo])
print(handle)

This takes a long time and becomes unusable if you want to have a PreviousTweet list of over 75 tweets. Does anyone know a better way to filter out old tweets when using Tweepy and the Cursor function?

Upvotes: 0

Views: 167

Answers (1)

Oluwafemi Sule
Oluwafemi Sule

Reputation: 38962

You can pass the since_id argument to the cursor. This allows fetching status that is more recent than the specified ID ( http://docs.tweepy.org/en/v3.5.0/api.html#API.user_timeline)

try:
    since_id = previous_tweets[-1]
except IndexError:
    since_id = None

for handle in handles:
    last_tweet = None
    for status in tweepy.Cursor(
      api.user_timeline, screen_name=handle, 
      tweet_mode="extended", since_id=since_id
    ).items():
     # ... persist tweets to flat file or database  
     last_tweet_id = status.id

    # this persists the last_tweet_id in memory.
    # you may find that persisting this to a database a better way to go.
    previous_tweets.append(last_tweet_id)

Upvotes: 2

Related Questions