CODEWITHSUNDEEP
pythonapitwittertweepytweetstream
David Yi
David Yi

Reputation: 401

Transferring Twitter Tweets to a txt file

from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import json 
from pprint import pprint

data_file = open('twitter.json')  
data = json.load(data_file)
##Json file with all the ckey, csecret, atoken, and asecret
pprint(data)

#consumer key, consumer secret, access token, access secret.
ckey = data["ckey"]
csecret = data["csecret"]
atoken = data["atoken"]
asecret = data["asecret"]

class listener(StreamListener):

def on_data(self, data):
    all_data = json.loads(data)       
    tweet = all_data["text"]        
    username = all_data["user"]["screen_name"]
    print((username,tweet))
    return True

def on_error(self, status):
    print (status)


auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)

The code above is all standard in accessing the twitter api. However, I need to transfer the tweets obtained from twitter to a .txt file. I tried using the code below twitterStream = Stream(auth, listener())

fid = open("cats based tweets.txt","w")
for tweet in twitterStream.filter(track=[cats]):
        fid.write(tweet)
    fid.close()

I intend on finding all twitter tweets/reposts that include the keyword cats, which it does. However, it is supposed to also write a txt file that includes all the tweets but it doesn't. Can anyone tell me what I need to do it fix it.

EDIT : I used the code that you guys have written but it doesn't return all of the tweets. It prints out like 5 or 6 then the error

RuntimeError: No active exception to reraise

appears and I have no idea why. Why does this occur cause I know it shouldn't.

Upvotes: 0

Views: 8781

Answers (3)

blue_chip
blue_chip

Reputation: 746

I've done this in a project and my method involves changing the on_data method within the StreamListener object. My code looks like this:

class Listener(StreamListener):
    def __init__(self, api=None, path=None):
        #I don't remember exactly why I defined this.
        self.api = api
        #We'll need this later.
        self.path = path

    def on_data(self, data):
        all_data = json.loads(data)

        tweet = all_data["text"]        
        username = all_data["user"]["screen_name"]
        print((username,tweet))

        #Open, write and close your file.
        savefile = open(file_path, 'ab')
        savefile.write(tweet)
        savefile.close()

        return True

A few things in the actual code, not where you redefined Listener or on_data. In order:

  1. Define the file where you want to save. Let's call that variable the file_path. Don't forget to add the .txt extensions here.

  2. Call the Stream and the Listener:

    twitterStream = Stream(authorization, Listener(path=file_path))

  3. Use your filters. Mine are coordinates and I put the filter in a try, except so that my code doesn't stop. Here it is adapted for you:

    try:
    twitterStream.filter(track=[cats])
except Exception, e:
    print 'Failed filter() with this error:', str(e)

Now the text in the tweet should be written in the file whenever a text appears in the stream. Take a look at your file size and you should see it increase. Particularly, if your filter is about cats. Internet loves cats.

Upvotes: 1

ZdaR
ZdaR

Reputation: 22954

I guess there is a slight indentation error in the snippet you provided, However I will try to fix your error with 2 approaches, the first one is by correcting the indentation and the second one would be to change youron_data method

Approach 1:

fid = open("cats based tweets.txt","w")
for tweet in twitterStream.filter(track=[cats]):
    fid.write(tweet+"\n")
fid.close()

Or you could simply write the above code as :

with open("cats based tweets.txt","w") as fid:
    for tweet in twitterStream.filter(track=[cats]):
        fid.write(tweet+"\n")

Approach 2:

In the second approach we can change the on_data method so that when the program receives a new tweet it opens and file and directly writes to it , but for this we need to open the file in append mode, as opening the file in w writeable mode would overwrite the contents of the file again and again.

def on_data(self, data):
    all_data = json.loads(data)       
    tweet = all_data["text"]        
    username = all_data["user"]["screen_name"]
    print((username,tweet))
    with open("cats based tweets.txt","a") as fid:
        fid.write(tweet+"\n")
    return True

Upvotes: 1

Anandhakumar R
Anandhakumar R

Reputation: 391

See the below link then you will know about how to save the tweets to Database as well as to the our local file.

https://github.com/anandstarz/Scrapee/blob/master/tweets

Upvotes: 0

Related Questions