CODEWITHSUNDEEP
pythonjsontwittertweepy
Diego Marino
Diego Marino

Reputation: 79

Tweets streamed using tweepy, reading json file in python

I streamed tweets using the following code

class CustomStreamListener(tweepy.StreamListener):
    def on_data(self, data):
        try:
            with open('brasil.json', 'a') as f:
                f.write(data)
                return True
        except BaseException as e:
            print("Error on_data: %s" % str(e))
        return True

Now I have a json file (brasil.json). I want to open it on python to do sentiment analysis but I can't find a way. I managed to open the first tweet using this:

with open('brasil.json') as f:
    for line in f:
        tweets.append(json.loads(line))

but it doesn't read all the other tweets. Any idea?

Upvotes: 0

Views: 2514

Answers (1)

chickity china chinese chicken
chickity china chinese chicken

Reputation: 8047

From comments: after examining the contents of the json data-file, all the tweets are in the odd number if rows. The even numbers are blank.

This caused a json.decoder.JSONDecodeError.

There are two ways to handle this error, either read only the odd rows or use exception-handling.

using odd rows:

with open('brasil.json') as f:
    for n, line in enumerate(f, 1):
        if n % 2 == 1: # this line is in an odd-numbered row
            tweets.append(json.loads(line))

exception-handling:

with open('brasil.json', 'r') as f:
    for line in f:
        try:
            tweets.append(json.loads(line))
        except json.decoder.JSONDecodeError:
            pass # skip this line

try and see which one works best.

Upvotes: 1

Related Questions