CODEWITHSUNDEEP
pythonjson
echo140
echo140

Reputation: 21

Python error trying to append JSON file

I'm relatively new to Python so my issue may be a simple one to fix but after days of trying and searching the internet I cant find anything.

So I built a script to stream data from Twitter and store the collecting data into a json file so that I can later access it and do whatever. This script pulls the user credentials like consumer key, token, and access info from a separate file to authenticate (I'm sure there is a better and more secure way to do that, this is just a proof of concept at the moment) using this code:

with open('Twitter_Credentials.json', mode = 'a+') as tc:
            data = json.load(tc)
            if user not in data['names']:
                user_dict = dict()
                user_dict[user] = {'key':'','secret':'','token':'','token_secret':''}
                user_dict[user]['key'] = input('Twitter Consumer Key: ')
                user_dict[user]['secret'] = input('Twitter Consumer Secret: ')
                user_dict[user]['token'] = input('Twitter Access Token: ')
                user_dict[user]['token_secret'] = input('Twitter Access Secret: ')
                data['names'].append(user_dict)
                json.dump(data,tc, indent = 2, ensure_ascii = False)
                tc.close()

The issue I am having is that if I want to append another user and their credentials to this file I keep receiving this error:

File "(filepath)", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

THINGS I HAVE ALREADY TRIED:

Using 'r+' and 'w+' did not give me an error, but it did duplicate the original user so that they appeared multiple times. I want to eliminate that so that when it appends it does not duplicate. Any insight would be greatly appreciated. Thanks in advance.

Upvotes: 1

Views: 1116

Answers (1)

abarnert
abarnert

Reputation: 365767

A JSON file is a file containing a single JSON document. If you append another JSON string to it, it's no longer a JSON file.

As the docs say:

Note Unlike pickle and marshal, JSON is not a framed protocol, so trying to serialize multiple objects with repeated calls to dump() using the same fp will result in an invalid JSON file.

If you aren't actually trying to store multiple documents in one file, the fix is easy: what you want to do is open the file, load it, modify the data, then open the file again and overwrite it. Like this:

with open('Twitter_Credentials.json', mode = 'r') as tc:
    data = json.load(tc)
if user not in data['names']:
    # blah blah
    with open('Twitter_Credentials.json', mode = 'w') as tc:
        json.dump(data, tc, indent = 2, ensure_ascii = False)

Notice that I'm using w mode, not a, because we want to overwrite the old file with the new one, not tack stuff onto the end of it.

If you are trying to store multiple documents, then you can't do that with JSON. Fortunately, there are very simple framed protocols based on JSON—JSONlines, NDJ, etc.—that are commonly used. There are three or four different such formats with minor differences, but the key to all of them is that each JSON document goes on a line by itself, with newlines between the documents.

But using ensure_ascii=False means you're not escaping newlines in strings, and indent=2 means you're adding more newlines between fields within the document, and then you aren't doing anything to write a newline after each document. So, your output isn't valid JSONlines any more than it's valid JSON.

Also, even if you fix all that, you're doing a single json.load, which would read only the first document out of a JSONlines file, and then doing json.dump to the same file, which would write a second document after that one, overwriting whatever's there. You could easily end up overwriting, say, half of the previous second document, leaving the other half behind as garbage to read later. So, you need to rethink your logic. At minimum, you want to do the same thing as above, opening the file twice:

with open('Twitter_Credentials.json', mode = 'r') as tc:
    data = json.load(tc)
if user not in data['names']:
    # blah blah
    with open('Twitter_Credentials.json', mode = 'a') as tc:
        json.dump(data, tc)
        tc.write('\n')

This time I'm using a mode, because this time we do want to tack a new line onto the end of an existing file.

Upvotes: 3

Related Questions