CODEWITHSUNDEEP
pythonjsonpython-3.xamazon-s3
muazfaiz
muazfaiz

Reputation: 5021

Python write JSON to S3

Here is my list of dicts which I need to write to S3 in JSON.

data = [{"name": "abc", "age": 23}, {"name": "def", "age": 21}, {"name": "fgh", "age": 34}]

I want to write it as example.json in the following format

{"name": "abc", "age": 23}
{"name": "def", "age": 21}
{"name": "fgh", "age": 34}

So, writing each dict on one line and use \n as line break. This code works for me locally:

import json
with open('example.json', 'w') as f:
    for d in data:
        json.dump(d, f, ensure_ascii=False)
        f.write('\n')

Now I don't want to save the file locally but to S3 directly line by line or anyway such that the desired format is preserved. I know we can use json.dumps() directly to write to S3 like this

import json
import boto3

s3 = boto3.client('s3')
s3.put_object(
     Body=str(json.dumps(data))
     Bucket='your_bucket_name'
     Key='your_key_here'
)

But I want to preserve the format which this would not do. How should I move forward with this?

Upvotes: 1

Views: 9511

Answers (1)

Daniel Scott
Daniel Scott

Reputation: 7903

Is there anything wrong with just generating the string and writing to S3?

data_string = ""
for d in data:
    data_string += json.dumps(d, ensure_ascii=False)
    data_string += "\n"

s3 = boto3.client('s3')
s3.put_object(
     Body=data_string
     Bucket='your_bucket_name'
     Key='your_key_here'
)

Upvotes: 4

Related Questions