CODEWITHSUNDEEP
pythonboto3
Learning
Learning

Reputation: 786

Writing json to file in s3 bucket

This code writes json to a file in s3, what i wanted to achieve is instead of opening data.json file and writing to s3 (sample.json) file,

how do i pass the json directly and write to a file in s3 ?

import boto3

s3 = boto3.resource('s3', aws_access_key_id='aws_key', aws_secret_access_key='aws_sec_key')
s3.Object('mybucket', 'sample.json').put(Body=open('data.json', 'rb'))

Upvotes: 70

Views: 170129

Answers (4)

Manuel Montoya
Manuel Montoya

Reputation: 1446

An alternative to Joseph McCombs answer can be achieved using s3fs.

from s3fs import S3FileSystem

json_object = {'test': 3.14}
path_to_s3_object = 's3://your-s3-bucket/your_json_filename.json'

s3 = S3FileSystem()
with s3.open(path_to_s3_object, 'w') as file:
    json.dump(json_object, file)

Upvotes: 6

Joseph McCombs
Joseph McCombs

Reputation: 831

I don't know if anyone is still attempting to use this thread, but I was attempting to upload a JSON to s3 trying to use the method above, which didnt quite work for me. Boto and s3 might have changed since 2018, but this achieved the results for me:

import json
import boto3

s3 = boto3.client('s3')
json_object = 'your_json_object here'
s3.put_object(
     Body=json.dumps(json_object),
     Bucket='your_bucket_name',
     Key='your_key_here'
)

Upvotes: 83

Uwe Bretschneider
Uwe Bretschneider

Reputation: 1281

I'm not sure, if I get the question right. You just want to write JSON data to a file using Boto3? The following code writes a python dictionary to a JSON file.

import json
import boto3    
s3 = boto3.resource('s3')
s3object = s3.Object('your-bucket-name', 'your_file.json')

s3object.put(
    Body=(bytes(json.dumps(json_data).encode('UTF-8')))
)

Upvotes: 102

Usman Mutawakil
Usman Mutawakil

Reputation: 5259

Amazon S3 is an object store (File store in reality). The primary operations are PUT and GET. You can not add data into an existing object in S3. You can only replace the entire object itself.

For a list of available operations you can perform on s3 see this link http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectOps.html

Upvotes: 8

Related Questions