CODEWITHSUNDEEP
amazon-s3boto
Jeremy Dunck
Jeremy Dunck

Reputation: 5894

Stream stdout to s3

I have a stream (suppose it's stdout from a generating process) which I would like to send to s3 using boto.

Could I have a code example, please?

Upvotes: 4

Views: 1698

Answers (2)

Paul M
Paul M

Reputation: 2046

I think the answer is that you cannot easily stream data into s3 unless you know the size of the data you are streaming because s3 requires the size of the object to be known beforehand. You can definitely stream data out of s3 but to get data in, you'll need to buffer in memory or disk unless you are lucky enough to know the size.

Upvotes: 6

Tony Gibbs
Tony Gibbs

Reputation: 2489

Pretty much from the tutorial: http://boto.s3.amazonaws.com/s3_tut.html#storing-data

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import subprocess

conn = S3Connection()
buckets = conn.get_all_buckets()
bucket = buckets[0]   # assuming you have one bucket created

output = subprocess.check_output(["echo", "Hello World!"])
k = Key(bucket)
k.key = 'test'
k.set_contents_from_string(output)  # should be saved on S3

k.get_contents_as_string()          # should retrieve the string

Upvotes: 1

Related Questions