CODEWITHSUNDEEP
amazon-web-servicesboto3amazon-glacier
mkomarinski
mkomarinski

Reputation: 113

boto3 upload a string to glacier file

My workflow has a tar file downloaded from S3, expanded then I optionally want to upload it into a glacier vault. Given there's other files in the S3 bucket, I don't want to use life-cycle management. I have all this working under boto and am now slowly upgrading to boto3

I've recently discovered that rather than downloading into an on-disk file I can download into a string object and operate on it, which makes uncompressing a lot faster since I don't need to touch the disk.

s3 = boto3.client('s3')
response = s3.get_object(Bucket=bucket,Key=path)
my_file = tarfile.open(fileobj=(StringIO(response['Body'].read())))
my_file.extractall(path="EXTRACTPATH")

If I want to upload to glacier via boto3, here's what I have:

glacier = boto3.client('glacier', region_name='MYREGION')
archive = glacier.upload_archive(vaultName='MYVAULT', archiveDescription=filename, body=response['Body'].read())

And that nets me:

botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the UploadArchive operation: Invalid Content-Length: 0

Any thoughts?

Upvotes: 1

Views: 1296

Answers (1)

Jordon Phillips
Jordon Phillips

Reputation: 16003

The StreamingBody is a non-seekable stream, it reads directly from the socket so you only get one read. You'll need to save the bytes if you want to use them in multiple locations.

Upvotes: 1

Related Questions