CODEWITHSUNDEEP
python-2.7amazon-web-servicesamazon-s3boto3
Sivakumar M
Sivakumar M

Reputation: 31

uploading file to specific folder in S3 bucket using boto3

My code is working. The only issue I'm facing is that I cannot specify the folder within the S3 bucket that I would like to place my file in. Here is what I have:

with open("/hadoop/prodtest/tips/ut/s3/test_audit_log.txt", "rb") as f:
    s3.upload_fileobj(f, "us-east-1-tip-s3-stage", "BlueConnect/test_audit_log.txt")

Upvotes: 2

Views: 9186

Answers (2)

Adil Hindistan
Adil Hindistan

Reputation: 6605

Explanation from @danimal captures pretty much everything. If you wanted to just create a folder-like object in s3, you can simply specify that folder-name and end it with "/", so that when you look at it from the console, it will look like a folder.

It's rather useless, an empty object, without a body (consider it as a key with null value) just for eye-candy but if you really want to do it, you can. 1) You can create it on the console interactively, as it gives you that option 2_ You can use aws sdk. boto3 has put_object method for s3 client, where you specify the key as "your_folder_name/", see example below:

import boto3
session = boto3.Session() # I assume you know how to provide credentials etc.
s3 = session.client('s3', 'us-east-1')

bucket = s3.create_bucket('my-test-bucket')
response = s3.put_object(Bucket='my-test-bucket', Key='my_pretty_folder/' # note the ending "/"

And there you have your bucket.

Again, when you are uploading a file you specify "my-test-bucket/my_file" and what you did there is create a "key" with name "my-test-bucket/my_file" and put the content of your file as its "value".

In this case you have 2 objects in the bucket. First object has null body and looks like a folder, while the second one looks like it is inside that but as @danimal pointed out in reality you created 2 keys in the same flat hierarchy, it just "looks-like" what we are used to seeing in a file system.

If you delete the file, you still have the other objects, so on the aws console, it looks like folder is still there but no files inside.

If you skipped creating the folder and simply uploaded the file like you did, you would still see the folder structure in AWS Console but you have a single object at that point.

When you however list the objects from command line, you would see a single object and if you delete it on the console it looks like folder is gone too.

Upvotes: 4

danimal
danimal

Reputation: 1697

Files ('objects') in S3 are actually stored by their 'Key' (~folders+filename) in a flat structure in a bucket. If you place slashes (/) in your key then S3 represents this to the user as though it is a marker for a folder structure, but those folders don't actually exist in S3, they are just a convenience for the user and allow for the usual folder navigation familiar from most file systems.

So, as your code stands, although it appears you are putting a file called test_audit_log.txt in a folder called BlueConnect, you are actually just placing an object, representing your file, in the us-east-1-tip-s3-stage bucket with a key of BlueConnect/test_audit_log.txt. In order then to (seem to) put it in a new folder, simply make the key whatever the full path to the file should be, for example:

# upload_fileobj(file, bucket, key)
s3.upload_fileobj(f, "us-east-1-tip-s3-stage", "folder1/folder2/test_audit_log.txt")

In this example, the 'key' of the object is folder1/folder2/test_audit_log.txt which you can think of as the file test_audit_log.txt, inside the folder folder1 which is inside the folder folder2 - this is how it will appear on S3, in a folder structure, which will generally be different and separate from your local machine's folder structure.

Upvotes: 3

Related Questions