How to create folder on S3 from Ec2 instance

Question

I want to create folder in S3 bucket from Ec2 instacne . I tried the put object but its not working . Is there any way of creating folder on s3 from ec2 instace using cli.

Answer 1

You don't need to create a folder to put item in it. For example, just run something like the below command and s3 will create the folders if they don't exist

aws s3 cp ./testfile s3://yourBucketName/any/path/you/like

If you want to use cp recursively you can specify --recursive option, or use "aws s3 sync".

If your command does not work, then you may have permission issues. Paste your error so that we can help you.

Note: "empty folders" in the console are actually "0-byte object with a key that's set to the folder name that you provided" (from the doc

Answer 2

There is no cli command that allows you to simply create a folder in an s3 bucket. To create this folder I would use the following command, which creates an empty file, with nothing inside. But if you delete the file you will delete the folder as long as you have not added anything else afterwards

aws s3api put-object --bucket bucket_name --key folder_name/empty.csv

Answer 3

aws s3 sync <folder_name> s3://<you-prefix>/<some_other_folder>/<folder_name>

Answer 4

aws s3api put-object --bucket bucketname --key foldername/  

This command works like a charm. Courtesy AWS Support.

Answer 5

And bare in mind that, S3 is an object store. It doesn't deal with folder.

If you create /xyz/ and upload a file call /xyz/foo.txt , those are actually 2 different object. if you delete /xyz/ , it will not delete /xyz/foo.txt.

S3 console allow you to "create folder", but after you play with it, you will notice , you CANNOT RENAME folder, or do ANYTHING that you can play with a folder (like moving a tree structure, recursively specify access rights)

In S3, there is something call "PREFIX" where the API allow you to list/filter file with particular "prefix", that let you deal with abstraction.

As mentioned above, since you CANNOT do anything like a file system folder, if you want to perform task like moving one folder to another folder, You need to write your own code to "rewrite" the file name(To be specific, it is "Key" in S3) , i.e. copy it to new object name and delete the old object.

If you want build advance control on S3, you may choose any of the AWS SDK to do it. https://aws.amazon.com/tools/

You can play around with the API function call put_object() (naming varied depends on SDK language) and proof those facts (which most is found inside AWS documentation)

update: Since @Tom raise up the issues.

You cannot create an virtual folder using AWS cli (Maybe @Tom can show how), only ways to do that is using AWS SDK put_object()

Let's try this First I create dummy file in shell

echo "dummy">test.txt

Then try use python aws sdk

import boto3 
s3=boto3.client("s3")
s3.create_bucket(Bucket="dummy")
# now create so call xyz/ "empty virtual folder"
s3.put_object(Bucket="dummy", Key="xyz/")
# now I put above file name to S3 , call xyz/test.txt
# First I must open the file, because put_object only take bytes or file object
myfile=open("test.txt")
s3.put_object(Bucket="dummy", Key="xyz/test.txt")

Now, go to your command shell, fire up your AWS CLI (or continue to play with boto3)

# check everything
aws s3 ls s3://dummy --recursive

#now delete the so call "folder" 
aws s3 rm s3://dummy/xyz/

# And you see the file "xyz/test.txt" is still there 
aws s3 s3://dummy --recursive

Answer 6

You can find the commands here from official blog of AWS:

http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html

And there are different other tools available which can be used to create Bucket/ folders in S3. One of the known tool is S3Browser which is available for windows servers. Install it on your EC2 instance and provide your AWS access key and secret keys to access the S3. This tool provide simple UI to do that.