CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2amazon-s3aws-cli
Dormidont
Dormidont

Reputation: 233

aws s3 cp clobbers files?

Um, not quite sure what to make out of this.
I am trying to download 50 files from S3 to EC2 machine. I ran:

for i in `seq -f "%05g" 51 101`; do (aws s3 cp ${S3_DIR}part-${i}.gz . &); done

A few minutes later, I checked on pgrep -f aws and found 50 processes running. Moreover, all files were created and started to download (large files, so expected to take a while to download).
At the end, however, I got only a subset of files:

$ ls
part-00051.gz  part-00055.gz  part-00058.gz  part-00068.gz  part-00070.gz  part-00074.gz  part-00078.gz  part-00081.gz  part-00087.gz  part-00091.gz  part-00097.gz  part-00099.gz  part-00101.gz
part-00054.gz  part-00056.gz  part-00066.gz  part-00069.gz  part-00071.gz  part-00075.gz  part-00080.gz  part-00084.gz  part-00089.gz  part-00096.gz  part-00098.gz  part-00100.gz

Where is the rest??
I did not see any errors, but I saw these for successfully completed files (and these are the files that are shown in the ls output above):

download: s3://my/path/part-00075.gz to ./part-00075.gz

Upvotes: 1

Views: 188

Answers (1)

John Rotenstein
John Rotenstein

Reputation: 269121

If you are copying many objects to/from S3, you might try the --recursive option to instruct aws-cli to copy multiple objects:

aws s3 cp s3://bucket-name/ . --recursive --exclude "*" --include "part-*.gz"

Upvotes: 2

Related Questions