CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2
LearningSlowly
LearningSlowly

Reputation: 9451

AWS S3 Sync function - missing files

I am transferring files from an S3 bucket to an EC2 instance.

Using the AWS CLI, I can see that there are 219 files of ~43GB total size in the S3 bucket in question

[IN]
    aws s3api list-objects --bucket resultsdata --output json --query "[sum(Contents[].Size), length(Contents[])]"

[OUT]

[
    43367899797,
    219
]

I have been using the AWS S3 sync command to transfer the files from the bucket to the EC2 machine.

When I run this I successfully see the downloading of the individual files within the bucket. However - it appears to end prematurly.

Completed 706 of 1550 part(s) with 36 file(s) remaining

When I go to the download location and carry out a count of files

ind /home/ec2-user -type f | wc -l

I get 198 files and not 219.

I have run the AWS sync command multiple times

aws s3 sync s3://MYBUCKETNAME .

What am I missing?

Edit

I have ruled out there being duplications in the S3 bucket

Edit 2

Deleting the files from ec2 and trying to use cp - 

aws s3 cp s3://mybucket . --recursive

I get 

Completed 4286 of 5084 part(s) with 34 file(s) remaining

But - when I count the files I get a different number again, 186.

So sync gives me 198, cp gives me 186. Both are short of the 219 files in the S3 bucket. Anyone?!

Upvotes: 4

Views: 2335

Answers (1)

LearningSlowly
LearningSlowly

Reputation: 9451

I had no space left on the device!

I would have expected an error message in this case. A new mounted volume has solved the issue

Upvotes: 3

Related Questions