CODEWITHSUNDEEP
amazon-s3amazon-ec2aws-cli
Mo.
Mo.

Reputation: 42493

AWS CLI S3 Copy stopping without any error

I'm trying to download a large csv file (100GB) from S3 into a mounted drive on an EC2 instance.

I have downloaded AWS-CLI and used the command: aws s3 cp s3://bucket/file.csv . --region=us-east-1 and it alls seems to be working great and downloads about 70GB worth of data and then suddenly it just seems to stop and exists without any message or error, deleteing the part of the file it created. 

Completed 642 of 916 part(s) with 1 file(s) remaining

and then it just stops.

Why could this be and how can I get around it? I tried using s3cmd get but it too times out and gets very slow to the point where it almost stops downloading anything.

Any ideas? Network issue? disk issue? API issue?

Thanks

Upvotes: 4

Views: 2816

Answers (1)

Harshavardhana
Harshavardhana

Reputation: 1428

Alternatively you can try https://github.com/minio/mc which implements both resumable downloads and uploads.

Additionally it also provides sessions for your ongoing operation, in case of any errors it gracefully saves the session and you can continue from where it left off.

mc is written in Golang and released under Apache license v2. Please provide us feedback.

Upvotes: 5

Related Questions