CODEWITHSUNDEEP
linuxamazon-web-servicesamazon-ec2amazon-s3
hdf
hdf

Reputation: 255

Uploading a huge file from ec2 to s3 fails

I'm trying to upload 160 Gb file from ec2 to s3 using

s3cmd put --continue-put FILE s3://bucket/FILE

but every time uploading interrupts with the message:

FILE -> s3://bucket/FILE [part 10001 of 10538, 15MB] 8192 of 15728640 0% in 1s 6.01 kB/s  failed

ERROR: Upload of 'FILE' part 10001 failed. Aborting multipart upload.
ERROR: Upload of 'FILE' failed too many times. Skipping that file.

The target bucket does exist.

What is the issue's reason?

Are there any other ways to upload the file?

Thanks.

Upvotes: 1

Views: 560

Answers (2)

Julio Faerman
Julio Faerman

Reputation: 13501

You can have up to 10000 upload parts per object, so it fails on part 10001. Using larger parts may solve the issue.

Upvotes: 2

ericzma
ericzma

Reputation: 763

"huge"---is it 10s or 100s of GBs? s3 limits the object size to 5GB and uploading may fail if it exceeds the size limitation.

Upvotes: 0

Related Questions