CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
nicq
nicq

Reputation: 2304

S3: how to upload large number of files

I have to upload 100k small files (Total size: 200MB).

I've tried to do that via web browser (AWS Console), but for the first 15 min, i've uploaded only 2MB.

What is the fastest way to upload 100k small files to S3?

Upvotes: 19

Views: 16801

Answers (3)

Melissa
Melissa

Reputation: 765

On MacOS, a command line tool that worked for me to sync a folder with more than 600 000 files (totalling 4 GB) is s4cmd:

brew install s4cmd
s4cmd sync /path/to/local/folder s3://your-s3-bucket/path

I was not able to use s3-parallel-put, because this tool is Linux-only.

I normally use aws s3 sync or aws s3 cp, but these only resulted in heavy system resource utilization without any visible progress.

Upvotes: 1

Synesso
Synesso

Reputation: 38958

I have 99,561 small files totalling 466MB and experimented with uploading these to S3 as quickly as possible from my m4.16xlarge (64 CPU) EC2 instance.

  • aws s3 sync took 10 minutes
  • CyberDuck promised to take 17 minutes, but hung. I terminated it after 45 minutes.
  • CloudBerry seems to be windows only at first glance, so I did not test it.
  • s3-parallel-put --put=stupid took 9 minutes
  • s3-parallel-put --put=stupid --processes=64 took 1 minute
  • s3-parallel-put --put=stupid --processes=256 took 19 seconds
  • s3-parallel-put --put=stupid --processes=512 took 18 seconds

I'll be moving forward with a s3-parallel-put solution.

Upvotes: 27

Mark B
Mark B

Reputation: 200446

Install the AWS CLI tool, then use the aws s3 cp command or the aws s3 sync command to upload all your files to S3.

Alternatively you could look into using third-party S3 clients such as Cyberduck and CloudBerry Explorer.

Upvotes: 14

Related Questions