CODEWITHSUNDEEP
amazon-web-servicesamazon-s3
volume one
volume one

Reputation: 7563

How to improve slow uploading to Amazon S3

I am in the process of converting a website to upload user submitted photos and videos directly to Amazon S3 instead of to the local server.

When I worked with small image files, it seemed fine... but now that I try with larger images say over 1MB, the upload speed is too slow and unusable.

What are the common strategies to overcome this? Do websites like YouTube first upload to their local server and then move to Google Cloud or something? If I have thousands of users uploading at the same time, my local server won't be able to store so many files hence I was moving to S3 before realising how slow it was.

Any suggestions please?

Upvotes: 32

Views: 69815

Answers (2)

bittap
bittap

Reputation: 548

In my case, Because Too many log is printed in console,The Amazon S3 upload was very slow. A 36MB file took for uploading 3 minute.

2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "[0x98][\r][0xd3][0xe][0xf9][0xe][0x3][0xe][0xdf][0x9][0xde][0x5]0[0x4]F[0x1][0xaa][0xfd][0xb0][0xfa][0x8d][0xf8][0xfa][0xf9][0xde][0xfa][0xb7][0xf9]E[0xfa][0xf][0xfa]*[0xfb][0xce][0xfb][0xc4][0xfa][0xed][0xf9]X[0xfa][0xee][0xfb][0xfa][0xfc][0x8a][0xfd][0x10][0xff][0x94][0xff]:[0x1]L[0x5]$[0x8]Z[\n]" [org.apache.http.impl.conn.Wire:73]
2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "#[0xc]&[0xb][0xac][0x9]g[0x9]+[0x9][0xae][0x7][0xf2][0x5][0x11][0x6]8[0x5]<[0x3][0x13][0x0][0x9d][0xfe]7[0xff]%[0xfe][0xc2][0xfb]J[0xfa][0xac][0xf9][0x8b][0xf7]8[0xf4][0xc1][0xf1][0xdd][0xf0][[0xf1][0xb7][0xf3]/[0xf5]/[0xf7][0x8a][0xfb]J[0x2][0xe4][0x7][0xd1][\n]" [org.apache.http.impl.conn.Wire:73]
2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "[0xc2][0xc][0x99][\r],[\r][[0xe]T[\r][0x80][0x9][0x10][0x7][0xad][0x2]][0xfd][0x8f][0xf8]([0xf6]^[0xf4][0x96][0xf2]?[0xf0]D[0xef][0xb2][0xf0][0xac][0xf2][0xb3][0xf3][0xbc][0xf4],[0xf8]l[0xfb]l[0xff][0xfa][0x3][0x82][0x7][0xd8][\n]" [org.apache.http.impl.conn.Wire:73]
2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "[0xb3][\r][0xb7][\r][[\r]=[\r][0xf7][0xb][0xb3][\n]" [org.apache.http.impl.conn.Wire:73]
2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "([0xb][0x9c][\n]" [org.apache.http.impl.conn.Wire:73]
2022-07-12T12:58:48 DEBUG - http-outgoing-2 >> "5[\n]" [org.apache.http.impl.conn.Wire:73]
....

Therefore, I turn off the log in logback.xml

<logger name="org.apache.http.wire" level="WARN"/>

and then The 36MB file took for uploading from 3 minute to 9 seconds.

Upvotes: 5

Mark B
Mark B

Reputation: 200562

I would take the following steps:

  • Enable Transfer Acceleration on your S3 bucket.

  • Change your application to upload files in multiple parts, using S3 Multipart Upload, and use multi-threading to upload more than one part at a time.

Upvotes: 38

Related Questions