CODEWITHSUNDEEP
amazon-web-servicesamazon-s3dataset
user11184680
user11184680

Reputation:

Is there a way to expedite the uploading of files on AWS S3?

I am trying to use AWS S3 and read the data to Jupyter Notebook. The total file size is 42 MB, however the time it is taking to get into S3 is very high. After 5 hours, only 22% has been completed and the estimated time to complete is 12 hours. Are there any other ways to either effectively upload to S3, or use other platform that possibly provides higher speed?

AWS Uploading Progress

Upvotes: 0

Views: 99

Answers (1)

Maurice
Maurice

Reputation: 13108

The upload bandwidth is determined by many factors:

  • Your local internet connection
  • Any VPN the traffic goes through
  • The public internet
  • S3-Bandwidth

Typically the last two aren't your problem (especially for 42MB), but the first two may be.

If you upload data to an S3-Bucket in a region that's far away from you, you can take a look at S3 Transfer-Acceleration, which allows you to send the data to the nearest CloudFront edge location, from where it traverses the global AWS backbone to the destination region, but I doubt this is going to help you much with the problem given the size of data, as it's most likely one of the first two.

Upvotes: 1

Related Questions