CODEWITHSUNDEEP
amazon-web-servicesamazon-s3amazon-ec2
unknownid
unknownid

Reputation: 105

Setting up AWS for data processing S3 or EBS?

Hey there I am new to AWS and trying to piece together the best way to do this.

I have thousands of photos I'd like to upload and process on AWS. The software is Agisoft Photoscan and is run in stages. So for the first stage i'd like to use an instance that is geared towards CPU/Memory usage and the second stage geared towards GPU/Memory.

What is the best way to do this? Do I create a new volume for each project in EC2 and attach that volume to each instance when I need to? I see people saying to use S3, do I just create a bucket for each project and then attach the bucket to my instances?

Sorry for the basic questions, the more I read the more questions I seem to have,

Upvotes: 2

Views: 300

Answers (2)

Kumaresh Babu N S
Kumaresh Babu N S

Reputation: 1708

You can go with AWS S3 to upload photos. AWS S3 is similar like Google Drive.

If you want to use AWS EBS volumes instead of S3. The problem you may face is,

  • EBS volumes is accessible within availability zone but not within region also means you have to create snapshots to transfer another availability zone. But S3 is global.
  • EBS volumes are not designed for storing multimedia files. It is like hard drive. Once you launch an EC2 instance need to attach EBS volumes.

As per best practice, you use AWS S3.

Based on your case view, you can create bucket for each project or you can use single bucket with multiple folders to identify the projects.

Create an AWS IAM role with S3 access permission and attach it to EC2 instance. No need of using AWS Credentials in the project. EC2 instance will use role to access S3 and role doesn't have permanent credentials, it will keep rotating it.

Upvotes: 1

E.J. Brennan
E.J. Brennan

Reputation: 46879

I'd recommend starting with s3 and seeing if it works - will be cheaper and easier to setup. Switch to EBS volumes if you need to, but I doubt you will need to.

You could create a bucket for each project, or you could just create a bucket a segregate the images based on the file-name prefix (i.e. project1-image001.jpg).

You don't 'attach' buckets to EC2, but you should assign an IAM role to the instances as you create them, and then you can grant that IAM role permissions to access the S3 bucket(s) of your choice.

Since you don't have a lot of AWS experience, keep things simple, and using S3 is about as simple as it gets.

Upvotes: 2

Related Questions