CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-s3amazon-ec2
connor449
connor449

Reputation: 1679

How to run code on aws ec2 with local input and output

I have code on aws ec2. Right now, it accepts input and output files from s3. Its an inefficient process. I have to upload input file to s3, copy s3 to ec2, run program, copy output files from ec2 to s3, then download locally.

Is there a way to run the code on ec2 and accept a local file as input and then have the output saved on my local machine?

Upvotes: 0

Views: 1377

Answers (2)

Markonick
Markonick

Reputation: 328

@John Rotenstein we have solved the problem of loading 60MB+ models into Lambdas by attaching AWS EFS volumes via VPC. Also solves the problem with large libs such as Tensorflow, opencv etc. Basically lambda layers almost become redundant and you can really sit back and relax, this saved us days if not weeks of tweaking, building and cherry picking library components from source allowing us to concentrate on the real problem. Beats loading from S3 everytime too. The EFS approach would require an ec2 instance obviously.

Upvotes: 0

John Rotenstein
John Rotenstein

Reputation: 270134

It appears that your scenario is:

  • Some software on an Amazon EC2 instance is used to process data on the local disk
  • You are manually transferring that data to/from the instance via Amazon S3

An Amazon EC2 instance is just like any other computer. It runs the same operating system and the same software as you would on a server in your company. However, it does benefit from being in the cloud in that it has easy access to other services (such as Amazon S3) and resources can be turned off to save expense.

Optimize current process

In sticking with the current process, you could improve it with some simple automation:

  • Upload your data to Amazon S3 via an AWS Command-Line Interface (CLI) command, such as: aws s3 cp file.txt s3://my-bucket/input/
  • Execute a script on the EC2 process that will:
  • Download the file, eg: aws s3 cp s3://my-bucket/input/file.txt .
  • Process the file
  • Copy the results to S3, eg: aws s3 cp file.txt s3://my-bucket/output/
  • Download the results to your own computer, eg: aws s3 cp s3://my-bucket/output/file.txt .

Use scp to copy files

Assuming that you are connect to a Linux instance, you could automate via:

  • Use scp to copy the file to the EC2 instance (which is very similar to the SSH command)
  • Use ssh with a [remote command(https://malcontentcomics.com/systemsboy/2006/07/send-remote-commands-via-ssh.html) parameter to trigger the remote process
  • Use scp to copy the file down once complete

Re-architect to use AWS Lambda

If the job that runs on the data is suitable for being run as an AWS Lambda function, then the flow would be:

  • Upload the data to Amazon S3
  • This automatically triggers the Lambda function, which processes the data and stores the result
  • Download the result from Amazon S3

Please note that an AWS Lambda function runs for a maximum of 15 minutes and has a limit of 512MB of temporary disk space. (This can be expanded by using Amazon EFS is needed.)

Something in-between

There are other ways to upload/download data, such as running a web server on the EC2 instance and interacting via a web browser, or using AWS Systems Manager Run Command to trigger the process on the EC2 instance. Such a choice would be based on how much you are permitted to modify what is running on the instance and your technical capabilities.

Upvotes: 2

Related Questions