Run long java processes on demand with AWS

Question

I want to process a large csv (millions of lines) with a Java application on AWS, and write the results in another csv.

The application is packaged in a single jar and can be run with some shell command java -jar myJar.jar -option1 -option2.

The application could be called anytime depending on a user uploading a csv, triggering the application.

• What I thought of first was to upload the csv on s3, and having a lambda function triggering every time it is uploaded. This lambda runs the jar (also stored on s3), and then writes the results on s3.

Problem : It works for small files, but lambda functions are limited in execution time, RAM, CPU and temp file storage. They actually are made for short processes.

• The other solution is to have an ec2 cluster running, with an app waiting for a trigger.

Problem : Having a cluster running, even when idle, means paying for it.

Is there a solution to run this jar without having coding its equivalent in a custom AWS technology?

EDIT : To answer the comments

• The "real-time" need is that it would be meant to be triggered by end users and they would be waiting for the results to be displayed. A few minutes is fine thought, and I'll try the ec2 solution suggested by @stdunbar to see how it goes.
• I know that 1M is small, and the lambda/s3 solution described above works well on a 5M-lines csv. But I am looking for some solution that scales well, that would be used when running the jar locally will be a matter of hours and that we could not afford to run it locally anymore. (I am still at an experimentation stage here, and a complete beginner on cloud solutions)

Answer 1

There are multiple part where you can make it work more efficiently and saves money.

Require coding:

• If it's not absolutely necessary to process the whole 1M lines at once/together , try to break it to smaller pieces.
• Write an Lambda that react to CSV creation, the Lambda spawn EC2s on your behalf and send the jobs to EC2 to process ( Quite the configuration works needed, I believe )

Less coding required:

• You wouldn't need to have the whole EC2 cluster running, just one small instance and scale up as the workload raise, same with the solution below
• Go for Elastic Beanstalk. They do the auto scaling for you, you just upload the .jar

Note that the biggest Lambda is quite powerful, at the moment, 3000MB RAM with equivalent CPU power and it gives you 15 minutes to do a task. To keep one T2.Medium ( 4Gb RAM, 2vCore) running 24/7 a month would cost you ~ $38

Or Both:

• You can keep a sleeping/stopped EC2 instance that would cost you a fraction of the idle/waiting instance. Lambda can start the instance. Auto-Scale will scale the amount of instances up and down. Cloudwatch can put the final instance back to "Stopped" after some period of CPU idling