CODEWITHSUNDEEP
google-app-enginegoogle-cloud-platformapp-engine-flexible
billDickens
billDickens

Reputation: 111

Google Serverless function - Can it run a process or just a single program?

We have a shell script that runs a combination of java and python programs through multiple steps. In AWS, when we need to run more than one concurrently, we send a parameter set, and a request to the VMs all prebuilt (serverlist is CV-1, CV-2,CV-3 etc) with this in mind. Several of these processes can be running at once, they don't have any inter-dependencies except the database updates. We have had up to 16 VM running at the same time with no issues, with one controlling instance that just keeps track of when each one is done.

We are looking for still more scalability. It would be nice if we didn't need to have the 16 machines "running" and waiting for something to kick them off. We would like to be able to kick off as many as we need, but equally interested in NOT paying for them when they are idle. They are idle more than 80% of the time. But once we need them, we need them fast as some of these can take a while to finish. The machines are all the same size. That may be a problem also.

I was working through the google server less examples, and while it seems to be what I need, it isn't completely clear if it works the way I was hoping. I am hoping that I can put all of the programs (python,Java, shell) in a container and instantiate an instance of the machine on the fly. I only need to pass it 3 parameters and that looks easy enough.

Where I am getting confused, is what it is capable of running. I have yet to see an example where it kicks off a shell script or a python program. Lots of Java "hello world" which for my level of expertise right now is fine, but I don't see anything that runs a shell script. Even a python "script" would be acceptable and might even be better.

Each script runs between 10 min and 5 hours depending on size of the input. So a startup of 4-5 min is not going to bother us. Most are in the 20 min range, about 5% are in the 4+hour range.

The questions is really: Is this approach going to work with Google serverless ? If its a no, then the rest of these don't matter overmuch !

Thanks

Upvotes: 0

Views: 101

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75735

The Serverless Compute products (Cloud Run, Cloud Function, App Engine) answer to HTTP request. So you need to expose an HTTP server to use these services. No background processing is possible, and you are limited in time

  • 9 minutes with Cloud Functions
  • 60 minutes with Cloud Run
  • 24h with App Engine standard with basic or manual scalling + Cloud Task

If you need to run background processes, or no https processes, you can cheat with Cloud Build (use your container and run the command that you want in a fake Build) or you can use AI Platform custom training to run your container for hours.

This latest solution is the only one that support GPUs capacity.

You will need to update or redesign your processes to fit this model. I think the portability isn't immediate according to your requirements.

Upvotes: 1

Related Questions