CODEWITHSUNDEEP
dockergoogle-cloud-platformcloud
me--
me--

Reputation: 2148

Scheduled, auto-scaled jobs in Google Cloud Platform

I'm kinda new to cloud development, so go easy on me. I'm not even sure what I'm asking is possible.

I'm trying to figure out how to schedule a job (every hour or two probably) to execute a worker process that will be packaged inside a custom Docker image. The worker process will check for any "scheduled work" it needs to perform (as requested by users of the application) by connecting to the application's data store, then execute any outstanding work. Once done, I want everything to tear down until the next schedule interval, so I'm not incurring any costs during the down time.

One complication is that the scheduled work will be of different types. Ideally, I want to spin up one instance of my image per type, passing that type into the worker process, so it knows what type of scheduled work it is dealing with. Once it's done with its type of scheduled work, it can exit, even if other containers are still running their scheduled work.

Summary of requirements

Things I've tried/considered

K8s

Seems to require that the cluster be running at all times. It's also unclear to me how I would have different nodes in a cluster responsible for different types of scheduled work.

AppEngine and Cloud Scheduler

I'd need to use the Flexible environment because of my bespoke Docker image. According to this, that means I have to leave at least one instance running at all times.

Again, I'm not sure how I would scale it such that different instances deal with different types of scheduled work.

Questions

So, I have a few questions about all this:

  1. First and foremost, is this even possible with GCP? My research thus far suggests it is not.
  2. What would be the best approach to getting as close to my requirements as possible?

Upvotes: 0

Views: 297

Answers (1)

David Maze
David Maze

Reputation: 158908

In principle the workload is a good match for Kubernetes. You can set up a Kubernetes CronJob to run each kind of worker once an hour; you'd create a separate CronJob with separate environment variables or command-line parameters for each kind of workload; provided you have a Docker image registry (like GCR) you can run any custom-built Docker image.

The one trick here is that you're talking about scaling containers up and down, but on GKE you're paying for nodes. In GKE the cluster autoscaler will automatically create and remove nodes for you. The FAQ sounds like scaling up quickly is a little more important to it than scaling down, with a target of getting enough capacity to start everything within 60 seconds. It will scale down nodes that are less than 50% in use for 10 minutes.

If your scheduled jobs are a large fraction of your total workload, then perhaps this would result in nodes being spun up and spun down with perhaps 50% average utilization, with a new ramp-up every hour. This turns out to meet your billing requirements too (or at least be better than leaving the whole cluster running all the time). The GKE pricing documentation says:

GKE uses Google Compute Engine instances for nodes in the cluster. You are billed for each of those instances according to Compute Engine's pricing, until the nodes are deleted. Compute Engine resources are billed on a per-second basis with a 1 minute minimum usage cost.

If cost is the primary driver of this, the optimal situation is one where you never have an idle node. The most straightforward way to approach this would be to spin up a dedicated GCE instance for each task and then destroy it when the task finishes. GCE has support for indirectly running containers on instances which could be a good match for your task; you'd have to provide your own job scheduler and give it the ability to purchase and terminate GCE instances. That then becomes a tradeoff for how much money you want to spend developing a custom solution, vs. how much money you want to spend on cloud resources.

Upvotes: 2

Related Questions