CODEWITHSUNDEEP
croncluster-computing
Jonathan
Jonathan

Reputation: 986

Crons for Clusters

Just a quick question that has been bothering me today. I own five servers, all have the exact same image and run behind a load balancer. I want to run a process heavy cron on these servers every half an hour.

I don't want to put the cron on each machine, as it is resource heavy and would block all incoming connections for a good thirty seconds. In addition, I don't really want to put the cron on one machine, just to make sure it is redundant and it will be run.

My possible solutions to this would be to have a remote service that would run the cron, just by way of accessing a URL that would trigger it; I think that would be the most feasible at this point.

I'm really curious as to what other solutions might be available.

Thanks for your time!

Upvotes: 1

Views: 261

Answers (1)

Keith Thompson
Keith Thompson

Reputation: 263257

You could set up staggered cron jobs on your 5 machines, so it runs every 2.5 hours on each of your 5 machines. Probably the cleanest way to do that is to schedule a job to run every 30 minutes, and have the job itself be a script that runs conditionally, depending on the current time and which machine it's on.

Or, if you have some kind of batch scheduling system, you could run a cron job on one system that submits a batch job, letting the scheduling system choose which server to use. This has the advantage that, assuming your batch system works properly, the job should still run if one of your servers is down. You'll likely need to set up some environment variables in your cron job to let it use the batch system properly.

Upvotes: 1

Related Questions