CODEWITHSUNDEEP
spring-bootspring-scheduled
P.Das
P.Das

Reputation: 53

Spring and scheduled tasks on different Data Centers

I have one spring scheduler , which I will be deploying in 2 different data center. My data centers will be in active and passive mode. I am looking for a mechanism where passive data center scheduler start working where that data center become active . We can do it using manually changing some configurations to true/false but , I am looking for a automated process.

-Initial state:

Data center A active - Scheduler M is running.
Data center B passive - Scheduler M is turned off.

-May be after 3 days.

Data center A passive - Scheduler M turned off.
Data center B active - Scheduler M is starting

Upvotes: 0

Views: 641

Answers (1)

Julian
Julian

Reputation: 4055

I don't know your business requirements but unless you want multiple instances running but only one active, the purpose you will have a load balancer would be to spread the load to multiple instances of the same application rather to stick with only one instance.

Anyway I think an easy way of doing this without using a very sophisticated mechanism (coming with a lot of complexity depending where you run your application) would be this:

  1. Have shared location such as a semaphore table in your database storing the ID of the application instance owning the scheduler process
  2. Have a timeout set for each task. Say if the scheduler is supposed to run every two minutes set the timeout to two minutes.
  3. Have your schedulers always kick off on all application instances
  4. Once the tasks kicks off first check if it is the one owning the processing. If yes do the work, if not go at point 7.
  5. After doing the work record the time stamp of the task completion in the semaphore table
  6. Wait for the time to pass for the next kick off
  7. If not the one owning the processing check when the task last run in the semaphore table. If the time since last run is greater than the timeout set for that process take the ownership of the process (recording your application instance id in the semaphore table)

We applied this and it ran very well with one of our applications. In reality it was much more complex than explained above as we had a lot of application instances and we had to avoid starting an ownership battle between them. To address this we put in place a "Permission to process request" concept so no matter how many instances wanted to take control it was only one which was granted.

For another application with similar requirements we used a much much easier way to achieve this but the price we paid was some extra learning curve in using ILock from Hazelcast IMGB framework. That is really very easy but keep in mind the Hazelcat community edition comes with absolutely no security and paying for a Hazelcast license just to achieve this may be a bit of expense.

Again all depends on you use case, for us the semaphore table was good enough in first scenario but prove bad in the second one as the multiple processes trying to update the same table at the same time ended up with a lot of database contention which took us to Hazelcast.

Other ideas would be a custom health check implementation that could trigger activating one scheduler or the other depending of response received.

Hope that helps, just ideas from our experience. Good luck.

Upvotes: 2

Related Questions