Running a Cron job on Hazelcast DurableExecutorService

Question

I am trying to run cron schedules on DurableExecutorService on Hazelcast. My idea is that if one node goes down with its schedule, other nod having backup can pickup and resume the CRON. This is what I am doing

        String cron = "0/5 * * * * *";

        config.setInstanceName(name);
        config.getDurableExecutorConfig("exec").setCapacity(200).setDurability(2).setPoolSize(8);

        HazelcastInstance instance = Hazelcast.newHazelcastInstance(config);
        DurableExecutorService executorService = instance.getDurableExecutorService("exec");

        executorService.executeOnKeyOwner(new EchoTask(name, cron), name);

The I use a Spring CRON scheduler to actually run the CRON job.

public class EchoTask implements Runnable, Serializable {

    private final String msg;
    private final String cronExpression;

    EchoTask(String msg, String cronExpression) {
        this.msg = msg;
        this.cronExpression = cronExpression;
    }
    public void run() {

            ThreadPoolTaskScheduler scheduler = new ThreadPoolTaskScheduler();
            scheduler.initialize();
            scheduler.schedule(new Runnable() {

                @Override
                public void run() {

                    System.out.println("Executing" + msg);

                }
            }, new CronTrigger(cronExpression));

        }
}

Now, I run this twice. So in effect, there are 2 instances running.

1. Executinginstance-1 on one instance
2. Executinginstance-2 on another instance

Now, My understanding is that if I go and kill one instance, lets say 1, then the CRON of node1 should migrate to node2. However, this is not happening.

I do get this message though when I kill the node

INFO: [192.168.122.1]:5707 [dev] [3.9.3] Committing/rolling-back alive transactions of Member [192.168.122.1]:5709 - 26ed879b-8ce5-4d58-832c-28d2df3f7f87, UUID: 26ed879b-8ce5-4d58-832c-28d2df3f7f87

I am sure, I am missing something here. Can someone pls guide?

Edit 1: I verified that for normal tasks this behavior works, but it does not for some reason work for Spring CRON

Edit 2 One doubt I have is, that ThreadPoolTaskScheduler is not serializable for some reason.

Failed to serialize org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler

I suspect, that why it is not getting persisted on the ringbuffer. Any idea how I can make it Serializable. I checked the code and ThreadPoolTaskScheduler already implements Serializable

Answer 1

As answered by Thomas Kountis at https://groups.google.com/forum/#!topic/hazelcast/id8AcvWyR5I

Hi,

As Jaromir said above, you are basically executing this line in your runnable scheduler.schedule(...). Once this instruction is finished, your Spring scheduler is working in the background, but from the DurableExecutor perspective the task is finished. Once a task is finished, it gets replaced by its result, in your case there is no result, so null. If you kill that owner member, the backup is promoted, but there is no task anymore, since we replaced it in the previous step. The backup knows, that the task completed. Does this make sense?

I can't thing of any non-abusive way to accomplish what you are looking for, except from maybe a naive CronSchedulerinside theIScheduledExecutor. Imagine a periodic task, that runs every second or so, and it holds a Map<CronExpression, Runnable> in it. During its run cycle, it checks the expression to assert if there is any runnable ready, and if so, runs it another Executor (Durable or Scheduled). Since this is a periodic task, it will work well with out IScheduledExecutor and it will be durable upon failures.

Hope that helps! We will be looking in adding native support of cron-expressions in IScheduledExecutor in the future.

Thanks

Answer 2

executorService.executeOnKeyOwner(new EchoTask(name, cron), name) will run on the key owner node. If you do not have backups enabled and kill the owner node then Hazelcast has no way of knowing that the key ever existed in cluster, hence no durability.

See the following code:

public class DurableExecutorServiceTest {

    DurableExecutorServiceTest() {
        Config config = new Config();
        config.getDurableExecutorConfig("MyService").setDurability(2).setCapacity(200).setPoolSize(2);

        HazelcastInstance hc = Hazelcast.newHazelcastInstance(config);
        hc.getMap("MyMap").put("Key-1", "Value-1");

        DurableExecutorService service = hc.getDurableExecutorService("MyService");
        service.executeOnKeyOwner(new MyRunnable(), "Key-1");

    }

    public static void main(String[] args) {
        new DurableExecutorServiceTest();
    }
}

class MyRunnable implements Runnable, Serializable {
    public void run() {
        int i = 0;
        while(true) {
            try {
                TimeUnit.SECONDS.sleep(3);
                System.out.println("Printing in Durable executor service: "+i++);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

You first need to fire up one Hazelcast instance before executing this code. When you launch this, it joins the previously running node and the runnable gets executed on the owner node of the key. Now kill the node that is printing the message and see if the other remaining node picks up the runnable.

If you are seeing something else in your setup then you might want to dig into your Spring cron jobs.