How to process time-consuming tasks in a distributed way?

Question

We need to process tasks that take a lot of time (parsing huge xml files and inserting the data into db) by multiple nodes. The nodes will not be many, we are even going to start with a single node.

The files are going to be read from an FTP. The job will be scheduled and will happen once a day. What would be a good and easy way to distribute the processing?

My current draft thoughts are:

• use hazelcast distributed ConcurrentMap - it handles the synchronization of the map behind the scene
• each node takes the whole list of files and shuffles it
• then each node iterates the list, using .putIfAbsent(..) - if the file is not in the map, process it. If it is in the map, it means another node is processing it, so try the next file.

That way:

• nodes will never be idle
• if a node fails, other nodes will still process everything (just slower)
• since the files are big and it takes a lot of time to process them, there will be little contention overhead in the .putIfAbsent(..) call and the underlying synchronization (shuffling is meant to improve this as well)
• it requires minimal configuration and does not require installing and managing additional services (like a message queue)

I'm not sure if this is the best approach though. Is it OK? What can be improved? Is there a better one?

Answer 1

Based on your comments I would suggest considering using JMS, (like ActiveMQ which I found the simplest to use/develop with)

It can be stand-alone, redundant and/or embedded.

You can add messages to a Queue and consume from any number of nodes. With auto-commit turned off a failing nodes messages are returned to the queue automatically.