We generate pdf files with data regarding monthly financial balance of tens of thousands of clients. At it's peak (100.000 files at the end of year), the process may take as long as 5 days to complete using distribute the load between 5 servers. The distribution of workload is a manual process (eg. server 1 generates pdf for clients 1 to 20.000, server 2 from 20.001 to 40.000, and so on). We use Java, so we would like to use a Java tool or framework in a fashion similar to BOINC (BOINC is not coded in Java), which would allow us to have: a server automatically distributing data and code to workers, and managing them workers (plain dual core desktops using free (or not so) cpu cycles) receiving code sent by server and executing it using the data as input, giving some status feedback, and finally doing something with the output, eg. sending pdf generated files to network storage. For now, we don't want to try heavyweight all-powerful toolkits such as Globus Toolkit. ¿Do you know any java framework/tools suitable to our needs? Open source would be nice, but we don't discard commercial solutions.

There are many: http://www.jppf.org/ https://jxta.dev.java.net/ https://java.net/projects/cajo/pages/Home http://www.dmoz.org/Computers/Programming/Languages/Java/Class_Libraries/Networking_and_Distributed_Computing/ Google is your best friend here.

javaframeworksparallel-processingdistributed

user58312

Reputation:

Java framework/tool for simple distributed computing problem

We generate pdf files with data regarding monthly financial balance of tens of thousands of clients. At it's peak (100.000 files at the end of year), the process may take as long as 5 days to complete using distribute the load between 5 servers. The distribution of workload is a manual process (eg. server 1 generates pdf for clients 1 to 20.000, server 2 from 20.001 to 40.000, and so on).

We use Java, so we would like to use a Java tool or framework in a fashion similar to BOINC (BOINC is not coded in Java), which would allow us to have:

a server automatically distributing data and code to workers, and managing them
workers (plain dual core desktops using free (or not so) cpu cycles) receiving code sent by server and executing it using the data as input, giving some status feedback, and finally doing something with the output, eg. sending pdf generated files to network storage.

For now, we don't want to try heavyweight all-powerful toolkits such as Globus Toolkit.

¿Do you know any java framework/tools suitable to our needs? Open source would be nice, but we don't discard commercial solutions.

Upvotes: 4