CODEWITHSUNDEEP
c++cmpidistributed-computing
Keith
Keith

Reputation: 131

C/C++ framework for distributed computing (MPI?)

I'm investigating as to whether there is a framework/library that will help me implement a distributed computing system.

I have a master that has a large amount of data split up into files of a few hundred megabytes. The files would be chunked up into ~1MB pieces and distributed to workers for processing. Once initialized, the processing on each worker is dependent on state information obtained from the previous chunk, so workers must stay alive throughout the entire process, and the master needs to be able to send the right chunks to the right workers. One other thing to note is that this system is only a piece of a larger processing chain.

I did a little bit of looking into MPI (specifically Open MPI), but I'm not sure if it is the right fit. It seems to be geared to sending small messages (a few bytes), though I did find some charts that show it's throughput increases with larger files (up to 1/5 MB).

I'm concerned that there might not be a way to maintain the state unless it was constantly sent back and forth in messages. Looking at the structure of some MPI examples, it looked like master (rank 0) and workers (ranks 1-n) were a part of the same piece of and their actions were determined by conditionals. Can I have the workers stay alive (maintaining state) and wait for more messages to arrive?

Now that I'm writing this I'm thinking it would work. The rank 1...n section would just be a loop with a blocking receive followed by the processing code. The state would be maintained in that loop until a "no more data" message was received at which point it would send back the results. I might be beginning to grasp the MPI structure here...

My other question about MPI is how to actually run the code. Remember that this system is part of a larger system, so it needs to be called from some other code. The examples I've seen make use of mpirun, with which you can specify how the number of processors, or a hosts file. Can I get the same behavior by calling my MPI function from other code?

So my question is is MPI the right framework here? Is there something better suited to this task, or am I going to be doing this from scratch?

Upvotes: 2

Views: 4496

Answers (4)

sebhtml
sebhtml

Reputation: 1

RayPlatform is a MPI framework for C++. You need to define plugins for your application (like modules in Linux).

RayPlatform is licensed under the LGPLv3. Link: https://github.com/sebhtml/RayPlatform

It is well documented also.

An example application using RayPlatform: https://github.com/sebhtml/RayPlatform-example

edit: added link

Upvotes: -1

Tom
Tom

Reputation: 5299

If you are getting up and running with a cluster and mpi, then I recommend having a look at boost mpi. Its a c++ wrapper over an underlying mpi library (such as openmpi or mpich2). I found it very useful.

Your idea of sending messages back and forward, with each node requesting a new message when it is finished until a handshake saying "no more messages" is provided sounds a good one. I had a similar idea, and got a simple version up and running. I just put it on github for you in case you want to have a look. https://github.com/thshorrock/mpi_manager. Most of the code is in the header file: https://github.com/thshorrock/mpi_manager/blob/master/include/mpi_manager/mpi_manager.hpp

Note, this was just a bit of code that was used to get me up and running, its not fully documented, and not a final version but its fairly short, works fine for my purposes and should provide a starting point for you.

Upvotes: 2

duedl0r
duedl0r

Reputation: 9424

Have a look at FastFlow. They use a data flow model to process data. It is extremely efficient if this model is suitable for you.

Upvotes: 1

unkulunkulu
unkulunkulu

Reputation: 11912

MPI seems reasonable option for your task. It uses the SPMD architecture, meaning you have the same program executing simultaneously on possibly distributed or even heterogeneous system. So the choice of process with rank 0 being the master and others being the workers is not mandatory, you can choose other patterns.

If you want to provide state for your application, you can use a constantly living MPI application and master process sending commands to worker ones over time. You probably should also consider saving that state to disk in order to provide more robustness to failures.

Running of an MPI process is done initially by mpirun. For example, you create some program program.c, then compile it using mpicc -o program program.c. Then you have to run mpirun -np 20 ./program <params> to run 20 processes. You will have 20 independent processes each having its own rank, so further progress is upon your application. The way these 20 processes will be distributed among nodes/processors is controlled by things like hostfile etc, should look at the documentation more closely.

If you want your code to be reusable, i.e. runnable from another MPI program, you generally should at least learn what MPI Communicator is and how to create/use one. There're articles on the net, keywords being "Creating MPI library".

If the code using your library is not to be in MPI itself, it's no huge problem, your program in MPI is not limited to MPI in communication. It just should communicate inside it's logic through MPI. You can call any program using mpirun, unless it tries calls to MPI library, it won't notice that it's being run under MPI.

Upvotes: 2

Related Questions