CODEWITHSUNDEEP
parallel-processingmakefileclearcasegnu-make
ankush
ankush

Reputation: 63

Gnu make on multiple host machine?

In clearmake there is an option to pass host names so that to run multiple jobs on these hosts but in gmake there is no option to pass multiple hosts although multiple jobs can be passed. I want to know how i can mimic this clearmake functionality?

Upvotes: 6

Views: 1346

Answers (4)

Stefano Vespucci
Stefano Vespucci

Reputation: 160

Another alternative is [makeflow] (http://ccl.cse.nd.edu/software/makeflow/). It allows to run a make (simplified) file to many HPC infrastructures (e.g. SLURM, Torque)

Upvotes: 0

MadScientist
MadScientist

Reputation: 101081

The usual answer for people looking to distribute builds run with standard make tools like GNU make across multiple systems is to use something like distcc.

Upvotes: 8

reinierpost
reinierpost

Reputation: 8611

As sfjac writes, GNU Make does not provide any support for distributing processes across machines. So you'd have to set up such support yourself.

One way to do this is by changing the $(SHELL) to something that can take a shell command and execute it on a different host, and also picks the host to use, such that the total work is distributed in a reasonable way. You could write a script that invokes ssh and does some bookkeeping to pick hosts.

Another issue is how to communicate the prerequisites and results across the machines. Let's assume all your Makefile ever does is look at files and create other files depending on their contents.

One option is to put everything on a shared filesystem, but that may produce synchronization issues (depending on how you share the files and how your recipes deal with such issues). Another option is to let each host use a local filesystem, and somehow mirror it (with rsync?) before and after execution of each command (but only the files read or written by the command). How does clearmake solve this problem?

So how easy it is to make this work reliably strongly depends on how complicated your Makefiles and their recipes are.

I notice there exist some so-called distributed shells that may or may not make life easier - I've never used them.

Upvotes: 1

sfjac
sfjac

Reputation: 7304

GNU make can run multiple processes on a single machine (see the -j option) but it is not set up to do parallel makes across multiple machines, which I think is what you are wanting to do. There is a -C gnu option for clearmake that will cause it to emulate GNU make, if you're just after some of GNU make's features (I don't have experience with this option but noticed it in the documentation. YMMV.).

Upvotes: 0

Related Questions