CODEWITHSUNDEEP
multithreadingmakefilethreadpoolgnu-make
TPPZ
TPPZ

Reputation: 4891

Makefile target running parallel targets and waiting for them to complete before doing more stuff

I am trying to have a "parent target" waiting for "parallel children targets" to complete before doing something in the "body" of the "parent target".

In the following example the message @echo "Print this only after *ALL* the previous PARALLEL tasks are done!" should be printed only after all the targets foo-task1, foo-task2 and foo-task3 have completed.

If I remove the ampersand (&) at the end of those "children targets" then they become serial, but that's not the expected behavior. They should behave similar to a "thread pool" or "process pool". I don't know in advance which of those targets will complete last, so I can not remove the ampersand (&) from the bottleneck target.

Those targets represent HTTP calls to a remote REST API that could require variable amounts of time to complete. I would like to wait for these "parallel REST calls" to see if they succeeded before printing the final message in foo-parallel.

These "parallel targets" are returning stdout/stderr so I wish the parent target could wait for them and fail in case there are some errors.

How could I do this?

foo-task1:
    @bash -c 'sleep 1 && echo "task1" &'

foo-task2:
    @bash -c 'sleep 1 && echo "task2" &'

foo-task3:
    @bash -c 'sleep 2 && echo "task3" &'

foo-parallel: foo-task3 foo-task2 foo-task1
    @echo "Print this only after *ALL* the previous PARALLEL tasks are done!"

The command make version:

$ make --version
GNU Make 4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

edit:

This slightly different target definition (I changed the sleep times just to make sure the messages print in the delayed order rather than the order in the code):

foo-parallel:
    @bash -c 'sleep 5 && echo "task1" & \
                        sleep 1 && echo "task2" & \
                        sleep 2 && echo "task3" & \
                        wait; \
                        echo "done here!"'

prints:

$ make foo-parallel 
task2
task3
task1
done here!

but I need to put the logic of all those task1, task2 and task3 inside the same target. I would like to split those tasks into different targets.

Upvotes: 1

Views: 5115

Answers (2)

MadScientist
MadScientist

Reputation: 100781

You cannot do this, without using make's -j option. By default make builds one target, waits for it to complete, then builds the next one. Only with -j will it support multiple targets running at the same time.

foo-parallel: foo-task3 foo-task2 foo-task1
        @echo "Print this only after *ALL* the previous PARALLEL tasks are done!"

foo-task1:
        @sleep 1 && echo "task1"

foo-task2:
        @sleep 1 && echo "task2"

foo-task3:
        @sleep 2 && echo "task3"

Then run: make -j Or if you want to restrict the number of parallel builds, use make -j2 or however many you want.

If you REALLY want to avoid typing the -j you can use recursive make:

foo-parallel:
        @$(MAKE) -j foo-task1 foo-task2 foo-task3
        @echo "Print this only after *ALL* the previous PARALLEL tasks are done!"

foo-task1:
        @sleep 1 && echo "task1"

foo-task2:
        @sleep 1 && echo "task2"

foo-task3:
        @sleep 2 && echo "task3"

However this might cause odd warnings if someone does invoke make -j. Generally I think it's better to leave it up to the person invoking make as to how much parallelism they want, for their particular system.

Upvotes: 4

uzsolt
uzsolt

Reputation: 6027

I think you want to use the -j option:

foo-task1:
    @sleep 1
    @echo "task1"

foo-task2:
    @sleep 1
    @echo "task2"

foo-task3:
    @sleep 2
    @echo "task3"

foo-parallel: foo-task3 foo-task2 foo-task1
    @echo "Print this only after *ALL* the previous PARALLEL tasks are done!"

And run make -j3 foo-parallel - the make will run parallel with correct dependency. See more Parallel Execution section:

GNU make knows how to execute several recipes at once. Normally, make will execute only one recipe at a time, waiting for it to finish before executing the next. However, the ‘-j’ or ‘--jobs’ option tells make to execute many recipes simultaneously.

Upvotes: 2

Related Questions