CODEWITHSUNDEEP
makefilegnu-make
Ani
Ani

Reputation: 1526

sub make with foreach fails randomly (with higher J factor)

I am using GNU make, where I have a top level makefile, which invokes another makefile, for different types of builds, like:

LIST_OF_TYPES: 32 64 ...

tgt-name: deps
          $(foreach i,$(LIST_OF_TYPES), \
            $(MAKE) -f $(MY_MAKEFILE) ARCH=$i mylib;)

when running with higher j factor like -j100 etc, one of the build fails, but the return value is still 0 so, I cannot make out if the build really did work! Is there anything wrong with the way I'm using the foreach construct? or its just the higher j with foreach which is causing problems?

Upvotes: 0

Views: 704

Answers (2)

bobbogo
bobbogo

Reputation: 15493

The top-level makefile you have listed is safe under -j (though badly sub-optimal). After the expansion of the $(foreach...), make effectively sees:

tgt-name: deps
    $(MAKE) -f $(MY_MAKEFILE) ARCH=32 mylib; $(MAKE) -f $(MY_MAKEFILE) ARCH=64 mylib; ...

When one of these sub-makes fails (due to mis-handling of -j), the failure is not reported to the top level make. You need to use something like:

tgt-name: deps
    $(MAKE) -f $(MY_MAKEFILE) ARCH=32 mylib && $(MAKE) -f $(MY_MAKEFILE) ARCH=64 mylib && ... && :

The && tells bash to exit immediately with an error if the previous command fails. (The : at the end is the bash builtin that does nothing but issue a successful exit—it will simplify your writing of the $(foreach ...).)

EDIT:

The proper way to this of course is to use make dependencies, not serial processing in the shell. You want make to see something like:

tgt-name: # default target

.PHONY: target-32
target-32: deps
    $(MAKE) -f ${MY_MAKEFILE} arch=32 mylib

.PHONY: target-64
target-64: deps
    $(MAKE) -f ${MY_MAKEFILE} arch=64 mylib

# etc. etc.

tgt-name: target-32 target-64
    @echo $@ Success

This is -j safe. Under -j make will make all of the target-% at the same time. Nice. (Though in this case it seems that your $MY_MAKEFILE is not -j safe (naughty!).) A few macros to replace the boiler plate:

LIST_OF_TYPES := 32 64 ...
LIST_OF_TARGETS := $(add-prefix,target-,${LIST_OF_TYPES})

tgt-name: # default target

.PHONY: ${LIST_OF_TARGETS}
${LIST_OF_TARGETS}: target-%: deps # Static Pattern Rule will (conveniently) set $*
    $(MAKE) -f ${MY_MAKEFILE} arch=$* mylib

tgt-name: ${LIST_OF_TARGETS}
    @echo $@ Success

P.S. I suspect that you should be marking tgt-name as .PHONY

Upvotes: 1

Olaf Dietsche
Olaf Dietsche

Reputation: 74078

I've never seen this kind of use of foreach, but it seems to work for you. Usually I use a bash for loop

tgt-name: deps
    for i in $(LIST_OF_TYPES); do $(MAKE) -f $(MY_MAKEFILE) ARCH=$$i mylib; done

But this is not the problem, since in either case the makes are run sequentially, AFAICS.

Since you build a library, there's the possible clash of two objects being inserted into an archive simultaneously. When this happens the archive might become corrupted.

As with every parallel execution, be it make jobs or threads, you must protect the shared resources (the archive in your case). You must add the objects at the end of the library build or protect the insertion with some lock (e.g. man lockfile or similar).

There might be other problems, of course. Look out for the simultaneous access to shared resources (object files, archives, ...) or incomplete defined dependencies.

Update:

foreach seems not to be a problem. Set LIST_OF_TYPES to a single type (e.g. 32 only) and then do a make -j100 mylib. If the problem is with the building of a single archive, it will fail with only one type as well.

You can also test with make ARCH=32 -j100 mylib. This should show the problem too.

Upvotes: 0

Related Questions