CODEWITHSUNDEEP
linuxmakefilegnu-make
Mikko Rantalainen
Mikko Rantalainen

Reputation: 15955

GNU Make: how to default to parallel build?

We have a Makefile where dependencies are marked correcty and it would run nicely on any number of CPU cores. We currently start the Makefile with

CPUS ?= $(shell nproc)
MAKEFLAGS += -j $(CPUS) -l $(CPUS) -s

and this used to work nicely in GNU Make 4.1 or lesser. However, GNU Make 4.2 or newer will emit following warning:

warning: -j12 forced in makefile: resetting jobserver mode.

(the number in the warning obviously depends on CPU cores in your system). The processing seems to result in correct output but I don't want console spam. Fortunately, this only happens for recursive use where a target contains rules that read $(MAKE) ... but this still happens often enough to spam the console.

Is it possible to default to nproc cores (or maybe nproc + 1 might be optimal to overlap some IO with CPU load?) if top level make is executed without the -j and -l flags without triggering any warnings?

The Makefile supports a lot of targets and I would want to default all the targets to parallel processing scaled according to the current system.

Upvotes: 3

Views: 1256

Answers (1)

Mikko Rantalainen
Mikko Rantalainen

Reputation: 15955

I was able create this hack:

ifndef MAKEFLAGS
CPUS ?= $(shell nproc)
MAKEFLAGS += -j $(CPUS) -l $(CPUS) -s
$(info Note: running on $(CPUS) CPU cores by default, use flag -j to override.) 
endif

It seems to work correctly in sense if I run make alone, I get parallel execution on multiple CPUs and if I run make -j2 I get execution on two cores. I don't know if this would cause additional side-effects if some other make flags have been used but in worst case it should default to basic GNU Make behavior of running on single core only and lose the -s flag in this example.

Note that according to the documentation one should be able to write

ifeq (,$(findstring j,$(MAKEFLAGS)))
CPUS ?= $(shell nproc)
MAKEFLAGS += -j $(CPUS)
endif

to only adjust CPU count if not otherwise defined but in reality this doesn't work because for some yet unknown reason $(MAKEFLAGS) doesn't include the -j flag outside the recipes (I tested with GNU Make 4.1)!

Try Makefile like this:

$(warning MAKEFLAGS=$(MAKEFLAGS))
$(warning $(shell printf "[%s] " $(MAKEFLAGS)))
default:
        @printf "MAKEFLAGS in recipe: "
        @printf "[%s] " $(MAKEFLAGS)
        @echo

and the output of make -j4 will be

Makefile:1: MAKEFLAGS=
Makefile:2: [] 
MAKEFLAGS in recipe: [-j] [--jobserver-fds=3,4]

Note how builtin functions such as findstring or shell printf cannot access the -j flag. As a result, you have to assume that if MAKEFLAGS has been set, the end user probably knows how he wants to set the flags and probably has set the -j correctly.

Upvotes: 3

Related Questions