CODEWITHSUNDEEP
c++makefilecmakelinker
MateuszL
MateuszL

Reputation: 2983

compile concurrently, link serially

I have big cmake/c++/linux project: a lot of small static libraries, a few big and interdependent static libraries, a few big executable binaries. Single binary with debug symbols is several GB. There are ~10 such binaries (worker, testA, testB, testC...). Compilation usually takes more time than we would like, but we have fast build server and we use make -j20. The worst though is linking. Single linking takes about 60 seconds and 4GB RAM. But when all final binaries are linked at the same time (happens often when 1 small sublibrary was modified, little to recompile, a lot to relink), 10 linkers use 40GB RAM (for 1 developer, there may be more) and very long time. IO is most likely the bottleneck.

We have many developers on 1 strong server and everybody uses make -j20 -l30 so that we don't overload CPU. But we don't have method for limiting number of concurrent linkers. It would be great to limit number of working linkers globally on server, but per make invocation would help as well. Ideally make -j20 -l30 --concurrent-linkers=2. Is it possible?

We use gold linker. We are in progress of separating smaller, independent modules, but this will take a long time.

Upvotes: 3

Views: 374

Answers (1)

Renaud Pacalet
Renaud Pacalet

Reputation: 28945

You could try something like:

$ cat Makefile
OBJS := foo bar baz...
EXES := qux quux quuz...

.PHONY: all

all: $(OBJS)
    $(MAKE) -j $(concurrent-linkers) $(EXES)

$(OBJS): ...
    <compile>

$(EXES): ...
    <link>

And call it with:

$ make -j20 -l30 concurrent-linkers=2

Basically, it separates the build in two make invocations, one for compilation and one for link, with different -j options. The main drawback is that all compilations must be finished before the first link starts. A better solution would be to design a simple link job server (a simple shell script with a bit of flock and tag files would make it) and delegate it the link jobs. But if you can live with this...

Demo with a dummy Makefile:

$ cat Makefile
OBJS := a b c d e f
EXES := u v w x y z

.PHONY: all

all: $(OBJS)
    $(MAKE) -j $(concurrent-linkers) $(EXES)

$(OBJS) $(EXES):
    @printf 'building $@...\n' && sleep 2 && printf 'done\n' && touch $@
$ make -j20 -l30 concurrent-linkers=2
building a...
building d...
building b...
building c...
building e...
building f...
done
done
done
done
done
done
make -j 2 u v w x y z
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: Entering directory 'foobar'
building u...
building v...
done
done
building w...
building x...
done
done
building y...
building z...
done
done
make[1]: Leaving directory 'foobar'

As you can see all $(OBJS) targets are built in parallel while the $(EXES) targets are built 2 (maximum) at a time.

EDIT If your makefile is generated by CMake there are at least two options:

  1. Tune your CMake files such that CMake generates two different makefiles: one for compilation and one for link. Then write a simple wrapper makefile like:

    .PHONY: myAll myCompile

myAll: myCompile
    $(MAKE) -j $(concurrent-linkers) -f Makefile.link

myCompile:
    $(MAKE) -f Makefile.compilation

  2. Convince CMake (if it is not already the case) to generate a makefile that defines two make variables: one (OBJS) set to the list of all object files and one (EXES) set to the list of all executable. Then write a simple wrapper makefile like:

    .DEFAULT_GOAL := myAll

include CMake.generated.Makefile

.PHONY: myAll

myAll: $(OBJS)
    $(MAKE) -j $(concurrent-linkers) $(EXES)

    A very similar solution exists if, instead, CMake generates two phony targets, one for all object files and the other for all executable:

    .DEFAULT_GOAL := myAll

include CMake.generated.Makefile

.PHONY: myAll

myAll: cmake-target-for-compilation
    $(MAKE) -j $(concurrent-linkers) cmake-target-for-link

Upvotes: 2

Related Questions