How do I optimise Makefiles so as not to repeat over and over

Question

I am new to using make and have been trying to understand how to properly configure the make file. In particular I think that there is a much better way of handling rules and dependants that follow a pattern. But I cannot really understand the manual in this regard.

I now have a working makefile (on OSX and Linux, I assume with gnu make), but expect that it can be shortened considerably by the use of techniques which I don't understand. It has repeating patterns for different directories all over it. Can you please show me how to make the following better, and let me know which facility is being used for each shortening.

# Include this file NOT stored in repository which defines which environment to use
include Makefile.local
# Work out the pas version
PAS=${shell git describe --abbrev=0}
# Version of node being used
VERSION = 7.6.0

\.dockerimage: access/.dockerimage evening/.dockerimage server/.dockerimage
    touch .dockerimage
server/.dockerimage: client/client/.dockerimage server/.env $(shell find server  \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)
    docker image build -t server:${PAS} -t server:latest server
    touch server/.dockerimage
server/.env: environments/common/server.env environments/${enviro}/server.env
    cat environments/common/server.env > server/.env
    cat environments/${enviro}/server.env >> server/.env
access/.dockerimage: request/.dockerimage access/.env $(shell find access  \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)
    docker image build -t access:${PAS} -t access:latest access
    touch access/.dockerimage
access/.env: environments/common/access.env environments/${enviro}/access.env
    cat environments/common/access.env > access/.env
    cat environments/${enviro}/access.env >> access/.env
evening/.dockerimage: request/.dockerimage evening/.env $(shell find evening  \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)
    docker image build -t evening:${PAS} -t evening:latest evening
    touch evening/.dockerimage
evening/.env: environments/common/evening.env environments/${enviro}/evening.env
    cat environments/common/evening.env > evening/.env
    cat environments/${enviro}/evening.env >> evening/.env
pcode/.dockerimage: libs/.dockerimage pcode/.env $(shell find pcode  \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)
    docker image build -t pcode:${PAS} -t pcode:latest pcode
    touch pcode/.dockerimage
pcode/.env: environments/common/pcode.env environments/${enviro}/pcode.env
    cat environments/common/pcode.env > pcode/.env
    cat environments/${enviro}/pcode.env >> pcode/.env
test-client/.dockerimage: client/client/.dockerimage
    docker image build -t test-client:${PAS} test-client:latest test-client
    touch test-client/.dockerimage
test-server/.dockerimage: services/.dockerimage
    docker image build -t test-server:${PAS} -t test-server:latest test-server
    touch test-server/.dockerimage
client/client/.dockerimage: services/.dockerimage $(shell find client/client \! -name .dockerimage -type f)
    docker image build -t client:${PAS} -t client:latest client/client
    touch client/client/.dockerimage
services/.dockerimage: client/.dockerimage $(shell find services \! -name .dockerimage -type f -print0)
    docker image build -t services:${PAS} -t services:latest services  --build-arg PAS_VERSION=${PAS}
    touch services/.dockerimage
client/.dockerimage: libs/.dockerimage $(shell find client \! -name .dockerimage -depth 1 -type f)
    docker image build -t components:${PAS} -t components:latest client
    touch client/.dockerimage
libs/.dockerimage: request/.dockerimage $(shell find libs \! -name .dockerimage -type f)
    docker image build -t libs:${PAS} -t libs:latest libs
    touch libs/.dockerimage
request/.dockerimage: node/.dockerimage $(shell find request \! -name .dockerimage -type f)
    docker image build -t request:${PAS} -t request:latest request
    touch request/.dockerimage
node/.dockerimage: node/Dockerfile-${ARCH} node/.dockerignore
    docker image build -f node/Dockerfile-${ARCH} -t node:${VERSION} -t node:latest node
    touch node/.dockerimage
clean: clean-images clean-above clean-env
    rm node/.dockerimage
    docker image rm -f node:latest
    docker image rm -f node:${VERSION}
clean-above:
    for dir in server access evening pcode client/client services test-client test-server servies client libs request; \
        do rm $$dir/.dockerimage; done
clean-env:
    for dir in access evening pcode server; do rm $$dir/.env; done
clean-images: clean-above
    for dir in access access evening pcode client/client services test-client test-server services client libs request; \
     do docker image rm -f $$dir:latest;  docker image rm -f $$dir:${PAS}; done
.PHONY: run clean clean-above clean-images clean-env

It would be also useful for it to run under git-bash (adding make.exe to it) on windows, although not essential.

Answer 1

Thanks to @MadScientist, I have been exploring implicit patterns. I had one problem with expansion that was solved in another SO question/answer. I thought for completeness sake I would post where I've got to. Since the original question I have added quite a lot more options to the makefile, so its not a one for one comparision.

# Include this file NOT stored in repository which defines which environment to use
include Makefile.local
#work out architecture from envonment
ARCH := ${shell cat environments/${enviro}/arch}
# Work out the pas version
PAS := ${shell git describe --abbrev=0}
# None standard dependancies for each of the images
server-IMAGEDEPS := libs/database/index.js libs/log/index.js libs/utils/index.js\
    $(shell find services/manager -type f -not -name package.json -print0) $(shell find services/web -type f -not -name package.json) \
    $(shell find client -type f -not -name .bowerrc -not -name bower.json -not -name package.json -not -name DOCKBUILDfile -print0)
server-BASEDEPS := services/.dockerimage
server-LINKDEPS := pasv5-database pasv5-manager pasv5-web
access-IMAGEDEPS := libs/request/index.js
access-BASEDEPS := libs/.dockerimage
access-LINKDEPS := pasv5-request
evening-IMAGEDEPS := libs/request/index.js libs/log/index.js
evening-BASEDEPS := libs/.dockerimage
evening-LINKDEPS := pasv5-request pasv5-log
pcode-IMAGEDEPS := libs/database/index.js libs/log/index.js libs/utils/index.js
pcode-BASEDEPS := libs/.dockerimage
pcode-LINKDEPS := pasv5-database pasv5-log pasv5-utils
manage-LINKDEPS := pasv5-utils pasv5-log
web-LINKDEPS := pasv5-log
daily-LINKDEPS := pasv5-database

LINKMODULEDIRS := libs/database libs/log libs/utils libs/request services/manager services/web
NODEMODULEDIRS := $(LINKMODULEDIRS) access evening pcode server daily

# Standard docker build command
DOCKBUILD = docker image build -t $(@D)-base:stable $(@D); touch $@

all: server/.dockerimage evening/.dockerimage pcode/.dockerimage access/.dockerimage

daily/.env: pcode/.env
    cp pcode/.env daily/.env
services/.dockerimage: client/.dockerimage services/Dockerfile services/package.json services/manager/package.json services/web/package.json
    @(DOCKBUILD)
client/.dockerimage: libs/.dockerimage client/Dockerfile client/package.json client/.bowerrc client/bower.json
    @(DOCKBUILD)
libs/.dockerimage: node/.dockerimage libs/Dockerfile libs/package.json libs/database/package.docker libs/log/package.json libs/utils/package.json \
    libs/request/package.json libs/request/akc-crt.pem
    @(DOCKBUILD)
node/.dockerimage: node/Dockerfile-${ARCH} node/.dockerignore
    docker image build -f node/Dockerfile-${ARCH} -t node:latest node
    $(eval VERSION := $(shell docker run node:latest node --version | cut -c 2- ))
    docker image tag node:latest node:${VERSION}
    docker image tag node:latest docker.hartley-consultants.com/node:${VERSION}
    touch $@
# stable version of node
node-stable:
    docker image tag node:latest node:stable
clean: clean-images clean-above clean-env
    rm node/.dockerimage
    docker image rm -f node:latest
    docker image rm -f node:${VERSION}
clean-above:
    for dir in server access evening pcode client services libs; do [ -f $$dir/.dockerimage ] && rm $$dir/.dockerimage; [ -f $$dir/.dockerbase ] && rm $$dir/.dockerbase; done; exit 0
clean-env:
    for dir in access evening pcode server; do rm $$dir/.env; done
clean-images: clean-above
    for img in access evening pcode server; \
     do docker image rm -f $$img-base:stable;  docker image rm -f $$img-${enviro}:${PAS}; rm $$img/.dockerimage; rm $$img/.dockerbase; done
    for img in services client libs; do docker image rm -f $$img:stable; rm $$img/.dockerimage; done

# sets up to run system without docker images
local: server/.env evening/.env pcode/.env access/.env daily/.env $(foreach dir,$(NODEMODULEDIRS), $(dir)/node_modules)
    npm install -g bower && cd client && bower install

pasv5-database:
    cd libs/database; npm link
pasv5-log:
    cd libs/log; npm link
pasv5-utils:
    cd libs/utils; npm link
pasv5-request:
    cd libs/request; npm link
pasv5-manager:
    cd services/manager; npm-link
pasv5-web:
    cd services/web; npm-link
clean-local:
    $(foreach dir,$(NODEMODULEDIRS), $(shell rm -rf $(dir)/node_modules))
.SECONDEXPANSION:
# pattern rules
%/.env: envionments/common/%.env environments/${enviro}/%.env; cat $^ > $@
%/.dockerbase: %/Dockerfile %/package.docker %/.env $$(%-BASEDEPS)
    $(DOCKBUILD)
%/.dockerimage: %/.dockerbase %/server.js Dockerfile-% $$(%-IMAGEDEPS)
    docker image build -t $(@D)-${enviro}:${PAS} --build-arg PAS_VERSION=${PAS} -f Dockerfile-${@D} .
ifeq ($(enviro), production)
    docker tag $(@D)-production:${PAS} docker.hartley-consultants.com/pas/$(@D):${PAS}
endif
    touch $@
%/node_modules: $$(%-LINKDEPS)
    cd $*; npm install; for link in $^ ; do npm link $$link ; done


.PHONY: all clean clean-above clean-images clean-env node-stable local clean-local pasv5-database pasv5-log pasv5-utils pasv5-request pasv5-manager pasv5-web

Answer 2

Simplifying of makefiles gives good results when iterative. Start with smaller steps two wrap repeated patterns in functions/variables/implicit rules and do it again and again. The following is my attempt to simplify some of rules in your makefile:

some_prerequisites = $1/.env $(shell find $1 \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)

environment_recipe = cat environments/common/$1.env environments/${enviro}/$1.env > $@
dockerimage_recipe = docker image build -t $1:${PAS} -t $1:latest $1 && touch $@

\.dockerimage: access/.dockerimage evening/.dockerimage server/.dockerimage
    touch $@
server/.dockerimage: client/client/.dockerimage $(call some_prerequisites, server)
    $(call dockerimage_recipe,server)
server/.env: environments/common/server.env environments/${enviro}/server.env
    $(call environment_recipe,server)
access/.dockerimage: request/.dockerimage $(call some_prerequisites, access)
    $(call dockerimage_recipe,access)
access/.env: environments/common/access.env environments/${enviro}/access.env
    $(call environment_recipe,access)
evening/.dockerimage: request/.dockerimage $(call some_prerequisites, evening)
    $(call dockerimage_recipe,evening)
evening/.env: environments/common/evening.env environments/${enviro}/evening.env
    $(call environment_recipe,evening)
pcode/.dockerimage: libs/.dockerimage $(call some_prerequisites, pcode)
    $(call dockerimage_recipe,pcode)
pcode/.env: environments/common/pcode.env environments/${enviro}/pcode.env
    $(call environment_recipe,pcode)

At the first iteration I introduced some_prerequisites function. Then, when makefile becomes slightly less messed I added environment_recipe and dockerimage_recipe.

This is #2 iteration:

some_prerequisites = $1/.env $(shell find $1 \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)

environment_recipe = cat environments/common/$1.env environments/${enviro}/$1.env > $@
dockerimage_recipe = docker image build -t $1:${PAS} -t $1:latest $1 && touch $@

\.dockerimage: access/.dockerimage evening/.dockerimage server/.dockerimage
    touch $@
server/.dockerimage: client/client/.dockerimage $(call some_prerequisites, server)
    $(call dockerimage_recipe,server)
access/.dockerimage: request/.dockerimage $(call some_prerequisites, access)
    $(call dockerimage_recipe,access)
evening/.dockerimage: request/.dockerimage $(call some_prerequisites, evening)
    $(call dockerimage_recipe,evening)
pcode/.dockerimage: libs/.dockerimage $(call some_prerequisites, pcode)
    $(call dockerimage_recipe,pcode)

$(foreach e, pcode evening access server \
    $(eval $e/.env: environments/common/$e.env environments/${enviro}/$e.env; \
        $$(call environment_recipe,$e) \
    ) \
)

This is #3 iteration:

some_prerequisites = $1/.env $(shell find $1 \! -name pacakge.json \! -name .dockerimage -depth 1 -type f)

environment_recipe = cat environments/common/$1.env environments/${enviro}/$1.env > $@
dockerimage_recipe = docker image build -t $1:${PAS} -t $1:latest $1 && touch $@

\.dockerimage: access/.dockerimage evening/.dockerimage server/.dockerimage
    touch $@
server/.dockerimage: client/client/.dockerimage
access/.dockerimage: request/.dockerimage
evening/.dockerimage: request/.dockerimage
pcode/.dockerimage: libs/.dockerimage

$(foreach e, pcode evening access server \
    $(eval $e/.env: environments/common/$e.env environments/${enviro}/$e.env; \
        $$(call environment_recipe,$e) \
    ) \
    $(eval $e/.dockerimage: $(call some_prerequisites, $e); \
        $$(call dockerimage_recipe,$e) \
    )\
)

To my mind this is sufficient simplification for some of rules in your makefile. Other can be simplified following the same pattern.

Answer 3

StackOverflow is not really a place to ask for someone to rewrite your code.

I will say two things: first, to reduce duplication you need to learn about make variables: you can put a lot of the duplicated items here in variables then use those instead of writing everything multiple times. This is simple to do.

Second, you should look into implicit rules, most particularly pattern rules. It looks like there's enough overlap between your rules that you should be able create a few different pattern rules rather than write a separate explicit rule for every target. This is a slightly more advanced topic.

You can make an attempt at modifying your makefile and see what happens: trial and error is often the best way to learn and it's cheap and easy to run make multiple times. If you run into problems you can't solve, now you have an appropriate, specific question for StackOverflow :).

Oh one last thing: you should consider using simply expanded variables (i.e., use := for assignment not =), especially for your $(shell ...) invocations; it will be much more efficient.