How to force make to always update and re-read an included makefile?

Question

I have a Makefile that generates a makefile.mk, which is subsequently included in the main makefile. Because the set of input files (*.in in this example) can change at any time, I need the makefile.mk to be regenerated once for each make run.

The include documentation states:

Once it has finished reading makefiles, make will try to remake any [included makefiles] that are out of date or don’t exist.

And on How Makefiles Are Remade:

[A]fter reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it [...], it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again.

Based on that, I thought the following should work:

include makefile.mk

makefile.mk:
    echo "--- Begin makefile.mk ---"
    @( \
        shopt -q -s nullglob; \
        INPUTS=(*.in); \
        echo "all:" $${INPUTS//.in/.out}; \
        for f in $$INPUTS; do \
            echo "$${f/.in/.out}: $$f"; \
            echo -e '\tcp $$< $$@'; \
        done \
    ) | tee $@
    echo "--- End makefile.mk ---"

(I know I can use pattern rules and wildcards in this simple example. In the real thing, makefile.mk has more complicated dependencies and is generated by a Python script.)

So we drop this into an empty directory and run it:

$ ls
Makefile
$ make
--- Begin makefile.mk ---
all:
--- End makefile.mk ---
make: Nothing to be done for 'all'.

So far so good. What happens if we create an input file?

$ touch test.in
$ ls
Makefile  makefile.mk  test.in
$ make
make: Nothing to be done for 'all'.

Because makefile.mk has no dependencies, it doesn't get updated. Bad! How about forcing it to always be considered out of date, by marking it phony:

.PHONY: makefile.mk

After adding this line, indeed makefile.mk is being regenerated:

$ make
--- Begin makefile.mk ---
all: test.out
test.out: test.in
    cp $< $@
--- End makefile.mk ---
make: Nothing to be done for 'all'.

However, it seems it's not being re-read, because make claims there's nothing to be done for all even though its prerequisite, test.out, doesn't exist.

Running make a second time does the right thing:

$ make
--- Begin makefile.mk ---
all: test.out
test.out: test.in
    cp $< $@
--- End makefile.mk ---
cp test.in test.out

Why doesn't make restart and read the updated makefile.mk, as the documentation seems to promise?

I found this related answer, which suggests that .PHONY is to blame, and offers a solution like this:

makefile.mk: force
    ...

.PHONY: force
force:

However, if I try that, make gets into an infinite loop, remaking makefile.mk over and over.

Answer 1

I see two ways to solve this:

• Use a unique suffix for generated makefile, so that it never exists when make is run, like:

  UNIQUE := $(shell date +%s.%N)
  include makefile.mk.$(UNIQUE)
  makefile.mk.$(UNIQUE): […]
  

• Generate makefile.mk as before with force), but make its inclusion (and other parts of the makefile) conditional on some make variable, so that it is not performed by default. Guard the recipe for makefile.mk by the opposite condition. Then invoke make recursively with this variable to activate the inclusion and disable the rebuild rule.

---

Edit by Thomas: Using the built-in MAKE_RESTARTS variable solves the infinite loop quite effectively.

include makefile.mk

ifndef MAKE_RESTARTS
makefile.mk: .FORCE
    @( \
        shopt -q -s nullglob; \
        INPUTS=(*.in); \
        echo "all:" $${INPUTS//.in/.out}; \
        for f in $$INPUTS; do \
            echo "$${f/.in/.out}: $$f"; \
            echo -e '\tcp $$< $$@'; \
        done \
    ) | tee $@

.PHONY: .FORCE
.FORCE:
endif