Recursive Makefile Does Not Recognize Dependency Changes in Sub-Directory

Question

I am aware that there is a lot of debate around recursive makefiles. That being understood, I'm still trying to get one to work.

Context: I have the following project setup:

quendor
  src
    zmachine
      Makefile
      main.c
      zmachine.h
  Makefile

So I use the root-level Makefile to call another Makefile in a directory. (My actual project has other directories that generate other library files but, for now, the above showcases the problem with the minimum amount of context.

The makefile setup I have in place does work in that it generates .o and .d files in the zmachine directory. Then a library (zmachine.a) is built correctly.

Problem: The problem I have is that when I run the makefile at the root level again, it will not pick up if there have been changes to either the .c or .h files in the subdirectory. I just get this message:

make: Nothing to be done for `zmachine_lib'.

Example of My Code:

Here is the root level Makefile:

CC := gcc

ifeq ($(CC), gcc)
    CFLAGS += -std=c11 -Wall -Wextra -Wpedantic -Wconversion -Wmissing-prototypes -Wshadow -MMD -MP
    LDFLAGS +=
    OPT +=
endif

AR ?= $(shell which ar)
RANLIB ?= $(shell which ranlib)

export CC
export AR
export RANLIB
export CFLAGS

SRC_DIR = src

ZMACHINE_DIR = $(SRC_DIR)/zmachine
ZMACHINE_LIB = $(ZMACHINE_DIR)/zmachine.a

SUB_DIRS = $(ZMACHINE_DIR)

SUB_CLEAN = $(SUB_DIRS:%=%-clean)

# Targets

zmachine_lib : $(ZMACHINE_LIB)

$(ZMACHINE_LIB):
    $(MAKE) -C $(ZMACHINE_DIR)

clean : $(SUB_CLEAN)

$(SUB_CLEAN):
    -$(MAKE) -C $(@:%-clean=%) clean

Here is the Makefile from the zmachine directory:

SOURCES := $(wildcard *.c)
HEADERS = $(wildcard *.h)
OBJECTS := $(SOURCES:%.c=%.o)

DEPS := $(OBJECTS:%.o=%.d)

TARGET = zmachine.a

ARFLAGS = rc

$(TARGET): $(OBJECTS)
    $(AR) $(ARFLAGS) $@ $?
    $(RANLIB) $@
    @echo "** Finished building Z-Machine architecture."

%.o: %.c
    $(CC) $(CFLAGS) -fPIC -c $< -o $@

clean:
    $(RM) $(TARGET) $(OBJECTS) $(DEPS)

-include $(DEPS)

Again, the logic of the Makefiles works in terms of generating the correct output and library files. So I seem to have got that part right.

What I can't get to work is having the combined Makefiles, working together, recognize that changes have been made and thus recompiling is necessary.

For example, if I change main.c or zmachine.h, running make again won't recognize that the change has happened, thus no recompilation is trigered.

What I Tried:

I did some searching for "recursive makefiles dependencies" and related searches but I couldn't find anything that showed me how to handle dependency changes in this specific context.

I did try running make -d to look at the files and dates that make was using. It gave me this output, which wasn't as helpful to me:

 No implicit rule found for `zmachine_lib'.
  Considering target file `src/zmachine/zmachine.a'.
   Finished prerequisites of target file `src/zmachine/zmachine.a'.
  No need to remake target `src/zmachine/zmachine.a'.
 Finished prerequisites of target file `zmachine_lib'.
Must remake target `zmachine_lib'.
Successfully remade target file `zmachine_lib'.
make: Nothing to be done for `zmachine_lib'.

I realize this must be because I do not have the file set up to recognize what depends on what. But I feel I do with this:

zmachine_lib : $(ZMACHINE_LIB)

$(ZMACHINE_LIB):
    $(MAKE) -C $(ZMACHINE_DIR)

Here zmachine_lib is the only target in this example. It depends on $(ZMACHINE_LIB) which, as you can see, I have set up to call the Makefile in the subdirectory.

So I'm guessing it's this bit of calling $(MAKE) that is perhaps obfuscating changes made at the subdirectory level since, for the $(ZMACHINE_LIB) target would appear to be already satisfied (from the perspective of the top-level Makefile).

I did find this makefile with dependency on a shared library sub project, which does seem like it's very close to my issue. But I can't see how to utilize that solution in my case because of the fact that the logic is distributed over the two Makefiles. I did try to change my target in the sub-Makefile to this:

$(TARGET): $(SOURCES)
  ...

Basically changing $(OBJECTS) to $(SOURCES), which is what it seems like that other solution was suggesting. But that does not work for me; dependency changes are still not recognized.

Answer 1

I am seeing what I think is one other possible answer to this. I'm finding that if I put the following in my root-level Makefile:

.PHONY : $(ZMACHINE_LIB)

It seems to work. Specifically, with that, let's say I start fresh -- no files generated. Then I do:

make zmachine_lib

That correctly compiles everything in the zmachine directory and builds the library zmachine.a Then if I run that same command again:

make zmachine_lib

I get:

'zmachine.a' is up to date

Perfect. That makes sense. Nothing changed, so everything is up to date.

Now I make a change to the main.c file in the zmachine directory and try again:

make zmachine_lib

Sure enough! The change is recognized and recompilation takes place. The same thing happens if I change zmachine.h.

So it looks like just treating the $(ZMACHINE_LIB) as a phony target forces the makefile in the subdirectory to actually be read -- even if zmachine.a exists.

I don't know if what I'm doing is considered bad practice here but I read the oft-mentioned Recursive Make Considered Harmful and it seems the situation is a little more nuanced than just saying "recursive makefiles are harmful." That said, I can see why people take the stance they do. (Actually, I can see it on both sides of the issue.)

Answer 2

I'm actually marking John's answer as the answer, but I figured I would include one of my own as well.

I found the following How can I get the dependencies of a target in a recursive makefile? which has the perfect solution in a sense:

"Rewrite it to be non-recursive or just live with it."

That pretty much is it, really. But live with what? Of all the material out there that seems to belabor the points, none seem to just state simply what would probably most help people like myself coming across this. To wit:

Calling one makefile from within another makefile (i.e., using recursive make build process) can be ineffective and inefficient because neither makefile will have the full overview of dependencies. This means the only way you can make sure your compilation is fully up to date with all changes is to do a make clean, effectively rebuilding the project each time.

Had I seen that, I would have known immediately that what I was dealing with was something that simply isn't possible. And while there are solutions that you can attempt, most of those will really just be introducing other problems. Which then goes back to that simple comment: "Rewrite it to be non-recursive or just live with it."

Examples of how to rewrite it non-recursively would also probably go a long way.

Answer 3

I am aware that there is a lot of debate around recursive makefiles.

I'm not sure there is so much debate, really. Recursive make has some pretty well known limitations. Non-recursive make has different, largely complementary, limitations.

That being understood, I'm still trying to get one to work.

Lots of people do. But although you may understand that recursive make has known issues, you do not seem to understand their nature, because you are asking about a manifestation of one of main ones.

Problem: The problem I have is that when I run the makefile at the root level again, it will not pick up if there have been changes to either the .c or .h files in the subdirectory.

No, it doesn't. That is to be expected with your makefile.

Recursive make serves large projects by dividing build information into more manageable pieces and keeping it close to the sources being built. Among the main costs of doing so is that details, especially dependency details, are compartmentalized. In your case, the top-level make knows that a target src/zmachine/zmachine.a is to be built, but no dependencies for that target are known to it. As a result, that make considers the target out of date only if it does not exist. So yes, it will not recognize that target as being out of date relative to its actual sources. That a sub-make would update it if run is irrelevant, because the sub-make never runs.

The usual, more practicable approach to recursive make is to recurse unconditionally. Something like this:

SUBDIRS = src doc

all: $(SUBDIRS)

$(SUBDIRS):
        $(MAKE) -C $@

Where there are dependencies among the targets managed by different make runs, those still need to be modeled somehow, else you will sometimes need multiple runs of the overall make for everything to be updated. Ideally, that's handled on a directory-by-directory basis, not a specific-target basis.

But that's a bit oversimplified, I'm afraid, because one generally wants recursion to work for most or all top-level targets, not just for the default target. You should consider looking at someone else's working recursive make build system to see how they do it.