GNU make many output files from single dependency which might match others

Question

I tried looking for answers to this question, so I apologize in advance if this is a duplicate of a question I didn't find. Also sorry that I cannot directly provide the code that I am working with (it would require a lot of environmental dependencies, anyway).

I have a sequence of actions, which all depend on the success of the previous actions, and also don't need to be repeated unless they are out of date. A make solution seemed like the proper one. I've come up with a solution that does almost all of it. Here is the sequence of steps I am trying to replicate, with the output of each step listed below its input:

• ZIP file
  • extract to package/
• package/directory/*.comp
  • execute uncomp.py to create a .uncomp file from a .comp file
• Everything works fine up to this point
• package/directory/*.uncomp
  • For *.uncomp files, execute script1 to produce a .html file
  • For *_ext.uncomp files, execute script2 to produce numbered *_ext.##.png file(s)
    • Multiple numbered files (_ext.0.png, _ext.1.png, _ext.2.png) are possible, and may not be present at the time make is run. However, make should know that they are the output of the previous step, and only run this recipe if these files (a) don't exist or (b) any are older than the *_ext.uncomp file.

I have put together a Makefile which does almost what I'm looking for, except that it delegates all of the last portion (numbered files) to a shell script which I could program to look at file times, but that defeats the purpose of using make in the first place, in my opinion.

Environment

Debian 8.8 (x86)

GNU Make 4.0
Built for x86_64-pc-linux-gnu

My Question

What rules and recipes can I use to inform GNU make of the relationship between the *_ext.uncomp files and the _ext.##.png files so that those recipes only get executed as necessary (and say 'Target is up-to-date' if all .png files are at least as new as the _ext.uncomp file), that won't also apply to the *.uncomp files, and that will still work of there are no .png files in the output?

I will also need to indicate the relationship between non-_ext files and their corresponding HTML counterparts. So that script1 only gets executed when the HTML file is out of date or doesn't exist. This recipe/rule should not pay attention to _ext.uncomp files.

Any other advice on my Makefile would also be appreciated, because I am not overly familiar with it.

Generalized contents of my current Makefile

.PHONY : all
all : package package/directory/*.uncomp
    ./process $^

%.comp.uncomp : %.comp package
    python uncomp.py $<

package : *.zip
    rm -rf package/
    unzip *.zip -d package/

Contents of the process script

This script should no longer exist if all the goals of the question are met (make will handle everything). It works great, but it always processes .uncomp files no matter what, even if the output from them already exists and is newer than the source.

#!/bin/bash

if [ $# -lt 2 ]; then
    echo "$0 expects at least 2 arguments"
    exit 1
fi

# Discard the first agrument, it's always 'package'
shift

# Iterate over each of the remaining arguments
while [ $# -gt 0 ]; do
    if [[ $1 == *_ext.uncomp ]] ; then
        python script2 $1
    elif [[ $1 == *.uncomp ]] ; then
        python script1 $1
    else
        echo "Warning: Unknown file type: $1"
    fi
    shift
done

Answer 1

I learned a lot about GNU make trying to get this to work. I discovered that the solution to my problem was in not overthinking it.

The most important realization was that I didn't need make to track all of the numbered output files, but just the first one (if the first one is out of date or missing, they all will be, and they all get re-extracted by the script, so a 1:1 relationship was all I needed to indicate there).

I found out that GNU make 3.82 and later uses "shortest stem first" order instead of definition order when matching pattern rules. To make my file compatible with both versions, I made sure to define the most specific stems first.

After that it was a matter of setting up some implicit rules, and just telling make what to expect to be able to find—the concept is a little backwards to my way of thinking which is why I had some trouble at first (look for this file that doesn't exist yet; now, here's a way to make it from a file that does exist). The end result, fully functional:

PACKAGE      := package
COMP         := .comp
UNCOMP       := .comp.uncomp
PNG0         := .comp.0.png
TXT          := .comp.txt
SUFFIX       := _ext

COMPFILES     = $(wildcard $(PACKAGE)/subdir/*$(COMP))
UNCOMPFILES   = $(COMPFILES:$(COMP)=$(UNCOMP))
SUFFIXFILES   = $(filter %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
PNGFILES      = $(SUFFIXFILES:$(UNCOMP)=$(PNG0))
NOSUFFIXFILES = $(filter-out %$(SUFFIX)$(UNCOMP),$(UNCOMPFILES))
TXTFILES      = $(NOSUFFIXFILES:$(UNCOMP)=$(TXT))

.PHONY : all
all : pngs txts htaccess

.PHONY : txts
txts : $(TXTFILES)

.PHONY : pngs
pngs : $(PNGFILES)

.PHONY : uncomp
uncomp : $(UNCOMPFILES)
    make pngs
    make txts

.PHONY : htaccess
htaccess : $(PACKAGE)/.htaccess

%$(SUFFIX)$(PNG0) : %$(SUFFIX)$(UNCOMP)
    @# Ignore failures when extracting PNG files
    -python script1.py $<

%$(TXT) : %$(UNCOMP)
    @# Ignore failures when dumping TXT files
    -python script2.py $< > $@

%$(UNCOMP) : %$(COMP)
    @# Ignore decompression failure
    -python uncomp.py $<

$(PACKAGE)/.htaccess : .htaccess | $(PACKAGE)
    cp .htaccess $(PACKAGE)/

$(PACKAGE) : *.zip
    rm -rf $(PACKAGE)/
    unzip *.zip -d $(PACKAGE)/
    make uncomp

.PHONY : clean
clean :
    rm -rf $(PACKAGE)/