How to write a "bidirectional" makefile that keeps two files synchronized?

Question

I'm working with set of binary files that can be "decompiled" to or "compiled" from a set of source (text) files (both formats are developed in house BTW). Since both the binary and text files are checked into my repository, I use a small script to (de)compile all of them.

Our workflow usually involves editing the binary files directly, and decompiling the modified binaries to text. However, occasionally we need to edit the text files and compiling the changes to binaries.

The question: Can I make a single makefile that detects which set was modified more recently, and automatically issues (de)compile commands in either direction to keep both set of files up to sync? I prefer using common (GNU?) make features, but if there is a more specialized tool that works, I'm all ears.

(I could make two separate directives, "decompile-all" and "compile-all". I want to know if there's a single-command option.)

Answer 1

I've run across this use case in the context of XML. I'd like to have a relax ng schema in both xml and compact syntax, arranged so that editing either updates the other. First draft was quite messy - involved a lot of eval - but the updated version below is basically fine.

Make lets you associate multiple rules with a single target and runs them independently via the "double colon" syntax. Make is not very willing to accept circular dependencies.

Thus, for foo.rng and foo.rnc (the suffixes of the relax ng schema forms), create a target foo.whatever whose purpose is to represent the most recent state of those two input files. I've arbitrarily chosen to use the same format as .rng for it, it could be unrelated. Update it from whichever of the inputs are newer than the last time make ran. Then derive the two formats from that one file, and use those elsewhere in the build.

Also I've picked one of the formats as favoured, in the sense that when they're both out of date I make no attempt to pick the newer one, though that's implementable with more hassle.

# The idea here is to allow relax ng schemas in compact or xml formats.
# If both exist, editing either will update the other.
# If only one format exists the rest of the system can use either format.
# On ambiguity on what to do, the xml format schema wins.
# Driven by suffix, *.rng is xml syntax and *.rnc is compact syntax.
# 
# This file defines one function that takes a %-prefixed name of a schema from
# somewhere in the MAKEFILE_DIR rooted tree:
# $(call get_schema_name, %derived.rnc)
# The call returns the name of an up to date copy of that target, even if the only
# source for that schema in tree is in the the other format, which can be used
# for xmllint etc
#
# As a special case, if both exist and one is empty, it'll be updated from the other.
# This means one can "fix" a failed partial edit by clearing the current file and
# calling make to clobber it with the previous version. It also means one can create
# a file in the other format, next to the current one, populate that with make, edit
# it with changes reflected in the original, then delete it when done.
#
# If both formats are present, running this script is prone to putting them both in
# the output format of trang, which is roughly a pretty-printed / canonical form.

# Will only find schema under this root.
MAKEFILE_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST))))

# Go in search of files that look like schema in .rnc or .rng format
# Using $(SCHEMA_HIDDEN_DIR) as a working directory, the exported function
# returns names of targets under said working directory and clean deletes it
SCHEMA_HIDDEN_DIR := .schema/
SCHEMA_XML_DIR := $(SCHEMA_HIDDEN_DIR)xml/
SCHEMA_RNG_DIR := $(SCHEMA_HIDDEN_DIR)rng/
SCHEMA_RNC_DIR := $(SCHEMA_HIDDEN_DIR)rnc/

clean::
    rm -rf $(SCHEMA_HIDDEN_DIR)

SCHEMA_RWILDCARD = $(foreach d,$(wildcard $(1)*),$(call SCHEMA_RWILDCARD,$(d)/,$(2)) $(filter $(subst *,%,$(2)),$(d)))

# Ignore any under the codegen dir so running repeatedly doesn't accumulate more
SCHEMA_RAW_RNC_SOURCE := $(filter-out $(MAKEFILE_DIR)/$(SCHEMA_HIDDEN_DIR)%, $(call SCHEMA_RWILDCARD,$(MAKEFILE_DIR)/,*.rnc))
SCHEMA_RAW_RNG_SOURCE := $(filter-out $(MAKEFILE_DIR)/$(SCHEMA_HIDDEN_DIR)%, $(call SCHEMA_RWILDCARD,$(MAKEFILE_DIR)/,*.rng))

# Derive names under XML_DIR
SCHEMA_CHDIR =  $(subst $(MAKEFILE_DIR)/, $(SCHEMA_XML_DIR), $1)
SCHEMA_UPDATED_RNG := $(call SCHEMA_CHDIR, $(SCHEMA_RAW_RNG_SOURCE))
SCHEMA_UPDATED_RNC := $(call SCHEMA_CHDIR, $(SCHEMA_RAW_RNC_SOURCE:.rnc=.rng))

# Going to make one file for each schema under XML_DIR
SCHEMA_UPDATED := $(sort $(SCHEMA_UPDATED_RNG) $(SCHEMA_UPDATED_RNC))

# And then going to extract a RNG and a RNC format from it
SCHEMA_DERIVED_RNG := $(subst $(SCHEMA_XML_DIR), $(SCHEMA_RNG_DIR), $(SCHEMA_UPDATED))
SCHEMA_DERIVED_RNC := $(patsubst %.rng, %.rnc, $(subst $(SCHEMA_XML_DIR), $(SCHEMA_RNC_DIR), $(SCHEMA_UPDATED)))

# If both formats exist for a given schema, update the other one
# This is prone to putting them in the format trang pretty prints, which seems fine
# Note that if the rng and the rnc are both newer than the target tree, the rng format wins
# as it runs first and overwrites the other one
# The double colon :: is why this works out, along with not telling make that
# running the receipe clobbers the other candidate input file

$(SCHEMA_UPDATED_RNG):: $(SCHEMA_XML_DIR)%.rng: %.rng
#   @echo "Updating $@ from $<"
    @mkdir -p $(dir $@)
    @if ! test -s $< ; then trang -Irnc -Orng $(<:.rng=.rnc) $< ; fi
    @cp $< $@
    @$(if $(filter $@, $(SCHEMA_UPDATED_RNG)), @trang -Irng -Ornc $@ $(<:.rng=.rnc))
    @touch $@

$(SCHEMA_UPDATED_RNC):: $(SCHEMA_XML_DIR)%.rng: %.rnc
#   @echo "Updating $@ from $<"
    @mkdir -p $(dir $@)
    @if ! test -s $< ; then trang -Irng -Ornc $(<:.rnc=.rng) $< ; fi
    @trang -Irnc -Orng $< $@
    @$(if $(filter $@, $(SCHEMA_UPDATED_RNG)), cp $@ $(<:.rnc=.rng))
    @touch $@

# Made the single up to date file from one or both of the inputs. Derive the
# output in each format from it.

$(SCHEMA_DERIVED_RNG): $(SCHEMA_RNG_DIR)%.rng:  $(SCHEMA_XML_DIR)%.rng
    @mkdir -p $(dir $@)
    @cp $< $@

$(SCHEMA_DERIVED_RNC): $(SCHEMA_RNC_DIR)%.rnc:  $(SCHEMA_XML_DIR)%.rng
    @mkdir -p $(dir $@)
    @trang -Irng -Ornc $< $@


# Final result of all this, an accessor for a name of an up to date schema file
# in either format
get_schema_name = $(filter $1, $(SCHEMA_DERIVED_RNG)$(SCHEMA_DERIVED_RNC))

The premise is:

• use the double colon rules to deal with presence/absence of either of the possible source files
• update whichever source file was not used to update the derived file
• when both have changed, consistently throw away the one named secondary
• create output files in both formats from the one combining target

TODO:

• would rather do the empty file test without calling into shell
• this calls mkdir much more often than necessary

This solves my immediate problem of wanting both representations live at the same time and is a mere five years too late for the OP.

Answer 2

I don't see how that can work. Suppose it could be done in make; now you have two files foo.exe and foo.ini (you don't say what your actual filename patterns are). You run make and it sees that foo.exe is newer than foo.ini, so it decompiles the binary to build a new foo.ini. Now, you run make again and this time it sees that foo.ini is newer than foo.exe, because you just built the former, so it compiles foo.ini into foo.exe.

Etc. Every time you run make it will perform an operation on all the files because one or the other will always be out of date.

The only way this could work would be if you (a) tested to see if files did not have exactly identical time last modified times, and (b) had a way to reset the time on the compiled/decompiled file so that it was identical to the file it was built from, rather than "now" which is the default of course.

The answer is that make cannot be used for this situation. You could of course write yourself a small shell script that went through every file and tested that the last modified times were identical or not, and if not compiled them then used touch -m -r origin where origin is the file that had the newer modification time, so that both had the same modification time.