CODEWITHSUNDEEP
makefilegnu-make
nos
nos

Reputation: 20870

How to handle 'argument list too long' in Makefile recipe?

I use Makefile to control workflow dependency, and the following code causes 'argument too long' error. What would be a good workaround? I am using 3.82 BTW.

make: execvp: /bin/sh: Argument list too long

Here the logic is to merge all the prerequisites if they are all done (each prerequisite file would have a .done sentinel file if it is done/ready).

contains=$(foreach v,$2,$(if $(findstring $1,$v),$v,))

.SECONDEXPANSION:
%.merged: $$(call contains,_$$*_,$$(STEP1s))
    if (for x in $^; do [ -f $$x.done ] || exit; done); then \
        for f in $^; do cat $$f; echo; done > $@; fi

EDIT: The following recipe doesn't trigger 'argument too long' error.

for x in $^;do echo $$x;done

Upvotes: 0

Views: 3453

Answers (3)

nos
nos

Reputation: 20870

Thank you very much for pointing me to the right direction.

I think the problem is that after recipe expansion, if (for x in $^; do [ -f $$x.done ] || exit; done); becomes too long

I ended up with this solution, which is somewhat cheating. Here I ignore the prerequisites, and re-compute the prerequisites' sentinels. Then compare their count to an expected value

contains=$(foreach v,$2,$(if $(findstring $1,$v),$v,))

.SECONDEXPANSION:
%.merged: $$(call contains,_$$*_,$$(STEP1s))
    if [ $(words $(wildcard *_$(basename $@)_*.txt.done)) == $(n_input) ]; then\
        echo -n $^| xargs -I{} -d' ' sh -c "cat {}; echo " > $@;fi

Upvotes: 0

Renaud Pacalet
Renaud Pacalet

Reputation: 29250

Edit: From the various comments it seems that it is the recipe length itself that causes the error, not the length of the arguments of the for loops. A way to break it in much smaller pieces would be to introduce intermediate targets, forbid parallel execution, and run make twice, a first time to decide if the target shall be rebuilt and the second time to build it with as many different recipes as there are files to concatenate (not tested):

.NOTPARALLEL:

contains=$(foreach v,$2,$(if $(findstring $1,$v),$v$3,))

.SECONDEXPANSION:
ifneq ($(STEM),)
.PHONY: $$(call contains,_$$(STEM)_,$$(STEP1s),.cat)

$(STEM).merged: $$(call contains,_$$(STEM)_,$$(STEP1s),.cat)

%.cat:
    cat $* >> $(STEM).merged
    echo >> $(STEM).merged
else
%.merged: $$(call contains,_$$*_,$$(STEP1s),) $$(call contains,_$$*_,$$(STEP1s),.done)
    $(MAKE) $@ STEM=$*
endif

First version: Apparently $^ is too large for the shell. Here is a solution that uses make macros instead of shell loops and that declares the .done files as regular prerequisites instead of testing their existence in the recipe:

contains=$(foreach v,$2,$(if $(findstring $1,$v),$v $v.done,))

.SECONDEXPANSION:
%.merged: $$(call contains,_$$*_,$$(STEP1s))
    $(foreach f,$(filter-out,%.done,$^),cat "$f" >> "$@"; echo >> "$@";)

Upvotes: 1

MadScientist
MadScientist

Reputation: 100926

I don't think Renaud's suggestion meets the stated requirements, which as I understand it is the output is only generated if ALL the prerequisites have a .done file. Also the shell script generated is much longer than needed.

Maybe this will work:

contains=$(foreach v,$2,$(if $(findstring $1,$v),$v $v.done,))

.SECONDEXPANSION:
%.merged: $$(call contains,_$$*_,$$(STEP1s))
        $(if $(foreach $F,$^,$(if $(wildcard $F.done),,x)),, \
          for fn in $^; do (cat "$$fn"; echo) >> "$@"; done)

You might be able to make this more efficient by avoiding the foreach loop; if you can require GNU make 4.4 you can use intcmp to compare the lengths:

$(if $(intcmp $(words $^),$(words $(wildcard $(addsuffix .done,$^)))),, \
    ...dostuff...)

If not you can probably do the same thing with filtering:

$(if $(filter $(words $^),$(words $(wildcard $(addsuffix .done,$^)))), \
    ...dostuff...)

Note, none of this is tested.

Upvotes: 1

Related Questions