CODEWITHSUNDEEP
makefilegnu-make
GrAnd
GrAnd

Reputation: 10221

Using different list elements for each usage of implicit recipe

There was a regular Makefile. Now I'm trying to enhance it to execute implicit recipe on different hosts via ssh. I have to do that using regular make/gmake as it's prohibited to install any 3rd-party packages on machines in that network.
Here is the concept:

SHELL := bash

SOURCES = $(wildcard *.in)
TARGETS = $(SOURCES:.in=.out)

TARGET_HOSTS := localhost foo bar
ONE_OF_TARGET_HOSTS := localhost

ifeq (${PROCESS_MODE},)
    PROCESS_MODE := LOCAL
endif


all: $(TARGETS)

ifeq (${PROCESS_MODE},LOCAL)

  %.out: %.in
        echo "Running $@ on host $(shell hostname)"
        echo $* > $@
        # some other stuff to run

else ifeq (${PROCESS_MODE},SSH)

  DONT_PASS_ENVIRONMENT_VARIABLES := BASH.*|DISPLAY|EC_.+|HOST|HOSTNAME|MACHTYPE|OSTYPE|PROCESS_MODE|PPID|PWD|SHELL|SHLVL|SHELLOPTS|SSH_.+|TZ
  SHELL_ENV := $(shell export -p | cut -b 12- | grep = | grep -Ev "^($(DONT_PASS_ENVIRONMENT_VARIABLES))")
  %.out: %.in
        echo "Pushing job $@ on remote host $(ONE_OF_TARGET_HOSTS)"
        @ssh $(ONE_OF_TARGET_HOSTS) 'cd "$(shell pwd)" && env $(SHELL_ENV) $(MAKE) PROCESS_MODE=LOCAL $@'

endif

It works in general. By default it executes everything locally (as it did before). If I run make PROCESS_MODE=SSH it executes rule for each .in file via ssh.
The problem is - right now it temporarily uses the same host (which is stored in ONE_OF_TARGET_HOSTS variable) for all ssh spawns. But I need to run each instance of rule on different hosts, which are defined in TARGET_HOSTS variable.

Let's say there are several files: a.in, b.in, c.in, d.in, e.in, etc. I want them to be processed as:

`a.in` - on localhost, 
`b.in` - on foo, 
`c.in` - on bar, 
`d.in` - on localhost, 
`e.in` - on foo,
 and so on...

(the actual files order/assignment does not matter actually; they simply should be different and use all hosts evenly)

Is that possible? Or maybe is there any other way to achieve that?

Upvotes: 0

Views: 97

Answers (2)

GrAnd
GrAnd

Reputation: 10221

Meanwhile I came out to this solution:

COUNT_TARGETS := $(words $(TARGETS))
# repeat hosts in the list to have there enough hosts for all targets
define expand-hosts
  $(eval COUNT_TARGET_HOSTS := $(words $(TARGET_HOSTS)))
  $(if $(filter $(shell test $(COUNT_TARGET_HOSTS) -lt $(COUNT_TARGETS) && echo $$?),0), \
    $(eval TARGET_HOSTS += $(TARGET_HOSTS)) \
    $(call expand-hosts) \
  )
endef
$(eval $(call expand-hosts))

# assign host to all targets
assign-hosts = $(1): SSH_TARGET_HOST = $(2)
$(foreach i, $(shell seq $(words $(TARGETS))), \
  $(eval $(call assign-hosts,$(word $(i), $(TARGETS)),$(word $(i), $(TARGET_HOSTS)))) \
)

...
%.out: %.in
    @echo "Pushing job $@ on remote host $(SSH_TARGET_HOST)"
    @ssh $(SSH_TARGET_HOST) 'cd "$(shell pwd)" && env $(SHELL_ENV) $(MAKE) -j1 PROCESS_MODE=LOCAL $@'

This way allows me to have all weird stuff outside the recipe.
But I will likely adopt the way of hosts list expansion from @RenaudPacalet answer.

Upvotes: 0

Renaud Pacalet
Renaud Pacalet

Reputation: 29270

In the following GNU make solution we first build a balanced list of [name.out]host tokens from the list of name.out targets. You can use any strings X and Y instead of [ and ], as long as for any name.out, Xname.outY is not a prefix of another token. Adapt to your situation.

In the recipe of %.out we recover the corresponding token with $(filter [$@]%,$(TOKEN_LIST)) and extract the hostname part with $(patsubst [$@]%,%,...). We assign it to shell variable host and use this variable in the echo and ssh commands.

There are 2 important aspects to remember:

  • make expands the recipe before passing it to the shell. So, when using the value of a shell variable we must write $$host instead of $host. After the make expansion it will become $host, what we want to pass to the shell, and no just ost.

  • Each line of a recipe is executed by a different shell. In order to use a shell variable in several recipe lines we must join them together with ; (or &&, as you wish) such that they become one single line, executed by one single shell. But for better readability we can use the line continuation (with a trailing \).

TOO_LONG_HOST_LIST := $(foreach t,$(TARGETS),$(TARGET_HOSTS))
HOST_LIST := $(wordlist 1,$(words $(TARGETS)),$(TOO_LONG_HOST_LIST))
TOKEN_LIST := $(join $(patsubst %,[%],$(TARGETS)),$(HOST_LIST))
# ...
%.out: %.in
    @host=$(patsubst [$@]%,%,$(filter [$@]%,$(TOKEN_LIST))); \
    echo "Pushing job $@ on remote host $$host"; \
    ssh $$host 'cd "$(CURDIR)" && env $(SHELL_ENV) $(MAKE) PROCESS_MODE=LOCAL $@'

Demo:

$ touch a.in b.in c.in d.in e.in f.in g.in
$ make TARGETS="a.out b.out c.out d.out e.out f.out g.out" TARGET_HOSTS="1 2 3"
Pushing job a.out on remote host 1
Pushing job b.out on remote host 2
Pushing job c.out on remote host 3
Pushing job d.out on remote host 1
Pushing job e.out on remote host 2
Pushing job f.out on remote host 3
Pushing job g.out on remote host 1

Note: remember that the $$ and the line continuations (the trailing ; \) are essential to guarantee the proper expansion of the shell variable host and its availability in all lines of the recipe.

Note: using the shell make function in a recipe, which is already a shell script, is almost always wrong; I replaced $(shell pwd) by the GNU make variable $(CURDIR).

Upvotes: 1

Related Questions