Using GNU Parallel for cluster computing over LAN with rsync

Question

I have two machines, and I want to use GNU Parallel to have multiple processes 'cat' the contents of some text files from both machines.

I have the following setup.

On a local machine, in the same directory, I have the following files:

• cmd.sh - a bash file with contents: 'cat "$@"'
• test1.txt - a text file with contents: 'Test 1'
• test2.txt - a text file with contents: 'Test 2'
• test3.txt - a text file with contents: 'Test 3'

• nodefile - a text file with the following contents:

  2/:

  4/ dan@192.168.0.3

This is if I am using the nodefile example from wordpress link (below), and my IP is 192.168.0.2.

None of these files are replicated on the remote machine. I want to have multiple processes 'cat' the contents of each of the test?.txt files from both machines.

Preferably, this:

• Wouldn't leave any artifacts on the remote machine
• Would leave the contents of the local directory intact.

I have been able to execute multiprocessing commands remotely with the nodefile as per this wordpress example, but none involving file echoing remotely.

So far, I have something like the following:

parallel --sshloginfile nodefile --workdir . --basefile cmd.sh -a cmd.sh --trc ::: test1.txt test2.txt test3.txt

But this isn't working and is removing the files from my directory and not replacing them, as well as giving rsync errors. I (unfortunately) can't provide the errors at the moment, or replicate the setup.

I am very inexperienced with parallel, can anyone guide me on the syntax to accomplish this task? I haven't been able to find the answer (so far) in the man pages or on the web.

Running Ubuntu 16.04 LTS and using latest version of GNU Parallel.

Answer 1

You make a few mistakes:

• -a is used to give an input source. It is basically an alias for ::::
• you do not give the command to run after the options to GNU Parallel and before the :::
• --trc takes an argument (namely the file to transfer back). You do not have a file to transfer back, so use --transfer --cleanup instead.

So:

chmod +x cmd.sh
parallel --sshloginfile nodefile --workdir . --basefile cmd.sh --transfer --cleanup ./cmd.sh ::: test1.txt test2.txt test3.txt

It is unclear if you want to transfer anything to the remote machine, so maybe this is really the correct answer:

parallel --sshloginfile nodefile --nonall --workdir . ./cmd.sh test1.txt test2.txt test3.txt