For each line of input, emit that line alongside output from passing it to a command

Question

I have an input file with line per value, like the following:

100
200

I intend to pass each input value to an arbitrary command (here, abc), and generate an output file which has both the input and the associated output alongside each other.

Assuming that my command abc transforms 100 to tada and 200 to jjhu (the real production version will do something different), my intended output is:

100 tada
200 jjhu

Currently, I'm trying to use the following command:

cat jobs.txt | xargs -n 1 -I {} sh -c "echo {}; abc {}"

...but its output is:

100
tada
200
jjhu

How can this be corrected, to put the output on the same line as the input?

Answer 1

The simpler solution adhering to the OP question would be:

cat jobs.txt | xargs -n 1 -I {} sh -c 'echo -n "{} "; abc {}'

As explained alreay in this page, the problem was that the echo command outputs a trailing newline, if not instructed to avoid it. The "-n" argument to echo does that.

It is said that xargs is not good for this. May be, however it does the job:

File jobs.txt

100
200

File abc

echo "a$1/"

/tmp$ cat jobs.txt | xargs -n 1 -I {} sh -c 'echo -n "{} "; ./abc {}'
100 a100/
200 a200/
/tmp$

UPDATE from comment below (plus other miscellaneous things in this page): this code it's not safe, being prone to data injection. Moreover, the -n 1 is redundant and, maybe, there can be other issues. So, even if this answer is THE correct reply to the OP question:

How can this be corrected, to put the output on the same line as the input?

it is strongly adviced to NOT USE this code.

Answer 2

As has been stated, xargs is not the right tool for the job:

• Charles Duffy's helpful answer shows a pure shell solution that is easy to extend, but suitable only for smallish files (shell loops are slow).
  It is the right solution if you need shell functionality for every command, as the OP does, such as when calling 2 commands per input argument.

• If echoing values is all you need (which turned out to be not the OP's use case, but that wasn't initially obvious), Jonathan Leffler's helpful answer shows a sed command that is a generally faster alternative for larger files.

As for why your xargs command didn't work:

The only immediate problem was that echo terminates its output with a trailing newline, which you can portably work around with printf.

Also, -n 1 is redundant, because it is overruled by -I {}, which always passes 1 argument and uses every input line as a whole as an argument, substituting all occurrences (or, per POSIX, up to at least 5) of placeholder {} with that one argument.

Finally, it's generally more robust and safer to pass {} as a positional parameter to the shell (sh) and double-quote its use in the shell script.
This prevents both unintended modification of the argument and malicious attempts to inject commands.

If we put it all together, we get:

xargs -I {} sh -c 'printf "%s " "$1"; abc "$1"' - {} < jobs.txt

Note the dummy parameter -, which is there because the first argument binds to $0 inside the shell, not $1.

Charles points out that letting the script passed to sh -c handle the looping over the input obviates the need for -I {} / -n 1 altogether and also greatly improves performance, because (typically) only a single sh process is needed.

xargs sh -c 'for a; do printf "%s " "$a"; abc "$a"; done' - < jobs.txt

^{Generally, if the input lines had embedded whitespace and you still wanted to treat each line as a single argument, you'd have to use -d'\n', but note that that won't work with BSD/macOS xargs.}

Note that this in effect an inline variant of Charles' own answer.

Answer 3

Revised question

For the revised question, you can do one of a couple of things. One is to use Charles Duffy's answer.

Another option is to use paste (which was what I suggested in my earliest answer). Assuming that the command abc is benighted enough to only accept a single argument at a time (that's a pain; are you sure you can't get it upgraded to handle multiple arguments in a single invocation?), then you might use:

while read -r line
do
    abc "$line"
done < jobs.txt | paste jobs.txt -

If the command accepts multiple arguments and generates one line of output per argument, then you could use:

xargs abc < jobs.txt | paste jobs.txt -

If the command generates multiple lines of output per argument, then you need to fall back on:

while read -r line
do echo "$line" $(abc "$line")
done < jobs.txt

as the argument processing before echo is invoked will flatten the output from abc to a single line. There are carefully and deliberately no quotes around the outside of the $(…) here. If the output contains characters that can screw up shell scripts, you have to work harder — it too is doable (echo "$line $(abc "$line" | tr '\n' ' ')", for instance).

Which of these many options is best depends on details that have not been revealed — any of them could be correct for your task.

Previous version of the question

It appeared that the input file was:

100
200

and the desired output was:

100 abc100
200 abc200

You can use sed to achieve this result with ease:

sed 's/.*/& abc&/' jobs.txt

Replace the contents of each line read with that line, a space, the letters abc and the line again.

Answer 4

Don't use xargs for this task at all. It's inefficient with -n 1 (running a new and separate shell process for each line of input), and opens you up to security vulnerabilities (consider if your 100 were $(rm -rf $HOME) with some escaping to make it act as a single word).

The following can run within a single shell instance, assuming that your input file has one value to each line:

while IFS= read -r line; do
  printf '%s %s\n' "$line" "$(abc "$line")"
done <jobs.txt