CODEWITHSUNDEEP
bashshellstdinpastecut
Dargor
Dargor

Reputation: 653

Pipe a lot of files to stdin, extract first columns, then combine those in a new file

Suppose we have these two files:

$ cat ABC.txt 
ABC DEF

$ cat PQR.txt 
PQR XTZ

And we want to form a new file with the 1st column of each file. This can be achieved by:

$ paste -d ' ' <(cut -d ' ' -f 1 ABC.txt) <(cut -d ' ' -f 1 PQR.txt )
ABC PQR

But I want to use this with tons of files in the input, not only ABC.txt and PQR.TXT, but a lot of them. How we can generalize this situation to pass each file in the collection to cut and then pass all the outputs to paste (I know that this may be done better with awk but I want to know how to solve this using this approach).

Edit 1

I have discovered a dirty, dirty way of doing this:

$ str='';  for i in *.txt; \
           do    str="${str} <(cut -d ' ' -f 1 ${i})"; \
           done ; \
  str="paste -d ' ' $str"; \
  eval $str

But please, free my soul with an answer that does not involve going to Computer Science Hell.

Edit 2

Each file can have n rows, if this matters.

Upvotes: 7

Views: 558

Answers (3)

webb
webb

Reputation: 4340

After a closer look, I see that @that-other-guy's answer is awesome, but here also is another dirty dirty way that's roughly the same under the hood.

eval "paste -d' ' "$(find *.txt -printf " <(cut -d' ' -f1 '%f')")

Upvotes: 1

agc
agc

Reputation: 8406

Given space delimited input files, and provided ':' is a safe delimiter, (i.e. if there are no colons in the input), this paste to sed one-liner works:

paste -d':' *.txt | sed 's/ [^:]*$//;s/ [^:]*:*/ /g;s/://g'

(POSIX, with no eval, exec, bashisms, subshells, or loops.)

Upvotes: 1

that other guy
that other guy

Reputation: 123570

Process substitution <(somecommand) doesn't pipe to stdin, it actually opens a pipe on a separate file descriptor, e.g. 63, and passes in /dev/fd/63. When this "file" is opened, the kernel* duplicates the fd instead of opening a real file.

We can do something similar by opening a bunch of file descriptors and then passing them to the command:

# Start subshell so all files are automatically closed
(
  fds=()
  n=0
  # Open a new fd for each process subtitution
  for file in ./*.txt
  do
    exec {fds[n++]}< <(cut -d ' ' -f 1 "$file")
  done

  # fds now contain a list of fds like 12 14
  # prepend "/dev/fd/" to all of them
  parameters=( "${fds[@]/#//dev/fd/}" )

  paste -d ' ' "${parameters[@]}"
)

{var}< file is bash's syntax for dynamic file descriptor assignment. like var=4; exec 4< file; but without having to hardcode the 4 and instead let bash pick a free file descriptor. exec opens it in the current shell.

* Linux, FreeBSD, OpenBSD and XNU/OSX anyways. This is not POSIX, but neither is <(..)

Upvotes: 5

Related Questions