CODEWITHSUNDEEP
linuxfileunixcommand-lineubuntu
Legend
Legend

Reputation: 116810

How do I insert a new line before concatenating?

I have about 80000 files which I am trying to concatenate. This one:

cat files_*.raw >> All

is extremely fast whereas the following:

for f in `ls files_*.raw`; do cat $f >> All; done;

is extremely slow. Because of this reason, I am trying to stick with the first option except that I need to be able to insert a new line after each file is concatenated to All. Is there any fast way of doing this?

Upvotes: 2

Views: 1141

Answers (4)

Michał Šrajer
Michał Šrajer

Reputation: 31182

Each time awk opens another file to process, the FRN equals 0, so:

awk '(0==FRN){print ""} {print}' files_*.raw >> All

Note, it's all done in one awk process. Performance should be close to the cat command from the question.

Upvotes: 1

Soren
Soren

Reputation: 14688

What about 

ls files_*.raw | xargs -L1 sed -e '$s/$/\n/' >>ALL

That will insert an extra newline at the end of each file as you concat them.

And a parallel version if you don't care about the order of concatenation:

find ./ -name "*.raw" -print | xargs -n1 -P4 sed -e '$s/$/\n/' >>All

Upvotes: 3

ritesh
ritesh

Reputation: 992

The second command might be slow because you are opening the 'All' file for append 80000 times vs. 1 time in the first command. Try a simple variant of the second command:

for f in `ls files_*.raw`; do cat $f ; echo '' ; done >> All

Upvotes: 2

trojanfoe
trojanfoe

Reputation: 122391

I don't know why it would be slow, but I don't think you have much choice:

for f in `ls files_*.raw`; do cat $f >> All; echo '' >> All; done

Upvotes: 1

Related Questions