CODEWITHSUNDEEP
bash
user3612121
user3612121

Reputation: 102

Concatenating text files in bash

I have many text files with only one line float value in one folder and I would like to concatenate them in bash in order for example: file_1.txt, file_2.txt ...file_N.txt. I would like to have them in one txt file in the order from 1 to N. Could someone please help me ? Here is the code I have but it just concatenates them in random manner. Thank you

for file in *.txt
do 
  cat ${file} >>  output.txt  
done

Upvotes: 3

Views: 1080

Answers (7)

Mark Setchell
Mark Setchell

Reputation: 207405

This works for me...

for i in $(seq 0 $N); do [[ -f file_$i.txt ]] && cat file_$i.txt; done > newfile

Or, more concisely

for i in $(seq 0 $N); do cat file_$i.txt 2> /dev/null ;done > newfile

Upvotes: 1

mklement0
mklement0

Reputation: 437197

Both solutions work well for the specific case at hand, but not generally in that they'll break with filenames with embedded spaces or other metacharacters (characters that, when used unquoted, have special meaning to the shell).

Here are solutions that work with filenames with embedded spaces, etc.:

Preferable solution for systems where sort -z and xargs -0 are supported (e.g., Linux, OSX, *BSD):

printf "%s\0" file_*.txt | sort -z -t_ -k2,2n  | xargs -0 cat > out.txt

Uses NUL (null character, 0x0) to separate the filenames and so safely preserves their boundaries.

This is the most robust solution, because it even handles filename with embedded newlines correctly (although such filenames are very rare in practice). Unfortunately, sort -z and xargs -0 are not POSIX-compliant.

POSIX-compliant solution, using xargs -I:

printf "%s\n" file_*.txt | sort -t_ -k2,2n  | xargs -I % cat % > out.txt

Processing is line-based, and due to use of -I, cat is invoked once per input filename, making this method slower than the one above.

Upvotes: 0

jrjc
jrjc

Reputation: 21873

for file in *.txt
do 
  cat ${file} >>  output.txt  
done

This works for me as well as :

for file in *.txt
do 
  cat $file >>  output.txt  
done

You don't need {}

But the simpler is still : 

cat file*.txt > output.txt

So if you have more than 9 files as suggested in the comment, you can do one of the following :

files=$(ls file*txt | sort -t"_" -k2g)
files=$(find . -name "file*txt" | sort -t "_" -k2g)
files=$(printf "%s\n" file_*.txt | sort -k1.6n) # Thanks to glenn jackman

and then:

cat $files

or 

cat $(find . -name "file*txt" | sort -t "_" -k2g)

Best is still to number your files correctly, so file_01.txt if you have less than 100 files, et file_001.txt if less than 1000, an so on.

example :

ls file*txt
file_1.txt  file_2.txt  file_3.txt  file_4.txt  file_5.txt  file_10.txt

They contain only their corresponding number.

$ cat $files
1
2
3
4
5
10

Upvotes: 2

David W.
David W.

Reputation: 107040

As others have pointed out, if you have files file_1, file_2, file_3... file_123283, the internal BASH sorting of these files will put file_11 before file_2 because they're sorted by text and not numerically.

You can use sort to get the order you want. Assuming that your files are file_#...

cat $(ls -1 file_* | sort -t_ -k2,2n)
  • The ls -1 lists your files out on one per line.
  • sort -t_ says to break the sorting fields down by underscores. This makes the second sorting field the numeric part of the file name.
  • -k2,2n says to sort by the second field numerically.

Then, you concatenate out all of the files together.

One issue is that you may end up filling up your command line buffer if you have a whole lot of files. Before cat can get the file names, the $(...) must first be expanded.

Upvotes: 1

glenn jackman
glenn jackman

Reputation: 246764

As much as I recommend against parsing the output of ls, here we go.

ls has a "version sort" option that will sort numbered files like you want. See below for a demo.

To concatenate, you want:

ls -v file*.txt | xargs cat > output

$ touch file{1..20}.txt
$ ls
file1.txt   file12.txt  file15.txt  file18.txt  file20.txt  file5.txt  file8.txt
file10.txt  file13.txt  file16.txt  file19.txt  file3.txt   file6.txt  file9.txt
file11.txt  file14.txt  file17.txt  file2.txt   file4.txt   file7.txt
$ ls -1
file1.txt
file10.txt
file11.txt
file12.txt
file13.txt
file14.txt
file15.txt
file16.txt
file17.txt
file18.txt
file19.txt
file2.txt
file20.txt
file3.txt
file4.txt
file5.txt
file6.txt
file7.txt
file8.txt
file9.txt
$ ls -1v
file1.txt
file2.txt
file3.txt
file4.txt
file5.txt
file6.txt
file7.txt
file8.txt
file9.txt
file10.txt
file11.txt
file12.txt
file13.txt
file14.txt
file15.txt
file16.txt
file17.txt
file18.txt
file19.txt
file20.txt

Upvotes: 4

Technext
Technext

Reputation: 8107

Use this:

find . -type f -name "file*.txt" | sort -V | xargs cat -- >final_file

If the files are numbered, then sorting doesn't happen in the natural way that we human expect. For that to happen, you will have to use -V option with sort command.

Upvotes: 1

user1830432
user1830432

Reputation: 690

You can use ls for listing files:

for file in `ls *.txt`
do·
  cat ${file} >>  output
done

Some sort techniques are discussed here: Unix's 'ls' sort by name

Upvotes: 0

Related Questions