CODEWITHSUNDEEP
bash
canadian_scholar
canadian_scholar

Reputation: 1325

Combining Directory of Text Files into CSV, One File Per Line

I have a large directory of text files, each rather complicated:

Say file1.txt:

Am no an listening depending up believing. Enough around remove to barton agreed regret in or it. Advantage mr estimable be commanded provision. Year well shot deny shew come now had. Shall downs stand marry taken his for out. Do related mr account brandon an up. Wrong for never ready ham these witty him. Our compass see age uncivil matters weather forbade her minutes. Ready how but truth son new under.

Am increasing at contrasted in favourable he considered astonished. As if made held in an shot. By it enough to valley desire do. Mrs chief great maids these which are ham match she. Abode to tried do thing maids. Doubtful disposed returned rejoiced to dashwood is so up.

And file2.txt:

Among going manor who did. Do ye is celebrated it sympathize considered. May ecstatic did surprise elegance the ignorant age. Own her miss cold last. It so numerous if he outlived disposal. How but sons mrs lady when. Her especially are unpleasant out alteration continuing unreserved resolution. Hence hopes noisy may china fully and. Am it regard stairs branch thirty length afford.

Blind would equal while oh mr do style. Lain led and fact none. One preferred sportsmen resolving the happiness continued. High at of in loud rich true. Oh conveying do immediate acuteness in he. Equally welcome her set nothing has gravity whether parties. Fertile suppose shyness mr up pointed in staying on respect.

What I need to do is to create a new file, say allfiles.txt that is:

Am no an listening depending up believing. Enough around remove to barton agreed regret in or it. Advantage mr estimable be commanded provision. Year well shot deny shew come now had. Shall downs stand marry taken his for out. Do related mr account brandon an up. Wrong for never ready ham these witty him. Our compass see age uncivil matters weather forbade her minutes. Ready how but truth son new under. Am increasing at contrasted in favourable he considered astonished. As if made held in an shot. By it enough to valley desire do. Mrs chief great maids these which are ham match she. Abode to tried do thing maids. Doubtful disposed returned rejoiced to dashwood is so up. 

Among going manor who did. Do ye is celebrated it sympathize considered. May ecstatic did surprise elegance the ignorant age. Own her miss cold last. It so numerous if he outlived disposal. How but sons mrs lady when. Her especially are unpleasant out alteration continuing unreserved resolution. Hence hopes noisy may china fully and. Am it regard stairs branch thirty length afford. Blind would equal while oh mr do style. Lain led and fact none. One preferred sportsmen resolving the happiness continued. High at of in loud rich true. Oh conveying do immediate acuteness in he. Equally welcome her set nothing has gravity whether parties. Fertile suppose shyness mr up pointed in staying on respect.

This file is just two lines in this case, the full text on each. I have searched the archives but cannot seem to find an implementation for this in bash.

Upvotes: 1

Views: 425

Answers (8)

glenn jackman
glenn jackman

Reputation: 246764

awk '
    FNR == 1 && FILENAME != ARGV[1] {print "\n"}
    {printf "%s",$0}
    END {print ""}
' *.txt > allfiles.txt

Upvotes: 1

potong
potong

Reputation: 58371

This might work for you:

for file in *.txt ;do paste -s "$file"; done | sed 's/^ *//;s/  */ /g'

Upvotes: 1

msw
msw

Reputation: 43487

Here's a pure INTERCAL implementation, no bash, tr, or cat required:

        PLEASE DO ,1 <- #1
        DO .4 <- #0
        DO .5 <- #0
        DO COME FROM (30)
        PLEASE ABSTAIN FROM (40)
        DO WRITE IN ,1
        DO .1 <- ,1SUB#1
        DO (10) NEXT
        PLEASE GIVE UP
(20)    PLEASE RESUME '?.1$#256'~'#256$#256'
(10)    DO (20) NEXT
        DO FORGET #1
        PLEASE DO .2 <- .4
        DO (1000) NEXT
        DO .4 <- .3~#255
        PLEASE DO .3 <- !3~#15'$!3~#240'
        DO .3 <- !3~#15'$!3~#240'
        DO .2 <- !3~#15'$!3~#240'
        PLEASE DO .1 <- .5
        DO (1010) NEXT
        DO .5 <- .2
        DO ,1SUB#1 <- .3
(30)    PLEASE READ OUT ,1
        PLEASE NOTE: having had pressing business at the local pub
(40)    the author got bored with this implementation

Upvotes: 4

gniourf_gniourf
gniourf_gniourf

Reputation: 46823

Here a pure solution: no cat, tr, awk, etc...

Besides, it will have a nicely output format: you won't get double spaces, or beginning or trailing spaces as with the methods provided in the other answers.

for f in *.txt; do
    # There are purposely no quotes for $(<"$f")
    echo $(<"$f")
    echo
done > newfile

The only caveat is if a file starts with -e, -E or -n: these characters won't be output: they will be slurped by echo considering it's an option. But I guess this is very unlikely to happen!

The trick is to use echo $l with no quotes!

Using this trick, here's how you can use cat in a funny way to achieve what you want (but this time it's not a pure solution): same thing, it's a funny no-use of quotes!

for f in *.txt; do
    # There are purposely no quotes for $(<"$f")
    cat <<< $(<"$f")
    echo
done > newfile

If you only have two files, say file1.txt and file2.txt you can do without a loop and a single cat command:

# there's purposely a lack of quotes
cat <<< $(<file1.txt)$'\n\n'$(<file2.txt) > newfile

or with a single echo (and same caveat as above), and pure :

# there's purposely a lack of quotes
echo $(<file1.txt)$'\n\n'$(<file2.txt) > newfile

Note. I added comments to specify that there are no quotes as every bash programmer should feel uncomfortable when reading these unquoted parts!

Note2. Can you do shorter?

Upvotes: 1

zmo
zmo

Reputation: 24812

touch allfiles.txt # create allfiles.txt
for f in *.txt; do # for each file of the current directory
    cat "$f" | tr '\n' ' ' >> allfiles.txt; # append the content of that file to allfiles.txt
    echo >> allfiles.txt # insert a new line
done

Upvotes: 5

Ansgar Wiechers
Ansgar Wiechers

Reputation: 200213

Combined Perl/bash solution:

for f in *.txt; do 
  perl -ne 'chomp; print "$_ "; END{ print "\n" }' "$f"
done > output.txt

Perl-only solution

#!/usr/bin/env perl

use strict;
use warnings;

foreach my $file (<*.txt>) {
  open FILE, "<$file" or die $!;
  while (<FILE>) {
    chomp;
    print "$_ ";
  }
  close FILE;
  print "\n";
}

Upvotes: 3

jaypal singh
jaypal singh

Reputation: 77085

With awk

awk 'FILENAME!=f&&NR>1{print "\n"}{FILENAME=f}1' ORS='' file1.txt file2.txt > allfiles.txt

Upvotes: 3

Barmar
Barmar

Reputation: 780798

for file in dir/* #Process all files in directory
do
   tr '\n' ' ' < "$file" # Remove newlines
   echo ''   # Add newline between files
done > newfile # Write all the output of the loop to the newfile

Upvotes: 5

Related Questions