CODEWITHSUNDEEP
unixawkprintf
user2498657
user2498657

Reputation: 377

how to split a large csv file in unix command line

I am just splitting a very large csv file in to parts. When ever i run the following command. the doesn't completely split rather returns me the following error. how can i avoid the split the whole file. 

       awk -F, '{print > $2}' test1.csv 

       awk: YY1 makes too many open files
       input record number 31608, file test1.csv
       source line number 1

Upvotes: 8

Views: 5272

Answers (2)

PaulProgrammer
PaulProgrammer

Reputation: 17630

You must have a lot of lines. Are you sure that the second row repeats enough to put those records into an individual file? Anyway, awk is holding the files open until the end. You'll need a process that can close the file handles when not in use.

Perl to the rescue. Again.

#!perl    
while( <> ) {
    @content = split /,/, $_;
    open ( OUT, ">> $content[1]") or die "whoops: $!";
    print OUT $_;
    close OUT;
}

usage: script.pl your_monster_file.csv

outputs the entire line into a file named the same as the value of the second CSV column in the current directory, assuming no quoted fields etc.

Upvotes: 0

Chris Seymour
Chris Seymour

Reputation: 85785

Just close the files after writing:

awk -F, '{print > $2; close($2)}' test1.csv

Upvotes: 17

Related Questions