CODEWITHSUNDEEP
perl
Scott Holtzman
Scott Holtzman

Reputation: 27259

perl Tie::File to remove lines in text file

I have the following code to remove lines out of a text file I am using. However, it is not actually removing the lines I want. I have done a lot of testing here and confirmed that it will remove the lines if I get the line number right. However, the $. seems to be duplicating the actual line number (as when I print on it, it says there are double the number of lines there actually are!)

#!/usr/bin/perl
# find_company_codes.pl

use strict;
use warnings;
use diagnostics;

use Tie::File;

my $split_file = "//server/folder/splits_table.txt";

# remove fields where only MAS has rollcall
my @file;
my @lines;

tie @file, 'Tie::File', $split_file or die $_;

foreach (@file) {

    @lines = split(",",$_);

    next if $lines[0] eq "ESDB Link Processing Reg-Prd-CCD";
    next if $lines[2] ne "";
    next if $lines[3] ne "";
    next if $lines[4] ne "";
    next if $lines[5] ne "";
    next if $lines[6] ne "";

    splice(@file, $.,1);

}

untie @file;

p.s. - I know I can use this, however, I thought using 'Tie::File` would be a little more straight to the point in this case.

Upvotes: 1

Views: 1911

Answers (1)

ikegami
ikegami

Reputation: 386331

You're not reading from a file, you're iterating over an array. $. makes no sense. First, we need to switch to indexes.

for (0..$#file) {
    my @fields = split(/,/, $file[$_]);

    next if $fields[0] eq "ESDB Link Processing Reg-Prd-CCD";
    next if $fields[2] ne "";
    next if $fields[3] ne "";
    next if $fields[4] ne "";
    next if $fields[5] ne "";
    next if $fields[6] ne "";

    splice(@file, $_, 1);
}

Except that doesn't work either. When you splice, you shift the entire array, so you must not advance the index that turn, and you need to finish the loop one index sooner. Fix:

my $i = 0;
while ($i < $#file) {
    my @fields = split(/,/, $file[$i]);

    if (   $fields[0] ne "ESDB Link Processing Reg-Prd-CCD"
        && $fields[2] eq ""
        && $fields[3] eq ""
        && $fields[4] eq ""
        && $fields[5] eq ""
        && $fields[6] eq ""
    ) {
        splice(@file, $i, 1);
    } else {
        ++$i;
    }
}

Good, that works, but it's CRAZY SLOW. Every splice can cause the rest of the file to be read and rewritten. Yet another inappropriate use of Tie::File! The following will be incredibly faster.

use strict;
use warnings;

my $qfn = "//rosefs19w/sal...";

my @file;
{
    open(my $fh, '<', $qfn)
        or die("Can't open $qfn: $!\n");
    @file = <$fh>;
}

my $i = 0;
while ($i < $#file) {
    my @fields = split(/,/, $file[$i]);

    if (   $fields[0] ne "ESDB Link Processing Reg-Prd-CCD"
        && $fields[2] eq ""
        && $fields[3] eq ""
        && $fields[4] eq ""
        && $fields[5] eq ""
        && $fields[6] eq ""
    ) {
        splice(@file, $i, 1);
    } else {
        ++$i;
    }
}

{
    open(my $fh, '>', $qfn)
        or die("Can't create $qfn: $!\n");
    print $fh @file;
}

That's right. Add 3 extra lines of code and remove 2, and your code becomes 100, 1000, 10000 times faster!

Upvotes: 3

Related Questions