CODEWITHSUNDEEP
regexperldelete-row
chinmoy khaund
chinmoy khaund

Reputation: 119

perl to delete consecutive duplicate lines

I want to delete the consecutive duplicate lines. i.e. for example

**test.txt**
car
speed is good
bike 
slower than car
plane
super fast
super fast
bullet train 
super fast

This removes all the duplicate lines except the 1st occurance.

perl -ne 'print unless $a{$_}++'

But i want the ouput to be

    **test.txt**
    car
    speed is good
    bike 
    slower than car
    plane
    super fast
    bullet train 
    super fast

I tried this oneliner but this doesnt do anything but just prints the input.

perl -00 -F'<\w+>|</\w+>' -i.bak -lane 'foreach(@F){if ($_=~/\w+/ && ($a ne $_)){print "$_";$a=$_;}}'

How to do this???

Upvotes: 3

Views: 1788

Answers (4)

mivk
mivk

Reputation: 14824

I also wanted to keep track of how many duplicates were suppressed and only skip consecutive duplicates.

While this is not exactly what the OP asked, it is a variant which others may find useful:

perl -ne 'if (defined($pr) && ($_ eq $pr)) {$cnt++;} else {print "... (+$cnt)\n" if ($cnt); print; $cnt=0; $pr=$_;}'

It produced something like this with my data (a database restore log): 

COPY 9
COPY 0
... (+2)
COPY 5
COPY 0
... (+1)
COPY 24
ALTER TABLE
... (+23)
CREATE INDEX
... (+73)

Upvotes: 0

Steve
Steve

Reputation: 54392

Why don't you just use uniq?

uniq file.txt

Results:

car
speed is good
bike 
slower than car
plane
super fast
bullet train 
super fast

You can also do this with awk:

awk 'line != $0; { line = $0 }' file.txt

Upvotes: 10

Vipul Ved Prakash
Vipul Ved Prakash

Reputation: 96

$ perl -ne 'print $_ unless $_ eq $prev; $prev = $_'

Upvotes: 5

epsalon
epsalon

Reputation: 2294

Try:

perl -ne 'print unless (defined($prev) && ($_ eq $prev)); $prev=$_'

Upvotes: 5

Related Questions