CODEWITHSUNDEEP
bashshellsed
Shivanshu
Shivanshu

Reputation: 894

Removing current line in bash during read

I'm using bash to read a file and after doing opeation on particular line ,i need to delete that line from input file.

Can you please suggest some way to do so using sed or any other way ?

i've tried using sed command like this :-

#!/bin/sh

file=/Volumes/workplace/GeneratedRules.ion

while read line;do
printf "%s\n" "$line"
sed  '1d' $file
done <$file

my aim in this program is to read one line and then deleting it.

Input :-

AllIsWell
LetsHopeForBest
YouCanMakeIt

but the output , i got is more weird than i thought.

output :-

AllIsWell
LetsHopeForBest
YouCanMakeIt
LetsHopeForBest
LetsHopeForBest
YouCanMakeIt
YouCanMakeIt
LetsHopeForBest
YouCanMakeIt

but i need to output as :

AllIsWell 
LetsHopeForBest
YouCanMakeIt

as i want to delete line after reading it.

NOTE :- i have simplified my problem here . The actual usecase is :-

I need to perform some bunch of operation on line except reading that and the input file is too long and my operation got fails in some way in between .So i want those lines which i have read to be deleted so that if i start the process again , it will not start from the beginning but at the point where it got stuck.

please help.

Upvotes: 0

Views: 2674

Answers (3)

dawg
dawg

Reputation: 103884

As stated in comments, there are many issues with altering the file you are currently reading from. Don't do it.

You could just keep track of which lines you have dealt with in the first loop (with a counter) then use sed to delete those lines after your first loop has processed them.

A simple example:

cd /tmp

echo 'Line 1
Line 2
Line 3
Line 4
Line 5' >file

echo "file before:"
cat file

cnt=1
while IFS= read -r line || [[ -n $line ]]; do 
    printf "'%s' processed\n" "$line"; 
    if [ "$cnt" -ge 3 ]; then
        break
    fi  
    let "cnt+=1"
done <file

sed -i '' "1,${cnt}d" file 

echo "file after:"
cat file

Prints:

file before:
Line 1
Line 2
Line 3
Line 4
Line 5
'Line 1' processed
'Line 2' processed
'Line 3' processed
file after:
Line 4
Line 5

Another method is to use something like awk, ruby or perl to 'slurp' the file and then feed that slurpped file line-by-line to your Bash while loop. The file can then be modified in your loop since the other process has already fully read and closed the file:

# Note: This is SLOWER and USES MORE MEMORY...
echo "file before:"
cat file

while IFS= read -r line; do 
    printf "'%s' processed\n" "$line"; 
    sed -i '' "1d" file
done < <(awk -v cnt=3 'NR>cnt{next} 
            {arr[NR]=$0}
        END { for(i=1;i<=cnt;i++) print arr[i] }' file)
echo "file after:"
cat file

# same output

Note:

Please make sure you polish up on how to use bash to read a stream line-by-line for less surprises.

Read HERE and HERE for more.

Upvotes: 0

Jeff Holt
Jeff Holt

Reputation: 3190

You effectively said you want your process to be restartable. Depending upon how you define the successful completion of an iteration of your while loop, you should store a line number in a separate file, x, that indicates how many lines you have successfully processed. If the file doesn't exist, then assume you would start reading at line 1.

Otherwise, you would get the content of x into variable n and then you would start reading $file at line $n + 1.

How you start reading at a particular line depends on constraints we don't know yet.

One way you could do it is to use sed to put lines $n + 1 into a temporary file, remove $file and then move the temporary file to $file before your while loop begins.

There are other solutions but each one might not elegantly satisfy your constraints.

But you'll want to carefully consider what happens if some other process is modifying the content of $file and when it is changing the content. If it only changes the content before or after your bash script is running, then you're probably ok to continue down this path. Otherwise, you have a synchronization problem to solve.

Upvotes: 3

Blackdoor
Blackdoor

Reputation: 982

It needs a option -i, and if you need backup the file, just assign a suffix to the option. see the man sed

#!/bin/sh

file=/Volumes/workplace/GeneratedRules.ion

while read line;do
printf "%s\n" "$line"
sed -i '1d' $file
done <$file

Upvotes: -1

Related Questions