CODEWITHSUNDEEP
bashsed
ddfrenk
ddfrenk

Reputation: 23

pass sed a long list of line numbers to remove from a file

I am trying to remove 500+ non-consecutive lines from a very large file with sed.

I have the lines stored in a list.txt file but I cant't use it in a for loop

for i in `cat list`; do  echo 'sed -i -e ' \'"$i"d\'' huge_file.txt' ; done

because line numbers in the original file would change every time sed removes one and exits.

I should do: 

sed -i -e '1d;2d;93572277d;93572278d; ......;nth '  huge_file.txt

Is there a way to pass that list to sed in a file?

Upvotes: 1

Views: 587

Answers (2)

RomanPerekhrest
RomanPerekhrest

Reputation: 92854

If it's just for a single particular task (not frequent) you may use the following GNU sed approach (assuming that numbers in list.txt are separated with newline \n):

sed -i "$(sed -z 's/\n/d;/g' list.txt)" huge_file.txt

Upvotes: 2

Kent
Kent

Reputation: 195079

you can try with awk:

awk -v s="2,3,..,n" 'BEGIN{n=split(s,t,",");for(i=1;i<=n;i++)d[t[i]]=1}
                    !d[NR]' huge.txt

You pass the comma-separated line numbers to awk by -v, in awk split it in array, and check each line, if the line number in the array, skip.

Test it with small file, if it worked as you expected, you can do:

awk -v '....' '....' huge.txt > tmp.txt && mv tmp.txt huge.txt

to write the change back to your original input file.

update

If you have 500 line numbers in another file, say, each number in a line, you can:

awk 'NR==FNR{a[$0]=1;next}!a[FNR]' ln.txt huge.txt

Upvotes: 3

Related Questions