CODEWITHSUNDEEP
unixsed
Neha Singh
Neha Singh

Reputation: 73

Sed command in unix is not working on file size greater than 3GB

Sed command is not working in Unix for file size greater that 3GB, to remove carriage return and new line character from the file.

I am trying to remove the new line character from a pipe delimited file.

code:

sed -i ':a;N;$!ba;s/\n|/|/g'   File.txt

It's a pipe delimited file so I am first searching for the end of line then I am replacing the new line character with a pipe.

Sample Input :

Test|A|B|C
|D

After the replace - Expected Output :

Test|A|B|C|D

Sed command is working normally on all the files whose size is less than 3 GB, but not file files with bigger size.

Upvotes: 4

Views: 1543

Answers (1)

Thor
Thor

Reputation: 47089

The issue is that your sed command reads the whole file into memory first:

:a        # label a
N         # add next line to pattern space
$!ba      # if not on last line ($!) branch to label a

This will eventually fail with sufficiently large input, because pattern space is held in main memory.

Given the simple replacement you need, doing the replacement on pairs of lines would enough, e.g.:

:a        # label a
N         # add next line to pattern space
s/\n|/|/  # substitute new-line pipe by pipe
ta        # if last substitution was successful branch to label a
P         # print first line of pattern space
D         # delete first line of pattern space

As a one-liner:

sed -n ':a; N; s/\n|/|/; ta; P; D' File.txt

Upvotes: 3

Related Questions