Using output piped from sed

Question

I have a sed command that is capturing a single line with sometext. The line in the file it is capturing ends with a linefeed. I am trying to utilize this variable in a pipeline, however, when I attempt to echo, or use it with other commands requiring an input, the result is a blank. Ex: sed '1,1!d' somefile.txt | echo "$1", I know the variable itself is not empty as I can replace echo "$1" with cat $1 and see the correct printout.

edit - I have tried piping to a tr -d and removing the newline. I have confirmed the newline character is gone, yet echos still show blank. Cats do not.

edit 2 - I piped the variable into an if statement ... | if [[ -z $1 ]]; then cat $1; fi it hits the if, is determined to be empty, so runs the cat, which prints a non-empty line to console. If the variable is empty why is cat still printing out information?

What is causing this inconsistency and how can I solve my problem? The ultimate goal is to run the output of one sed, through another to replace specific lines in a target file.

sed '1,1!d' somefile.txt | sed '2,1s/.*/'$1'/' targetfile.txt

Contents of somefile.txt:

these
are
words

Contents of targetfile.txt:

The next line should say these
This line should say these
The previous line should say these

Output of echo after sed:

<empty>

Output of cat after sed:

these

Output of 2nd sed, using input from 1st:

The next line should say these

the previous line should say these

Answer 1

You are confused about arguments and input data. Look at this:

$ echo "$1"

$ echo "foo" | if [[ -z $1 ]]; then cat $1; fi
foo

The first argument to my shell, $1 is empty so if [[ -z $1 ]] succeeds. The reason that cat $1 produces output is that you have a fundamental shell programming error in that statement - you aren't quoting your variable, $1. The correct syntax isn't cat $1, it's cat "$1". Look at the difference:

$ echo "foo" | if [[ -z $1 ]]; then cat "$1"; fi
cat: '': No such file or directory

We can simplify the code to make what's happening clearer:

$ echo "foo" | cat $1
foo

$ echo "foo" | cat "$1"
cat: '': No such file or directory

The reason that echo "foo" | cat $1 produces output is that the unquoted $1 is expanded by the shell to nothing before cat is called so that statement is equivalent to just echo "foo" | cat and so cat just copies the input coming in from the pipe to it's output.

On the other hand echo "foo" | cat "$1" generates an error because the shell expands "$1" to the null string before cat is called and so it's then asking cat to open a file named <null> and that of course does not exist, hence the error.

Always quote your shell variables unless you have a specific reason not to and fully understand all of the implications. Read a shell man page and/or google that if you're not sure what those implications are.

wrt another part of your code you have:

sed '1,1!d' somefile.txt | echo "$1"

but, unlike cat, echo neither reads it's input from a pipe nor from a file name passed as an argument. The input to echo is just the list of string arguments you provide it so while echo "foo" | cat will cause cat to read the input stream containing foo and output it, echo "foo" | echo will produce no output because echo isn't designed to read input from a pipe and so it'll just print a null string since you gave it no arguments.

It's not clear what you're really trying to accomplish but I think you might want to replace the 2nd line of targetfile.txt with the first line of somefile.txt. If so that's just:

awk '
    NR==FNR { if (NR==1) new=$0; next }
    FNR==2  { $0 = new }
    { print }
' somefile.txt targetfile.txt

Do not try to use sed to do it or you'll find yourself in escaping/quoting hell because, unlike awk, sed does not understand literal strings, see Is it possible to escape regex metacharacters reliably with sed.

Answer 2

You appear to want to extract the first line from file1 and use it to replace the second line in file2.

At the moment, you are extracting that value from the first file with your first sed but sending it to the second sed on its stdin rather than as a parameter ($1).

Your description is confusing so I will use this as file1:

File 1 Line 1
File 1 Line 2
File 1 Line 3

And this as file2:

File 2 Line 1
File 2 Line 2
File 2 Line 3

There are many ways to do this.

---

Method 1

# Extract line1 from file1
extracted=$(sed '1!d' file1)
# Replace line 2 in file2 with extracted value
sed "2s/.*/$extracted/" file2

Not sure why I feel like a dentist now :-)

If you want to put it all on one line, as some folks like to do:

x=$(sed '1!d' file1) && sed "2s/.*/$x/" file2

---

Method 2

This one is a bit tricky. It uses the first sed to write a script for the second sed:

sed  's/^/2s|.*|/;s/$/|/;q' file1 | sed -f /dev/stdin file2

If you look at the first sed on its own you will see it is generating a script for the second one:

sed  's/^/2s|.*|/;s/$/|/;q' file1
2s|.*|File 1 Line 1|

If you look at the second sed, you will see it is executing a script passed on its standard input:

sed -f /dev/stdin ...

---

Method 3

Easier still is awk:

awk 'FNR==NR{if(NR==1)saved=$0;next} FNR==2{$0=saved}1' file1 file2
File 2 Line 1
File 1 Line 1
File 2 Line 3

All you need to notice is that I am passing 2 files to awk and that FNR==NR inside the script means that awk is currently processing the first file, because FNR is the line number in the current file and NR is the total number of lines awk has processed from all files so far. So, when processing the second file, NR is greater than FNR by the number of lines in the first file.