Getting head to display all but the last line of a file: command substitution and standard I/O redirection

Question

I have been trying to get the head utility to display all but the last line of standard input. The actual code that I needed is something along the lines of cat myfile.txt | head -n $(($(wc -l)-1)). But that didn't work. I'm doing this on Darwin/OS X which doesn't have the nice semantics of head -n -1 that would have gotten me similar output.

None of these variations work either.

cat myfile.txt | head -n $(wc -l | sed -E -e 's/\s//g')
echo "hello" | head -n $(wc -l | sed -E -e 's/\s//g')

I tested out more variations and in particular found this to work:

cat <<EOF | echo $(($(wc -l)-1))
>Hola
>Raul
>Como Esta
>Bueno?
>EOF
3

Here's something simpler that also works.

echo "hello world" | echo $(($(wc -w)+10))

This one understandably gives me an illegal line count error. But it at least tells me that the head program is not consuming the standard input before passing stuff on to the subshell/command substitution, a remote possibility, but one that I wanted to rule out anyway.

echo "hello" | head -n $(cat && echo 1)

What explains the behavior of head and wc and their interaction through subshells here? Thanks for your help.

Answer 1

Here is a one-liner that can get you the desired output, and it can be used more generally for getting all lines from a file except the last n lines.

grep -n "" myfile.txt \ # output the line number for each line
| sort -nr \            # reverse the file by using those line numbers
| sed '1,4d' \          # delete first 4 lines (last 4 of the original file)
| sort -n \             # reverse the reversed file (correct the line order)
| sed 's/^[0-9]*://'    # remove the added line numbers

Here is the above command in an actual single line and runnable (can't execute the above due to the added comments):

grep -n "" myfile.txt | sort -nr | sed '1,4d' | sort -n | sed 's/^[0-9]*://'

It's a little cumbersome, and this problem can be solved with more comprehensive commands like ghead, but when you can't or don't want to download such tools, it's nice to be able to do this with the more basic options. I've been in situations where it's simply not an option to get better tools.

Answer 2

For Mac OS X specifically, I found an answer from a comment to this Q&A.

Assuming you are using Homebrew, run brew install coreutils then use the ghead command:

cat myfile.txt | ghead -n -1

Or, equivalently:

ghead -n -1 myfile.txt

Lastly, see brew info coreutils if you'd like to use the commands without the g prefix (e.g., head instead of ghead).

Answer 3

head is the wrong tool. If you want to see all but the last line, use:

sed \$d

The reason that

# Sample of incorrect code:
echo "hello" | head -n $(wc -l | sed -E -e 's/\s//g')

fails is that wc consumes all of the input and there is nothing left for head to see. wc inherits its stdin from the subshell in which it is running, which is reading from the output of the echo. Once it consumes the input, it returns and then head tries to read the data...but it is all gone. If you want to read the input twice, the data will have to be saved somewhere.

Answer 4

cat myfile.txt | echo $(($(wc -l)-1))

This works. It's overly complicated: you could just write echo $(($(wc -l)-1)) <myfile.txt or echo $(($(wc -l <myfile.txt)-1)). The problem is the way you're using it.

cat myfile.txt | head -n $(wc -l | sed -E -e 's/\s//g')

wc consumes all the input as it's counting the lines. So there is no data left to read in the pipe by the time head is started.

If your input comes from a file, you can redirect both wc and head from that file.

head -n $(($(wc -l <myfile.txt) - 1)) <myfile.txt

If your data may come from a pipe, you need to duplicate it. The usual tool to duplicate a stream is tee, but that isn't enough here, because the two outputs from tee are produced at the same rate, whereas here wc needs to fully consume its output before head can start. So instead, you'll need to use a single tool that can detect the last line, which is a more efficient approach anyway.

Conveniently, sed offers a way of matching the last line. Either printing all lines but the last, or suppressing the last output line, will work:

sed -n '$! p'
sed '$ d'

Answer 5

awk 'NR>1{print p}{p=$0}'

For this job, an awk one-liner is a bit longer than a sed one.

Answer 6

Using sed:

sed '$d' filename

will delete the last line of the file.

$ seq 1 10 | sed '$d' 
1
2
3
4
5
6
7
8
9

Answer 7

head -n -1 will give you all except the last line of its input.