Shell one liner to prepend to a file

Question

This is probably a complex solution.

I am looking for a simple operator like ">>", but for prepending.

I am afraid it does not exist. I'll have to do something like

 mv myfile tmp
 cat myheader tmp > myfile

Anything smarter?

Answer 1

Here's my one-liner with the filenames of a prepending file (file1) and a prepended file (file2) entered only once:

vi "+0r file1" +wq file2

Unfortunately, a tab completion doesn't work within double quotes.

Answer 2

The hack below was a quick off-the-cuff answer which worked and received lots of upvotes. Then, as the question became more popular and more time passed, people started reporting that it sorta worked but weird things could happen, or it just didn't work at all. Such fun.

I recommend the 'sponge' solution posted by user222 as Sponge is part of 'moreutils' and probably on your system by default. (echo 'foo' && cat yourfile) | sponge yourfile

The solution below exploits the exact implementation of file descriptors on your system and, because implementation varies significantly between nixes, it's success is entirely system dependent, definitively non-portable, and should not be relied upon for anything even vaguely important. Sponge uses the /tmp filesystem but condenses the task to a single command.

Now, with all that out of the way the original answer was:

---

Creating another file descriptor for the file (exec 3<> yourfile) thence writing to that (>&3) seems to overcome the read/write on same file dilemma. Works for me on 600K files with awk. However trying the same trick using 'cat' fails.

Passing the prependage as a variable to awk (-v TEXT="$text") overcomes the literal quotes problem which prevents doing this trick with 'sed'.

#!/bin/bash
text="Hello world
What's up?"

exec 3<> yourfile && awk -v TEXT="$text" 'BEGIN {print TEXT}{print}' yourfile >&3

Answer 3

The simplest solution I found is:

cat myHeader myFile | tee myFile

or

echo "<line to add>" | cat - myFile | tee myFile

Notes:

• Use echo -n if you want to append just a piece of text (not a full line).
• Add &> /dev/null to the end if you don't want to see the output (the generated file).
• This can be used to append a shebang to the file. Example:

  # make it executable (use u+x to allow only current user)
  chmod +x cropImage.ts
  # append the shebang
  echo '#''!'/usr/bin/env ts-node | cat - cropImage.ts | tee cropImage.ts &> /dev/null
  # execute it
  ./cropImage.ts myImage.png
  

Answer 4

for dash / ash:

echo "hello\n$(cat myfile)" > myfile

example:

$ echo "line" > myfile
$ cat myfile
line
$ echo "line1\n$(cat myfile)" > myfile
$ cat myfile
line1
line

Answer 5

You can use perl command line:

perl -i -0777 -pe 's/^/my_header/' tmp

Where -i will create an inline replacement of the file and -0777 will slurp the whole file and make ^ match only the beginning. -pe will print all the lines

Or if my_header is a file:

perl -i -0777 -pe 's/^/`cat my_header`/e' tmp

Where the /e will allow an eval of code in the substitution.

Answer 6

A solution with printf:

new_line='the line you want to add'
target_file='/file you/want to/write to'

printf "%s\n$(cat ${target_file})" "${new_line}" > "${target_file}"

You could also do:

printf "${new_line}\n$(cat ${target_file})" > "${target_file}"

But in that case you have to be sure there aren’t any % anywhere, including the contents of target file, as that can be interpreted and screw up your results.

Answer 7

Quick and dirty, buffer everything in memory with python:

$ echo two > file
$ echo one | python -c "import sys; f=open(sys.argv[1]).read(); open(sys.argv[1],'w').write(sys.stdin.read()+f)" file
$ cat file
one
two
$ # or creating a shortcut...
$ alias prepend='python -c "import sys; f=open(sys.argv[1]).read(); open(sys.argv[1],\"w\").write(sys.stdin.read()+f)"'
$ echo zero | prepend file
$ cat file
zero
one
two

Answer 8

Using a bash heredoc you can avoid the need for a tmp file:

cat <<-EOF > myfile
  $(echo this is prepended)
  $(cat myfile)
EOF

This works because $(cat myfile) is evaluated when the bash script is evaluated, before the cat with redirect is executed.

Answer 9

If you have a large file (few hundred kilobytes in my case) and access to python, this is much quicker than cat pipe solutions:

python -c 'f = "filename"; t = open(f).read(); open(f, "w").write("text to prepend " + t)'

Answer 10

IMHO there is no shell solution (and will never be one) that would work consistently and reliably whatever the sizes of the two files myheader and myfile. The reason is that if you want to do that without recurring to a temporary file (and without letting the shell recur silently to a temporary file, e.g. through constructs like exec 3<>myfile, piping to tee, etc.

The "real" solution you are looking for needs to fiddle with the filesystem, and so it's not available in userspace and would be platform-dependent: you're asking to modify the filesystem pointer in use by myfile to the current value of the filesystem pointer for myheader and replace in the filesystem the EOF of myheader with a chained link to the current filesystem address pointed by myfile. This is not trivial and obviously can not be done by a non-superuser, and probably not by the superuser either... Play with inodes, etc.

You can more or less fake this using loop devices, though. See for instance this SO thread.

Answer 11

EDIT: This is broken. See Weird behavior when prepending to a file with cat and tee

The workaround to the overwrite problem is using tee:

cat header main | tee main > /dev/null

Answer 12

With $( command ) you can write the output of a command into a variable. So I did it in three commands in one line and no temp file.

originalContent=$(cat targetfile) && echo "text to prepend" > targetfile && echo "$originalContent" >> targetfile

Answer 13

If you're scripting in BASH, actually, you can just issue:

cat - yourfile  /tmp/out && mv /tmp/out yourfile

That's actually in the Complex Example you yourself posted in your own question.

Answer 14

Mostly for fun/shell golf, but

ex -c '0r myheader|x' myfile

will do the trick, and there are no pipelines or redirections. Of course, vi/ex isn't really for noninteractive use, so vi will flash up briefly.

Answer 15

If you need this on computers you control, install the package "moreutils" and use "sponge". Then you can do:

cat header myfile | sponge myfile

Answer 16

This still uses a temp file, but at least it is on one line:

echo "text" | cat - yourfile > /tmp/out && mv /tmp/out yourfile

Credit: BASH: Prepend A Text / Lines To a File

Answer 17

sed -i -e "1s/^/new first line\n/" old_file.txt

Answer 18

WARNING: this needs a bit more work to meet the OP's needs.

There should be a way to make the sed approach by @shixilun work despite his misgivings. There must be a bash command to escape whitespace when reading a file into a sed substitute string (e.g. replace newline characters with '\n'. Shell commands vis and cat can deal with nonprintable characters, but not whitespace, so this won't solve the OP's problem:

sed -i -e "1s/^/$(cat file_with_header.txt)/" file_to_be_prepended.txt

fails due to the raw newlines in the substitute script, which need to be prepended with a line continuation character () and perhaps followed by an &, to keep the shell and sed happy, like this SO answer

sed has a size limit of 40K for non-global search-replace commands (no trailing /g after the pattern) so would likely avoid the scary buffer overrun problems of awk that anonymous warned of.

Answer 19

I think this is the cleanest variation of ed:

cat myheader | { echo '0a'; cat ; echo -e ".\nw";} | ed myfile

as a function:

function prepend() { { echo '0a'; cat ; echo -e ".\nw";} | ed $1; }

cat myheader | prepend myfile

Answer 20

sed -i -e '1rmyheader' -e '1{h;d}' -e '2{x;G}' myfile

Answer 21

Like Daniel Velkov suggests, use tee.
To me, that's simple smart solution:

{ echo foo; cat bar; } | tee bar > /dev/null

Answer 22

When you start trying to do things that become difficult in shell-script, I would strongly suggest looking into rewriting the script in a "proper" scripting language (Python/Perl/Ruby/etc)

As for prepending a line to a file, it's not possible to do this via piping, as when you do anything like cat blah.txt | grep something > blah.txt, it inadvertently blanks the file. There is a small utility command called sponge you can install (you do cat blah.txt | grep something | sponge blah.txt and it buffers the contents of the file, then writes it to the file). It is similar to a temp file but you dont have to do that explicitly. but I would say that's a "worse" requirement than, say, Perl.

There may be a way to do it via awk, or similar, but if you have to use shell-script, I think a temp file is by far the easiest(/only?) way..

Answer 23

John Mee: your method is not guaranteed to work, and will probably fail if you prepend more than 4096 byte of stuff (at least that's what happens with gnu awk, but I suppose other implementations will have similar constraints). Not only will it fail in that case, but it will enter an endless loop where it will read its own output, thereby making the file grow until all the available space is filled.

Try it for yourself:

exec 3<>myfile && awk 'BEGIN{for(i=1;i<=1100;i++)print i}{print}' myfile >&3

(warning: kill it after a while or it will fill the filesystem)

Moreover, it's very dangerous to edit files that way, and it's very bad advice, as if something happens while the file is being edited (crash, disk full) you're almost guaranteed to be left with the file in an inconsistent state.

Answer 24

Here's what I discovered:

echo -e "header \n$(cat file)" >file

Answer 25

variables, ftw?

NEWFILE=$(echo deb http://mirror.csesoc.unsw.edu.au/ubuntu/ $(lsb_release -cs) main universe restricted multiverse && cat /etc/apt/sources.list)
echo "$NEWFILE" | sudo tee /etc/apt/sources.list

Answer 26

I'm liking @fluffle's ed approach the best. After all, any tool's command line switches vs scripted editor commands are essentially the same thing here; not seeing a scripted editor solution "cleanliness" being any lesser or whatnot.

Here's my one-liner appended to .git/hooks/prepare-commit-msg to prepend an in-repo .gitmessage file to commit messages:

echo -e "1r $PWD/.gitmessage\n.\nw" | ed -s "$1"

Example .gitmessage:

# Commit message formatting samples:
#       runlevels: boot +consolekit -zfs-fuse
#

I'm doing 1r instead of 0r, because that will leave the empty ready-to-write line on top of the file from the original template. Don't put an empty line on top of your .gitmessage then, you will end up with two empty lines. -s suppresses diagnostic info output of ed.

In connection with going through this, I discovered that for vim-buffs it is also good to have:

[core]
        editor = vim -c ':normal gg'

Answer 27

A variant on cb0's solution for "no temp file" to prepend fixed text:

echo "text to prepend" | cat - file_to_be_modified | ( cat > file_to_be_modified ) 

Again this relies on sub-shell execution - the (..) - to avoid the cat refusing to have the same file for input and output.

Note: Liked this solution. However, in my Mac the original file is lost (thought it shouldn't but it does). That could be fixed by writing your solution as: echo "text to prepend" | cat - file_to_be_modified | cat > tmp_file; mv tmp_file file_to_be_modified

Answer 28

The one which I use. This one allows you to specify order, extra chars, etc in the way you like it:

echo -e "TEXTFIRSt\n$(< header)\n$(< my.txt)" > my.txt

P.S: only it's not working if files contains text with backslash, cause it gets interpreted as escape characters

Answer 29

current=`cat my_file` && echo 'my_string' > my_file && echo $current >> my_file

where "my_file" is the file to prepend "my_string" to.

Answer 30

Why not simply use the ed command (as already suggested by fluffle here)?

ed reads the whole file into memory and automatically performs an in-place file edit!

So, if your file is not that huge ...

# cf. "Editing files with the ed text editor from scripts.",
# http://wiki.bash-hackers.org/doku.php?id=howto:edit-ed

prepend() {
   printf '%s\n' H 1i "${1}" . wq | ed -s "${2}"
}

echo 'Hello, world!' > myfile
prepend 'line to prepend' myfile

Yet another workaround would be using open file handles as suggested by Jürgen Hötzel in Redirect output from sed 's/c/d/' myFile to myFile

echo cat > manipulate.txt
exec 3<manipulate.txt
# Prevent open file from being truncated:
rm manipulate.txt
sed 's/cat/dog/' <&3 > manipulate.txt

All this could be put on a single line, of course.