CODEWITHSUNDEEP
python
ewok
ewok

Reputation: 21513

copying one file's contents to another in python

I've been taught the best way to read a file in python is to do something like:

with open('file.txt', 'r') as f1:
    for line in f1:
        do_something()

But I have been thinking. If my goal is to copy the contents of one file completely to another, are there any dangers of doing this:

with open('file2.txt', 'w+') as output, open('file.txt', 'r') as input:
    output.write(input.read())

Is it possible for this to behave in some way I don't expect?

Along the same lines, how would I handle the problem if the file is a binary file, rather than a text file. In this case, there would be no newline characters, so readline() or for line in file wouldn't work (right?).

EDIT Yes, I know about shutil. There are many better ways to copy a file if that is exactly what I want to do. I want to know about the potential risks, if any, of this approach specifically, because I may need to do more advanced things than simply copying one file to another (such as copying several files into a single one).

Upvotes: 13

Views: 18705

Answers (3)

Seoester
Seoester

Reputation: 1236

Please note that the shutil module also contains copyfileobj(), basically implemented like Barmar's answer.

Or, to answer your question:

from shutil import copyfileobj

with open('file2.txt', 'wb') as output, open('file.txt', 'rb') as input:
    copyfileobj(input, output)

would be my suggestion. It avoids re-implementing the buffering mechanism and, should the implementation of the standard library improve, your code wins as well.

On Unix, there also is a non-standardised syscall called sendfile. It is used mostly for sending data from an open file to a socket (serving HTTP requests, etc.).

Linux allows using it for copying data between regular files as well though. Other platforms don't, check the Python doc and your man pages. By using a syscall the kernel copies the content without the need of copying buffers to and from userland.

The os module offers os.sendfile() since Python 3.3. You could use it like:

import io
import os

with open('file2.txt', 'wb') as output, open('file.txt', 'rb') as input:
    offset = 0 # instructs sendfile to start reading at start of input
    input_size = input.seek(0, io.SEEK_END)
    os.sendfile(output.fileno(), input.fileno(), offset, input_size)

Otherwise, there is a package on PyPi, pysendfile, implementing the syscall. It works exactly as above, just replace os.sendfile with sendfile.sendfile (and import sendfile).

Upvotes: 16

Barmar
Barmar

Reputation: 782785

The only potential problem with your output.write(input.read()) version is if the size of the file is too large to hold all of it in memory. You can use a loop that reads smaller batches.

with open('file2.txt', 'wb+') as output, open('file.txt', 'rb') as input:
    while True:
        data = input.read(100000)
        if data == '':  # end of file reached
            break
        output.write(data)

This will work for both text and binary files. But you need to add the b modifier to the modes for portable operation on binary files.

Upvotes: 11

user2390182
user2390182

Reputation: 73498

While this may not completely answer your question, but for plain copying without any other processing of file contents, you should consider other means, e.g. the shutil module:

shutil.copy('file.txt', 'file2.txt')

Upvotes: 4

Related Questions