CODEWITHSUNDEEP
pythonfile-writing
shikhanshu
shikhanshu

Reputation: 1548

Pythonic way to write a large number of lines to a file

I need to auto-generate a somewhat large Makefile using a Python script. The number of lines is expected to be relatively large. The routine for writing to the file is composed of nested loops, whole bunch of conditions etc.

My options:

  1. Start with an empty string and keep appending the lines to it and finally write the huge string to the file using file.write (pro: only single write operation, con: huge string will take up memory)

  2. Start with an empty list and keep appending the lines to it and finally use file.writelines (pro: single write operation (?), con: huge list takes up memory)

  3. Write each line to the file as it is constructed (pro: no large memory consumed, con: huge number of write operations)

What is the idiomatic/recommended way of writing large number of lines to a file?

Upvotes: 1

Views: 1722

Answers (2)

ShadowRanger
ShadowRanger

Reputation: 155403

Option #3 is usually the best; normal file objects are buffered, so you won't be performing excessive system calls by writing as you receive data to write.

Alternatively, you can mix option #2 and #3; don't build intermediate lists and call .writelines on them, make the code that would produce said lists a generator function (having it yield values as it goes) or generator expression, and pass that to .writelines. It's functionally equivalent to #3 in most cases, but it pushes the work of iterating the generator to the C layer, removing a lot of Python byte code processing overhead. It's usually meaningless in the context of file I/O costs though.

Upvotes: 2

Ethan Furman
Ethan Furman

Reputation: 69041

Option 3: Write the lines as you generate them.

Writes are already buffered; you don't have to do it manually as in options 1 and 2.

Upvotes: 3

Related Questions