CODEWITHSUNDEEP
pythonlinuxappend
NevilleDNZ
NevilleDNZ

Reputation: 1279

Appending to the end of a file in a concurrent environment

What steps need to be taken to ensure that "full" lines are always correctly appended to the end of a file if multiple of the following (example) program are running concurrently.

#!/usr/bin/env python
import random
passwd_text=open("passwd.txt","a+")
u=("jsmith:x:1001:1000:Joe Smith,Room 1007,(234)555-8917,(234)555-0077,[email protected]:/home/jsmith:/bin/sh",
  "jdoe:x:1002:1000:Jane Doe,Room 1004,(234)555-8914,(234)555-0044,[email protected]:/home/jdoe:/bin/sh",
  "xyz:x:1003:1000:X Yz,Room 1003,(234)555-8913,(234)555-0033,[email protected]:/home/xyz:/bin/sh")
for i in range(random.randint(1,2)):
  print >> passwd_text, random.choice(u)
passwd_text.close()

And: Can an "all or nothing" append be guaranteed (on linux/unix) even if the the disk becomes full, or "ulimit -f" has been set?

(Note similar question: How do you append to a file?)

Upvotes: 7

Views: 6606

Answers (2)

hwjp
hwjp

Reputation: 16081

I think the discussion of this "bug" in python's normal open function suggests that you don't get the POSIX atomic guarantee, but you do if you use

with io.open('myfile', 'a') as f:
    f.write('stuff')

http://docs.python.org/2/library/io.html#io.open

if the operating system implements its write sys call correctly...

http://bugs.python.org/issue15723

Upvotes: 10

user405725
user405725

Reputation:

You have to lock the file in order to ensure that nobody else is writing to it at the same time. See File Locking and lockfile or posixfile for more details.

UPDATE: And you cannot write more data into the file if the disk is full. I am not sure about Python's implementation of output re-direction, but write system call can write less bytes than requested.

Upvotes: 3

Related Questions