How do I atomically create a locked file in Linux?

Question

Scenario: I have many processes running that need to fetch files over the net. If the file is already downloaded, I want it cached on disk. If another process is downloading the file, block until it is finished downloading.

I've been trying to find the easiest way to do to this. The obvious way is to:

create file w/ an exclusive lock active on it only if it doesn't exist (O_CREAT | O_EXCL)
if file exists already:
   open file and acquire exclusive lock
else:
   download to newly created file
release lock

This system accomplishes the above goals with (seemingly) no race conditions

Unfortunately, I couldn't find documentation on how to use open(), etc. to create a file that is locked in Linux. If I split the create step into:

open w/ O_CREAT | O_EXCL
flock

a race condition now exists between the create and lock (non-creating process acquires the lock before the creator does).

I realize I could use an external lock file per file (e.g. filename + '.lock), which I acquire before attempting to create filename, but this feels.. inelegant (and I need to now worry about how to files that actually have a .lock suffix!)

Is there anyway to atomically create and lock it (as Windows offers) or is the external lockfile method pretty much what is standard/required?

Answer 1

This is possible on BSDs with O_SHLOCK or O_EXLOCK, but can be done on a new enough Linux using O_TMPFILE and linkat.

Since I was working with python when I found this here's what I did with an example program that initialises the file from stdin.

#!/usr/bin/python3

import fcntl
import os
import os.path
import pathlib
import shutil
import sys


def locked_file_opener(path, flags):
    # A dirfd (and not AT_FDCWD or None) has to be passed to os.link
    # to get python to use linkat() instead of link()
    dirpath = os.path.dirname(path) or "." # dirname("./.") returns ""
    dfd = os.open(dirpath, os.O_RDONLY)
    try:
        # O_TMPFILE is added to open a temp file in the directory
        new_flags = flags | os.O_TMPFILE
        # O_CREAT is removed since it's incompatible with O_TMPFILE
        new_flags &= ~os.O_CREAT
        # and O_EXCL is removed since that's handled by linkat
        new_flags &= ~os.O_EXCL

        fd = os.open(dirpath, new_flags)
        try:
            fcntl.flock(fd, fcntl.LOCK_EX)
            os.link(f"/proc/self/fd/{fd}", path, follow_symlinks=True, src_dir_fd=dfd, dst_dir_fd=dfd)
        except Exception as e:
            os.close(fd)
            raise
        return fd
    finally:
        os.close(dfd)


if __name__ == "__main__":
    import shutil
    import sys
    try:
        with open(sys.argv[1], "w", opener=locked_file_opener) as f:
            shutil.copyfileobj(sys.stdin, f)
            fcntl.flock(f, fcntl.LOCK_SH)
    except FileExistsError as e:
        # Another process created the file first
        with open(sys.argv[1], "r") as f:
            # Wait for creator process to finish
            fcntl.flock(f, fcntl.LOCK_SH)

If you're working with syscalls directly it's something like

int open_locked(const char *path) {
    int ret = -1;
    char *basep = strdup(path);
    char *dirp = strdup(path);
    int fd = -1;
    const char *base = basename(basep);
    const char *dir = dirname(dirp);
    char *proc = NULL;
    // An invalid path
    if (strcmp(base, ".") == 0 || strcmp(base, "/") == 0) {
        goto cleanup_alloc;
    }

    fd = open(dir, O_RDWR|O_TMPFILE);
    if (fd == -1) {
        goto cleanup_alloc;
    }

    flock(fd, LOCK_EX); // Should be impossible to fail

    if (asprintf(&proc, "/proc/self/fd/%d", ret) < 0) {
        goto cleanup_fd;
    }
    if (linkat(AT_FDCWD, proc, AT_FDCWD, path, AT_SYMLINK_FOLLOW)) < 0) {
        goto cleanup_proc;
    }
    ret = fd;
    fd = -1;

cleanup_proc:
    free(proc);
cleanup_fd;
    if (fd != -1) {
        close(fd);
    }
cleanup_alloc:
    free(dirp);
    free(basep);
    return ret;
}

Answer 2

I'm struggling with how to solve a similar problem at the moment, which is what brought me to your question. As I see it, the essence is:

int fd = open(path, O_CREAT|O_RDWR|O_EXCL, mode);
if (fd == -1)
    {
    /* File already exists. */
    the_file_already_exists(fd);
    }
else
    {
    /* I just now created the file.  Now I'll lock it. */

    /* But first I'll deliberately create a race condition!! */
    deliberately_fork_another_process_that_handles_file(path);

    int code = flock(fd,LOCK_EX);
    if (code < 0)
        {
        perror("flock");
        exit(1);
        }

    /* I now have the exclusive lock.  I can write to the file at will --
    or CAN I??  See below. */
    write_to_the_file_at_will(fd);
    }

Obviously in real life I would never deliberately create that race condition, but its equivalent could certainly happen by accident in a real system. That other process might, for example, open the file for reading, obtain a shared lock on it, and read the file. It would see an empty file. That could mean that a write operation is in progress, but it might mean that the file is simply empty and that's the correct and final answer.

If empty files are not allowed, the reader could simply behave exactly the way it would behave if the file was missing. After all, if the reader had started a millisecond earlier it would have failed to open the file anyway. In this case the reader needs to check if the file is empty after it opens it.

If empty files ARE allowed, then you're in a bit of a quandary and I have no ready answer for that.

The problem I have is that when a file is first created, I want to write some sort of default value into it, because I want to "auto-initialize" a fresh system without having to pre-create every possible file it might need. That other process handling the file might itself have already initialized it! For all I know, three other processes might have also run in the meantime and altered the value. In that case I certainly do not want to "write to the file at will" after obtaining the exclusive lock, because I will clobber all those changes.

I suppose the answer is for my code above to ensure that the file is empty before writing to it. If it is NOT empty, then the code should behave exactly as if the file already existed: i.e., it should call:

the_file_already_exists(fd);

Perhaps the bottom line to all of this discussion is that every process which handles the file in any way should check to see if it is empty and behave accordingly. Again though, if empty files ARE allowed, then I can't yet think of any guaranteed solution. None of this would be necessary if there were some way to create the file and lock it as a single atomic sequence, but I don't think there is any way to do that.

Answer 3

Why don't you use a lockfile utility?

Examples

Suppose you want to make sure that access to the file "important" is serialised, i.e., no more than one program or shell script should be allowed to access it. For simplicity's sake, let's suppose that it is a shell script. In this case you could solve it like this:

...
lockfile important.lock
...
access_"important"_to_your_hearts_content
...
rm -f important.lock
...

Answer 4

The race exists anyway. If the file may or may not exist then you have to test for its existence before trying to lock it. But if the file is your mutex, then you can't possibly do that and the space between "if file exists already" (false) and "download to newly created file" is unconstrained. Another process could come by and create the file and start downloading before your download begins, and you would clobber it.

Basically don't use fcntl locks here, use the existence of the file itself. open() with O_CREAT and O_EXCL will fail if the file already exists, telling you that someone else got there first.