CODEWITHSUNDEEP
pythonmultithreadingflaskgunicorn
Leevi L
Leevi L

Reputation: 1789

How to prevent gunicorn workers from crashing when they try to access the same files

My Flask app logs requests in separate json files because they are fairly big and would just clutter the usual text logs (e.g. app.logger.info("..")). It keeps the last n requests in some folder.

More precisely a function globs all json files and while there are more than n it removes one at a time and then adds the new request:

logfiles = sorted(glob.glob("*.json"))

while len(logfiles) >= n:
    os.remove(logfiles.pop(0))

I employ the app with a gunicorn server and noticed that, for an identical request, it sometimes crashes and sometimes succeeds. A crash is followed by the info:

[18935] [INFO] Handling signal: int
[18935] [INFO] Shutting down: Master

I am guessing that two gunicorn workers are simultaneously accessing the log folder and try to delete the same file. So one of the two will crash because the file is already gone. I am aware of (=heard of) this type of problem when programming with multiple threads but I have so far never had to solve it myself.

I guess I will have to lock the folder and create a queue of workers that are trying to add / remove files to the folder? Whats the best way to go about it? Should I use one of the available modules like threading or Queue? Will they run smoothly with Flask and gunicorn? Or should I implement my own queue?

PS: This is a follow up question to How to log large requests as files?

Upvotes: 0

Views: 2828

Answers (1)

YossIrving
YossIrving

Reputation: 11

When the python run-time attempts to access a file that doesn't exist, it throws an exception. Your code does not have a way to handle that exception and as a result, the gunicorn thread throws an interrupt signal and crashes. The most basic way of handling this is to use a try block to encapsulate the code that attempts to access/delete the file:

try:
    # Code that tries to access the file
    while len(logfiles) >= n:
        os.remove(logfiles.pop(0))
except:
    # Code that you want to execute after the operation fails

This works for removing pages, but you might want to also use something like filelock if you run into other errors while trying to write to files.

Upvotes: 1

Related Questions