CODEWITHSUNDEEP
pythonherokuflaskshared-memorygunicorn
Benjy Wiener
Benjy Wiener

Reputation: 1255

Share mutable data between Flask workers

I have a Flask app with some user-modifiable config values that are stored in a database. The values won’t change often, so I don’t want to fetch them from the database each time I get a request (the values are needed for every request), but I also need changes to propagate immediately to all workers.

The two general approaches I’ve come up with for solving this issue are:

  1. Somehow have a shared local copy of the values that can be modified by each worker. Then when the user changes the value, the worker will update both the database and the local variable. shelve may be a good option for this approach.
  2. Somehow notify each worker that the values need to be reloaded from the database.

What would be the best way to do this?

(Note: the app is served by Gunicorn on a Heroku Hobby Dyno.)

Upvotes: 3

Views: 2912

Answers (1)

Benjy Wiener
Benjy Wiener

Reputation: 1255

In the end I learned about Python’s multiprocessing module, which provides, among other things, a fairly simple way to share memory between processes.

In my case I needed to share a string, so I initialized a shared string with

import multiprocessing as mp
...
shared_str = mp.Array('c', 100)

This creates a wrapper for a shared c-string of length 100. The multiprocessing.Array constructor takes a type argument (either a ctypes class or a type-code string and a length. I can then access the shared string with shared_str.value.

I run gunicorn with the --preload option, so the shared memory is allocated when my app is created, and then each process can access it with its own copy of shared_str.

This method will work for other ctypes objects as well (and even non-ctypes objects, using pickle). See the documentation for multiprocessing here, and specifically, the sharedctypes submodule.

Upvotes: 3

Related Questions