Seeking suggestion/solution for an unexpected app behavior due to Gunicorn multiple workers in Django+Nginx+Gunicorn environment

Question

I've recently bothered by an unexpected Django application behavior in the production environment of Django+Nginx+Gunicorn+MySQL. This behavior does not exist on Django development server so I barely foundd it before deployment. However I've sort of figured out a work-around after analyzing the production logs but I wonder if there's a better way to solve this issue, or my design is flawed so that I'll need to redesign.

To simplify my questions, I separated and extracted the scenario where this behavior occurs: the application uses one class-based view with one model to feed one number to the front-end on one page load (users click next to next page load, the app feeds next number). The number is from a list of five numbers. These five numbers are feeded in order. When all five are feeded, the list will be shuffled and the five numbers in the shuffled list will be feeded in order again.

For example, I have an list [0, 1, 2, 3, 4]. The first page load will feed 0 to the front-end by reading the URL parameters (so the view knows it's getting started). On front-end, 0 is shown, and click next to the next page. The next page will show 1. Same behavior until the page shows 5. When click next at this moment, the list [0, 1, 2, 3, 4] will be shuffled, for example, to [4, 2, 1, 0, 3]. And 4 will be shown on the page. Then click next, next page shows 2, etc. Until 3. Then shuffle. Then the same process.

The model is for letting view know which number to feed. I simplified the model as the other fields are irrelevant. Here's my simplified model.

#models.py
class Question(models.Model):
    q_set = models.PositiveIntegerField()
    q_index = models.PositiveIntegerField()

In views.py and outside the class-based view, an list of numbers QUESTION_VALUES are defined and shuffled initially. The class-based view uses get() to read the 'set' and 'index' parameters in the URL and to decide the index of the number for feeding to front end and to shuffle the list when all numbers in current list are feeded. The view uses post() to decide the parameters for the next page. In short, the page 'GET' the number from the view and 'POST' the current parameters back to the view so that view decides the parameters for next page. Here's my simplified view.

#views.py
QUESTION_VALUES = [0, 1, 2, 3, 4]
random.shuffle(QUESTION_VALUES)

class QuestionView(TemplateView):
    model = Question
    template_name = 'question.html'

    def get(self, request, *args, **kwargs):
        q = get_object_or_404(Question, q_set=kwargs['set'], q_index=kwargs['index'])
        q_index = int(q.q_index)
        if q_index < 5:
            q_val = QUESTION_VALUES[q_index - 1]
        elif q_index == 5:
            q_val = QUESTION_VALUES[q_index - 1]
            random.shuffle(QUESTION_VALUES)
        return render(request, self.template_name, locals())

    def post(self, request, *args, **kwargs):
        q = get_object_or_404(Question, q_set=kwargs['set'], q_index=kwargs['index'])
        q_set = int(kwargs['set'])
        q_index = int(kwargs['index'])
        if 'next-button' in request.POST:
            if q_index and q_index < 5:
                q_index += 1
                return HttpResponseRedirect(reverse('question', kwargs={'set': q_set,'index': q_index}))
            elif q_index and q_index == 5:
                if q_set and q_set < 5:
                    q_set += 1
                    q_index = 1
                    return HttpResponseRedirect(reverse('question', kwargs={'set': q_set,'index': q_index}))
                elif q_set and q_set == 5:
                    return HttpResponseRedirect('end')

On django development server, everything works just fine. One page shows one number, click next, next page shows the next number in current list. After all five numbers are feeded, the list gets shuffled. The next page shows the first number in the shuffled list (set #2). The application will be directed to 'end' when click 'next' on the pages shows the fifth number of set #5. **More importantly, different number is shown within a set. This means it's impossible to show 0, 0, 1, 2, 3 on five pages, respectively. **

However, when I was testing the deployed app, I found that showing 0, 0, 1, 2, 3 on five pages respectively occurred, which was unexpected. I didn't know what was wrong so that I added prints to views.py and check the logs by Gunicorn. The reason I found is that I have 3 sync workers for handling requests on Gunicorn and I set it to 3 just following the setup tutorials which say (2 x Number of CPUs) + 1. I found when one worker is handling request and processing the view and in the middle of feeding numbers within one set of list, it can exit for "refreshing" and let another worker continue to handle the request. I understand each worker process would have an interpreter so at this point the current list gets lost and the successor worker which has another interpreter does not know the current list so it gets a new shuffled list just before entering the class-based view. My current work-around is to set the number of workers to 1 so that this single worker always knows the current list. And now it's working fine.

My first question is that if there's better way to solve this issue while still having 3 workers. Although one worker seems enough for the current app traffic I still would like to know the best practice in my scenario. My second question is that if there's better design of view methods for this number-feeding logic.

My production environment is Django 1.11, Gunicorn 19.7.1, Nginx 1.10.3, MySQL 5.6, and Ubuntu 16.04. The app is deployed on a single CPU droplet by DigitalOcean.

I'll greatly appreciate any suggestion and discussion.

Answer 1

You have more problems than you think; the orders are specific to the workers but are shared by all requests to each worker, so each user will get the same order that happens to be set for the worker they hit for each request.

You shouldn't try and store data in the process like this. Data that needs to persist between requests should go in the session; that way it's specific to a user and will be available no matter which worker they hit.