CODEWITHSUNDEEP
pythondjangohttpgunicornuvicorn
aroooo
aroooo

Reputation: 5056

Understanding slow Django performance

My webserver is scaling and I'm using Sentry Performance to try and better understand where things are slow. One thing that I don't quite understand is where the slowness might be coming from when the total reported query times are dramatically different than the time it takes to get a response.

For example, one of my endpoints is taking 40s to finally get a response to the user. You'll see the total response time took an incredible 44,000ms even though all the work was done in about 1s (which is still slow, but not 44s slow).

Edit: as a general note, all of my endpoints are doing this– the peculiar thing is those dashed dots at the end of the timeline where any of the actual Django / database hits begin. It just sits blank for a whole minute before Django begins doing anything. Is this a potential guincorn or Uvicorn configuration error?

Additional info:

AppUvicornWorker:

class AppUvicornWorker(UvicornWorker):
    CONFIG_KWARGS = {"loop": "uvloop", "http": "httptools", "lifespan": "off"}

class UserSerializer(serializer.ModelSerializer):
    class Meta:
        model = User
        fields = ["id", "username", "name"]

class CommentSerializer(serializer.ModelSerializer):
    user = UserSerializer()

    class Meta:
        model = Comment
        fields = ["id", "text", "user"]


class CommentsViewSet(viewsets.ViewSet):
    def list(self, request):
        queryset = Comments.objects.all()
        serializer = CommentSerializer(queryset, many=True)
        return Response(serializer.data)

enter image description here

Upvotes: 6

Views: 2502

Answers (2)

gabn88
gabn88

Reputation: 811

We had similar problems when our load increased. The issue is that Django is a synchronous framework, so each request has to be responded to by a single thread. This thread cannot do extra work in between (apart from maybe some multiprocessing work, but that is out of the context).

So what essentially is happening. Every second about 100 requests come in. The server can handle 17 requests per second (each requests takes 1 second). So it will take at least 5 seconds to handle all those requests. But in those 5 seconds another 400 extra requests are added, which take another 23 seconds. All in all the server cannot handle the load and will reach some kind of ceiling, which in your case is 40 seconds, after which probably also some requests are ignored /failed.

What you could do:

  1. Add more workers, might need a hardware upgrade (more cpu cores)
  2. Scale horizontally, if your app is 12-factor (https://12factor.net/) this should be relatively easy, might need more hardware, instead of bigger hardware. Could be virtual servers. Look into docker/kubernetes for this.
  3. Bring down the time requests take. Although one second might seem fast, it could be possible to make it faster. For example, reducing this time by half increases your throughput by a factor 2.

Upvotes: 1

willeM_ Van Onsem
willeM_ Van Onsem

Reputation: 476493

One of the reasons that this is slow is because for each Comment, it has to make a separate query to fetch the User data. You can boost efficiency with .select_related(…) [Django-doc]:

class CommentsViewSet(viewsets.ViewSet):
    
    def list(self, request):
        queryset = Comments.objects.select_related('user')
        serializer = CommentSerializer(queryset, many=True)
        return Response(serializer.data)

This will retrieve the data of the user in the same query, and thus prevent the N+1 query problem.

Note: normally a Django model is given a singular name, so Comment instead of Comments.

Upvotes: 1

Related Questions