CODEWITHSUNDEEP
nginxflaskgoogle-cloud-platformgoogle-kubernetes-engineuwsgi
Olgun Kaya
Olgun Kaya

Reputation: 2579

GCP deployment with nginx - uwsgi - flask fails

I have a very simple flask app that is deployed on GKE and exposed via google external load balancer. And getting random 502 responses from the backend-service (added a custom headers on backend-service and nginx to make sure the source and I can see the backend-service's header but not nginx's)

The setup is;

LB -> backend-service -> neg -> pod (nginx -> uwsgi) where pod is the application built using flask and deployed via uwsgi and nginx.

The scenario is to handle image uploads in simple-secured way. Sender sends me a token with upload request.

My flask app

  1. receive request and check the sent token via another service using "requests".
  2. If token valid, proceed to handle the image and return 200
  3. If token is not valid, stop and send back a 401 response.

First, I got suspicious about the 200 and 401's. And reverted all responses to 200. Following some of the expected responses, server starts to respond 502 and keep sending it. "Some of the messages at the very beginning succeeded".

nginx error logs contains below lines

2023/02/08 18:22:29 [error] 10#10: *145 readv() failed (104: Connection reset by peer) while reading upstream, client: 35.191.17.139, server: _, request: "POST /api/v1/imageUpload/image HTTP/1.1", upstream: "uwsgi://127.0.0.1:21270", host: "example-host.com"

my uwsgi.ini file is as below;

[uwsgi]
    socket = 127.0.0.1:21270
    master
    processes = 8
    threads = 1
    buffer-size = 32768
    stats = 127.0.0.1:21290
    log-maxsize = 104857600
    logdate
    log-reopen
    log-x-forwarded-for
    uid = image_processor
    gid = image_processor
    need-app
    chdir = /server/
    wsgi-file = image_processor_application.py
    callable = app
    py-auto-reload = 1
    pidfile = /tmp/uwsgi-imgproc-py.pid

my nginx.conf is as below

location ~ ^/api/ {
        client_max_body_size 15M;
        include uwsgi_params;
        uwsgi_pass 127.0.0.1:21270;
    }

Lastly, my app has a healthcheck method with simple JSON response. It does no extra stuff and simply returns. This never fails as explained above.

Edit : my nginx access logs in the pod shows the response as 401 while the client receives 502.

Upvotes: 0

Views: 156

Answers (1)

Olgun Kaya
Olgun Kaya

Reputation: 2579

for those who gonna face with the same issue, the problem was post data reading (or not reading).

nginx was expecting to get post data read by the proxied, in our case uwsgi, app. But according to my logic I was not reading it in some cases and returning back the response.

Setting uwsgi post-buffering solved the issue.

post-buffering = %(16 * 1024 * 1024)

Which led me to this solution;

https://stackoverflow.com/a/26765936/631965 Nginx uwsgi (104: Connection reset by peer) while reading response header from upstream

Upvotes: 1

Related Questions