CODEWITHSUNDEEP
pythondockerazure-web-app-servicegrpcgrpc-python
Jinjing Arima
Jinjing Arima

Reputation: 1

GRPC python server deployment on Azure app service not working

I've been stuck with this issue in a past couple days so was hoping if I can get someone's insights... I have a python grpc server in docker which works fine locally. The problem is that once its deployed on Azure App Service, the request always returns 503 service unavailable. Log Stream I see is below:

023-09-06T17:57:27.440709740Z Server started, listening on 50051

2023-09-06T18:07:31.176838969Z Server started, listening on 50051

2023-09-06T18:17:30.829370909Z Server started, listening on 50051

2023-09-06T18:27:31.834088285Z Server started, listening on 50051

2023-09-06T18:37:32.414292994Z Server started, listening on 50051

/home/LogFiles/2023_09_06_ln1sdlwk0000CQ_docker.log  (https://mysite.scm.azurewebsites.net/api/vfs/LogFiles/2023_09_06_ln1sdlwk0000CQ_docker.log)

2023-09-06T18:39:38.365Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3. Elapsed time = 127.1765748 sec

2023-09-06T18:39:55.833Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 144.6447916 sec**

2023-09-06T18:40:10.908Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 159.7197992 sec**

2023-09-06T18:40:25.989Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 174.8003644 sec**

2023-09-06T18:40:42.860Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 191.6716851 sec**

2023-09-06T18:40:58.381Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 207.1929065 sec**

2023-09-06T18:41:13.464Z INFO  - Waiting for response to warmup request for container mysite_0_ce82c4e3**. Elapsed time = 222.2752554 sec**

2023-09-06T18:41:21.372Z ERROR - Container mysite_0_ce82c4e3****for site mysite did not start within expected time limit. Elapsed time = 230.1834844 sec

2023-09-06T18:41:21.381Z ERROR - Container mysite_0_ce82c4e3****didn't respond to HTTP pings on port: 50051, failing site start. See container logs for debugging.

2023-09-06T18:41:21.388Z INFO  - Stopping site mysite because it failed during startup.

I followed this doc and configured every setting as it says to change. https://github.com/Azure/app-service-linux-docs/blob/master/HowTo/gRPC/Python/Flask/README.md But the grpc call to the deployed endpoint returns 503. I believe 503 is because the app is stopped after the warmup reuqests not returning its response according to the log stream. I'm not sure why the warmup request is not returning its response. Does anyone know how I can get around/fix this? Any help will be appreciated!

I tried this deployment Steps:

  1. Enable HTTP version The first setting you'll need to configure is the HTTP version

Navigate to Configuration under Settings in the left pane of your web app Click on the General Settings tab and scroll down to Platform settings Go to the HTTP version drop-down and select 2.0 Click save This will restart your application and configure the front end to allow clients to make HTTP/2 calls.

  1. Enable HTTP 2.0 Proxy Next, you'll need to configure the HTTP 2.0 Proxy:

Under the same Platform settings section, find the HTTP 2.0 Proxy setting and switch it to On. Click save Once turned on, this setting will configure your site to be forwarded HTTP/2 requests.

  1. Add HTTP20_ONLY_PORT application setting Earlier, we configured the application to listen to a specific HTTP/2 only port. Here we'll add an app setting HTTP20_ONLY_PORT and put the value as the port number we used earlier.

Navigate to the Configuration under Settings on the left pane of your web app. Under Application Settings, click on New application setting Add the following app setting to your application Name = HTTP20_ONLY_PORT Value = 50051 This setting will communicate to your web app which port is specified to listen over HTTP/2 only.

  1. Add SCM_DO_BUILD_DURING_DEPLOYMENT application setting To ensure our application runs a pip install to resolve all our dependencies named in requirements.txt, we'll need to set the following application setting.

Navigate to the Configuration under Settings on the left pane of your web app. Under Application Settings, click on New application setting Add the following app setting to your application Name = SCM_DO_BUILD_DURING_DEPLOYMENT Value = true 6. Add a custom startup command (I didn't follow this step since I'm deploying with a docker and the docker has the startup command) To ensure your application starts up properly, we'll need to set a custom startup command to kick off our grpc and flask server.

Navigate to the Configuration under Settings on the left pane of your web app. Under General Settings, add the following Startup Command python app.py 7. Save your app configuration Click the Save button at the top of the Configuration page. This will restart your application and apply all updated application settings.

Upvotes: 0

Views: 499

Answers (1)

Hamzah
Hamzah

Reputation: 1

I faced the same issue. I believe it is because during startup, Azure App Service sends HTTP 1.1 pings to the server and expects a response. However, the gRPC service is listening for HTTP 2 requests. So the Azure App Service gets no response and stops the container.

To solve this, I added a very simple flask server:

from flask import Flask

app = Flask(__name__)

## Required for HTTP 1.1 pings by Azure App Service
@app.route('/')
def func():
    return "Hello!"

I then ran the flask server and gRPC server in parallel:

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    support_pb2_grpc.add_SupportServiceServicer_to_server(SupportService(), server)
    port = os.environ.get('PORT', '50051')
    server.add_insecure_port(f'[::]:{port}')
    server.start()
    return server


if __name__ == '__main__':
    ## Run Flask (HTTP 1.1) and gRPC (HTTP 2) servers in parallel
    grpc_server = serve()
    app.run(host="0.0.0.0", port=8000)

(Please note that this is not production-ready, just a proof-of-concept).

Then in my Azure App Service configuration, I set HTTP20_ONLY_PORT=50051 and WEBSITES_PORT=8000.

And that fixed the issue. The app service was able to start successfully.

Upvotes: 0

Related Questions