CODEWITHSUNDEEP
kubernetesazure-functionsserverlessazure-akskeda
Chayan Bansal
Chayan Bansal

Reputation: 2085

terminationGracePeriodSeconds not working for queue triggered function with KEDA

I am running a KEDA enabled Queue Triggered Azure Function for which terminationGracePeriodSeconds has been set to 900 seconds (15 minutes). The function execution time varies between 30 seconds and 600 seconds (10 minutes). So grace period of 15 minutes seems to be reasonable.

Is there a possibility that after the SIGTERM is issued, the pod takes up a new request after finishing the current one if there is still some grace period left.

I have come across situations where the function execution stops abruptly, leaving the message unprocessed.

Here are the specifications in the deployment:

spec:
      containers:
        - image: ####.azurecr.io/azurefunctionqueuetriggeredk8s
          name: queuetrigcontainer
          ports:
          - containerPort: 80
          resources:
            requests:
              memory: "500Mi"
              cpu: "700m"
            limits:
              memory: "600Mi"
              cpu: "700m"
      nodeSelector:
        agentpool: testuserpool
      terminationGracePeriodSeconds: 900

Is there any plausible explanation for this behavior? How this can be resolved?

EDIT: The container logs show "Application is shutting down..."

enter image description here

Upvotes: 1

Views: 1182

Answers (2)

Karthikeyan VK
Karthikeyan VK

Reputation: 6006

Try increasing your RAM and CPU Size, it might cause some weird Terminating issues too. Write debug logs see there is bug in your code.

Upvotes: -1

Luca Ghersi
Luca Ghersi

Reputation: 3321

Looks like this is a non-implemented event on the code side, and not necessarily something related to your configuration. The most probable explanation is the SIGTERM is not handled and the function keeps processing events, no matter what.

There are reports here on GitHub (https://github.com/Azure/azure-functions-host/issues/5365) about kind of the same situation with a timeout.

If you have the chance you could handle the ApplicationStopping and ApplicationStopped events on the IApplicationLifetime object and stop the function from processing if the app is in a stopping state.

Upvotes: 1

Related Questions