CODEWITHSUNDEEP
google-cloud-mlgoogle-cloud-vertex-ai
flip
flip

Reputation: 11

502/Timeout in vertex AI custom fastapi/uvicorn prediction container

I am receiving a 'Error: 14 UNAVAILABLE: 502:Bad Gateway' timeout when calling my custom container from both the node and python client for predictions in vertex AI. The vertex prediction endpoint does work for short predictions. The prediction also does complete, I just get a preemptive response.

In the node client I was originally getting a '4 DEADLINE EXCEEDED', but set the call option timeout higher.

Now I just get a 'Error: 14 UNAVAILABLE: 502:Bad Gateway' from both clients. This is very frustrating, why is it timing out? Where can it be changed?

The most IMPORTANT thing to note is that the prediction does finish, it has to upload a file to google cloud storage and it does do that, and the logs show that the endpoint runs for as long as it needs to, but I am getting a preemptive 502 from Vertex which ruins my workflow for long running predictions. So, the question is, why would I be getting a 502? I am assuming it comes from some internal timeout in GCP.

Another note, I do have a health endpoint.

Help much appreciated.

Upvotes: 1

Views: 990

Answers (1)

Magne Johansen
Magne Johansen

Reputation: 1

I had the same issue using fastapi and uvicorn and asked google support. They quoted that there is a 60 seconds timeout limit in vertex ai online prediction. As I understand this is not something we can change.

Upvotes: 0

Related Questions