GCP Cloud Run produces intermittend 503 errors on long running requests with http2

Question

When calling my cloud run service (http2 enabled on the service!) for large files and canceling some requests, I get intermittent 503 errors. This problem is not load related (tested with apache bench). I've tried to lower the parallel request count to 1. It was first experienced by our tester who got CORS errors due to 503 errors from the api server.

My test setup with curl: request a large file (5 parallel requests) during these long running requests, I start and cancel (SIGTERM) curl requests to the same file

I experience 503 errors intermittently. 20 runs, 5 parallel downloads (100 requests total), around 5-15 are producing 503 errors

My setup: Node 14, Fastify 3.15

Answer 1

After long testing and head scratching the solution was simple. The http2SessionTimeout of node (set through fastify options) was the solution. Default is 5seconds. That produced the above errors. I've tried the node default of 0. That produced an unusable cloud run server.

The solution was to set the http2SessionTimeout to 10 minutes.