GCP EventArc : Response Code 500 when using PubSub trigger with Cloud Run

Question

I am using PubSub to trigger Cloud Run using EventArc. The PubSub subscripton thus has the Push Endpoint as https://<my cloud run url>/?__GCP_CloudEventsMode=CUSTOM_PUBSUB_projects%2F<my project ID>%2Ftopics%2F<my pubsub topic>. I noticed that while testing bulk of messages, I got "requestMethod": "POST", "status": 500, "requestUrl": "https://<my cloud run url>/?__GCP_CloudEventsMode=CUSTOM_PUBSUB_projects%2F<my project ID>%2Ftopics%2F<my pubsub topic>" for some of the messages. The messages were redelivered and hence were processed. However, I would like to know why I was getting this message and how to prevent it from happening. Can you please help ?

Edit: Full Error log

{
  "insertId": "61cc8fbf0000cbfca245d9e8",
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "https://<my cloud run url>/?__GCP_CloudEventsMode=CUSTOM_PUBSUB_projects%2F<my project ID>%2Ftopics%2F<my pubsub topic>",
    "requestSize": "2125",
    "status": 500,
    "responseSize": "807",
    "userAgent": "APIs-Google; (+https://developers.google.com/webmasters/APIs-Google.html)",
    "remoteIp": "< hidden>",
    "serverIp": "< hidden>",
    "latency": "0.026223558s",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "location": "us-central1",
      "project_id": "<my project ID>",
      "configuration_name": "<config id>",
      "service_name": "<service name>",
      "revision_name": "<service revision name>"
    }
  },
  "timestamp": "2021-12-29T16:41:35.052220Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "<instance ID>"
  },
  "logName": "projects/<my project ID>/logs/run.googleapis.com%2Frequests",
  "trace": "projects/<my project ID>/traces/aff551446a5eb4d6e32cf6fe0b43dbbe",
  "receiveTimestamp": "2021-12-29T16:41:35.135561206Z"
}

Answer 1

Why are you getting this message?

When Pub/Sub delivers a message to a push endpoint, Pub/Sub sends the message in the body of a POST request. The body of the request is a JSON object and the message data is in the message.data field. To receive messages from push subscriptions, use a webhook and process the POST requests that Pub/Sub sends to the push endpoint. The push endpoint must be a publicly accessible HTTPS address. The server for the push endpoint must have a valid SSL certificate signed by a certificate authority. The Pub/Sub service delivers messages to push endpoints from the same Google Cloud region that the Pub/Sub service stores the messages.

Success codes such as 102, 200, 201, 202, 204 acknowledge complete processing of the Pub/Sub message. Any code other than that such as HTTP 400 or 500 will result in negative acknowledgement. If you send a negative acknowledgement or the acknowledgment deadline expires, Pub/Sub resends the message.

From this documentation, If a push subscriber sends negative acknowledgments (like in your case), Pub/Sub might deliver messages using a push backoff. When Pub/Sub uses a push backoff, it stops delivering messages for 100 milliseconds to 60 seconds and then starts delivering messages again. The push backoff is an exponential backoff that prevents a push subscriber from receiving messages that it can't process. The amount of time that Pub/Sub stops delivering messages depends on the number of negative acknowledgments that push subscribers send or if set to default PubSub will try resending the message immediately. However, the conditions that prevented message acknowledgement may not have had time to change when using immediate redelivery, resulting in the message being unacknowledged again, and the message being continuously redelivered.

What can we do to prevent it from happening ?

• You can set subscription properties when you create or update a subscription. We can modify/specify explicitly acknowledgement deadline, expiration period and retry policy as per our choice.

1. If the acknowledgement deadline expires or a subscriber responds with a negative acknowledgement, Pub/Sub can send the message again using exponential backoff. If the retry policy isn't set, Pub/Sub resends the message as soon as the acknowledgement deadline expires or a subscriber responds with a negative acknowledgement. So you should go ahead and set the minimum back off duration (don’t keep it default) and keep the default maximum back off duration as 600 seconds.

2. 10 seconds is the default acknowledgement deadline. The maximum is 10 minutes. We can increase the acknowledgement deadline to more than default so that our messages don’t expire.

3. If all of your messages are expected to take a longer time to process, then it is best to increase the ack deadline. If only a couple of messages will be slow and most will be processed quickly, then it's better to use the modifyAckDeadline call to increase the ack deadline on a per-message basis.

Check if your push subscription uses authentication. If yes, then Pub/Sub service should sign a JSON Web Token (JWT) and send the JWT in the authorization header of the push request. The JWT should include claims and a signature.

• The claims must be accurate.
• The Pub/Sub service should sign the claims.

If subscribers use a firewall, they can't receive push requests. To receive push requests, you must turn off the firewall and verify the JWT.

• You must also grant Pub/Sub the permissions needed to create tokens for your service account. Pub/Sub creates and maintains a special service account for your project: service-{PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com where {PROJECT_NUMBER} is the project that contains the subscription. This service account needs the Service Account Token Creator role. If you use the Cloud Console to set up the subscription for push authentication or you use a project created after April 8, 2021, the role is granted automatically as part of the Pub/Sub service agent role. Otherwise, you must explicitly grant the role to the account.Also a push subscription configured with a service account should have the role roles/run.invoker which binds it to a particular Cloud Run service that can invoke Cloud Run service.