CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-rungoogle-iam
Kevin Danikowski
Kevin Danikowski

Reputation: 5196

CloudRun Service to Service returning 403 After Setup

I have a service to service set up that I completed using the google cloud tutorial (https://cloud.google.com/run/docs/authenticating/service-to-service#nodejs)

  1. Changed the cloudrun Service account to have roles/run.invoker (they both share the same role)

  2. Make a request to get the access token: http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://XXXX-XXXX-XXXX-xx.a.run.app'

  3. (failing) Use that access token to make a request at https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint with the access token: axios.post('https://XXXX-XXXX-XXXX-xx.a.run.app/my-endpoint', {myData}, {headers: {Authorization: 'Bearer eyJhbGciOiJSUz.....'}})

However, on step 3, making the call to my service, I receive a 403 error, any thoughts on what I missed?

Note: I have tried deploying my invoked service with --allow-unauthenticated and without it. I am not using a custom domain, I am using the CloudRun created url.

PS: If I change the ingress from internal and load balancer to all it works, however I'm not sure if this is correct to do.

Upvotes: 1

Views: 3118

Answers (2)

Kevin Danikowski
Kevin Danikowski

Reputation: 5196

Solution was to add a VPC Connector and route all traffic through it. I added this to the deploy script --vpc-egress all-traffic. Originally I had --vpc-egress private-ranges-only to connect to redis MemoryStore, however this was insufficient to connect to my other service (internal only ingress).

Credit to excellent insight from @JohnHanley and @GuillaumeBlaquiere

Interesting Note About NodeJS: My container wouldn't start when I switched the --vpc-egress to all-traffic, and I had no idea why because there were no logs. It turns out running node v16.2 caused some weird issues with --vpc-egress all-traffic that I couldn't debug, so downgrading to 14.7 allowed the container to start.

Upvotes: 0

John Hanley
John Hanley

Reputation: 81356

The HTTP 403 Forbidden error message when accessing your Cloud Run service means that your client is not authorized to invoke this service.

You have not granted the service account permission to call the receiving service. Your question states that you added roles/run.invoker but the error message indicates you did not complete this step correctly.

  1. Go to the Google Cloud Console.
  2. Select the receiving service (this is the Cloud Run service you are calling).
  3. Click Show Info Panel in the top right corner to show the Permissions tab.
  4. In the Add members field, enter the identity of the calling service.
  5. Select the Cloud Run Invoker role from the Select a role drop-down menu.
  6. Click Add.

Note: When requesting the Identity Token, do not specify the custom domain. Your question's wording is confusing on that point.

[UPDATE]

The OP has enabled internal and load balancer. This requires setting up Serverless VPC Access.

Connecting to a VPC network

Upvotes: 3

Related Questions