Cloud Scheduler doesn't work with custom Cloud Run domain

Question

I'm hosting the backend for an internal admin tool on Cloud Run. Since only admins of the GCP project should be able to access this tool, I followed the instructions here to enable IAP for Cloud Run by setting up a load balancer with a static external IP (and custom domain), restricting ingress to "Internal and Cloud Load Balancing", and allowing public unauthenticated access for the Cloud Run service since IAP is handling the authentication and authorization.

Now I'm trying to set up some cron jobs on Cloud Scheduler, for which I've provided an endpoint corresponding to my custom domain (say https://customdomain.com/endpoint), along with a service account email that allows OIDC tokens to be generated. The audience for the OIDC token is set automatically to the same custom domain URL. However, as reported on this thread, there seems to be a bug with Cloud Scheduler that only allows run.app audiences - anything else (including custom domains) results in a 401 UNAUTHENTICATED. This happens even if I set my target URL to https://customdomain.com/endpoint but my audience to https://cloud-run-service.a.run.app/endpoint. Of course, I can't change my target URL to https://cloud-run-service.a.run.app/endpoint since it doesn't allow direct traffic not coming through the load balancer.

Has anyone been in this situation or know of any workarounds? Thanks!

Answer 1

I understand your issue is, In Cloud Scheduler, the OIDC token that is sent to the Cloud Run Service only works if the Audience is the Cloud Run-provided URL, not the Custom Domain URL.

1. Doesn't work: URL: https://service-url.customdomain.com | Audience: https://https://service-url.customdomain.com
2. Works: URL: https://service-url.customdomain.com | Audience: https://example-abcdefg.a.run.app
3. Works: URL: https://example-abcdefg.a.run.app | Audience: https://example-abcdefg.a.run.app

Google is aware of the issue and is working on allowing them to specify custom audiences for Cloud Run services, which will solve your problem.

Right now as per the latest update on May, 2022 we're about to ship custom audiences for Cloud Run. Please fill out this form if you are interested in being an early tester for "custom audiences for Cloud Run."

Currently, to authenticate the caller via Cloud IAM, you must pass in JWT token with the audience field set to the full URL of the service, such as https://example-abcdefg.a.run.app. With this capability, you can specify a custom domain as the audience field in the OAuth token instead of the original service URL enable a service deployed in multiple regions to accept a common audience field

Issue tracker reference : https://issuetracker.google.com/182490050

Answer 2

After hours of painful debugging, here's the solution for anyone with the same issue. While it's still true that custom domains mapped to the Cloud Run service don't work as the OIDC audience, neither does the Cloud Run-provided run.app URL when using IAP in front of a load balancer. It turns out the expected audience in such cases is the IAP Client ID. You can find this under Credentials -> APIs and Services -> OAuth 2.0 Client IDs -> <IAP service name>. Just manually set the OIDC audience to this exact string and things should start working!

Answer 3

I believe you can still set the target URL (while configuring Cloud Scheduler) to the run.app/endpoint of your Cloud Run service by making use of service accounts

1. First create a service account for Cloud Scheduler

2. Then give this service account permission to invoke your Cloud Run Service

See Google's documentation here