Deploy CloudFunctions in VPC SC with cloud build

Question

On GCP project, protected using VPC Service Control,I tried to deploy cloud functions with Cloud Build, but got error when deploy with gcloud command.

Procedure

1 add cloudbuild service account in VPC SC

written in docs, I need to add my-project-number@cloudbuild.gserviceaccount.com in limit access

$ gcloud access-context-manager levels describe MY-ACCESS-LEVEL-POLICY

     basic:
      conditions:
      - ipSubnetworks:
        - xxx.xxx.xxx.xxx ( my local machine IP)
        members:
        - serviceAccount:my-project-number@cloudbuild.gserviceaccount.com
        - serviceAccount:service-my-project-number@gcf-admin-robot.iam.gserviceaccount.com

other article says I also need to add service-my-project-number@gcf-admin-robot.iam.gserviceaccount.com so I did it.

2 set VPC SC restricted service

I already have service perimeters, whose restrictedServices contains

 - cloudbuild.googleapis.com,
 - cloudfunctions.googleapis.com,

3 deploy with gcloud command

this function need to connect CloudSQL, so I use VPC access connector.

 $ gcloud functions deploy my-function --project=MY-PROJECT --region=asia-northeast1 --runtime=go113 --trigger-event=google.storage.object.finalize --trigger-resource=MY-GCS-BUCKET --source=/path/to/source --entry-point=MyFunction --vpc-connector=projects/MY-PROJECT/locations/asia-northeast1/connectors/MY-VPC-ACCESS-CONNECTOR --egress-settings=all --ingress-settings=internal-only 

doc says as below, which looks correctly set on my deploy command.

• All functions must use a Serverless VPC Access connector. See Connecting to a VPC network for more information.
• All functions must allow only traffic from internal sources. See Ingress settings for more information.
• All functions must route all outgoing traffic through the VPC network. See Egress settings for more information.

What happens

After executing gcloud command, I got this error.

Deploying function (may take a while - up to 2 minutes)...failed.                                                                     
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Build failed: Unable to build your function due to VPC Service Controls. The Cloud Build service account associated with this function needs an appropriate access level on the service perimeter. Please grant access to the Cloud Build service account: my-project-number@cloudbuild.gserviceaccount.com’ by following the instructions at https://cloud.google.com/functions/docs/securing/using-vpc-service-controls#grant-build-access.

But on Procedure 1. I added my-project-number@cloudbuild.gserviceaccount.com in Access Level Policy doc

Once, I tried to modify Access Level Policy; currently it's "AND" condition of IP and IAM, so changed condition as below; I thought cloud build runs not on my IP address (xxx.xxx.xxx.xxx/32). But it didn't change anything.

※Updated: change 0.0.0.0/32 to 0.0.0.0/0

$ gcloud access-context-manager levels describe MY-ACCESS-LEVEL-POLICY

     basic:
      conditions:
      - ipSubnetworks:
        - 0.0.0.0/0 (allow any IP address)
        - xxx.xxx.xxx.xxx/32 ( my local machine IP)
        members:
        - serviceAccount:my-project-number@cloudbuild.gserviceaccount.com
        - serviceAccount:service-my-project-number@gcf-admin-robot.iam.gserviceaccount.com

2021 Sep.21st updated ===

I got CloudStorage.object.get error before cloudfunctions deploy. Same situation was written in troubleshooting

Also, I figured out that my access manager policy's condition, which is IPs AND members, is the reason. If I remove IP condition, or set condition operator OR, then Cloud functions deploy run as I expected. Which means, serviceaccount called GCP method from unknown IP address, not my local machine's IP.

==== 2021 Sep.21st updated

Question

2021 Sep.21st updated

What configuration was missing to deploy?

What IP address must be added to ipSubnetworks for cloudbuild-serviceaccount to call method in VPC SC?

Answer 1

Unfortunately since there is no helpful log that will guide us to understand what is causing the issue, I suggest you to open a case in Google Cloud Platform support. I work with the Support team and we can help you further by investigating the VPC logs.

Alternatively, maybe you could continue the troubleshooting if you run Cloud Function without fully hardened configuration, and trying to find the callerIp checking the logs.

Answer 2

According to the official documentation:

VPC Service Controls protection is available only for builds run in private pools.

So in order to make the cloud build service account able to access the VPC SC you need to set up a private pool.

If this does not work, you could also, as suggested in the official documentation, use a user-specified service accounts instead as it is inside the VPC Service Controls perimeter as long as your project is also inside the perimeter.