Serverless VPC access connector is in a bad shape

Question

Our project is using a Serverless VPC access connector to allow access to DB over private IP from cloud functions and cloud runs. It was working flawlessly for a few months, but today I tried to deploy one of the functions that use such a connector and I got the message:

VPC connector projects/xxxx/locations/us-central1/connectors/vpc-connector is not ready yet or does not exist. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation.

I went to the Serverless VPC access view and found out that indeed the connector has a red marking on it. When I hover on it it says

Connector is in a bad state, manual deletion recommended

but I don't know for what reason, Link to logs doesn't show anything for the past 3 months.

I tried to google about the such error but without success. I also tried to search through logs but also didn't find anything relevant.

I'm looking for any hints:

• Why it happened?
• How to fix it? I don't want to recreate the connector, it is related to many functions, and cloud runs

Answer 1

The only resolution I can find is to add the Editor role before adding the connector, with its 8963 excess permissions.

The error message Error code 7, message: Operation failed: Google APIs Service Agent (<PROJECT_NUMBER>@cloudservices.gserviceaccount.com) needs editor role in the project. is surprising as basic roles are not even allowed in some organization policies. You would think a more specific role should be sufficient to create a VPC serverless connector.

You can't add conditions like Expiry to a basic role so don't forget to close this security hole before the SecOps team comes after you.

If your VPC is in the service project of a Shared VPC then you will need to add this role to the host project before creating the connector. That's not clear in the message. Giving a service account from a service project Editor role on the network host project might raise a few eyebrows in SecOps if they haven't already blocked such a path. That would force you to put the connector in the host project instead.

You will also need to apply the "roles/compute.networkUser" to a bunch of service accounts on the host project's subnet that this connector sits on. Don't apply these bindings to the project but directly to the subnet (which may be on the host project),

This is not a definitive list of service accounts as it depends on what's using the connector. App engine default service account for the first, cloud functions for the second) :

• "serviceAccount:${var.project_id}@appspot.gserviceaccount.com"
• "serviceAccount:service-${var.project_number}@gcf-admin-robot.iam.gserviceaccount.com"
• "serviceAccount:${var.project_number}@cloudservices.gserviceaccount.com"
• "serviceAccount:service-${var.project_number}@gcp-sa-- vpcaccess.iam.gserviceaccount.com"

You can't see subnet IAM bindings on the console, but gcloud can show them

 gcloud compute networks  subnets get-iam-policy projects/VPC_HOST_PROJECT/regions/REGION/subnetworks/my-subnet 

If any of the permissions are not set, the VPC will be created but left in a bad state. You need Serverless VPC Access Admin role to delete it (or Editor role), but even that won't help if one of your services is actively using the connector and you get a resource in use error.

Good luck [re]creating one.

Answer 2

This week, I also got the Error code 7 when I attempted to destroy my VPC connector implementation. Initially, I tried to delete it with my Terraform configuration, and then via the GCP Console UI. Both didn't work. The full error was as follows:

Error waiting for Deleting Connector: Error code 7, message: Operation failed: Forbidden

What probably caused my problem was the IAM policy implementation. During the development, I first created the VPC connector, and then the IAM policy. When I implemented the policy it deleted the full IAM policy for the full GCP project.

This way, I deleted the GCP service account attached to the vpcaccess.googleapis.com, which is service-<PROJECT_NUMBER>@gcp-sa-vpcaccess.iam.gserviceaccount.com. By re-adding the service account and give it the roles/vpcaccess.serviceAgent role, I could delete my VPC access connector without the above error.

Hope this helps.

Answer 3

As the issue was blocking us from the deployment of cloud functions I was forced to recreate the connector.

But this time API returned an error:

Error: Error waiting to create Connector: Error waiting for Creating Connector: Error code 7, message: Operation failed: Google APIs Service Agent (<PROJECT_NUMBER>@cloudservices.gserviceaccount.com) needs editor role in the project.

After adding such permission old connector started to work again...

Before there was no such requirement, but it changed in meantime.

Spooky, one time something works other not.