Connecting to a Cloud SQL instance from VM with public and private IPs - how to ensure the right network interface is chosen for the connection?

Question

What I'm trying to set up:

• Cloud SQL instance with private IP, Postgresql database
• A VM with a public IP, but also one private IP on same VPC network as the SQL instance is on (VM, SQL instance and VPC are all in the same region)
• VM has a service account with sufficient Cloud SQL client/viewer permissions
• Possibility to connect from VM to SQL instance.

What happens?

• Any attempt to actually use the connection, from for example psql client or db-migrate, simply hangs - for example psql --host 10.78.0.3 -U gcp-network-issue-demo-staging-db-user gcp-network-issue-demo-staging-database will not prompt for a password, just sit there.
• If I remove the VM's public IP address from the setup, it connects fine. However, I need a publicly accessible VM for other services to connect to it..

I assume the psql connection attempt goes through the wrong network interface or something (this may be just my ignorance about network stuff speaking) - how can I get this working? What am I missing?

PS: this is basically same problem as Connecting to Google Cloud SQL instance on private IP from a VM with both private and public IPs fails but commenters there seem to want one Terraform-related and one connection-issue-related question.

Some screenshots:

VM IPs:

DB IPs:

Network config for VM:

Private IP config for DB instance:

This is the setup of the private network:

I don't understand why the private IP of the DB instance (10.78.0.3) is not an IP from the range of the private network (10.2.0.0-10.2.0.24, right?)..? Is that my problem?

Answer 1

To answer your question:

I don't understand why the private IP of the DB instance (10.78.0.3) is not an IP from the range of the private network (10.2.0.0-10.2.0.24, right?)..?

The Cloud SQL instance is assigned an IP address from the allocated range. When you setup a private services access a VPC peering is created between your VPC gcp-network-issue-demo-staging-network and the service producer VPC network that uses the allocated range 10.78.0.0/16

Also, looking at your VM network config, I see that the VM has two Nics in two different VPCs (default and gcp-network-issue-demo-staging-network). In your case, you can use only one Nic.

As a next step, make sure that your VM is using only the VPC network that you have used to create the private connection. Once that done you should be able to connect to the Cloud SQL instance IP using the command bellow:

telnet 10.78.0.3 3306