How to connect Airflow 2 in Composer 3 to Cloud SQL via Private Service Connect (PSC)?

Question

I'm using Google Cloud Composer v3 (Airflow 2) and need help with networking. I'm trying to connect Airflow DAGs running in Composer to a Cloud SQL PostgreSQL instance, which is accessed via Private Service Connect (PSC) in a shared VPC network.

Here's the setup:

• Cloud SQL is in a customer project, and it uses Private Service Connect (PSC) for access.
• Composer v3 (Airflow) UI and Cloud SQL are in the same customer project.
• Airflow resources (schedulers, workers, etc.) run in a tenant project according to Composer v3 architecture: Composer v3 Architecture.

I'm not sure how to establish a connection between the Airflow resources in the tenant project and Cloud SQL (via PSC) in the customer project.

This my diagram:

What I’ve tried: I attempted to add the Composer environment to the same shared VPC in network config but seems create new network attachaments into private service connect, also I have tried without network in both case i got below error:

I wrote a Python DAG to connect to Cloud SQL using the Cloud SQL Python Connector but didn’t receive a response — it seems to hang indefinitely.

I tried to connect using the Cloud SQL Proxy with Airflow connexion with this script, I got the following error:

psycopg2.OperationalError: connection to server at "127.0.0.1", port 58149 failed: server closed the connection unexpectedly.
This probably means the server terminated abnormally before or while processing the request.

Any guidance or examples would be greatly appreciated! thank you.

Answer 1

I share my solution:

I have used cloud sql proxy v2.13.0

• Set up a PSC endpoint in the shared VPC (if it already exists).
• Add a dedicated PSC endpoint for the Composer v3 VPC network.

If you’re working within an organization, you’ll need firewall rules for both ingress and egress, as follows:

• Ingress Rule:

direction = "INGRESS"
allow {
    protocol = "tcp"
    ports    = ["5432", "3307", "3306"]
}
source_ranges = ["YOUR_RANGE/16", "IP_ADDRESS_WORKER_COMPOSER_V3/10"]

• Egress Rule:

direction = "EGRESS"
allow {
    protocol = "tcp"
    ports    = ["5432", "3307", "3306"]
}
destination_ranges= ["YOUR_RANGE/16", "IP_ADDRESS_WORKER_COMPOSER_V3/10"]

Additional Steps:

1. Create the PSC endpoint.
2. Configure a DNS managed zone and add a DNS record.
3. First to do step 1 and 2, next add the network and subnetwork configuration to your Composer v3 environment, otherwise you have networking issues.

For those on free trial or without organization structure, this setup may differ slightly, but the above firewall rules are generally required into organization structure. You can find a detailed example in my GitHub repository: GitHub - Cloud SQL and Composer v3 with PSC. with free trial or without organization structure.

Please feel free to share any additional solutions or improvements. Your feedback is appreciated!