GCP AI Platform Pipelines - KFP - Use managed storage - cloudsqlproxy issue

Question

I have problem to setup GCP AI Platform Pipelines - Kubeflow Pipelines v1.4.1 with "Use managed storage" None of configuration doesn't allow me (errors on KFP service) to use "Use managed storage", but let focus on specific one described below.

My setup is:

• private cluster with version "1.17.15-gke.800" in custom network/subnetwork and workload identity enabled
• private mysql 8.0 in the same custom network/subnetwork as GKE
• GCS
• "Use managed storage" where there is specified all required information: GCS name, connection name for DB, user, password and db prefix
• workload identity used (to simplify all KSA (K8s service account) were mapped to single Google service account with owner role)

Be aware that setup with GKE and KFP works fine without option "Use managed storage", but in case of using "Use managed storage" it fails:

1. ml-pipeline:

• status:
  • Pod errors: CrashLoopBackOff
  • Does not have minimum availability
• logs:

packets.go:36: unexpected EOF
CETdriver: bad connection

2. metadata-grpc-deployment:

• status:
  • Pod errors: CrashLoopBackOff
  • Does not have minimum availability
• logs:

Non-OK-status: status status: Internal: mysql_real_connect failed: errno: 2013, error: Lost connection to MySQL server at 'handshake: reading inital communication packet', system error: 11MetadataStore cannot be created with the given connection config.

3. cloudsqlproxy & mysql (the same logs):

• status: okay
• logs:

couldn't connect to "<db_connection_string>": googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
More details:
Reason: authError, Message: Invalid Credentials

It seems that there is issue in cloudsqlproxy service/pod, which cannot establish connection to DB. Is it a bug in this feature or some wrong configuration?

Answer 1

I had the same problem and fixed it by:

1. Enabling the GCP SQL API.
2. Writing out the username 'root' in the managed storage options. The default is not 'root' as stated in the description.