CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-sqlgcloud
Adam Hopkinson
Adam Hopkinson

Reputation: 28793

gcloud sql import csv fails with permissions error when import file contains wildcard

According to the Google Cloud Platform SQL docs, I should be able to both export to and import from sharded files in a GCS bucket by putting a * in the filename.

If I import a single file, it works fine:

gcloud sql import csv sql-instance-name gs://gcsbucketname/data/led/led_finance_view_000000000000.csv --project=project-name --database=finance --table=Import_Test -q
Importing data into Cloud SQL instance...done.
Imported data from [gs://gcsbucketname/data/led/led_finance_view_000000000000.csv] into [https://www.googleapis.com/sql/v1beta4/projects/project-name/instances/sql-instance-name].

But if I import a sharded file, it throws a permissions error:

gcloud sql import csv sql-instance-name gs://gcsbucketname/data/led/led_finance_view_*.csv --project=project-name --database=finance --table=Import_Test -q
ERROR: (gcloud.sql.import.csv) HTTPError 403: The service account does not have the required permissions for the bucket.

I can confirm that these commands are running under the same user, using the same bucket & SQL instance.

I think the '403 service account permissions' error is probably a bug in GCP and therefore a red herring - but why won't it let me import the sharded file?

Upvotes: 1

Views: 2195

Answers (2)

Arturo Ramirez Alvillo
Arturo Ramirez Alvillo

Reputation: 21

You have to get the user id (of creation) of your current database instance, in order to get that user id, type something like this in your terminal:

Command: gcloud sql instances describe short [name/db instance]

After that , go to the gcs project -> open your bucket -> add member permitions and you have to add the user id that you got in first command, add role for Storage Admin for that user id.

and that´s all.

Upvotes: 2

alextru
alextru

Reputation: 449

Currently, CloudSQL doesn't incorporate the possibility of importing several CSV files at once using wildcards. I have opened a public feature request for this. You can star to make it gain visibility.

Meanwhile, as a workaround, you can either have a script to run the import command once for each file, or join the CSV files before importing.

Upvotes: 2

Related Questions