CODEWITHSUNDEEP
pythongoogle-cloud-platformgoogle-cloud-storagegoogle-cloud-sql
Jayita Banerjee
Jayita Banerjee

Reputation: 31

Read data from GCS and load into CLOUD SQL

I am new in GCP as well as Python.

I am trying to read csv files which is present in google cloud storage and write to data into cloud sql table using Python.Can anyone help on that.Any help will be appreciated.

Thanks in advancs

Upvotes: 1

Views: 2756

Answers (2)

Yaakov Bressler
Yaakov Bressler

Reputation: 12168

I like to use google-cloud-storage when operating with GCS. The package is basically a wrapper for GCloud's API.

Here's how you might use this library:

from google.cloud import storage

# create the client
path_to_service_account = "path/foo.json"
client = storage.Client.from_service_account_json(path_to_service_account)

# get the bucket
bucket_name = "my-bucket"
bucket = client.lookup_bucket(bucket_name)

# loop through the bucket to get the resource you want
for resource in bucket.list_blobs(prefix="dir-name/"):
    # file type doesn't need to be a csv...
    if resource.endswith("my_file.csv"):
        my_blob = bucket.get_blob(resource)
        my_blob_name = resource.split("/")[-1]
        my_blob.download_to_filename(os.path.join("save-dir", my_blob_name))

# finally, load the file from local storage:
...

Upvotes: 1

guillaume blaquiere
guillaume blaquiere

Reputation: 76073

You shouldn't read and load the data if you haven't update/cleaning to perform on data. You can use the Cloud SQL load CSV from Cloud Storage capability. It also work PostgreSQL (change on top on the page for this)

Do you need code example for calling a REST API in Python? (it's quite basic today, but maybe the security can annoy you!)

Upvotes: 2

Related Questions