CODEWITHSUNDEEP
python-3.xgoogle-cloud-platform
MAC
MAC

Reputation: 1515

How to upload folder from local to GCP bucket using python

I am following this link and getting some error:

How to upload folder on Google Cloud Storage using Python API

I have saved model in container environment and from there I want to copy to GCP bucket.

Here is my code:

storage_client = storage.Client(project='*****')
def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

   bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):
        
        print(local_file)


        
        print("this is bucket",bucket)
        blob = bucket.blob(gcs_path)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")

path="/pythonPackage/trainer/model_mlm_demo" #this is local absolute path where my folder is. Folder name is **model_mlm_demo**
buc="py*****" #this is my GCP bucket address
gcs="model_mlm_demo2/" #this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

/pythonPackage/trainer/model_mlm_demo has 3 files in it config, model.bin and arguments.bin`

ERROR

The codes doesn't throw any error, but there is no files uploaded in GCP bucket. It just creates empty folder.

enter image description here

Upvotes: 1

Views: 5680

Answers (4)

Tsvi Sabo
Tsvi Sabo

Reputation: 675

I just came across the gcsfs library which seems to be also about better interfaces

You could copy an entire directory into a gcs location like this:


def upload_to_gcs(src_dir: str, gcs_dst: str):
    fs = gcsfs.GCSFileSystem()
    fs.put(src_dir, gcs_dst, recursive=True)

Upvotes: 1

MAC
MAC

Reputation: 1515

I figured out a way using subprocess to upload model artefacts in GCP bucket.

import subprocess

subprocess.call('gsutil cp -r source_folder_in_local gs://*****/folder_name', shell=True, stdout=subprocess.PIPE)

If gsutil is not installed. You can install using this link:

https://cloud.google.com/storage/docs/gsutil_install

Upvotes: 0

Priyashree Bhadra
Priyashree Bhadra

Reputation: 3597

I have reproduced your issue and the below code snippet works fine. I have updated the code based on folders and names you have mentioned in the question. Let me know if you have any issues.

import os
import glob
from google.cloud import storage
storage_client = storage.Client(project='')

def upload_local_directory_to_gcs(local_path, bucket, gcs_path):

    bucket = storage_client.bucket(bucket)

    assert os.path.isdir(local_path)
    for local_file in glob.glob(local_path + '/**'):

        print(local_file)

        print("this is bucket", bucket)
        filename=local_file.split('/')[-1]
        blob = bucket.blob(gcs_path+filename)
        print("here")
        blob.upload_from_filename(local_file)
        print("done")


# this is local absolute path where my folder is. Folder name is **model_mlm_demo**
path = "/pythonPackage/trainer/model_mlm_demo"
buc = "py*****"  # this is my GCP bucket address
gcs = "model_mlm_demo2/"  # this is the new folder that I want to store files in GCP

upload_local_directory_to_gcs(local_path=path, bucket=buc, gcs_path=gcs)

Upvotes: 1

Zhong Dai
Zhong Dai

Reputation: 514

What I can see the error is, you don't need to pass the gs:// as the bucket parameter. Actually, here is an example you may need to check out,

https://cloud.google.com/storage/docs/uploading-objects#storage-upload-object-python

def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    blob.upload_from_filename(source_file_name)

    print(
        "File {} uploaded to {}.".format(
            source_file_name, destination_blob_name
        )
    )

Upvotes: 1

Related Questions