CODEWITHSUNDEEP
python-3.xgoogle-cloud-functionsgoogle-cloud-storagefile-transfer
Sid
Sid

Reputation: 163

Transfer files from Google Cloud Storage to Windows VM instance using Cloud Functions

I am trying to setup a Cloud Function which moves and delete the files from a storage bucket to a Windows instance which is on the same project. For sure, we can get it working if we run it locally from the instance and using gsutil.

But how do we get the VM path encoded in the cloud function scripts in local folder's place?

I also had the local folder shared inside the VM.

Much appreciate your inputs.

Below is the code,

import logging
from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="serviceaccount.json"

#logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG) 
bucket_name = 'bucket name'
#File_name = 'filename'
# Instance/VM location where the files should be downloaded into (VM NAME)
folder = '//VW123456789/Cloud-Files' 

storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blobs = bucket.list_blobs() #List all objects that satisfy the filter.

# Download the files inside windows-VM on GCP 
def download_to_local():
 logging.info('File download Started...Please wait for the job to complete!')
 # Create this folder locally if not exists
 if not os.path.exists(folder):
    os.makedirs(folder)
 # Iterating through for loop one by one using API call
 for blob in blobs:
    logging.info('Blobs: {}'.format(blob.name))
    destination_files = '{}/{}'.format(folder, blob.name) 
    blob.download_to_filename(destination_files)
    logging.info('Exported {} to {}'.format(blob.name, destination_files))
    blob.delete()
if __name__ == '__main__':
 download_to_local()

Thank you!

Upvotes: 2

Views: 1017

Answers (1)

John Hanley
John Hanley

Reputation: 81396

There are a few methods to copy files to/from Windows servers. None of these methods are simple to implement in a Cloud Function.

Windows CIFS File Share

This method involves enabling Windows Shares. AFAIK there is no simple SAMBA client that could be implemented in Cloud Functions.

SFTP

This method requires setting up the Windows Server SSH server and an SSH key pair for the client (Cloud Function). There are Python SSH client libraries (paramiko) that might work with Cloud Functions. Transferring files using SFTP is easy to implement via paramiko.

REST API Server

This method requires creating your own software that provides a REST API (or similar technology) that Cloud Functions can call via HTTPS. You will need to manage authorization. Implementing your own API and security poses a significant risk.

RECOMMENDATION

Cloud Functions is the wrong service for interfacing with Windows Servers. I recommend creating an HTTP endpoint on the Windows Server that is invoked instead of Cloud Function. Now you have removed Windows Server authorization from your design equation. Your Python code can run directly on Windows interfacing with Cloud Storage.

Upvotes: 4

Related Questions