CODEWITHSUNDEEP
pythonsftppysftpgoogle-cloud-storage
Chandra
Chandra

Reputation: 7

Upload file to Google bucket directly from SFTP server using Python

I am trying to upload file from SFTP server to GCS bucket using cloud function. But this code not working. I am able to sftp. But when I try to upload file in GCS bucket, it doesn't work, and the requirement is to use cloud function with Python.

Any help will be appreciated. Here is the sample code I am trying. This code is working except sftp.get("test_report.csv", bucket_destination). Please help.

destination_bucket ="gs://test-bucket/reports"
with pysftp.Connection(host, username, password=sftp_password) as sftp:
    print ("Connection successfully established ... ")

    # Switch to a remote directory
    sftp.cwd('/test/outgoing/')
   
    bucket_destination = "destination_bucket"

    sftp.cwd('/test/outgoing/')
    if sftp.exists("test_report.csv"):
        sftp.get("test_report.csv", bucket_destination)
    else:
        print("doesnt exist")

Upvotes: 0

Views: 8332

Answers (3)

Martin Prikryl
Martin Prikryl

Reputation: 202158

The pysftp cannot work with GCP directly.

Imo, you cannot actually upload a file directly from SFTP to GCP anyhow, at least not from a code running on yet another machine. But you can transfer the file without storing it on the intermediate machine, using pysftp Connection.open (or better using Paramiko SFTPClient.open) and GCS API Blob.upload_from_file. That's what many actually mean by "directly".

client = storage.Client(credentials=credentials, project='myproject')
bucket = client.get_bucket('mybucket')
blob = bucket.blob('test_report.csv')

with sftp.open('test_report.csv', bufsize=32768) as f:
    blob.upload_from_file(f)

For the rest of the GCP code, see How to upload a file to Google Cloud Storage on Python 3?

For the purpose of bufsize, see Reading file opened with Python Paramiko SFTPClient.open method is slow.

Consider not using pysftp, it's dead project. Use Paramiko directly (the code will be mostly the same). See pysftp vs. Paramiko and Convert paramiko.sftp_file.SFTPFile object to unicode*.

Upvotes: 2

Jeff
Jeff

Reputation: 1558

This is exactly what we built SFTP Gateway for. We have a lot of customers that still want to use SFTP, but we needed to write files directly to Google Cloud Storage. Files don't get saved temporarily on another machine. The data is streamed directly from the SFTP Client (python, filezilla, or any other client), straight to GCS.

https://console.cloud.google.com/marketplace/product/thorn-technologies-public/sftp-gateway?project=thorn-technologies-public

Full disclosure, this is our product and we use it for all our consulting clients. We are happy to help you get it setup if you want to try it.

Upvotes: 0

Chandra
Chandra

Reputation: 7

i got the solution based on your reply thanks here is the code

               bucket = client.get_bucket(destination_bucket)
              `blob = bucket.blob(destination_folder +filename)

                with sftp.open(sftp_filename, bufsize=32768) as f:
                    blob.upload_from_file(f)

Upvotes: -1

Related Questions