CODEWITHSUNDEEP
python-3.xgoogle-cloud-platformgoogle-cloud-functionsgoogle-cloud-storage
atron12
atron12

Reputation: 43

Writing a new file to a Google Cloud Storage bucket from a Google Cloud Function (Python)

I am trying to write a new file (not upload an existing file) to a Google Cloud Storage bucket from inside a Python Google Cloud Function.

Any ideas would be much appreciated.

Thanks.

Upvotes: 4

Views: 16333

Answers (3)

guillaume blaquiere
guillaume blaquiere

Reputation: 75745

You have to create your file locally and then to push it to GCS. You can't create a file dynamically in GCS by using open.

For this, you can write in the /tmp directory which is an in memory file system. By the way, you will never be able to create a file bigger than the amount of the memory allowed to your function minus the memory footprint of your code. With a function with 2Gb, you can expect a max file size of about 1.5Gb.

Note: GCS is not a file system, and you don't have to use it like this

EDIT 1

Things have changed since my answer:

  • It's now possible to write in any directory in the container (not only the /tmp)
  • You can stream write a file in GCS, as well as you receive it in streaming mode on CLoud Run. Here a sample to stream write to GCS.

Note: stream write deactivate the checksum validation. Therefore, you won't have integrity checks at the end of the file stream write.

Upvotes: 2

moojen
moojen

Reputation: 1352

You can now write files directly to Google Cloud Storage. It is no longer necessary to create a file locally and then upload it.

You can use the blob.open() as follows:

from google.cloud import storage

def write_file():
    client = storage.Client()
    bucket = client.get_bucket('bucket-name')
    blob = bucket.blob('path/to/new-blob.txt')
    with blob.open(mode='w') as f:
        for line in object: 
            f.write(line)

You can find more examples and snippets here: https://github.com/googleapis/python-storage/tree/main/samples/snippets

Upvotes: 5

André Duarte
André Duarte

Reputation: 305

 from google.cloud import storage
 import io

 # bucket name
 bucket = "my_bucket_name"

 # Get the bucket that the file will be uploaded to.
 storage_client = storage.Client()
 bucket = storage_client.get_bucket(bucket)

 # Create a new blob and upload the file's content.
 my_file = bucket.blob('media/teste_file01.txt')

 # create in memory file
 output = io.StringIO("This is a test \n")

 # upload from string
 my_file.upload_from_string(output.read(), content_type="text/plain")

 output.close()

 # list created files
 blobs = storage_client.list_blobs(bucket)
 for blob in blobs:
     print(blob.name)

# Make the blob publicly viewable.
my_file.make_public()

Upvotes: 3

Related Questions