How to create Email notification for changes to files in storage bucket

Question

How can I create email notifications to an email address (john.citizen@gmail.com) ( when there are changes to files in a storage bucket i.e New file added, Append, overwrite or failure to update? I am just starting with GCP.

Answer 1

#Only change dataset name

def process_request(event, context):
    try:
    
        # Change the bigquery_dataset Name according to what you have created.
        bigquery_dataset_name = 'Your dataset name'

        # Don't Change Anything from here on.
        # When creating the function trigger type event = storage bucket

        source_bucket_name = event['bucket']
        blob_name = event['name']
        # call function to notify bucket updates
       #send_text_message_to_teams("{} has been received in {}".format(blob_name, source_bucket_name))
        storage_client = storage.Client()
        bigquery_client = bigquery.Client()
        source_bucket = storage_client.bucket(source_bucket_name) 
        source_blob = source_bucket.blob(blob_name)
        
        #check if file type is csv the define job_config,uri, filename, tablename and table id AND then load the job
        if source_blob.name.split('.')[-1] == 'csv':
            job_config = bigquery.LoadJobConfig(
                    skip_leading_rows=1,
                    autodetect=True,
                    source_format=bigquery.SourceFormat.CSV,
                    write_disposition=bigquery.WriteDisposition.WRITE_TRUNCATE)
            uri = 'gs://{}/{}'.format(source_bucket_name, source_blob.name)
            file_name = '.'.join(source_blob.name.split('/')[-1].split('.')[0:-1])
            table_name = ''.join([character if character.isalnum() else '_' for character in file_name])
            table_id = '{}.{}.{}'.format(bigquery_client.project, bigquery_dataset_name, table_name)
            print('Trasferring {} into {}'.format(source_blob.name, table_id))
            
        #load job using details above   
            load_job = bigquery_client.load_table_from_uri(uri, table_id, job_config=job_config)
            load_job.result()
            print("table updated")
            print('{} has been processed.'.format(source_blob.name))
            # call function to notify table updates
            #send_text_message_to_teams("{} has been updated".format(table_id))
        else:
            print('{} is not a csv.'.format(source_blob.name))
    except Exception as e:
         # call function to notify failures
         #send_text_message_to_teams("function-uploadcsv has encoutered an issue. The details are {}".format(e))

Answer 2

GCP has no 'mail-me' when something changes in cloud storage but you can receive notifications in your app and send an email from there.

There are two ways to do that:

• Object Change Notifications will send an HTTP POST to your app.

• Pub/Sub storage notifications (reccommended by Google). - It posts a pub/sub message when a file is created, modified, or deleted. Pubsub can do HTTP Posts, trigger cloud functions, trigger cloud run (like functions, but dockerized), or be polled.

Google has also a Sending mails tutorial.

There is an edge case you might find helpful:

If

• the volume is very low and
• the file creaton/update/delete happen one-by-one and
• you don't mind which file has been changed / created / updated and
• losing a notification is not critical

Then you could:

• Set up a pubsub queue with low retention (<5 minutes).
• Set up an alert when the queue has more than one message.
• And Google will send you an email when this condition happens.