CODEWITHSUNDEEP
azuresnowflake-cloud-data-platform
Andrei Budaes
Andrei Budaes

Reputation: 665

Is there a way to setup a Snowpipe on multiple Azure Storage Accounts?

My company set up multiple storage accounts (about 30) for an app with the same container name on the same Azure tenant ID (ex. azure://myaccount1.blob.core.windows.net/samecontainername, azure://myaccount2.blob.core.windows.net/samecontainername, azure://myaccount3.blob.core.windows.net/samecontainername)

I've created Snowpipes in the past and the process was:

  1. Create a queue on the account and container
  2. Create an Event Grid linked to that account and container and with a destination set as the created queue
  3. Create a Notification Integration in Snowflake (to a queue related to that account and container)
  4. Authenticate the service principal app Snowflake created (if it is the first time on that tenant ID)
  5. Create a stage using the account and blob url (azure://myaccount2.blob.core.windows.net/samecontainername) and using the notification integration created
  6. Create a pipe using the notification integration created and copy from the stage created into a variant table (for Json files)

The question is, is there a way to simplify this so I won't have to do the steps above 30 times? Each account contains the same container name

Upvotes: 0

Views: 272

Answers (1)

Andrei Budaes
Andrei Budaes

Reputation: 665

Seems not as snowpipe can have one NOTIFICATION INTEGRATION per queue. However, we have setup a copy statement from all accounts into one not used for PROD on which we set up a NOTIFICATION INTEGRATION to Snowflake and then a PIPE using that integration

Upvotes: 0

Related Questions