CODEWITHSUNDEEP
snowflake-cloud-data-platformsnowflake-schema
Unnikrishnan Raju
Unnikrishnan Raju

Reputation: 19

SNOWPIPE in snowflake is 100% reliable or not?

I have used snowpipe to retrieve data from AWS S3 to Snowflake, but in my case, its not working as expected. Sometimes the files are not processing into snowflake.

Is there any alternate methods available for the same?

Upvotes: 0

Views: 1471

Answers (2)

Hans Henrik Eriksen
Hans Henrik Eriksen

Reputation: 2850

The event handling from AWS S3 has been said to be unreliable in the way that events might arrive several minutes late (this is an AWS issue, but affects Snowpipe).

The remedy is to schedule a task to periodically (minimum daily) do:

ALTER PIPE my_pipe REFRESH [ PREFIX = '<path>' ];

Please use a prefix to avoid scanning large S3 buckets for unprocessed items. Also watch for announcements from Snowflake about when the S3 event issue is fixed by Amazon, so you can delete any unnecessary REFRESH tasks.

If you have eg. a YYYY/MM/DD/ bucket structure this unfortunately means you have to create a Stored Procedure to run the command with a dynamic PREFIX...

I use this combination (PIPE/REFRESH TASK) for my Snowpipes.

Upvotes: 2

Simon D
Simon D

Reputation: 6229

To answer your question: Yes. I've used it in the past on multiple occasions in production (AWS) and it has worked as expected.

Upvotes: 0

Related Questions