CODEWITHSUNDEEP
change-data-capturesnowflake-cloud-data-platform
2tone_tony
2tone_tony

Reputation: 47

Change data capture in snowflake

I was wondering if anyone knows if snowflake can do change data capture, and how it would be done.

I can't seem to find any documentation on it.

Thanks

Upvotes: 1

Views: 2439

Answers (2)

MARTHA VIJAY KUMAR
MARTHA VIJAY KUMAR

Reputation: 31

If you want to use a cost effective system, go with existing cdc tools like hvr-software which is providing its best as per my experience with them.

Best thing in hvr is that it has many connectors like s3, snowflake and inbuilt hive support to create tables on top of the data which we load.

Note: ofcourse snowflake supports the stream with kafka or any other source but why do we need to make our cluster up and running all the time, instead cdc tool runs on a single ec2 instance.

Upvotes: 0

Rick
Rick

Reputation: 2110

Change Data Capture (CDC) in Snowflake can be done with streams: https://docs.snowflake.net/manuals/user-guide/streams.html

For CDC with an external source you rely on data integration providers like Alooma, Matillion, Golden Gate or implement it on your own.

Plus, Snowflake also integrates with Kafka. So that you "only" need to push changes to Kafka for integration with Snowflake: https://docs.snowflake.net/manuals/user-guide/kafka-connector.html

Upvotes: 3

Related Questions