CODEWITHSUNDEEP
snowflake-cloud-data-platform
Olegq
Olegq

Reputation: 13

Does SnowFlake support any other ways for insert data into internal table stage besides "PUT" files?

According to the SnowFlake documentation, I found one way to insert data into an internal table stage - only files with the command PUT. But I'm a little confused by this because my data will be generated inside the AWS cloud and there is only one place where I can save data files - s3. And I can use this s3 as external stage, so I don't need internal stage at all. Do I understand everything correctly? And if the data is created inside the cloud, should you use an external stage?

My final goal - use stage as place where I insert increment data and merge this from here to raw table.

Upvotes: 1

Views: 437

Answers (2)

Eric Lin
Eric Lin

Reputation: 1510

Yes, your understanding is correct. If you data is already in S3 bucket, then just creating an external stage on top of S3 is the easiest way. There is no need for an internal stage.

Upvotes: 0

Simeon Pilgrim
Simeon Pilgrim

Reputation: 25938

If you have and external location (aka S3 bucket) with the data already existing in there, this is an external stage.

Create an external stage object, to make reading from the bucket (aka permissions hidden away). And just COPY FROM the external stage name object.

But if you don't currently have a bucket, with permissions and you have an app that is reading/generating data. Instead of setting up S3, Snowflake allows you to push the data to "internal stages" where all the S3 security etc is handled for you.

In our primary usage, we want all our data to live for ever in S3 just incase "something bad happened in snowflake" and we have bucket, and life cycle policies in place. So we put all our data in those S3 buckets, then we read from the files after they where written (our app that triggered the data write, triggered the COPY command in snowflake.)

And for the "incremental data" we put each set of files into a staging table, and then did the transforms/merges into our internal representations. Thus this is ELT (extract, load, transform) verse the classic in app base ETL

Upvotes: 0

Related Questions