CODEWITHSUNDEEP
snowflake-cloud-data-platformdata-ingestiondata-lake
The_Third_Eye
The_Third_Eye

Reputation: 183

How to dynamically choose a Virtual Warehouse Based on the data, in S3

I have been pondering on the following requirement, where i would like to increase the warehouse, size based on the File Size, in the S3.

0 MB to 1024 MB -> Small Warehouse
1025 MB to 10250 MB (1+Gb to 10Gb) -> Medium Warehouse
10251 MB to 5TB (10Gb - 5TB) --> Which would be, S3 Object Max Storage File. 
(Large Warehouse)

So, is there any way to programatically, perform these tasks, i.e i knew that we could write a snow procedure, in JS and iterate as required, based on the source file size, but just wanted to know, is there any other way around.

Upvotes: 0

Views: 519

Answers (1)

Gokhan Atil
Gokhan Atil

Reputation: 10134

There is no other way than writing a stored procedure, or Snowscript block.

More important point is, using a larger warehouse on a single file will NOT improve the performance. The nodes/servers in your warehouse will wait idle (at least when reading the file).

So you shouldn't just focus on the size of the file, you should also consider the number of files to ingest/read when choosing the warehouse size.

Upvotes: 0

Related Questions