Reputation: 34
How snowflake internally stores the Data into Micro partitions
I am more curious to understand the Snowflake process to store the data into Micro-partitions. As Far as i know, snowflake each partition size would be 50-500MB.
Suppose I have a file size worth 1GB and i wanted to load this data into snowflake. Can some one explain me the internal process/steps snowflake does to store the data into micro partitons
Upvotes: 0
Views: 587
Answers (2)
Reputation: 426
To optimize the number of parallel operations for a load, Snowflake recommends files roughly 10 MB to 100 MB in size, compressed. Splitting large files into a greater number of smaller files distributes the load among the servers in an active warehouse and increases performance.
Upvotes: 0
Reputation: 6229
Snowflake's micro-partition file format is proprietary so you're not going to get much more information than is already in the documentation (short of someone breaching their employment contract from Snowflake).
Upvotes: 4
Related Questions
- Regarding the burden on Snowflake's database storage layer
- Snowflake Partitioning Vs Manual Clustering
- Actual physical storage size used by snowflake
- What "cloud storage " means in snowflake database storage layer?
- How delete works in snowflake partitions
- Partition level Access on Data stored in snowdfl
- Does Snowflake store all data in S3?
- Where are snowflake tables stored?
- What is the difference between Micro-partitions and Data Clustering
- snowflake support micro partition cluster by hour?