CODEWITHSUNDEEP
snowflake-cloud-data-platform
danD
danD

Reputation: 716

How to increase the snowpipe warehouse size

I know that snowpipe uses internal warehouse snowpipe.

However there is a requirement where I will be needed to load big single file like 30GB of size using snowpipe (continuous data ingestion, Files arrive in S3 once every week but any time).

And we do not have control to split file or change format.

Is there a way that I can define snowpipe warehouse size to large or XLarge.

or if not then what should be the best way to handle this scenario so that the file is loaded faster

Upvotes: 1

Views: 447

Answers (2)

Mike Walton
Mike Walton

Reputation: 7339

Snowpipe operates no differently from running a COPY command directly. The size of the warehouse has NO effect on the speed at which a single file will load. The benefit of a larger warehouse is the number of files that can be loaded in parallel, which is why the best practice is to split your files into smaller pieces. If you truly have no ability to split the files (or create a Lambda process that splits your files that land on S3 for you), then you have no control on how fast that file will load into Snowflake.

I would explore whether you're able to run a command using Lambda or some other scripting that would split a file that lands on S3 into multiple files and places those smaller files into the Snowpipe S3 bucket. If you can do this, Snowpipe will load the files in parallel and be far more efficient for you.

Upvotes: 1

Nat Taylor
Nat Taylor

Reputation: 1108

Snowpipe uses Snowflake-supplied compute resources, so no there is no way to set the size of the warehouse.

Upvotes: 0

Related Questions