CODEWITHSUNDEEP
copysnowflake-cloud-data-platform
SJE
SJE

Reputation: 21

Snowflake: copy function and LAST_MODIFIED date parameter

Currently the copy function ignores all files whom LAST_MODIFIED date attribute is older than 64days from the current date. I am looking for changing this parameter (--> 30 days) but I don't find how to do that and how?

Any help would be appreciate :)

Regards

Upvotes: 2

Views: 770

Answers (1)

NickW
NickW

Reputation: 9798

Your statement is not strictly true, it is more a by-product of what Snowflake is actually doing.

Snowflake tries to ensure that files are not loaded more than once. In order to do this it maintains metadata about the files it has loaded but deletes this after 64 days. Therefore, if a file is more than 64 days old then Snowflake doesn't know its status and, by default, wont load it as it doesnt know if it has already loaded it or not.

You can force Snowflake to load these files that have an unknown status but obviously this does carry the risk of loading the same file twice. This is described in detail here

You cannot change how long Snowflake holds this metadata (always 64 days). If files exist in a stage they will be loaded, or not, based on the core rules and any parameter values set in the COPY INTO statement

Upvotes: 3

Related Questions