CODEWITHSUNDEEP
azureazure-data-factorydelta-lake
euh
euh

Reputation: 461

Read just the latest version of delta file on blob storage in Azure Data Factory

I have a delta file (consisting of meta data and fragmented parquet files) that I save with databricks to Azure Blob Storage. Later, I am trying to read that file with Azure Data Factory Pipeline but when using copy activity it reads all the data in that delta instead of the latest version (as specified by meta data).

How do I just read one version from delta file on a blob storage?

Upvotes: 0

Views: 1154

Answers (1)

JayashankarGS
JayashankarGS

Reputation: 8125

You can use dataflow for getting data for specific version.

create a new dataflow activity.

enter image description here

select Inline as source type and Delta in inline dataset type.

enter image description here

Next go to Source options tab. enter image description here

Here, select your delta Folder path, Time travel as Query by version, then give your version.

This gives you the result. And use this dataflow in your pipeline.

Upvotes: 1

Related Questions