CODEWITHSUNDEEP
lookupazure-data-factory
Shivam Yadav
Shivam Yadav

Reputation: 23

How to get row count in file using Azure Lookup Activity

I am reading the data file and RecordCount file having counts of record in data file. I am using lookup Activity to get the counts in data file and comparing it with the count of RecordCount file. This approach is working well and I can compare the records when we have count less than 5000. When data file has Count is more than 5000, it's considering only 5000 records and my pipeline is aborting because of count mismatch. eg:

Datafile count: 7500

RecordCount file: 7500

Though counts are equal but Lookup will consider only 5000 records and will give a mismatch.

How can I achieve this.

Upvotes: 0

Views: 3584

Answers (1)

Mark Kromer MSFT
Mark Kromer MSFT

Reputation: 3838

Add a Data Flow to your pipeline before the Lookup. Source = ADLS Gen2, Sink = ADLS Gen2. Add a Surrogate Key transformation, call the new column as "mycounter". Add an Aggregate transformation and call the new column as "rowcount" with a formula of max(mycounter). In the Sink, output just the "rowcount" column. You'll now have a new dataset that is just the row count from any file. You can consume that row count as a single-line Lookup activity in the pipeline directly after your data flow.

Upvotes: 0

Related Questions