CODEWITHSUNDEEP
azure-data-lakeu-sql
DraganB
DraganB

Reputation: 1138

Merge files from Data lake store

I have a package that daily imports a file to Data lake store. So that is the same file with different values(same columns etc). My idea is to merge those files into a single file on Data lake, for a monthly report. I want to investigate U-SQL, so my question is:
Is that possible to do with U-SQL?
If its not possible is there any other options to do that?

Upvotes: 0

Views: 2328

Answers (1)

Amit Kulkarni
Amit Kulkarni

Reputation: 965

It is very easily possible to merge records from two files and write a new file. Here are the steps

  1. Read all of the new file using EXTRACT
  2. Read all the records of the current master file using EXTRACT
  3. Use UNION ALL to merge the records: https://msdn.microsoft.com/en-us/library/azure/mt621340.aspx
  4. Write output to master file using OUTPUT statement

For a quick U-SQL tutorial go here: https://learn.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-get-started

Upvotes: 2

Related Questions