CODEWITHSUNDEEP
azureazure-sql-databaseazure-blob-storageazure-synapsepolybase
Jo Joseph
Jo Joseph

Reputation: 59

Incremental Loads in Polybase

I have created an EXTERNAL TABLE for PolyBase to load data from BLOB storage to Azure SQL Data Warehouse. I have below questions regarding the file load.

1) If there is incremental growth of data then how will the PolyBase work. If there is daily sample file like 

test_2019_02_01.incr
test_2019_02_02.incr

then do we need to create drop and recreate external table every time. if not how do I make this work

CREATE EXTERNAL TABLE [dbo].[sample]
 (
 [social] varchar(30) NOT NULL,
 [address] varchar(30) NOT NULL
 )
 with (
 LOCATION='/TEST/',
 DATA_SOURCE=POLYTEST,
 FILE_FORMAT=POLY_TEST_HEADER,
 REJECT_TYPE=VALUE,
 REJECT_VALUE=0
 )

2) Can I write UNIX scripts in PolyBase Azure BLOB storage?

Please share your thoughts on this issue.

Upvotes: 1

Views: 763

Answers (1)

Alberto Morillo
Alberto Morillo

Reputation: 15658

You can create a table on your Azure SQL Data Warehouse where the data coming from the files will reside, then create a list of the files located on Azure Storage and iterate thru that list and creating and dropping an external table for each item on the list. You can flag each item already processed. Every time you create an external table, you load the data into the Azure Data Warehouse table, then drop the external table. Here you will find a full example.

Upvotes: 1

Related Questions