CODEWITHSUNDEEP
azure-data-factoryserverlessazure-synapse-analytics
Developer
Developer

Reputation: 1

Using synapse spark db as a source in copy activity

I have a synapse spark db using the delta format. When I try to create a linked service using either Synapse or Sql using its sql end point it does not allow it. Has any one come across that?

I need to move the data from a synapse lake database to an on prem db. Therefore I was planning to use a copy activity. I can access the parquet files on the data lake using ADLS G2 service. Is there a way of using synapse spark db end point?

Upvotes: 0

Views: 400

Answers (1)

Dileep Raj Narayan Thumula
Dileep Raj Narayan Thumula

Reputation: 3250

In Azure Synapse Analytics, connecting directly to a lake database,similar to connecting to a dedicated SQL pool, is not possible. Instead, lake databases in Azure Synapse Analytics store their data within an Azure Data Lake Storage account. This is achieved by utilizing a linked service to the data lake storage account. By default, the data lake account created during the creation of the Synapse workspace is used to store all the data of the lake database.

  • When working with Azure Synapse, creating a linked service using Synapse or SQL endpoints for a Synapse Spark DB in Delta format is currently not supported.

You can refer to this official Microsoft documentation to understand about the Lake Databases.

  • One option is to use the Copy Activity in Azure Data Factory. Since you mentioned that you can access the parquet files on the data lake using the ADLS G2 service, you can leverage this to move the data.The Copy Activity supports various data sources and destinations,including Parquet files.

enter image description here As you mentioned you need to move parquet files to on prem I have used synapse ADLS linked service and used Azure SQL database as sink. enter image description here

Copy activity: enter image description here

enter image description here

To access the data stored in the Azure Data Lake database in Azure Synapse, you can create a linked service to the data lake storage account. This will allow you to establish a connection and interact with the data lake.

By creating a linked service, you will have the necessary access and permissions to query and work with the data stored in the Azure Data Lake database.

Upvotes: 0

Related Questions