CODEWITHSUNDEEP
apache-sparkazure-cosmosdbazure-cosmosdb-sqlapi
Steven Lai
Steven Lai

Reputation: 148

Azure CosmosDB Spark OLTP Connector with Managed Identity

We are trying to ingest some data from DataLake to Azure Cosmos DB and Spark OLTP Connector seems to be the easiest to use.

But due to the company's policy, we are not supposed to use the master keys and we usually use managed identity for the applications. I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code as:

    CosmosAsyncClient client = CosmosClientBuilder
     .credential(new DefaultAzureCredentialBuilder().build())
     .buildAsyncClient();

Is there anyway to setup the connector to use the same authentication mechanism with managed identity?

Upvotes: 0

Views: 836

Answers (2)

Matias Quaranta
Matias Quaranta

Reputation: 15603

Currently, the Spark Connector does not support MSI. I see you correctly created the Issue on the repo that holds the source code: https://github.com/Azure/azure-sdk-for-java/issues/29958

That will surely be used for tracking purposes or at least linking to the workitem that tracks the progress on that area. The feature will be available in the future but there is currently no ETA.

Upvotes: 1

Pratik Lad
Pratik Lad

Reputation: 8341

I see the Cosmos DB Java client builder has the 'TokenCredential' option with sample code

In CosmosAsyncClient you also have to mention the maker key. there is no such way to use managed identities.

we are not supposed to use the master keys and we usually use managed identity for the applications.

As you want to transfer data from Datalake to CosmosDB with Managed Identities you can use Copy Data Tool in Azur data factory. Create Linked service for cosmos db and in authentication type select Managed identity either system or user.

enter image description here

You can refer this So Thread by @KarthikBhyresh-MT for more understanding on Copy data tool.

Upvotes: 1

Related Questions