Error connecting to Cosmos DB from Databricks: AnalysisException: Catalog 'cosmoscatalog' not found

Question

I've followed the documentation from start to end.

I'm trying to connect to a CosmosDb so I can write data to it.

My Databricks cluster runtime version is: 11.3 LTS

I have installed the cosmos DB Spark connector: com.azure.cosmos.spark:azure-cosmos-spark_3-3_2-12:4.15.0 per the documentation.

I have the following code:

cosmosEndpoint = "MyEndpoint"  
cosmosMasterKey = "MyMasterKey"  
cosmosDatabaseName = "SampleDB"  
cosmosContainerName = "testContainer"

#Configure Catalog Api to be used

spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog") spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosEndpoint) spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosMasterKey)

#create an Azure Cosmos DB database using catalog api

spark.sql("CREATE DATABASE IF NOT EXISTS cosmosCatalog.{};".format(cosmosDatabaseName))

#create an Azure Cosmos DB container using catalog api

spark.sql("CREATE TABLE IF NOT EXISTS cosmosCatalog.{}.{} using cosmos.oltp TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '1100')".format(cosmosDatabaseName, cosmosContainerName)) `

I get the error AnalysisException: Catalog 'cosmoscatalog' not found. I have followed the documentation from start to end. Does anyone know why this error occurs?

Answer 1

While connecting Cosmos Db SQL API to Databricks make sure you install the library on cluster correctly most of the time the error we are getting because if libraries are not installed properly.

For connection give appropriate credentials from cosmos db

Code that I tried for connecting Cosmos Db SQL API to Databricks:

cosmosEndpoint1 = "Endpoints"
cosmosMasterKey1 = "masterkey"
cosmosDatabaseName1 = "demo3DB"
cosmosContainerName1 = "demo3Container"

#Configure Catalog Api to be used

spark.conf.set("spark.sql.catalog.cosmosCatalog", "com.azure.cosmos.spark.CosmosCatalog")
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountEndpoint", cosmosEndpoint1)
spark.conf.set("spark.sql.catalog.cosmosCatalog.spark.cosmos.accountKey", cosmosMasterKey1)

#create an Azure Cosmos DB database using catalog api

spark.sql("CREATE DATABASE IF NOT EXISTS cosmosCatalog.{};".format(cosmosDatabaseName1))

#create an Azure Cosmos DB container using catalog api

spark.sql("CREATE TABLE IF NOT EXISTS cosmosCatalog.{}.{} using cosmos.oltp TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '1100')".format(cosmosDatabaseName1, cosmosContainerName1))

Output: