CODEWITHSUNDEEP
azure-cosmosdbazure-databricksfeature-storedatabricks-unity-catalog
Samlex
Samlex

Reputation: 191

Publish features to cosmos dB using Azure Databricks Feature Store Client fails on workspace with unity catalog enabled

we are trying to create an online feature store using cosmosdb following this documentation: https://learn.microsoft.com/en-us/azure/databricks/machine-learning/feature-store/publish-features .

But I get an error when I publish the table to cosmosdb: AnalysisException: Catalog 'cosmoscatalog' not found. The issue only happens when using unity-enabled workspaces. I can publish using a non-unity enabled workspace.

P.S. If I create the table using the non-unity enabled workspace, then the unity-enabled workspace can update the cosmosdb. But the unity-enabled worskpace cannot create the cosmos container/database using the fs.publish_table.

I tried the following code:

from databricks.feature_store.online_store_spec import AzureCosmosDBSpec
from databricks.feature_store.client import FeatureStoreClient

fs = FeatureStoreClient()

account_uri = "https://online-feature-store.documents.azure.com:443/"

# Specify the online store.
online_store_spec = AzureCosmosDBSpec(
  account_uri=account_uri,
  write_secret_prefix="secret/write-cosmos",
  read_secret_prefix="secret/read-cosmos",
  database_name="online_feature_store_example",
  container_name="feature_store_online_wine_features"
)

# Push the feature table to online store.
fs.publish_table("online_feature_store_example.wine_static_features", online_store_spec, mode='merge')

The following code works on workspaces without unity catalog enabled. However, on a unity-catalog enabled workspace, it trhows an error: AnalysisException: Catalog 'cosmoscatalog' not found

Upvotes: 0

Views: 492

Answers (2)

Xiao
Xiao

Reputation: 21

Databricks engineer here. Thanks for the question. Because both Unity Catalog and the official CosmosDB Spark connector modify Spark catalogs, they could conflict and cause Feature Store’s publish to CosmosDB to fail.

We are aware of the issue and are working on it. Meanwhile your workaround of using a non-UC workspace to create table first works (great find btw!). @sinpais's answer above also works.

Upvotes: 2

sinpais
sinpais

Reputation: 26

You need to create the database and container in CosmosDB with the name you are specifying in AzureCosmosDBSpec.

Upvotes: 1

Related Questions