CODEWITHSUNDEEP
azure-cosmosdbazure-cosmosdb-tables
BeGreen
BeGreen

Reputation: 951

How to alter TBLPROPERTIES in cosmosdb

In cosmosDB, I need to insert a large amount of data in a new container.

create_table_sql = f"""
        CREATE TABLE IF NOT EXISTS cosmosCatalog.`{cosmosDatabaseName}`.{cosmosContainerName} 
        USING cosmos.oltp
        OPTIONS(spark.cosmos.database = '{cosmosDatabaseName}')
        TBLPROPERTIES(partitionKeyPath = '/id', manualThroughput = '10000', indexingPolicy = 'AllProperties', defaultTtlInSeconds = '-1');
        """
spark.sql(create_table_sql)

# Read data with spark
data = (
    spark.read.format("csv")
    .options(header="True", inferSchema="True", delimiter=";")
    .load(spark_file_path)
    )

cfg = {
        "spark.cosmos.accountEndpoint": "https://XXXXXXXXXX.documents.azure.com:443/",
        "spark.cosmos.accountKey": "XXXXXXXXXXXXXXXXXXXXXX",
        "spark.cosmos.database": cosmosDatabaseName,
        "spark.cosmos.container": cosmosContainerName,
    }

data.write.format("cosmos.oltp").options(**cfg).mode("APPEND").save()

Then after this insert I would like to change the Manual Throughput of this container.

alter_table = f"""
        ALTER TABLE cosmosCatalog.`{cosmosDatabaseName}`.{cosmosContainerName} 
        SET TBLPROPERTIES( manualThroughput = '400');
        """
spark.sql(alter_table)

Py4JJavaError: An error occurred while calling o342.sql. : java.lang.UnsupportedOperationException

I find no documentation online on how to change TBLPROPERTIES for a cosmosdb table in sparkSQL. I know I can edit it on the Azure Portal and also with azure cli, but I would like to keep it in sparkSQL.

Upvotes: 1

Views: 135

Answers (1)

Sajeetharan
Sajeetharan

Reputation: 222692

This is not supported with the spark connector for NOSQL API , you might need to track the issue here. So you might need to do it through CLI command or portal or SDK (Java)

FYI : Cosmos NOSQL API container is not similar to Table in SQL, so alter commands will not work.

Upvotes: 1

Related Questions