CODEWITHSUNDEEP
pysparkazure-synapseazure-data-explorerazure-synapse-analytics
DxG
DxG

Reputation: 209

Exception while writing data to kusto from Azure Synapse notebook

While writing to Kusto from Azure Synapse notebook using the below pyspark code. I get an exception if the table already exists.

df.write \
    .format("com.microsoft.kusto.spark.synapse.datasource") \
    .option("spark.synapse.linkedService", "<link service name>") \
    .option("kustoDatabase", "<Database name>") \
    .option("kustoTable", "<Table name>") \
    .option("tableCreateOptions","CreateIfNotExist") \
    .mode("Append") \
    .save()

Exception:

java.lang.ClassCastException: java.lang.String cannot be cast to org.json.JSONObject
    at com.microsoft.kusto.spark.datasink.KustoWriter$.$anonfun$write$3(KustoWriter.scala:69)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.Iterator.foreach(Iterator.scala:943)
    at scala.collection.Iterator.foreach$(Iterator.scala:943)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)

Versions in synapse pool Spark 3.3 Python 3.10 Scala 2.12.17 Java 11

Upvotes: 1

Views: 258

Answers (1)

ShaikMaheer
ShaikMaheer

Reputation: 273

From the error message, it seems your dataframe has a column with string data and you are trying to load that column into Kusto table column, where its datatype is object type.

Check your column datatypes in dataframe and Kusto table and make sure they are matching and then perform write operation.

Upvotes: 1

Related Questions