CODEWITHSUNDEEP
pythondatabrickssql-data-warehouse
Brian Custer
Brian Custer

Reputation: 35

Error while using the write method of a dataframe object on databricks

I am trying to write some data to Azure SQL Data Warehouse using Azure Databricks and Python. The code I am using is as follows:

salesdf.write\
   .format('com.databricks.spark.sqldw')\
   .option('url', sqlDwUrlSmall)\
   .option('dbtable', 'StgSales')\
   .option( 'forward_spark_azure_storage_credentials','True')\
   .option('tempdir', tempDir)\
   .mode('overwrite')\
   .save()

This method fails without a specific error message. The traceback doesn't make any sense either. I'm pretty sure I have all of the options and format stated correctly but it is still not working.

Upvotes: 1

Views: 757

Answers (1)

Ritesh
Ritesh

Reputation: 1036

Not sure, but I think the issue is with syntax. This is what I found in official documentation.

df.write \
  .format("com.databricks.spark.sqldw") \
  .option("url", "jdbc:sqlserver://<the-rest-of-the-connection-string>") \
  .option("forwardSparkAzureStorageCredentials", "true") \
  .option("dbTable", "my_table_in_dw_copy") \
  .option("tempDir", "wasbs://<your-container-name>@<your-storage-account-name>.blob.core.windows.net/<your-directory-name>") \
  .save()

Check .option("forwardSparkAzureStorageCredentials", "true") in your code, it seems you have put (_) underscore.

Upvotes: 1

Related Questions