CODEWITHSUNDEEP
apache-sparkpysparkdatabricksdelta-lake
efsee
efsee

Reputation: 629

Can't use "update" in outputMode() when writing stream data in spark

I'm trying to write stream data in spark to delta format, but it looks like it won't allow me to use update in outputMode(), below is my code and error message:

deltaStreamingQuery = (eventsDF
  .writeStream
  .format("delta")
  .option("checkpointLocation", checkpointPath)
  .outputMode("update")
  .queryName("stream_1p")
  .start(writePath)
)

AnalysisException: 'Data source com.databricks.sql.transaction.tahoe.sources.DeltaDataSource does not support Update output mode;'```

Upvotes: 2

Views: 5741

Answers (1)

thePurplePython
thePurplePython

Reputation: 2767

Currently Databricks Delta only supports append and complete as outputMode for sinks. append will add new rows to the table and complete will overwrite the table so perhaps this is what you are looking for to incorporate updates.

The official documentation is here => https://docs.databricks.com/delta/delta-streaming.html

Upvotes: 2

Related Questions