CODEWITHSUNDEEP
pysparkdatabricksspark-structured-streamingdelta-lake
sachin thana
sachin thana

Reputation: 21

WriteStream is not able to write Data in Delta Table

I am trying to connect Streaming Json files from the streaming path using the below code

Schema1= "customerId STRING,orderId STRING,products ARRAY<STRUCT<productId: STRING,quantity: STRING,soldPrice: STRING>>,salesRepId STRING,shippingAddress STRUCT<address: STRING,attention: String,city: STRING,state: STRING,zip: STRING>,submittedAt TIMESTAMP";
streamingDF = (spark.readStream.schema(Schema1)\
  .option("maxFilesPerTrigger", 1).json(stream_path))

After few transformations in streamingDF Streaming Dataset and trying to write to a Delta Table using below code

streamingDF.writeStream.outputMode("append")\
  .option("checkpointLocation", orders_checkpoint_path)\
  .partitionBy("submitted_yyyy_mm")\
  .table("sachin")

But those records are not inserted into our delta table and also when I checked the dashboard it shows that numInputRows is 0

Screenshot of streaming while writestream being executed

Why those records are not append into delta table?

Upvotes: 2

Views: 5796

Answers (1)

user3322581
user3322581

Reputation: 210

instead of using .table() use .start() then path the table path instead of the table name:

   streamingDF.writeStream.outputMode("append")
  .option("checkpointLocation", orders_checkpoint_path)
  .partitionBy("submitted_yyyy_mm")
  .start("/pathtotable/sachin")

Upvotes: 2

Related Questions