CODEWITHSUNDEEP
apache-sparkhivespark-structured-streaming
Naman Agarwal
Naman Agarwal

Reputation: 654

How to write a Streaming Structured Stream into Hive directly?

I want to achieve something like this :

df.writeStream
.saveAsTable("dbname.tablename")
.format("parquet")
.option("path", "/user/hive/warehouse/abc/")
.option("checkpointLocation", "/checkpoint_path") 
.outputMode("append")
.start()

I am open to suggestions. I know Kafka Connect could be one of the options but how to achieve this using Spark. A possible workaround may be what I am looking for. Thanks in Advance !!

Upvotes: 1

Views: 786

Answers (1)

Ged
Ged

Reputation: 18013

Spark Structured Streaming does not support writing the result of a streaming query to a Hive table directly. You must write to paths.

For 2.4 they say try foreachBatch, but I have not tried it.

Upvotes: 2

Related Questions