CODEWITHSUNDEEP
spark-structured-streamingelasticsearch-hadoop
Gokulraj
Gokulraj

Reputation: 520

How to set dynamic doc id in elasticsearch sink using spark structured streaming

In elasticsearch write sink how should I add doc id with dynamic values from the dataset field. In my case I need to set doc id based on a particular field from the formatted dataset. Came across "es.mapping.id" but how would I get values from my dataset?

Upvotes: 1

Views: 361

Answers (1)

Gokulraj
Gokulraj

Reputation: 520

Found that you can achieve this by just specifying the field name as the value for "es.mapping.id"

StreamingQuery query = finalData.writeStream()
                .outputMode(OutputMode.Append())
                .format("org.elasticsearch.spark.sql")
                .option("es.mapping.id", "input_key")        
                .option("checkpointLocation","/tmp/spark-checkpoint")
                .start("spark_index/doc");

Upvotes: 0

Related Questions