CODEWITHSUNDEEP
apache-sparkapache-kafkaspark-streaming
ramanKC
ramanKC

Reputation: 317

Does Spark Structured Streaming Kafka Writer supports writing data to particular partition?

Does Spark Structured Streaming's Kafka Writer supports writing data to particular partition? In Spark Structured Streaming Documentation, it is no where mentioned that writing data to specific partition is not supported.

Also I can't see an option to pass "partition id" in section "Writing Data to Kafka"

If it is not supported, any future plans to support or reasons why this is not supported.

Upvotes: 1

Views: 348

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191681

The keys determine which partition to write to - no, you can't hard-code a partition value within Spark's write methods.

Spark does allow you to configure kafka.partitioner.class, though, which would allow you to define the partition number based on the keys of the data

Kafka’s own configurations can be set via DataStreamReader.option with kafka. prefix, e.g, stream.option("kafka.bootstrap.servers", "host:port"). For possible kafka parameters, see ... Kafka producer config docs for parameters related to writing data.

Upvotes: 3

Related Questions