CODEWITHSUNDEEP
apache-kafkaconfluent-platformapache-kafka-connect
vijay kumar vijju
vijay kumar vijju

Reputation: 133

Kafka Connect HDFS Sink Issue

I am trying to stream data using Kafka-Connect with HDFS Sink Connector. Both Standalone and Distributed modes are running fine but its writing into HDFS only once (based on flush-size) and not streaming later on. Please help if I'm missing some thing.

Confluent 2.0.0 & Kafka 0.9.0

Upvotes: 4

Views: 1088

Answers (1)

Gaurav Mishra
Gaurav Mishra

Reputation: 1019

I faced this issue long back.Just check below parameter is missing

Connect-hdfs-sink properties

"logs.dir":"/hdfs_directory/data/log"

"request.timeout.ms":"310000"

"offset.flush.interval.ms":"5000"

"heartbeat.interval.ms":"60000"

"session.timeout.ms":"300000

"max.poll.records":"100"

Upvotes: 2

Related Questions