CODEWITHSUNDEEP
javaapache-kafkakafka-producer-apiapache-kafka-connect
Pawan
Pawan

Reputation: 53

Read a CSV file in real time using Kafka Connect

how can I integrate Kafka connect with Kafka producer in java so that I can read a CSV file in real time? I am having a hard time finding any source related to this.

Right now I am using scala-ide to run a simple Kafka producer class, but have no idea on how to use it with Kafka connect.

Upvotes: 4

Views: 9626

Answers (4)

Reda Arté Kejji
Reda Arté Kejji

Reputation: 131

The issue with the casual spooldir connector is that you need to have the csv file inside the pod/ container of kafka-connect. It might not be secure for large sized files.

Upvotes: 0

fhussonnois
fhussonnois

Reputation: 1727

Just another Kafka Connect plugin for reading CSV files : https://github.com/streamthoughts/kafka-connect-file-pulse

Hope this project can help people looking for similar solution.

Disclaimer: I am one of the contributors to this project

Upvotes: 0

Robin Moffatt
Robin Moffatt

Reputation: 32090

A Kafka Connect connector for reading CSV files already exists: https://github.com/jcustenborder/kafka-connect-spooldir.

Yuo can see an example of it in action here: https://www.confluent.io/blog/ksql-in-action-enriching-csv-events-with-data-from-rdbms-into-AWS/

Disclaimer: I wrote the above article, and work for Confluent on whose blog it was published

Upvotes: 4

Jakub
Jakub

Reputation: 4006

Kafka Connect has a producer already built in. You just need to use the right connector plugin. For reading a CSV file, the FileStreamSource connector which is part of Kafka should work. You can start the Kafka Connect probably in the standalone mode (see Kafka docs) and configure the connector. The example config files which are part of Kafka should help you to get started.

Upvotes: 1

Related Questions