How to send the data from Kafka to ElasticSearch using python

Question

i am using Python Language. I have csv file, I need to convert into json and send to kafka and then to ElasticSearch.

I am able to convert Csv to Json and send to kafka consumer. How to get the data from Kafka Consumer to ElasticSearch

Answer 1

You can use Logstash with Kafka input plugin to send data to Elasticsearch. It's easy to configure and run. https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

Answer 2

You don't need to write any Python. Kafka itself provides a streaming integration API, called Kafka Connect. To use it, you just configure it with JSON files.

• Ingest CSV file into Kafka with https://github.com/jcustenborder/kafka-connect-spooldir
  • Example of it in use: https://www.confluent.io/blog/ksql-in-action-enriching-csv-events-with-data-from-rdbms-into-AWS/
• Stream data from Kafka topic to Elasticsearch https://www.confluent.io/connector/kafka-connect-elasticsearch/
  • Example of it in use: https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2

Disclaimer: I work for Confluent, and wrote the above articles.