CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connect
Sandesh
Sandesh

Reputation: 438

kafka data to elaticsearch

I am trying to send data from apache kafka producer to elastic search using kafka connect elasticsearch

Getting following error in elasticsearch:

[2017-12-21T11:00:54,979][DEBUG][o.e.a.b.TransportShardBulkAction] 
[pageviews7][0] failed to execute bulk item (index) BulkShardRequest 
[[pageviews7][0]] containing [index {[pageviews7][kafkaconnect]
[pageviews7+0+0], source[{"key1":"value1"}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to find 
type parsed [string] for [key1]

Following is my kafka-connect-elasticsearch properties file:

name=elasticsearch-sink
connector.class=
io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=pageviews7
key.ignore=true
connection.url=http://localhost:9200
type.name=kafkaconnect
schemas.enable=false
schema.ignore=true

Upvotes: 1

Views: 746

Answers (3)

Robin Moffatt
Robin Moffatt

Reputation: 32090

What version of Elasticsearch are you running? You may fine that setting schema.ignore=true in your Kafka Connect connector config will workaround this error.

Upvotes: 0

Antonios Chalkiopoulos
Antonios Chalkiopoulos

Reputation: 868

Try the stream-reactor Kafka connectors for Elastic-Search. One uses the TCP protocol and the other one the HTTP protocol, and they have been battle-tested in Production sinking 5+ Billion events / day into ES indexes, and they are pretty easy to configure and well documented

https://github.com/landoop/stream-reactor

Upvotes: 1

Ehud Lev
Ehud Lev

Reputation: 2901

This looks like a problem with indexing the document into Elasticsearch:

  1. Try to post your document into ES manually
  2. You should get the same error. Which means you have some problem with your index mapping.
  3. If the index has a problem with the mapping as I suspect, you can either write to another index using topic.index.map property or delete the current index (only if you don't care about the data).

Example of how to curl manually into ES:

curl -XPOST <ES-url>/pageviews7/test_id -d '{"key1":"value1"}'

Delete you current inedex:

curl -XDELETE <ES-url>/pageviews7

Upvotes: 0

Related Questions