CODEWITHSUNDEEP
dockerelasticsearchapache-kafkaconfluent-platform
Andrii Fedorenko
Andrii Fedorenko

Reputation: 152

Push own id. Confluent kafka connect elasticsearch docker

I'm using confluentinc/cp-kafka-connect docker image. I'm trying to send JSON file to kafka, with elasticsearch id.

{"_id":10000725, "_source": {"createdByIdentity":"tu_adminn","createdBy":"Admin Testuser"}}

here is my connector

{
  "name": "test-connector",
  "config": {
    "connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
    "tasks.max": "1",
    "topics": "andrii",
    "key.ignore": "false",
    "schema.ignore": "true",
    "connection.url": "http://elasticsearch:9200",
    "type.name": "test-type",
    "name": "elasticsearch-sink"
  }
}

When i'm using key.ignore = true it's generates some weird id. How can i pass exactly my id and source?

Upvotes: 0

Views: 471

Answers (1)

Robin Moffatt
Robin Moffatt

Reputation: 32090

Per the docs:

  • If you specify key.ignore=true then Kafka Connect will use a composite key of your message's kafka topic, partition, and offset -- this is the "weird id" that you're seeing.

  • If you want to use your own ID for the created Elasticsearch document, you can set key.ignore=false and Kafka Connect will use the key of the Kafka message as the ID.

If your Kafka message does not have the appropriate key for what you want to do, you will need to set it. One option is to use something like KSQL: 

CREATE STREAM target AS SELECT * FROM source PARTITION BY _id

Disclaimer: I work for Confluent, the company behind the open-source KSQL project

Upvotes: 2

Related Questions