CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connectconfluent-platform
hithendra sharma
hithendra sharma

Reputation: 23

Kafka connect elastic search ID creation for multiple fields not working

I am asking this question as there was no answer in the original case: Elastic Kafka Connector, ID Creation.

I have a similar situation.

Elastic search table to create a record for a single field, but not for multiple fields when request sent through kafkaconnect.

Getting exception "Key is used as document id and can not be null" in elastic search.

My Connector Configurations:

{
 "name": "test-connector33",
 "config": {
 "connector.class":"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
 "tasks.max": "1",
 "topics": "test-connector33",
 "connection.url": "http://localhost:9200",
 "type.name": "aggregator",
 "schema.ignore": "true",
 "topic.schema.ignore": "true",
  "topic.key.ignore": "false",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": "false", 
 "key.converter": "org.apache.kafka.connect.json.JsonConverter",
 "key.converter.schemas.enable": "false",
 "key.ignore":"false",
 "name": "test-connector33",
"transforms": "InsertKey,extractKey",
"transforms.InsertKey.type":"org.apache.kafka.connect.transforms.ValueToKey",
"transforms.InsertKey.fields":"customerId,city",
"transforms.extractKey.type":"org.apache.kafka.connect.transforms.ExtractField$Key",
"transforms.extractKey.field":"customerId,city"
}}

Any idea how to resolve this?

Thanks in advance!

Upvotes: 0

Views: 528

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191748

org.apache.kafka.connect.transforms.ExtractField$Key only supports single fields.

Pretend your JSON object was a HashMap<String, Object>. You cannot find the field customerId,city, and so that map.get(field) operation returns null, therefore setting the field to be null.

If you want to send keys via the console producer, you are welcome to do that by adding --property print.key=true as a flag, then typing the key, press tab, then putting the value. If you want to echo data into the process, then you can also set --property key.separator='|', for a vertical bar as well as add --property parse.key=true

Upvotes: 1

Related Questions