CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connect
Viacheslav Shalamov
Viacheslav Shalamov

Reputation: 4427

Kafka-connect elasticsearch auto-lowercase topic name for for index

I'm using elasticsearch sink kafka-connector to index messages from multiple kafka topics to elasticsearch. I have my topics with camelCase naming, and I can't change it. So when starting up the ES sink connector, it does not index anything because elaticsearch has problems with non-lowercase index names.

I know I can use topic.index.map property to manually convert topic name to index.
topic.index.map=myTopic1:mytopic1, myTopic2:mytopic2,...

Is there a way to convert to lowercase automatically? I have dozens of topics to convert, and I suspect it to be around hundred soon.

Upvotes: 1

Views: 629

Answers (2)

Robin Moffatt
Robin Moffatt

Reputation: 32110

As of recent versions of the Elasticsearch sink connector this is done automatically. The PR that fixed this was https://github.com/confluentinc/kafka-connect-elasticsearch/pull/251

Upvotes: 1

Viacheslav Shalamov
Viacheslav Shalamov

Reputation: 4427

Found out that since 5.1 they do that automatically, if mapping is not specified for the topic.
from here:

final String indexOverride = topicToIndexMap.get(topic);
String index = indexOverride != null ? indexOverride : topic.toLowerCase();

See this commit for details.

Upvotes: 1

Related Questions