CODEWITHSUNDEEP
elasticsearchapache-kafkaapache-kafka-connectconfluent-platform
Rahul
Rahul

Reputation: 1

Topic mapping when streaming from Kafka to Elasticsearch

When I transfer or stream two and three tables then I can easily map in Elasticsearch but can I map automatically map topics to index

I have streamed data from PostgreSQL to ES by mapping manually topic.index.map=topic1:index1,topic2:index2, etc.

Can I map automatically whatever topics send by producer then consumer consume in ES connector automatically?

Upvotes: 0

Views: 292

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191738

By default, the topics map directly to an index of the same name.

If you want "better" control, you can use RegexRouter in a transforms property

To quote the docs

topic.index.map

This option is now deprecated. A future version may remove it completely. Please use single message transforms, such as RegexRouter, to map topic names to index names

If you cannot capture a single regex for each topic in the connector, then run more connectors with a different pattern

Upvotes: 1

Related Questions