CODEWITHSUNDEEP
apache-kafkalogstashelastic-stackfilebeatlogstash-file
Jinal Kothari
Jinal Kothari

Reputation: 1

Why logstash gives filebeat metadata for kafka input plugin?

I'm running a logstash pipeline using kafka input plugin with following configuration

kafka {
    bootstrap_servers => "xxxx"
        topics => ['topic1', 'topic2', 'topic3' ... , 'topic50']
        auto_offset_reset => 'latest'
        group_id => 'xyz'
        decorate_events => 'basic'
    }

I want a field with kafka @metadata value so I added a new field - kafka_topic using

mutate {
        add_field => { 'kafka_topic' => '%{[@metadata]}' }
    }

but kafka_topic field contains some values of filebeat

{"beat":"filebeat","type":"_doc","version":"7.16.0"}
{“beat":"filebeat","topic":"a","type":"doc","version":"6.8.3"}
{“beat":"filebeat","topic":"b","type":"doc","version":"6.8.3"}
{“kafka”:{“topic":"c","consumer_group":"x","partition":11,"offset":5403823,"key":null,"timestamp":1690370271634}}
{“kafka":{"topic":"d","consumer_group":"x","partition":11,"offset":5403824,"key":null,"timestamp":1690370294548}}

I'm unable to understand why it is giving filebeat metadata for some topics

Upvotes: 0

Views: 117

Answers (0)

Related Questions