CODEWITHSUNDEEP
elasticsearchlogstash
James Jiang
James Jiang

Reputation: 2183

Can I delete the message field from Logstash?

I have a basic Logstash -> Elasticsearch setup, and it turns out the 'message' field is not required after the logstash filter done its job - storing this raw message field to elasticsearch is only adding unnecessary data to storage imo.

Can I safely delete this field and would it cause any trouble to ES? advices or readings are welcome, thanks all.

Upvotes: 24

Views: 26118

Answers (3)

Graham Hannington
Graham Hannington

Reputation: 1957

I would have added the following as a comment to the answer by Ben Lim, but I do not know how to add a code block in a comment, or even whether that is possible...

If you can use a combination of input and codec that does not create a message field, then you do not need to remove it.

For example, the following combination of input and codec (JSON Lines over TCP) does not create a message field:

input {
  tcp {
    port => 5044
    codec => json_lines
  }
}
output {
  elasticsearch {
    hosts => ["localhost"]
    document_type => "mytype"
    index => "myindex"
  }
}

Upvotes: 0

Steve
Steve

Reputation: 7098

You can also do this within the json filter.

filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

Upvotes: 11

Ban-Chuan Lim
Ban-Chuan Lim

Reputation: 7890

No, it will not cause any trouble to ES. You can delete message field if it is redundant or unused.

You can add this filter to end of the filters.

mutate
{
     remove_field => [ "message" ]
}

Upvotes: 38

Related Questions