CODEWITHSUNDEEP
redisduplicatesjrubylogstash
Kali_89
Kali_89

Reputation: 617

Multiple Logstash instances causing duplication of lines

We're receiving logs using Logstash with the following configuration:

input {
  udp {
    type => "logs"
    port => 12203
  }
}

filter {
  grok {
    type => "tracker"
     pattern => '%{GREEDYDATA:message}'
  }
  date {
    type => "tracker"
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
  }
}

output{
    tcp{
         type => "logs"
         host => "host"
         port => 12203
    }
}

We're then picking the logs up on the machine "host" with the following settings:

input {
      tcp {
                      type => "logs"
                      port => 12203
                        }
}


output {
    pipe {
        command => "python /usr/lib/piperedis.py"
    }
}

From here, we're doing parsing of the lines and putting them into a Redis database. However, we've discovered an interesting problem.

Logstash 'wraps' the log message in a JSON style package i.e.:

{\"@source\":\"source/\",\"@tags\":[],\"@fields\":{\"timestamp\":[\"2013-09-16 15:50:47,440\"],\"thread\":[\"ajp-8009-7\"],\"level\":[\"INFO\"],\"classname\":[\"classname\"],\"message\":[\"message"\]}}

We then, on receiving it and passing it on on the next machine, take that as the message and put it in another wrapper! We're only interested in the actual log message and none of the other stuff (source path, source, tags, fields, timestamp e.t.c.)

Is there a way we can use filters or something to do this? We've looked through the documentation but can't find any way to just pass the raw log lines between instances of Logstash.

Thanks,

Matt

Upvotes: 0

Views: 1255

Answers (2)

Vor
Vor

Reputation: 35129

Why not just extract those messages from stdout?

line = sys.stdin.readline()
line_json = json.loads(line)
line_json['message'] # will be your @message

Upvotes: 0

tomdee
tomdee

Reputation: 2439

The logstash documentation is wrong - it indicates that the default "codec" is plain but in fact it doesn't use a codec - it uses an output format.

To get a simpler output, change your output to something like

output {
    pipe {
        command => "python /usr/lib/piperedis.py"
        message_format =>  "%{message}"
    }
}

Upvotes: 1

Related Questions