CODEWITHSUNDEEP
logstashlogstash-configuration
Rexmuller
Rexmuller

Reputation: 49

Logstash - parse array of JSON

I'm trying to parse SendGrid Webhook events using Logstash. The issue is that Logstash's output is not an array of JSON, but only JSON. Square brackets are missing.

The reason I'm doing this is GeoIP location and UserAgent parsing for analytics.

I am posting to 127.0.0.1:3000, then I want to forward the output to 127.0.0.1:8080. 8080 is just a basic Express server which prints requests/responses and it sends final data to ElasticSearch.

This is the input:

[
  {
    "email": "[email protected]",
    "event": "click",
    "ip": "8.8.8.8",
    "sg_event_id": "WS1wXXhERnefBsqEt5FSFA",
    "sg_message_id": "mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0",
    "timestamp": 1596484698,
    "url": "http://10.0.0.6/ServiceCenter/view",
    "url_offset": { "index": 1, "type": "html" },
    "useragent": "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
  }
]

This is the output:

{
  sg_event_id: 'WS1wXXhERnefBsqEt5FSFA',
  event: 'click',
  email: '[email protected]',
  sg_message_id: 'mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0',
  timestamp: 1596484698,
  useragent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
  ip: '8.8.8.8',
  url: 'http://10.0.0.6/ServiceCenter/view',
  url_offset: { index: 1, type: 'html' }
}

This is my config:

input {
  http {
    host => "127.0.0.1"
    port => "8080"
  }
}

filter {
  mutate {
    remove_field => [ "@version", "@timestamp", "headers", "host" ]
  }
}

output {
  http {
    http_method => "post"
    url => "http://127.0.0.1:3000"
  }
  stdout {
    codec => rubydebug
  }
}

Upvotes: 0

Views: 1079

Answers (1)

Rexmuller
Rexmuller

Reputation: 49

So finally, I've found an workaround to do it using the json_encode filter.

  1. Used ruby code to store keys and values to @DATA[oldJSON]
  2. Used json_encode plugin on @DATA[oldJSON] and save the results to @DATA[newJSON]
  3. Important: Set http output format => message and content_type => "application/json; charset=UTF-8". Default is text/plain and we don't want this.
  4. Set message value to '[ %{[@DATA][newJSON]} ]'

Config:

input {
  http {
    host => "127.0.0.1"
    port => "8080"
  }
}

filter {
  mutate {
    remove_field => [
      "@version",
      "@timestamp",
      "headers",
      "host"
    ]
  }
      ruby {
        code => '
            event.to_hash.each { |k,v|
                event.set("[@DATA][oldJSON][#{k}]", v)
    }
        '
  }
    json_encode {
      source => "[@DATA][oldJSON]"
      target => "[@DATA][newJSON]"
  }
  
}

output { 
  http {
    http_method => "post"
    url => "http://127.0.0.1:3000"
    format => message
    content_type => "application/json; charset=UTF-8"
    message => '[ %{[@DATA][newJSON]} ]'
    }
  }

Output:

[
  {
    ip: '8.8.8.8',
    sg_message_id: 'mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0',
    url_offset: { type: 'html', index: 1 },
    sg_event_id: 'WS1wXXhERnefBsqEt5FSFA',
    email: '[email protected]',
    event: 'click',
    url: 'http://10.0.0.6/ServiceCenter/view',
    timestamp: 1596484698,
    useragent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko'
  }
]

Maybe somebody will find this useful.

Upvotes: 1

Related Questions