CODEWITHSUNDEEP
elasticsearchlogstashlogstash-configuration
Ajay Gupta
Ajay Gupta

Reputation: 157

parse json array string using logstash

I want to parse this json using logstash.

{"name":"bob","last":"builder", "atts":"{\"a\":111, \"b\":222}"}

{ "name" => "bob", "last" => "builder" "atts" => { "a" => 111, "b" => 222} }

Upvotes: 0

Views: 4670

Answers (1)

Dhruv Madhok
Dhruv Madhok

Reputation: 54

Two options!

Parsing JSON using Logstash

If you want to parse JSON using logstash- would refer to the logstash plugin here:

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

To achieve this- you'd be toying with the filter part of your logstash.conf:

filter {
   json {
     source => "message"
   }
}

there are more examples of json decoding in that link.

Parsing JSON using Filebeat

Your other option would be to decode json on the filebeat side before it gets into logstash. Relevant links:

https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html

https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/5

https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/7

https://discuss.elastic.co/t/how-to-read-json-file-using-filebeat-and-send-it-to-elasticsearch/91802

Here's a sample filebeat.yml for this situation:

filebeat.inputs:
  - type: log
    paths:
      - 'path to the log directory you want to track'
    enter code here
    input_type: log
    json.keys_under_root: true
    json.add_error_key: true
    fields:
        log_type: 'type of log'

    processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true

    - add_tags:
        tags:
            - 'tag in elastic'

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

setup.template.settings:
  index.number_of_shards: 1

output.logstash:
  # The Logstash hosts
  hosts: ["where logstash is running"]
  index: 'your index'

  codec.json:
    pretty: true
    escape_html: false

#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- decode_json_fields:
    fields: ["message"]
    process_array: true

json.keys_under_root: true
json.add_error_key: true

and 

processors:
    - decode_json_fields:
        fields: ["message"]
        process_array: true

does the trick.

Upvotes: 2

Related Questions