CODEWITHSUNDEEP
elasticsearchfilebeat
Optimize
Optimize

Reputation: 103

Use filebeat to ingest JSON log file

So I have a log file that each line of it is a json object. I want to be able to send this log files directly to elasticsearch, and then hopefully elastic would ingest the data.

I'm pretty sure I need to declare a specific template for this. However, I'm not sure how, and would be happy to have some guidance of how to do it right.

Upvotes: 0

Views: 1399

Answers (1)

Surendra Deshpande
Surendra Deshpande

Reputation: 428

   #Filebeat Configuration
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -

      paths:
        #- /var/log/*.log
        - ${applicationLogsPath}
      document_type: application_logs

      # Mutiline can be used for log messages spanning multiple lines.
      multiline:

        # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
        pattern: ^%{TIMESTAMP_ISO8601}

        # Defines if the pattern set under pattern should be negated or not. Default is false.
        negate: true

        # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
        # that was (not) matched before or after or as long as a pattern is not matched based on negate.
        # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
        match: after

    # Additional prospector
    -
      paths:
        - ${iisLogsPath}
      document_type: iis_logs

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

output:

  ### Logstash as output
  elasticsearch:
    # The elasticsearch hosts
    hosts: ["${elasticsearchHost}:9200"]

    # Number of workers per Logstash host.
    #worker: 1

    # The maximum number of events to bulk into a single batch window. The
    # default is 2048.
    #bulk_max_size: 2048

This is a default template I use to ingest logs into elasticsearch through Filebeat.. You can also send the logs to logstash and filter your logs to capture information that is necessary and then let logstash forward the logs to Elasticsearch..

Please let me know if you need anything else..

Thanks,

Upvotes: 2

Related Questions