CODEWITHSUNDEEP
elastic-stackfilebeat
AGNEL NANDAPURAPU
AGNEL NANDAPURAPU

Reputation: 1

How can we make single line jsons in filebeat output file

I used FileBeat to convert apache2 logs and dump into an output file. the generate output file has a JSON body for each apache2 log message. but here I need JSON body in a single line instead of parsing.

With the current format, I'm not able to fetch each JSON body from the output file, if we keep line wise JSON, then we can split each line of the output file, and can take the each JSON body easily.

I modified the filebeat.yml file output configurations. like below

output.file: path: "/tmp/logstojson/apache2/" filename: filebeat permissions: 0600 codec.json: pretty: false

I didn't find the expected format in the output file.

Actual output file data : 

{ "@timestamp": "2019-04-09T13:12:47.106Z", 

  "@metadata": {

    "beat": "filebeat",

    "type": "doc",

    "version": "6.6.2",

    "pipeline": "filebeat-6.6.2-system-auth-pipeline"

  },

  "fileset": {

    "module": "system",

    "name": "auth"

  },

  "prospector": {

    "type": "log"

  },

  "input": {

    "type": "log"

  },

  "source": "/var/log/auth.log",

  "log": {

    "file": {

      "path": "/var/log/auth.log"

    }

  },

  "message": "Apr  9 13:12:45 ip-172-31-22-12 sudo: 
pam_unix(sudo:session): session closed for user root",

  "event": {

    "dataset": "system.auth"

  },

  "beat": {

    "version": "6.6.2",

    "name": "ip-172-31-22-12",

    "hostname": "ip-172-31-22-12"

  },

  "host": {

    "name": "ip-172-31-22-12"

  },

  "offset": 537068

}

expected output file data : 

{ "@timestamp": "2019-04-09T13:12:47.106Z", "@metadata": {  "beat": "filebeat", "type": "doc", "version": "6.6.2",  "pipeline": "filebeat-6.6.2-system-auth-pipeline" }, "fileset": { "module": "system", "name": "auth" }, "prospector": { "type": "log" }, "input": { "type": "log" }, "source": "/var/log/auth.log", "log": { "file": {  "path": "/var/log/auth.log" } }, "message": "Apr  9 13:12:45 ip-172-31-22-12 sudo: pam_unix(sudo:session): session closed for user root", "event": { "dataset": "system.auth" }, "beat": { "version": "6.6.2", "name": "ip-172-31-22-12",  "hostname": "ip-172-31-22-12"  }, "host": { "name": "ip-172-31-22-12" }, "offset": 537068}

Upvotes: 0

Views: 1169

Answers (1)

littledaxter
littledaxter

Reputation: 121

Not a full answer but it's normal if you don't see a difference when setting the codec.json: pretty: false since the default value is already false (see: filebeat codec doc)

the other solution i could see would be to manualy format your json output via a codec.format: string: '%{[@timestamp]} %{[message]}'

But that's far from clean or efficient (if not totally impossible if your input has some level of variation or complexity)

Maybe FB supports other codec not documented in that page; ES's doc can lack precision sometimes so that could be worth investigating.

On another note: if you need that kind of transformation/formatting capabilities on your logs, you might want to consider using logstash instead (or, if you also need FB's lightweight transportation, use both FB and LS). LS supports a file or filebeat input and a file output and it's basically just a more complete ETL utility than FB (which is more of a lightweight forwarder). You should definitively consider it for these kind of issues. (see LS file output doc: By default, this output writes one event per line in json format. You can customise the line format using the line codec like)

Upvotes: 0

Related Questions