CODEWITHSUNDEEP
linuxbashcsvlogging
oppa_
oppa_

Reputation: 15

Convert linux log into csv

I'm newbie on linux. I'm so sorry for asking this question again. But I am really appreciate if someone could help me on this. I have trouble on how to convert my linux log to csv file for more readable.

I have apache log as bellow:

[Sun Mar 01 06:01:30 2015] [error] [client 123.456.789.012] File does not exist: /var/www/html/

How can I separate them by column, using: Date (Sun Mar 01 06:01:30 2015), IP (123.456.789.012) only IP, Error Message (File does not exist) and Target (/var/www/html/)?

Thank you

Upvotes: 0

Views: 5324

Answers (3)

Leaderofrock
Leaderofrock

Reputation: 1

I use report command line utility: https://github.com/MatteoGuadrini/pyreports#command-line

$ cat mylog.yml
reports:
  - report:
    input:
     manager: 'log'
     filename: '/tmp/test_log.log'
     # Apache http log format
     params:
       pattern: '([(\d\.)]+) (.*) \[(.*?)\] (.*?) (\d+) (\d+) (.*?) (.*?) (\(.*?\))'
       headers: ['ip', 'user', 'date', 'req', 'ret', 'size', 'url', 'browser', 'host']
    output:
      manager: 'csv'
      filename: '/tmp/mylog.csv'

$ report mylog.yaml

I follow this docs to learn YAML syntax: https://pyreports.readthedocs.io/en/latest/dev/cli.html#command-line-interface

Upvotes: 0

John Damm Sørensen
John Damm Sørensen

Reputation: 82

The easiest way would be to use your own Logformat string. You can modify the standard LogFormat to use TAB instead of space as separator. The standard or Common Log Format usually named combined LogFormat looks like this: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

If you want a TAB separated file simply use this CustomLog statement in your server configuration file: Customlog logs/tabbed-logfile "%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%b\t\"%{Referer}i\"\t\"%{User-Agent}i\""

Upvotes: 1

Steephen
Steephen

Reputation: 15824

There are many ways t achieve it in shell script. Will describe the method in detail and will give a sample example.

You have to identify the delimiter to partition your string and either you can use awk or sed command to partition the fields according to the delimiter

For example in you case you can consider ']' as delimiter s to break the line using the delimiter command will be as follows:

cat logfile | awk -F']' '{print "$1, $2, $3"}' > new_log_file.csv

Upvotes: 1

Related Questions