CODEWITHSUNDEEP
pythonnode.js
Christophermp
Christophermp

Reputation: 186

How to parse log data from file in nodejs or Python?

I have several servers that provide me a file with hundreds of lines in this format:

Jun  8 19:17:52 icmp-73260f user.info SM: SM- Security log event: Playout::CPLEnd

Jun  8 19:17:52 icmp-73260f user.info SM: SM- Security log event: Playout::PlayoutComplete

Jun  8 19:17:52 icmp-73260f user.debug SM: IMB Event- End of track CRC values: ffbbffbb - 00c7ffbb - 54c783e4 - 00e483e4

Jun  8 19:17:52 icmp-73260f user.debug SM: IMB Controller- Notify STOPPED state for frame 28465

How could i parse this in python or Nodejs?

The plan is to split them up in Date, Device, User info, then message.

I plan to store this data in a DB, but that part i have covered.

Upvotes: 0

Views: 223

Answers (1)

Wytamma Wirth
Wytamma Wirth

Reputation: 563

Here's a simple way to do this in Python. However, it's not very robust to changes in the log structure. I suggest you look into regex if you think the log structure will change a lot.

from datetime import datetime

logs = []

with open("log.txt", "r") as log_file:
  for line in log_file:
    line_list = line.split()
    log = {}
    date = datetime.strptime(" ".join(line_list[:3]), '%b %d %H:%M:%S')
    log['Date'] = date.replace(year=2019) # no year in log
    log['Device'] = line_list[3]
    log['User_info'] = line_list[4]
    log['Message'] = " ".join(line_list[6:])
    logs.append(log)

print(logs)

[   {   'Date': datetime.datetime(2019, 6, 8, 19, 17, 52),
        'Device': 'icmp-73260f',
        'User_info': 'user.info',
        'message': 'SM- Security log event: Playout::CPLEnd'},
    {   'Date': datetime.datetime(2019, 6, 8, 19, 17, 52),
        'Device': 'icmp-73260f',
        'User_info': 'user.info',
        'message': 'SM- Security log event: Playout::PlayoutComplete'},
    {   'Date': datetime.datetime(2019, 6, 8, 19, 17, 52),
        'Device': 'icmp-73260f',
        'User_info': 'user.debug',
        'message': 'IMB Event- End of track CRC values: ffbbffbb - 00c7ffbb - '
                   '54c783e4 - 00e483e4'},
    {   'Date': datetime.datetime(2019, 6, 8, 19, 17, 52),
        'Device': 'icmp-73260f',
        'User_info': 'user.debug',
        'message': 'IMB Controller- Notify STOPPED state for frame 28465'}]

Upvotes: 2

Related Questions