Parsing text to csv in python with headers- F5 LTM

Question

I'm trying to audit a lot of f5 configuration and im having a difficult time parsing the data below, i tried modifying the code below but it throws me an error. I'm a complete noob on python and this is my first time automating a task like this. thanks

Data:

Ltm::Virtual Server: acme.com
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  Destination      : 10.1.1.2:80
Ltm::Virtual Server: foo.com
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  Destination      : 10.100.11.15:80
Ltm::Virtual Server: hamhamspam.com
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  Destination      : 10.200.8.17:443

Expected Output

Virtual Server  Availability    State   Reason                                  Destination
acme.com        offline         enabled The children pool member(s) are down    10.1.1.2:80
foo.com         available       enabled The virtual server is available         10.100.11.15:80
hamhamspam.com  offline         enabled The children pool member(s) are down    10.200.8.17:443

import csv

def convert_to_dict(line, header):
    d = {}
    for cell in header:
        d[cell] = ''

    row = line.strip().split(':')
    for cell in row:
        if cell:
            key, value = cell.split(':')
            d[key] = value

    return d

def extract_fields(logfile):
    fields = set()
    for line in logfile:
        row = line.strip().split(':')
        for cell in row:
            if cell:
                key, value = cell.split(':')
                fields.add(key)

    logfile.seek(0)
    return sorted(list(fields))

if __name__ == '__main__':
    with open('ltm.txt', 'r') as logfile:
        with open('report.csv', 'wb') as csvfile:
            csvwriter = csv.writer(csvfile)

            header = extract_fields(logfile)
            csvwriter.writerow(header)

            for line in logfile:
                d = convert_to_dict(line, header)
                csvwriter.writerow([d[cell] for cell in header])

Error encountered:

error:
C:\Users\test\PycharmProjects\F5Parser\venv\Scripts\python.exe C:/Users/test/PycharmProjects/F5Parser/vip_ltm_parser
Traceback (most recent call last):
  File "C:/Users/test/PycharmProjects/F5Parser/vip_ltm_parser", line 33, in <module>
    header = extract_fields(logfile)
  File "C:/Users/test/PycharmProjects/F5Parser/vip_ltm_parser", line 22, in extract_fields
    key, value = cell.split(':')
ValueError: not enough values to unpack (expected 2, got 1)

Process finished with exit code 1

Answer 1

working code below:

lines_data = []
with open('ltm.txt', 'r') as logfile:
    pairs = {}
    for line in logfile:
        new_entry = False
        if line[0].isspace():
          fields = line.strip().split(':')
        else:
          double_colon_idx = line.find('::')
          line = line[double_colon_idx+2:]
          new_entry = True
        fields = line.strip().split(':')
        if new_entry and pairs:
          lines_data.append(pairs)
          pairs = {}
        if len(fields) >= 2:
          key = fields[0]
          value = fields[1]
          pairs[key.strip()] = value.strip()
    lines_data.append(pairs)
headers = lines_data[0].keys()
header_str = ','.join(headers)
with open('report.csv','w') as out:
  out.write(header_str + '\n')
  for entry in lines_data:
    _line = []
    for key in headers:
      _line.append(entry[key])
    out.write(','.join(_line) + '\n')

ltm.txt

Ltm::Virtual Server: acme.com
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  Destination      : 10.1.1.2:80
Ltm::Virtual Server: foo.com
  Availability     : available
  State            : enabled
  Reason           : The virtual server is available
  Destination      : 10.100.11.15:80
Ltm::Virtual Server: hamhamspam.com
  Availability     : offline
  State            : enabled
  Reason           : The children pool member(s) are down
  Destination      : 10.200.8.17:443

report.csv

Virtual Server,Availability,State,Reason,Destination
acme.com,offline,enabled,The children pool member(s) are down,10.1.1.2
foo.com,available,enabled,The virtual server is available,10.100.11.15
hamhamspam.com,offline,enabled,The children pool member(s) are down,10.200.8.17

Answer 2

You have different problem in your code.

1. you try to split on ':' while one of the keys does contain colons. You should split on ': '

2. you consistently try to split twice, first time the line (which is correct) then each field (which is wrong). You should remove one split and strip after the split:

    ...
    for line in logfile:
        row = line.split(': ')
        key = row[0].strip()
        fields.add(key)
   

   Same for the other function

3. You process each line separately, while lines beginning with a space character are continuation lines. You should only extract a pair key, value and return it:

    def extract_pair(line):
        key, value = line.split(': ')
        return key.strip(), value.strip()
   

   Then in you main, you have to process the continuation lines

        ...
        d = None
        for line in logfile:
            key, value = extract_pair(line)
            if line[0].isspace():
                d[key] = value       # continuation line: update d
            else:
                if d is not None:    # write full previous row
                    csvwriter.writerow([d[cell] for cell in header])
                d = {key: value}     # initial line: reset d
        if d is not None:            # process last row
            csvwriter.writerow([d[cell] for cell in header])