Convert output from command or file to json

Question

My script delivers the following outout which i need to convert into json structure. Convert via Perl or other on AIX is part of the path I suppose. Now im reading lots of docs and doing trials (=no success so far)..

Any help/direction is appreciated :-)

TEST                            STATUS          FAIL REASON
----                            ------          -----------
Security Profile                PASS            
Group Check                     FAIL            Errors detected in group definition files
User Check                      FAIL            Errors detected in user definition files
Service hostmibd                FAIL            Active
Service aixmibd                 FAIL            Active
Subserver shell                 FAIL            Active
Subserver kshell                FAIL            Active
Subserver login                 FAIL            Active
Subserver exec                  FAIL            Active
Subserver bootps                FAIL            Active
Subserver tftp                  FAIL            Active
Subserver ntalk                 FAIL            Active
SNMP version                    FAIL            snmpdv3ne
SNMP community                  PASS            
SSHD status                     PASS            

Answer 1

Fixed length data is always a good sign for unpack. You'll need the JSON module for the conversion. This program reads from STDIN and takes files as arguments.

#!/usr/bin/env perl
use warnings;
use strict;
use JSON;

my @names = qw<test status fail_reason>;
my @return;
while(<>){
  next if $. < 3;     # skip the header
  chomp;
  my (%obj);
  @obj{@names} = grep {$_} unpack 'A32A16A*';
  push @return, \%obj;
}

print encode_json \@return;

output:

[{"status":"PASS","fail_reason":null,"test":"Security Profile"},{"status":"FAIL","test":"Group Check","fail_reason":"Errors detected in group definition files"},{"test":"User Check","fail_reason":"Errors detected in user definition files","status":"FAIL"},{"test":"Service hostmibd","fail_reason":"Active","status":"FAIL"},{"status":"FAIL","test":"Service aixmibd","fail_reason":"Active"},{"test":"Subserver shell","fail_reason":"Active","status":"FAIL"},{"status":"FAIL","fail_reason":"Active","test":"Subserver kshell"},{"status":"FAIL","test":"Subserver login","fail_reason":"Active"},{"fail_reason":"Active","test":"Subserver exec","status":"FAIL"},{"status":"FAIL","test":"Subserver bootps","fail_reason":"Active"},{"fail_reason":"Active","test":"Subserver tftp","status":"FAIL"},{"fail_reason":"Active","test":"Subserver ntalk","status":"FAIL"},{"fail_reason":"snmpdv3ne","test":"SNMP version","status":"FAIL"},{"test":"SNMP community","fail_reason":null,"status":"PASS"},{"test":"SSHD status","fail_reason":null,"status":"PASS"}]

Edits:

with this line you can exlude empty fields (if the status is pass, there will be no fail reason).

    delete $obj{$_} foreach grep {$obj{$_} eq '' or not defined $obj{$_} keys %obj; 

If you have more files like this with different field sizes, here is a version that extracts the length of the fields itself:

my (@return, @names, $head_line, $pack_template);
while (<>) {
  chomp;
  if ($. == 1) {      # set the header line
    $head_line = $_;
  }
  elsif($. == 2){
                        # extract the lengths from the second line
    my @lengths = map(length, (/(-+\s*)/g));
    $lengths[-1]= '*';  # replace the last length with * as a catch rest
    $pack_template = join '', map {"A$_"} @lengths;
    @names = unpack $pack_template, $head_line;
  } else {
    my (%obj);
    @obj{@names} = unpack $pack_template;
    push @return, \%obj;
  }
}

Answer 2

This program reads your sample data from the DATA file handle and builds a JSON data structure from it.

The position of the data in each column is established by examining the line of hyphens beneath the headers. The fields are extracted from each line using unpack together with a template derived from those positions.

The JSON module is used to test the result by converting it into a Perl data structure, the result of which is dumped using Data::Dump.

Double quotes embedded within the data is supported.

Hopefully you will be able to amend this code to read from your desired input file instead of from DATA, and do whatever you want with the resulting JSON data.

use strict;
use warnings;
use 5.014;    # For non-destructive tr/// and s///

my $headers = <DATA>;
my $dashes = <DATA>;
my @offsets;
push @offsets, $-[0] while $dashes =~ /-+/g;

my @widths = map { $offsets[$_]-$offsets[$_-1] } 1 .. $#offsets;
push @widths, '*';
my $unpack = join ' ', map "A$_", @widths;

my @headers = map { lc =~ tr/ /_/r } unpack $unpack, $headers;

my @lines;

while (<DATA>) {
  next unless /\S/;
  my @fields = map s/"/\\"/gr, unpack $unpack, $_;
  push @lines, '  {' . join(', ', map qq{"$headers[$_]":"$fields[$_]"}, 0 .. $#fields). '}';
}

my $json = "[\n" . join(",\n", @lines) . "\n]\n";

print $json, "\n\n";

use JSON;
use Data::Dump;
dd from_json $json;


__DATA__
TEST                            STATUS          FAIL REASON
----                            ------          -----------
Security Profile                PASS            
Group Check                     FAIL            Errors detected in group definition files
User Check                      FAIL            Errors detected in user definition files
Service hostmibd                FAIL            Active
Service aixmibd                 FAIL            Active
Subserver shell                 FAIL            Active
Subserver kshell                FAIL            Active
Subserver login                 FAIL            Active
Subserver exec                  FAIL            Active
Subserver bootps                FAIL            Active
Subserver tftp                  FAIL            Active
Subserver ntalk                 FAIL            Active
SNMP version                    FAIL            snmpdv3ne
SNMP community                  PASS            
SSHD status                     PASS            

output JSON

[
  {"test":"Security Profile", "status":" PASS", "fail_reason":""},
  {"test":"Group Check", "status":"FAIL", "fail_reason":"Errors detected in group definition files"},
  {"test":"User Check", "status":"FAIL", "fail_reason":"Errors detected in user definition files"},
  {"test":"Service hostmibd", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Service aixmibd", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver shell", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver kshell", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver login", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver exec", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver bootps", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver tftp", "status":"FAIL", "fail_reason":"Active"},
  {"test":"Subserver ntalk", "status":"FAIL", "fail_reason":"Active"},
  {"test":"SNMP version", "status":"FAIL", "fail_reason":"snmpdv3ne"},
  {"test":"SNMP community", "status":"PASS", "fail_reason":""},
  {"test":"SSHD status", "status":"PASS", "fail_reason":""}
]

output Perl data

[
  { fail_reason => "", status => " PASS", test => "Security Profile" },
  {
    fail_reason => "Errors detected in group definition files",
    status => "FAIL",
    test => "Group Check",
  },
  {
    fail_reason => "Errors detected in user definition files",
    status => "FAIL",
    test => "User Check",
  },
  { fail_reason => "Active", status => "FAIL", test => "Service hostmibd" },
  { fail_reason => "Active", status => "FAIL", test => "Service aixmibd" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver shell" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver kshell" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver login" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver exec" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver bootps" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver tftp" },
  { fail_reason => "Active", status => "FAIL", test => "Subserver ntalk" },
  { fail_reason => "snmpdv3ne", status => "FAIL", test => "SNMP version" },
  { fail_reason => "", status => "PASS", test => "SNMP community" },
  { fail_reason => "", status => "PASS", test => "SSHD status" },
]