CODEWITHSUNDEEP
pythonjsonpandasdataframe
OverflowingTheGlass
OverflowingTheGlass

Reputation: 2434

Pandas Dataframe to JSON Hierarchy

I have exhaustively reviewed/attempted implementations all the other questions on SO corresponding to this challenge and have yet to reach a solution.

Question: how do I convert employee and supervisor pairs into a hierarchical JSON structure to be used for a D3 visualization? There are an unknown number of levels, so it has to be dynamic.

I have a dataframe with five columns (yes, I realize this isn't the actual hierarchy of The Office):

  Employee_FN Employee_LN Supervisor_FN Supervisor_LN  Level
0     Michael       Scott          None          None      0
1         Jim     Halpert       Michael         Scott      1
2      Dwight     Schrute       Michael         Scott      1
3     Stanley      Hudson           Jim       Halpert      2
4         Pam     Beasley           Jim       Halpert      2
5        Ryan      Howard           Pam       Beasley      3
6       Kelly      Kapoor          Ryan        Howard      4
7    Meredith      Palmer          Ryan        Howard      4

Desired Output Snapshot:

{
  "Employee_FN": "Michael",
  "Employee_LN": "Scott",
  "Level": "0",
  "Reports": [{
        "Employee_FN": "Jim",
        "Employee_LN": "Halpert",
        "Level": "1",
        "Reports": [{
              "Employee_FN": "Stanley",
              "Employee_LN": "Hudson",
              "Level": "2",
            }, {
              "Employee_FN": "Pam",
              "Employee_LN": "Beasley",
              "Level": "2",
            }]
        }]
}

Current State:

j = (df.groupby(['Level','Employee_FN','Employee_LN'], as_index=False)
             .apply(lambda x: x[['Level','Employee_FN','Employee_LN']].to_dict('r'))
             .reset_index()
             .rename(columns={0:'Reports'})
             .to_json(orient='records'))

print(json.dumps(json.loads(j), indent=2, sort_keys=True))

Current Output:

[
  {
    "Employee_FN": "Michael",
    "Employee_LN": "Scott",
    "Level": 0,
    "Reports": [
      {
        "Employee_FN": "Michael",
        "Employee_LN": "Scott",
        "Level": 0
      }
    ]
  },
  {
    "Employee_FN": "Dwight",
    "Employee_LN": "Schrute",
    "Level": 1,
    "Reports": [
      {
        "Employee_FN": "Dwight",
        "Employee_LN": "Schrute",
        "Level": 1
      }
    ]
  },
  {
    "Employee_FN": "Jim",
    "Employee_LN": "Halpert",
    "Level": 1,
    "Reports": [
      {
        "Employee_FN": "Jim",
        "Employee_LN": "Halpert",
        "Level": 1
      }
    ]
  },
  {
    "Employee_FN": "Pam",
    "Employee_LN": "Beasley",
    "Level": 2,
    "Reports": [
      {
        "Employee_FN": "Pam",
        "Employee_LN": "Beasley",
        "Level": 2
      }
    ]
  },
  {
    "Employee_FN": "Stanley",
    "Employee_LN": "Hudson",
    "Level": 2,
    "Reports": [
      {
        "Employee_FN": "Stanley",
        "Employee_LN": "Hudson",
        "Level": 2
      }
    ]
  },
  {
    "Employee_FN": "Ryan",
    "Employee_LN": "Howard",
    "Level": 3,
    "Reports": [
      {
        "Employee_FN": "Ryan",
        "Employee_LN": "Howard",
        "Level": 3
      }
    ]
  },
  {
    "Employee_FN": "Kelly",
    "Employee_LN": "Kapoor",
    "Level": 4,
    "Reports": [
      {
        "Employee_FN": "Kelly",
        "Employee_LN": "Kapoor",
        "Level": 4
      }
    ]
  },
  {
    "Employee_FN": "Meredith",
    "Employee_LN": "Palmer",
    "Level": 4,
    "Reports": [
      {
        "Employee_FN": "Meredith",
        "Employee_LN": "Palmer",
        "Level": 4
      }
    ]
  }
]

Problems:

  1. Each person only has themselves as children
  2. The whole JSON structure appears to be in a dict - I believe it has to be enclosed by {} to be readable

I have tried switched around the groupby and lambda elements in various configurations to reach the desired output as well. Any and all insight would be greatly appreciated! Thank you!

Update:

I changed my code block to this:

j = (df.groupby(['Level','Supervisor_FN','Supervisor_LN'], as_index=False)
             .apply(lambda x: x[['Level','Employee_FN','Employee_LN']].to_dict('r'))
             .reset_index()
             .rename(columns={0:'Reports'})
             .rename(columns={'Supervisor_FN':'Employee_FN'})
             .rename(columns={'Supervisor_LN':'Employee_LN'})
             .to_json(orient='records'))

print(json.dumps(json.loads(j), indent=2, sort_keys=True))

The new output is this:

[
  {
    "Employee_FN": "Michael",
    "Employee_LN": "Scott",
    "Level": 1,
    "Reports": [
      {
        "Employee_FN": "Jim",
        "Employee_LN": "Halpert",
        "Level": 1
      },
      {
        "Employee_FN": "Dwight",
        "Employee_LN": "Schrute",
        "Level": 1
      }
    ]
  },
  {
    "Employee_FN": "Jim",
    "Employee_LN": "Halpert",
    "Level": 2,
    "Reports": [
      {
        "Employee_FN": "Stanley",
        "Employee_LN": "Hudson",
        "Level": 2
      },
      {
        "Employee_FN": "Pam",
        "Employee_LN": "Beasley",
        "Level": 2
      }
    ]
  },
  {
    "Employee_FN": "Pam",
    "Employee_LN": "Beasley",
    "Level": 3,
    "Reports": [
      {
        "Employee_FN": "Ryan",
        "Employee_LN": "Howard",
        "Level": 3
      }
    ]
  },
  {
    "Employee_FN": "Ryan",
    "Employee_LN": "Howard",
    "Level": 4,
    "Reports": [
      {
        "Employee_FN": "Kelly",
        "Employee_LN": "Kapoor",
        "Level": 4
      },
      {
        "Employee_FN": "Meredith",
        "Employee_LN": "Palmer",
        "Level": 4
      }
    ]
  }
]

Problems:

  1. The Level matches the underlying employee for both the underlying employee and the supervisor
  2. The nesting only goes one level deep

Upvotes: 5

Views: 3249

Answers (1)

Igor Raush
Igor Raush

Reputation: 15240

This type of problem isn't particularly well-suited for Pandas; the data structure you're going after is recursive, not tabular.

Here is one possible solution.

from operator import itemgetter

employee_key = itemgetter('Employee_FN', 'Employee_LN')
supervisor_key = itemgetter('Supervisor_FN', 'Supervisor_LN')

def subset(dict_, keys):
    return {k: dict_[k] for k in keys}

# store employee references
cache = {}

# iterate over employees sorted by level, so supervisors are cached before reports
for row in df.sort_values('Level').to_dict('records'):

    # look up employee/supervisor references
    employee = cache.setdefault(employee_key(row), subset(row, keys=('Employee_FN', 'Employee_LN', 'Level')))
    supervisor = cache.get(supervisor_key(row), {})

    # link reports to employee
    supervisor.setdefault('Reports', []).append(employee)

# grab only top-level employees
[rec for key, rec in cache.iteritems() if rec['Level'] == 0]

[{'Employee_FN': 'Michael',
  'Employee_LN': 'Scott',
  'Level': 0,
  'Reports': [{'Employee_FN': 'Jim',
    'Employee_LN': 'Halpert',
    'Level': 1,
    'Reports': [{'Employee_FN': 'Stanley',
      'Employee_LN': 'Hudson',
      'Level': 2},
     {'Employee_FN': 'Pam',
      'Employee_LN': 'Beasley',
      'Level': 2,
      'Reports': [{'Employee_FN': 'Ryan',
        'Employee_LN': 'Howard',
        'Level': 3,
        'Reports': [{'Employee_FN': 'Kelly',
          'Employee_LN': 'Kapoor',
          'Level': 4},
         {'Employee_FN': 'Meredith',
          'Employee_LN': 'Palmer',
          'Level': 4}]}]}]},
   {'Employee_FN': 'Dwight', 'Employee_LN': 'Schrute', 'Level': 1}]}]

Upvotes: 4

Related Questions