nesting with namedtuple

Question

I'm having trouble getting my data in the form that I'd like in python.

Basically I have a program that reads in binary data and provides functions for plotting and analysis on said data.

My data has main headings and then subheadings that could be any number of varied datatypes.

I'd like to be able to access my data like for example:

>>> a = myDatafile.readit()
>>> a.elements.hydrogen.distributionfunction
(a big array)
>>> a.elements.hydrogen.mass
1
>>> a.elements.carbon.mass
12

but I don't know the names of the atoms until runtime.

I've tried using namedtuple, for example after I've read in all the atom names:

self.elements = namedtuple('elements',elementlist)

Where elementlist is a list of strings for example ('hydrogen','carbon'). But the problem is I can't nest these using for example:

for i in range(0,self.nelements):
    self.elements[i] = namedtuple('details',['ux','uy','uz','mass','distributionfunction'])

and then be able to access the values through for example

self.elements.electron.distributionfunction.

Maybe I'm doing this completely wrong. I'm fairly inexperienced with python. I know this would be easy to do if I wasn't bothered about naming the variables dynamically.

I hope I've made myself clear with what I'm trying to achieve!

Answer 1

I had the same issue with nested json but needed to be able to serialise the output with pickle which doesn't like you creating objects on the fly.

I've taken @bren's answer and enhanced it so that the resulting structure will be serialisable with pickle. You have to save references to each of the structures you create to globals so that pickle can keep tabs on them.

##############################################
class Json2Struct:
    '''
    Convert mappings to nested namedtuples

    Usage:
        jStruct = Json2Struct('JS').json2Struct(json)
    '''
##############################################

    def __init__(self, name):
        self.namePrefix = name
        self.nameSuffix = 0


    def json2Struct(self, jsonObj):  # thank you https://gist.github.com/hangtwenty/5960435
        """ 
        Convert mappings to namedtuples recursively. 
        """
        if isinstance(jsonObj, Mapping):
            for key, value in list(jsonObj.items()):
                jsonObj[key] = self.json2Struct(value)
            return self.namedtuple_wrapper(**jsonObj)
        elif isinstance(jsonObj, list):
            return [self.json2Struct(item) for item in jsonObj]
        return jsonObj


    def namedtuple_wrapper(self, **kwargs):
        self.nameSuffix += 1
        name = self.namePrefix + str(self.nameSuffix)

        Jstruct = namedtuple(name, kwargs)
        globals()[name] = Jstruct

        return Jstruct(**kwargs)

The example below should work as follows and also be serialisable:

stuff = {'data': {'elements': {'hydrogen': {'distributionfunction': 'foo'}, 
  'nitrogen': {'xyzfunction': 'bar', 
    'distributionfunction': 'baz'}
  },
  'compound': {'water': {'distributionfunction': 'lorem'}, 
    'hcl': {'xyzfunction': 'ipsum'}}}
 }

example = Json2Struct('JS').json2Struct(stuff)

example.data.elements.hydrogen.distributionfunction  # 'foo'

Answer 2

Here's a method for recursively creating namedtuples from nested data.

from collections import Mapping, namedtuple


def namedtuplify(mapping, name='NT'):  # thank you https://gist.github.com/hangtwenty/5960435
    """ Convert mappings to namedtuples recursively. """
    if isinstance(mapping, Mapping):
        for key, value in list(mapping.items()):
            mapping[key] = namedtuplify(value)
        return namedtuple_wrapper(name, **mapping)
    elif isinstance(mapping, list):
        return [namedtuplify(item) for item in mapping]
    return mapping

def namedtuple_wrapper(name, **kwargs):
    wrap = namedtuple(name, kwargs)
    return wrap(**kwargs)


stuff = {'data': {'elements': {'hydrogen': {'distributionfunction': 'foo'}, 
  'nitrogen': {'xyzfunction': 'bar', 
    'distributionfunction': 'baz'}
  },
  'compound': {'water': {'distributionfunction': 'lorem'}, 
    'hcl': {'xyzfunction': 'ipsum'}}}
 }

example = namedtuplify(stuff)

example.data.elements.hydrogen.distributionfunction  # 'foo'

Answer 3

Without knowing your data, we can only give a generic solution.

Considering the first two lines contains the headings and Sub-Heading reading it somehow you determined the hierarchy. All you have to do is to create an hierarchical dictionary.

For example, extending your example

data.elements.hydrogen.distributionfunction
data.elements.nitrogen.xyzfunction
data.elements.nitrogen.distributionfunction
data.compound.water.distributionfunction
data.compound.hcl.xyzfunction

So we have to create a dictionary as such

{'data':{'elements':{'hydrogen':{'distributionfunction':<something>}
                     'nitrogen':{'xyzfunction':<something>,
                           'distributionfunction':<something>}
                }
       compound:{'water':{'distributionfunction':<something>}
                 'hcl':{'xyzfunction':<something>}
                }
       }
 }

how you will populate the dictionary depends on the data which is difficult to say now. But the keys to the dictionary you should populate from the headers, and somehow you have to map the data to the respective value in the empty slot's of the dictionary.

Once the map is populated, you can access it as

 yourDict['data']['compound']['hcl']['xyzfunction']

Answer 4

If your element name are dynamic and obtained from the data at runtime, you can assign them to a dict and access like this

elements['hydrogen'].mass

but if you want dotted notation you can create attributes at run time e.g.

from collections import namedtuple

class Elements(object):
    def add_element(self, elementname, element):
        setattr(self, elementname, element)

Element = namedtuple('Element', ['ux','uy','uz','mass','distributionfunction'])

elements = Elements()
for data in [('hydrogen',1,1,1,1,1), ('helium',2,2,2,2,2), ('carbon',3,3,3,3,3)]:
    elementname = data[0]
    element = Element._make(data[1:])
    elements.add_element(elementname, element)

print elements.hydrogen.mass
print elements.carbon.distributionfunction

Here I am assuming the data you have, but with data in any other format you can do similar tricks