CODEWITHSUNDEEP
pythonpython-3.xamazon-web-servicesamazon-dynamodb
Robbert
Robbert

Reputation: 6582

Malformed Node or String with Dynamodb Returned item

I have been experimenting with DynamoDB in AWS and have decided it's the wrong tool for me. While working in AWS, I uploaded a large number of records to the system that I am now trying to retrieve using Python. I admit, I'm completely new to Python and what I've cobbled together is from examples found around the internet. I'm working with Office 365 Audit logs that have many fields and when I retrieve them from Dynamo, I found that decimal fields are coming back as decimal('1').

{
  'ItemType': 'File',
  'SourceFileName': 'xxx',
  'UserKey': 'xxx',
  'RecordType': Decimal('6'),
  'UserId': 'xxx',
  'ClientIP': 'xxx',
  'CorrelationId': 'xxx',
  'ObjectId': 'xxx',
  'Version': Decimal('1'),
  'Site': 'xxx',
  'WebId': 'xxx',
  'SourceFileExtension': 'js',
  'SiteUrl': 'xxx',
  'Workload': 'SharePoint',
  'SourceRelativeUrl': 'xxx',
  'EventSource': 'SharePoint',
  'ListId': 'xxx',
  'OrganizationId': 'xxx',
  'Operation': 'FileAccessed',
  'UserAgent': 'xxx',
  'ListItemUniqueId': 'xxx',
  'CreationTime': '2019-07-26T21:54:45',
  'UserType': Decimal('0'),
  'Id': 'xxx',
  'CustomUniqueId': False
}

In troubleshooting, I found this handy function that converts Decimal('0') to an actual decimal

class DecimalEncoder(json.JSONEncoder):
    def default(self, o):

        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)

and this function is called

response = table.scan(Limit=10)
for i in response['Items']:
    d = ast.literal_eval((json.dumps(i, cls=DecimalEncoder)))

For the most part this works fine, unless i has a boolean field (like the example above). Then I get this error

malformed node or string: <_ast.Name object at 0x04B3F778>

As far as I understand with Python, the boolean values of True and False are capitalized correctly.. I considered adding the the DecimalEncoder default method but this wouldn't do much since it's literally returning the same value

elif isinstance(o,bool):
    return True if o else False

I did find by removing the key, I got no error

if 'CustomUniqueId' in i:
    del i['CustomUniqueId']

However, don't want to lose this field. I'm at a loss as to how to resolve this.

Upvotes: 1

Views: 220

Answers (1)

JD D
JD D

Reputation: 8097

The problem is that you are converting your Python object to a JSON string (i.e. the json.dumps command) and then trying to interpret that string as Python dictionary (the ast.literal_eval takes a string and tries to run it as Python code).

A python dictionary looks very similar to a JSON object but they are not 100% compatible. In JSON, boolean is presented as true and false and in Python it's represented as True and False.

I way to solve this would be to use json.loads which reads a JSON string and converts it into a Python equivalent:

d = json.loads((json.dumps(i, cls=DecimalEncoder)))

That being said, this seems like a lot of hoops/processing to convert Decimal class to native floats and would recommend just using Decimal directly or converting to a float (i.e. float(d['RecordType'])) if you need to pass them to something that expects a float.

Upvotes: 1

Related Questions