Pymongo: Cannot encode object of type decimal.Decimal?

Question

After trying to insert_one into a collection. I receive this error:

bson.errors.InvalidDocument: cannot encode object: Decimal('0.16020'), of type: <class 'decimal.Decimal'>

The code runs fine when the JSON does not include the decimal.Decimal object. If there is a solution can you kindly consider coding it in a recursive manner to made the whole of the python dictionary json_dic compatible to be inserted into MongoDB (as there is more than once instance of the class decimal.Decimal in the json.dic entries).

EDIT 1: Here is the JSON I am dealing with

import simplejson as json
from pymongo import MongoClient

json_string = '{"A" : {"B" : [{"C" : {"Horz" : 0.181665435,"Vert" : 0.178799435}}]}}'

json_dict = json.loads(json_string)
this_collection.insert_one(json_dict)

This produces bson.errors.InvalidDocument: cannot encode object: Decimal('0.181665435'), of type: <class 'decimal.Decimal'>

EDIT 2: Unfortunately my example above simplified my existing JSON too much and the answer provided by @Belly Buster (despite working fine with the Json above) thows an error:

AttributeError: 'decimal.Decimal' object has no attribute 'items'

with my actual JSON, so I am providing the full JSON here hopefully to find out what is wrong (also as a screen-shot):

json_string = 
'
{
  "Setting" : {
    "GridOptions" : {
      "Student" : "HighSchool",
      "Lesson" : 1,
      "Attended" : true
    },
    "Grades" : [
      80,
      50.75
    ],
    "Count" : 2,
    "Check" : "Coursework",
    "Passed" : true
  },
  "Slides" : [
    {
      "Type" : "ABC",
      "Duration" : 1.5
    },
    {
      "Type" : "DEF",
      "Duration" : 0.5
    }
  ],
  "Work" : {
    "Class" : [
      {
        "Time" : 123456789,
        "Marks" : {
          "A" : 50,
          "B" : 100
        }
      }
    ],
    "CourseWorkDetail" : [
      {
        "Test" : {
          "Mark" : 0.987654321
        },
        "ReadingDate" : "Feb162006",
        "Reading" : 300.001,
        "Values" : [
          [
            0.98765
          ],
          [
            -0.98765
          ]
        ]
      },
      {
        "Test" : {
          "Mark" : 0.123456789
        },
        "ReadingDate" : "Jan052010",
        "Reading" : 200.005,
        "Values" : [
          [
            0.12345
          ],
          [
            -0.12345
          ]
        ]
      }
    ]
  },
  "listing" : 5
}
'

Edit 3: Complimentary to the answer below, you can iterate recursively in a dictionary like this and use the function from the answer

def iterdict(dict_items, debug_out):
    for k, v in dict_items.items():
        if isinstance(v):
            iterdict(v)
        else:
            dict_items[k] = convert_decimal(v)
    return dict_items

Answer 1

The accepted answer here is partially correct but fails our tests written with pytest with the example data:

example = {
    "string_field": "some",
    "string_array_list": ["samatta"],
    "decimal_containing_list": ["a", Decimal(8)],
    "empty_list": [],
    "empty_dict": {},
    "direct_decimal": Decimal(0),
    "nested_decimal_dict": {"very_important-field": Decimal(1)},
    "number_field": 1,
    "nested_number_field": {"very_important-field": 1},
    "number_list": [1, 2],
    "decimal_containing_field_list": [{"very_important": Decimal(1)}],
}

where nested arrays are the case. I improved it a little bit, and current one passess the tests:

from typing import Union
    
def convert_decimal(item: Union[dict, list]) -> Union[dict, list]:
    if item is None:
        return None

    elif isinstance(item, list):
        for index, item in enumerate(item):
            if isinstance(item, Decimal):
                item[index] = Decimal128(str(item))
            if isinstance(item, dict):
                convert_decimal(item)
        return item

    elif isinstance(item, dict):
        for k, v in list(item.items()):
            if isinstance(v, dict):
                convert_decimal(v)
            elif isinstance(v, list):
                convert_decimal(v)
            elif isinstance(v, Decimal):
                item[k] = Decimal128(str(v))

        return item

when you print it out it gives,

{
    "string_field": "some",
    "string_array_list": ["samatta"],
    "decimal_containing_list": ["a", Decimal128("8")],
    "empty_list": [],
    "empty_dict": {},
    "direct_decimal": Decimal128("0"),
    "nested_decimal_dict": {"very_important-field": Decimal128("1")},
    "number_field": 1,
    "nested_number_field": {"very_important-field": 1},
    "number_list": [1, 2],
    "decimal_containing_field_list": [{"very_important": Decimal128("1")}],
}

Answer 2

I would recommend just adding a codec to automatically convert the data types at insertion. If you recursively change the data types to use the Decimal128 object you might break compatibility with existing code.

You can follow the tutorial to create a simple decimal.Decimal codec in the pymongo docs here

Answer 3

from bson.decimal128 import Decimal128, create_decimal128_context
from decimal import localcontext

decimal128_ctx = create_decimal128_context()
with localcontext(decimal128_ctx) as ctx:
    horiz_val = Decimal128(ctx.create_decimal("0.181665435"))
    vert_val = Decimal128(ctx.create_decimal("0.178799435"))

doc = { 'A': { 'B': [ { 'C': { 'Horiz': horiz_val, 'Vert': vert_val } } ] } }
result = collection.insert_one(doc)
# result.inserted_id

pprint.pprint(list(collection.find()))

[ {'A': {'B': [{'C': {'Horiz': Decimal128('0.181665435'),
                      'Vert': Decimal128('0.178799435')}}]},
  '_id': ObjectId('5ea79adb915cbf3c46f5d4ae')} ]

NOTES:

• PyMongo's decimal128 - Support for BSON Decimal128
• Python's decimal module provides support for decimal floating point arithmetic.

From the PyMongo's decimal128 documentation:

To ensure the result of a calculation can always be stored as BSON Decimal128 use the context returned by create_decimal128_context() (NOTE: as shown in the example code above).

Answer 4

EDIT:

The convert_decimal() function will perform the conversion of Decimal to Decimal128 within a complex dict structure:

import simplejson as json
from pymongo import MongoClient
from decimal import Decimal
from bson.decimal128 import Decimal128

def convert_decimal(dict_item):
    # This function iterates a dictionary looking for types of Decimal and converts them to Decimal128
    # Embedded dictionaries and lists are called recursively.
    if dict_item is None: return None

    for k, v in list(dict_item.items()):
        if isinstance(v, dict):
            convert_decimal(v)
        elif isinstance(v, list):
            for l in v:
                convert_decimal(l)
        elif isinstance(v, Decimal):
            dict_item[k] = Decimal128(str(v))

    return dict_item

db = MongoClient()['mydatabase']
json_string = '{"A" : {"B" : [{"C" : {"Horz" : 0.181665435,"Vert" : 0.178799435}}]}}'
json_dict = json.loads(json_string, use_decimal=True)
db.this_collection.insert_one(convert_decimal(json_dict))
print(db.this_collection.find_one())

gives:

{'_id': ObjectId('5ea743aa297c9ccd52d33e05'), 'A': {'B': [{'C': {'Horz': Decimal128('0.181665435'), 'Vert': Decimal128('0.178799435')}}]}}

ORIGINAL:

To convert a decimal to a Decimal128 that MongoDB will be happy with, convert it to a string and then to a Decimal128. This snippet may help:

from pymongo import MongoClient
from decimal import Decimal
from bson.decimal128 import Decimal128

db = MongoClient()['mydatabase']
your_number = Decimal('234.56')
your_number_128 = Decimal128(str(your_number))
db.mycollection.insert_one({'Number': your_number_128})
print(db.mycollection.find_one())

gives:

{'_id': ObjectId('5ea6ec9b52619c7b39b851cb'), 'Number': Decimal128('234.56')}

Answer 5

Pymongo doesn't recognize Decimal - that's why you are getting the error.

The correct pymongo insert is coll.insert_one({"number1": Decimal128('8.916')}).

You'll also need the import - from bson import Decimal128

Now, if you want to process your JSON file without changing Decimal to Decimal128`, you could modify the import statement.

from bson import Decimal128 as Decimal

coll.insert_one({"number1": Decimal('8.916')})