How is the best approach to validate data in python?

Question

I am a newbie in Python, and I´m trying to find the best way to validate data.

I have an object of type "well" that has as attributes other objects. Data can come from a XML file or via code. An example can be seen bellow.

class Well:
    def __init__ (self, name, group):
        self.__name   = name     # Required
        self.__group  = group    # Required
        self.__operate_list = [] # Optional
        self.__monitor_list = [] # Optional
        self.__geometry = None   # Optional
        self.__perf = None       # Optional
        ...

class Operate:

    # *OPERATE (*MAX|*MIN) type value (action)
    # *OPERATE (*PENALTY) type (mode) value (action)
    # *OPERATE (*WCUTBACK) type mode v1 (v2 (v3)) (action)

    def __init__ (self, att:str, type_:str, value: [], mode=None, action=None):
        self.__att = att
        self.__type_ = type_
        self.__mode = mode
        self.__value_list = value
        self.__action = action        

To validate an "operate" for example, i need to check a lot of restrictions and valid values for each attribute. For instance, I have a list of valid "type_" strings and I should assert that type_ is in this list.

1) The best approach to do this is in the constructor? Should I create a method to do this validation? Or should I create a new class only to validade data?

2) Where should I create these lists of valid values? In the constructor? As global variables?

Answer 1

You can do this with pyfields, which can be seen as an industrialized version of wwii's answer.

You define a field using field(). Fields can be mandatory or optional, and in case the default value is mutable you can specify a default factory (I saw that you use empty lists as default values in your example). Helper function copy_value helps you cover the most common case:

from pyfields import field, copy_value, init_fields
from valid8.validation_lib import is_in

class Well(object):
    name = field()                                        # Required
    group = field()                                       # Required
    operate_list = field(default_factory=copy_value([]))  # Optional
    monitor_list = field(default_factory=copy_value([]))  # Optional
    geometry = field(default=None)                        # Optional
    perf = field(default=None)                            # Optional

Then you can optionally add validation on top of your fields. You can both validate type (with check_type=True) and value (with validators). Validators can rely on existing callables such as is_in as shown below, but generally can leverage any validation callable. Finally the constructor can be generated for you, as shown below:

valid_types = ('type_A', 'type_B')

class Operate(object):
    att = field()                                  # Required
    type_: str = field(check_type=True, validators=is_in(valid_types))   # Required
    value = field(default_factory=copy_value([]))  # Optional
    mode = field(default=None)                     # Optional
    action = field(default=None)                   # Optional

    @init_fields
    def __init__(self):
        pass

o = Operate(att="foo", type_='type_A')  # ok
o.type_ = 1  # <-- raises TypeError: Invalid value type provided

bad_o = Operate(att="foo", type_='type_WRONG')  # <-- raises ValidationError: NotInAllowedValues: x in ('type_A', 'type_B') does not hold for x=type_WRONG

See pyfields documentation for details. I'm the author by the way ;)

Answer 2

You could do it with descriptors. The only advantage I can contrive is that it puts the validation in another class - making the class that uses it less verbose. Unfortunately you would have to make one for each attribute with unique validations, Unless you want to include options for membership tests and/or instance-of tests which shouldn't make it too complicated.

from weakref import WeakKeyDictionary

class RestrictedAttribute:
    """A descriptor that restricts values"""
    def __init__(self, restrictions):
        self.restrictions = restrictions
        self.data = WeakKeyDictionary()

    def __get__(self, instance, owner):
        return self.data.get(instance, None)

    def __set__(self, instance, value):
        if value not in self.restrictions:
            raise ValueError(f'{value} is not allowed')
        self.data[instance] = value

When used the descriptor instance must be assigned as a class attribute

class Operate:

    __type_ = RestrictedAttribute(('red','blue'))

    def __init__ (self, att:str, type_:str, value: [], mode=None, action=None):
        self.__att = att
        self.__type_ = type_
        self.__mode = mode
        self.__value_list = value
        self.__action = action        

In use:

In [15]: o = Operate('f',type_='blue',value=[1,2])

In [16]: o._Operate__type_
Out[16]: 'blue'

In [17]: o._Operate__type_ = 'green'
Traceback (most recent call last):

  File "<ipython-input-17-b412cfaa0cb0>", line 1, in <module>
    o._Operate__type_ = 'green'

  File "P:/pyProjects3/tmp1.py", line 28, in __set__
    raise ValueError(msg)

ValueError: green is not allowed

Answer 3

You can use getter and setter methods via the use of the property function:

class Operate:
    def __init__(self, type):
        self.type = type

    @property
    def type(self):
        return self._type

    @type.setter
    def type(self, value):
        assert value in ('abc', 'xyz')
        self._type = value

so that:

o = Operate(type='123')

would result in:

Traceback (most recent call last):
  File "test.py", line 18, in <module>
    o = Operate(type='123')
  File "test.py", line 8, in __init__
    self.type = type
  File "test.py", line 15, in type
    assert value in ('abc', 'xyz')
AssertionError

Answer 4

assert isinstance(obj) 

Is how you test the type of an object.

if item in container: ...

Is how you would test if an object is in a container.

Whether you do this in the init method or in another method is up to you, it depends which looks cleaner to you, or if you would need to reuse the functionality.

The list of valid values could be passed into the init method or hardcoded into the init method. It can also be a global property of the class.