How to override a python3 __sub__ function so the data type isn't changed

Question

I'm trying to subclass the datetime class so my main code looks cleaner. However, doing any arithmetic on my subclass changes the data type back to datetime.datetime.

I took my original code and cut it down to a minimal example.

from datetime import datetime, timedelta

class worldtime(datetime):
   UTC = True
   tz_offset = timedelta(hours = 4)

   def __new__(cls, *args, **kwargs):
      #kwargs['tzinfo'] = dateutil.tz.tzutc()
      return super().__new__(cls, *args, **kwargs)

   def is_UTC(self):
      return self.UTC

   def to_local(self):
      print(f"type(self): {type(self)}")
      if self.UTC is True:
         self = self - self.tz_offset
         print(f"type(self): {type(self)}")
         self.UTC = False
         return self

dt = worldtime(2019, 8, 26, 12, 0, 0)
print (f"dt = {dt}   is_UTC(): {dt.is_UTC()}")
print (f"type(dt): {type(dt)}")
print (f"dir(dt): {dir(dt)}")
dt = dt.to_local()

The moment I subtract the tz_offset timedelta, the type of the object changes back to datetime.datetime:

dt = 2019-08-26 12:00:00   is_UTC(): True
type(dt): <class '__main__.worldtime'>
dir(dt): ['UTC', '__add__', '__class__', '__delattr__', '__dict__', 
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', 
'__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', 
'__le__', '__lt__', '__module__', '__ne__', '__new__', '__radd__', 
'__reduce__', '__reduce_ex__', '__repr__', '__rsub__', '__setattr__', 
'__sizeof__', '__str__', '__sub__', '__subclasshook__', '__weakref__', 
'astimezone', 'combine', 'ctime', 'date', 'day', 'dst', 'fold', 
'fromisoformat', 'fromordinal', 'fromtimestamp', 'hour', 'is_UTC', 
'isocalendar', 'isoformat', 'isoweekday', 'max', 'microsecond', 'min', 
'minute', 'month', 'now', 'replace', 'resolution', 'second', 'strftime', 
'strptime', 'time', 'timestamp', 'timetuple', 'timetz', 'to_local', 
'today', 'toordinal', 'tz_offset', 'tzinfo', 'tzname', 'utcfromtimestamp', 
'utcnow', 'utcoffset', 'utctimetuple', 'weekday', 'year']
type(self): <class '__main__.worldtime'>
type(self): <class 'datetime.datetime'>
Traceback (most recent call last):
  File "testwt.py", line 33, in <module>
    dt.to_local()
  File "testwt.py", line 27, in to_local
    self.UTC = False
AttributeError: 'datetime.datetime' object has no attribute 'UTC'

I can admit to being new to subclasses in python. While I have seen other posts that seem to talk about this problem, there are no examples to follow. The best I have seen is that I have to override the __sub__ operator, but I'm uncertain how to do that and ensure the returned object is the correct type. Again, there aren't any clear code examples to work with...

Update: corrected a minor error in the example code because worldtime.to_local() needs to return the new instance back to the main code.

Answer 1

The important line is this one, within the to_local() method:

self = self - self.tz_offset

Instead of changing self (this worldtime object) so that it now represents the local time, you are actually setting it to be a completely new object, specifically, the result of self - self.tz_offset.

So why isn't the result of that a worldtime object?

Note that the types of object in this calculation are worldtime - timedelta. At the moment you haven't done anything to specify how to perform subtraction on your worldtime class, so worldtime automatically inherits its subtraction behavior from its parent class (datetime). But this means it gets treated like an ordinary datetime object (after all, it is really a datetime, just with a couple of extra attributes and methods).

So Python carries out a datetime - timedelta calculation, and the result is a datetime object, which it then assigns to self. Which is why your worldtime object seems to be 'changing' into a datetime.

How can we make it work?

There are two options:

1) Update our object instead of creating a new one

If we know our offset will always just be some hours, we could do something like:

def to_local(self):
    if self.UTC is True:
        self.hour = self.hour + self.tz_offset.hours
        self.UTC = False

BUT this WON'T work because (contrary to what I initially expected!):

1. tz_offset doesn't have a hours attribute (when you create a timedelta it stores the time as days, seconds and microseconds)
2. datetime objects don't let you set the hour directly like this

We could try changing the _hour attribute (which is how datetime stores its time internally), but changing 'private' attributes like this is generally a bad idea. Plus, we still have to turn tz_offset back into hours to do that calculation, and what happens if we later want to have an offset with hours and minutes? And we need to make sure our offset isn't taking us across a date boundary... (and probably other issues we haven't thought of!)

Better to let datetime do what it's good at, so:

2a) Let datetime handle the subtraction, but turn the result back into a worldtime

def to_local(self):
    if self.UTC is True:
        new_time = self - self.tz_offset
        self = worldtime(
            new_time.year,
            new_time.month,
            new_time.day,
            new_time.hour,
            new_time.minute,
            new_time.second,
        )
        self.UTC = False

Alternatively, as you mentioned, you could define the __sub__() special method to define what the - operator does on our worldtime objects.

2b) Override the - operator with __sub__()

Let's leave to_local() as

def to_local(self):
    if self.UTC is True:
        self = self - self.tz_offset
        self.UTC = False

But change how that - behaves. Here, we're basically moving what we did in 2a into a separate method called __sub__() (as in subtraction). This means that when Python hits the -, it passes the left and right operands into the __sub__() special method as self and other (respectively), and then returns the result of the method.

    def __sub__(self, other):
    new_time = self - other
    return worldtime(
        new_time.year,
        new_time.month,
        new_time.day,
        new_time.hour,
        new_time.minute,
        new_time.second,
    )

BUT when we run this, we get an error like this:

RecursionError: maximum recursion depth exceeded

What happened?

When Python hits the self - self.tz_offset in to_local(), it calls __sub__(self, self.tz_offset). So far, so good. But when it gets to self - other within __sub__(), we're still doing subtraction on a worldtime object, so Python dutifully calls __sub__(self, other) again...and again, and again, and gets stuck in an infinite loop!

We don't want that. Instead, once we're in __sub__() we just want to do normal datetime subtraction. So it should look like this:

    def __sub__(self, other):
    new_time = super().__sub__(other)
    return worldtime(
        new_time.year,
        new_time.month,
        new_time.day,
        new_time.hour,
        new_time.minute,
        new_time.second,
    )

Here, super().__sub__(other) means we're using the __sub__() method on the parent class instead. Here, that's datetime, so we get a datetime object back, and can create a new worldtime object from that.

---

The whole thing (with your print statements) now looks like this:

from datetime import datetime, timedelta


class worldtime(datetime):
    UTC = True
    tz_offset = timedelta(hours = -4)

    def __new__(cls, *args, **kwargs):
        #kwargs['tzinfo'] = dateutil.tz.tzutc()
        return super().__new__(cls, *args, **kwargs)

    def is_UTC(self):
        return self.UTC

    def to_local(self):
        print(f"type(self): {type(self)}")
        if self.UTC is True:
            self = self - self.tz_offset
            print(f"type(self): {type(self)}")
            print(self)
            self.UTC = False

    def __sub__(self, other):
        new_time = super().__sub__(other)
        return worldtime(
            new_time.year,
            new_time.month,
            new_time.day,
            new_time.hour,
            new_time.minute,
            new_time.second,
        )


dt = worldtime(2019, 8, 26, 12, 0, 0)
print (f"dt = {dt}   is_UTC(): {dt.is_UTC()}")
print (f"type(dt): {type(dt)}")
print (f"dir(dt): {dir(dt)}")
dt.to_local()

(I changed to 4-space tabs, as is standard in Python)

---

BUT...Is this the best way to do this?

Hopefully that's answered your questions about subclassing in Python.

But reflecting on the problem, I'm not sure if this is the best way to go. Subclassing built-ins can be complicated and easy to get wrong, datetimes themselves are already complicated and easy to get wrong. Subclassing datetime makes less sense as it's not straightforward to change them after creation, and creating a new object and setting it to self doesn't feel very neat.

I wonder if it would be better to use composition instead of inheritance. So worldtime would store a datetime object internally, and you can operate on that, and use the timezone support in the datetime module to manage your timezone conversion, and maybe just do it on-the-fly for returning the local time.

Something like:

from datetime import datetime, timedelta, timezone


class WorldTime:
    OFFSET = timedelta(hours=-4)

    # assumes input time is in UTC, not local time
    def __init__(self, year, month=None, day=None, hour=0, minute=0, second=0,
                 microsecond=0, tzinfo=timezone.utc, *, fold=0):
        self.dt_in_utc = datetime(year, month, day, hour, minute, second,
                                  microsecond, tzinfo, fold=fold)

    # convert to our timezone, and then make naive ("local time")
    def to_local(self):
        return self.dt_in_utc.astimezone(timezone(self.OFFSET)).replace(tzinfo=None)


dt = WorldTime(2019, 8, 26, 12, 0, 0)
print(dt.to_local())

# Gives:
# 2019-08-26 08:00:00

I've made it so that to_local() returns a datetime object, which you can then print out, or do whatever you want to with afterwards.

---

• This question covers something similar: Convert a python UTC datetime to a local datetime using only python standard library?
• This reference is quite good: https://howchoo.com/g/ywi5m2vkodk/working-with-datetime-objects-and-timezones-in-python

---

Edit

I had another experiment with inheriting from datetime, and I think the following should work:

from datetime import datetime, timedelta, timezone


class WorldTime(datetime):
    OFFSET = timedelta(hours=-4)

    def __new__(cls, *args, tzinfo=timezone.utc, **kwargs):
        return super().__new__(cls, *args, tzinfo=tzinfo, **kwargs)

    def __add__(self, other):
        result = super().__add__(other)
        return WorldTime(*result.timetuple()[:6], tzinfo=result.tzinfo,
                          fold=result.fold)

    def __sub__(self, other):
        "Subtract two datetimes, or a datetime and a timedelta."
        if not isinstance(other, datetime):
            if isinstance(other, timedelta):
                return self + -other
            return NotImplemented
        return super().__sub__(other)

    def to_local(self):
        return self.astimezone(timezone(self.OFFSET)).replace(tzinfo=None)

dt = WorldTime(2019, 8, 26, 12, 0, 0)
print(dt)
print(dt.to_local())  # local time
print(dt + timedelta(days=20, hours=7))  # 20 days, 7 hours in the future
print(dt - timedelta(days=40, hours=16))  # 40 days, 16 hours in the past
print(dt - WorldTime(2018, 12, 25, 15, 0, 0))  # time since 3pm last Christmas Day


# Output:
# 2019-08-26 12:00:00+00:00  # WorldTime
# 2019-08-26 08:00:00  # datetime
# 2019-09-15 19:00:00+00:00  # WorldTime
# 2019-07-16 20:00:00+00:00  # WorldTime
# 243 days, 21:00:00  # timedelta

So it looks like addition and subtraction of timedeltas returns a WorldTime object, and we can find the difference between two WorldTime objects as a timedelta.

This isn't rigorously tested, however, so proceed with caution!

Answer 2

The result of subtracting (sub-) classes of datetime will always return a datetime instance. It becomes obvious when taking a look at the implementation for __add__(self, other) in the datetime module (as __sub__(self, other) essentially just forwards the computation to the addition function when subtracting a timedelta instance from a datetime instance):

class datetime(date):

    ...

    def __sub__(self, other):
        "Subtract two datetimes, or a datetime and a timedelta."
        if not isinstance(other, datetime):
            if isinstance(other, timedelta):  # This is True in our case
                return self + -other  # This is calling the __add__ function
            return NotImplemented

        # The remainder of the __sub__ function is omitted as we are 
        # focussing on the case in which a timedelta instance is subtracted 
        # from a datetime instance.

    def __add__(self, other):
        "Add a datetime and a timedelta."
        if not isinstance(other, timedelta):
            return NotImplemented
        delta = timedelta(self.toordinal(),
                          hours=self._hour,
                          minutes=self._minute,
                          seconds=self._second,
                          microseconds=self._microsecond)
        delta += other
        hour, rem = divmod(delta.seconds, 3600)
        minute, second = divmod(rem, 60)
        if 0 < delta.days <= _MAXORDINAL:
            return type(self).combine(date.fromordinal(delta.days),
                                      time(hour, minute, second,
                                           delta.microseconds,
                                           tzinfo=self._tzinfo))
        raise OverflowError("result out of range")

The key here is that the _add__ function creates a new timedelta instance, and then uses the .combine() function to create a new output.

I'll show you two examples on how you can address this behaviour:

1. Overwriting the class method combine(cps, date, time, tzinfo=True):

   class worldtime
   
       ...
   
       @classmethod
       def combine(cls, date, time, tzinfo=True):
           "Construct a datetime from a given date and a given time."
           if not isinstance(date, _date_class):
               raise TypeError("date argument must be a date instance")
           if not isinstance(time, _time_class):
               raise TypeError("time argument must be a time instance")
           if tzinfo is True:
               tzinfo = time.tzinfo
           return cls(date.year, date.month, date.day,
                      time.hour, time.minute, time.second, time.microsecond,
                      tzinfo, fold=time.fold)
   

   This should now call the constructor for worldtime instead of the parent class datetime and return an object of worldtime. As the combine function is called from many of the existing magic methods it should hopefully cover other cases (and arithmetic operations).

2. Overwriting the __sub__(self, other) method:

   class worldtime:
   
       ...
   
       def __sub__(self, other):
           # the subtraction will turn sub into an instance of datetime
           # as we‘re calling the original subtraction function of datetime
           sub = super(worldtime, self).__sub__(other)
   
           # timetuple returns the parameters (year, month, day, etc.) 
           # and we need the first six parameters only to create a new instance.
           return worldtime(*sub.timetuple()[:6])
   

   This will convert the difference between self and other (which has turned into datetime) back into an instance of worldtime using its constructor.

The first option is likely to be cleaner as it will be applied to all arithmetic functions of datetime. The second option would require you to add more special cases to other arithmetic operations and may lead to greater implementation and maintenance efforts.