How do I get Python dataclass InitVar fields to work with typing.get_type_hints while also using annotations?

Question

When messing with Python dataclasses, I ran into this odd error that's pretty easy to reproduce.

from __future__ import annotations

import dataclasses as dc
import typing

@dc.dataclass
class Test:
    foo: dc.InitVar[int]

print(typing.get_type_hints(Test))

Running this gets you the following:

Traceback (most recent call last):
  File "test.py", line 11, in <module>
    print(typing.get_type_hints(Test))
  File "C:\Program Files\Python310\lib\typing.py", line 1804, in get_type_hints
    value = _eval_type(value, base_globals, base_locals)
  File "C:\Program Files\Python310\lib\typing.py", line 324, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "C:\Program Files\Python310\lib\typing.py", line 687, in _evaluate
    type_ =_type_check(
  File "C:\Program Files\Python310\lib\typing.py", line 173, in _type_check
    raise TypeError(f"{msg} Got {arg!r:.100}.")
TypeError: Forward references must evaluate to types. Got dataclasses.InitVar[int].

Without from __future__ import annotations, it seems to work fine; but in the actual code I'm making use of that import in a couple different type hints. Is there no way to make it so that the annotations import doesn't break this?

Answer 1

So I was actually able to replicate this exact same behavior in my Python 3.10 environment, and frankly was sort of surprised that I was able to do so. The issue, at least from the surface, seems to be with InitVar and with how typing.get_type_hints resolves such non-generic types.

Anyways, before we get too deep into the weeds, it's worth clarifying a bit about how the from __future__ import annotations works. You can read more about it in the PEP that introduces it into the wild, but essentially the story "in a nutshell" is that the __future__ import converts all annotations in the module where it is used into forward-declared annotations, i.e. ones that are wrapped in a single quotes ' to render all type annotations as string values.

So then with all type annotations converted to strings, what typing.get_type_hints actually does is to resolve those ForwardRef types -- which is essentially the typing library's way of identifying annotations that are wrapped in strings -- using a class or module's globals namespace, along with an optional locals namespace if provided.

Here's a simple example to basically bring home all that was discussed above. All I'm doing here, is instead of using from __future__ import annotations at the top of the module, I'm manually going in and forward declaring all annotations by wrapping them in strings. It's worth noting that this is essentially the same as how it appears in the question above.

import typing
from dataclasses import dataclass, InitVar


@dataclass
class Test:
    foo: 'InitVar[int]'


print(typing.get_type_hints(Test))

If curious, you can also try with a __future__ import and without forward declaring the annotations manually, and then inspect the Test.__annotations__ object to confirm that the end result is the same as how I've defined it above.

In either case, we run into the same error below, also as noted in the OP above:

Traceback (most recent call last):
    print(typing.get_type_hints(Test))
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 1804, in get_type_hints
    value = _eval_type(value, base_globals, base_locals)
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 324, in _eval_type
    return t._evaluate(globalns, localns, recursive_guard)
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 687, in _evaluate
    type_ =_type_check(
  File "C:\Users\USER\.pyenv\pyenv-win\versions\3.10.0\lib\typing.py", line 173, in _type_check
    raise TypeError(f"{msg} Got {arg!r:.100}.")
TypeError: Forward references must evaluate to types. Got dataclasses.InitVar[int].

Let's note the stack trace as it's certainly to useful to know where things went wrong. However, we'll likely want to explore exactly why the dataclasses.InitVar usage resulted in this strange and unusual error in the first place, which is actually what we'll look at to start with.

So what's up with dataclasses.InitVar?

The TL;DR here is there's a problem with subscripted dataclasses.InitVar usage specifically. Anyway, let's look at only the relevant parts of how InitVar is defined in Python 3.10:

class InitVar:

    def __init__(self, type):
        self.type = type
    
    def __class_getitem__(cls, type):
        return InitVar(type)

Note that the __class_getitem__ is the method that is called when we subscript the class in an annotation, for example like InitVar[str]. This calls InitVar.__class_getitem__(str) which returns InitVar(str).

So the actual problem here is, the subscripted InitVar[int] usage returns an InitVar object, rather than the underlying type, which is the InitVar class itself.

So typing.get_type_hints is causing an error here because it sees an InitVar instance in the resolved type annotation, rather than the InitVar class itself, which is a valid type as it's a Python class essentially.

Hmm... but what seems to be the most straightforward way to resolve this?

The (Patchwork) Road to a Solution

If you check out the source code of typing.get_type_hints at least in Python 3.10, you'll notice that it's converting all string annotations to ForwardRef objects explictly, and then calling ForwardRef._evaluate on each one:

for name, value in ann.items():
    ...
    if isinstance(value, str):
        value = ForwardRef(value, is_argument=False)
>>  value = _eval_type(value, base_globals, base_locals)

What the ForwardRef._evaluate method does is eval the contained reference using the class or module globals, and then internally call typing._type_check to check the reference contained in the ForwardRef object. This does a couple things like validating that the reference is of a Generic type from the typing module, which definitely aren't of interest here, since InitVar is explicitly defined is a non-generic type, at least in 3.10.

The relevant bits of typing._type_check are shown below:

    if isinstance(arg, _SpecialForm) or arg in (Generic, Protocol):
        raise TypeError(f"Plain {arg} is not valid as type argument")
    if isinstance(arg, (type, TypeVar, ForwardRef, types.UnionType, ParamSpec)):
        return arg
    if not callable(arg):
>>      raise TypeError(f"{msg} Got {arg!r:.100}.")

It's the last line shown above, raise TypeError(...) which seems to return the error message that we're running into. If you check the last condition that the _type_check function checks, you can kind of guess how we can implement the simplest possible workaround in our case:

if not callable(arg):

If we glance a little briefly into the documentation for the callable builtin, we get our first concrete hint of a possible solution we can use:

def callable(i_e_, some_kind_of_function): # real signature unknown; restored from __doc__
    """
    Return whether the object is callable (i.e., some kind of function).
    
    Note that classes are callable, as are instances of classes with a
    __call__() method.
    """

So, simply put, all we need to do is define a __call__ method under the dataclasses.InitVar class. This can be a stub method, essentially a no-op, but at a minimum the class must define this method so that it can be considered a callable, and thus the typing module can accept it as a valid reference type in a ForwardRef object.

Finally, here's the same example as in the OP, but slightly modified to add a new line which patches dataclasses.InitVar to add the necessary method, as a stub:

from __future__ import annotations

import typing
from dataclasses import dataclass, InitVar


@dataclass
class Test:
    foo: InitVar[int]


# can also be defined as:
#   setattr(InitVar, '__call__', lambda *args: None)
InitVar.__call__ = lambda *args: None

print(typing.get_type_hints(Test))

The example now seems to work as expected, without any errors raised by the typing.get_type_hints method, when forward declaring any subscripted InitVar annotations.