What's the point of using TypeVar?

Question

When speaking about Generics, Python gives the following example:

from collections.abc import Sequence
from typing import TypeVar

T = TypeVar('T')      # Declare type variable

def first(l: Sequence[T]) -> T:   # Generic function
    return l[0]

Could someone please explain what TypeVar does in this case? For example, if it can be anything, why not just give it the value Any? And if it's constrained, why not give it a Union value? In other words, what is the usefulness of using TypeVar(...)?

I suppose in reviewing the above, it's used when an element of some sort of sub-element? For example, it could be:

[{set1}, {set2}, {set3}, ...]

And that type would be Sequence[set] -> set

But something like this:

[1, 2, 3, ...]

Would have type Sequence[int] -> int. Are there any other usages besides this "item-in-iterable"?

Answer 1

Just wanted to say that with new version of Python (3.12+) it is no longer necessary to use TypeVars.

So, you can write, for example:

def foo[T](bar: T) → None:
     # code

instead of:

T = TypeVar('T')
def foo(bar: T) → None:
     # code

You can see some examples here: https://mypy.readthedocs.io/en/stable/generics.html#generic-methods-and-generic-self

Answer 2

The best example is with pydantic.

Imagine I have a function that implements pydantic, and I want that function to be able to handle my pydantic type for records retrieved as a dict from firestore. That code may look something like this:

class MyModel(BaseModel):
    ...

class MyClass:
    def get_records(...) -> Generator[MyModel, None, None]:
        for record in self.client.collection("MyModel").where(...).stream(...):
            body = record.to_dict()
            if body:
                yield MyModel.model_validate(body)

Now that's great and all, but then what if I have multiple models, then I have to define a function for each one, right? Pretty annoying.

Ok what if I use a Union.

class MyModel(BaseModel):
    ...

class MyModel2(BaseModel):
    ...

T: TypeAlias = Union[MyModel, MyModel2]

class MyClass:
    def get_records(
        my_union_type: Type[T],
        collection_name: str
    ) -> Generator[T, None, None]:
        for record in self.client.collection(collection_name).where(...).stream(...):
            body = record.to_dict()
            if body:
                if isinstance(my_union_type, MyModel):
                    yield MyModel.model_validate(body)
                elif isinstance(my_union_type, MyModel2):
                    yield MyModel2.model_validate(body)

As you can see we can now use our new type, but its a bit messy, no? We are calling the same method model_validate but the only problem is, we can't refer to the type in a dynamic way and just say "a unknown basemodel" like this.

And, what's more, for each type we want it to handle we have to repeat this same logic...

Step in TypeVar... TypeVar allows us to specify a variable for a Type, as a opposed to defining a type as a type.

If that doesn't make sense, think of it this way:

# this is a type stored in a variable
my_model_as_a_type = MyModel

# this is an instance stored in a variable
mymodel_as_an_instance = MyModel()

As you can see, a type is literally the thing that defines what that object looks like, but it cannot be used as that object, because it is not an instance of it.

An instance is that thing, initialised in memory, with all the functions and whatever else that implementation of its type has defined.

So, moving onto typevar... how can we improve our code then?

Well, with a few simple changes we can make our function accept any basemodel, but not explicitly say it has to be BaseModel itself, e.g. can be a child or whatever and that the caller of the function can see the type returned to them...

class MyModel(BaseModel):
    ...

class MyModel2(BaseModel):
    ...

T = TypeVar("T", bound=BaseModel)

class MyClass:
    def get_records(
        model_type: Type[T],
        collection_path: str
    ) -> Generator[T, None, None]:
        adapter = TypeAdapter(model_type)
        for record in self.client.collection(collection_path).where(...).stream(...):
            body = record.to_dict()
            if body:
                yield adapter.validate_python(body)

Now when you pass in a model to this function, the object you get back will be an instance from the firestore collection of the type you gave it. And typecheckers are happy, and your colleagues are happy because now they know what the type being returned is, instead of it "possibly" being a union and the output being a "variable" of whatever model happens to be passed in and leaving it to the caller to then figure out which of the union types it is, you can have functions that are defined to only operate on specific collections for specific types...

So we could then have get_my_model_records and get_my_model_2_records function that yield the results of get_records as another generator, and merely provide their specific types to it and whatever else specific implementation details you need, meaning the consumers of your API for these two methods, know exactly what objects they will get back when they call these functions and can easily see that each method is specific for each type.

Answer 3

The purpose of the TypeVar in this context is to say that the function returns a specific type that is related to the argument's type.

For example, if you did:

a = first([1, 2, 3]) + "foo"

you would get an error, because in this expression T becomes bound to the type int, and so you'd get an error about adding an int and a str.

If you annotated first with Any types as you describe, this would not produce a mypy error (and hence you'd get a TypeError at runtime instead), because the return value of first would always simply be Any.

See the mypy documentation on generics for a lot more examples of how to use typevars: https://mypy.readthedocs.io/en/stable/generics.html