CODEWITHSUNDEEP
pythongenericsstatic-analysismypy
Intrastellar Explorer
Intrastellar Explorer

Reputation: 2481

Python mypy checking of return types for TypeVar(bound=Union[A, B]) doesn't error vs TypeVar(A, B) does error

I am stuck trying to understand bounding of TypeVar when using it in two different ways:

Here is the code I am using:

#!/usr/bin/env python3.6

"""Figuring out why enum is saying incompatible return type."""


from enum import IntEnum, EnumMeta
from typing import TypeVar, Union


class Enum1(IntEnum):

    MEMBER1 = 1
    MEMBER2 = 2


class Enum2(IntEnum):

    MEMBER3 = 3
    MEMBER4 = 4


# Enums = TypeVar("Enums", bound=Union[Enum1, Enum2])  # Case 1... Success
Enums = TypeVar("Enums", Enum1, Enum2)  # Case 2... error: Incompatible return value


def _enum_to_num(val: int, cast_enum: EnumMeta) -> Enums:
    return cast_enum(val)


def get_some_enum(val: int) -> Enum1:
    return _enum_to_num(val, Enum1)


def get_another_enum(val: int) -> Enum2:
    return _enum_to_num(val, Enum2)  # line 35

When running mypy==0.770:

This case is very similar to this question: Difference between TypeVar('T', A, B) and TypeVar('T', bound=Union[A, B])

The answer explains when using case 1(bound=Union[Enum1, Enum2]), the following is legal:

  1. Union[Enum1, Enum2]
  2. Enum1
  3. Enum2

And when using case 2 (A, B), the following is legal:

  1. Enum1
  2. Enum2

However, I don't think this answer explains my problem, I am not using the Union case.

Can anyone please tell me what's going on?

Upvotes: 6

Views: 2193

Answers (2)

rocky
rocky

Reputation: 7098

I write first a little about what mypy sees and reports, and follow with the question of whether this is a mypy bug.

The message:

Incompatible return value type (got "Enum1", expected "Enum2")

means here that roughly that a Enum2 or a subtype of that is expected. Enum2 is the declared return value of get_another_enum(). However mypy thinks that the function call _enum_to_num() is returning an Enum1 type.

The "roughly" part is because there are exceptions for type checking when a type is unbound, or is an Any, or Union type; but that doesn't apply in this example.

Mypy decides that the function cast_enum() in _enum_to_num() is returning the first type listed in Enums — I guess as a static type checker, it has to pick one and that's what it does.

So if you switch the order around in the Enums assignment and write: 

Enums = TypeVar("Enums", Enum2, Enum1)  # Case 2... error: Incompatible return value

Then line 35 will succeed, but the return in get_some_enum() will fail with the message: 

error: Incompatible return value type (got "Enum2", expected "Enum1")

As to whether this is a mypy bug, it is hard to tell...

There is no dynamic type error that you can find here using the type() or ininstance() functions; running the code works as expected too.

On the other hand, Python never checks the return type whether at compile time or at run time: you could change the return type of _enum_to_none() to be None and that would still be valid as far as the Python interpreter is concerned.

The question then comes down to: in the static type system imposed by mypy, is this a bug? (I don't think PEP 484, 526, or other numbers try to address this).

Someone more qualified should answer the question of whether this is a bug that should be caught by a static analyzer, mypy in particular.

See Ken Hung's answer for a way to be more explicit and remove mypy's error.

Upvotes: 1

Ken Hung
Ken Hung

Reputation: 782

I think the error occurs because the type checker does not have enough information to infer the return type by looking at the types of input arguments. Although the handling may be improved.

Suppose you have a simple generic function:

Enums = TypeVar("Enums", Enum1, Enum2)

def add(x: Enums, y: Enums) -> Enums:
    return x

The type checker can infer the return type by type of input arguments:

add(Enum2.MEMBER3, Enum2.MEMBER4) # ok, return Enum2
add(Enum1.MEMBER1, Enum1.MEMBER2) # ok, return Enum1

add(Enum2.MEMBER3, Enum1.MEMBER2) # not ok

Look at your function _enum_to_num again, the type checker has no means to infer the return type, it just doesn't know what type will be returned because it doesn't know what type will be returned by cast_enum:

def _enum_to_num(val: int, cast_enum: EnumMeta) -> Enums:
    return cast_enum(val)

The idea of static type checking is that it evaluates the code without execution, it investigates the types of variables, not the dynamic values. By looking at the type of cast_enum, that is EnumMeta, the type checker cannot tell whether cast_enum will return Enums or not. Looks like it just assumes it will return Enum1, and it causes the error in _enum_to_num(val, Enum2).

You know that _enum_to_num(val, Enum2) will return Enum2 because you know the value of cast_enum is Enum2. The value is something that type checker doesn't touch in general. It may be confusing, the value of the variable cast_enum is Enum2, while the type of cast_enum is EnumMeta, although Enum2 is a type.

This issue can be solved by telling the type checker that types will be passed through cast_enum using typing.Type:

from typing import TypeVar, Union, Type

...

def _enum_to_num(val: int, cast_enum: Type[Enums]) -> Enums:
    return cast_enum(val)

The error will disappear because now the type checker can infer the return type.

Upvotes: 3

Related Questions