Type annotation for overloads that exclude types (something vs everything else)

Question

I'm trying to use @overload to communicate the different ways of calling a function, but what is easily communicated in the code with a simple else statement is not possible in the type annotations. Without the "else" MyPy (correctly) complains that the overload versions mismatch (see the snippet below for example).

error: Overloaded function signatures 1 and 2 overlap with incompatible return types

Did I understand correctly that there is no good solution for this problem?

eg. here is a simple example:

ListOrTuple = TypeVar("ListOrTuple", List, Tuple)
# unfortunately, typing doesn't support "anything else" at the moment
# https://github.com/python/typing/issues/599#issuecomment-586007066
AnythingElse = TypeVar("AnythingElse")
# what I would like to have is something like AnythingElse= TypeVar("AnythingElse", Not[List,Tuple])

@overload
def as_list(val: ListOrTuple) -> ListOrTuple:
    ...

@overload
def as_list(val: AnythingElse) -> List[AnythingElse]:
    ...

def as_list(val):
    """Return list/tuple as is, otherwise wrap in a list

    >>> as_list("test")
    ['test']
    """
    return val if isinstance(val, (list, tuple)) else [val]

Answer 1

This question is old enough, but this problem arises in multiple questions. This is intended to be a duplicate target, because the question is formulated in generic and reusable way. Other questions asking about overloads with type "X or not X" can be closed as dupes of this.

This problem (expressing type not X or Any \ X) bound to overloads is not planned for any mypy milestone.

First, the solution: just write both overloads as you have done, e.g. (I reduced imports and the overall complexity to simplify the example)

from typing import Any, TypeVar

ListOrTuple = TypeVar("ListOrTuple", list[Any], tuple[Any, ...])
AnythingElse = TypeVar("AnythingElse")

@overload
def as_list(val: ListOrTuple) -> ListOrTuple: ...  # type: ignore[misc]
@overload
def as_list(val: AnythingElse) -> list[AnythingElse]: ...

... and this works as intended. The error message about overlapping signatures is more like a lint warning than a real type error [1-2]. All you need is to ignore the error line like in example above. And everything will work great. Almost: see details below.

This is also explained in mypy official docs.

How does it work

In short ([4]):

• When overloaded function is called, mypy goes over every overload and checks if there is a match. If arguments passed to function are compatible with types of overload parameters, mypy picks this overload and processes it further as regular function (if TypeVars are involved, things are getting much more difficult - in fact mypy tries to substitute them and picks an overload only if this procedure succeeded). Here we consider A compatible to B if and only if A is a (nominal or structural, non-strict) subtype of B. What does it mean for us? If there are multiple matches, the first one will be used. Thus to say (int) -> str; (float which is not int) -> float we have to define two overloads, (int) -> str; (float) -> float in this order. Are you happy? I'm not, mypy fails to treat int as a float subtype in overloads, which is a reported bug.

• When overloaded definition is parsed and interpreted by mypy, it verifies its formal correctness. If arguments are not overlapping, overloads are OK. Otherwise, the more narrow are argument types, the more narrow return type should be - otherwise mypy complaints. Two different errors may be emitted here (numbers may differ with more signatures):

  1. Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader. This is a bad sign: your second overload is just completely ignored. Basically it means that you need to swap the signatures to make both work and to get another message instead:
  2. Overloaded function signatures 1 and 2 overlap with incompatible return types. This is much better! Basically it says "Are you sure that it is what you want? Strictly speaking, without implementation-specific assumptions any of overloads can be picked by PEP484-compatible type checker, plus there are some edge cases!". You can safely say "yes, I mean this!" and ignore the message. But beware (see below section)!

Why is the error message there?

Well, I cheated a bit when saying "everything will be fine" above. The main problem is that you can declare a variable to have wider type. If return types are not compatible, then you have actually tweaked type checker to produce some output not matching run-time behaviour. Like this:

class A: pass
class B(A): pass
@overload
def maybe_ok_1(x: B) -> int: ...  # E: Overloaded function signatures 1 and 2 overlap with incompatible return types
@overload
def maybe_ok_1(x: A) -> str: ...
def maybe_ok_1(x): return 0 if isinstance(x, B) else 'A'

# So far, so good:
reveal_type(maybe_ok_1(B()))  # N: revealed type is "builtins.int"
reveal_type(maybe_ok_1(A()))  # N: revealed type is "builtins.str"
# But this can be dangerous:
# This is `B`, and actual return is `int` - but `mypy` doesn't think so.
x: A = B()
reveal_type(maybe_ok_1(x))  # N: Revealed type is "builtins.str"  # Ooops!

Note that this isn't limited to "manual" upcasting as your argument may be passed to a function accepting a wider type:

# cont. previous snippet
def do_work(x: A) -> str:
    return maybe_ok_1(x) + "foo"

do_work(B())  # Oops, this raises a TypeError

Examples

Finally, check out a few examples for some practical view:

class A: pass
class B(A): pass

# Absolutely fine, no overlaps
@overload
def ok_1(x: str) -> int: ...
@overload
def ok_1(x: int) -> str: ...
def ok_1(x): return '1' if isinstance(x, int) else 1

reveal_type(ok_1(1))
reveal_type(ok_1('1'))

# Should use `TypeVar` instead, but this is a synthetic example - forgive me:)
@overload
def ok_2(x: B) -> B: ...
@overload
def ok_2(x: A) -> A: ...
def ok_2(x): return x

reveal_type(ok_2(A()))  # N: revealed type is "__main__.A"
reveal_type(ok_2(B()))  # N: revealed type is "__main__.B"

# But try to reverse the previous example - it is much worse!
@overload
def bad_1(x: A) -> A: ...
@overload
def bad_1(x: B) -> B: ...  # E: Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader  [misc]
def bad_1(x): return x

reveal_type(bad_1(B()))  # N: Revealed type is "__main__.A"  # Oops! Though, still true
reveal_type(bad_1(A()))  # N: Revealed type is "__main__.A"

# Now let's make it completely invalid:
@overload
def bad_2(x: A) -> int: ...
@overload
def bad_2(x: B) -> str: ...  # E: Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader  [misc]
def bad_2(x): return 'B' if isinstance(x, B) else 1

reveal_type(bad_2(B()))  # N: Revealed type is "builtins.int"  # Oops! The actual return is 'B'
reveal_type(bad_2(A()))  # N: Revealed type is "buitlins.int"


# Now watch something similar to ok_2, but with incompatible returns (we may want to ignore defn line)
@overload
def maybe_ok_1(x: B) -> int: ...  # E: Overloaded function signatures 1 and 2 overlap with incompatible return types
@overload
def maybe_ok_1(x: A) -> str: ...
def maybe_ok_1(x): return 0 if isinstance(x, B) else 'A'

# So far, so good:
reveal_type(maybe_ok_1(B()))  # N: revealed type is "builtins.int"
reveal_type(maybe_ok_1(A()))  # N: revealed type is "builtins.str"
# But this can be dangerous:
# This is `B`, and actual return is `int` - but `mypy` doesn't think so.
x: A = B()
reveal_type(maybe_ok_1(x))  # N: Revealed type is "builtins.str"  # Ooops!

Here's playground link so you can tweak the code to see what can be changed.

References from mypy issue tracker:

1. Just go on #12759
2. Just go on #13805
3. Special cases aren't special enough to break the rules? But they still keep it open...
4. Read me! This contains a short explanation of overload implementation
5. Ordering matters!

Answer 2

Use bound="Union[List, Tuple]":

from typing import Any, List, Tuple, TypeVar, Union, overload


ListOrTuple = TypeVar("ListOrTuple", bound="Union[List, Tuple]")
AnythingElse = TypeVar("AnythingElse")


@overload
def as_list(val: ListOrTuple) -> ListOrTuple:
    pass


@overload
def as_list(val: AnythingElse) -> List[AnythingElse]:
    pass


def as_list(val: Any) -> Any:
    """Return list/tuple as is, otherwise wrap in a list

    >>> as_list("test")
    ['test']
    """
    return val if isinstance(val, (list, tuple)) else [val]


a = as_list(2)  # it's List[int]
b = as_list('2')  # it's List[str]
c = as_list(['2', '3'])  # it's List[str]

Answer 3

This is the work-around that I have. It works well enough for me but I don't like it at all.

# attempt to list all the "other" possible types
AnythingElse = TypeVar("AnythingElse", Set, Mapping, type, int, str, None, Callable, Set, Deque, ByteString)
ListOrTuple = TypeVar("ListOrTuple", List, Tuple, Sequence)


@overload
def as_list(val: ListOrTuple) -> ListOrTuple:
    ...

@overload
def as_list(val: AnythingElse) -> List[AnythingElse]:
    ...

def as_list(val):
    """Return list/tuple as is, otherwise wrap in a list

    >>> as_list("test")
    ['test']
    """
    return val if isinstance(val, (list, tuple)) else [val]