CODEWITHSUNDEEP
pythonpython-3.xmypypython-typing
Brad Solomon
Brad Solomon

Reputation: 40918

Extending stub file for a third-party library/module

I am using the yarl library's URL object.

It has a quasi-private attribute, ._val, which is a urllib.parse.SplitResult object but has no type annotation in yarl/__init__.pyi. (Understandably so, if the developer does not want to formally make it part of the public API.)

However, I have chosen to use URL._val at my own risk. A dummy example:

# urltest.py
from urllib.parse import SplitResult
from typing import Tuple

from yarl import URL


def foo(u: URL) -> Tuple[str, str, str]:
    sr: SplitResult = u._val
    return sr[:3]

But mypy doesn't like this, because it complains:

$ mypy urltest.py
"URL" has no attribute "_val"

So, how can I, within my own project, "tack on" (or extend) an instance attribute annotation to URL so that it can be used through the rest of my project? i.e.

from yarl import URL

URL._val: SplitResult
# ...

(mypy does not like this either; "Type cannot be declared in assignment to non-self attribute.")

I've tried creating a new stub file, in stubs/yarl/__init__.pyi:

from urllib.parse import SplitResult

class URL:
    _val: SplitResult

And then setting export MYPYPATH='.../stubs' as described in stub files. However, this overrides, not extends, the existing annotations, so everything but ._val throws an error:

error: "URL" has no attribute "with_scheme"
error: "URL" has no attribute "host"
error: "URL" has no attribute "fragment"

...and so on.

Upvotes: 14

Views: 7705

Answers (4)

adam.hendry
adam.hendry

Reputation: 5653

Unfortunately, I don't think there's really a way of making "partial" changes to the type hints for some 3rd party library -- at least, not with mypy.

Actually, there is. Per PEP 561, the first place type checkers "SHOULD" look for stubs is in the $PATH:

  1. Stubs or Python source manually put in the beginning of the path. Type checkers SHOULD provide this to allow the user complete control of which stubs to use, and to patch broken stubs/inline types from packages. In mypy the $MYPYPATH environment variable can be used for this.

Hence, fill $MYPYPATH with a list of paths to extra directories where mypy should look for stubs and put your fixes there. You "SHOULD" be able to simply overwrite the section that is failing with proper types. Per the mypy docs:

These stub files do not need to be complete! A good strategy is to use stubgen, a program that comes bundled with mypy, to generate a first rough draft of the stubs. You can then iterate on just the parts of the library you need.

You "SHOULDN'T" even have to use stubgen, but try it out (you may have to use stubgen if you need the other type hints from the package, though I'm not sure). Even if you do, worst case, run stubgen on the file and overwrite the part of the stub that's broken.

Upvotes: 3

David Gilbertson
David Gilbertson

Reputation: 4883

One option is to create a new class, based on the class you want to 'extend'. I do this for Pandas DataFrame objects when I want autocomplete for the data I'm working with.

import pandas as pd

class TitanicDataFrame(pd.DataFrame):
    PassengerId: pd.Series
    Survived: pd.Series
    Name: pd.Series
    Sex: pd.Series
    Age: pd.Series


df: TitanicDataFrame = pd.read_csv('data/titanic.csv')
mean_age = df.Age.mean()

Note that the TitanicDataFrame class isn't actually used (as a class), it's only used as the type (thus ignored at runtime).

Upvotes: 2

Michael0x2a
Michael0x2a

Reputation: 64228

Unfortunately, I don't think there's really a way of making "partial" changes to the type hints for some 3rd party library -- at least, not with mypy.

I would instead try one of the following three options:

  1. Just # type: ignore the attribute access:

    def foo(u: URL) -> Tuple[str, str, str]:
    sr: SplitResult = u._val  # type: ignore
    return sr[:3]

    This type-ignore will suppress any error messages that are generated on that line. If you're going to take this approach, I'd also recommend running mypy with the --warn-unused-ignores flag, which will report any redundant and unused # type: ignore statements. It's unlikely this particular # type: ignore will become redundant as mypy updates/as the stubs for your third party library updates, but it's a nice flag to enable just in general.

  2. Talk to the maintainer of this library and see if they're willing to either add a type hint for this attribute (even if it's private), or to expose this information via some new API.

    If it helps, there is some precedent for adding type hints even for private or undocumented attributes in Typeshed, the repository of types for the standard library -- see the "What to include" section in their contribution guidelines.

  3. If the library maintainer isn't willing to add this attribute, you could always just fork the stubs for this library, make the change to the forked stubs, and start using that.

I would personally try solution 2 first, followed by solution 1, but that's just me.

Upvotes: 4

chepner
chepner

Reputation: 531948

One possibility is to simply ignore the type of u for this assignment:

def foo(u: URL) -> Tuple[str, str, str, str]:
    sr: SplitResult = typing.cast(typing.Any, u)._val
    return sr[:3]

mypy will assume you know what you are doing, and that u has a _val attribute with type str.

Upvotes: 1

Related Questions