CODEWITHSUNDEEP
pythonpython-3.xgenerator
mick.io
mick.io

Reputation: 511

Proper use Generator typing

I'm attempting to add typing to a method that returns a generator. Whenever I run this program with the return type specified, a TypeError is raised.

Adding quotes or removing the typing fixes the error, but this seems like hack. Surely there is a correct way of doing this.

def inbox_files(self) -> "Generator[RecordsFile]":
    ...

# OR

def inbox_files(self):
    ...

from typing import Generator, List
from .records_file import RecordsFile

Class Marshaller:

    ...

    def inbox_files(self) -> Generator[RecordsFile]:
        return self._search_directory(self._inbox)

    def _search_directory(self, directory: str) -> RecordsFile:
        for item_name in listdir(directory):
            item_path = path.join(item_name, directory)
            if path.isdir(item_path):
                yield from self._search_directory(item_path)
            elif path.isfile(item_path):
                yield RecordsFile(item_path)
            else:
                print(f"[WARN] Unknown item found: {item_path}")

The following stack trace is produced:

Traceback (most recent call last):
  File "./bin/data_marshal", line 8, in <module>
    from src.app import App
  File "./src/app.py", line 9, in <module>
    from .marshaller import Marshaller
  File "./src/marshaller.py", line 9, in <module>
    class Marshaller:
  File "./src/marshaller.py", line 29, in Marshaller
    def inbox_files(self) -> Generator[RecordsFile]:
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 254, in inner
    return func(*args, **kwds)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 630, in __getitem__
    _check_generic(self, params)
  File "/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/lib/python3.7/typing.py", line 208, in _check_generic
    raise TypeError(f"Too {'many' if alen > elen else 'few'} parameters for {cls};"
TypeError: Too few parameters for typing.Generator; actual 1, expected 3

¯\_(ツ)_/¯

Upvotes: 33

Views: 13471

Answers (4)

chepner
chepner

Reputation: 531993

Update: Python 3.13 now allows for the send and result types to default to None if not provided.

You have to explicitly specify the send type and the return type, even if both are None.

def inbox_files(self) -> Generator[RecordsFile,None,None]:
    return self._search_directory(self._inbox)

Note that the yield type is what you might think of as the return type. The send type is the type of value you can pass to the generator's send method. The return type is the type of value that could be embedded in the StopIteration exception raised by next after all possible value have been yielded. Consider:

def foo():
    yield 3
    return "hi"

f = foo()

The first call to next(f) will return 3; the second will raise StopIteration("hi"). )

A generator that you cannot send into or return from is simply an iterable or an iterator (either, apparently can be used).

def inbox_files(self) -> Iterable[RecordsFile]:  # Or Iterator[RecordsFile]
    return self._search_directory(self._inbox)

_search_directory itself also returns a generator/iterable, not an instance of RecordsFile:

def _search_directory(self, directory: str) -> Iterable[RecordsFile]:

Upvotes: 40

Craynic Cai
Craynic Cai

Reputation: 397

From Python 3.13 you can write such code:

from collections.abc import Generator

def my_func(arg1) -> Generator[RecordsFile]:
    ...

which is much simpler.

Ref

Upvotes: 1

Oleksandr Boiko
Oleksandr Boiko

Reputation: 352

If using Generator[RecordsFile, None, None] too often is too cumbersome, you can alias a type:

from typing import Generator, TypeVar

T = TypeVar('T')
type Gen[T] = Generator[T, None, None]

# now use `Gen[RecordsFile]` as an alias for `Generator[RecordsFile, None, None]`

Upvotes: 0

Dennis O&#39;Connor
Dennis O&#39;Connor

Reputation: 11

This answer was useful, but I was confused since I was sure I had used Generator[] with just one parameter in the past and it worked.

I traced it back to using "from __future__ import annotations". Only one parameter seems to be required in that case.

Upvotes: 0

Related Questions