How should Poetry scripts used in the build process be stored in the project?

Question

Let's say that you're using Poetry to manage a Python PyPI package. As it stands, your project has a Makefile that contains procedures for managing the installation, unit testing, and linting of your project. However, since this goes against the spirit of Poetry where you ideally have one configuration file to rule them all (pyproject.toml), and because said Makefile can be annoying on Windows builds, you would like to instead move the Makefile functionality to be managed by Poetry directly by making use of Poetry's support for setuptools scripts.

Here's an example of such a Makefile:

PYMODULE := awesome_sauce
TESTS := tests
INSTALL_STAMP := .install.stamp
POETRY := $(shell command -v poetry 2> /dev/null)
MYPY := $(shell command -v mypy 2> /dev/null)

.DEFAULT_GOAL := help

.PHONY: all
all: install lint test

.PHONY: help
help:
    @echo "Please use 'make <target>', where <target> is one of"
    @echo ""
    @echo "  install     install packages and prepare environment"
    @echo "  lint        run the code linters"
    @echo "  test        run all the tests"
    @echo "  all         install, lint, and test the project"
    @echo "  clean       remove all temporary files listed in .gitignore"
    @echo ""
    @echo "Check the Makefile to know exactly what each target is doing."
    @echo "Most actions are configured in 'pyproject.toml'."

install: $(INSTALL_STAMP)
$(INSTALL_STAMP): pyproject.toml
    @if [ -z $(POETRY) ]; then echo "Poetry could not be found. See https://python-poetry.org/docs/"; exit 2; fi
    $(POETRY) run pip install --upgrade pip setuptools
    $(POETRY) install
    touch $(INSTALL_STAMP)

.PHONY: lint
lint: $(INSTALL_STAMP)
    # Configured in pyproject.toml
    # Skips mypy if not installed
    @if [ -z $(MYPY) ]; then echo "Mypy not found, skipping..."; else echo "Running Mypy..."; $(POETRY) run mypy $(PYMODULE) $(TESTS); fi
    @echo "Running Flake8..."; $(POETRY) run pflake8 # This is not a typo
    @echo "Running Pylint..."; $(POETRY) run pylint $(PYMODULE)

.PHONY: test
test: $(INSTALL_STAMP)
    # Configured in pyproject.toml
    $(POETRY) run pytest

.PHONY: clean
clean:
    # Delete all files in .gitignore
    git clean -Xdf

But then, you face another problem; due to how these scripts operate, they're expected to be in a package or module of their own, so the first idea would be to simply include the scripts as part of the project package itself:

📦awesome_sauce
 ┣ 📂awesome_sauce
 ┃ ┣ 📂poetry_scripts
 ┃ ┃ ┣ 📜__init__.py
 ┃ ┃ ┣ 📜run_unit_tests.py
 ┃ ┃ ┗ 📜run_linters.py
 ┃ ┣ 📜__init__.py
 ┃ ┗ 📜sauce.py
 ┣ 📂docs
 ┣ 📂tests
 ┣ 📜.gitignore
 ┣ 📜poetry.lock
 ┗ 📜pyproject.toml

In this case, you could add the following to pyproject.toml

[tool.poetry.scripts]
tests = 'awesome_sauce.poetry_scripts.run_unit_tests:main'
linters = 'awesome_sauce.poetry_scripts.run_linters:main'

and they could be run as

poetry run tests
poetry run linters

But this isn't ideal; now your build configuration is part of the package and presumably goes straight into production, unless you pull off something fancy during the CI/CD process. It's dead weight to the end-users, and needlessly adds stuff to the package namespace.

Then, another idea would be to have everything in a top-level script:

📦awesome_sauce
 ┣ 📂awesome_sauce
 ┃ ┣ 📜__init__.py
 ┃ ┗ 📜sauce.py
 ┣ 📂docs
 ┣ 📂tests
 ┣ 📜.gitignore
 ┣ 📜poetry.lock
 ┣ 📜poetry_scripts.py
 ┗ 📜pyproject.toml

which would change the lines in pyproject.toml to

[tool.poetry.scripts]
tests = 'poetry_scripts:run_unit_tests'
linters = 'poetry_scripts:run_linters'

but this isn't necessarily ideal either, because now you potentially have a relatively lengthy script sitting at the project root, and it can be subjectively ugly. Though at least now it shouldn't be part of the production package, which is a net gain.

These two approaches could be combined, and the script package in the first version would simply be moved to the root, meaning the project now consists of two different packages.

📦awesome_sauce
 ┣ 📂awesome_sauce
 ┃ ┣ 📜__init__.py
 ┃ ┗ 📜sauce.py
 ┣ 📂docs
 ┣ 📂poetry_scripts
 ┃ ┣ 📜__init__.py
 ┃ ┣ 📜run_unit_tests.py
 ┃ ┗ 📜run_linters.py
 ┣ 📂tests
 ┣ 📜.gitignore
 ┣ 📜poetry.lock
 ┗ 📜pyproject.toml

[tool.poetry.scripts]
tests = 'poetry_scripts.run_unit_tests:main'
linters = 'poetry_scripts.run_linters:main'

but it isn't clear whether this is a better approach.

Does a consensus exist for handling the Poetry/setuptools scripts within a given project? If so, I don't believe that has been previously documented in public.

Answer 1

I don't believe there is a clear consensus on this.

However I've been working on a solution called Poe the Poet which is intended to be a better fit than make for most projects; it integrates nicely with poetry, works seamlessly cross-platform (depending how you define your tasks), and it's self documenting.

Firstly, it's important to be aware that I don't think [tool.poetry.scripts] is what you want at all, because anything you define there will actually be installed into the PATH of any environment where your package is installed!

What poethepoet allows you to do is define tasks in your pyproject.toml like so:

[tool.poe.tasks]
test = "pytest -v"

Which can then be invoked with the poe CLI tool like

poe test tests/features # extra args are appended to the command

Or if you want to define the task logic in a python module (and make the task self documenting while you're at it) you can do something like:

[tool.poe.tasks.lint]
script = "poetry_scripts.run_linters:main"
help   = "Check code style and such"

Running poe without any arguments prints documentation including available tasks and their help messages.

As for the project layout, I'd suggest putting all your python code under ./src to get something like:

📦awesome_sauce
 ┣ 📂src
 ┃ ┣ 📂awesome_sauce
 ┃ ┃ ┣ 📜__init__.py
 ┃ ┃ ┗ 📜sauce.py
 ┃ ┣ 📂scripts
 ┃ ┃ ┣ 📜__init__.py
 ┃ ┃ ┣ 📜run_unit_tests.py
 ┃ ┃ ┗ 📜run_linters.py
 ┣ 📂docs
 ┣ 📂tests
 ┣ 📜.gitignore
 ┣ 📜poetry.lock
 ┗ 📜pyproject.toml

and then setting the following in the poetry section of your pyproject.toml so that awesome_sauce is included in your build, but everything else in src will be ignored by poetry build

packages = [{include = "awesome_sauce", from = "src"}]

It also works as a poetry plugin.