Is requirements.txt still needed when using pyproject.toml?

Question

Since mid 2022 it is now possible to get rid of setup.py, setup.cfg in favor of pyproject.toml. Editable installs work with recent versions of setuptools and pip and even the official packaging tutorial switched away from setup.py to pyproject.toml.

However, documentation regarding requirements.txt seems to be have been also removed, and I wonder where to put the pinned requirements now?

As a refresher: It used to be common practice to put the dependencies (without version pinning) in setup.py avoiding issues when this package gets installed with other packages needing the same dependencies but with conflicting version requirements. For packaging libraries a setup.py was usually sufficient.

For deployments (i.e. non libraries) you usually also provided a requirements.txt with version-pinned dependencies. So you don't accidentally get the latest and greatest but the exact versions of dependencies that that package has been tested with.

So my question is, did anything change? Do you still put the pinned requirements in the requirements.txt when used together with pyproject.toml? Or is there an extra section for that in pyproject.toml? Is there some documentation on that somewhere?

Answer 1

Context

I found this SO question "Is requirements.txt still needed when using pyproject.toml" and have a different use case compared to the one described by the above question. Instead of opening an extra SO question with the same title, I decided to add this answer as kind of a "long comment", supporting users in similar situations.

In my use case I do not install my code as a library but use it as an application.

Solution

It is possible to install dependencies from pyproject.toml by using hatchling as a build-backend and defining the dependencies in the sections dependencies and [project.optional-dependencies].

Install the dependencies from pyproject.toml with

pip install .[dev]

Example for the corresponding sections in pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = 'my_application_name'
version = '0.0.1'
requires-python = ">=3.11.0"

dependencies = [
    'cryptography==41.0.1',
]

[project.optional-dependencies]
linting = [
  'pylint==2.17.4'
]
formatting = [
  'black[d]==23.3.0'
]
dev = ['my_application_name[linting, formatting]']

Related:

https://hatch.pypa.io/latest/meta/faq/#libraries-vs-applications

https://peps.python.org/pep-0665/

https://github.com/pypa/packaging.python.org/issues/685

https://peps.python.org/pep-0631/

https://stackoverflow.com/a/75503961/2876079

https://pip.pypa.io/en/stable/cli/pip_install/#options

https://github.com/winpython/winpython/issues/1128

How to write a minimally working pyproject.toml file that can install packages?

Answer 2

So my question is, did anything change? Do you still put the pinned requirements in the requirements.txt when used together with pyproject.toml? Or is there an extra section for that in pyproject.toml? Is there some documentation on that somewhere?

Nothing changed. Pinned dependencies do not belong in the project's packaging metadata, which means that pinned dependencies do not belong in the dependencies list of the [project] section of the pyproject.toml file. As of today there is no standardized section of pyproject.toml that I know of for pinned dependencies. This document even though quite outdated still applies in the spirit: "abstract dependencies" belong in the package metadata (for example in the [project] section of pyproject.toml) and "concrete dependencies" can be listed in a requirements.txt file if needed (or in any other file that serves a similar purpose like Poetry's poetry.lock, or Pipenv's Pipfile.lock to name only two of them, depending on the project's development workflow and tooling).

In short and simplified, the abstract dependencies are about the what should be installed, and the concrete dependencies are about the how to install (from which index, which version specifically, which build, which distribution, and so on).

Abstract dependencies are meant to be valid for anyone who installs your project (library). But the concrete dependencies will very likely vary from one user to another, or from one computer to another, they might even change over time. For example, some users might not have access to the same package indexes as others (PyPI is not available in some regions, some companies use private internal mirror repositories). Some dependencies need very specific builds that depend on the hardware available on the computer (PyTorch, for example).

Answer 3

Quoting myself from here

My current assumption is: [...] you put your (mostly unpinned) dependencies to pyproject.toml instead of setup.py, so you library can be installed as a dependency of something else without causing much troubles because of issues resolving version constraints.

On top of that, for "deployable applications" (for lack of a better term), you still want to maintain a separate requirements.txt with exact version pinning.

Which has been confirmed by a Python Packaging Authority (PyPA) member and clarification of PyPA's recommendations should be updated accordingly at some point.

Answer 4

I suggest switching to poetry, it's way better than a standard pip for dependency management. And because it uses pyproject.toml your dependencies and configs are in one place so it's easier to manage everything 🙂

Answer 5

This is the pip documentation for pyproject.toml

...This file contains build system requirements and information, which are used by pip to build the package.

So this is not the correct place. Looking at the side bar we can see there is an entry for Requirements File Format which is the "old" requirements.txt file