How do you create a fully-fledged Python package?

Question

When creating a Python package, you can simply write the code, build the package, and share it on PyPI. But how do you do that?

How do you create a Python package?
How do you publish it?

And then, what if you want to go further?

How do you set up CI/CD for it?
How do you test it and check code coverage?
How do you lint it?
How do you automate everything you can?

RGS · Accepted Answer

Preamble

When you've published dozens of packages, you know how to answer these questions in ways that suit your workflow(s) and taste. But answering these questions for the first time can be quite difficult, time consuming, and frustrating!

That's why I spent days researching ways of doing these things, which I then published as a blog article called How to create a Python package in 2022.

That article, and this answer, document my findings for when I wanted to publish my package extendedjson

Overview

Here is an overview of some tools you can use and the steps you can take, in the order I followed them while discovering all of this.

Disclaimer: other alternative tools exist (usually) & most of the steps here are not mandatory.

Use Poetry for dependency management
Use GitHub to host the code
Use pre-commit to ensure committed code is linted & formatted well
Use Test PyPI to test uploading your package (which will make it installable with pip)
Use Scriv for changelog management
Upload to the real PyPI
Use pytest to test your Python code
Use tox to automate linting, formatting, and testing across Python versions
- black
- isort
- pylint
- flake8 with mccabe
Add code coverage with coverage.py
Set up CI/CD with GitHub Actions
- run linters and tests
- trigger automatically on pull requests and commits
- integrate with Codecov for coverage reports
- publish to PyPI automatically
Add cool README badges
Tidy up a bit
- set tox to use pre-commit
- remove duplicate work between tox and pre-commit hooks
- remove some redundancy in CI/CD

Steps

Here is an overview of the things you can do and more or less how to do it. Again, thorough instructions plus the rationale of why I picked certain tools, methods, etc, can be found in the reference article.

Use Poetry for dependency management.
- poetry init initialises a project in a directory or poetry new dirname creates a new directory structure for you
- do poetry install to install all your dependencies
- poetry add packagename can be used to add packagename as a dependency, use -D if it's a development dependency (i.e., you need it while developing the package, but the package users won't need it. For example, black is a nice example of a development dependency)
Set up a repository on GitHub to host your code.
Set up pre-commit hooks to ensure your code is always properly formatted and it passes linting. This goes on .pre-commit-config.yaml. E.g., the YAML below checks TOML and YAML files, ensures all files end with a newline, makes sure the end-of-line marker is consistent across all files, and then runs black and isort on your code.

# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: check-toml
      - id: check-yaml
      - id: end-of-file-fixer
      - id: mixed-line-ending
  - repo: https://github.com/psf/black
    rev: 22.3.0
    hooks:
      - id: black
  - repo: https://github.com/PyCQA/isort
    rev: 5.10.1
    hooks:
      - id: isort
        args: ["--profile", "black"]

Configure Poetry to use the Test PyPI to make sure you can publish a package and it is downloadable & installable.
- Tell Poetry about Test PyPI with poetry config repositories.testpypi https://test.pypi.org/legacy/
- Log in to Test PyPI, get an API token, and tell Poetry to use it with poetry config http-basic.testpypi __token__ pypi-your-api-token-here (the __token__ is a literal and shouldn't be replaced, your token goes after that).
- Build poetry build and upload your package poetry publish -r testpypi
Manage your CHANGELOG with Scriv
- run scriv create before any substantial commit and edit the file that pops up
- run scriv collect before any release to collect all fragments into one changelog
Configure Poetry to use PyPI
- login to PyPI and get an API token
- tell Poetry about it with poetry config pypi-token.pypi pypi-your-token-here
- build & publish your package in one fell swoop with poetry publish --build
Do a victory lap: try pip install yourpackagename to make sure everything is going great ;)
Publish a GH release that matches what you uploaded to PyPI
Write tests. There are many options out there. Pytest is simple, versatile, and not too verbose.
- write tests in a directory tests/
- start test files with test_...
- actual tests are functions with a name starting with test_...
- use assertions (assert) to check for things (tests fail when asserting something Falsy); notice sometimes you don't even need to import pytest in your test files; e.g.:

# In tests/test_basic_example.py

def this_test_would_definitely_fail():
    assert 5 > 10

def this_test_would_definitely_pass():
    assert 5 > 0

run tests with the command pytest
Automate testing, linting, and formatting, with tox.
- tox creates virtual environments for separate Python versions and can run essentially what you tell it to. Configuration goes in tox.ini. You can also embed it in the file pyproject.toml, but as of writing this, that's only supported if you add a string that actually represents the .ini configuration, which is ugly. Example tox.ini:

[tox]
isolated_build = True
envlist = py38,py39,py310

[testenv]
deps =
    black
    pytest
commands =
    black --check extendedjson
    pytest .

The environments py38 to py310 are automatically understood by tox to represent different Python versions (you guess which ones). The header [testenv] defines configurations for all those environments that tox knows about. We install the dependencies listed in deps = ... and then run the commands listed in commands = ....

run tox with tox for all environments or tox -e py39 to pick a specific environment
Add code coverage with coverage.py
- run tests and check coverage with coverage run --source=yourpackage --branch -m pytest .
- create a nice HTML report with coverage html
- add this to tox
Create a GitHub action that runs linting and testing on commits and pull requests
- GH Actions are just YAML files in .github/workflows
- this example GH action runs tox on multiple Python versions

# .github/workflows/build.yaml
name: Your amazing CI name

# Run automatically on...
on:
  push:  # pushes...
    branches: [ main ]  # to the main branch... and
  pull_request:  # on pull requests...
    branches: [ main ]  # against the main branch.

# What jobs does this workflow run?
jobs:
  build:  # There's a job called “build” which
    runs-on: ubuntu-latest  # runs on an Ubuntu machine
    strategy:
      matrix:  # that goes through
        python-version: ["3.8", "3.9", "3.10"]  # these Python versions.

    steps:  # The job “build” has multiple steps:
      - name: Checkout sources
        uses: actions/checkout@v2  # Checkout the repository into the runner,

      - name: Setup Python
        uses: actions/setup-python@v2  # then set up Python,
        with:
          python-version: ${{ matrix.python-version }}  # with the version that is currently “selected”...

      - name: Install dependencies
        run: |  # Then run these commands
          python -m pip install --upgrade pip
          python -m pip install tox tox-gh-actions  # install two dependencies...

      - name: Run tox
        run: tox  # and finally run tox.

Notice that, above, we installed tox and a plugin called tox-gh-actions. This plugin will make tox aware of the Python version that is set up in the GH action runner, which will allow us to specify which environments tox will run in that case. We just need to set a correspondence in the file tox.ini:

# tox.ini
# ...
[gh-actions]
python =
    3.8: py38
    3.9: py39
    3.10: py310

Integrate with Codecov for nice coverage reports in the pull requests.
- log in to Codecov with GitHub and give permissions
- add Codecov's action to the YAML from before after tox runs (it's tox that generates the local coverage report data) and add/change a coverage command to generate an xml report (it's a format that Codecov understands)

# ...
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v2
        with:
            fail_ci_if_error: true

Add a GH Action to publish to PyPI automatically
- just set up a YAML file that does your manual steps of building and publishing with Poetry when a new release is made
- create a PyPI token to be used by GitHub
- add it as a secret in your repository
- configure Poetry in the action to use that secret

name: Publish to PyPI

on:
  release:
    types: [ published ]
    branches: [ main ]
  workflow_dispatch:

jobs:
  build-and-publish:
    runs-on: ubuntu-latest

    steps:
      # Checkout and set up Python

      - name: Install poetry and dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install poetry

      - name: Configure poetry
        env:
          pypi_token: ${{ secrets.PyPI_TOKEN }}  # You set this manually as a secret in your repository
        run: poetry config pypi-token.pypi $pypi_token

      - name: Build and publish
        run: poetry publish --build

Add cool badges to your README file like
Tidy up a bit
- run linting through tox on pre-commit to deduplicate effort and run your preferred versions of the linters/formatters/...
- separate linting/formatting from testing in tox as a separate environment
- check coverage only once as a separate tox environment

How do you create a fully-fledged Python package?

Answers (1)

Preamble

Overview

Steps

Related Questions