CODEWITHSUNDEEP
pythondockerpypipython-poetryaws-codeartifact
Khan
Khan

Reputation: 1518

Poetry project lock file and Docker container installation

I have a Python project with Poetry which i would like to upload to CodeArtifact (like any other Python Repository) and then install it into a Docker container. My Python project has dependencies in the pyproject.toml like

[tool.poetry.dependencies]
pandas = "^2.1.4"
pytest-mock = "^3.12.0"

the ^ means that even higher versions are accepted. But as soon as I have my project working I also have a poetry.lock file. This prevents higher versions from being installed with poetry install. Now when everything is working, I run poetry build and upload the lib to CodeArtifact.

If I want to install this library into a Docker container, I could run pip install and specify the version of my lib.

My questions are now:

Upvotes: 1

Views: 1446

Answers (1)

FlightPlan
FlightPlan

Reputation: 742

If you've run poetry build to create a package and uploaded that package to codeartifact, that package is now ready to be installed by a package manager (eg pip install or poetry add), regardless of whether the "caller" is in a container.

Making your container dependencies mirror your local project

If you want to make your dependency versions follow the poetry.lock and exactly mirror those in your local project, you can install Poetry in your container, copy in the poetry.lock from your local project, and do poetry install in the container.

Here's an example (several, actually) of how this is done.

How dependency resolution works

Dependency management tools like Pip and Poetry will look at the dependencies specified in the pyproject.toml of each of the dependencies in your project, compare them with each other, and determine which dependency versions will satisfy the project (and dependency requirements) without causing conflicts.

In your case, this means that the version of Pandas that gets installed will change based on your environment and the other dependencies in the project.

A concrete example:

  • You're in project Foo. You do poetry build, making Foo an installable package
  • Foo package has within it a pyproject.toml, that specifies its dependency requirements
  • A different project: "Bar" now installs Foo via Poetry or Pip (etc). Let's say we did pip install foo

The behavior is as follows:

  • Pip looks at the pyproject.toml files in each of Bar's dependency packages (and their subdependency packages)
  • Pip determine what "maximum version" of each subdependency can be installed and still:
    1. Meet all package requirements
    2. Not create conflicts

These dependency managers are "smart" in that they will upgrade dependencies only as high as the dependencies will support.

For example:

  • You install package bar that relies on mydep = <=2.0.5
  • Now you install package qux that relies on mydep = <=2.1.0
  • Mydep will be installed, but only as far as version 2.0.5.

How to prevent higher versions

IMPORTANT: Pip and Poetry respect semantic versioning. Semantic versioning highlights breaking changes to libraries (changes in public API, etc), so dependency managers know "just how far they can upgrade" without breaking your code. It is not recommended to manually edit versions requirements - this is what dependency managers like Poetry are for!

Now with that caveat, you can use "less than or equals" notation in your requirements files (typically pyproject.toml) to "lock" a dependency at a particular version or set a cap. Eg.mydep = <=2.0.5 will go no higher than 2.0.5 and mydep = =2.0.5 will be exactly 2.0.5. Then if you have poetry for example, you'd run poetry update mydep to lock that change into your poetry.lock file.

Upvotes: 2

Related Questions