Install the latest version of my package from working directory into my local environment using Python's poetry

Question

It's extremely useful for the development workflow to be able to build and install the latest version of a package into a local environment. You can then interactively validate and debug by importing this latest version into a Python shell or a Jupyter notebook. The problem is I've recently adopted Poetry and cannot figure how to do this now. So...

How do I install the latest version of my package from the current working directory into my local environment using Poetry?

Moving on from setuptools

Back in the day, I used to always use setuptools and it worked great. I'd put a setup.py file in the root of my repository, create a virtual environment (let's say using conda) for the project, and do...

pip install -e .

From here, I could fire up a python shell, or even configure a jupyter kernel to use this virtual environment, and I'd always have the latest version of my package to interact with.

Now setuptools has its limitations, and we've since moved on to Poetry to more tightly control dependencies and handle more sophisticated build needs and such.

The problem with poetry

If you look up what's the pip install -e . equivalent in poetry you will find this issue. Looks like the creator of poetry thinks installing directly from source like this is a hack and has no interest in supporting it. (BTW: I've tried poetry build and then pulling out the setup.py file like he suggested and it does not work)

Linking directly to source is not necessary, I'm willing to run an install command to get the latest version of the package. And when I do this with poetry, it appears to work.

cd root/of/my/project
poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: my-project (0.4.8) <-- this is the latest version according to the source code in the working directory

The problem is that if I open a Python shell and try and import my package for instance, it is linked to the last version of my package that installed from a remote artifact repository (via pip install my-package) – not what's in my working directory.

python
...
>>> import my_package
>>> my_package.__version__
'0.4.7`

Now, even though I'm using poetry I'm using a conda environment to specify the Python version for my project and then installing, and using, poetry from inside that.

source activate my-package
(my-package) ... $ poetry update

I also know that poetry (not very transparently) can create and manage its own virtual environment on your behalf. I thought maybe the reason this is not working is because I need to be inside this environment (whereas I was only inside my conda environment, while poetry is installing the 0.4.8 version of my package within the virtual environment it manages).

I tried both shell and run to test this out. I get the same result.

poetry shell
Virtual environment already activated: /Users/U6020643/.conda/envs/my-package

Python 3.8.5
...
>>> import my_package
>>> my_package.__version__
'0.4.7`

What gives?

Answer 1

The way I fixed this: I stopped using conda to manage the Python version for projects that involve poetry and instead use pyenv.

Below is how I made that work. This was very helpful!

1. Removing conda as default environment manager.

This involved removing conda base activate from ~/.bash_profile.

Now open a new shell and verify that there's no conda environment prefix, e.g. (base) ... $.

2. Installing pyenv using Homebrew

Had been awhile, and an OS upgrade or two, since I interacted with Homebrew. Needed to do some housekeeping.

brew cleanup # This made it so that brew update didn't take forever
brew update
brew upgrade
brew cleanup

Then...

brew install pyenv

3. Install Python version(s)

Let's say you need/want Python 3.

pyenv install 3.8.5

4. Set the local Python version for your project

cd your/project/root/
pyenv local 3.8.5

5. Install poetry

See here.

6. Use it

Now install and use shell. Check the version -> hey it works!

cd your/project/root
poetry install
poetry shell
my_package --version  # Package has a CLI.