How to handle `pip install` inside a pipenv environment (not reflected in the Pipfile, only in the `pipenv graph`)?

Question

If someone accidentally uses pip install instead of pipenv install inside a pipenv environment, that package does not get reflected in the list of packages on the Pipfile nor in the Pipfile.lock.

The problem with that is you might go to deployment with this Pipfile.lock thinking you have everything you need when actually you have a missing package.

I was looking through the documentation https://pipenv.pypa.io/ to find out what actually happens when you run pip install instead of pipenv install (even by mistake) and I couldn't find an explanation for this.

If you run pipenv graph it actually shows you the packages installed via pip! So I know pipenv is somehow aware of these packages. But what do I need to do to make those reflect in the Pipfile?

Answer 1

First, let's clarify that the pipenv install command is just a wrapper for pip. If you install with --verbose, you'll see that it's also just using pip install and putting the packages in the same, activated virtual environment. So the answer to

I was looking through the documentation https://pipenv.pypa.io/ to find out what actually happens when you run pip install instead of pipenv install (even by mistake)

is simply pipenv-specific operations will not be done. This includes updating the Pipfile and the Pipfile.lock (which is one of the main reasons to use pipenv in the first place) to have deterministic builds. The Pipfile you can update yourself manually, but for Pipfile.lock...you can't.

If you run pipenv graph it actually shows you the packages installed via pip!

Yes, because as I said, they both just use pip. Both methods will install the packages in the same virtual environment and pipenv graph is just checking that same env. The packages will be stored in a folder under lib/pythonX.Y/site-packages, whether with pipenv or plain pip.

Now on to your actual question:

But what do I need to do to make those reflect in the Pipfile?

D Malan's comment of using pipenv clean is a good approach. From the docs:

$ pipenv clean --help
Usage: pipenv clean [OPTIONS]

  Uninstalls all packages not specified in Pipfile.lock.

Options:
  --bare           Minimal output.
  --dry-run        Just output unneeded packages.
  ...

As described, you just need to run that command to check for inconsistencies. Add the --dry-run command so that it only reports, not actually uninstalls them.

You can then make a script for that, like this Bash script:

#!/usr/local/bin/bash

echo "Checking if there are packages in venv not in Pipfile.lock"

# Get packages pipenv did not find in Pipfile.lock
# NOTE:
#   Here, mapfile requires Bash 4.x syntax
#   For alternatives: https://stackoverflow.com/a/32931403/2745495
mapfile -t packages < <(pipenv clean --dry-run)

if [ ${#packages[@]} -eq 0 ]; then
    echo "All good!"
else
    echo "Found ${#packages[@]} not in Pipfile.lock!"
    for pkg in "${packages[@]}"; do
        echo "  ${pkg}"
    done
    echo ""
    echo "Check if they need to be 'pipenv install'-ed or deleted with 'pipenv clean'"

    # Make sure script exits with a non-zero code here
    exit 1
fi

Running it on a test env with both a pip install-ed package (ex. mypy) and a pipenv install-ed package (ex. flake8):

(my-test-repo) $ pipenv graph
flake8==3.8.4
  - mccabe [required: >=0.6.0,<0.7.0, installed: 0.6.1]
  - pycodestyle [required: >=2.6.0a1,<2.7.0, installed: 2.6.0]
  - pyflakes [required: >=2.2.0,<2.3.0, installed: 2.2.0]
mypy==0.790
  - mypy-extensions [required: >=0.4.3,<0.5.0, installed: 0.4.3]
  - typed-ast [required: >=1.4.0,<1.5.0, installed: 1.4.1]
  - typing-extensions [required: >=3.7.4, installed: 3.7.4.3]

(my-test-repo) $ cat Pipfile.lock | grep mypy

(my-test-repo) $ ./check.sh 
Checking if there are packages in venv not in Pipfile.lock
Found 4 not in Pipfile.lock!
  typing-extensions
  typed-ast
  mypy
  mypy-extensions

Check if they need to be 'pipenv install'-ed or deleted with 'pipenv clean'

(my-test-repo) $ pipenv install mypy
...
✔ Success! 
Updated Pipfile.lock (e60379)!
Installing dependencies from Pipfile.lock (e60379)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:

(my-test-repo) $ ./check.sh 
Checking if there are packages in venv not in Pipfile.lock
All good!

To solve the problem of

you might go to deployment with this Pipfile.lock thinking you have everything you need when actually you have a missing package.

If you are using Git, make the script part of your git pre-commit hook.

The pre-commit hook is run first, before you even type in a commit message. It’s used to inspect the snapshot that’s about to be committed, to see if you’ve forgotten something, to make sure tests run, or to examine whatever you need to inspect in the code. Exiting non-zero from this hook aborts the commit, although you can bypass it with git commit --no-verify.

(my-test-repo) $ cp check.sh .git/hooks/pre-commit
(my-test-repo) $ chmod +x .git/hooks/pre-commit

(my-test-repo) $ git add .
(my-test-repo) $ git commit
Checking if there are packages in venv not in Pipfile.lock
Found 4 not in Pipfile.lock!
  typing-extensions
  mypy
  mypy-extensions
  typed-ast

Check if they need to be 'pipenv install'-ed or deleted with 'pipenv clean'
(my-test-repo) $

The commit is aborted when inconsistencies are found, preventing you from committing a possibly incomplete Pipfile.lock.