Python venv projects randomly stop being able to find python3 binary

Question

I have a macbook pro running Catalina 10.15.6.

I have a consistent problem with python venv environments failing.

When I start a python project I run this in the project directory:

python3 -m venv .

This creates the environment. I activate the environment with . bin/activate and running which python3 gives me something like this:

/Users/qotsa42/Projects/web/project_name/bin/python3

In other words, it's using the python binary from the virtual environment. All is good.

I have a couple dozen projects on my computer created exactly this way. Sometimes, for no discernible reason and with no apparent pattern, the activation script no longer loads the python binary.

For example, I just tried to activate an environment for a project I haven't touched in a few months. which pip3 gives me this:

/Users/qotsa42/Projects/web/project_name/bin/pip3

But which python3 gives me this:

/usr/local/bin/python3

I cannot for the life of me figure out what is stopping the virtual environment from loading the right python binary.

In this particular project, I made my main.py file executable and put a bash shebang at the top pointed towards the virtual environment's binary. When I run the script, it gives me this:

/Users/qotsa42/Projects/web/project_name/bin/python3: bad interpreter: No such file or directory

Popping into the bin directory of the faulty environment and doing ls -la gives me this:

total 64
drwxr-xr-x  13 qotsa42  DOMAIN\Domain Users   416B Jul 23 13:54 ./
drwxr-xr-x  15 qotsa42  DOMAIN\Domain Users   480B Sep 16 12:41 ../
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   2.2K Jul 23 13:54 activate
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   1.3K Jul 23 13:54 activate.csh
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   2.4K Jul 23 13:54 activate.fish
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   269B Jul 23 13:54 easy_install*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   269B Jul 23 13:54 easy_install-3.7*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   260B Jul 23 13:54 pip*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   260B Jul 23 13:54 pip3*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   260B Jul 23 13:54 pip3.7*
lrwxr-xr-x   1 qotsa42  DOMAIN\Domain Users     9B Jul 23 13:54 python@ -> python3.7
lrwxr-xr-x   1 qotsa42  DOMAIN\Domain Users     9B Jul 23 13:54 python3@ -> python3.7
lrwxr-xr-x   1 qotsa42  DOMAIN\Domain Users    35B Jul 23 13:54 python3.7@ -> /usr/local/opt/python/bin/python3.7

Now here's the same ls -la command on a project where the virtual environment still functions:

total 136
drwxr-xr-x  19 qotsa42  DOMAIN\Domain Users   608B Sep 14 14:53 ./
drwxr-xr-x  13 qotsa42  DOMAIN\Domain Users   416B Sep 14 15:15 ../
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   8.6K Aug 31 12:00 Activate.ps1
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   2.2K Aug 31 12:00 activate
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   1.3K Aug 31 12:00 activate.csh
-rw-r--r--   1 qotsa42  DOMAIN\Domain Users   2.4K Aug 31 12:00 activate.fish
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   260B Sep 14 14:53 black*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   255B Sep 14 14:53 black-primer*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   261B Sep 14 14:53 blackd*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   270B Aug 31 12:00 easy_install*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   270B Aug 31 12:00 easy_install-3.8*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   256B Aug 31 12:01 f2py*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   256B Aug 31 12:01 f2py3*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   256B Aug 31 12:01 f2py3.8*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   261B Aug 31 12:01 pip*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   261B Aug 31 12:01 pip3*
-rwxr-xr-x   1 qotsa42  DOMAIN\Domain Users   261B Aug 31 12:01 pip3.8*
lrwxr-xr-x   1 qotsa42  DOMAIN\Domain Users     7B Aug 31 12:00 python@ -> python3
lrwxr-xr-x   1 qotsa42  DOMAIN\Domain Users    22B Aug 31 12:00 python3@ -> /usr/local/bin/python3

It's possible that an upgrade to the OS or Brew upgrade broke all my virtual envs and I'm just now noticing, but that seems dubious to me. I appreciate any help in advance.

ALSO: Deleting all the environment files/directories and reinstalling the virtual environment has fixed this in the past but I'd rather not deal with all that if possible.

It appears the issue is in the simlinks to the python binaries. There is no python3.7 binary in the /usr/local/opt/python/bin directory, but there is a python3.8 one.

Running /usr/local/opt/python/bin/python3.8 ./main.py works but it doesn't load the modules I've installed via pip3 -- i.e. I get ModuleNotFoundError: No module named 'pyodbc'

Running python3 -m venv --upgrade . outputs:

Error: [Errno 2] No such file or directory: '/Users/qotsa42/Projects/web/import_csv/bin/python3'

Answer 1

what does your activate script look like? It's primary purpose is to slot in the Python at the start of the PATH.

This is mine:

VIRTUAL_ENV="/Users/myuser/kds2/py2/venv38"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"

before running activate, I have this in my PATH (I used a utility to split it line by line):

0 /Users/myuser/.cargo/bin
1 /Users/myuser/.nvm/versions/node/v10.15.0/bin
2 /opt/local/bin

after activate

0 /Users/myuser/kds2/py2/venv38/bin
1 /Users/myuser/.cargo/bin
2 /Users/myuser/.nvm/versions/node/v10.15.0/bin

So, basically, all it does it prefix the PATH with $VIRTUAL_ENV/bin

I suggest you look a bit at the activate and try to see what it up to. Could anything be wrong with $VIRTUAL_ENV? It's really hardcoded to the activate location.

printf "\n$VIRTUALENV:$VIRTUALENV:\n" might give you a clue to what's goind on.

VIRTUALENV=$(fnp $BASH_SOURCE/../..) would do the trick just as welk,

Now, the next thing is what does your actual directory look like:

If I cd right above that venv, I have this tree structure:

tree -d -L 2 venv

venv
├── bin
│   └── __pycache__
├── cx_Oracle-doc
├── include
├── lib
│   └── python3.6
├── man
│   └── man1
└── share
    ├── doc
    ├── jupyter
    └── man

Now, yours may differ. 3.6 is probably a leftover in mine. But there is a directory structure and it is filled with files.

let's cd to the bin, i.e. my first PATH entry after activate-ion

and ls -l python

(venv38) myuser@bin$ ls -l python
lrwxr-xr-x  1 myuser  staff  9 May 29 13:15 python -> python3.8

that's just to make it version-neutral.

Let's ls -l python3.8

lrwxr-xr-x  1 myuser  staff  73 May 29 13:15 python3.8 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8

which is where macports has put Python. If you are using brew, it ought to be somewhere like /usr/local/ (I think).

And, yes, I can launch it:

(venv38) myuser@bin$ python3.8
Python 3.8.1 (default, Jan  5 2020, 21:32:35)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Now, any one of these may be failing, but need to work through steps to verify what's going on:

• PATH being set
• venv/bin contents
• actual Python being pointed to in venv/bin

Then you can reason about the cause of your problem.

and, if it is that 3.7 vs 3.8 item you are wondering about maybe you can fix by just rm python then ln -s /usr/local/opt/python/bin/python3.8 python.

but before you do that, take a look at the timestamp for python, the one presumably pointing at 3.7. the symlink carries the time the link was created. does that ring a bell? did you install something around that time? maybe something to do with pip?

Now, I have had Python 3.6/3.7/3.8 coexist mostly without problems. But if you can, maybe consider standardizing on 1 version unless you get your venv stabilized?