CODEWITHSUNDEEP
pythonsubprocessvirtualenvironment
ESPeach
ESPeach

Reputation: 321

Python calling subprocess that requires no virtual environment

I have a Python 3.6 script that calls out to a third-party tool using subprocess.

main_script.py:

#!/usr/bin/env python
import subprocess
result = subprocess.run(['third-party-tool', '-arg1'], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

The problem is, main_script.py must be run from within a virtual environment, and third-party-tool must be run from no virtual environment whatsoever.

I don't know much about third-party-tool, except that it is on my path. Calling it while I have a virtual environment active causes it to jam up and throw an exception later on. I do not know if it uses the default python binary or it it spins up its own virtual env and does stuff in there. It is not a Python script, but apparently calls one somehow.

How do I tell subprocess to drop out of my virtual environment and run the command in the default shell environment?

I've examined a couple of similar questions:

Upvotes: 5

Views: 2277

Answers (2)

ESPeach
ESPeach

Reputation: 321

Thanks for your help, nullUser; your solution is a concise and correct answer to my question.

However, when I tried it out, my third-party-tool now fails for some other (unknown) reason. There was probably some other environment variable I don't know about that's getting lost with the new shell. Fortunately, I found an alternate solution which I'll share for anyone else struggling.

My Solution

As far as I can tell, the only difference that entering the virtual environment does to my environment is add a new path to my PATH variable, and add the variable VIRTUAL_ENV.

I can replicate the outside-virtual-environment behavior by creating a copy of my environment where I:

  • delete that VIRTUAL_ENV environment variable and
  • remove the python prefix from PATH.

Example

my_script.py

my_script.py Implements my solution:

#!/usr/bin/env python
import subprocess, os, sys

env = os.environ.copy()
if hasattr(sys,'real_prefix'):
    # If in virtual environment, gotta forge a copy of the environment, where we:
    # Delete the VIRTUAL_ENV variable.
    del(env['VIRTUAL_ENV'])

    # Delete the "/home/me/.python_venv/main/bin:" from the front of my PATH variable.
    orig_path = env['PATH']
    virtual_env_prefix = sys.prefix + '/bin:'
    env['PATH'] = orig_path.replace(virtual_env_prefix, '')

# Pass the environment into the third party tool, modified if and when required.
subprocess.run(['./third-party-tool'], shell=False, env=env)

third-party-tool

third-party-tool is mocked out as a script that tells you if it's in a virtual environment and prints out environment variables. In this example, third-party-tool is a Python script but in general it might not be.

#!/usr/bin/env python
# third-party-tool
import sys, os
in_venv = hasattr(sys, 'real_prefix')
print('This is third-party Tool and you {} in a virtual environment.'.format("ARE" if in_venv else "ARE NOT"))
os.system('env')

Testing

Now I try calling third-party-tool from outside virtual environment, inside virtual environment, and from the python script in the virtual environment, capturing the output.

[me@host ~]$ ./third-party-tool > without_venv.txt
# Now I activate virtual environment
(main) [me@host ~]$ ./third-party-tool > within_venv.txt
(main) [me@host ~]$ ./my_script.py > within_venv_from_python.txt

Note: the outputs looks like this: This is third-party Tool and you ARE NOT in a virtual environment. (Proceeds to pring out a list of KEY=VALUE environment variables)

I use my favorite diff tool and compare the outputs. within_venv_from_python.txt is identical to without_venv.txt, which is a good sign (in both cases, third-party-tool runs with same environment variables, and indicates it is not living in the matrix). After implementing this solution, my actual third-party-tool appears to be working.

Upvotes: 3

nullUser
nullUser

Reputation: 1833

From the documentation of subprocess:

https://docs.python.org/3/library/subprocess.html

The accepted args are

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None,
    capture_output=False, shell=False, cwd=None, timeout=None, check=False,
    encoding=None, errors=None, text=None, env=None, universal_newlines=None)

In particular,

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process’ environment. It is passed directly to Popen.

Thus, passing an empty dictionary env={} (start with empty environment) and using bash --login (run as login shell, which reads env defaults) should do the trick.

subprocess.run(['bash', '--login', '-c', '/full/path/to/third-party-tool', '-arg1'], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env={})

Upvotes: 4

Related Questions