Calling OS' Python Executable from another Python Process

Question

I am using subprocess.Popen to call an external Python script from my installed Application, bundled using PyInstaller. The format of this command is something like this:

subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"],
                 stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                 stderr=subprocess.PIPE, shell=False)

I have successfully tested this on both Windows and MacOS - both can run the external script. However, on other Posix OS' I get the following error:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
ImportError: No module named site

Now, I realize that this question has been asked before, and generally it is down to PYTHONHOME having an incorrect value. However, if I run the command in a format similar to:

PYTHONHOME=/usr /usr/bin/python /path/to/exe/SDK.py

I get no errors in the logs, but the script SDK.py doesn't execute.

Therefore to prove that this isn't an issue with my script; I installed my own version of Python onto the machine - after doing this, the script executed successfully. I tested with both /usr/bin/python and /home/vagrant/Python-2.7.15/python, without needing to specify a PYTHONHOME in the subprocess command.

However, I still need to allow users to execute the SDK.py script using the built in, OS version of Python.

So to do this I have tried these other things:

1. Using sys.executable in the subprocess.Popen call to execute SDK.py.

(The value of which was: /opt/program_name/lib/program_name)

2. Forcing PYTHONPATH & PYTHONHOME to be empty by explicitly setting the environment:

   import os
   env = os.environ.copy()
   env['PYTHONHOME'] = ''
   env['PYTHONPATH'] = ''
   subprocess.Popen(["/usr/bin/python", "/path/to/exe/SDK.py"], env=env)
   

Can anyone explain how I can call the system version of Python from an external subprocess like this?

EDIT: Output when calling Python in verbose mode (from the CLI):

>>> from sys import executable
>>> executable
'/usr/bin/python'
>>> import _csv
# trying _csv.so
# trying _csvmodule.so
# trying _csv.py
# trying _csv.pyc
# trying /usr/lib64/python2.7/_csv.so
# trying /usr/lib64/python2.7/_csvmodule.so
# trying /usr/lib64/python2.7/_csv.py
# trying /usr/lib64/python2.7/_csv.pyc
# trying /usr/lib64/python2.7/plat-linux2/_csv.so
# trying /usr/lib64/python2.7/plat-linux2/_csvmodule.so
# trying /usr/lib64/python2.7/plat-linux2/_csv.py
# trying /usr/lib64/python2.7/plat-linux2/_csv.pyc
# trying /usr/lib64/python2.7/lib-dynload/_csv.so
dlopen("/usr/lib64/python2.7/lib-dynload/_csv.so", 2);
import _csv # dynamically loaded from /usr/lib64/python2.7/lib-dynload/_csv.so

Answer 1

This is not a full answer, but for the sake of completeness I will document here the two things that worked - and which approach we went with in the end.

1. For some strange reason, when my colleague did this:

env = os.environ.copy() env['LD_LIBRARY_PATH'] = env['PATH']

It allowed us to call the subprocess when we passed this env into the Popen() call like so:

self.process = subprocess.Popen(sdk_executable_command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False, env=env)

We did not investigate this further because we went with the second solution described below. However, it would lead me to believe that these issues may have been down to the predefined setting of LD_LIBRARY_PATH.

2. When we created a virtualenv for python using the pip package of the same name, and we called the SDK using this python executable - the SDK would successfully run. Reason for this is still unknown, but because this is an SDK: we decided it was appropriate for the eventual SDK Developer to run this in an environment separate from the 'out of the box' Python in Linux.

Answer 2

I'll try and walk you trough the process of checking you Python installation on CentOS 7.

First of all, open some terminal window and verify the installation of the python RPMs:

$ rpm -V python python-libs

If you get no output, everything is fine. Next check the environment variables:

$ env | grep PYTHON

There should be no output as well. If there is some mention of PYTHONHOME or PYTHONPATH, unset them. Now check what Python thinks about its default path:

$ /usr/bin/python
Python 2.7.5 (default, Jul 13 2018, 13:06:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from sys import path
>>> print path
['', '/usr/lib64/python27.zip', '/usr/lib64/python2.7', '/usr/lib64/python2.7/plat-linux2', '/usr/lib64/python2.7/lib-tk', '/usr/lib64/python2.7/lib-old', '/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages', '/usr/lib64/python2.7/site-packages/gtk-2.0', '/usr/lib/python2.7/site-packages']

You should also be able to import the site module without any problems:

>>> import site
>>> print site
<module 'site' from '/usr/lib64/python2.7/site.pyc'>

If everything is OK, try to run your script from the very same command line where you performed the tests.