How to install and import Python modules at runtime

Question

I want to write a script to automatically setup a brand new ubuntu installation and install a django-based app. Since the script will be run on a new server, the Python script needs to automatically install some required modules.

Here is the script.

#!/usr/bin/env python

import subprocess
import os
import sys

def pip_install(mod):
    print subprocess.check_output("pip install %s" % mod, shell=True)

if __name__ == "__main__":
    if os.getuid() != 0:
        print "Sorry, you need to run the script as root."
        sys.exit()

    try:
        import pexpect
    except:
        pip_install('pexpect') 
        import pexpect        

    # More code here...

The installation of pexpect is success, however the next line import pexpect is failed. I think its because at runtime the code doesn't aware about the newly installed pexpect.

How to install and import Python modules at runtime? I'm open to another approaches.

Answer 1

I had the same issue but none of Google's searches helped. After hours debugging, I found that it may be because the sys.path is not reloaded with new installation directory.

In my case on my Ubuntu Docker, I want to import dns.resolver at runtime for Python3.8 (pre-installed). I also created ubuntu user and run all things with this user (including my Python script).

• Before installing, sys.path doesn't have /home/ubuntu/.local/lib/python3.8/site-packages since I didn't install anything.
• While installing with subprocess or pip.main like above, it creates /home/ubuntu/.local/lib/python3.8/site-packages (as user installation).
• After installing , the sys.path should be refreshed to include this new location.

Since the sys.path is managed by site module, we should reload it (ref HERE):

    import site
    from importlib import reload
    reload(site)

The full block for anyone that needs:

import subprocess
import sys

try:
    import dns.resolver
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "dnspython"])
    import site
    from importlib import reload
    reload(site)
    import dns.resolver

I'm not Python developer so these code can be simplify more. This may help in cases such as fresh CI/CD environment for DevOps engineers like me.

Answer 2

For those who are using pip version greater than 10.x, there is no main function for pip so the alternative approach is using import pip._internal as pip instead of import pip like :

Updated answer of Paulo

import pip._internal as pip

def install(package):
    pip.main(['install', package])

if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Answer 3

I actually made a module for this exact purpose (impstall)

It's really easy to use:

import impstall
impstall.now('pexpect')
impstall.now('wx', pipName='wxPython')

Github link for issues/contributions

Answer 4

I solved my problem using the imp module.

#!/usr/bin/env python

import pip
import imp

def install_and_load(package):
    pip.main(['install', package])

    path = '/usr/local/lib/python2.7/dist-packages'
    if path not in sys.path:
        sys.path.append(path)

    f, fname, desc = imp.find_module(package)
    return imp.load(package, f, fname, desc)

if __name__ == "__main__":
    try:
        import pexpect
    except:
        pexpect = install_and_load('pexpect')

    # More code...

Actually the code is less than ideal, since I need to hardcode the Python module directory. But since the script is intended for a known target system, I think that is ok.

Answer 5

You can import pip instead of using subprocess:

import pip

def install(package):
    pip.main(['install', package])

# Example
if __name__ == '__main__':
    try:
        import pexpect
    except ImportError:
        install('pexpect')
        import pexpect

Another take:

import pip

def import_with_auto_install(package):
    try:
        return __import__(package)
    except ImportError:
        pip.main(['install', package])
    return __import__(package)

# Example
if __name__ == '__main__':
    pexpect = import_with_auto_install('pexpect')
    print(pexpect)

[edit]

You should consider using a requirements.txt along with pip. Seems like you are trying to automate deployments (and this is good!), in my tool belt I have also virtualenvwrapper, vagrant and ansible.

This is the output for me:

(test)root@vagrant:~/test# pip uninstall pexpect
Uninstalling pexpect:
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/ANSI.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/FSM.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/fdpexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect-2.4-py2.6.egg-info
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/pxssh.pyc
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.py
  /usr/lib/python-environments/test/lib/python2.6/site-packages/screen.pyc
Proceed (y/n)? y
  Successfully uninstalled pexpect
(test)root@vagrant:~/test# python test.py
Downloading/unpacking pexpect
  Downloading pexpect-2.4.tar.gz (113Kb): 113Kb downloaded
  Running setup.py egg_info for package pexpect
Installing collected packages: pexpect
  Running setup.py install for pexpect
Successfully installed pexpect
Cleaning up...
<module 'pexpect' from '/usr/lib/python-environments/test/lib/python2.6/site-packages/pexpect.pyc'>
(test)root@vagrant:~/test#