CODEWITHSUNDEEP
python
cerhovice
cerhovice

Reputation: 686

how to control the import paths in Python?

I am new to Python. I am at a company where they built a large system in Python. They use a proprietary system to manage the paths when the system is running, but now I have been asked to build a standalone script that interacts with some of the code in their system. Sadly, my standalone script won't be running under the path-manager they use, so I need to figure out the paths on my own.

So, for instance, I have this line:

from hark.tasks import REPLY_LINE

This is actually copied from some of their older code. In this case, the script can find hark, but hark has an __init__.py file, and that is where the problems start. So I get this:

meg/src/python2/hark/hark/__init__.py in <module>()
      5 from flask import jsonify, render_template, request
      6 import jinja2
----> 7 import logbook, logbook.compat
      8 
      9 from healthhark.context import Ghost, g

The project that they built actually includes logbook 3 times. If I do:

find . -name "*logbook*"

I see: 

meg/zurge/opt/python2.7/lib/python2.7/site-packages/logbook
meg/zurge/opt/python2.7-hark/lib/python2.7/site-packages/logbook
meg/zurge/opt/python3.4/lib/python3.4/site-packages/logbook

Like I said, they have a proprietary path manager that usually tells each piece of code where it can find the packages that it should include, but I am building a standalone app.

I don't know much about Python, but I am wondering if their is an idiomatic and Pythonic way of including packages that are in such distant directories?

And, before anyone suggests pip install, we don't rely on global installs at all.

Upvotes: 4

Views: 774

Answers (1)

Matthew Franglen
Matthew Franglen

Reputation: 4532

The best solution would probably be virtualenv or virtualenvwrapper. This would allow you to define an environment which contains all of the libraries that your script requires. This would not be global.

This can be done as follows:

  • Create a requirements.txt file defining the required libraries
  • Install pip and virtualenvwrapper
  • source /usr/local/bin/virtualenvwrapper.sh
  • workon hark_task_script_env || ( mkvirtualenv hark_task_script_env && pip install -r requirements.txt )
  • python your-script.py

The virtualenvwrapper installs the environment into a folder in your home directory. The plain virtualenv library installs the environment into a folder in the project. Other than that they are equivalent.

I would really recommend using them over a proprietry package manager. If you must use the proprietry package manager then it is reasonable to have access to the package loader for it!

If this really isn't satisfactory then you can hack a package loader as follows (assuming you use a *nix system). This is a shell script written in zsh. Hopefully it is clear enough for you to rewrite it in a supported shell if your system does not have that available:

#!/bin/zsh

setopt extended_glob

function find_package_init_files () {
    locate __init__.py
}

# Get the containing folder of a file or a folder
function file_or_folder_to_parent_folder () {
    while read file_or_folder
    do
        echo ${file_or_folder:h}
    done
}

# Exclude folders where the parent folder also has an __init__.py file in it
function exclude_inner_packages () {
    while read init_file_folder
    do
        init_file_parent_folder=${init_file_folder:h}
        if [ ! -e ${init_file_parent_folder}/__init__.py ]
        then
            echo ${init_file_folder}
        fi
    done
}

# This produces an array of all folders
# that contain at least one python package
function get_distinct_python_package_folders () {
    find_package_init_files |
        file_or_folder_to_parent_folder |
        exclude_inner_packages |
        file_or_folder_to_parent_folder |
        sort |
        uniq
}

PYTHONPATH=${(j/:/)$(get_distinct_python_package_folders)} YOUR_SCRIPT_HERE

You may well need to update this script to put the default python path first, and remember that this is an incredibly clumsy approach. If there are multiple versions of libraries installed on a system then the one that you may end up using will be ill defined.

Using a proper package manager, even if that is the proprietry one, is the best way.

Upvotes: 1

Related Questions