CODEWITHSUNDEEP
python-2.7pybind11openimageio
Jao
Jao

Reputation: 586

Is it possible to create a standalone file to import a python library created with pybind?

I hope I'm clear in my question, if not please tell me.

I am using OpenImageIO's python bindings (pybind11) for some scripts that will run on hundreds of computers. Unfortunately it took me a lot of time to install OpenImageIO and make it work with my Python2 installation. I'd like to know if there's a way to create a file/folder that I could send to other computers so they can install the Python module simply with "pip install file/folder".

Thanks ofr your help

Upvotes: 0

Views: 427

Answers (1)

Stuart Berg
Stuart Berg

Reputation: 18141

Are you running the scripts on a compute cluster with a shared filesystem? If so, then there's no need to create separate installations of python for each machine. The simplest solution is to create ONE python environment in a location that is accessible by all of your machines. An easy way to create a Python environment in a non-system location is to use Miniconda. Install it to a shared (network) location, and create an environment for all of your machines to use.

If your machines do NOT have a shared file system, then you'll need to somehow reproduce the environment on all of them independently. In this case, there's no simple way to do that with pip.**

But if you can use conda instead, then there's a very straightforward solution. First, install everything you need into a single conda environment. Then you have a choice: You can export the list of conda packages, or simply copy the entire conda environment directory to the other machines.

OpenImageIO is available from the conda-forge channel, a community-developed repository of conda packages. The name of the package is py-openimageio. They have stopped updating the python-2.7 version, but the old versions are still available.

Here's how to do it.

  1. Install Miniconda-2.7
  1. Create a new environment with python 2.7, OpenImageIO, and any other packages you need:
conda create -n jao-stuff -c conda-forge py-openimageio python=2.7
conda activate jao-stuff
python -c "import OpenImageIO; print('It works!')"

  1. Do ONE of the following:

    a. Export the list of packages in your environment:

    conda env export -n jao-stuff -f jao-stuff-packages.yaml

    Then, on the other machines, install Miniconda, then create the environments using the package list from the previous step:

    conda create -n jao-stuff --file jao-stuff-packages.yaml

    OR

    b. Just copy all of the files in the environment to the other machines, and run them directly. Conda environments are self-contained (except for a few low-level system libraries), so you can usually just copy the whole thing to another machine and run it without any further install step.

    tar czf jao-stuff.tar.gz $(conda info --prefix)/envs/jao-stuff

    On the other machine, unpack the tarball anywhere and just run the python executable it contains:

    tar xzf jao-stuff.tar.gz
jao-stuff/bin/python -c "import OpenImageIO; print('It works!')"

**That's because OpenImageIO is a C++ project, with several C++ dependencies, and they don't provide binaries in the wheel format. I don't blame them -- pip is not well suited to this use-case, even with wheels. Conda, on the other hand, was designed for exactly this use-case, and works perfectly for it.

Upvotes: 1

Related Questions