CODEWITHSUNDEEP
pythonubuntupipvirtualenv
NeuronQ
NeuronQ

Reputation: 8195

Mix Python virtualenv packages with distro packages?

What is a good way to handle the case when you use Python virtualenv but you want some of the packages installed via your distro's package manager?

Let's say you need lxml but because you can't get pip install lxml to work on Ubuntu. and you really don't want to waste time on this, so you just do a apt-get install python-lxml.

Now, you can create a virtaulenv with --system-site-packages and have access to the system-wide installed pre-compiled lxml now. But you'll also drag in all the other system wide packages that you don't need! And yes, there will be quite a bunch of packages that will be installed outside virtualenv, either via sudo pip ... or sudo apt-get python-..., so no "just keep the system clean and install everything you can in virtualenvs, so that --system-site-packages won't drag too many packages with it" is not a solution form me here.

So, is there a way to install just some particular system-site-packages?

Upvotes: 4

Views: 720

Answers (1)

selurvedu
selurvedu

Reputation: 501

I always use pip install --user packagename – it doesn't require sudo.

If you don't have pip yet, or either OS's pip or OS's easy_install doesn't work properly, first install setuptools in user's home directory, and then use its easy_install for pip.

$ wget https://bootstrap.pypa.io/ez_setup.py -O - | python3 - --user
$ easy_install-3.4 --user pip

On the other side, if you decide to use pip inside virtualenv, don't use --user option for an obvious reason.

Doing it the way you described

For example, I want to import module curl that's present in OS.

$ pip3 install --user virtualenv
$ python3 -m virtualenv myvenv
$ cd myvenv
$ source bin/activate
(myvenv)$ python3
>>> import curl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named curl

There is no such module inside virtualenv. Just to make sure I looked into what's in sys.path.

>>> import sys
>>> sys.path
['',
'/home/username/myvenv/lib/python27.zip',
'/home/username/myvenv/lib/python2.7',
'/home/username/myvenv/lib/python2.7/lib-dynload',
'/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2',
'/home/username/myvenv/lib/python2.7/site-packages']

Notice that there is no /usr/lib/python3.4/site-packages, just as you wanted, but /usr/lib/python3.4 is present. If that's not okay for you, use --always-copy option when creating a virtualenv.

In my further steps I create symbolic links to curl module and its dependencies.

>>> ^D # Press Control-D
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/curl lib/python3.4/site-packages/

Notice that there is no trailing slash in the second argument (file or directory which symlink points to), but there is a slash in the third argument (where symlink should be created). That's because I create a symlink inside of the directory lib/python3.4/site-packages. Without a trailing slash, it would try to replace the directory.

Let's check if that works.

(myvenv)$ python3
>>> import curl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/username/myvenv/lib/python3.4/site-packages/curl/__init__.py", line 9, in <module>
    import sys, pycurl
ImportError: No module named 'pycurl'

Nope. curl depends on pycurl. We need to go deeper.

>>> ^D
(myvenv)$ ls -1 /usr/lib/python3.4/site-packages/pycurl*
/usr/lib/python3.4/site-packages/pycurl-7.19.3.1-py3.4.egg-info
/usr/lib/python3.4/site-packages/pycurl.cpython-34m.so
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/pycurl-7.19.3.1-py3.4.egg-info lib/python3.4/site-packages/
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/pycurl.cpython-34m.so lib/python3.4/site-packages/
(myvenv)$ python3
>>> import curl
>>> curl
<module 'curl' from '/home/username/myvenv/lib/python3.4/site-packages/curl/__init__.py'>

At last.

Is this what you need?

Personally, I strongly discourage you from doing this because, firstly, this is a perversion.

Moreover, I doubt this method works on everything. The depth of dependencies can be scary.

Secondly, I see no reason in avoiding --system-site-packages that works like a charm in all cases I'm acknowledged about.

Packages installed by pip in user's home directory have higher priority than system-wide packages, thus, they are taken first when doing import. In virtualenv, they are only imported if virtualenv was created with --system-site-packages option. Packages installed by pip inside virtualenv have even higher priority.

Honestly, I wrote this answer just to ask, why would you want to mess with all this instead of getting pip to work. I can't yet write comments on SO, so that's why I wrote a complete answer. :)

But you'll also drag in all the other system wide packages that you don't need!

No, you won't. You load a module into memory only when doing import. If you mean the package will be present in virtualenv – once again, it won't. If you had looked inside of myvenv/lib/python3.4/site-packages/ you would have seen that there are only two packages inside - pip and setuptools (the latter one is separated into several directories though). The Python interpreter inside virtualenv simply loads system-wide modules just like an ordinary interpreter does.

P.S. The code above also applies to Python 2.7. Just replace pip3 with pip2, python3 with python2, 3.4 with 2.7, etc.

P.P.S. Instead of symlinking, you can copy the modules with their dependencies into virtualenv to the corresponding directories. This makes sense when using --always-copy, so that your virtualenv becomes portable.

P.P.P.S. If you decide to use virtualenv with --system-site-packages and pip, and, let's say, you want to have requests package newer than the already installed one in /usr/lib, use pip install -I, as described here.

Upvotes: 1

Related Questions