Issue with pip and virtualenv when specifying python version

Question

I'm using virtualenv (1.11.6) to manage my python projects, I created a new one but I noticed that it uses an old python version (2.7.2 instead of 2.7.8), so I recreated a virtualenv by using the -p option:

virtualenv -p /opt/local/bin/python ENV

Now the project uses python 2.7.8, but if I try to install any python library using pip 1.5.6 (scrapy for example) I get

Command python setup.py egg_info failed with error code 1

This does not happen if I create a virtualenv without specifying the python version... what could be the reason of this behavior? How can I create a virtualenv using the python version of my choice, without issues?

ps: I'm using OS X and I've installed python 2.7.8 using macports

EDIT: the problem is somehow related to the cryptography lib which scrapy depends on:

running build_ext

building '_Cryptography_cffi_4ed9e37dx4000d087' extension

creating /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography

creating /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat

creating /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings

creating /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__

/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -Os -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.c -o /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.o

/usr/bin/clang -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names -L/opt/local/lib/db46 /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.o -lcrypto -lssl -o /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so

Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/setup.py", line 174, in <module>

    "test": PyTest,

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup

    dist.run_commands()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "<string>", line 15, in replacement_run

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 186, in find_sources

    mm.run()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 246, in run

    self.add_defaults()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 282, in add_defaults

    sdist.add_defaults(self)

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/lib/python2.7/site-packages/setuptools/command/sdist.py", line 167, in add_defaults

    build_py = self.get_finalized_command('build_py')

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command

    cmd_obj.ensure_finalized()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/lib/python2.7/site-packages/setuptools/command/build_py.py", line 26, in finalize_options

    orig.build_py.finalize_options(self)

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_py.py", line 46, in finalize_options

    ('force', 'force'))

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 298, in set_undefined_options

    src_cmd_obj.ensure_finalized()

  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized

    self.finalize_options()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/setup.py", line 88, in finalize_options

    self.distribution.ext_modules = get_ext_modules()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/setup.py", line 68, in get_ext_modules

    OpenSSLBinding().ffi.verifier.get_extension(),

  File "cryptography/hazmat/bindings/openssl/binding.py", line 89, in __init__

    self._ensure_ffi_initialized()

  File "cryptography/hazmat/bindings/openssl/binding.py", line 109, in _ensure_ffi_initialized

    libraries=libraries,

  File "cryptography/hazmat/bindings/utils.py", line 80, in build_ffi

    extra_link_args=extra_link_args,

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cffi-0.8.6-py2.7-macosx-10.8-x86_64.egg/cffi/api.py", line 340, in verify

    lib = self.verifier.load_library()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cffi-0.8.6-py2.7-macosx-10.8-x86_64.egg/cffi/verifier.py", line 75, in load_library

    return self._load_library()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cffi-0.8.6-py2.7-macosx-10.8-x86_64.egg/cffi/verifier.py", line 151, in _load_library

    return self._vengine.load_library()

  File "/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cffi-0.8.6-py2.7-macosx-10.8-x86_64.egg/cffi/vengine_cpy.py", line 149, in load_library

    raise ffiplatform.VerificationError(error)

cffi.ffiplatform.VerificationError: importing '/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so': dlopen(/Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so, 2): Symbol not found: _CRYPTO_malloc_debug_init

  Referenced from: /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so

  Expected in: flat namespace

 in /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography/cryptography/hazmat/bindings/__pycache__/_Cryptography_cffi_4ed9e37dx4000d087.so

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /Users/davidezanotti/PycharmProjects/scrapy_test/ENV/build/cryptography
Storing debug log for failure in /Users/davidezanotti/.pip/pip.log

it seems like I miss some dev tools or similar... but I don't understand why I can install it successfully if I create an environment without specifying the python version :/

Answer 1

Problems with virtualenv and python versions are sometimes linked to libraries, that python packages have in dependencies.

This post describes, how to solve problem with cryptography dependency.