scipy on os x lion

Question

I'm am trying to get my python/numpy/scipy working environment running on OS X Lion.

I've already managed to compile numpy and scipy from their latest sourcecode versions with the usual

python setup.py build
python setup.py install

I do run the Python 2.7.2 version downloaded from python.org:

tobi-mbp:~ tobi$ python
Python 2.7.2 (v2.7.2:8527427914a2, Jun 11 2011, 15:22:34) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

If i than import numpy it shows:

>>> import numpy
>>> numpy.__version__
'1.5.1'
>>> numpy.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/__init__.pyc'
>>> 

the system instalation of numpy. Whereas

>>> import scipy
>>> scipy.__version__
'0.10.0.dev'
>>> scipy.__file__
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/scipy/__init__.pyc'
>>>

gives the installed version of scipy. That means the scipy.test() fails and actually gets stuck.

I did try pip and easy_install as well but the fail to compile scipy and with numpy the same strange bahavior occours. Same for the precompiled binarys provided on sourcefourge.

As this seems to be an issue with PYTHONPATH:

>>> import sys, os, pprint
>>> pprint.pprint(sys.path)
['',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/distribute-0.6.19-py2.7.egg',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/Library/Python/2.7/site-packages/nose-1.1.2-py2.7.egg',
 '/Library/Python/2.7/site-packages/pip-1.0.2-py2.7.egg',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
 '/Library/Python/2.7/site-packages']
>>> pprint.pprint([p for p in os.environ.items() if p[0].startswith('PYTHON')])
[]
>>> 

And a list of the contents of site-packages

tobi-mbp:site-packages tobi$ ls -l
total 752
-rw-rw-r--   1 root  admin     119 12 Jun 00:25 README
drwxr-xr-x  10 root  admin     340  6 Aug 19:47 distribute-0.6.19-py2.7.egg
-rw-r--r--   1 tobi  admin     237  6 Aug 19:48 easy-install.pth
drwxrwxr-x  48 tobi  admin    1632  6 Aug 18:19 numpy
drwxrwxr-x   6 tobi  admin     204 20 Jul 20:44 numpy-1.6.1-py2.7.egg-info
-rw-r--r--   1 root  admin    1670  6 Aug 18:19 numpy-2.0.0.dev_26aa3cf-py2.7.egg-info
drwxr-xr-x   4 root  admin     136  6 Aug 19:48 pip-1.0.2-py2.7.egg
drwxr-xr-x   8 tobi  admin     272  6 Aug 19:48 readline-6.2.0-py2.7.egg-info
-rwxr-xr-x   1 tobi  admin  357048  6 Aug 19:48 readline.so
drwxrwxr-x  42 tobi  admin    1428  6 Aug 18:21 scipy
-rw-r--r--   1 root  admin    1768  6 Aug 18:21 scipy-0.10.0.dev-py2.7.egg-info
drwxrwxr-x   6 tobi  admin     204 27 Feb 14:00 scipy-0.9.0-py2.7.egg-info
-rw-r--r--   1 tobi  admin     144  6 Aug 19:47 setuptools-0.6c11-py2.7.egg-info
-rw-r--r--   1 tobi  admin      30  6 Aug 19:47 setuptools.pth

tobi-mbp:site-packages tobi$ more easy-install.pth
import sys; sys.__plen = len(sys.path)
./distribute-0.6.19-py2.7.egg
./pip-1.0.2-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

tobi-mbp:site-packages tobi$ more setuptools.pth 
./distribute-0.6.19-py2.7.egg

I do see the System Extras folder in the path, now the question is why and how I can get rid of it.

thanks for your help

cheers Tobi

Answer 1

There appears to be a path issue somewhere. Perhaps you are setting PYTHONPATH? When you are running the python.org 2.7.2, the Apple-supplied Python 2.7.1's Extras directory should not be on sys.path. To help figure out what's going on, launch the Python 2.7.2 and examine the following:

>>> import sys, os, pprint
>>> pprint.pprint(sys.path)
>>> pprint.pprint([p for p in os.environ.items() if p[0].startswith('PYTHON')])

Also the contents of the site-packages directory:

$ cd /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/
$ ls -l
$ more easy-install.pth

and the contents of any other .pth files there.

If it is still not clear what is going on, edit your answer to show the results of the above.

UPDATE:

Thanks for providing the requested additional info. It confirms that sys.path does include the Extras directory from the Apple-supplied system Python 2.7. There is no obvious reason why that should be happening. Without more information, I can only speculate. As unlikely as it may seem, my best guess at this point is that you accidentally copied some files from the one Python to the other. The Apple-supplied Python has a patch in site.py to add the Extras directory:

$ cd /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
$ ls -l site.py
-rw-r--r--  1 root  wheel  19929 Jun 16 17:03 site.py
$ grep Extras site.py
            sitepackages.append(os.path.join(prefix, "Extras", "lib", "python"))

You should not see that in the python.org Python:

$ cd /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
$ ls -l site.py
-rw-rw-r--  1 root  admin  20288 Jun 11 15:25 site.py
$ grep Extras site.py
$

You could also do a more complete recursive search of all of the directories on sys.path for pth files but that seems unlikely to find anything.

You also appear to have installed two versions each of numpy and scipy in your site-packages, one the latest released version and the other cutting-edge version. I don't know whether you took steps to clear out the numpy and scipy directories between installs or whether that can cause problems but it might.

At this point, without more information, I would recommend that you completely delete the 2.7.2 installation and start over:

$ sudo rm -rf /Library/Frameworks/Python.framework/Versions/2.7

You might consider using another approach like the binary numpy/scipy installers or install using MacPorts. A better source of advice might be found on one of the project's mailing lists. Good luck!