Python, Pandas, and Pico: Unable to import Pandas, but NumPy is no problem

Question

I've managed to construct a simple app utilizing the Pico framework (https://github.com/fergalwalsh/pico). My frontend is connecting to my backend without any difficulties. Below is my Python file, which at the moment simply returns/renders a string, using a client-side input value, "name".

from __future__ import absolute_import
import sys
import pico
import numpy as np

# import sklearn
# import pandas as pd
from api2 import aloha

from pico import PicoApp

@pico.expose()
def hello(name):

   a = np.arange(15).reshape(3, 5)
   # a = np.arrange('data', 'field').reshape(3,5)

return "hello %s, %s" %(name, a)

app = PicoApp()
app.register_module(__name__)

(It also returns a NumPy array, simply because I'm testing what I can import into the file.)

All my packages are installed just fine, via Anaconda in /site-packages, which is in the python3.6 directory.

Oddly, the app runs fine; it can import NumPy. It breaks, however, when I try to import Pandas or SKLearn. I've tried manually copying and pasting NumPy into /Library/Python/2.7/site-packages, which actually breaks the app. But NumPy works in the app when it is only located in Anaconda's /site-packages.

I've tried altering app.register(__name__) to app.register('api'), which is the name of the Python file (api.py), based on another Question/Answer here. I've also tried reinstalling Pandas with sudo -H pip install pandas, but all the requirements are already satisfied.

This is the error that is thrown when I try to include Pandas in api.py:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Library/Python/2.7/site-packages/pico/server.py", line 31, in <module>
    app = import_string(module_name)
  File "/Library/Python/2.7/site-packages/werkzeug/utils.py", line 443, in import_string
    sys.exc_info()[2])
  File "/Library/Python/2.7/site-packages/werkzeug/utils.py", line 431, in import_string
    module = import_string(module_name)
  File "/Library/Python/2.7/site-packages/werkzeug/utils.py", line 443, in import_string
    sys.exc_info()[2])
  File "/Library/Python/2.7/site-packages/werkzeug/utils.py", line 418, in import_string
    __import__(import_name)
  File "./api.py", line 6, in <module>
    import pandas as pd
  File "/Library/Python/2.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/Library/Python/2.7/site-packages/pandas/compat/numpy/__init__.py", line 24, in <module>
    'this pandas version'.format(_np_version))
werkzeug.utils.ImportStringError: import_string() failed for 'api.app'. Possible reasons are:

- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;

Debugged import:

- 'api' not found.

Original exception:

ImportStringError: import_string() failed for 'api'. Possible reasons are:

- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;

Debugged import:

- 'api' not found.

Original exception:

ImportError: this version of pandas is incompatible with numpy < 1.9.0
your numpy version is 1.8.0rc1.
Please upgrade numpy to >= 1.9.0 to use this pandas version

When I run which python, it points to /Users/richardscheiwe/anaconda3/bin/python. Also, I have NumPy v.1.15 installed, and I can't find any other NumPy folder(s). When I try moving a version of NumPy to Library/Python/2.7/site-packages, I get this error:

ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: cannot import name multiarray

I guess I need to somehow point the app's Python to Anaconda's Python 3.6 version, but I don't know how to do that. Pico is also available in Anaconda's /site-packages directory, but it isn't pointing there.

Any help is greatly appreciated. I've scoured StackOverflow and GitHub.

Answer 1

You don't mention how you are starting the pico app but I assume you are doing like this:

python -m pico.server api

In this case it will simply use whatever python is in your path. If it is python3 in /Users/richardscheiwe/anaconda3/bin/python but you are getting errors referring to /Library/Python/2.7/ then there is some problem with your anaconda installation/paths in your environment.

There is nothing different with pico to running a plain python script but I suggest you create a simplified script without pico (literally just import pandas) to work out your environment issues with simpler error messages.

Answer 2

If I'm reading this correctly the error seems to come from trying to use a version of NumPy built for running on python 2.6 while your app is running using Python3.

Try removing NumPy using; "sudo pip uninstall numpy" and then use "pip -H install Numpy" to try reinstalling it and seeing if it correctly finds the Python3 version of Numpy