CODEWITHSUNDEEP
c++pythonhadoopshared-libraries
ffriend
ffriend

Reputation: 28492

Pydoop: No module named _hdfs_*

I was able to build and install Pydoop without errors, so, for example, I can do the following: 

>>> import pydoop
>>> pydoop.__version__
'0.10.0'

However, when I try to import main Pydoop modules such as pipes or hdfs I'm getting ImportError: 

>>> import pydoop.hdfs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydoop/hdfs/__init__.py", line 79, in <module>
    from fs import hdfs, default_is_local
  File "pydoop/hdfs/fs.py", line 28, in <module>
    hdfs_ext = pydoop.import_version_specific_module("_hdfs")
  File "pydoop/__init__.py", line 111, in import_version_specific_module
    return import_module(complete_mod_name(name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named _hdfs_2_0_0_cdh_4_3_0

In addition, when I try to use pydoop script I'm getting such a hint: 

...
ImportError: /usr/local/lib/python2.7/dist-packages/pydoop/_pipes_2_0_0_cdh_4_3_0.so: undefined symbol: BIO_s_mem

BIO_s_mem is a symbol from libssl (OpenSSL), so I guess Pydoop can't find this shared library. I made sure it is available, ends with .so (as opposed to, say, .so.1) and is in LD_LIBRARY_PATH.

So what may be the reason for this error? How can I fix it (build options? environment variables?)

Any help is appreciated.

Upvotes: 0

Views: 1685

Answers (2)

thetainted1
thetainted1

Reputation: 471

Not sure about the pipes error, but I ran into your issue with _hdfs_2_0_0_cdh_4_3_0 (mine was a different version of hadoop, but I believe the problem is similar).

The setup.py script seems to want to make an egg file in /usr/local/lib/python2.7/dist-packages for pydoop, but the setup requires that it just be a folder (which will have that _hdfs_2_0_0_cdh_4_3_0.so file in it).

The solution is pretty simple: just delete /usr/local/lib/python2.7/dist-packages/pydoop-0.11.1.egg-info or the equivalent for you version.

Upvotes: 1

simleo
simleo

Reputation: 2975

What OS version are you using? Try setting LD_PRELOAD to the path of your libssl, e.g.:

export LD_PRELOAD=/lib/x86_64-linux-gnu/libssl.so.1.0.0

Upvotes: 1

Related Questions