ImportError: Permission Denied while using LXML

Question

I've been having a ton of trouble using LXML, after installing it from https://pypi.python.org/pypi/lxml/3.2.1 using Easy_Install-2.7. I installed it on Windows using cygwin, and at first the package seemed to be okay. However upon further testing I ran into problems.

When I run code with:

import lxml

it works completely fine. But as soon as I try:

import lxml.etree

I get this error:

Traceback (most recent call last):
  File "D:\Nick_Code\NewsScraper\testdummy.py", line 7, in <module>
    import lxml.etree
  File "/usr/lib/python2.7/site-packages/lxml-3.2.0-py2.7-cygwin-1.7.20-i686.egg/lxml/etree.py", line 7, in <module>
    __bootstrap__()
  File "/usr/lib/python2.7/site-packages/lxml-3.2.0-py2.7-cygwin-1.7.20-i686.egg/lxml/etree.py", line 6, in __bootstrap__
    imp.load_dynamic(__name__,__file__)
ImportError: Permission denied

I've been trying to find information/work arounds for quite a while but no success. Please let me know if you have any insight or need information.

Thanks!

Michael

Answer 1

This is not a solid answer. But I will highlight several of the problems involved for obtaining a solution. Most likely the problem above, is like a cancer caused by several factors acting catastrophically together.

I have the same exact problem as in the OP, when attempting to use the native Cygwin supplied Python packages on my Windows Vista machine. Being new to Python I have spent several days in trying to get this to work, and understand why it is not working. But all my Google-fu returned nothing but countless dead ends. So here's my take on this.

There are many reasons why Python could have trouble under Cygwin, some which you can do something about and some which are beyond most peoples control. What it boils down to, are the following key issues:

1. Windows is a complete mess when it comes to file permissions, and Cygwin cannot handle windows file permissions very well. So what you see in Cygwin is far from the whole story.

2. Windows is shamefully character case-independent which causes loads of trouble, especially when you need to (cross)compile anything that was originally developed under *nix based system (i.e. everything). In fact, if you attempt extracting any archive that contains files whose names differ only in capitalization. (I.e. "makefile" vs "Makefile" etc.) files under Windows or Cygwin, you loose all but one of the files. in case their You need to enable case-sensitivity to do anything more than "hello world" *nix compilations.

3. Windows handles symlinks completely different than Cygwin. And if your ZIP, TAR etc. archives contain any symlinks, they will be broken after extraction to Windows environment.

4. Sloppy code practices, where developer have not properly tested their creations on various environments, or carefully set proper file permissions to their *.tar.gz collections. Including correct dependency specifications, or mentioning whether or not binaries has been statically linked etc.

For the full gory details and further (Win-Cygwin) issues, look HERE.

At first I tried to use Cygwin's own Python without any additional packages, and nstalling lxml using PIP and easy_install. Then I tried to use Cygwin's own libxml2, libxslt and xml python packages, and I had the same problems.

At first, after installing the static windows binaries (as suggested elsewhere), I got this error:

File "/usr/lib/python2.7/site-packages/lxml-3.2.4-py2.7-cygwin-1.7.24-i686.egg/lxml/etree.py", line 6, in __bootstrap__
    imp.load_dynamic(__name__,__file__)
ImportError: Permission denied
Aborted (core dumped)

Then I investigated the file permissions and changed those with: chmod -R 755 /usr/lib/python2.7/

I got one step further to isolate problem to an apparently missing file. And enabling verbose and diagnostic mode's didn't help much either.

File "/usr/lib/python2.7/site-packages/lxml-3.2.4-py2.7-cygwin-1.7.24-i686.egg/lxml/etree.py", line 6, in __bootstrap__
    imp.load_dynamic(__name__,__file__)
ImportError: No such file or directory
Aborted (core dumped)

HERE is the exact statement specification:

Load and initialize a module implemented as a dynamically loadable shared library and return its module object. If the module was already initialized, it will be initialized again. Re-initialization involves copying the __dict__ attribute of the cached instance of the module over the value used in the module cached in sys.modules. The pathname argument must point to the shared library. The name argument is used to construct the name of the initialization function: an external C function called initname() in the shared library is called. The optional file argument is ignored. (Note: using shared libraries is highly system dependent, and not all systems support it.)

So I started reading on the lxml website which clearly state lxml's dependencies on both libxml2 and libxslt, and unless they are statically linked, they also depend on iconv and zlib. So you're lead to believe you need to install all of these. Don't! Continue reading. But if you're going to build from sources (as easy_install may try to do) you'll need everything, including the development header libraries: libxml2-devel, libxslt-devel. Another place states that you also need Cython and install with:

easy_install lxml==dev 

The dependencies are shown in this picture from HERE:

So you think you may get away with something like:

STATIC_DEPS=true pip install lxml

But that doesn't do it either. Probably because the libraries used to compile Cygwin's Python have to be the same as those for compiling lxml. But I don't know. Notice how the lxml package refers to Cygwin "1.7.24". My Cygwin is already "1.7.25" and you can check this with uname -a. Then you can check your static python executable with file and ldd. Then you understand that this also depend on the C-compiler used for building python/cygwin under Windows or *nix. Smelling a nightmare I decided that building my own was not the way to go. So next I tried to install the Python libraries (supplied as executables) meant for Windows Python. This didn't work since I never had windows native Python installed, and I was greeted with an error that the installed could not find Python in my registry. I could of course just extract the executable, but I wouldn't know where to put the binaries without the installer. So I had another idea...

There are 3 possible solutions to getting this to work, as far as I can see.

1. The easy way of installing a Windows native Python interpreter. You loose some native Cygwin functionality, unless you install in correct place: /usr/lib/python2.7 and make sure Cygwin can find it and use it. This also uses a different file-permissions, case-sensitivity and character set (UTF-16LE) than Cygwin (UTF-8), potentially creating many other issues down the line! Difficulty: Easy

2. Continue hacking the Cygwin's Python, to make it work with the binary libraries used in (1). But this requires:

   • a) Uninstall and remove all Cygwin Python packages, except bare Python interpreter.
   • b) Remove all PIP and easy install traces.
   • c) Hacking the Windows registry to pretend to have Python27 installed:
   • HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\InstallPath C:\Python27\
   • HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7\PythonPath C:\Python27\Lib;C:\Python27\DLLs;C:\Python27\Lib\lib-tk
   • HKEY_CLASSES_ROOT...
   • d) Install the Windows binary libraries.
   • e) All the rest should now hopefully work with PIP or easy_install. Difficulty: Medium!

3. Doing it properly by compiling Python and all libraries from scratch. Difficulty: Hard!

I successfully did (1), but I still think (2) is the smarter way of doing it, but I have not tested it, which is why I don't consider this as a good answer. BTW. One more quirk, I have to run the interpreter with: python.exe -E to avoid an annoying: "SyntaxError: invalid syntax" when hitting return!

Conclusion:

Apparently, you don't need the libxml2 and libxslt python packages to use lxml! In my case I needed Scrapy, so I also had to install a few other packages.

$ pip.exe list
cssselect (0.9.1)
lxml (3.2.4)
pip (1.4.1)
pyOpenSSL (0.11)
pywin32 (218)
queuelib (1.1.1)
Scrapy (0.20.0)
setuptools (1.4.1)
six (1.4.1)
Twisted (13.2.0)
w3lib (1.5)
zope.interface (4.0.5)


$ll /cygdrive/c/Python27/Lib/site-packages/
adodbapi
cssselect
isapi
lxml
OpenSSL
pip
pythonwin
pywin32_system32
queuelib
scrapy
twisted
w3lib
win32
win32com
win32comext
zope
cssselect-0.9.1-py2.7.egg-info
lxml-3.2.4-py2.7.egg-info
pip-1.4.1-py2.7.egg-info
queuelib-1.1.1-py2.7.egg-info
Scrapy-0.20.0-py2.7.egg-info
six-1.4.1-py2.7.egg-info
Twisted-13.2.0-py2.7.egg-info
w3lib-1.5-py2.7.egg-info
zope.interface-4.0.5-py2.7.egg-info
PyWin32.chm
setuptools-1.4.1-py2.7.egg
pyOpenSSL-0.11-py2.7.egg-info
pywin32-218-py2.7.egg-info
easy-install.pth
pywin32.pth
setuptools.pth
zope.interface-4.0.5-py2.7-nspkg.pth
pythoncom.py
six.py
pythoncom.pyc
six.pyc
pythoncom.pyo
pywin32.version.txt
README.txt

Useful References:

• HERE
• HERE
• HERE HERE HERE HERE HERE