CODEWITHSUNDEEP
pythonmacosopensslscrapyosx-snow-leopard
Garret C
Garret C

Reputation: 45

Scrapy error--ImportError: Library not loaded

I'm trying to do my first bit of scraping with Scrapy using a really simple demo from https://github.com/scrapy/dirbot and I get an ImportError: Library not loaded, as follows.

gchrist1$ scrapy crawl dmoz

2015-12-20 17:48:33 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)

2015-12-20 17:48:33 [scrapy] INFO: Optional features available: ssl, http11, boto

2015-12-20 17:48:33 [scrapy] INFO: Overridden settings: {'DEFAULT_ITEM_CLASS': 'dirbot.items.Website', 'NEWSPIDER_MODULE': 'dirbot.spiders', 'SPIDER_MODULES': ['dirbot.spiders']}

Traceback (most recent call last):

File "/Users/gchrist1/anaconda/bin/scrapy", line 11, in <module>
sys.exit(execute())

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/cmdline.py", line 143, in execute
_run_print_help(parser, _run_command, cmd, args, opts)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
func(*a, **kw)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/cmdline.py", line 150, in _run_command
cmd.run(args, opts)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
self.crawler_process.crawl(spname, **opts.spargs)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 150, in crawl
crawler = self._create_crawler(crawler_or_spidercls)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 166, in _create_crawler
return Crawler(spidercls, self.settings)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/crawler.py", line 46, in __init__
self.extensions = ExtensionManager.from_crawler(self)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 56, in from_crawler
return cls.from_settings(crawler.settings, crawler)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/middleware.py", line 32, in from_settings
mwcls = load_object(clspath)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
mod = import_module(module)

File "/Users/gchrist1/anaconda/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/extensions/memusage.py", line 16, in <module>
from scrapy.mail import MailSender

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/scrapy/mail.py", line 22, in <module>
from twisted.internet import defer, reactor, ssl

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
from OpenSSL import SSL

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
from OpenSSL import rand, crypto, SSL

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
from OpenSSL._util import (

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/OpenSSL/_util.py", line 6, in <module>
from cryptography.hazmat.bindings.openssl.binding import Binding

File "/Users/gchrist1/anaconda/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 13, in <module>
from cryptography.hazmat.bindings._openssl import ffi, lib

ImportError: dlopen(/Users/gchrist1/anaconda/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Library not loaded: libcrypto.1.0.0.dylib

Referenced from: /Users/gchrist1/anaconda/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

Reason: image not found

I'm using Mac OS 10.6.8, with Python 2.7.6 |Anaconda 1.9.1 (x86_64)| (default, Jan 10 2014, 11:23:15) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin

I assume the error has something to do with not having OpenSSL, so I tried both brew install openssl and pip install openssl. The former seemed to work:

Warning: openssl-1.0.2e already installed

but the latter did not: 

pip install OpenSSL
Downloading/unpacking OpenSSL
  Could not find any downloads that satisfy the requirement OpenSSL
Cleaning up...
No distributions at all found for OpenSSL
Storing debug log for failure in /Users/gchrist1/.pip/pip.log)

But I still get the same error. I also tried what is recommended at https://raspberrypi.stackexchange.com/questions/28176/python-and-openssl-error-on-import but got the same errors.

Upvotes: 2

Views: 664

Answers (2)

Shuai Zhang
Shuai Zhang

Reputation: 2061

Perhaps you should try to upgrade pip and start over:

easy_install -U pip

Upvotes: 2

jww
jww

Reputation: 102346

ImportError: dlopen(/Users/gchrist1/anaconda/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Library not loaded: libcrypto.1.0.0.dylib

On OS X, the various packages like Python and Scrapy should probably be using an install_name so the proper version of the library is found at runtime. It sounds like they are not using it during build/install time.

Perform a find for libcrypto.1.0.0.dylib:

find <some directory> -name libcrypto.1.0.0.dylib

Once you get a path, use DYLD_LIBRARY_PATH to set the path prior to running the other programs. DYLD_LIBRARY_PATH is like Linux's LD_LIBRARY_PATH. Also see Apple's dyld(1) man pages.

Upvotes: 1

Related Questions