CODEWITHSUNDEEP
pythonweb-scrapingweb-crawlerscrapy
Arkan Kalu
Arkan Kalu

Reputation: 403

Scrapy middleware setup

I am trying to access public proxy using scrapy to get some data. I get the following error when i try to run the code:

ImportError: Error loading object 'craiglist.middlewares.ProxyMiddleware': No module named middlewares

I've created middlewares.py file with following code:

import base64
# Start your middleware class
class ProxyMiddleware(object):
    # overwrite process request
    def process_request(self, request, spider):
        # Set the location of the proxy
        request.meta['proxy'] = "http://124.200.36.150:8118"

settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'craiglist.middlewares.ProxyMiddleware': 100,
}

Any help is welcomed.

Upvotes: 1

Views: 839

Answers (1)

alecxe
alecxe

Reputation: 474003

With this setup, you need to move middlewares.py one level up into craiglist package.

Upvotes: 1

Related Questions