CODEWITHSUNDEEP
python-2.7scrapypyquery
user3218442
user3218442

Reputation: 21

How to use Pyquery with scrapy?

My objective is to use pyquery with scrapy, apparently from scrapy.selector import PyQuerySelector returns ImportError: cannot import name PyQuerySelector when I crawl the spider.

I followed this specific gist https://gist.github.com/joehillen/795180 to implement pyquery.

Any suggestions or tutorials that can help me get this job done?

Upvotes: 2

Views: 895

Answers (1)

Raghu
Raghu

Reputation: 323

You declare a class and make your rules and in the callback attribute of rule extractor give parse_item by default the scrapy goes parse() function

def parse_item(self, response):
    pyquery_obj = PyQuery(response.body)
    header = self.get_header(pyquery_obj)
    return {
        'header': header,
    }


def get_header(self, pyquery_obj):
    return pyquery_obj('#page_head').text()

Upvotes: 1

Related Questions