CODEWITHSUNDEEP
pythondjangoscrapy
Abhimanyu
Abhimanyu

Reputation: 81

Scrapy Crawls only 1st page

heya I am making a Project using scrapy in which I need to scrap the business details from a business directory http://directory.thesun.co.uk/find/uk/computer-repair
the problem I am facing is: when I am trying to crawl the page my crawler fetches the details of only 1st page whereas I need to fetch the details of the rest 9 pages also; that is all 10 pages.. i am showing below my Spider code and items.py and settings .py please see my code and help me to solve it

spider code::

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from project2.items import Project2Item

class ProjectSpider(BaseSpider):
    name = "project2spider"
    allowed_domains = ["http://directory.thesun.co.uk/"]
    start_urls = [
        "http://directory.thesun.co.uk/find/uk/computer-repair"
    ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites = hxs.select('//div[@class="abTbl "]')
        items = []
        for site in sites:
            item = Project2Item()
            item['Catogory'] = site.select('span[@class="icListBusType"]/text()').extract()
            item['Bussiness_name'] = site.select('a/@title').extract()
            item['Description'] = site.select('span[last()]/text()').extract()
            item['Number'] = site.select('span[@class="searchInfoLabel"]/span/@id').extract()
            item['Web_url'] = site.select('span[@class="searchInfoLabel"]/a/@href').extract()
            item['adress_name'] = site.select('span[@class="searchInfoLabel"]/span/text()').extract()
            item['Photo_name'] = site.select('img/@alt').extract()
            item['Photo_path'] = site.select('img/@src').extract()
            items.append(item)
        return items

My items.py code is as follows:: 

from scrapy.item import Item, Field

class Project2Item(Item):
    Catogory = Field()
    Bussiness_name = Field()
    Description = Field()
    Number = Field()
    Web_url = Field()
    adress_name = Field()
    Photo_name = Field()
    Photo_path = Field()

my settings.py is:::

BOT_NAME = 'project2'

SPIDER_MODULES = ['project2.spiders']
NEWSPIDER_MODULE = 'project2.spiders'

please help me to extract details from other pages too...

Upvotes: 1

Views: 160

Answers (1)

Eugene Nagorny
Eugene Nagorny

Reputation: 1666

Fetching description .select('span/text()') you are selecting text from ALL spans in //div[@class="abTbl "]. To extract last span you can use 'span[last()]/text()' xpath

btw this http://www.w3schools.com/xpath/xpath_syntax.asp should help you with XPathes

Upvotes: 1

Related Questions