CODEWITHSUNDEEP
python-2.7scrapy
Dipendra bhatt
Dipendra bhatt

Reputation: 759

My scrapy spider is not working

I tried to make a spider that crawls tripadvisor.in to extract some data but i don't know why its not working. My project name is spidey.Here is the spider i made::

import scrapy
from scrapy.selector import Selector
from spidey.items import tripad


class DmozSpider(scrapy.Spider):
    name="spidey"
    allowed_domains=["https://www.tripadvisor.in"]
    start_urls=['https://www.tripadvisor.in/Attractions-g297604-Activities-Goa.html']
    def parse(self, response):
    sel=Selector(response)

    sites=sel.xpath('//div[@id="FILTERED_LIST"]/div[@class="tmHide"]/div[@class="element_wrap"]/div[@class="wrap al_border attraction_element"]/div[@class="entry al_offer_group"]/div[@class="property_title"]').extract()
    items=[]
    for site in sites:
        item=tripad()
        item['name']=site.xpath('//h1[@id="HEADING" class="heading_name"]/text()').extract()
        items.append(item)

    return items

Upvotes: 1

Views: 106

Answers (1)

Djunzu
Djunzu

Reputation: 498

Well, I will point two errors. There may be more.

  1. As @Rafael said, allowed_domains is wrong.
  2. Indentation is absolutely important in Python. Yours is wrong.

favorite

I tried to make a spider that crawls tripadvisor.in to extract some data but i don't know why its not working. My project name is spidey.Here is the spider i made::

import scrapy
from scrapy.selector import Selector
from spidey.items import tripad

class DmozSpider(scrapy.Spider):
    name="spidey"
    allowed_domains=["tripadvisor.in"]
    start_urls=['https://www.tripadvisor.in/Attractions-g297604-Activities-Goa.html']
    def parse(self, response):
        sel=Selector(response)

        sites=sel.xpath('//div[@id="FILTERED_LIST"]/div[@class="tmHide"]/div[@class="element_wrap"]/div[@class="wrap al_border attraction_element"]/div[@class="entry al_offer_group"]/div[@class="property_title"]').extract()
        # I prefer to yield items:
        for site in sites:
            item=tripad()
            item['name']=site.xpath('//h1[@id="HEADING" class="heading_name"]/text()').extract()
            yield item

Upvotes: 1

Related Questions