CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingscrapy
codewithawais
codewithawais

Reputation: 579

Scrapy crawler returning duplicate values

Below is my complete code. I don't know why it's returning a lot of duplicates. Any fix, please? I am trying to request all the regions from this link "https://www.compass.com/agents" and extract the agent's info. 

# -*- coding: utf-8 -*-
import scrapy


class MainSpider(scrapy.Spider):
    name = 'main'
    start_urls = ["https://www.compass.com/agents"]

    def parse(self, response):
        regions = response.xpath('//ul[@class="geoLinks-list textIntent-caption1--strong"]/li')
        for each in regions:
            region_link = each.xpath('.//a/@href').get()
            region_name = each.xpath('.//a/text()').get()

            yield response.follow(url=region_link, callback=self.parse_data, meta={"region_text": region_name})


    def parse_data(self, response):
        region = response.request.meta["region_text"]

        agents = response.xpath('//div[@class="agentCard-contact"]')
        for agent in agents:
            name = agent.xpath('normalize-space(//div[@class="agentCard-contact"]/a/text())').get()
            profile_link = agent.xpath('//div[@class="agentCard-contact"]/a/@href').get()
            email = agent.xpath('//a[@class="textIntent-body agentCard-email"]/@href').get()
            mobile = agent.xpath('//a[@class="textIntent-body agentCard-phone"]/@href').get()

            yield {
                "Name": name,
                "Profile_link": profile_link,
                "Email": email,
                "Mobile": mobile,
                "Region": region,
            }

Upvotes: 0

Views: 290

Answers (1)

Naved Ansari
Naved Ansari

Reputation: 36

I thing there is the issue with your xpath. Change your xpath with the below and try: name = agent.xpath('normalize-space(.//a[@class="textIntent-headline1 agentCard-name"]/text())').get() profile_link = agent.xpath('.//a[@class="textIntent-headline1 agentCard-name"]/@href').get() email = agent.xpath('.//a[@class="textIntent-body agentCard-email"]/@href').get() mobile = agent.xpath('.//a[@class="textIntent-body agentCard-phone"]/@href').get()

Upvotes: 1

Related Questions