CODEWITHSUNDEEP
pythonhttp-redirectxpathscrapy
romankk
romankk

Reputation: 51

Scrapy unable to parse links

Unable to collect links properly. Keep getting partial links from the page. How to get my parser to work?

import scrapy


class GlobaldriveruSpider(scrapy.Spider):
    name = 'globaldriveru'
    allowed_domains = ['globaldrive.ru']
    start_urls = ['https://globaldrive.ru/moskva/motory/?items_per_page=500']

    def parse(self, response):
        links = response.xpath('//div[@class="ty-grid-list__item-name"]/a/@href').get()
        for link in links:
            yield scrapy.Request(response.urljoin(link), callback=self.parse_products, dont_filter=True)
            #yield scrapy.Request(link, callback=self.parse_products, dont_filter=True)

    def parse_products(self, response):
     #       for parse_products in response.xpath('//div[contains(@class, "container-fluid  products_block_page")]'):
        item = dict()
        item['title'] = response.xpath('//h1[@class="ty-product-block-title"]/text()').extract_first()
        yield item

Here is some output log

[]
2019-04-29 16:21:12 [scrapy.core.engine] INFO: Spider opened
2019-04-29 16:21:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2019-04-29 16:21:12 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6024
2019-04-29 16:21:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://globaldrive.ru/robots.txt> (referer: None)
2019-04-29 16:21:17 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://globaldrive.ru/moskva/motory/?items_per_page=500> (referer: None)
2019-04-29 16:21:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://globaldrive.ru/h/> from <GET https://globaldrive.ru/h>
2019-04-29 16:21:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://globaldrive.ru/-/> from <GET https://globaldrive.ru/->
2019-04-29 16:21:18 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://globaldrive.ru/%d0%b9/> from <GET https://globaldrive.ru/%D0%B9>
2019-04-29 16:21:18 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET http://globaldrive.ru/%d1%80/> from <GET https://globaldrive.ru/%D1%80>

Upvotes: 1

Views: 72

Answers (1)

vezunchik
vezunchik

Reputation: 3717

Replace .get() with .extract() in parse function, right now you're iterating through one link letter by letter, but you need just to extract all the links.

def parse(self, response):
    links = response.xpath('//div[@class="ty-grid-list__item-name"]/a/@href').extract()  # <- here
    for link in links:
        yield scrapy.Request(response.urljoin(link), self.parse_products)

Upvotes: 1

Related Questions