Gabriel Alejandro
Gabriel Alejandro

Reputation: 13

python issue to join relative url to absolute url for img

I'm facing the following issues with my current code to make it work. I just concatenate the URL but its not working:

Current relative path (this is what I get with normal response.xpath crawl):

/imagename.jpg

This is my current code:

class MercadoSpider(CrawlSpider):
    name = 'extractor'
    item_count = 0

    rules = {
        # Para cada item
        Rule(LinkExtractor(allow = (), restrict_xpaths = ('//*[@id="main-container"]/div/div[2]/div[1]/ul/li[7]/a'))),
        Rule(LinkExtractor(allow =(), restrict_xpaths = ('//*[@id="main-container"]/div/div[2]/div[2]/div/div/div/h4/a')),
                            callback = 'parse_item', follow = False)
    }

    def parse_item(self, response):
        ml_item = MercadoItem()
        ml_item['titulo'] = response.xpath('normalize-space(//*[@id="main-container"]/div/div[2]/div[1]/div[2]/h2)').extract_first()
        ml_item['sku'] = response.xpath('normalize-space(//*[@id="main-container"]/div/div[2]/div[1]/div[2]/ul/li[2]/a)').extract()
        ml_item['marca'] = response.xpath('normalize-space(//*[@id="main-container"]/div/div[2]/div[1]/div[2]/ul/li[1]/a)').extract()
        ml_item['tecnologia'] = response.xpath('normalize-space(//*[@id="DetailedSpecs"]/table/tbody/tr[4]/td)').extract_first()
        ml_item['tipo'] = response.xpath('normalize-space(//*[@id="DetailedSpecs"]/table/tbody/tr[3]/td)').extract()
        ml_item['precio'] = response.xpath('normalize-space(//*[@id="main-container"]/div/div[2]/div[1]/div[2]/div[1]/span[2])').extract()
        ml_item['color'] = response.xpath('normalize-space(//*[@id="mainC"]/div/div/div/div/ul/li/b)').extract()
        ml_item['potencia'] = response.xpath('normalize-space(//*[@id="ProductReview"]/div/div/div/dl/dd/strong)').extract()
        ml_item['condicion'] = response.xpath('normalize-space(//*[@class="stock in-stock"])').extract_first()
        ml_item['desc_corta'] = response.xpath('normalize-space(//*[@id="tab-additional_information"])').extract()
        ml_item['descripcion'] = response.xpath('normalize-space(//*[@id="main-container"]/div/div[2]/div[2]/div)').extract()
        ml_item['id_publicacion'] = response.xpath('normalize-space(//*[@id="mainC"]/div/div/div[11]/div[1]/ul/li[1]/b)').extract()
        #imagenes del producto
        xpath1 = 'http://www.website.com.ar'
        xpath2 = response.xpath('//*[@id="main-container"]/div/div[2]/div[1]/div[1]/p/img/@src').extract_first()
        ml_item['image_urls'] = xpath1 + xpath2
        ml_item['image_name'] = response.xpath('//*[@id="main-container"]/div/div[2]/div[1]/div[1]/p/img/@src').extract()
        #info de la tienda o vendedor
        ml_item['categoria'] = response.xpath('normalize-space(//*[@class="woocommerce-breadcrumb breadcrumbs"])').extract_first()
        self.item_count += 1
        if self.item_count > 10000:
            raise CloseSpider('item_exceeded')
        yield ml_item

Upvotes: 0

Views: 109

Answers (1)

E. Amanatov
E. Amanatov

Reputation: 46

try

absolute_url = response.urljoin(your_url_from_xpath)

scrapy documentation

Upvotes: 1

Related Questions