CODEWITHSUNDEEP
pythonweb-scrapingscrapy
xuriguer
xuriguer

Reputation: 77

Scrapy - Scrape mutliple urls when only change the last parameter

I want to scrape some urls that only change the last parameter, called 'asin'. The url is -> https://www.amazon.es/dp/'+asin

I have the list of these asin in a csv. I'm trying to get the scrape with this code with any result.

import scrapy
from csv import DictReader

class CommentSpider(scrapy.Spider):
     name = 'comments'
     allowed_domains = ['amazon.es']
     def start_requests(self):
         with open("adresses.csv") as asin:
              for i in DictReader(asin):
                  link= ('https://www.amazon.es/dp/'+i)

                  yield Request(url = link,
                        callback = self.parse,
                        method ='GET')

    def parse(self, response):
         items = AmazonItem()

         Product_name = response.css('#productTitle.a-size-large').css('::text').extract()
         Rating= response.css('.a-icon-alt::text')[0].extract()
         Comments =  response.css('#acrCustomerReviewText').css('::text').extract()

         items['Product_name'] = Product_name
         items['Rating'] = Rating
         items['Comments'] = Comments

         yield items

Can you hlep me with the script, please?

Thanks!

Upvotes: 0

Views: 48

Answers (1)

Umair Ayub
Umair Ayub

Reputation: 21241

TypeError: can only concatenate str (not "collections.OrderedDict") to str

That error says it all, you cannot concatenate string and a dictionary

You can print that i variable to see what it has, you need to change your code something like below

for i in DictReader(asin):
    self.logger.info(i)
    link= ('https://www.amazon.es/dp/'+i['column_name_in_your_csv'])

    yield Request(url = link,
        callback = self.parse,
        method ='GET')

Upvotes: 1

Related Questions