CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Luka Jozić
Luka Jozić

Reputation: 164

Scrapy Request returns wrong currency

Im trying to scrape this website for information. Everything is fine except for the price. On the website and in the html the price is listed in USD. But when I run the code below it returns it in GBP.

price = response.xpath('//*[@id="page"]/div[2]/div/div[1]/div[2]/div/div[2]/div/p').extract()

I have tried different variations of the following:

headers = {
    'USER_AGENT': "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
    'Accept-Language': 'en-US,en;q=0.9',
}


def start_requests(self):
    for url in self.start_urls:
        yield scrapy.Request(url=url, headers=headers, callback=self.parse, cookies={'hn.globale.currency': 'USD', 'hn.globale.country': 'US'})

But its always coming out as GBP. How can I change it to get USD? I can also get the product data as json but its all in GBP there as well.

Upvotes: 0

Views: 120

Answers (1)

Tim Roberts
Tim Roberts

Reputation: 54733

That page is all built dynamically with Javascript and configured on the fly. If you look at the page, you'll see the "USD" selection at upper left, but that text is not in the source. Unless you are executing the Javascript when you scrape, you can't get the conversions. If they have a JSON API, there must be a way to configure it, but they'd have to tell you that.

Upvotes: 1

Related Questions