CODEWITHSUNDEEP
amazon-web-servicesweb-scrapingcurrency
PharaohAtem
PharaohAtem

Reputation: 13

How to force web-scraping script to get results based off my country and not the AWS's server location?

I have a Steam web-scraper that when I use on my own PC, gives me the results in my own currency, but when I use it hosted off AWS (US servers) it gives it in USD.

game = input("Game: ")

html = requests.get("https://store.steampowered.com/search/?term=" + game +"&category1=998")
doc = lxml.html.fromstring(html.content)

title = doc.xpath('//*[@id="search_resultsRows"]/a[1]/div[2]/div[1]/span/text()')[0]
discount = doc.xpath('.//div[@class="col search_discount  responsive_secondrow"]/text()')

if not discount:
    discount = "No discount"
    price = doc.xpath('.//div[@class="col search_price  responsive_secondrow"]/text()')
else:
    price = doc.xpath('.//div[@class="col search_price discounted responsive_secondrow"]/text()')

price_split = price[0].split()
price_final = " ".join(price_split)

print("{}: {} ({})".format(title, price_final, discount))

Let's say the game is Assassin's Creed Odyssey. On my own PC, I'd get:

Assassin's Creed® Odyssey: CDN $79.99 (No discount)

When I use it off Amazon's AWS, I'd get the US version:

Assassin's Creed® Odyssey: $59.99 (No discount)

How can I make it so that I always get the Canadian result regardless of where I host the script?

Upvotes: 0

Views: 284

Answers (1)

F_SO_K
F_SO_K

Reputation: 14849

It really depends what you are scraping and how they have configured their website. They will probably be changing their content based on one of two things;

  1. Your IP location. If your server IP is in the US, they might show you prices in USD. In this case you can; host the script on a server in the location you want to use (e.g. Canada) or setup a http proxy in the chosen location and run your http traffic through that. Sounds easier just to spin up a server in the right location.
  2. This site could be using user-agent data. The site could be changing the content based on the Accept-Language http header you send. This is unlikely but it would be easy to check/test by experimenting with updating your Accept-Language header.

You should also inspect the website. Usually a website will display content based on your IP location, but provide you with an option to switch to a different location. If this is the case you can

  1. Setup your script to 'click' on the right location before starting to scrape
  2. Look at how the website manage your location choice. Its probably stored in a cookie, in which case you can setup your script to send the required cookie.

Upvotes: 1

Related Questions