CODEWITHSUNDEEP
pythonbeautifulsouppython-requests
Kohlinger
Kohlinger

Reputation: 21

amazon price web scraping with python, requests and bs4

I have a question about web scraping the price of an amazon article. I tried to get the price of an article and it works unfortunately not always. I get randomly the status code 503 (server unavailable). I can work around that problem with a while loop that ends if status code == 200. I want to understand the main problem of the unavailable server, so I can maybe fix the main problem and not work around of it. The problem occurs only on amazon so far.

here is my code for a 10 times test. The request usually fails 2/10 times

import requests
from bs4 import BeautifulSoup


for i in range(10):
    page = requests.get("https://www.amazon.de/Bloodborne-Game-Year-PlayStation-4/dp/B016ZU4FIQ/ref=sr_1_3?ie=UTF8&qid=1519566642&sr=8-3&keywords=bloodborne+ps4")

    if page.status_code != 200:
        print("Error status code: " + str(page.status_code))
        continue

    soup = BeautifulSoup(page.content, "html.parser")

    price = soup.find(id="priceblock_ourprice", class_="a-size-medium a-color-price")


    price_string = price.get_text()

    print(price_string)

Upvotes: 2

Views: 10366

Answers (2)

serkan
serkan

Reputation: 1

it's old but for anyone trying this:

you should use user-agent, here is the example code

headers = {
            myuseragent
                   }

        page = requests.get(url, headers=headers)

you can find your user agent in here

user-agent

Upvotes: 0

SIM
SIM

Reputation: 22440

Try the below script. It should get you the price.

import requests
from bs4 import BeautifulSoup

URL = "https://www.amazon.de/Bloodborne-Game-Year-PlayStation-4/dp/B016ZU4FIQ/ref=sr_1_3?ie=UTF8&qid=1519566642&sr=8-3&keywords=bloodborne+ps4"
page = requests.get(URL,headers={"User-Agent":"Defined"})
soup = BeautifulSoup(page.content, "html.parser")
price = soup.find(id="priceblock_ourprice").get_text()
print(price)

Output:

EUR 34,99

Upvotes: 7

Related Questions