CODEWITHSUNDEEP
pythonweb-scraping
mukul dhiman
mukul dhiman

Reputation: 37

Web Scraping Authorization required

I am trying to web scrape https://gomechanic.in/gurgaon/car-repair/hyundai-creta/petrol this website. I want price of each service. In network, I also got the API which is giving all the data. The API has the name get-service-detials-by-category. API Link: https://gomechanic.app/api/v2/oauth/customer/get-services-details-by-category?car_id=135&city_id=1&category_id=0&user_car_id=null . You can see this link in network section of inspect element. But when I am calling it in python, it is showing login required. But data is visible in response section.

My Python code:

url = "https://gomechanic.app/api/v2/oauth/customer/get-services-details-by-category?car_id=135&city_id=1&category_id=0&user_car_id=null"
header = {
    "Accept": "*/*",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Authorization": "Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiJiNGJjM2NhZjVkMWVhOTlkYzk2YjQzM2NjYzQzMDI0ZTAyM2I0MGM2YjQ5ZjExN2JjMDk5OGY2MWU3ZDI1ZjM2MTU1YWU5ZDIxNjE2ZTc5NSIsInNjb3BlcyI6W10sInN1YiI6IjE2MzM5MTcwNjIzOSIsImV4cCI6MTYzNjUwOTA2Mi4wLCJhdWQiOiIzIiwibmJmIjoxNjMzOTE3MDYyLjAsImlhdCI6MTYzMzkxNzA2Mi4wfQ.IUP_RJVt6mqC5EMO3HKd-2iX69_dSgBEE-jJ0pg26bizK5EBvf48d0ZRiAcwPX6bNWAIkxH7hfqA0Zq0pu1SymyeDVpxmOB2D7H5t7aj1DqhgawxD7ZgoY6Q_nyA1SAmltbeFSAIf2mwVlQV4H-pdH74qiaIG1ij9kRsBpdLxSMqpas1Vy9mQN_8W5csu24gIjvPYAdaT6w6qxjCxrlbT24EJ0hswPCy4_h12AlpZKYs_oVAqHMKgVcyi9jSeXOS_KD8Kwbcx1hNVtYVblBg5xcezm8RtP8tcJ4XgHoqXWEmI349SEb1s8wZX4u1LtEKNovMWkBwWQr8_jBPNSy7rDRHnNAvT5h2u-x-1AlnN-JFLLsz9rCWLRoypG-_1-1Y46lOUAFgjVB1L4IvPJ3zk1dxjDNJPtxzV3e-GJWVv5qlHw3g3cTlTd05r4ab-PDj314K4Ft7P9RaLdgtrcdrpO_bbs00BBN7Vo87dPFL_NHl37FmWvsh6pQ1rCa6bkQKpCZTFxgcriKTjwYeeC2XLKpnJm26PQ5ALIQQIq4EEE2LZq7N8jZ-FEtg5ozTmQ9HxkZbG12LTZzbD472OmuuSDxRlWzyTF4ObMs0PyA8dyVRTYtYT8l1juxc71TRPzG_cnTVrbjCI-rbCqvvFokGosoC_VJdiQlt3Dau6t1IrmY",
    "Connection": "keep-alive",
    "Content-Type": "application/json",
    "Host": "gomechanic.app",
    "Origin": "https://gomechanic.in",
    "Referer": "https://gomechanic.in/",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "Windows",
    "Sec-Fetch-Dest": "empty",
    "Sec-Fetch-Mode": "cors",
    "Sec-Fetch-Site": "cross-site",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36"
}
page = requests.get(url, headers=header)
response = requests.get(url)
data = json.loads(response.text)

Note: The Authorization key keeps on changing so please update it before running the code. You can find it in header section of inspect element.

Screenshots for reference: Data I needError I am getting

I want all the the data of response section

Upvotes: 2

Views: 677

Answers (1)

Yash
Yash

Reputation: 1281

You are already getting the correct output at page = requests.get(url, headers=header) and you are printing output of response = requests.get(url) where you are not passing header. Also authorization key does not change here.

You can use the following code to get package and price.

op={}
page = requests.get(url, headers=header)
for d in page.json().get("data"):
    for service in d.get("services"):
        for package in service.get("package_details"):
                op[service.get("name")]=package.get("total")
print(op)

output:

{'Basic Service': 3199, 'Standard Service': 4199, 'Comprehensive Service': 5499}

Upvotes: 2

Related Questions