CODEWITHSUNDEEP
pythonseleniumweb-scrapingbeautifulsoup
doomdaam
doomdaam

Reputation: 783

How to get access to XHR data using python script?

I have the following data in this URL:

https://forsikringsguiden.dk/signalr/poll?transport=longPolling&messageId=d-3E730B44-EB%2C0%7CTX%2C1%7CTY%2C1%7CTZ%2C0&clientProtocol=1.4&connectionToken=81u0%2BTRKNqfgWWdC72ld4wHwbmFGpkCXZRiPe3a1mJPmKvrJpUhBiq3qOThYE04iotxtecRVERe7QA2OyL0GugQoS2nyzbnlcvGZpV1JSav5XqGY0OCrpBKVp0vqXToc&connectionData=%5B%7B%22name%22%3A%22insuranceofferrequesthub%22%7D%5D&tid=5&_=1572867896650

How would I access the data using a python script?

Below is the header information. I believe using a combination of the data I could generate a URL, but I think it's a risk to merge all the different information to end up with a final URL.

Let me know if additional information is missing, I don't know what else I need include in my question.

The data I need is the "companydisplayname", "basicprice", and "discountprice".

Below image is from the inspect of the main URL, which requires you to fill out some forms before information is added. It compared insurance prices:

https://forsikringsguiden.dk/#!/bilforsikring/resultatside

enter image description here

Upvotes: 0

Views: 337

Answers (1)

skaul05
skaul05

Reputation: 2344

You can use the following script. First, we can use requests to fetch the JSON and then perform operations on it to fetch the required results:

import requests
request_data = requests.get(url ="https://forsikringsguiden.dk/signalr/poll?transport=longPolling&messageId=d-3E730B44-EB%2C0%7CTX%2C1%7CTY%2C1%7CTZ%2C0&clientProtocol=1.4&connectionToken=81u0%2BTRKNqfgWWdC72ld4wHwbmFGpkCXZRiPe3a1mJPmKvrJpUhBiq3qOThYE04iotxtecRVERe7QA2OyL0GugQoS2nyzbnlcvGZpV1JSav5XqGY0OCrpBKVp0vqXToc&connectionData=%5B%7B%22name%22%3A%22insuranceofferrequesthub%22%7D%5D&tid=5&_=1572867896650")
request_json = request_data.json()

# Now Perform Operations
for a_key in data["M"]:
    for a_value in a_key["A"]:
        # Checking whether companydisplayname, basicprice and discounted price exists or not
        if "offers" in a_value and "companydisplayname" in a_value["offers"][0] and "basicprice" in a_value and "discountedprice" in a_value:
            print "Company Name : ",a_value["offers"][0]["companydisplayname"]
            print "Basic Price : ",a_value["basicprice"]
            print "Discounted Price : ",a_value["discountedprice"]

Output:

Company Name :  Topdanmark
Basic Price :  8360
Discounted Price :  7003
Company Name :  OK Forsikring
Basic Price :  6473
Discounted Price :  6473
Company Name :  GF Forsikring
Basic Price :  6737
Discounted Price :  6737
Company Name :  Nykredit Forsikring A/S
Basic Price :  5215
Discounted Price :  4707
Company Name :  Gjensidige Forsikring A/S
Basic Price :  5215
Discounted Price :  4707
Company Name :  If Skadeforsikring
Basic Price :  4938
Discounted Price :  4670
Company Name :  PenSam
Basic Price :  4691
Discounted Price :  4691
Company Name :  Runa Forsikring
Basic Price :  999999
Discounted Price :  3877
Company Name :  Bauta Forsikring
Basic Price :  999999
Discounted Price :  3877
Company Name :  Alm. Brand
Basic Price :  4252
Discounted Price :  3633
Company Name :  Alka Forsikring
Basic Price :  6151
Discounted Price :  6151
Company Name :  Tryg
Basic Price :  7324
Discounted Price :  5884
Company Name :  FDM Forsikring
Basic Price :  4227
Discounted Price :  4227
Company Name :  Lærerstandens Brandforsikring
Basic Price :  999999
Discounted Price :  3877

Hope this will help you out!!!

Upvotes: 1

Related Questions