CODEWITHSUNDEEP
web-scrapingscrapy
user3381431
user3381431

Reputation: 93

scrapy scrape data from interactive chart

I'm trying to scrape the data from this interactive chart which is located at the bottom of the website below: https://www.marsh.com/us/insights/research/global-insurance-market-index-q4-2020.html

I've used developer tools in chrome but cannot find the data points in the elements tab.

Would appreciate if someone can take a look and tell me if the data points are stored on the page somewhere.

Upvotes: 0

Views: 348

Answers (1)

Murat Demir
Murat Demir

Reputation: 716

The website is printing an excel file's data. So you don't have to try to find charts data output. I write a scraping script for you.

import scrapy,os,wget
from xlrd import open_workbook

class MarshSpider(scrapy.Spider):
    name = 'marsh'
    allowed_domains = ['www.marsh.com']
    start_urls = ['https://www.marsh.com/us/insights/research/global-insurance-market-index-q4-2020.html']

    def parse(self, response):
        xlsx_url = response.xpath('//div[contains(@class,"htmleditor")]//@data-csv-url').get() #Get the xlsx URL here
        main_url = "https://www.marsh.com"
        file = wget.download(main_url+xlsx_url) #download the url
        data = open_workbook(file) #open in workbook
        worksheet = data.sheet_by_index(0)

        for row in range(1,worksheet.nrows):
            yield{
                "Global Insurance Composite Renewal Rate":worksheet.cell(row,1).value,
                "Global Casualty Insurance Renewal Rate":worksheet.cell(row,2).value,
            }
        os.remove(file)

Upvotes: 1

Related Questions