CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
gabboshow
gabboshow

Reputation: 5549

get data from MarketWatch

Using BeautifulSoup I am trying to scrape MarketWatch

from bs4 import BeautifulSoup
import requests
import pandas


url = "https://www.marketwatch.com/investing/stock/khc/profile"
# Make a GET request to fetch the raw HTML content
html_content = requests.get(url).text

soup = BeautifulSoup(html_content, "lxml")

Now I would like to extract the "P/E Current" and the "Price to Sales Ratio" that in the html are in the class

[...]
<div class="sixwide addgutter">

    <div class="block threewide addgutter">

        <h2>Valuation</h2>    
                <div class="section">    
            <p class="column">P/E Current</p>    
            <p class="data lastcolumn">19.27</p>    
        </div>    
                <div class="section">    
            <p class="column">P/E Ratio (with extraordinary items)</p>    
            <p class="data lastcolumn">19.55</p>    
        </div>    
                <div class="section">
            <p class="column">P/E Ratio (without extraordinary items)</p>
            <p class="data lastcolumn">20.00</p>
        </div>
                <div class="section">
            <p class="column">Price to Sales Ratio</p>
            <p class="data lastcolumn">1.55</p>
        </div>
                <div class="section">
            <p class="column">Price to Book Ratio</p>
            <p class="data lastcolumn">0.75</p>
        </div>
        [...]

How can I get them?

I use the command

section = soup.findAll('div', {'class' : 'section'})

but then I don't know how to go ahead to get the values I am interested in, can you help?

Upvotes: 1

Views: 1943

Answers (2)

0buz
0buz

Reputation: 3503

This solution will get data from all sections under div with class "sixwide addgutter". The result is in the form of list of dictionaries:

soup = BeautifulSoup(req.content, 'lxml')

base = soup.find('div', attrs={'class' : 'sixwide addgutter'})
section = base.find_all('div', attrs={'class' : 'section'})

all_data=[]
for item in section:
    data = {}
    data['name']=item.p.text
    data['value']=item.p.findNext('p').text
    all_data.append(data)

Output sample:

enter image description here

If you want the specifics, "P/E Current" and "Price to Sales Ratio" are all_data[0], all_data[3] respectively.

Upvotes: 1

Nico M&#252;ller
Nico M&#252;ller

Reputation: 1874

You can do the following

  1. find all sections
  2. loop through the sections and find its p elements
  3. if the p contains your search text, get its next sibling with the value.
url = "https://www.marketwatch.com/investing/stock/khc/profile"
html_content = requests.get(url).text

soup = BeautifulSoup(html_content, "lxml")
sections = soup.findAll('div', attrs={'class' : 'section'})
for section in sections : 
    ps = section.find('p')
    if "P/E Current" in ps.getText() or "Price to Sales Ratio" in ps.getText(): 
    val = ps.nextSibling.nextSibling
    print(f"{ps.getText()}: {val.getText()}")

OUT: P/E Current: 19.27
     Price to Sales Ratio: 1.55

Upvotes: 1

Related Questions