CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
user11035529
user11035529

Reputation: 35

Use BeautifulSoup to scrape multiple websites

I try to get tickers for multiple websites using BeautifulSoup. I tried the following code with a loop but when I run the output it only gives me one ticker for one website: 

url = ['https://finance.yahoo.com/quote/AAPL/key-statistics/', 'https://finance.yahoo.com/quote/BOX/key-statistics/']

for pg in url: 
    page = requests.get(pg)

soup = BeautifulSoup(page.content, "html.parser")

ticker = soup.find("h1", attrs={"data-reactid":"7"}).text

ticker

Output:

Out[147]: 'BOX - Box, Inc.'

I then tried to use append function: 

data = [ ]
data.append(ticker)

but still gives me only one result. Anything wrong here?

Upvotes: 1

Views: 4484

Answers (2)

KunduK
KunduK

Reputation: 33384

Your code is perfect.What you have done you have kept soup outside the for loop hence it is taking only last url rather all urls. Now try this.

url = ['https://finance.yahoo.com/quote/AAPL/key-statistics/', 'https://finance.yahoo.com/quote/BOX/key-statistics/']

for pg in url:
    page = requests.get(pg)
    soup = BeautifulSoup(page.content, "html.parser")
    ticker = soup.find("h1", attrs={"data-reactid":"7"}).text
    print("Output :- " + ticker)

Output:-

Output :- AAPL - Apple Inc.
Output :- BOX - Box, Inc.

Upvotes: 1

Maaz
Maaz

Reputation: 2445

Your code is not correctly indent, but when I run this:

from bs4 import BeautifulSoup
import requests

url = ['https://finance.yahoo.com/quote/AAPL/key-statistics/', 'https://finance.yahoo.com/quote/BOX/key-statistics/']

data = []
for pg in url:
    page = requests.get(pg)
    soup = BeautifulSoup(page.content, "html.parser")
    ticker = soup.find("h1", attrs={"data-reactid":"7"}).text
    data.append(ticker)

print(data)

I get this:

['AAPL - Apple Inc.', 'BOX - Box, Inc.']

Upvotes: 3

Related Questions