CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Abhishek Pal
Abhishek Pal

Reputation: 51

Extracting links from html from the link of the following website

I want to extract the link

/stocks/company_info/stock_news.php?sc_id=CHC&scat=&pageno=2&next=0&durationType=Y&Year=2018&duration=1&news_type=

from the html of the page

http://www.moneycontrol.com/company-article/piramalenterprises/news/PH05#PH05

The following is the code that is used

url_list = "http://www.moneycontrol.com/company-article/piramalenterprises/news/PH05#PH05"
html = requests.get(url_list)
soup = BeautifulSoup(html.text,'html.parser')
link = soup.find_all('a')
print(link)

using beautiful soup. How would I go about it, using find_all('a") doesn't return the required link in the returned html.

Upvotes: 0

Views: 104

Answers (2)

KunduK
KunduK

Reputation: 33384

Please try this to get Exact Url you want.

import bs4 as bs
import requests
import re


sauce = requests.get('https://www.moneycontrol.com/stocks/company_info/stock_news.php?sc_id=CHC&durationType=Y&Year=2018')

soup = bs.BeautifulSoup(sauce.text, 'html.parser')

for a in soup.find_all('a', href=re.compile("company_info")):
   # print(a['href'])
    if 'pageno' in a['href']:
        print(a['href'])

output:

/stocks/company_info/stock_news.php?sc_id=CHC&scat=&pageno=2&next=0&durationType=Y&Year=2018&duration=1&news_type=
/stocks/company_info/stock_news.php?sc_id=CHC&scat=&pageno=3&next=0&durationType=Y&Year=2018&duration=1&news_type=

Upvotes: 1

Maaz
Maaz

Reputation: 2445

You just have to use the get method to find the href attribute:

from bs4 import BeautifulSoup as soup
import requests

url_list = "http://www.moneycontrol.com/company-article/piramalenterprises/news/PH05#PH05"
html = requests.get(url_list)
page= soup(html.text,'html.parser')
link = page.find_all('a')
for l in link:
    print(l.get('href'))

Upvotes: 1

Related Questions