CODEWITHSUNDEEP
pythonbeautifulsoup
ponderwonder
ponderwonder

Reputation: 127

How to get a list of urls with BeautifulSoup from a href on a webpage

I am trying to get a list of report names and URLs on a webpage. The report name part works just fine. However, I can't get the list of URLs. Not sure what went wrong. Any ideas? Many thanks!

import requests
from bs4 import BeautifulSoup

link = 'https://www.globalpartnership.org/results/monitoring-evaluation'
res = requests.get(link)
soup = BeautifulSoup(res.text,'html.parser') 

reports = soup.find_all('ul',class_='three-col-list')

for report in reports:
    tag_url = report.find('li')
    url=tag_url.find('a')['href']

    print(report.text)
    print(url)

Upvotes: 1

Views: 59

Answers (1)

Yaakov Bressler
Yaakov Bressler

Reputation: 12018

Instead, iterate as follows:

reports = soup.find_all('ul',class_='three-col-list')

for report in reports:

    for tag_url in report.find_all('li'):
      url=tag_url.find("a",{"href":True})["href"]
      print(tag_url.text)
      print(url)

Upvotes: 1

Related Questions