CODEWITHSUNDEEP
pythonpython-2.7beautifulsoup
wan mohd payed
wan mohd payed

Reputation: 171

How to get the link inside the li tag?

I have this code:

import urllib
from bs4 import BeautifulSoup
url = "http://download.cnet.com/windows/"
pageHtml = urllib.urlopen(url)
soup = BeautifulSoup(pageHtml)
for a in soup.select("div.catFlyout a[href]"):
    print "http://download.cnet.com"+a["href"]

But this code did not give the correct output. The correct output should be like this:

http://download.cnet.com/windows/security-software/
http://download.cnet.com/windows/browsers/
http://download.cnet.com/windows/business-software/
..
..
http://download.cnet.com/windows/video-software/

Upvotes: 1

Views: 92

Answers (1)

alecxe
alecxe

Reputation: 473863

There are some relative and absolute links in the list, prepend base url only if the link starts with http:

for a in soup.select("div.catFlyout a[href]"):
    if not a["href"].startswith("http"):
        print "http://download.cnet.com"+a["href"]
    else:
        print a["href"]

Or, use urlparse to check if link is absolute or not (taken from here):

import urllib
import urlparse
from bs4 import BeautifulSoup

def is_absolute(url):
    return bool(urlparse.urlparse(url).scheme)

url = "http://download.cnet.com/windows/"
pageHtml = urllib.urlopen(url)
soup = BeautifulSoup(pageHtml)
for a in soup.select("div.catFlyout a[href]"):
    if not is_absolute(a['href']):
        print "http://download.cnet.com"+a["href"]
    else:
        print a["href"]

Upvotes: 1

Related Questions