CODEWITHSUNDEEP
pythonbeautifulsoupweb-crawler
Tasos
Tasos

Reputation: 7587

Crawler with BeautifulSoup

I am trying to create a web crawler for student research. I have already finish it, but I would like to tell me if the way I use is the best one. (probably it isn't :p)

The crawler is for the cnn site and the only thing I want to get, is the text of the news.

Here is an example link: link

Here is my code:

def cnn_crawler(link):
    req = urllib2.Request(link, headers={'User-Agent' : "Magic Browser"}) 
    usock = urllib2.urlopen(req)
    encoding = usock.headers.getparam('charset')
    page = usock.read().decode(encoding)
    usock.close()

    soup = BeautifulSoup(page)
    div = soup.find('div', attrs={'class': 'cnn_strycntntlft'})
    text = div.find_all('p')
    text.remove(soup.find('p', attrs={'class': 'cnn_strycbftrtxt'}))
    final = ""
    for entry in text:
            final = final + entry.get_text() + " "
    return final

Upvotes: 0

Views: 2608

Answers (1)

Gunjan
Gunjan

Reputation: 2805

You can try using Goose packege if its just for text extraction

https://github.com/grangier/python-goose

here is the link. it works perfect if u just need text

Upvotes: 1

Related Questions