CODEWITHSUNDEEP
pythonweb-crawlerweb-scrapingbeautifulsoup
torayeff
torayeff

Reputation: 9702

BeautifulSoup does not work for some web sites

I have this sript:

import urrlib2
from bs4 import BeautifulSoup
url = "http://www.shoptop.ru/"
page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)
divs = soup.findAll('a')
print divs

For this web site, it prints empty list? What can be problem? I am running on Ubuntu 12.04

Upvotes: 3

Views: 6189

Answers (2)

Surya Kasturi
Surya Kasturi

Reputation: 4731

Actually there are quite couple of bugs in BeautifulSoup which might raise some unknown errors. I had a similar issue when working on apache using lxml parser

So, just try to use other couple of parsers mentioned in the documentation

soup = BeautifulSoup(page, "html.parser")

This should work!

Upvotes: 3

Stefan
Stefan

Reputation: 3041

It looks like you have a few mistakes in your code urrlib2 should be urllib2, I've fixed the code for you and this works using BeautifulSoup 3

import urllib2
from BeautifulSoup import BeautifulSoup
url = "http://www.shoptop.ru/"
page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)
divs = soup.findAll('a')
print divs

Upvotes: -2

Related Questions