CODEWITHSUNDEEP
python-3.xbeautifulsoup
Abhijit
Abhijit

Reputation: 23

Web Scraping with BeautifulSoup by class returns empty list

I am trying to extract anchor tag inside ... (screenshot attached below) with BeautifulSoup but getting empty list while only anchor tag is working.

I read the BeautifulSoup documentation and tried the select() method and find_all() method but still giving empty list.

>>> import requests, webbrowser, bs4
>>> res = requests.get('https://www.google.com/search?q=beautiful+soup')
>>> soup = bs4.BeautifulSoup(res.text, 'html.parser')
>>> elems = soup.select('.r a')
>>> len(elems)
0
>>> elems = soup.select('a')
>>> len(elems)
68
>>> elems = soup.select('.r')
>>> len(elems)
0
>>> soup.find_all('a', class_='r')
[]
>>> soup.select('[class~=r]')
[]
>>> soup.find_all('a', class_='r')
[]
>>> soup.find_all('a', _class='r')
[]
>>> soup.find_all('a', {'class_':'r'})
[]
>>> soup.find_all('a', {'_class':'r'})
[]

Div with as class r

Upvotes: 1

Views: 127

Answers (1)

Dev Khadka
Dev Khadka

Reputation: 5471

It looks like google.com generates class name randomly, may be to discourage scraping. Your code works on other site

import requests, webbrowser, bs4
res = requests.get('https://html.com')
soup = bs4.BeautifulSoup(res.text, 'html.parser')
elems = soup.select('.post-single p')
len(elems)

Upvotes: 1

Related Questions