CODEWITHSUNDEEP
python
nick lin
nick lin

Reputation: 1

Why doesn't soup.select('.r a') find anything in my google search?

I'm trying out the lucky.py project in this book, https://automatetheboringstuff.com/chapter11/. The program runs fine, but I can't get beautifulsoup to select the correct links.

What I've tried:

I've tried soup.select('div') and it chooses all the links from the top.

Tried soup.select('span div') and it selects all the sublinks on each search result.

Looked up lots over questions, but none of them seems to answer why soup.select('.r a') doesn't work or how to fix it.

When I enter print(linkElems) in the code, it shows me an empty dictionary.

This is my code:

#! /usr/bin/env python3
import requests, sys, webbrowser, bs4

print('Googling...')    # display text while downloading the Google page

res = requests.get('https://google.com/search?q=' + ' '.join(sys.argv[1:]))
res.raise_for_status()

soup = bs4.BeautifulSoup(res.text, features="html.parser")

linkElems = soup.select('.r a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('https://google.com' + linkElems[i].get('href'))

I'm expecting it to open the first 5 links of the google search in new tabs, but nothing comes up because the selector isn't working properly.

Upvotes: 0

Views: 669

Answers (1)

YoungMin Park
YoungMin Park

Reputation: 1189

It seems like class r (.r) means tag for one link.
If class r has only one a tag, multiple links can't be opened.

So, you might need to search upper tag like "div tag + id='search'", for example, div#search

Then, returned object will contain all a tags because element of "div#search" is located upper than all a tags

Upvotes: 0

Related Questions