CODEWITHSUNDEEP
pythonparsingbeautifulsouppython-requests
Hadi GhahremanNezhad
Hadi GhahremanNezhad

Reputation: 2455

How to retrieve yahoo search results?

I am trying to search Yahoo for a query using this code:

import requests
from bs4 import BeautifulSoup

query = "deep"
yahoo = "https://search.yahoo.com/search?q=" + query + "&n=" + str(10)
raw_page = requests.get(yahoo)

soup = BeautifulSoup(raw_page.text)

for link in soup.find_all(attrs={"class": "ac-algo fz-l ac-21th lh-24"}):
    print (link.text, link.get('href'))

But this does not work and the result is empty. How can I get 10 first search results?

Upvotes: 2

Views: 1091

Answers (2)

Nazim Kerimbekov
Nazim Kerimbekov

Reputation: 4783

Here are the main problems with your code:

  • When using Beautiful soup you should always include a parser (e.g. BeautifulSoup(raw_page.text, "lxml"))

  • You were searching for the wrong class, it's " ac-algo fz-l ac-21th lh-24" not "ac-algo fz-l ac-21th lh-24" (notice the space in the begining)

All in all your code should look like this:

import requests
from bs4 import BeautifulSoup

query = "deep"
yahoo = "https://search.yahoo.com/search?q=" + query + "&n=" + str(10)
raw_page = requests.get(yahoo)

soup = BeautifulSoup(raw_page.text, "lxml")
for link in soup.find_all(attrs={"class": " ac-algo fz-l ac-21th lh-24"}):
    print(link.text, link.get('href'))

Hope this helps

Upvotes: 2

KunduK
KunduK

Reputation: 33384

You can use Css Selector to find all links which is must faster.

import requests
from bs4 import BeautifulSoup

query = "deep"
yahoo = "https://search.yahoo.com/search?q=" + query + "&n=" + str(10)
raw_page = requests.get(yahoo)

soup = BeautifulSoup(raw_page.text,'lxml')

for link in soup.select(".ac-algo.fz-l.ac-21th.lh-24"):
    print (link.text, link['href'])

Output :

(Deep | Definition of Deep by Merriam-Webster', 'https://www.merriam-webster.com/dictionary/deep')
(Connecticut Department of Energy & Environmental Protection', 'https://www.ct.gov/deep/site/default.asp')
(Deep | Define Deep at Dictionary.com', 'https://www.dictionary.com/browse/deep')
(Deep - definition of deep by The Free Dictionary', 'https://www.thefreedictionary.com/deep')
(Deep (2017) - IMDb', 'https://www.imdb.com/title/tt4105584/')
(Deep Synonyms, Deep Antonyms | Merriam-Webster Thesaurus', 'https://www.merriam-webster.com/thesaurus/deep')
(Deep Synonyms, Deep Antonyms | Thesaurus.com', 'https://www.thesaurus.com/browse/deep')
(DEEP: Fishing - Connecticut', 'https://www.ct.gov/deep/cwp/view.asp?q=322708')
(Deep Deep Deep - YouTube', 'https://www.youtube.com/watch?v=oZhwagxWzOc')
(deep - English-Spanish Dictionary - WordReference.com', 'https://www.wordreference.com/es/translation.asp?tranword=deep')

Upvotes: 3

Related Questions