CODEWITHSUNDEEP
pythonpython-3.xweb-scraping
zeroFighter
zeroFighter

Reputation: 123

Get all the urls from the internet related to a given keyword using Python

I am learning scraping using BeautifulSoup. What I want to do is to get all available urls related to a keyword from the internet.

Is there any way to do that?

Upvotes: 0

Views: 2007

Answers (1)

Himanshu dua
Himanshu dua

Reputation: 2523

You can use requests python library to search on google. Install requests by pip install requests You can use google to search anything and parse the result with Beautifulsoup. Following code search the query on google and after that BeautifulSoup is used to get the URLs that Google returned.

import requests
import urllib
from bs4 import BeautifulSoup

query = 'any search term'


r = requests.get('https://www.google.com/search?q={}'.format(query))
soup = BeautifulSoup(r.text, "html.parser")
links = []
for item in soup.find_all('h3', attrs={'class' : 'r'}):
    links.append(item.a['href'])

print(links)

Upvotes: 3

Related Questions