CODEWITHSUNDEEP
python
Matthieu Mollard
Matthieu Mollard

Reputation: 43

Python - Get Result of Google Search


My goal is to create a small sript that find all the result of a google search but in "raw".
I don't speak english very well so i prefer to give an exemple to show you what i would like :
I Type : elephant
The script return
www.elephant.com
www.bluelephant.com
www.ebay.com/elephant
.....

I was thinking about urllib.request but the return value will not be usable to that ! I found some tutorials but not adapted at all to my wish ! Like i told you my goal is to have an .txt file as output witch contains alls the website who match with my query !
Thanks all

Upvotes: 2

Views: 11585

Answers (4)

payam.1991
payam.1991

Reputation: 91

I just add 2 points to "9th Dimension" answer.

  1. Use this guide to find your Custom Search Engine ID
  2. A small modification should be made in the second line of the code: as the following, the "version" should be added as an argument

service = build('customsearch','v1',developerKey= my_api_key)

Upvotes: 1

9th Dimension
9th Dimension

Reputation: 305

These seem to change often, so hopefully this answer remains useful for a little while...

First, you'll need to create a Google Custom Search, by visiting their site or following the instructions provided here https://developers.google.com/custom-search/docs/tutorial/creatingcse.

This will provide you with both

  • Custom Search Engine ID
  • API Key

credentials which are needed to use the service.

In your python script, you'll want to import the following package:

from googleapiclient.discovery import build

which will enable you to create a build object:

service = build("customsearch", developerKey=my_api_key)

According to the docs, this constructs a resource for interacting with the API.

When you want to return search results, call execute() on service's cse().list() method:

res = service.cse().list(q=my_search_keyword, cx=my_cse_id, **kwargs).execute()

to return a list of search results, where each result is a dictionary object. The i'th result's URL can be accessed with the "link" key:

ithresult = res[i]['link']

Note that you can only return 10 results in a single call, so make use of the start keyword argument in .list(), and consider embedding this call in a loop to generate several links at a time.

You should be able to find plenty of SO answers about saving your search results to a text file.

N.B. One more thing that confused me at first - presumably you'll want to search the entire web, and not just a single site. But when creating your CSE you will be asked to specify a single site, or list of sites, to search. Don't worry, just enter any old thing, you can delete it later. Even Google endorses this hack:

Convert a search engine to search the entire web: On the Custom Search home page, click the search engine you want. Click Setup, and then click the Basics tab. Select Search the entire web but emphasize included sites. In the Sites to search section, delete the site you entered during the initial setup process.

Upvotes: 2

Daerdemandt
Daerdemandt

Reputation: 2371

You have 2 options - using API or make a request like a browser does and then parse HTML.

  • First option is rather tricky to set up and is limited - 100 free queries/day, then 1000 for $5.

  • Second option is easier but it violates Google's ToS.

Upvotes: 0

Hui-Yu Lee
Hui-Yu Lee

Reputation: 989

One simple way is to make a request to google search, then parse the html result. You can use some Python libraries such us Beautiful Soup to parse the html content easily, finally get the url link you need.

Upvotes: 4

Related Questions