CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoupgoogle-api
user12239916
user12239916

Reputation: 137

Webscraping Google Search Results Using Google API - Returns same result over and over again

My problem

Hi everyone

I am attempting to develop my very first web scraper using the Google API and Beautiful Soup in Python.

The aim is for the scraper to ask the user for an input, do a normal Google search and return a number of results in a nicely formatted way.

My current code succeeds in retrieving the data, but it only returns 1 result. When I try to do a loop on the results, in just keeps giving me the same result over and over again.

What I actually want is for the script to iterate over the different results on e.g. first page of Google search results and return all of these results. I am quite new to coding and I am guessing it's somewhere in my loop that I am making the mistake, but I guess it could also be something in the API?

I would very much appreciate some help here. Also the formatting of the code might be horrible, feel free to critique it so I can better at writing nice code :D

current code is

import sys
import urllib.request
import urllib.parse
import re
from urllib.request import urlopen as ureqs
from bs4 import BeautifulSoup as soup
from googleapiclient.discovery import build

# Google Personal Search Engine information
my_api_key = <key>
my_cse_id = <id>

# Google Search
def google_search(search_term, api_key, cse_id, **kwargs):
    service = build('customsearch', 'v1', developerKey=api_key)
    res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
    return res


# Setting up so that user can input query
query = input("enter the query\n")

# Getting into printing the results
results = google_search(query, my_api_key, my_cse_id)

print("\n*********Google Search Results*********\n")

for i in results:
    print("Title == " +results['items'][0]['title'])
    print("Link ==" +results['items'][0]['link'])
    snippet = results['items'][0]['snippet'].replace('\n', "")
    html_snippet = results['items'][0]['htmlSnippet'].replace('\n', "")
    html_snippet = html_snippet.replace("<b>", "")
    html_snippet = html_snippet.replace("</b>", "")
    html_snippet = html_snippet.replace("<br>", "")
    html_snippet = html_snippet.replace("&nbsp;…", ".")
    print("Description == " + snippet+html_snippet)

Upvotes: 1

Views: 668

Answers (1)

Olkin
Olkin

Reputation: 181

You iterate through results, but you never use object i inside the loop. Instead you always display results['items'][0]’s result

Upvotes: 0

Related Questions