CODEWITHSUNDEEP
pythonweb-scrapinggoogle-search-api
Auroshikha Dasgupta
Auroshikha Dasgupta

Reputation: 1

Is there any way to get more than 100 results at a time from a Google SERP API?

I've been stuck on this issue for so long. Basically I'm supposed to crawl throught the search results page and extract the urls of the first 10000 results. But with the APIs I can only get upto 100 at a time. I'm using Zenserp. Here is my code in Python:

import os
import requests
import csv
import json
import numpy as np
from bs4 import BeautifulSoup

headers = {"apikey": "xxxxxxx"}

params = (
   ("q","cat videos"),
   ("tbm","vid"),
   ("num","100"),
   ("start","100"),
);

response = requests.get('https://app.zenserp.com/api/v2/search', headers=headers, params=params);
output = response.text
print(output)

I've only been able to scrape 100 links. I'm supposed to find 10000 for my use case. Please help!

Upvotes: 0

Views: 1297

Answers (1)

Mathieu Delalandre
Mathieu Delalandre

Reputation: 16

Google has a scalable crawler able to analyse the entire web and to detect a large amount of content related to the queries of users (e.g. "cat videos" / 6.6 M of Search Engine Results “SERs”). However, these results are post-processed, filtered and aggregated to generate the Search Engine Result Pages “SERPs”. As a general trends only 20 to 30 pages (with 10 URLs per page) are returned to the user, whatever the SERs.

Examples of SERs vs. SERPs Google - 18th of June 2023

Two strategies could be applied for scalability

  1. to skip SERP API like Zenserp and to develop your own SERP scraper / crawler for a deep extraction (mainly to catch the omitted results). Here is an example in Python using the SEOquake tool http://mathieu.delalandre.free.fr/training/pgl-2022-2023.pdf

  2. Next, to handle multiple queries by extending / adapting your root query with the discovered URLs. Below is the list of the highest authority domains detected on the “cat videos” SERPs, combining “cat videos” with “en.wikipedia.org”, www.youtube.com, ….. “ascensionpress.com” will return deeper results on the considered domains.

List of the top 25 authority domains on the query "cat videos" Google - 26th of June 2023

Upvotes: 0

Related Questions