Haytorade
Haytorade

Reputation: 165

Python Bing Search API only showing 30-results (should be 2,900)

I'm trying to find founders in the web3 space, and I want to use Linkedin as the source so that I can pull their bios into a CSV and filter by keywords. Since LinkedIn's API is expensive, I'm using Bing to find the profiles then navigating to them.

First, I define the functions for Bing search.

import requests
import csv

subscription_key = 'SOMEKEY'
endpoint = 'https://api.bing.microsoft.com' + "/v7.0/search"


def _bing_search_once(query, market, count, offset):
    params = {'q': query, 'mkt': market, 'count': count, 'offset': offset, 'responseFilter': 'Webpages'}
    headers = {'Ocp-Apim-Subscription-Key': subscription_key}

    try:
        response = requests.get(endpoint, headers=headers, params=params)
        response.raise_for_status()

        print("Headers:")
        print(response.headers)

        print("JSONResponse:")
        json_response = response.json()
        if 'webPages' not in json_response:
            return [], 0
        web_pages = json_response['webPages']
        web_pages_count = web_pages['totalEstimatedMatches']
        page_list = web_pages['value']
        return page_list, web_pages_count
    except Exception as ex:
        raise ex


def bing_search(query, max_count, market='en-us'):
    web_pages_count = 0
    offset = 0
    elements = []
    page_count = 50 # min(50, max_count - offset)
    while offset < max_count:
        page_list, results_count = _bing_search_once(query, market, page_count, offset)
        web_pages_count = results_count
        elements += page_list
        offset += page_count
        if web_pages_count <= len(elements):
            break
    return web_pages_count, elements

Then, I attempt to call bing_search with a query; I limit it to 500 results; and I save the results to a csv:

query =  'site:linkedin.com/in intitle:founder "web3" -agent -VC'

demanded_result_count = 500
print('Search for linkedin profiles')

web_pages_results, results = bing_search(query, demanded_result_count)

with open('./search_result.csv', 'w') as file:
    writer = csv.writer(file)
    writer.writerow(['name', 'url', 'snippet', 'language'])
    for result in results:
        row = [
            result['name'],
            result['url'],
            result['snippet'],
            result['language']
        ]
        writer.writerow(row)

It will only pull 30-results, no matter the query. Checking the above query manually (on bing.com) gave me over 2,900 results, but I still only get 30 through the API.

Am I doing something wrong?

Upvotes: 0

Views: 238

Answers (0)

Related Questions