Reputation: 165
I'm trying to find founders in the web3 space, and I want to use Linkedin as the source so that I can pull their bios into a CSV and filter by keywords. Since LinkedIn's API is expensive, I'm using Bing to find the profiles then navigating to them.
First, I define the functions for Bing search.
import requests
import csv
subscription_key = 'SOMEKEY'
endpoint = 'https://api.bing.microsoft.com' + "/v7.0/search"
def _bing_search_once(query, market, count, offset):
params = {'q': query, 'mkt': market, 'count': count, 'offset': offset, 'responseFilter': 'Webpages'}
headers = {'Ocp-Apim-Subscription-Key': subscription_key}
try:
response = requests.get(endpoint, headers=headers, params=params)
response.raise_for_status()
print("Headers:")
print(response.headers)
print("JSONResponse:")
json_response = response.json()
if 'webPages' not in json_response:
return [], 0
web_pages = json_response['webPages']
web_pages_count = web_pages['totalEstimatedMatches']
page_list = web_pages['value']
return page_list, web_pages_count
except Exception as ex:
raise ex
def bing_search(query, max_count, market='en-us'):
web_pages_count = 0
offset = 0
elements = []
page_count = 50 # min(50, max_count - offset)
while offset < max_count:
page_list, results_count = _bing_search_once(query, market, page_count, offset)
web_pages_count = results_count
elements += page_list
offset += page_count
if web_pages_count <= len(elements):
break
return web_pages_count, elements
Then, I attempt to call bing_search with a query; I limit it to 500 results; and I save the results to a csv:
query = 'site:linkedin.com/in intitle:founder "web3" -agent -VC'
demanded_result_count = 500
print('Search for linkedin profiles')
web_pages_results, results = bing_search(query, demanded_result_count)
with open('./search_result.csv', 'w') as file:
writer = csv.writer(file)
writer.writerow(['name', 'url', 'snippet', 'language'])
for result in results:
row = [
result['name'],
result['url'],
result['snippet'],
result['language']
]
writer.writerow(row)
It will only pull 30-results, no matter the query. Checking the above query manually (on bing.com) gave me over 2,900 results, but I still only get 30 through the API.
Am I doing something wrong?
Upvotes: 0
Views: 238