CODEWITHSUNDEEP
pythonweb-scrapinglinkedin-api
Phuong Duyen Huynh Ngoc
Phuong Duyen Huynh Ngoc

Reputation: 520

scrape linkedin people search with python

I want to scrape the result of people search using linkedin. 

url='https://www.linkedin.com/search/results/people/?facetCurrentCompany=%5B%222525300%22%5D&facetGeoRegion=%5B%22fi%3A0%22%5D&keywords=python&origin=FACETED_SEARCH'
import bs4
import requests
res=requests.get(url)
soup=bs4.BeautifulSoup(res.text, 'lxml')

There is no error, but the problem is when I click on the link the result shows that there is one person matched my search criteria. and I cannot find that person in the soup result generated from Python code.

Does anyone know how to fix this?

Upvotes: 5

Views: 17383

Answers (3)

abhijithvijayan
abhijithvijayan

Reputation: 931

The Rest API method is not good for scraping, as it has several restrictions & limits.

Using Selenium for automation can scrape as much data as possible and even enables you to perform actions on LinkedIn.

For scraping I recommend using https://github.com/austinoboyle/scrape-linkedin-selenium. It covers most of the needs but has several bugs as of now(as LinkedIn updates their site frequently).

I am using modified version in a Flask Backend here

It's better to fork the library and use the scraping methods to your needs.

Upvotes: 1

NotSoShabby
NotSoShabby

Reputation: 3728

I would use an open source that already done the hard work and try to modify it to my needs. For example:

https://github.com/ericfourrier/scrape-linkedin

Note: this will only work for public data

Upvotes: 5

Tobias Lorenz
Tobias Lorenz

Reputation: 1448

You are trying to scrape data, which is available to logged in users only.

You should use the official LinkedIn REST API and authenticate via OAuth2. Give it a try: https://developer.linkedin.com/docs/rest-api

Upvotes: 7

Related Questions