CODEWITHSUNDEEP
htmlpython-3.xweb-scrapingbeautifulsoup
Jure Stabuc
Jure Stabuc

Reputation: 569

Links missing when scraping with BeautifulSoup

I would like to get all player links from whoscored. The problem is, BS cannot find any link with the class player-link. What would be the best way to get these links? They exist somewhere as they show up in inspector.

This is my code so far:

import requests
from bs4 import BeautifulSoup as BS
import re
from incapsula import IncapSession
import json

session = IncapSession()
links = "https://www.whoscored.com/Matches/1080633/Live/England-Premier-League-2016-2017-Manchester-United-Arsenal"
try:
    response = session.get(links).text
except IncapBlocked as e:
    raise
soup = BS(response, "html.parser")
player_site = soup.find("a", text = "Player Statistics")
player_link = player_site.get("href")
player_link = "https://www.whoscored.com" + player_link
try:
    response_players = session.get(player_link).text
except IncapBlocked as e:
    raise
soup_players = BS(response_players, "html.parser")
soup_players.find_all("a", class_ ="player-link")

Thanks for helping.

EDIT:

Now I read that I get different result because the page is partially loaded via JavaScript. How can I resolve this?

Upvotes: 0

Views: 681

Answers (1)

Bill Bell
Bill Bell

Reputation: 21663

You would want to use selenium for this.

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get('https://www.whoscored.com/Matches/1080633/LiveStatistics/England-Premier-League-2016-2017-Manchester-United-Arsenal')
>>> links = driver.find_elements_by_xpath('.//a[@class="player-link"]')
>>> for link in links:
...     link.get_attribute('href')
...     
'https://www.whoscored.com/Players/79554/Show/David-de-Gea'
'https://www.whoscored.com/Players/18296/Show/Antonio-Valencia'
'https://www.whoscored.com/Players/70050/Show/Marcos-Rojo'
'https://www.whoscored.com/Players/81726/Show/Phil-Jones'
'https://www.whoscored.com/Players/23220/Show/Matteo-Darmian'
'https://www.whoscored.com/Players/97752/Show/Paul-Pogba'
'https://www.whoscored.com/Players/71174/Show/Ander-Herrera'
'https://www.whoscored.com/Players/2115/Show/Michael-Carrick'
'https://www.whoscored.com/Players/25363/Show/Juan-Mata'
'https://www.whoscored.com/Players/300299/Show/Marcus-Rashford'
'https://www.whoscored.com/Players/122366/Show/Anthony-Martial'
'https://www.whoscored.com/Players/29544/Show/Morgan-Schneiderlin'
'https://www.whoscored.com/Players/70033/Show/Daley-Blind'
'https://www.whoscored.com/Players/3859/Show/Wayne-Rooney'
'https://www.whoscored.com/Players/33831/Show/Sergio-Romero'
'https://www.whoscored.com/Players/110154/Show/Memphis-Depay'
'https://www.whoscored.com/Players/109000/Show/Jesse-Lingard'
'https://www.whoscored.com/Players/8166/Show/Ashley-Young'
'https://www.whoscored.com/Players/6775/Show/Petr-Cech'
'https://www.whoscored.com/Players/91879/Show/Carl-Jenkinson'
'https://www.whoscored.com/Players/30051/Show/Laurent-Koscielny'
'https://www.whoscored.com/Players/80921/Show/Shkodran-Mustafi'
'https://www.whoscored.com/Players/23072/Show/Nacho-Monreal'
'https://www.whoscored.com/Players/125209/Show/Mohamed-Elneny'
'https://www.whoscored.com/Players/69738/Show/Francis-Coquelin'
'https://www.whoscored.com/Players/13796/Show/Theo-Walcott'
'https://www.whoscored.com/Players/13756/Show/Mesut-%C3%96zil'
'https://www.whoscored.com/Players/26820/Show/Aaron-Ramsey'
'https://www.whoscored.com/Players/25244/Show/Alexis-S%C3%A1nchez'
'https://www.whoscored.com/Players/24444/Show/Olivier-Giroud'
'https://www.whoscored.com/Players/89401/Show/Granit-Xhaka'
'https://www.whoscored.com/Players/84146/Show/Alex-Oxlade-Chamberlain'
'https://www.whoscored.com/Players/76810/Show/Gabriel'
'https://www.whoscored.com/Players/36745/Show/David-Ospina'
'https://www.whoscored.com/Players/136824/Show/Alex-Iwobi'
'https://www.whoscored.com/Players/27550/Show/Kieran-Gibbs'

Upvotes: 1

Related Questions