CODEWITHSUNDEEP
pythonajaxweb-scrapingbeautifulsouppython-requests
Luke
Luke

Reputation: 51

How to scape this AJAX webpage with python?

I'm trying to scrape the player table on the Fantasy Football League website using python, but I'm a bit of a noob.

I'm aware that the site is called this external URL to retrieve the player data: https://fantasy.premierleague.com/api/bootstrap-static/

My problem is, this is pretty much an undocumented API.How would you suggest finding out how to call this URL to get back:

  1. Player
  2. Name
  3. Cost
  4. Sell
  5. Form
  6. Pts

SIDE-NOTE: I see a lot of people talking about this 'API' on reddit, so its obviously possible - I'm just to stupid to figure it out how to replicate the AJAX call that the site makes.

Thanks a lot!

Upvotes: 1

Views: 64

Answers (1)

Jack Fleeting
Jack Fleeting

Reputation: 24930

Try it this way:

import requests
import json
import pandas as pd

resp = requests.get('https://fantasy.premierleague.com/api/bootstrap-static/')
data = json.loads(resp.text)  #the response is in json format

players =[] #initialize a list of all players
for i in data['elements']:
    #the relevant info is hidden in here
    player = [] #initialize a list of relevant items for each player
    player.append(i['second_name'])
    cost = i['now_cost']/10
    player.append(cost)
    sel = float(i['selected_by_percent'])
    player.append(sel)
    player.append(i['form'])
    player.append(i['total_points'])
    players.append(player)

#now load the list into a dataframe
columns = ['Player','Cost','Sel.','Form','Pts.']
df = pd.DataFrame(players, columns=columns)
df

This should output the relevant information for all 557 players.

Upvotes: 3

Related Questions