How to scrape data from HTML which changes with changing dropdown in python

Question

I have a page https://www.sportsbookreview.com/betting-odds/college-football/3771018/line-history/

Where I want to scrape line changes:

If I inspect the page I see that values are stored under span class="column-GaqNx" but I want to get these values only for Pinnacle not for others. How can I select it only to scrape this? As far as I see the only difference between elements in the dropdown is the img logo.

Thanks for the help.

Answer 1

The data is loaded dynamically via JavaScript, you can use this example to load the lines:

import json
import requests
from datetime import datetime, timedelta


url = r'https://www.sportsbookreview.com/ms-odds-v2/odds-v2-service?query={ openingLines(eid: 3771018, mtid: 401, paid: 20) lineHistory(eid: 3771018, mtid: 401, paid: 20, partid: [450, 1207]) { lines } }'
dt = timedelta(hours=4)  # adjust to your timezone

data = requests.get(url).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for d in data['data']['lineHistory']:
    t = datetime.fromtimestamp(int(d['lines'][0]['tim']) // 1000) - dt
    v1 = '{:>5} {:>5}'.format(d['lines'][0]['adj'], d['lines'][0]['ap'])
    v2 = '{:>5} {:>5}'.format(d['lines'][1]['adj'], d['lines'][1]['ap'])
    print('{:<20} {} {}'.format(str(t), v1, v2))

Prints:

2019-10-25 21:00:54  -10.5  -115  10.5   102
2019-10-25 20:58:29  -10.5  -108  10.5  -104
2019-10-25 20:52:01  -10.5  -111  10.5  -101
2019-10-25 20:50:54  -10.5  -114  10.5   101
2019-10-25 20:33:23  -10.5  -111  10.5  -101
2019-10-25 20:27:38  -10.5  -114  10.5   101
2019-10-25 20:17:45  -10.5  -110  10.5  -102
2019-10-25 20:15:19  -10.5  -112  10.5  -101
2019-10-25 20:12:21  -10.5  -108  10.5  -104
2019-10-25 20:11:27  -10.5  -114  10.5   102
2019-10-25 20:08:05   10.5   102 -10.5  -115
2019-10-25 20:06:12  -10.5  -111  10.5  -101
2019-10-25 20:05:40  -10.5  -115  10.5   102
2019-10-25 20:01:31  -10.5  -116  10.5   103
2019-10-25 19:56:14  -10.5  -112  10.5  -100
2019-10-25 19:49:26  -10.5  -113  10.5   100
2019-10-25 19:48:19  -10.5  -111  10.5  -101
2019-10-25 19:46:44  -10.5  -109  10.5  -103
2019-10-25 19:45:45  -10.5  -116  10.5   103
2019-10-25 19:43:39  -10.5  -112  10.5  -100
2019-10-25 19:11:31  -10.5  -108  10.5  -104
2019-10-25 19:04:48  -10.5  -107  10.5  -105
2019-10-25 18:55:59  -10.5  -109  10.5  -103
2019-10-25 18:53:24  -10.5  -108  10.5  -104
2019-10-25 18:27:45  -10.5  -115  10.5   102
2019-10-25 18:23:47  -10.5  -111  10.5  -101
2019-10-25 18:17:19   10.5  -101 -10.5  -112
2019-10-25 18:08:02  -10.5  -108  10.5  -104
2019-10-25 18:05:53  -10.5  -109  10.5  -103
2019-10-25 17:52:16  -10.5  -111  10.5  -101
2019-10-25 17:50:19   10.5   100 -10.5  -113
2019-10-25 17:48:36  -10.5  -113  10.5   101
2019-10-25 17:43:48  -10.5  -109  10.5  -103
2019-10-25 17:23:02  -10.5  -111  10.5  -101
2019-10-25 17:17:11  -10.5  -107  10.5  -105
2019-10-25 16:56:19  -10.5  -109  10.5  -104
2019-10-25 16:45:20    -10  -113    10   101
2019-10-25 16:41:04    -10  -112    10  -100
2019-10-25 16:38:51    -10  -108    10  -104
2019-10-25 16:28:00     10  -102   -10  -110
2019-10-25 16:15:03    -10  -110    10  -103
2019-10-25 16:13:03    -10  -108    10  -104
2019-10-25 15:41:00   11.5  -114 -11.5   102
2019-10-25 14:06:05  -11.5   102  11.5  -115
2019-10-25 13:00:02  -11.5  -106  11.5  -106
2019-10-25 12:40:16  -11.5  -107  11.5  -105
2019-10-25 12:36:47  -11.5  -110  11.5  -102
2019-10-25 11:35:33  -11.5  -102  11.5  -110
2019-10-25 10:48:21  -11.5  -101  11.5  -112
2019-10-25 09:51:31  -11.5   102  11.5  -114
2019-10-25 08:37:57  -11.5   103  11.5  -115
2019-10-25 05:27:04  -11.5  -101  11.5  -111
2019-10-25 00:07:42  -11.5  -104  11.5  -108
2019-10-24 19:49:35  -11.5  -106  11.5  -106
2019-10-24 19:46:51  -11.5  -107  11.5  -105
2019-10-24 19:45:27   13.5  -112 -13.5  -101
2019-10-24 19:43:28  -13.5  -101  13.5  -111
2019-10-24 17:05:15  -13.5  -102  13.5  -110
2019-10-24 16:32:13  -13.5  -103  13.5  -109
2019-10-24 16:29:46  -13.5  -107  13.5  -105
2019-10-24 15:40:49  -13.5  -111  13.5  -102
2019-10-24 11:41:56  -13.5  -110 -13.5  -111
2019-10-24 11:08:13   13.5  -102 -13.5  -110
2019-10-24 09:02:09  -13.5  -110  13.5  -103
2019-10-24 07:51:44  -13.5  -109  13.5  -103
2019-10-24 01:11:00  -13.5  -110  13.5  -103
2019-10-23 19:42:54   13.5  -103 -13.5  -109
2019-10-23 19:27:57  -13.5  -108  13.5  -104
2019-10-23 11:25:49  -13.5  -109  13.5  -103
2019-10-23 08:39:15  -13.5  -108  13.5  -104
2019-10-23 08:04:36  -13.5  -107  13.5  -105
2019-10-22 15:20:45  -13.5  -108  13.5  -106
2019-10-22 13:27:57    -14  -100    14  -114
2019-10-22 11:27:13    -14  -102    14  -112
2019-10-22 10:33:42    -13  -108    13  -106
2019-10-21 16:52:31    -13  -106    13  -108
2019-10-21 08:46:35    -13  -107    13  -107
2019-10-21 08:24:44    -13  -108    13  -108
2019-10-20 19:32:07    -12  -119    12   102
2019-10-20 18:11:56    -12  -108    12  -108