pynewbpy44
Reputation: 3
Python/BeautifulSoup only returning first table item
I am trying to scrape the data from a website but the below code will only pull the first row from the website's table despite being in a for loop. What am I missing?
import requests
from bs4 import BeautifulSoup
import pandas
import xlsxwriter
r = requests.get("https://www.fantasypros.com/nfl/stats/qb.php")
c = r.content
soup=BeautifulSoup(c, "html.parser")
all=soup.find_all("div",{"class":"mobile-table double-header"})
l=[]
for item in all:
d={}
d["Player"] = (item.find("a",{"class","player-name"}).text.strip())
l.append(d)
df=pandas.DataFrame(l)
df.to_csv("Output.csv")
Upvotes: 0
Views: 316
Answers (3)
chitown88
Reputation: 28565
Let pandas do the work.
import pandas as pd
url = 'https://www.fantasypros.com/nfl/stats/qb.php'
df = pd.read_html(url, header=1)[0]
df[['Player','Team']] = df['Player'].str.extract('(.*)\((.*)\)', expand=True)
df['Player'] = df['Player'].apply(lambda x: x.strip())
Output:
print(df.to_markdown(index=False))
| Rank | Player | CMP | ATT | PCT | YDS | Y/A | TD | INT | SACKS | ATT.1 | YDS.1 | TD.1 | FL | G | FPTS | FPTS/G | ROST | Team |
|-------:|:-------------------|------:|------:|------:|------:|------:|-----:|------:|--------:|--------:|--------:|-------:|-----:|----:|-------:|---------:|:-------|:-------|
| 1 | Josh Allen | 409 | 646 | 63.3 | 4407 | 6.8 | 36 | 15 | 26 | 122 | 763 | 6 | 3 | 17 | 417.7 | 24.6 | 99.9% | BUF |
| 2 | Justin Herbert | 443 | 672 | 65.9 | 5014 | 7.5 | 38 | 15 | 31 | 63 | 302 | 3 | 1 | 17 | 395.6 | 23.3 | 99.8% | LAC |
| 3 | Tom Brady | 485 | 719 | 67.5 | 5316 | 7.4 | 43 | 12 | 22 | 28 | 81 | 2 | 3 | 17 | 386.7 | 22.7 | 96.1% | TB |
| 4 | Patrick Mahomes II | 436 | 658 | 66.3 | 4828 | 7.3 | 37 | 13 | 28 | 66 | 381 | 2 | 4 | 17 | 374.2 | 22 | 99.9% | KC |
| 5 | Matthew Stafford | 404 | 601 | 67.2 | 4886 | 8.1 | 41 | 17 | 30 | 32 | 43 | 0 | 2 | 17 | 346.8 | 20.4 | 90.4% | LAR |
| 6 | Aaron Rodgers | 366 | 531 | 68.9 | 4115 | 7.7 | 37 | 4 | 30 | 33 | 101 | 3 | 0 | 16 | 336.3 | 21 | 93.8% | GB |
| 7 | Dak Prescott | 410 | 596 | 68.8 | 4449 | 7.5 | 37 | 10 | 30 | 48 | 146 | 1 | 6 | 16 | 330.4 | 20.7 | 93.9% | DAL |
| 8 | Joe Burrow | 366 | 520 | 70.4 | 4611 | 8.9 | 34 | 14 | 51 | 40 | 118 | 2 | 2 | 16 | 328.1 | 20.5 | 95.7% | CIN |
| 9 | Jalen Hurts | 265 | 432 | 61.3 | 3144 | 7.3 | 16 | 9 | 26 | 139 | 784 | 10 | 2 | 15 | 321.2 | 21.4 | 96.5% | PHI |
| 10 | Kyler Murray | 333 | 481 | 69.2 | 3787 | 7.9 | 24 | 10 | 31 | 88 | 423 | 5 | 0 | 14 | 310.5 | 22.2 | 97.3% | ARI |
| 11 | Kirk Cousins | 372 | 561 | 66.3 | 4221 | 7.5 | 33 | 7 | 28 | 29 | 115 | 1 | 2 | 16 | 307.3 | 19.2 | 76.1% | MIN |
| 12 | Ryan Tannehill | 357 | 531 | 67.2 | 3734 | 7 | 21 | 14 | 47 | 55 | 270 | 7 | 4 | 17 | 282.3 | 16.6 | 17.1% | TEN |
| 13 | Derek Carr | 428 | 626 | 68.4 | 4804 | 7.7 | 23 | 14 | 40 | 40 | 108 | 0 | 5 | 17 | 270.6 | 15.9 | 85.7% | LV |
| 14 | Carson Wentz | 322 | 516 | 62.4 | 3563 | 6.9 | 27 | 7 | 32 | 57 | 215 | 1 | 5 | 17 | 264.9 | 15.6 | 12.4% | WAS |
| 15 | Lamar Jackson | 246 | 382 | 64.4 | 2882 | 7.5 | 16 | 13 | 38 | 134 | 767 | 2 | 3 | 12 | 253 | 21.1 | 99.5% | BAL |
| 16 | Russell Wilson | 259 | 400 | 64.8 | 3113 | 7.8 | 25 | 6 | 33 | 43 | 183 | 2 | 1 | 14 | 248.7 | 17.8 | 93.1% | DEN |
| 17 | Jimmy Garoppolo | 301 | 441 | 68.3 | 3810 | 8.6 | 20 | 12 | 29 | 38 | 51 | 3 | 3 | 15 | 239.5 | 16 | 3.1% | SF |
| 18 | Mac Jones | 352 | 521 | 67.6 | 3801 | 7.3 | 22 | 13 | 28 | 44 | 129 | 0 | 3 | 17 | 238 | 14 | 28.3% | NE |
| 19 | Taylor Heinicke | 321 | 494 | 65 | 3419 | 6.9 | 20 | 15 | 38 | 60 | 313 | 1 | 2 | 16 | 236.7 | 14.8 | 0.6% | WAS |
| 20 | Matt Ryan | 375 | 560 | 67 | 3968 | 7.1 | 20 | 12 | 40 | 40 | 82 | 1 | 4 | 17 | 234.8 | 13.8 | 34.1% | IND |
| 21 | Ben Roethlisberger | 390 | 605 | 64.5 | 3740 | 6.2 | 22 | 10 | 38 | 20 | 5 | 1 | 5 | 16 | 228 | 14.3 | 13.6% | FA |
| 22 | Trevor Lawrence | 359 | 602 | 59.6 | 3641 | 6 | 12 | 17 | 32 | 73 | 334 | 2 | 5 | 17 | 216 | 12.7 | 49.0% | JAC |
| 23 | Teddy Bridgewater | 285 | 426 | 66.9 | 3052 | 7.2 | 18 | 7 | 31 | 30 | 106 | 2 | 1 | 14 | 209.8 | 15 | 0.8% | MIA |
| 24 | Jared Goff | 332 | 494 | 67.2 | 3245 | 6.6 | 19 | 8 | 35 | 17 | 87 | 0 | 6 | 14 | 202.4 | 14.5 | 13.2% | DET |
| 25 | Baker Mayfield | 253 | 418 | 60.5 | 3010 | 7.2 | 17 | 13 | 43 | 37 | 134 | 1 | 3 | 14 | 193.8 | 13.8 | 8.3% | CAR |
| 26 | Tua Tagovailoa | 263 | 388 | 67.8 | 2653 | 6.8 | 16 | 10 | 20 | 42 | 128 | 3 | 1 | 13 | 190.9 | 14.7 | 61.9% | MIA |
| 27 | Daniel Jones | 232 | 361 | 64.3 | 2428 | 6.7 | 10 | 7 | 22 | 62 | 298 | 2 | 3 | 11 | 173.6 | 15.8 | 17.5% | NYG |
| 28 | Sam Darnold | 243 | 406 | 59.9 | 2527 | 6.2 | 9 | 13 | 35 | 48 | 222 | 5 | 4 | 12 | 170.5 | 14.2 | 1.8% | CAR |
| 29 | Davis Mills | 263 | 394 | 66.8 | 2664 | 6.8 | 16 | 10 | 31 | 18 | 44 | 0 | 1 | 13 | 167 | 12.8 | 5.0% | HOU |
| 30 | Zach Wilson | 213 | 383 | 55.6 | 2334 | 6.1 | 9 | 11 | 44 | 29 | 185 | 4 | 1 | 13 | 162.9 | 12.5 | 12.2% | NYJ |
| 31 | Justin Fields | 159 | 270 | 58.9 | 1870 | 6.9 | 7 | 10 | 36 | 72 | 420 | 2 | 5 | 12 | 136.9 | 11.4 | 44.2% | CHI |
| 32 | Taysom Hill | 78 | 134 | 58.2 | 978 | 7.3 | 4 | 5 | 9 | 70 | 374 | 5 | 0 | 12 | 122.8 | 10.2 | 3.7% | NO |
| 33 | Jameis Winston | 95 | 161 | 59 | 1170 | 7.3 | 14 | 3 | 11 | 32 | 166 | 1 | 1 | 7 | 120.3 | 17.2 | 29.4% | NO |
| 34 | Trevor Siemian | 108 | 188 | 57.4 | 1154 | 6.1 | 11 | 3 | 9 | 9 | 20 | 1 | 1 | 6 | 93.2 | 15.5 | 0.1% | CHI |
| 35 | Andy Dalton | 149 | 236 | 63.1 | 1515 | 6.4 | 8 | 9 | 18 | 16 | 76 | 0 | 1 | 8 | 93.1 | 11.6 | 0.6% | NO |
| 36 | Cam Newton | 69 | 126 | 54.8 | 684 | 5.4 | 4 | 5 | 10 | 47 | 230 | 5 | 1 | 8 | 91.3 | 11.4 | 9.1% | FA |
| 37 | Tyrod Taylor | 91 | 150 | 60.7 | 966 | 6.4 | 5 | 5 | 13 | 19 | 151 | 3 | 0 | 6 | 86.7 | 14.5 | 0.6% | NYG |
| 38 | Tyler Huntley | 122 | 188 | 64.9 | 1081 | 5.8 | 3 | 4 | 18 | 47 | 294 | 2 | 3 | 7 | 86.7 | 12.4 | 0.7% | BAL |
| 39 | Jacoby Brissett | 141 | 225 | 62.7 | 1283 | 5.7 | 5 | 4 | 19 | 19 | 70 | 1 | 3 | 11 | 76.4 | 6.9 | 2.0% | CLE |
| 40 | Trey Lance | 41 | 71 | 57.7 | 603 | 8.5 | 5 | 2 | 4 | 38 | 168 | 1 | 0 | 6 | 67 | 11.2 | 87.4% | SF |
| 41 | Geno Smith | 65 | 95 | 68.4 | 702 | 7.4 | 5 | 1 | 13 | 9 | 42 | 1 | 1 | 4 | 55.3 | 13.8 | 1.7% | SEA |
| 42 | Drew Lock | 67 | 111 | 60.4 | 787 | 7.1 | 2 | 2 | 9 | 10 | 53 | 2 | 1 | 6 | 52.8 | 8.8 | 2.3% | SEA |
| 43 | Mike White | 88 | 132 | 66.7 | 953 | 7.2 | 5 | 8 | 4 | 5 | -1 | 0 | 0 | 4 | 52 | 13 | 0.6% | NYJ |
| 44 | Josh Johnson | 57 | 85 | 67.1 | 638 | 7.5 | 5 | 2 | 3 | 9 | 28 | 0 | 0 | 4 | 48.4 | 12.1 | 0.1% | DEN |
| 45 | Colt McCoy | 74 | 99 | 74.7 | 740 | 7.5 | 3 | 1 | 6 | 22 | 37 | 0 | 1 | 8 | 42.3 | 5.3 | 0.1% | ARI |
| 46 | Mike Glennon | 90 | 167 | 53.9 | 790 | 4.7 | 4 | 10 | 9 | 7 | 33 | 1 | 3 | 6 | 40.9 | 6.8 | 0.6% | FA |
| 47 | Gardner Minshew II | 41 | 60 | 68.3 | 439 | 7.3 | 4 | 1 | 5 | 9 | 21 | 0 | 0 | 3 | 34.6 | 11.5 | 0.7% | PHI |
| 48 | Tim Boyle | 61 | 94 | 64.9 | 526 | 5.6 | 3 | 6 | 0 | 2 | 13 | 0 | 0 | 4 | 30.4 | 7.6 | 0.1% | DET |
| 49 | Case Keenum | 47 | 72 | 65.3 | 462 | 6.4 | 3 | 1 | 5 | 12 | 22 | 0 | 1 | 5 | 29.6 | 5.9 | 0.1% | BUF |
| 50 | Cooper Rush | 30 | 47 | 63.8 | 422 | 9 | 3 | 1 | 3 | 9 | -8 | 0 | 1 | 4 | 25.1 | 6.3 | 0.1% | DAL |
| 51 | Jordan Love | 36 | 62 | 58.1 | 411 | 6.6 | 2 | 3 | 3 | 12 | 27 | 0 | 1 | 6 | 24.2 | 4 | 0.8% | GB |
| 52 | Joe Flacco | 27 | 42 | 64.3 | 338 | 8 | 3 | 0 | 2 | 2 | 3 | 0 | 1 | 2 | 23.8 | 11.9 | 0.7% | NYJ |
| 53 | Mason Rudolph | 35 | 58 | 60.3 | 277 | 4.8 | 1 | 1 | 0 | 5 | 53 | 0 | 0 | 2 | 19.4 | 9.7 | 0.6% | PIT |
| 54 | PJ Walker | 36 | 66 | 54.5 | 362 | 5.5 | 1 | 3 | 7 | 7 | 13 | 0 | 1 | 5 | 16.8 | 3.4 | 0.0% | CAR |
| 55 | Nick Foles | 24 | 35 | 68.6 | 250 | 7.1 | 1 | 0 | 4 | 4 | 8 | 0 | 0 | 1 | 16.8 | 16.8 | 0.1% | IND |
| 56 | Marcus Mariota | 1 | 2 | 50 | 4 | 2 | 0 | 0 | 0 | 13 | 87 | 1 | 0 | 7 | 14.9 | 2.1 | 5.8% | ATL |
| 57 | Jake Fromm | 27 | 60 | 45 | 210 | 3.5 | 1 | 3 | 6 | 8 | 65 | 0 | 1 | 3 | 13.9 | 4.6 | 1.0% | FA |
| 58 | Brandon Allen | 17 | 34 | 50 | 149 | 4.4 | 2 | 0 | 4 | 7 | -1 | 0 | 0 | 5 | 13.8 | 2.8 | 0.1% | CIN |
| 59 | Sean Mannion | 22 | 36 | 61.1 | 189 | 5.3 | 1 | 0 | 2 | 2 | 14 | 0 | 0 | 1 | 13 | 13 | 0.1% | MIN |
| 60 | Brian Hoyer | 9 | 11 | 81.8 | 227 | 20.6 | 1 | 0 | 0 | 11 | -8 | 0 | 0 | 5 | 12.3 | 2.5 | 0.0% | NE |
| 61 | Nick Mullens | 20 | 30 | 66.7 | 147 | 4.9 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 9.9 | 9.9 | 0.1% | LV |
| 62 | Mitch Trubisky | 6 | 8 | 75 | 43 | 5.4 | 0 | 1 | 0 | 13 | 24 | 1 | 0 | 5 | 9.1 | 1.8 | 4.2% | PIT |
| 63 | Kyle Allen | 12 | 19 | 63.2 | 120 | 6.3 | 1 | 0 | 2 | 2 | 11 | 0 | 1 | 2 | 7.9 | 4 | 0.1% | HOU |
| 64 | Garrett Gilbert | 20 | 31 | 64.5 | 194 | 6.3 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | 1 | 7.8 | 7.8 | 0.0% | FA |
| 65 | Ian Book | 12 | 20 | 60 | 135 | 6.8 | 0 | 2 | 8 | 3 | 6 | 0 | 0 | 1 | 4 | 4 | 0.0% | NO |
| 66 | Chad Henne | 11 | 16 | 68.8 | 82 | 5.1 | 0 | 0 | 0 | 8 | 0 | 0 | 0 | 4 | 3.3 | 0.8 | 0.1% | KC |
| 67 | Chris Streveler | 6 | 9 | 66.7 | 36 | 4 | 0 | 0 | 2 | 3 | 6 | 0 | 0 | 2 | 2 | 1 | 0.0% | NYJ |
| 68 | Blaine Gabbert | 7 | 11 | 63.6 | 67 | 6.1 | 0 | 0 | 1 | 9 | -7 | 0 | 0 | 6 | 1.9 | 0.3 | 0.1% | TB |
| 69 | C.J. Beathard | 2 | 2 | 100 | 33 | 16.5 | 0 | 0 | 0 | 1 | 2 | 0 | 0 | 2 | 1.5 | 0.8 | 0.0% | JAC |
| 70 | Sam Ehlinger | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 9 | 0 | 0 | 3 | 0.9 | 0.3 | 0.1% | IND |
| 71 | Ryan Fitzpatrick | 3 | 6 | 50 | 13 | 2.2 | 0 | 0 | 1 | 1 | 2 | 0 | 0 | 1 | 0.7 | 0.7 | 0.1% | FA |
| 72 | David Blough | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 6 | 0 | 0 | 1 | 0.6 | 0.6 | 0.0% | DET |
| 73 | Nathan Peterman | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 2 | 0 | 0 | 1 | 0.2 | 0.2 | 0.1% | CHI |
| 74 | Kellen Mond | 2 | 3 | 66.7 | 5 | 1.7 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0.2 | 0.1 | 0.2% | MIN |
| 75 | Stone Smartt | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | LAC |
| 76 | Jared Bernhardt | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | ATL |
| 77 | Chris Oladokun | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | PIT |
| 78 | Bryce Perkins | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | LAR |
| 79 | Shane Buechele | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | KC |
| 80 | Jacob Eason | 2 | 5 | 40 | 25 | 5 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0.0% | SEA |
| 81 | Jarrett Stidham | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | LV |
| 82 | John Lovett | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | MIA |
| 83 | Desmond Ridder | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2.1% | ATL |
| 84 | Reid Sinnett | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | PHI |
| 85 | Jack Coan | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | IND |
| 86 | Ben DiNucci | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | DAL |
| 87 | Jarrett Guarantano | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | ARI |
| 88 | Kyle Trask | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.2% | TB |
| 89 | K.J. Costello | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | NO |
| 90 | Brock Purdy | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | SF |
| 91 | Carson Strong | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | PHI |
| 92 | Sam Howell | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.7% | WAS |
| 93 | E.J. Perry | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | JAC |
| 94 | Skylar Thompson | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | MIA |
| 95 | Drew Plitt | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | CIN |
| 96 | Armani Rogers | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | WAS |
| 97 | Malik Willis | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1.7% | TEN |
| 98 | Bailey Zappe | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | NE |
| 99 | Dustin Crum | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | KC |
| 100 | Anthony Brown | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | BAL |
| 101 | Matt Corral | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.8% | CAR |
| 102 | Jake Browning | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | CIN |
| 103 | Kenny Pickett | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 5.2% | PIT |
| 104 | Chase Garbers | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | LV |
| 105 | Erik Ainge | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | NYJ |
| 106 | Hunter Cantwell | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | BAL |
| 107 | Ben Chappell | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | WAS |
| 108 | Adam Froman | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | ATL |
| 109 | Trace McSorley | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0.3% | ARI |
| 110 | Keith Null | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | CAR |
| 111 | Brett Ratliff | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | TEN |
| 112 | Brian Brohm | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | BUF |
| 113 | Joshua Dobbs | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.1% | CLE |
| 114 | Deshaun Watson | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 39.0% | CLE |
| 115 | Nate Sudfeld | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | SF |
| 116 | Will Grier | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | DAL |
| 117 | Brett Rypien | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0.1% | DEN |
| 118 | Easton Stick | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | LAC |
| 119 | Danny Etling | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | GB |
| 120 | Matt Barkley | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | BUF |
| 121 | Ryan Griffin | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | TB |
| 122 | Brett Smith | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | TB |
| 123 | Jeff Driskel | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0.0% | HOU |
| 124 | Kurt Benkert | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | -1 | 0 | 0 | 1 | -0.1 | -0.1 | 0.0% | FA |
| 125 | Chase Daniel | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | -2 | 0 | 0 | 1 | -0.2 | -0.2 | 0.1% | LAC |
| 126 | Davis Webb | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | -3 | 0 | 0 | 1 | -0.3 | -0.3 | 0.0% | NYG |
| 127 | Feleipe Franks | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 3 | 6 | 0 | 0 | 6 | -0.4 | -0.1 | 0.1% | ATL |
| 128 | Logan Woodside | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 6 | -6 | 0 | 0 | 4 | -0.6 | -0.2 | 0.0% | TEN |
| 129 | John Wolford | 1 | 4 | 25 | 5 | 1.3 | 0 | 1 | 1 | 2 | -1 | 0 | 0 | 2 | -0.9 | -0.5 | 0.1% | LAR |
| 130 | Josh Rosen | 2 | 11 | 18.2 | 19 | 1.7 | 0 | 2 | 0 | 0 | 0 | 0 | 0 | 3 | -1.2 | -0.4 | 0.0% | CLE |
Upvotes: 0
Barry the Platipus
Reputation: 10460
Of course, you can also do something like this:
import requests
from bs4 import BeautifulSoup
import pandas as pd
url = 'https://www.fantasypros.com/nfl/stats/qb.php'
r = requests.get(url)
soup = BeautifulSoup(r.text)
table = soup.select('table#data')
df = pd.read_html(str(table))[0]
print(df)
This returns that table in full:
Unnamed: 0_level_0 Unnamed: 1_level_0 PASSING RUSHING MISC
Rank Player CMP ATT PCT YDS Y/A TD INT SACKS ATT YDS TD FL G FPTS FPTS/G ROST
0 1 Josh Allen (BUF) 409 646 63.3 4407 6.8 36 15 26 122 763 6 3 17 417.7 24.6 99.9%
1 2 Justin Herbert (LAC) 443 672 65.9 5014 7.5 38 15 31 63 302 3 1 17 395.6 23.3 99.8%
2 3 Tom Brady (TB) 485 719 67.5 5316 7.4 43 12 22 28 81 2 3 17 386.7 22.7 96.1%
3 4 Patrick Mahomes II (KC) 436 658 66.3 4828 7.3 37 13 28 66 381 2 4 17 374.2 22.0 99.9%
4 5 Matthew Stafford (LAR) 404 601 67.2 4886 8.1 41 17 30 32 43 0 2 17 346.8 20.4 90.4%
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
Upvotes: 1
Lucas Martiniano
Reputation: 261
I used find to get the first table occurrence and then iterated over all div.player-name elements using find_all.
import requests
from bs4 import BeautifulSoup
import pandas
r = requests.get("https://www.fantasypros.com/nfl/stats/qb.php")
c = r.content
soup=BeautifulSoup(c, "html.parser")
table=soup.find("div",{"class":"mobile-table double-header"})
l = []
for item in table.find_all("a",{"class","player-name"}):
d={}
d["Player"] = item.text.strip()
l.append(d)
df=pandas.DataFrame(l)
df.to_csv("Output.csv")
Upvotes: 1
Related Questions
- Python iteration over table with beautifulsoup gives only first column
- Beautifulsoup: how to iterate a table
- Failing to scrape a table with beautifulsoup
- Pandas return empty dataframe when trying to scrape table
- Scraping from beautifulsoup in Python with a for loop just returns the last result
- Python BeautifulSoup Looping Through Table Data
- BeautifulSoup - only returning first table
- BeautifulSoup html table scrape - will only return last row
- Beautifulsoup for row loop only runs once?
- Python beautifulsoup iterate over table