CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
hudswideworld
hudswideworld

Reputation: 19

Parsing HTML from commented out section Python

I am attempting to scrape the data from the games_played_team table on this page http://www.pro-football-reference.com/teams/nwe/2016_roster.htm using Requests and BeautifulSoup

url = "http://www.pro-football-
reference.com/teams/nwe/2016_roster.htm"
r = requests.get(url)

soup = BeautifulSoup(r.content)
print soup.prettify()

returns

<!DOCTYPE html>
<html data-version="klecko-" data-root="/home/pfr/build" 
itemscope="itemscope" itemtype="http://schema.org/WebSite" lang="en" class="no-js">
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=2.0" />
  <link rel="dns-prefetch" href="https://d2p3bygnnzw9w3.cloudfront.net/req/201706021" />
  <link href="https://d2p3bygnnzw9w3.cloudfront.net" rel="preconnect" crossorigin="crossorigin" />
  <link href="http://d9kjk42l7bfqz.cloudfront.net/req/201607120" rel="preconnect" crossorigin="crossorigin" />
  <link href="https://www.google-analytics.com/" rel="preconnect" crossorigin="crossorigin" />
  <link href="https://www.googletagservices.com" rel="preconnect" crossorigin="crossorigin" />
  <script class="allowed">
   var sr_is_production = true;
function vjs_ready(e){"loading"!=document.readyState?e():document.addEventListener("DOMContentLoaded",e)}var log_performance=!1,sr_detect_operaMini=navigator.userAgent.indexOf("Opera Mini")&gt
;-1;if(sr_detect_operaMini){var el=document.querySelector("html");el.className=el.className.concat(" operamini")}var sr_detect_firefox=navigator.userAgent.indexOf("Firefox")&gt;-1;if(sr_detect
_firefox){var el=document.querySelector("html");el.className=el.className.concat(" firefox")}var sr_detect_firefoxMobile=navigator.userAgent.indexOf("Firefox")&gt;-1&amp;&amp;(navigator.userAg
ent.indexOf("Mobile")&gt;-1||navigator.userAgent.indexOf("Tablet")&gt;-1);if(sr_detect_firefoxMobile){var el=document.querySelector("html");el.className=el.className.concat(" firefox-mobile")}
var sr_detect_ie=function(){var e=window.navigator.userAgent;if(e.indexOf("Trident/7.0")&gt;0)return 11;if(e.indexOf("Trident/6.0")&gt;0)return 10;if(e.indexOf("Trident/5.0")&gt;0)return 9;for
(var t=3,n=document.createElement("div"),r=n.getElementsByTagName("i");n.innerHTML="&lt;!--[if gt IE "+ ++t+"]&gt;&lt;i&gt;&lt;/i&gt;
&lt;link rel="shortcut icon" href="https://d2p3bygnnzw9w3.cloudfront.net/req/201706021/favicons/pfr/favicon.ico"&gt;
  </script>
 </head>
</html>

When I look at the actual page source, it looks like the table I want is in a commented out block of HTML. I'm not entirely sure how to proceed and would appreciate any help

Upvotes: 0

Views: 92

Answers (1)

Pythonista
Pythonista

Reputation: 11615

Changing requests.get to requests.post is the easiest way here. Alternatively, you could just use Selenium if you wanted another easy solution.

Upvotes: 1

Related Questions