CODEWITHSUNDEEP
pythonxpathscrapy
tigsss
tigsss

Reputation: 31

Why can't scrapy find table using XPath given the ID?

I'm sort of new to programming, but I've been following this Scrapy tutorial to scrape NFL data here: http://brutalsimplicity.github.io/2016/07/25/scrapy.html.

We're using this link (https://www.pro-football-reference.com/boxscores/201409040sea.htm) and we want the big table of Play by Play at the very bottom.

In the tutorial, it says the table id we want is "pbp_data" but I found the same table and the id is now "pbp," with this code

<table class="sortable stats_table now_sortable sliding_cols" id="pbp" data-cols-to-freeze="2">

I've never used Python or XPath before so I just followed the direction they said which was to do response.xpath('//table[@id ="pbp"]') in my scrapy shell, but for some reason, nothing is showing up when I hit enter.

Is there a problem with my understanding of XPath or am I doing something wrong with scrapy?

Thanks!

Upvotes: 0

Views: 285

Answers (1)

Chillie
Chillie

Reputation: 1485

If you use view(response) you'll notice that the data you are trying to scrape is commented:

<div class="placeholder"></div>
<!--
   <div class="table_outer_container">
      <div class="overthrow table_container" id="div_pbp">
  <table class="sortable stats_table" id="pbp" data-cols-to-freeze=2>
... -->

Will this answer help you?

Upvotes: 2

Related Questions