Xpath() method not returning results using Scrapy

Question

I am using Python.org version 2.7 64 bit on Windows Vista 64 bit. I have some Scrapy code that is attempting to parse the table at this link headed 'Wayne Rooney's Match History:"...

http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney The code I have so far is this:

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "wiki"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]

    def parse(self, response):

        for row in response.selector.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]'):
        # Is this row contains goal symbols?
            list_of_goals = row.xpath('//span[@title="Goal"]')
            if list_of_goals:
                list = str(list_of_goals)
                print remove_tags(list).encode('utf-8')     

execute(['scrapy','crawl','wiki'])

This returns all the data from the table apart from the goal data (it doesn't return assists either, but I haven't added any logic for that yet. This code is a development of the original piece of code I had which did not return goals or assists either:

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.utils.markup import remove_tags
from scrapy.cmdline import execute
import re


class MySpider(Spider):
    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        titles = response.selector.xpath("normalize-space(//title)")
        for titles in titles:
            body = response.xpath('//table[@id="player-fixture"]//tr[td[@class="tournament"]]').extract()
            body2 = "".join(body)

            print remove_tags(body2).encode('utf-8')

execute(['scrapy','crawl','goal'])

The statement in the HTML source that indicates a goal is this:

Can anyone tell me why my code listed at the top does not return goals scored with that logic? is it something to do with the fact that a ball icon is used to denote goals rather than a word?

Thanks

furas · Accepted Answer

In first version you get only and there is no text so you get empty string as result - because you remove_tags().

Adding string "GOAL" for rows with "goal icon":

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:        
    list = str(list_of_goals)
    print remove_tags(list).encode('utf-8') + "GOAL" # <-- string

or (becasue there is no text in )

list_of_goals = row.xpath('//span[@title="Goal"]')

if list_of_goals:
    print "GOAL" # <-- string

EDIT:

I made my version with Scrapy 0.22.2.

Probably we use differrent version of Scrapy because some your functions did't work for me.

I prefer css selectors then xpath - they are simpler for me.

from scrapy.spider import Spider
from scrapy.selector import Selector
from scrapy.cmdline import execute


class MySpider(Spider):

    name = "goal"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/Players/3859/Fixtures/Wayne-Rooney"]    

    def parse(self, response):
        sel = Selector(response)

        #titles = sel.xpath("normalize-space(//title)")
        #print 'titles:', titles.extract()[0]
        print
        print 'titles:', "".join( sel.css("title::text").extract() ).strip()
        print

        #rows = sel.xpath('//table[@id="player-fixture"]//tbody//tr')
        rows = sel.css('table#player-fixture tbody tr')

        for row in rows:
            #print 'date:', row.xpath('.//td[@class="date"]/text()').extract()
            #print 'result:', row.xpath('.//td[@class="result"]/a/text()').extract()
            print 'date:', "".join( row.css('.date::text').extract() ).strip()
            print 'result:', "".join( row.css('.result a::text').extract() ).strip()
            print 'team_home:', "".join( row.css('.team.home a::text').extract() ).strip()
            print 'team_away:', "".join( row.css('.team.away a::text').extract() ).strip()
            print 'info:', "".join( row.css('.info::text').extract() ).strip(), "".join( row.css('.info::attr(title)').extract() ).strip()
            print 'rating:', "".join( row.css('.rating::text').extract() ).strip()
            print 'incidents:', ", ".join( row.css('.incidents-icon::attr(title)').extract() ).strip()
            print '-'*40

#execute(['scrapy','crawl','goal'])
execute(['scrapy','runspider','main.py'])

and part of result

titles: Wayne Rooney Match History | WhoScored.com

date: 17-08-2013
result: 1 : 4
team_home: Swansea
team_away: Manchester United
info: 28' Minutes played in this match
rating: 7.26
incidents: Assist, Assist
----------------------------------------
date: 26-08-2013
result: 0 : 0
team_home: Manchester United
team_away: Chelsea
info: 90' Minutes played in this match
rating: 7.03
incidents: 
----------------------------------------
date: 14-09-2013
result: 2 : 0
team_home: Manchester United
team_away: Crystal Palace
info: 90' Minutes played in this match
rating: 8.44
incidents: Man of the Match, Goal
----------------------------------------
date: 17-09-2013
result: 4 : 2
team_home: Manchester United
team_away: Bayer Leverkusen
info: 84' Minutes played in this match
rating: 9.18
incidents: Goal, Goal, Assist
----------------------------------------
date: 22-09-2013
result: 4 : 1
team_home: Manchester City
team_away: Manchester United
info: 90' Minutes played in this match
rating: 7.17
incidents: Goal, Yellow Card
----------------------------------------
date: 25-09-2013
result: 1 : 0
team_home: Manchester United
team_away: Liverpool
info: 90' Minutes played in this match
rating: 
incidents: Man of the Match, Assist
----------------------------------------

Xpath() method not returning results using Scrapy

Answers (1)

Related Questions