CODEWITHSUNDEEP
pythonpython-3.xxpathscrapyweb-crawler
kreesh
kreesh

Reputation: 73

Scrapy returning empty array from XPath

I'm trying to collect data on an athlete from this webpage: https://www.athletic.net/TrackAndField/Athlete.aspx?AID=7844096#!/L4. I've been able to collect the athlete's name, but am having difficulty collecting their school name using the same method. I know the school name is contained as text within a link inside the block, but it only returns an empty array.

Here is my code:

import scrapy

class AthletesSpider(scrapy.Spider):
    name = 'athletes'
    allowed_domains = ['athletic.net']
    start_urls = ['https://www.athletic.net/TrackAndField/Athlete.aspx?AID=7844096#!/L0']

    def parse(self, response):
        yield {
            'athlete_name' : response.xpath("//h2/text()").extract_first(),
            'school_name' : response.xpath("//h1/a/text()").extract_first()
        }

Am I missing something?

Upvotes: 1

Views: 113

Answers (1)

whackamadoodle3000
whackamadoodle3000

Reputation: 6748

Add a comma in your dictionary

import scrapy

class AthletesSpider(scrapy.Spider):
    name = 'athletes'
    allowed_domains = ['athletic.net']
    start_urls = ['https://www.athletic.net/TrackAndField/Athlete.aspx?AID=7844096#!/L0']

    def parse(self, response):
        yield {
            'athlete_name' : response.xpath("//h2/text()").extract_first(), <--here
            'school_name' : response.xpath("//h1/a/text()").extract_first()
        }

Upvotes: 2

Related Questions