CODEWITHSUNDEEP
pythonweb-scrapingscrapy
0cN
0cN

Reputation: 35

Scrapy collecting data in various steps

I'm trying to scrape data from a football website but i'm facing some difficulties.I've a series of links of two kinds:

So the bot should : login, and start crawling each of those links. This is my try:

    class fanta(CrawlSpider):

    from website.players_id import players_id  #list with all players id like p40239

    name = 'bot2'
    login_page = "https://loginpage.com/"


    #### HELPER ####
    prova = "https://websitefake.com/player/p40239"
    #This is the part that generates the 600 player profiles
    start_urls = [prova.replace("p40239", i) for i in players_id] 


    def start_requests(self): #LOGIN
        return [FormRequest(
            self.login_page,
            formdata = {'name':'aaa', 'pass':'aaa'},
            callback = self.logged_in)]

    def logged_in(self, response):
        if "Attenzione" in response.body: #Login check
            self.log("Could not log in")
        else:
            self.log("Logged In") #If logged in start scraping
            for url in self.start_urls:
                yield Request(url, callback=self.parse)

    #Scrape the data from the https://websitefake.com/player/p1234 page
    def parse(self, response):
        name = response.css("response name::text").extract()
        surname =response.css("response surname::text").extract()
        team_name =response.css("response team_name::text").extract()
        role = response.css("response role_name::text").extract()

        #Add /statistics after the "p1234" creating the url for parse_statistics
        p = re.findall("p\d+", response.url) 
        new_string = p[0] + "/statistics"
        url_replaced = re.sub("p\d+", new_string, response.url)

        #Creating the Request for https://websitefake.com/player/p1234/statistics within the items to pass through the parse_stats
        r= Request(url_replaced, callback=self.parse_stats, encoding="utf-8")
        r.meta['name'] = name
        r.meta['surname'] = surname
        r.meta['team_name'] = team_name
        r.meta['role'] = role
        yield r

    def parse_stats(self, response):
        player = Player()
        stats = response.xpath("/response/Stat").extract() #N. of stat tags
        for s in range(1,len(stats)+1):
            time = response.xpath("/response/Stat[{}]/timestamp/text()".format(s)).extract()
            player['name'] = response.meta['name']
            player['surname'] = response.meta['surname']
            player['team_name'] = response.meta['team_name']
            player['role'] = response.meta['role']
            #### DATA FROM THE STATISTICS PAGE ####
            yield player

The problem in this is that when i run the spider it keeps scraping with the parse method("player's page" and doesn't follow the callback parse_stats) so what i get is:

and not this:

I've tried everything came in my mind, probably i'm mistaked something with the yield, i don't know :S Thanks for all the future answers!

Upvotes: 1

Views: 94

Answers (1)

Frank Martin
Frank Martin

Reputation: 2594

You must not use CrawlSpider and parse together. Since you don't use any rules you probably want to use a normal Spider.

See the WARNING in the documentation

Upvotes: 1

Related Questions