CODEWITHSUNDEEP
python-2.7web-scrapingscrapy
Akshay Zankar
Akshay Zankar

Reputation: 11

Python Scrapy Click on html button

I am new to scrapy and using scrapy with python 2.7 for web automation. I want to click on a html button on a website which opens a login form. My problem is that I just want to click on a button and trasfer control to new page. I have read all similar questions but none found satisfactory because they all contain direct login or using selenium.

Below is HTML Code for button and I want to visit http://example.com/login where there is login page.

<div class="pull-left">
    <a href="http://example.com/login" class="emplink">Employers</a>

I have written code for to extract link. But how to visit that link and carry out next process. Below is My code.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = 'pro'
    url =  "http://login-page.com/"


def start_requests(self):
    yield scrapy.Request(self.url, self.parse_login)


def parse_login(self, response):
    employers = response.css("div.pull-left a::attr(href)").extract_first()
    print employers

Do I need to use "yield" Everytime and callback to new fuction for just visiting a link or there is other way to do it.

Upvotes: 0

Views: 2669

Answers (1)

Wilfredo
Wilfredo

Reputation: 1548

What you need is to yield a new request or easier make a response.follow like in the docs:

def parse_login(self, response):
    next_page = response.css("div.pull-left a::attr(href)").extract_first()
    if next_page is not None:
        yield response.follow(next_page, callback=self.next_page_parse)

About the callback, it depends basically on how easily can the page gets parsed, for example, check the general spiders section on the docs

Upvotes: 2

Related Questions