Python Selenium not working inside scrapy

Question

I have a script that scrapes ebay products but in the product page i need to select a dropdown input a zipcode and click get details button so i used selenium

but for some reason selenium is not working inside scrapy selenium worked fine when i tested the code in python interpreter

here is the example

the code works fine in practice but when running scrapy

it does not select the drop down and skips over to the next request

SCRAPY OUTPUT send keys not working because dropdown is not selected when running scrapy AND UNTIL DROPDOWN FIELD IS SELECTED THE ZIPCODE FIELD WILL NOT APPEAR

here is the code

    # -*- coding: utf-8 -*-
    import scrapy, time
    from urlparse import urljoin
    from selenium import webdriver
    
    
    class EbayspiderSpider(scrapy.Spider):
        name = "ebayspider"
        #allowed_domains = ["www.ebay.com"]
        start_urls = ['http://stores.ebay.com/Complete-Tractor/Other-/_i.html?_fsub=1&_sid=1159087983&_trksid=p4634.c0.m322']
    
        def parse(self, response):
     #INITIALLY STARTING THE WEB BROWSER
            self.driver = webdriver.Chrome()
            SET_SELECTOR = '.li.nol'
            for attr in response.css(SET_SELECTOR):
                linkse = '.v4lnk ::attr(href)'
                link = attr.css(linkse).extract_first()
                yield scrapy.Request(urljoin(response.url, link), callback=self.parse_link)
    
            next_page = 'td.next a ::attr(href)'
            next_page = response.css(next_page).extract_first()
            if next_page:
                yield scrapy.Request(urljoin(response.url, next_page), callback=self.parse)
    
        def parse_link(self, response):
       
   #GETTING THE URL
            self.driver.get(response.url)
   #CLICKING on the show button to show the form
            self.driver.find_element_by_id('viTabs_1').click()
   #SELECTING THE DROPDOWN
            el = self.driver.find_element_by_id('shCountry')
            for option in el.find_elements_by_tag_name('option'):
                if option.text == 'United States':
                    option.click()
                    break
   #FILLING IN THE ZIPCODE FIELD
            el = self.driver.find_element_by_id('shZipCode')
            el.send_keys("4883")
    #CLICKING THE GET BUTTON
            self.driver.find_element_by_name("getRates").click()
            price1 = self.driver.find_element_by_xpath('//*[@id="shippingSection"]/table/tbody/tr[2]/td[1]/div').text
            price2 = self.driver.find_element_by_xpath('//*[@id="shippingSection"]/table/tbody/tr[2]/td[2]/div').text
    
            try:
                print '
	', price1, '	', price2
            except:
                pass

SO what is happening here why is it working in interpreter but not inside scrapy ?

i do not see what is wrong, please point me to the right direction

EDIT:

HYML CODE FOR THE DROPDOWN

and

code


                    
                        
                              
                            Quantity:
                            
                        
                        
                            
                            Change country:
                            
                            
                        There are 15 items available. Please enter a number less than or equal to 15.
                        Select a valid country.
                    
                    
                        
                                                                   
                            ZIP Code:
                            
                                
                            
                            Please enter a valid ZIP Code.
                            Please enter 5 or 9 numbers for the ZIP Code.

Andersson · Accepted Answer

It seems that you're trying to send value to input field that is not currently visible. When you're using Python shell to execute commands, you works with already rendered page, so required input field is visible and everything works fine, but your script works differently... You can try to wait some time until it become visible as below:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait


element = wait(self.driver, 10).until(EC.presence_of_element_located((By.ID, "shCountry")))
select = Select(element)
select.select_by_visible_text("United States")
wait(self.driver, 10).until(EC.visibility_of_element_located((By.ID, "shZipCode"))).send_keys("4883")

Python Selenium not working inside scrapy

SCRAPY OUTPUT send keys not working because dropdown is not selected when running scrapy AND UNTIL DROPDOWN FIELD IS SELECTED THE ZIPCODE FIELD WILL NOT APPEAR

HYML CODE FOR THE DROPDOWN

Answers (1)

Related Questions