CODEWITHSUNDEEP
pythonseleniumpython-3.xselenium-webdriverphantomjs
Maciej
Maciej

Reputation: 133

PhantomJS - connect to GhostDriver error

I am scraping websites with PhantomJS and Selenium. My problem is that after about 50 checked URLs i have an error:

selenium.common.exceptions.WebDriverException: Message: Can not connect to GhostDriver

I have not any idea how to fix it, i tried two PhantomJS versions (1.9 and 1.98) and it does not work still. Have You any idea?

Here is the code I'm executing:

def get_car_price(self, car_url): 
    browser = webdriver.PhantomJS('C:\phantomjs.exe') 
    browser.get(car_url) 
    content = browser.page_source 
    browser.quit() 

    website = lh.fromstring(content) 
    for price in website.xpath('//*[@id="js_item_' + str(self.car_id) + '"]/div[1]/div[2]/div[2]/strong[2]'): 
        return price.text

Upvotes: 4

Views: 1641

Answers (1)

alecxe
alecxe

Reputation: 474191

Instead of opening/quitting the PhantomJS browser, keep it open and reuse it. Create it globally at the startup of your script and quit when script is about to finish.

Example:

class Service(object):
    def __init__(self):
        self.browser = webdriver.PhantomJS('C:\phantomjs.exe') 

    def get_car_price(self, car_url): 
        self.browser.get(car_url) 
        content = self.browser.page_source 

        website = lh.fromstring(content) 
        for price in website.xpath('//*[@id="js_item_' + str(self.car_id) + '"]/div[1]/div[2]/div[2]/strong[2]'): 
            return price.text

    def shutdown(self):
        self.browser.quit() 

service = Service()
try:
    for url in urls:
        print(service.get_car_price(url))
finally:
    service.shutdown()

Upvotes: 3

Related Questions