CODEWITHSUNDEEP
seleniumphantomjs
Ben
Ben

Reputation: 1581

Too many PhantomJS processes

I'm using the Selenium PhantomJS webdriver in a celery queue with two concurrent workers.

While I only have two concurrent workers, I often get about 50 PhantomJS processes in the activity monitor, with a huge memory footprint. These processes don't seem to close properly, even when I call the driver.quit() method.

My code looks as follows:

class Scrape:
    def __init__(self):
        self.driver = webdriver.PhantomJS()
        self.driver.implicitly_wait(10)
        self.driver.set_window_size(1280, 800)

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        self.driver.quit()

I'm using this with the with statement to ensure proper cleanup. Yet it doesn't seem to work. Any ideas on how to improve this?

Upvotes: 2

Views: 529

Answers (1)

timbre timbre
timbre timbre

Reputation: 13980

According to this issue, namely last 2 comments,

this issue will happen if phantomjs is installed with npm

So they suggest to install it using something else (apt-get / homebrew).

Upvotes: 3

Related Questions