CODEWITHSUNDEEP
pythonseleniumselenium-webdriverpython-3.5pythonanywhere
Matt Cremeens
Matt Cremeens

Reputation: 5151

Getting selenium to work on pythonanywhere

My understanding is that pythonanywhere supports a headless Firefox browser but you need

from pyvirtualdisplay import Display

And so you can connect using

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)

And I connect just fine. However, after I start using the driver with

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)
    wb=load_workbook(r'/home/hoozits728/mutual_fund_tracker/Mutual_Fund_Tracker.xlsx')
    ws=wb.get_sheet_by_name('Tactical')

    for i in range(3, ws.max_row+1):
        if ws.cell(row=i,column=2).value is not None:
            driver.get('https://finance.yahoo.com/quote/' + ws.cell(row=i,column=2).value + '/performance?ltr=1')
            oneyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-child(2) > div > div:nth-child(5) > span:nth-child(2)').text
            threeyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(6) > span:nth-of-type(2)').text
            fiveyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(7) > span:nth-of-type(2)').text
            ws.cell(row=i,column=10).value=oneyear
            ws.cell(row=i,column=11).value=threeyear
            ws.cell(row=i,column=12).value=fiveyear

           … and so on …

I get this error after just a little while

enter image description here

For what it's worth, this code works perfectly fine on my local machine. Also, I am a paying member, so there should be no whitelist issues.

Upvotes: 7

Views: 1196

Answers (2)

Matt Cremeens
Matt Cremeens

Reputation: 5151

It has recently come to my understanding that yahoo has blocked pythonanywhere from running any web scraping scripts. I assume this is true for all AWS servers and those who use them, but I am not 100% certain of this. I hope this helps anyone who comes across this question.

https://www.pythonanywhere.com/forums/topic/5724/#id_post_52307

Upvotes: 5

Glenn
Glenn

Reputation: 5786

You're getting that error because selenium is unable to connect to the browser you created. If you're running the first chunk of code, and then the second chunk of code, then the display has been close and that would probably cause the browser to crash.

You need to run the code that uses the browser inside the with block.

There is an example on the PythonAnywhere help pages that shows how to do all this in the most reliable way.

Upvotes: 0

Related Questions