CODEWITHSUNDEEP
pythonbeautifulsoup
Justin
Justin

Reputation: 1

BeautifulSoup cannot find any select tags

This is the website I'm trying to scrape data off of: https://finance.yahoo.com/quote/AAPL/options

I'm obtaining the date values in the drop down menu under this tag:

<select class="Fz(s)" data-reactid="5"></select>

Here is the code I'm trying to run: 

from bs4 import BeautifulSoup
from urllib2 import urlopen
optionsUrl = 'https://finance.yahoo.com/quote/aapl/options'
optionsPage = urlopen(optionsUrl)
soup = BeautifulSoup(optionsPage, 'lxml')
bigDates = soup.findAll('select' , {'class' : 'Fz(s)'})

My problem is that it doesn't find anything, even though I can see the elements when I inspect it in Chrome.

How can I get the dates in the selection drop-down menu?

Upvotes: 0

Views: 324

Answers (1)

Ajax1234
Ajax1234

Reputation: 71471

The dropdown menu contents are generated dynamically. Therefore, to access the dates, you can use selenium to access the contents:

Install selenium:

pip install selenium

Install the proper driver for your browser of choice:

http://selenium-python.readthedocs.io/installation.html#drivers

from selenium import webdriver 
from bs4 import BeautifulSoup as soup
driver = webdriver.Chrome('path/to/driver/')
driver.get('https://finance.yahoo.com/quote/aapl/options')
dates = [i.text for i in soup(driver.page_source, 'lxml').find_all('option')]

Output:

[u'February 16, 2018', u'February 23, 2018', u'March 2, 2018', u'March 9, 2018', u'March 23, 2018', u'March 29, 2018', u'April 20, 2018', u'May 18, 2018', u'June 15, 2018', u'July 20, 2018', u'September 21, 2018', u'January 18, 2019', u'January 17, 2020']

Upvotes: 1

Related Questions